SlowGuess/NAACL_2021_OCR huggingface dataset:
We further run experiments on the PTB dataset, which is a benchmark for the word ordering task. The results are shown in Table 3. We can observe
that various neural models outperform the traditional N-GRAM model and RNNLM. In these neural models, the results are consistent with those of Wikipedia.
5.5 Discussions
The impact of the number of noisy/missing words. To better understand the robustness of different models under the noise scenario and the missing scenario, we show how the performance changes as the number of noise or missing words changes in Figure 1. As seen, approaches based on self-attention always outperform other approaches in both scenarios, especially more significantly in the noise scenario. Besides, the performance of all models drops as the increases of the number of missing words or noisy words, but more sharply for the missing scenario. The results imply that: 1) In the bag-to-sentence transformation problems, the capability of neural models to resist noisy words is better than the capability to resist missing words; 2) It is still challenging for neural models to handle the bags where some information is missed.
The impact of bag size. We further study how the size of the input bag influences the performance of different models. Figure 2 illustrates how the
log p(y|x)
Case-1	Input: . largest animals bears the they in the land also are only native taiwan and Reference: they are also the largest land animals and the only native bears in taiwan.
Beam-1: they are also the only native animals and the largest land bears in taiwan.	-0.2602
Beam-2: they are also the only native land animals and the largest bears in taiwan.	-0.2708
Beam-3: they are also the largest land animals and the only native bears in taiwan.	-0.3183
Case-2	Input: a, engineering there . time mechanical chairman long he for served , as of Reference: there he served , for a long time , as chairman of mechanical engineering.
Beam-1: there , he served as chairman of mechanical engineering , for a long time.	-0.0797
Beam-2: there , he served for a long time , as chairman of mechanical engineering.	-0.0882
Beam-3: for a long time , there , he served as chairman of mechanical engineering.	-0.2041
Case-3	Input: their cuddy again however . sends interrupts and , exchange away ali Reference: however , cuddy interrupts their exchange again and sends ali away.
Beam-1: however , ali interrupts their exchange again and sends cuddy away.	-0.1829
Beam-2: however , cuddy interrupts their exchange again and sends ali away.	-0.2116
Beam-3: however , cuddy interrupts their exchange and sends ali away again.	-0.2187
Table 4: Generation examples of 3 different results via beam search in our AttM under normal scenario. We show the log generation probability for each beam candidate in the last column.
performance of AttM changes with respect to bags with different numbers of words in the normal scenario, where we bin test examples into buckets. We observe a similar trend for all models: they first remain stable when the bag size less than 8, and then decrease monotonically when the bag size keeps increasing. The reason might be that when only a few words are available in input bag, the model can well capture the meaning of the whole sentence, but when the bag becomes large enough, the semantic combination of words will become more complicated and the meaning of target sentence will be hard to be grasped. Besides, self-attention-based models always achieve the best performance, which is consistent with the result in Table 2.
Multiple plausible outputs. Actually, for the bag-to-sequence task when applied to language, a bag of words sometimes may correspond to multiple reasonable and grammatical outputs. Such a phenomenon is similar to response generation in dialog systems, where several responses can be reasonable. Table 4 shows the three generated results of AttM (the most strong model) through beam search. We can notice that all generated sentences are grammatical and reasonable. In case-1, the objects "animals" and "land bears" are exchangeable in terms of syntax; both "native" and "largest" can describe these objects. Our model prefers "the only native animals" and "the largest land bears". Since our model is a conditional language model learned from the training corpus, and the decoder reconstructs a sentence conditioned on the representation of the input bag of words. The joint probability of sentence-1 is larger than sentence-2. In case-2, "for a long time" and "there" are adverbials, and are
Figure 3: Performance of the neural model and human in terms of different scenarios.
position variable. However, the meaning of all generated sentences remains the same. In case-3, both "ali" and "cuddy" are names, thus they are undistinguishable in this situation. Our model assigns a higher probability to "ali interrupts their exchange again and sends cuddy away". Despite the lack of order information, neural models can still organize all possible sentences through beam search.
Neural Models Vs. Human. We are also curious about the ability of humans to organize sentences from a bag of words. We first binned the test set of the normal scenario into four buckets according to the size of the input bag, and then randomly selected 40 samples from each bucket. We invited humans to organize the target sentence regarding the input bag using crowd-sourcing. Each bag was randomly presented to 3 judges and we retain the answer with the highest BLEU score. Figure 3(a) illustrates the BLEU score of humans and the most competitive model AttM across different bag sizes. We observe that both the performance of humans and AttM become worse with the increase of the bag size, which is consistent with the result in Figure 2. Besides, AttM always shows better performance than human, but the performance gap
Annotated Types	Ratio
Synonymous (30%)	Exactly generated	16%
Two adverbials are exchanged	5%
Two coordinate clauses are exchanged	7%
Other reasons	2%
Non-synonymous (57%)	The subject and object are exchanged	5%
The logic is unreasonable.	33%
Other reasons	19%
Table 5: The statistical analysis of 100 randomly selected samples for AttM in normal scenario. We only show the result of the grammatical part, and the proportion of ungrammatical samples is  $13\%$ .
becomes smaller as the bag size decreases. This result indicates that humans are better at recognizing small bags than large bags.
Besides, we also study how noisy words and missing words impact the performance of humans and neural models. Based on the above test set randomly selected from the normal scenario, we randomly introduced 1 or 2 noisy words to the source bag denoting as noise-1, noise-2 respectively, and randomly removed 1 or 2 words from the source bag, denoting as missing-1, missing-2 respectively. We also invited humans to organize a sentence regarding the input bag using crowdsourcing. Figure 3(b) presents the results of each test set. We summarize our observations as follows: (1) Both the performance of human and AttM get worse when noisy words are introduced or some words are removed; (2) Compared with neural models, humans are more robust to noisy words and missing word in sentence organization; (3) The performance AttM is significantly better than humans, but becomes comparable with humans when 2 words are randomly removed from the input bag. The results imply that humans have a more strong background knowledge of language to guess the meaning of the target sentence and complete the cloze test.
Error analysis. To further analyze the quality of the generated sentence and in which case our model fails to recover the original sentence, we invite four educated annotators to judge the quality of 100 randomly sampled sentences<sup>5</sup> generated by AttM. Annotators were asked to judge whether a generated sentence is grammatical and the meaning of a generated sentence is the same as the ground truth. We can find that  $87\%$  of generated sentences are grammatical and  $30\%$  of sentences share the same meaning with the ground-truth. Among those grammat-
Figure 4: Visualization of attention weight in the 5-th head (left) and 6-th head (right) in self-attention. The target sentence is "the film was written and directed by larry cohen." "lysander" is a noisy word.
ical and synonymous samples,  $46.7\%$  (14/30) of sentences are not exactly the same with the ground truth in syntax. There are two main types of paraphrase: the position of adverbials is exchanged or the position of coordinate clauses is exchanged. Among those grammatical and non-synonymous samples, the logic of the majority sentences is unreasonable due to the position exchange of adverbials or coordinate clauses, and unreasonable combinations of semantic units. Besides, the semantics of some sentences are changed because of the exchange of the subject and the object.
Attention visualization. Figure 4 shows the visualization of attention weights of different heads in the 5-th block from the self-attention layer. We can observe that self-attention can capture combinatorial relations between the input words from different semantic aspects. For instance, "cohen" shows a strong correlation with "larry" in both heatmaps since "Larry Cohen" is the name of a famous director. Moreover, both "was" and "by" attend to "directed" and "written", composing the phrase "was written (directed) by". Such combinatorial relations can make the word representation more informative, which contributes to the representation learning of the bag of words. Additionally, we observe that almost all words but itself demonstrate weak correlations with the noisy word "lysander" in both heatmaps, demonstrating the advantages of our model to tolerate noisy words.
In this paper, we present an empirical study to investigate the ability of neural models to organize sentences from a bag of words under three typical scenarios. We conclude our discussion with the following findings:
- Self-attention is effective to capture the se
5We randomly select samples with a bag size greater than or equal to 10 since they contain more error cases.
mantic dependencies between words in the input bag and shows competitive performance in bag-to-sentence transformation.
- Neural models have a certain degree of capability to organize a sentence from a bag of words. However, it is still challenging for neural models to handle large bags or the bags where some information is missing.
- Compared with humans, neural models show a better capability to organize sentences from a bag of words, especially in terms of large bags. However, the performance of humans is more robust to noisy words or missing words than neural models.
We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Science Foundation of China (NSFC No. 61876196) and the National Key Research and Development Program of China (No. 2020AAA0105200). Rui Yan is supported as a Young Fellow of Beijing Institute of Artificial Intelligence (BAAI).
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Adria De Gispert, Marcus Tomalin, and Bill Byrne. 2014. Word ordering with phrase-based grammars. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 259-268.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159.
Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dolkar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473-1482.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pages 1243-1252.
Eva Hasler, Felix Stahlberg, Marcus Tomalin, Adri de Gispert, and Bill Byrne. 2017. A comparison of neural models for word ordering. In Proceedings of the 10th International Conference on Natural Language Generation, pages 208-212.
Jing He and Hongyu Liang. 2011. Word-reordering for statistical machine translation using trigram language model. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1288-1293.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Maximilian Ilse, Jakub Tomczak, and Max Welling. 2018. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136. PMLR.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches out: Proc. ACL-04 Workshop, pages 74-81.
Jiangming Liu and Yue Zhang. 2015. An empirical comparison between n-gram and syntactic language models for word ordering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 369-378.
Yijia Liu, Yue Zhang, Wanxiang Che, and Bing Qin. 2015. Transition-based syntactic linearization. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 113-122.
Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan AlRegib, and Hans Peter Graf. 2018. At- tend and interact: Higher-order object interactions for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6790-6800.
Lidia Mangu, Eric Brill, and Andreas Stolcke. 2000. Finding consensus in speech recognition: word error minimization and other applications of confusion networks. Computer Speech & Language, 14(4):373-400.
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, volume 2, page 3.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660.
Allen Schmaltz, Alexander M Rush, and Stuart M Shieber. 2016. Word ordering without syntax. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2319-2324.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 16, pages 3776-3784.
Cyrus Shaoul. 2010. The westbury lab wikipedia corpus. Edmonton, AB: University of Alberta.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28, pages 2440-2448.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2015. Order matters: Sequence to sequence for sets. In International Conference on Learning Representations.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189-198.
Songhua Xu, Shaohui Yang, and Francis Chi-Moon Lau. 2010. Keyword extraction and headline generation using novel word features. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1461-1466.
Bo Yang, Sen Wang, Andrew Markham, and Niki Trigoni. 2020. Robust attentional aggregation of deep feature sets for multi-view 3d reconstruction. International Journal of Computer Vision, 128(1):53-73.
Yue Zhang, Graeme Blackwood, and Stephen Clark. 2012. Syntax-based word ordering incorporating a large-scale language model. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 736-746. Association for Computational Linguistics.
Yue Zhang and Stephen Clark. 2011. Syntax-based grammaticality improvement using ccg and guided search. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1147-1157. Association for Computational Linguistics.