SlowGuess/NAACL_2022_OCR huggingface dataset:
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S Yu. 2021. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2277.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211-227, Online. Association for Computational Linguistics.
Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. 2019. Generating diverse translations with sentence codes. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1823-1827, Florence, Italy. Association for Computational Linguistics.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888-2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575. IEEE Computer Society.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense generation. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3056–3062, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020. ASER: A large-scale eventuality knowledge graph. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 201-211. ACM / IW3C2.
Chao Zhao, Marilyn Walker, and Snigdha Chaturvedi. 2020. Bridging the structural gap between encoding and decoding for data-to-text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2481-2491, Online. Association for Computational Linguistics.
Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. 2021. Pre-training text-to-text transformers for concept-centric common sense. In International Conference on Learning Representations.
The BART and T5 models are implemented using the Transformers library (Wolf et al., 2020). We fine-tune each model on the training data of COMMONGEN with Adam (Kingma and Ba, 2015). We set the learning rate as 2e-5 and adopt early stopping based on the loss of development set. The batch size of training is 64.
B Human Evaluation Details
We randomly select 100 test instances that had 5 concepts as input, since they are more challenging than those with fewer concepts. The three measures we used are 1) Rationality: whether or not the sentence is in accordance with commonsense; 2) Fluency: whether or not the sentence is fluent and has no grammatical errors; and 3) Succinctness: whether or not the sentence contains redundant words or repeated information.
The pairwise scores of those measures are calculated as follows. When comparing a certain approach to Planned-BART  $\mathbf{R_{\mathrm{ank}}}$ , we report the percentage of instances that were judged to be better/worse/same than those of Planned-BART  $\mathbf{R_{\mathrm{ank}}}$ , yielding a score ranging from -1 (unanimously worse) to 1 (unanimously better). For example, when evaluating the rationality scores, Unordered-BART  $\mathbf{V}_{\mathrm{V}_i}$  performs better/worse/same than Planned-BART  $\mathbf{R_{\mathrm{ank}}}$  for  $27\% /65\% /8\%$  of the instances, yielding a pairwise score as  $0.27 - 0.65 = -0.38$ .
C Permutation Invariance
To figure out to what extent Unordered-BART is permutation-invariant, we conduct the following
analysis. For each test instance, we feed all different input permutations to the Unordered-BART to obtain the corresponding output sentences. We measure the invariance of the outputs by computing the percentage of the most frequent output. If the concept order does not affect the output, all different permutations will lead to identical outputs and the percentage will be  $100\%$ . If half of the permutations obtain the same outputs, the percentage will be  $50\%$ . We measure the input invariance at the following two levels.
Sentence-level invariance For various permutations of a specific concept-set, if the percentage of the most frequent sentence is greater than an invariance threshold  $\alpha$ , we regard the model as input-invariant to that instance at the sentence level.
Skeleton-level invariance The sentence-level invariance requires the output sentences to be identical, which is strict and does not consider minor lexical differences (e.g., the difference in function words or modifiers). Therefore, we also report the skeleton-level invariance by measuring the percentage of the most frequent skeleton (mode skeleton). It reflects whether or not the model output will follow a certain order under different permutations, which is a more forgivable invariance measure compared to its sentence-level counterpart.
Figure 4: The histogram of COMMONGEN test set w.r.t. the percentage of most frequent sentences (left) and skeletons (right), respectively. We also show the cumulative distribution in blue lines. When  $\alpha = 0.9$ ,  $37\%$  and  $61\%$  of the test instances are invariant at the sentence and skeleton-level, respectively.
Figure 4 shows the distribution of COMMON-GEN test set w.r.t. the percentage of most frequent sentences (left) and skeletons (right). When setting  $\alpha = 0.9$ ,  $37\%$  of the test instances are invariant at the sentence level, and  $61\%$  are invariant at the skeleton level. This indicates that for  $61\%$  of the permutations, Unordered-BART can organize the concepts in one particular order irrespective of the input order.
Relation	Head	UAS	\( I_{lh} \)
v-dobj-n	10-7	83.97	13
v-prep-adp-pobj-n	10-7	82.61	13
n-nsubj-v	11-12	87.62	3
n-prep-adp-pobj-n	10-8	60.55	15
v-advcl-v	4-11	85.37	4
v-conj-v	10-1	83.07	2
n-nsubj-v-dobj-n	6-16	62.34	40
v-xcomp-v	8-15	91.32	19
n-conj-n	10-0	88.75	56
n-comp-n	1-8	83.31	24
Table 4: The functional heads for each relation, as well as the corresponding UAS and importance rank.
D Analysis of Local Attention
In addition to the analysis of global attention distribution, we also investigate local attention patterns, i.e., the attention weights between concept tokens. Previous works show that some attention heads can reflect certain aspects of syntactic and semantic relations between words (Clark et al., 2019; Htut et al., 2019). We want to investigate if the heads can track commonsense relations between concepts.
For this purpose, we first build gold relations between the concepts that capture commonsense knowledge. One option is to utilize ConceptNet relations between concepts (Lin et al., 2020). However, these relations connect only two concepts at a time disregarding the context information from other concepts. Consider {throw, catch, dog, frisbee} as an example. "Dog" may be "caught" but this relation is less plausible in this case because of the existence of "frisbee". When considering this context, humans provide references such as "The dog catches the frisbee when the boy throws it."
Another option is to use the dependency relations between words in the reference sentences, which can capture the commonly occurring relations between concepts while incorporating the context. For example, the relation "catch  $\xrightarrow{\text{obj}}$  frisbee" captures the commonsense that frisbee is often caught, Similar ideas are also adopted in Zhang et al. (2020). In particular, we extract the one-hop and two-hop dependency relations of all concept pairs from the references, and only keep the relations that appear in two or more references.
For attention probing, we use attention weights between input tokens to reflect the strength of their associations. Given two concepts  $c_{i}$  and  $c_{j,j > i}$ , we regard them as strongly associated under the attention head  $(l,h)$  if the attention weight  $\alpha_{ij}^{l,k}$  is the highest among the scores from all the other concepts  $c_{k\setminus \{i,j\}}$  to  $c_{j}$ .
Model	Output	Skeleton
Unordered-BART	A crowd of people watch and dance to the music.	crowd watch dance music
Planned-BART	A crowd of people are dancing to music while others watch.A man plays music and watches the crowd dance.A group of people dance to music as a crowd watches.A man watches a crowd of people dancing to music.	crowd dance music watchmusic watch crowd dance dance music crowd watchwatch crowd dance music
Human	The crowd likes to watch her dance to the music.The crowd watched the dance, and listed to the music.I watched as the crowd dance to the music.A person dancing to the music as a crowd of people watch.	crowd watch dance musiccrowd watch dance musicwatch crowd dance musicdance music crowd watch
Table 5: Sample texts generated by Unordered-BART, Planned-BART, and humans for the concept set {dance, music, crowd, watch}. The diversity of Planned-BART is more close to human generation.
Figure 5: UAS of commonsense relations from three BART models via attention probing. The performance of fine-tuned Unordered-BART  $>$  pre-trained Frozen-BART  $>$  randomly-initialized BART among all of the relations.
We choose the 10 most common dependency relations from the test set and report the Unlabeled Attachment Score (UAS) of attention probing in Figure 5. We also list the UAS of a randomly initialized BART and a pre-trained BART without fine-tuning for comparison.
Results in Figure 5 show that for some frequent and simple relations such as "v-dobj-n" and "n-subj-v", there is at least one attention head that tends to track them regardless of the differences in the permutation orders. For example, attention head "Layer-10 Head-7" tracks the "v-dobj-n" relation with a UAS of  $83.9\%$ . The comparison among the three models shows that the pre-trained BART already exceeds the randomly initialized model in tracking commonsense relations between words, and fine-tuning further strengthens those relations.
Figure 6: The cross-attention matrix of two permutations of the same concept-set produced by Unordered-BART. It's difficult for Unordered-BART to learn the optimal order of attention.
To demonstrate that these functional heads are important for generation, we use the expected sensitivity (Michel et al., 2019) of the model to each head to evaluate the head importance as
$$
I _ {l h} = \mathbb {E} _ {x \sim X} \left| \frac {\partial \mathcal {L} (x)}{\partial \xi_ {l h}} \right| \tag {2}
$$
where  $\mathcal{L}(x)$  is the loss of generation and  $\xi_{lh}$  is the mask variable for head  $l - h$  with values in  $\{0,1\}$ . The general idea is that the value change of important heads can have a larger impact on the model loss. Results are shown in Table 4. For most relations, the corresponding functional heads also have a high rank of importance. This consistency indicates that the model heavily relies on these heads when generating texts, and further demonstrates that the finetuned BART can capture the commonsense between concepts for generation.
E Impact on Repetition
The repetition of the unordered BART is caused by the order-agnostic property of its input. Since the input concepts are unordered, the decoder cannot pay attention to the input in a monotonic way (from left to right) during decoding, which may mislead the decoder to attend to the concepts that have been
previously generated. For example, on the left of Figure 6, the decoder attends to "tea" and "glass" twice during decoding, which achieves the local coherence but causes the global repetition issue and unnatural text. However, when modifying the input in another order, as shown in the right of Figure 6, the repetitive and unnatural expressions disappear. It indicates that the BART decoder has difficulty ordering the input globally, and providing a well-ordered plan as input can alleviate this issue. On the contrary, in Planned-BART, the decoder can assign attention weights monotonically to the input, and therefore reduce the repetition caused by re-attending the previous concepts.
F Impact on Diversity
Table 5 provides an example with the outputs created by both models and humans. Unordered-BART can create only one output due to the permutation invariance. Also, the object of watch is missing in its output. On the other hand, similar to the human-written output, the output of Planned-BART is more natural and diverse.