SlowGuess/NAACL_2022_OCR huggingface dataset:
Guihong Cao, and Songlin Hu. 2020. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8449-8456.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal, Swaroop Mishra, and Chitta Baral. 2019. Exploring ways to incorporate additional knowledge to improve natural language commonsense question answering. arXiv preprint arXiv:1909.08855.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018a. Improving language understanding by generative pre-training.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018b. Improving language understanding by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Hongyu Ren*, Weihua Hu*, and Jure Leskovec. 2020. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations.
Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. In Advances in Neural Information Processing Systems, volume 33, pages 19716-19726. Curran Associates, Inc.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019a. Atomic: An atlas of machine commonsense for if then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense
knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3056-3062, Online. Association for Computational Linguistics.
Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2020a. Entity context and relational paths for knowledge graph completion. arXiv preprint arXiv:2002.06757.
Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro A. Szekely, and Xiang Ren. 2020b. Connecting the dots: A knowledgeable path generator for commonsense question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4129-4140. Association for Computational Linguistics.
Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, and Michael Witbrock. 2019. Improving natural language inference using external knowledge in the science questions domain. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7208-7215.
Yubo Xie and Pearl Pu. 2021. How commonsense knowledge helps with natural language tasks: A survey of recent resources and methodologies. arXiv preprint arXiv:2108.04674.
Jun Yan, Mrigank Raman, Aaron Chan, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. 2020. Learning contextualized knowledge structures for commonsense reasoning. arXiv preprint arXiv:2010.12873.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge
graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020. Aser: A large-scale eventuality knowledge graph. In Proceedings of The Web Conference 2020, pages 201-211.
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2020. Rica: Evaluating robust inference capabilities based on commonsense axioms. arXiv preprint arXiv:2005.00782.
A Qualitative Comparison
Table 7 shows qualitative comparison between CoSe-Co and baselines on the CSQA dataset.
B Comparison with GPT-2 as backbone language model
We decided to use T5-base as a design choice as we were required to train a text-to-text model where given a sentence as input, the model has to generate the relevant path as output. Since T5-base is a text-to-text generation language model, we felt that it is a suitable choice.
Methods	100% Train
IHdev (%)	IHTest (%)
RoBERTa-large (w/o KG)	73.07 (±0.45)	68.69 (±0.56)
+ PGQA w/ GPT-2	77.53 (±0.47)	71.19 (±0.49)
+ CoSe-Co w/ GPT-2	77.90 (±0.37)	72.67 (±0.18)
+ PGQA w/ T5-base	77.56 (±0.32)	71.31 (±0.44)
+ CoSe-Co w/ T5-base	78.15 (±0.23)	72.87 (±0.31)
Table 6: Performance comparison between using T5-base and GPT-2 as backbone language model for PGQA and CoSe-Co for multi-choice QA task on CSQA dataset.
To empirically establish that improvements over PGQA are not due to using T5-base instead of GPT-2, we performed an experiment to replace T5-base with GPT-2 as the backbone language model of CoSe-Co. We train GPT-2 using the same sentence-path dataset as we used for T5-base by providing it as input the sentence followed by a [SEP] token and
adapting GPT-2 to generate the corresponding path. Additionally, we also experiment with replacing the language model in PGQA from GPT-2 to T5-base. Table 6 summarises the results obtained for multi-choice QA on CSQA where it can be seen that using GPT-2 vs T5 does not lead to noticeable changes in the performance. The test accuracy attained by CoSe-Co with T5-base is  $72.87\%$  which is almost the same as for CoSe-Co with GPT-2:  $72.67\%$ . A similar observation is seen for PGQA where using T5-base backbone gives  $71.31\%$  and using GPT-2 gives  $71.19\%$ . Further, we would like to highlight that CoSe-Co with GPT-2 backbone attains  $72.67\%$  accuracy and performs better than PGQA with GPT-2  $(71.19\%)$ .
Based on these observations, we can conclude that performance gains of CoSe-Co over PGQA are not due to using different backbone but because CoSe-Co is trained over semantically related sentence-commonsense pairs that enables it to generate contextually more relevant commonsense.
C Entity masking while training CoSe-Co
Table 8 shows the various kinds of paths obtained from CoSe-Co when trained with different values of  $p_{mask}$ , across the same original question. A number of observations can be made. First, the paths obtained from the variant which is trained without any masking ( $p_{mask} = 0.0$ ) produces inferences that enrich the overall context of certain entities in question but do not necessarily capture the inter-relation between them and thus the overall intention of the question. With the configurations that are trained with  $p_{mask} \neq 0$ , the various paths capture the overall context in an answer-oriented manner. These configurations also allow us to mask concepts in the original question such that CoSe-Co can exploit the unmasked entities to direct its generated paths in a manner that best suit the blank. This is evident from the second half of Table 8. When the interrogative element is masked in the first example, the paths are directed towards actually finding the best answer, while when 'Google maps' is replaced in the third example, the paths are clearly focused on predicting concepts related to GPS systems.
D Details of PGQA Baseline
PGQA (Wang et al., 2020b) leverages the commonsense paths generated by their path generator
Question	Predictions	Generated Paths
Where could you see an advert- isement while reading news?	PGQA	QA-GNN	Ours	PGQA	CoSe-Co
television	web page	web page	(read_news hassubevent read relatedto news atlocation television)	(spread_information_capableof advertisement atlocation web_page usedfor reading_news)
(read_news hassubevent read relatedto page)	(news这篇文章 is article atlocation web_page _receivesaction advertisement)
What can years of playing tennis lead to?	becoming tired	becoming tired	tennis elbow	(playing_tennis causes becoming_tired)	(injury_hassubevent playing_tennis hasprerequisite practice_taking_care_of_sports_equipment)
(play antonym fun usedfor playing_tennis causes tennis_elbow)	(playing_tennis hassubevent injury hasprerequisite practice_hasfirstsubevent be_better_at_new Things)
A person writes a check to a clerk, where does the clerk put them?	desk drawer	cash register	cash register	(put relatedto desk partof drawer)	(makepayments_capableof clerk desires check atlocation cashRegisters_usedfor to_pay_for_goods)
(check relatedto cash relatedto register)	(cashRegisters_usedfor clerk isa person desires clean_house hasprerequisite put_thingsinto Places)
Where could you find some large pieces of paper that are not for sale?	office supply store	cabinet	artist's studio	(large relatedto note relatedto paper relatedto office_supply)	(shredded_paper usedfor sale_hassubevent buying产品的_nohasproperty artist_studio)
(pieces relatedto part relatedto paper relatedto office_supply)	(writeletters_usedfor paper receivesaction sell_for-money atlocation store)
What do humans take in while breathing?	air	oxygen	oxygen	(humans relatedto air)	(breathing hassubevent inhale motivatedbygoal fresh_air_atlocation oxygen)
(breathing hassubevent air)	(inhaling_hassubevent breathing causes life _usedfor living_lifeshasprerequisite good_health)
(human relatedto breathing hassubevent oxygen)
Table 7: Comparison between predictions made by PGQA (Wang et al., 2020b), QA-GNN (Yasunaga et al., 2021), and CoSe-Co on a subset of CSQA's in-house test set (Talmor et al., 2019). Commonsense paths that are responsible for the corresponding predictions are also given for both the path-based models. Underlined portions represent the meaningful path sub-structures which direct the overlying model towards the correct answer.
module along with the question and candidate answer choices to perform multi-choice QA on CSQA dataset (Talmor et al., 2019). Specifically, given a question  $q$  with corresponding candidate answer choices set  $C = \{c_1, \dots, c_n\}$ , the PGQA framework generates commonsense inferences for each pair of answer choice  $c_i$  and entities extracted from  $q$ . A total of  $k$  paths corresponding to each answer choice  $c_i$  are obtained to get a resultant set of paths  $-P_{q-c_i}$ . Further, an average over the hidden representations corresponding to sequence of decoded tokens from the final layer of their path generator decoder are used as path embedding and combined as  $-H_S \in R^{k \times h_D}$  to represent the paths in  $P_{q-c_i}$ . Following this, they augment the choice into  $q$  by replacing the interrogative phrase in  $q$  with  $c_i$  to obtain  $q'$ . For instance, given the question 'Google maps and other GPS services have replaced what?', the answer choice 'atlas' is augmented into the question as: 'Google maps and other GPS services have replaced atlas.'
To embed the augmented question and corresponding answer choice, they use a pre-trained LM encoder  $E$  (such as RoBERTa (Liu et al., 2019)) to embed the query - '[CLS]  $q'$  [SEP]  $c_i$ ' corresponding to  $c_i$ . The representation corresponding
to [CLS] token is extracted from the final hidden layer as  $h_{US} \in R^{h_E}$ . In order to leverage relevant knowledge from the generated commonsense inferences, the question and choice embeddings are used to attend over generated paths as:
$$
\alpha_ {p} = S o f t m a x (t a n h (H _ {S} W ^ {A}) h _ {U S})
$$
$$
h _ {S ^ {\prime}} = \sum_ {h \in H _ {S}} \alpha_ {p} ^ {h} \cdot h
$$
where,  $W^{A} \in R^{h_{D} \times h_{E}}$ ,  $\alpha_{p} \in R^{k}$  and  $h_{S'} \in R^{h_{D}}$ . Finally, a linear layer is applied over the concatenation of  $\{h_{US}, h_{S'}\}$  to project it as a scalar. A softmax is taken over concatenation of scalars obtained corresponding to each answer choice to obtain their likelihood followed by cross entropy loss for training.
E.1 Relation Heuristics
As mentioned in §3.1, we employ heuristics on the basis of contained relations to perform filtering of ConceptNet paths. Particularly, we use the following rules:
Input	CoSe-Co Variant	CoSe-Co Outputs
Google maps and other GPS services have replaced what?	pmask=0.0	(orienteering_usedfor map_isanavigation_hascontext GPS_product gsrc)(satellite_gear-hascontextmaps capableof help_find_places_usedfor map_store_in_city)(similar_to_gps_hasproperty google_hascontextmaps capableof show_liveplaces)(geospatial_isa navigation_usedfor google_hascontextmaps capableof show_peopleplaces)
Google maps and other GPS services have replaced what?	pmask=1.0	(gps hascontextnavigation_usedfor satellite atlocation orbital_stations_in_space)(satellitejahstation_navigationscompasscapableofpoint_north_hassubeventdrivingcar)(navigation_usedfor satellite_isaGPS-hascontextnavigation_entailscompetition)(location_where_goGPCarlydefinedasnavigation_usedfor satellite_hascontextgps)
Google maps and other GPS services have replaced what?	pmask=0.5	(compass_isa google_maps usedfor navigation_hascontextgps isa navigating_map)(location_where_go_camping_isa location_usedfor map_product google_maps)(satellite_gear_isa GPS usedfor navigation_hascontextgps isa navigating_map)(navigation_maps_usedfor map_isa navigation_map hascontext navigation)
Google maps and other GPS services have replaced what?	pmask=0.33	(orienteering_usedfor maps_isa google_maps_hasprerequisitelookingsthrough_the telescope)(location_where_go_shopping_definedaswhere_go_shopping_usedfor map)(navigation_maps_isamaps_usedfor satellite locatednearplanet)(satellitejahstation_navigationsusedfor maps_hascontext google_maps capableof showlocations)
Google maps and other GPS services have replaced [MASK].	pmask=0.33	(gps hascontextmaps_usedfor satellite locatednearplanet)(navigation_maps isa navigation_usedfor compass capableof point_north_handle)(satellitejahstation_usedfor compass capableof point_north_or_south_hemispheres)(location_where_go_if_near_beach_definedasmap usedfor navigation_mannerof sport)
Google maps and other GPS services have [MASK] what?	pmask=0.33	(orienteering_usedfor map_isa google_maps_hascontextgps)(location_where_go_if你需要_to_definedaslocation_isa map usedfor information)(located_in_latin_america_receivesaction_israel(Language_latin_america)(navigation_maps usedfor find_place_hasprerequisitego_to_market)(satellitejahstation_usedfor maps capableof showlocations_andRoutes)
[MASK] and other GPS services have replaced what?	pmask=0.33	(transportation_system_isa GPS hascontextastronomy_fieldedmond_halley)(location_where_go_if_in'accident_usedfor map_atlocation GPS_systems)(radio_freqency Messaging_isa GPS hasproperty useful)(receiver partof radio_isagpshascontextnavigation_usedfor compass)
Table 8: Examples of commonsense inferences obtained for different input forms of the same question from CoSe-Co when trained with different values of  $p_{mask}$ . Potential answers which are observed in a path are highlighted, while context-enriching concepts are coloured.
1. We discard any path that uses the same two relations to connect any three neighbouring entities occurring in it. That is, for any subpath  $\{e_i,r_i,e_{i + 1},r_{i + 1},e_{i + 2}\}$  in a given path  $p$ , we only consider  $p$  as a part of our dataset if  $r_i\neq r_{i + 1}$ .
2. Following (Wang et al., 2020b), we do not consider paths that contain any relations from the set {HasContext, RelatedTo, Synonym, Antonym, DerivedFrom, FormOf, EtymologicallyDerivedFrom, EtymologicallyRelatedTo}. We observed that entities connected through these relations were often largely dissimilar and thus not useful for our case.
E.2 Multi-Choice QA
In §4.3, we discuss commonsense question answering task where we use framework developed by Wang et al. (2020b) and just replace the commonsense knowledge used by them with the paths generated by CoSe-Co. We use the same hyperparameters as used by them and mention them here
for reference. The model is trained on a batch size of 16, dropout of 0.1 for 15 epochs. A learning rate of 2e-6 is used for encoder LM (Roberta-large) used for embedding question and choice context and an lr of 1e-3 is used for remaining path attention and classification layer parameters. We perform the evaluation on CSQA (Talmor et al., 2019) dataset downloaded from here. The train split comprises of 8,500, dev split contains 1,221 and in-house test split contains 1,241 samples.
E.3 OpenCSR
In this section, we discuss the implementation details used for OpenCSR in §4.4. The dataset has been downloaded from here. The training splits of ARC, QASC, and OBQA datasets comprises of 5355, 6883, and 4199 samples respectively while the development split comprises of 562, 731, and 463 samples respectively. The test set is hidden and authors who proposed the task with reformulated dataset are yet to set up a leaderboard on the hidden test set. They run their proposed model DrFact (which is based on BERT-base and is the current
state-of-the-art on this task) on a single seed which takes about  $\sim 2-3$  days to train one model on a given dataset. While fine-tuning T5-base (with and without CoSe-Co knowledge), we train the model for 5 epochs with a learning rate of 5e-4, weight decay of 0.01 and batch size 8 using AdamW optimizer (Loshchilov and Hutter, 2017).
E.4 Paraphrase Generation
For paraphrase generation on MRPC (Dolan and Brockett, 2005) dataset, we fine-tune T5-base (with and without CoSe-Co knowledge) at a learning rate of 5e-4 for 5 epochs with weight decay of 0.01 and 4 gradient accumulation steps using AdamW (Loshchilov and Hutter, 2017) optimizer. The training set of MRPC comprises of 2,661 paraphrases while the test set comprises of 1,088 paraphrases. The dataset has been downloaded from here.
F Further Analysis of Generated Paths
- Correctness of Novel Triples: Since there is no ground truth to check the correctness of triple comprising of novel entities, we attempt to evaluate them by leveraging a commonsense knowledge base completion model - Bilinear AVG (Li et al., 2016) which has been shown to achieve an accuracy of  $92.5\%$  on knowledge completion task and is also used to score triples. We extract triples comprising of at least one novel entity from the paths generated by CoSe-Co for the test split of sentence-path dataset and provide the triple to Bilinear AVG to obtain a score. The average score over all the triples is 0.414 (on a scale of 0 to 1).
Further, we perform KG completion (predicting tail entity given head entity and relation of a KG triple) using CoSe-Co since it generates paths which essentially comprise of triples. We compare the performance with COMET (Bosselut et al., 2019). We consider test split of sentence-path dataset comprising of 11,264 paths and extract triples. We filter out triples appearing in training paths of CoSe-Co and train set triples of COMET yielding 717 test triples in total. CoSe-Co achieves an accuracy of  $24.12\%$  which is significantly better than COMET which provides accuracy of  $9.76\%$ . To perform comparison
with COMET (Bosselut et al., 2019) we take their code and pre-trained model from here.
- In Figure 3(b), greedy decoding cannot be compared for diversity with other methods since it generates only a single unique path.
- Since generated paths diversity estimates can be affected by path length, we measure the standard deviation of the number of entities in paths generated corresponding to test split sentences and found it to be 0.76 which shows that variance in the lengths of generated paths is very low (<1) and hence, the diversity of 0.43 (on a scale 0 to 1) attained by CoSe-Co is not due to length bias.
G Ethics statement