SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1: Commonsense paths generated by CoSe-Co for questions in CommonsenseQA data. Potential answers observed in path itself are highlighted, context-enriching concepts are coloured.
and ground truth paths (for a given test sample) as one uni-gram followed by determining BLEU score (Papineni et al., 2002) between them. To estimate Diversity, we extract top- $k = 5$  paths for each sentence, consider each pair combination amongst them and estimate fractional overlap (intersection over union of set of path entities) between them. Compliment of overlap  $(1 - overlap)$  followed by mean over entire test split conveys how diverse paths are. Figure 3 shows corresponding results. It is observed that paths generated using nucleus sampling are diverse but lack relevance, while an opposite trend is observed for top-k sampling. Diverse-path search provides best balance between relevance (0.436) and diversity (0.43). We estimate Novelty as a fraction of total entities in a generated path that are not present in any training path followed by averaging over test split. CoSe-Co attains a novelty of  $23.28\%$  which shows that good fraction of entities in generated path are novel. Further discussion on the quantitative analysis of generated paths can be found in appendix F. Table 1 shows a few examples of generated paths. CoSe-Co generates paths contextually relevant to question in addition to inferring novel entities.
4.3 Multi-Choice Question Answering
We perform multiple choice question answering on the CSQA dataset (Talmor et al., 2019). Here, a
question is given with 5 answer choices and the model has to predict the correct one. As an example, consider a question 'Where could you see an advertisement while reading news?' with answer choices 'television, bus, email, web page, and la villa'. One of the prior works for this task - PGQA (Wang et al., 2020b), comprises of a knowledge module which generates commonsense and a QA module which identifies correct choice using this knowledge (see appendix D for details). Since our aim is not to design an improved QA module but a better commonsense generator, for fair comparison with PGQA, we use their QA module with CoSe-Co. The QA module embeds the question + choices using RoBERTa (Liu et al., 2019) and uses the CLS token output to perform attention over path embeddings generated using the commonsense module. The output of attention module together with embedding of question and answer choices is used to predict the correct answer.
Table 2 shows results on CSQA which are usually averaged over 5 runs on this benchmark. We compare against several baselines broadly classified into ones using static KG such as MHGRN (Feng et al., 2020), QA-GNN (Yasunaga et al., 2021) etc. and others which train a dynamic path generator (PGQA) (Wang et al., 2020b) as commonsense module. We also compare with T5-base since it is backbone LM for CoSe-Co.
Methods	20% Train
IHtest (%)	60% Train
IHtest (%)	100% Train
IHdev (%)	IHtest (%)
T5-base (w/o KG) (Raffel et al., 2019)	-	-	61.88 (±0.08)	57.34 (±0.21)
RoBERTa-large (w/o KG)	46.25 (±0.63)	52.30 (±0.16)	73.07 (±0.45)	68.69 (±0.56)
+ RGCN (Schlichtkrull et al., 2018)	45.12 (±0.69)	54.71 (±0.37)	72.69 (±0.19)	68.41 (±0.66)
+ GconAttn (Wang et al., 2019)	47.95 (±0.11)	54.96 (±0.69)	72.61( ±0.39)	68.59 (±0.96)
+ KagNet (Lin et al., 2019)	-	-	73.47 (±0.22)	69.01 (±0.76)
+ RN (Santoro et al., 2017)	45.12 (±0.69)	54.23 (±0.28)	74.57 (±0.91)	69.08 (±0.21)
+ MHGRN (Feng et al., 2020)	-	-	74.45 (±0.10)	71.11 (±0.81)
+ PGQA (Wang et al., 2020b)	58.25 (±0.43)	69.66 (±0.97)	77.53 (±0.47)q	71.19 (±0.49)
+ QA-GNN (Yasunaga et al., 2021)	59.08 (±1.25)	68.70 (±0.62)	75.54 (±0.42)	72.29 (±0.43)p
+ CoSe-Co (Ours)	61.20 (±0.19)p,q	70.23 (±0.40)q	78.15 (±0.23)p,q	72.87 (±0.31)p,q
Table 2: Performance comparison on in-house dev (IHdev) and test (IHtest) split of CommonsenseQA dataset (Talmor et al., 2019). All scores are averaged across 5 runs. First row depicts amount of training data used. The second-best number for each column is underlined while best is in bold. Superscripts  $p$  and  $q$  denote statistically significant differences ( $p$ -value of 0.05) in comparison to two of our baselines- PGQA and QA-GNN, respectively.
When using entire training data, we observe that CoSe-Co performs better than all baselines<sup>2</sup> on test set. We outperform PGQA with a gain of  $1.68\%$  in accuracy on test split signifying the relevance and applicability of inferences generated by CoSe-Co. CoSe-Co performs better than QA-GNN (Yasunaga et al., 2021) also particularly in low training data regimes with performance gains of  $\sim 2\%$  (and  $\sim 3\%$  over PGQA) showing that while QA-GNN is more sensitive towards amount of training data used, CoSe-Co is more robust and helps in generalizing better. Qualitatively, consider the question - 'Where could you see an advertisement while reading news?' PGQA generates the path - 'read_news hassubevent read relatedto news atlocation television' ignoring the context that advertisement is being seen along with reading news and ends up predicting television as answer which is wrong. While CoSe-Co generates - 'spread_information_capableof advertisement atlocation web_page usedfor reading_news'. Here it can be seen that CoSe-Co identifies that seeing the advertisement and reading news is happening together and generates path accordingly to relate them with 'web page' which is the correct answer. We also conduct a thorough qualitative comparison (appendix A) where we observe that evaluators find CoSe-Co paths to be significantly more contextually relevant than PGQA.
We conduct a human study wherein we pre
sented evaluators with questions from CSQA dataset with corresponding commonsense paths generated by CoSe-Co and PGQA in an anonymized manner to compare the generative commonsense methods. We asked them to compare the paths based on their contextual relevance with the complete sentence and classify them into one of three categories - 1) 'CoSe-Co is better than PGQA', 2) 'PGQA is better than CoSe-Co', 3) 'Both are of the similar quality'. A total of 150 questions samples were randomly sampled from the test set and presented to 6 evaluators (25 samples each). Following are our observations:
- Number of samples where CoSe-Co is better: 62 (41.33% of 150 samples)
Number of samples where PGQA is better: 38 (25.33% of 150 samples)
- Number of samples where both are of similar quality: 50 (33.33% of 150 samples)
This shows that commonsense generated by CoSe-Co is found to be more relevant in human evaluation. Also, if we exclude neutral samples and consider the 100 samples where the commonsense paths generated by one of either approach is found to be better, CoSe-Co's paths are found to be more relevant in 62 samples (62% of 100 samples) while PGQA's paths are more relevant in 38 samples (38% of 100 samples).
We also study the effect of using a different generative LM (GPT-2 as used by PGQA) as backbone for CoSe-Co in appendix B and empirically
Results for PGQA and QA-GNN are reproduced using their official open-source implementations while numbers for other baselines have been taken from these two works.
Entity masking for training	Query template for path-sentence	Entity masking for inference
pmask	IHdev (%)	Query	IHdev (%)	Type	IHdev (%)
0.0	77.52 (±0.44)	Q1	77.69 (±0.43)	Interrogative	78.07 (±0.56)
0.50	77.38 (±0.40)
0.67	77.61 (±0.79)	Q2	77.25 (±0.64)	Random	77.90 (±0.84)
1.0	77.71 (±1.17)
0.33	78.15 (±0.23)	Q1 + Q2	78.15 (±0.23)	No Masking	78.15 (±0.23)
Table 3: Studying the effect of ablation variants through comparison on CommonsenseQA dev set.
establish that performance gains over PGQA are independent of which LM is used.
Entity masking during training As described in §3.2, a parameter  $p_{mask}$  is used to decide whether entities in an input sentence will be masked. We tune  $p_{mask}$  over the CSQA IHdev set and determine 0.33 as optimal value. Table 3 shows comparison where masking during training works better than not masking. We show qualitative analysis for different  $p_{mask}$  in appendix C. Further,  $0 < p_{mask} < 1$  ensures trained CoSe-Co can be used for both masked and unmasked inputs.
Path-sentence query templates As described in §3.1, we used two query templates— $Q1$  (includes relation information) and  $Q2$  (does not capture relations)—while creating our path-sentence paired dataset. Here we study the effect of using these different query templates (Table 3). We observe that training CoSe-Co on a combined dataset,  $Q1 + Q2$ , results in the best performance, followed by that on using  $Q1$  alone, that further outperforms  $Q2$ . This indicates the influence of including relation information in the training dataset.
Entity masking during inference Since CoSe-Co is given a masked sentence as input during training ( $p_{mask} = 0.33$ ), we explore the effect of similar type of masking during inference. Specifically, certain parts of input sentence can be replaced with masked token to enable CoSe-Co to generate paths that lead towards filling the mask. As reported in Table 3, the variant where no masking is done performs marginally better than when Interrogative or Random tokens in sentence are masked. Thus, by default we do not perform masking during inference unless otherwise stated.
4.4 OpenCSR: Open-Ended CommonSense Reasoning
In CSQA, often multiple choices are appropriate and model gets penalised unfairly if it predicts
suitable answer which does not match with single ground truth. To mitigate this, Lin et al. (2019) re-configured three multi-choice QA datasets for OpenCSR as a generative task where interrogative tokens are replaced with blanks (“_”) and a set of singleton tokens is labeled as ground truth. To generate a set of paths  $P$ , we use inference masking variant of CoSe-Co since question contains a blank. Given a question  $q$ , blank (“_”) is replaced with mask token. To inject our paths, we devise a supervised method where we adapt a separate T5-base model for OpenCSR such that concatenation of  $q$  and paths is given as input to T5 along with the prefix 'fill mask to answer question':'. T5 is trained to generate one of the answers in ground truth set. During inference, top-  $K$  answers, determined on basis of generation likelihood from T5 decoder, are taken as answer candidates.
Table 4 shows comparison between DrFact $^3$  (Lin et al., 2021a) (current state-of-the-art based on BERT-base) and our supervised method which uses CoSe-Co's paths. Specifically, we evaluate - 1) 'Paths from CoSe-Co' where generated paths are concatenated; and 2) 'Concepts from CoSe-Co' where only entities in generated paths are appended. Since our supervised method is based on pre-trained T5, for fair comparison and to probe if performance changes are due to T5, we compare against another baseline: T5-base fine tuned for OpenCSR without paths. We evaluate two metrics as used in Lin et al. (2021a): 1) Hits@K: Determined on basis of whether generated and ground truth answer sets have non-empty intersection; 2) Recall@K: Estimates how many predicted answers match at least one ground truth answer. We vary value of K to be \{10, 30, 50\}. We evaluate on three datasets - ARC (Clark et al., 2018), QASC (Khot et al., 2020), and OBQA (Mihaylov et al., 2018).
CoSe-Co performs significantly better than
3The authors communicated that the test set and leader board has not been released yet. Hence, we report results using the author provided code and validation set. Also, they run their models on single seed.
ARC	QASC	OBQA
Hits@K	H@10	H@30	H@50	H@10	H@30	H@50	H@10	H@30	H@50
DrFact (Lin et al., 2021a)	36.09	53.25	64.50	21.78	37.62	51.49	12.08	23.77	35.13
T5-base (Raffel et al., 2019)	49.70	67.46	71.01	33.66	47.52	53.47	17.42	29.55	37.88
+ CoSe-Co Paths	50.89	63.91	69.23	30.69	47.52	56.44	20.45	34.09	45.45
+ CoSe-Co Concepts	44.97	66.86	73.37	35.64	47.52	57.43	21.21	35.61	42.42
Recall@K	R@10	R@30	R@50	R@10	R@30	R@50	R@10	R@30	R@50
DrFact (Lin et al., 2021a)	12.60	21.05	27.27	12.38	22.28	29.70	6.12	11.85	16.51
T5-base (Raffel et al., 2019)	15.98	28.30	33.93	18.98	26.40	30.53	8.52	14.61	18.71
+ CoSe-Co Paths	16.87	27.45	33.73	17.49	28.05	33.33	9.90	16.53	22.42
+ CoSe-Co Concepts	15.12	28.99	35.21	19.64	28.05	33.00	9.96	17.35	21.10
both baselines on all datasets uniformly. Specifically, 'Concepts from CoSe-Co' usually performs better which shows entities in paths generated by CoSe-Co are useful. Our approach provides performance gains of upto  $8\%$ ,  $6\%$ ,  $10\%$  in Hits@50 and  $8\%$ ,  $3\%$ ,  $6\%$  in Recall@50 over DrFact on ARC, QASC and OBQA respectively. Even though T5-base baseline performs better than DrFact, commonsense from CoSe-Co augmented with T5 achieves new state of the art on this task with performance gains upto  $2.3\%$ ,  $3.9\%$ ,  $7.5\%$  in Hits@50 and  $1.2\%$ ,  $2.5\%$ ,  $3.7\%$  in Recall@50 over T5-base on ARC, QASC and OBQA respectively.
4.5 Effect of Concatenating CoSe-Co Knowledge in Generation Task
We explore augmenting CoSe-Co paths for text generation where our aim is not to obtain SOTA results but to analyse if it improves performance of a base model. Specifically, we study Paraphrase Generation: given a sentence, generate another sentence expressing same meaning using different words where commonsense is usually needed while rephrasing. Since T5 (Raffel et al., 2019) is designed for generation tasks, we fine-tune T5-base to generate annotated paraphrase given a sentence as input on MRPC dataset (Dolan and Brockett, 2005). Generated paths are appended as string to input. Please refer to appendix E.4 for elaborated implementation details and discussion.
Table 4: Performance comparison on Hits@K and Recall@K metrics for OpenCSR (Lin et al., 2021a) on ARC, QASC and OBQA datasets. DrFact is a BERT-based current state of the art method.
MRPC Paraphrase Generation
BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
T5-base	43.10	36.10	61.80	36.33	47.10
+ CoSe-Co Paths	44.50	36.70	62.50	37.34	48.50
Table 5: Using CoSe-Co Paths leads to improvements in paraphrase generation task on MRPC dataset. Generative commonsense methods like PGQA which rely on answer choices cannot be applied in tasks like paraphrase generation where entities are not available.
Table 5 summarises results evaluated through commonly used generation metrics - BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al., 2016). Amongst these, SPICE is considered to correlate most with human judgement. Using CoSe-Co paths results in better paraphrase generation as indicated by  $\sim 1 - 1.5\%$  improvement in most metrics.
We presented CoSe-Co, a framework to generate commonsense inferences that are relevant to the overall context of a given natural language text. We created a novel dataset of <sentence, commonsense paths> pairs for training CoSe-Co and make it available to the community<sup>4</sup>. Empirical evaluation shows that commonsense inferences generated by CoSe-Co are relevant, diverse, and also contain novel entities not present in KG. We augment knowledge generated by CoSe-Co in commonsense tasks such as Multi-Choice QA and Open-ended CommonSense Reasoning, achieving SoTA results for these tasks. Further, we also used CoSe-Co for NLP tasks such as paraphrase generation achieving improved performance. While, using ConceptNet as our base KG allowed us to perform an exhaustive fair comparison with a variety of benchmark methods—where the motivation is to provide more relevant knowledge (in symbolic form as in KG) to tasks—CoSe-Co can further be enhanced by utilizing other commonsense KGs. Our work can be extended to explore better ways of integrating the generated knowledge generically across a variety of KGs and LMs, and is a potential direction for future work.
An ethics statement regarding the released dataset has been discussed in Appendix G.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2503-2514, Osaka, Japan. The COLING 2016 Organizing Committee.
Maria Becker, Katharina Korfhage, Debjit Paul, and Anette Frank. 2021. Co-nect: A framework for revealing commonsense knowledge paths as explicitations of implicit knowledge in texts. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 21–32, Groningen, The Netherlands (online). Association for Computational Linguistics.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In International Conference on Learning Representations.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 4923-4931. AAAI Press.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multihop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309, Online. Association for Computational Linguistics.
Kelvin Guu, John Miller, and Percy Liang. 2015. Traversing knowledge graphs in vector space. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 318-327, Lisbon, Portugal. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Filip Ilievski, Pedro Szekely, and Bin Zhang. 2021. Cskg: The commonsense knowledge graph. In European Semantic Web Conference, pages 680-696. Springer.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8082-8090.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1813-1827, Online. Association for Computational Linguistics.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445-1455, Berlin, Germany. Association for Computational Linguistics.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829-2839, Hong Kong, China. Association for Computational Linguistics.
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William Cohen. 2021a. Differentiable open-ended commonsense reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4611-4625, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021b. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 1504-1515.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang,