SlowGuess/NAACL_2022_OCR huggingface dataset:
- PTM+GNN models integrate PTM with additional GNN-based KG encoders. Based on the same PTM (the above baseline), we consider five variants with different KG encoders: (1) Relation Network (RN) (Santoro et al., 2017) using a relational reasoning structure over the CSKG; (2) GcoAttn (Lin et al., 2019) using a graph concept attention model to aggregate entity information from the CSKG; (3) RGCN (Schlichtkrull et al., 2018) extending the GCN with relation-specific weights; (4) MHGRN (Feng et al., 2020a) using a GNN architecture reasoning over the CSKG that unifies both GNNs and path-based models; (5) QA-GNN (Yasunaga et al., 2021) using a GAT to perform jointly reasoning over the CSKG.
For all these methods, we adopt the same architecture and configuration for the PTM, so that we can examine the effect of different KG encoders.
We implement all PTMs based on HuggingFace Transformers (Wolf et al., 2020). For all the baselines, we keep the common hyper-parameters as identical as possible and set their special hyperparameters following the suggestions from the original papers. In our approach, we extract the relation paths with no more than 2 hops between the concept nodes from the question and the answer candidate. We tune the hidden dimension of MLPs from the path encoder in \{32, 64, 100\}, and the batch size in \{32, 48, 60, 120\}. The parameters of the model are optimized by RAdam (Liu et al., 2020), and the learning rate of the PTM and the KG encoder is also tuned in \{1e-4, 1e-5, 2e-5\} and \{1e-3, 1e-2\}, respectively. To accelerate the training process, we don't incorporate Dropout regularization in our model. All the above hyper-parameters are tuned on the development set.
5.3 Results Analysis
Following previous works (Yasunaga et al., 2021; Wang et al., 2021a), we take the results on CommonsenseQA and OpenBookQA as the main experiments to compare different methods. In order to test their robustness to data sparsity, we examine the performance under five different proportions of training data, i.e.,
Methods	CommonsenseQA	OpenBookQA
5%	10%	20%	50%	80%	100%	5%	10%	20%	50%	80%	100%
RoBERTa-large	29.66	42.84	58.47	66.13	68.47	\( 68.69^† \)	37.00	39.4	41.47	53.07	57.93	\( 64.8^† \)
+ RGCN	24.41	43.75	59.44	66.07	68.33	\( 68.41^† \)	38.67	37.53	43.67	56.33	63.73	\( 62.45^† \)
+ GconAttn	21.92	49.83	60.09	66.93	69.14	\( 68.59^† \)	38.60	36.13	43.93	50.87	57.87	\( 64.75^† \)
+ RN	23.77	34.09	59.90	65.62	67.37	\( 69.08^† \)	33.73	35.93	41.40	49.47	59.00	\( 65.20^† \)
+ MHGRN	29.01	32.02	50.23	68.09	70.83	\( 71.11^† \)	38.00	36.47	39.73	55.73	55.00	\( 66.85^† \)
+ QA-GNN	32.95	37.77	50.15	69.33	70.99	\( 73.41^† \)	33.53	35.07	42.40	54.53	52.47	\( 67.80^* \)
+ SAFE(Ours)	36.45	56.51	65.16	70.72	73.22	74.03	38.80	41.20	44.93	58.33	65.60	69.20
Table 3: Performance comparison on CommonsenseQA and OpenBookQA with different proportions of training data. We report the average test performance of three runs, and the best results are highlighted in bold. † indicates the reported results from Yasunaga et al. (2021). * indicates the reported results from Wang et al. (2021a)
Methods	SocialIQA	PIQA	CoPA
RoBERTa-large	78.25	77.53	67.60
+ GcoAttn	78.86	78.24	70.00
+ RN	78.45	76.88	70.20
+ MHGRN	78.11	77.15	71.60
+ QAGNN	78.10	78.24	68.40
+ SAFE (Ours)	78.86	79.43	71.60
$\{5\%, 10\%, 20\%, 50\%, 80\%, 100\%\}$ .
CommonsenseQA and OpenBookQA. The results of different methods on CommonsenseQA and OpenBookQA are presented in Table 3.
Comparing the results under the full-data setting (i.e.,  $100\%$  training data), we can see that all the PTM+GNN methods perform better than vanilla PTM (i.e., RoBERTa-large). It indicates that the KG encoder on the CSKG is able to incorporate useful knowledge information to improve PTMs on commonsense reasoning tasks. Additionally, among all the PTM+GNN baselines, QA-GNN performs the best. The major reason is that QA-GNN uses the PTM to estimate the importance of KG nodes and connects the QA context and the CSKG to form a joint graph, which is helpful to improve the reasoning ability on the CSKG. Finally, our method consistently outperforms all the baselines. Our approach incorporates a lightweight MLP architecture as the KG encoder with relation paths as features. It reduces the parameter redundancy of the KG encoder and focuses on the most essential features for reasoning, i.e., semantic relation paths. Such an approach is effective to enhance the commonsense reasoning capacity of PTMs.
Comparing the results under different sparsity
Table 4: Performance comparison on SocialIQA, PIQA, and CoPA (Dev accuracy).
Methods	BERT-large	AristoRoBERTa
Fine-tuned PTMs	59.00	78.40†
+ RGCN	45.40	74.60†
+ GconAttn	48.20	71.80†
+ RN	48.60	75.35†
+ MHGRN	46.20	80.60†
+ QA-GNN	58.47	82.77†
+ SAFE (Ours)	59.20	87.13
Table 5: Evaluation with other PTMs on OpenBookQA (average test accuracy of three runs). Methods with AristoRoBERTa use the textual evidence by Clark et al. (2020b) as an additional input to the QA context.  $\dagger$  indicates reported results in (Yasunaga et al., 2021).
ratios of training data, we can see that the performance substantially drops when the size of training data is reduced. While, our method performs consistently better than all baselines. It is because that our KG encoder consists of significantly fewer parameters than those of the baselines, which reduces the risk of overfitting and endows our approach with better robustness in data scarcity scenarios.
Other Commonsense Reasoning Datasets. To further verify the effectiveness of our method, we also compare the results of different methods on other commonsense reasoning datasets. These datasets are from different domains or different tasks. These results are shown in Table 4. Similarly, our approach also achieves the best performance in most cases. It indicates that our approach is generally effective for various commonsense reasoning datasets or tasks, by outperforming competitive but complicated baselines. Among all the datasets, our approach improves the performance of the PTM on CoPA dataset by a large margin. The reason is that CoPA is a small dataset with only 500 training
Figure 4: Analysis of different hidden dimension size of our SAFE model.
examples. Baselines with heavy architectures are easy to overfit on it. In contrast, our KG encoder is lightweight, which is more capable of resisting the overfitting issue.
5.4 Evaluation with Other PTMs
The major contribution of our approach lies in the lightweight KG encoder, which can be also used to enhance the commonsense reasoning capacity of various PTMs. To validate it, we examine the performance of our KG encoder when integrated with two other PTMs, i.e., BERT-large and AristotleBERTa, on OpenBookQA dataset.
As shown in Table 5, the BERT-large and AristotleBERTa enhanced by our KG encoder perform better than original PTMs. Especially, our KG encoder can improve the performance of AristotleBERTa by a large margin (with  $8.73\%$  improvement). These results show that our KG encoder is a general method to improve PTMs for commonsense reasoning. In contrast, when adapting other KG encoders to these two PTMs, the performance decreases in most cases. It is mainly because these KG encoders have complicated architectures, which may not be easily adapted to other PTMs.
5.5 Hyper-parameters Analysis
For hyper-parameter analysis, we study the hidden dimension size of the MLP in the path encoder. Concretely, we evaluate our model with varying values of the hidden dimension size on CommonsenseQA and OpenBookQA datasets using RoBERTa-large model. The results are shown in Figure 4. We can see that with the increase of the hidden dimension size, the performance improves at first and then drops to some extent. The possible reason lies in two aspects. On the one hand, a too
Figure 5: The generated feature values of relation path examples by the path encoder. Q and A denote the concept nodes from the question and the answer candidate, respectively.
small hidden dimension size makes the path encoder hard to represent sufficient information from relation paths for commonsense reasoning. On the other hand, a larger hidden dimension size enlarges the parameter number of our KG encoder, which increases the risk of overfitting that may cause performance degradation.
We propose a rather simple KG encoder to effectively utilize the relation features from the CSKG, which first computes the feature values of the relation paths and then aggregates these values as the confidence score of the question and choice from the perspective of KG. In this way, we can generate a table in advance that maps each type of relation path into its feature value that reflects its contribution to the confidence score. Based on this table, it is convenient to directly judge the importance of the relation path and quickly assess the confidence about if the choice is the answer to the question from the perspective of KG. Figure 5 shows some path-value examples on CommonsenseQA dataset. As we can see, the path with a higher value indeed provide more persuasive evidence (e.g., causes and capableof) that indicates the choice is more likely to be the answer to the question. In contrast, the path with a lower value usually represents an ambiguous relationship (e.g., relatedto), which contributes less to the judge of whether the choice is the answer.
We review the related studies in two aspects, i.e., commonsense reasoning and KG-enhanced pretrained models.
Commonsense Reasoning. Commonsense reasoning tasks aim to evaluate the understanding
of commonsense knowledge (Davis and Marcus, 2015), e.g., physical commonsense (Zellers et al., 2019), which are mostly formulated as a multichoice QA problem. Early studies either rely on explicit text features (Clark et al., 2016) to capture the relations between the question and answer candidates, or adopt neural networks (e.g., DNN or LSTM) (Yu et al., 2014; Chen et al., 2017) to model the implicit correlation features. Recently, pre-trained models (PTM) (Devlin et al., 2019; Liu et al., 2019) have achieved remarkable performance on commonsense reasoning tasks. Furthermore, a surge of works incorporate external knowledge resources to further improve the reasoning performance. Among them, CSKG (e.g., ConceptNet (Speer et al., 2017)) has been widely studied, and existing works mainly adopt graph neural networks to learn useful commonsense knowledge from the CSKG to enhance PTMs. Based on these works, we systemically study what is necessarily needed from CSKGs for improving PTMs. Our analysis leads to an important finding that relation features mainly contribute to the performance improvement, and we design a lightweight MLP architecture to simplify the KG encoder.
KG-Enhanced Pre-trained Models. Recently, a series of works focus on enhancing PTMs with external KGs to improve the performance on factual knowledge understanding (Sun et al., 2020; Wang et al., 2021b) and knowledge reasoning tasks (Talmor et al., 2019; Zhang et al., 2019; He et al., 2020). These works inject the structured knowledge from the external KG into PTMs in either pre-training or fine-tuning stage. The first class of works mainly focus on devising knowledge-aware pre-training tasks (Wang et al., 2021b; Zhang et al., 2019) to improve the understanding of entities or triples from the KG, e.g., knowledge completion (Wang et al., 2021b) and denoising entity auto-encoder (Zhang et al., 2019). Another class of works adopt task-specific KG encoders to enhance PTMs during fine-tuning, e.g., path-based relation network (Feng et al., 2020a) and GNN (Yasunaga et al., 2021). Different from them, we aim to directly enhance PTMs with a KG encoder on the downstream commonsense reasoning tasks, and design a rather simple yet effective KG encoder.
In this work, we study how the external commonsense knowledge graphs (CSKGs) are utilized to
improve the reasoning capacity of pre-trained models (PTMs). Our work makes an important contribution to understanding and enhancing the commonsense reasoning capacity of PTMs. Our results show that relation paths from the CSKG are the key to performance improvement. Based on this finding, we design a rather simple MLP-based KG encoder with relation paths from the CSKG as features, which can be generally integrated with various PTMs for commonsense reasoning tasks. Such a lightweight KG encoder has significantly fewer than  $1\%$  trainable parameters compared to previous GNN-based KG encoders. Experimental results on five commonsense reasoning datasets demonstrate the effectiveness of our approach.
In future work, we will study how to effectively leverage the commonsense knowledge from large-scale unstructured data to improve PTMs. We will also try to apply our approach to other knowledge-intensive tasks, e.g., knowledge graph completion and knowledge graph based question answering (Lan et al., 2021).
This work primarily investigates how external commonsense knowledge graphs (CSKGs) enhance the commonsense reasoning capacity of pre-trained models (PTMs) and proposes a simple but effective KG encoder on CSKGs to enhance PTMs. A potential problem derives from using PTMs and CSKGs in our approach. PTMs have been shown to capture certain biases from the data that have been pre-trained on (Bender et al., 2021). And existing works (Mehrabi et al., 2021) have found that CSKGs are likely to contain biased concepts derived from human annotations. However, a comprehensive analysis of such biases is outside of the scope of this work. It is a compelling direction to investigate to what extent the combination of CSKGs and PTMs can help mitigate such biases. An alternative consideration is to consider filtering biased concepts in the process of subgraph extraction from the CSKG. By devising proper rules, it is promising to reduce the influence of biased concepts on our approach.
This work was partially supported by Beijing Natural Science Foundation under Grant No. 4222027, and National Natural Science Foundation of China under Grant No. 61872369, Beijing Outstand
ing Young Scientist Program under Grant No. BJWZYJH012019100020098. This work is also supported by Beijing Academy of Artificial Intelligence (BAAI). Xin Zhao is the corresponding author.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1657-1668.
Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groeneveld, Michal Guerquin, and Michael Schmitz. 2020a. From 'f' to 'a' on the N.Y. regents science exams: An overview of the aristo project. AI Mag., pages 39-53.
Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groeneveld, Michal Guerquin, and Michael Schmitz. 2020b. From 'f' to 'a' on the N.Y. regents science exams: An overview of the aristo project. AI Mag., 41(4):39-53.
Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D. Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2580-2586.
Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92-103.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020a. Scalable multi-hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1295-1309.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020b. Scalable multi-hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309.
Gaole He, Junyi Li, Wayne Xin Zhao, Peiju Liu, and JiRong Wen. 2020. Mining implicit entity preference from user-item interaction data for knowledge graph completion via adversarial learning. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 740-751. ACM / IW3C2.
Ian T. Jolliffe. 1986. *Principal Component Analysis*. Springer Series in Statistics.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. A survey on complex knowledge base question answering: Methods, challenges and solutions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 4483-4491. ijcai.org.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2829-2839.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3243-3253.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. arXiv preprint arXiv:2103.11320.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2381-2391.
Tuan-Phong Nguyen, Simon Razniewski, Julien Romero, and Gerhard Weikum. 2021. Refined commonsense knowledge from large-scale web contents. arXiv preprint arXiv:2112.04596.
Yilin Niu, Fei Huang, Jiaming Liang, Wenkai Chen, Xiaoyan Zhu, and Minlie Huang. 2021. A semantic-based method for unsupervised commonsense question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3037-3049.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *Logical Formalizations of Commonsense Reasoning*, Papers from the 2011 AAAI Spring Symposium, Technical Report SS-11-06, Stanford, California, USA, March 21-23, 2011.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Tim Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4967-4976.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019a. ATOMIC: an atlas of machine commonsense for if-then reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The
Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 3027-3035.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Common-sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4462-4472.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In *The Semantic Web - 15th International Conference*, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843, pages 593-607.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444-4451.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 3660-3670.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4149-4158.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of ai through gamification.
Robert Endre Tarjan. 1972. Depth-first search and linear graph algorithms. SIAM J. Comput., pages 146-160.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.