SlowGuess/NAACL_2022_OCR huggingface dataset:
Jinhao Jiang $^{1,3*}$ , Kun Zhou $^{2,3*}$ , Wayne Xin Zhao $^{1,3,4\dagger}$  and Ji-Rong Wen $^{1,2,3}$
<sup>1</sup>Gaoling School of Artificial Intelligence, Renmin University of China.
$^{3}$ Beijing Key Laboratory of Big Data Management and Analysis Methods.
$^{4}$ Beijing Academy of Artificial Intelligence, Beijing, 100084, China.
jiangjinhao@ruc.edu.cn, francis_kun_zhou@163.com,
batmanfly@gmail.com, jrwen@ruc.edu.cn
Commonsense reasoning in natural language is a desired ability of artificial intelligent systems. For solving complex commonsense reasoning tasks, a typical solution is to enhance pre-trained language models (PTMs) with a knowledge-aware graph neural network (GNN) encoder that models a commonsense knowledge graph (CSKG). Despite the effectiveness, these approaches are built on heavy architectures, and can't clearly explain how external knowledge resources improve the reasoning capacity of PTMs. Considering this issue, we conduct a deep empirical analysis, and find that it is indeed relation features from CSKGs (but not node features) that mainly contribute to the performance improvement of PTMs. Based on this finding, we design a simple MLP-based knowledge encoder that utilizes statistical relation paths as features. Extensive experiments conducted on five benchmarks demonstrate the effectiveness of our approach, which also largely reduces the parameters for encoding CSKGs. Our codes and data are publicly available at https://github.com/RUCAIBox/SAFE.
In the era of artificial intelligence, it is desirable that intelligent systems can be empowered by the capacity of commonsense reasoning in natural language. For this purpose, a surge of commonsense reasoning tasks and datasets are proposed to evaluate and improve such an ability of NLP models, e.g., CommonsenseQA (Talmor et al., 2019) and SocialIQA (Sap et al., 2019b). Although large-scale pre-trained models (PTMs) (Devlin et al., 2019; Liu et al., 2019) have surpassed human performance in a number of NLP benchmarks, it is still hard for PTMs to accurately capture and understand commonsense knowledge for accomplishing complex reasoning tasks (Talmor et al., 2021).
In order to enhance the reasoning capacity, commonsense knowledge graphs (CSKGs) (e.g., ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019a)) have been adopted for injecting external commonsense knowledge into PTMs. By conducting entity linking to CSKGs, existing methods (Yasunaga et al., 2021; Feng et al., 2020a) aim to capture the structured knowledge semantics via knowledge graph (KG) encoders (e.g., graph neural network (GNN) (Velickovic et al., 2018; Kipf and Welling, 2017)), and then integrate the KG encoders for improving the commonsense reasoning capacity of PTMs (Yasunaga et al., 2021).
Despite the effectiveness, these approaches are built on highly complicated network architectures (involving both PTMs and GNNs). Thus, it is difficult to explain how and why external commonsense knowledge improves the commonsense reasoning capacity of PTMs. Besides, existing CSKGs (Mehrabi et al., 2021; Nguyen et al., 2021) are mostly crowdsourced from massive selected resources (e.g., books, encyclopedias, and scraped web corpus), containing a wide variety of content. Without a clear understanding of how these external resources should be utilized, it is likely to incorporate irrelevant concepts or even knowledge biases (Mehrabi et al., 2021; Nguyen et al., 2021) into PTMs, which might hurt the reasoning performance. Indeed, some researchers have noted this issue and questioned whether existing GNN-based modules are over-complicated for commonsense reasoning (Wang et al., 2021a). Furthermore, they find that even a simple graph neural counter can outperform existing GNN modules on CommonsenseQA and OpenBookQA benchmarks.
However, existing studies can't well answer the fundamental questions about knowledge utilization for commonsense reasoning: How do external knowledge resources enhance the commonsense reasoning capacity of PTMs? What is necessarily required from external knowledge resources
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1730 - 1741
for PTMs? Since the simplified knowledge-aware GNN has already yielded performance improvement on the CommonsenseQA (Wang et al., 2021a), we speculate that there might be a simpler solution if we could identify the essential knowledge for commonsense reasoning.
Focusing on this issue, we think about designing the solution by further simplifying the KG encoder. Based on our empirical analysis, we observe a surprising result that it is indeed relation features from CSKGs, but not node features, that are the key to the task of commonsense reasoning (See more details in Section 3). According to this finding, we propose a rather simple approach to leveraging external knowledge resources for enhancing the commonsense reasoning capacity of PTMs. Instead of using a heavy GNN architecture, we design a lightweight KG encoder fully based on the multilayer perceptron (MLP), which utilizes Statistical relation pAth from CSKGs as FEatures, namely SAFE. We find that semantic relation paths can provide useful knowledge evidences for PTMs, which is the key information for helping commonsense reasoning. By conducting extensive experiments on five benchmark datasets, our approach achieves superior or competitive performance compared with state-of-the-art methods, especially when training data is limited. Besides the performance improvement, our approach largely reduces the parameters for encoding CSKGs (fewer than  $1\%$  trainable parameters compared to GNN-based KG encoders (Yasunaga et al., 2021)).
Our main contributions can be summarized as follows: (1) We empirically find that relation features from CSKGs are the key to the task of commonsense reasoning; (2) We design a simple MLP-based architecture with relation paths as features for enhancing the commonsense reasoning capacity of PTMs; (3) Extensive experiments conducted on five benchmark datasets demonstrate the effectiveness of our proposed approach, which also largely reduces the parameters of the KG encoder.
2 Task Description
According to pioneer works (Talmor et al., 2019; Mihaylov et al., 2018), the commonsense reasoning task can be generally described as a multichoice question answering problem: given a natural language question  $q$  and a set of  $n$  choices  $\{c_1, \dots, c_n\}$  as the answer candidates, the goal is to select the most proper choice  $c^{\star}$  from these can-
didates to answer the question based on necessary commonsense knowledge.
To explicitly capture commonsense knowledge, external commonsense knowledge graphs (CSKGs) have often been utilized in this task, e.g., ConceptNet (Speer et al., 2017). A CSKG can be formally described as a multi-relational graph  $\mathcal{G} = (\mathcal{V},\mathcal{R},\mathcal{E})$ , where  $\mathcal{V}$  is the set of all concepts (or entity) nodes (e.g., hair and water),  $\mathcal{R}$  is the set of relation types (e.g., related to and at location), and  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{R} \times \mathcal{V}$  is the set of relational links that connect two concept nodes in  $\mathcal{V}$ .
Following prior studies (Lin et al., 2019), we solve the commonsense reasoning task in a knowledge-aware setting, where a CSKG  $\mathcal{G}$  is available as input. We first link the mentioned concepts from the question and the answer candidates to the CSKG, so that we can leverage the rich semantic knowledge from the CSKG for commonsense reasoning. Based on the linked concepts in the question and each answer candidate, we further extract their neighbouring nodes from  $\mathcal{G}$  and the relational links that connect them, to compose a subgraph  $\mathcal{G}^{q,c_i}$  for characterizing the commonsense knowledge about the question  $q$  and the answer candidate  $c_i$ .
3 Empirical Analysis on the Commonsense KG Encoder
In this section, we conduct an empirical study to investigate how the external KG encoder helps PTMs with commonsense reasoning.
3.1 Analysis Setup
To conduct the analysis experiments, we select QA-GNN (Yasunaga et al., 2021), a representative approach that integrates PTM with GNN for the commonsense QA task, as the studied model. We adopt the CommonsenseQA (Talmor et al., 2019) and OpenBookQA (Mihaylov et al., 2018), two of the most widely used commonsense reasoning benchmarks, for evaluation, with the same data split setting in (Lin et al., 2019).
We perform two analysis experiments: one examines the effect of the commonsense KG encoder, and the other one examines the effect of different features in the commonsense KG encoder. To be specific, the two experiments focus on two key questions about commonsense reasoning: (1) what is the effect of the commonsense KG encoder on PTMs? (2) what is the key information within the
(a)CommonsenQA
(b) OpenBookQA
commonsense KG encoder?
3.2 Results and Findings
Next, we conduct the experiments and present our findings of commonsense reasoning.
Effect of Commonsense KG Encoder. Since existing studies have widely utilized a GNN module to encode the commonsense knowledge, we examine its contribution to the improvement of reasoning performance. We consider comparing three variants of QA-GNN: (A) PTM-Only that directly removes the GNN module and degenerates into a pure PTM, (B) PTM-Pred that trains the PTM and GNN simultaneously but only makes the prediction with the PTM module, and (C) GNN-Pred that trains the PTM and GNN simultaneously but only makes the prediction with the GNN module.
The comparison results are shown in Figure 1. As we can see, using the predictions solely based on the GNN module (i.e., GNN-Pred) can only answer a relatively minor proportion of the questions (no more than  $60\%$  in CommonsenseQA). As a comparison, when trained independently (i.e., PTM-Only) or jointly with the GNN module (i.e., PTM-Pred), the PTM module can answer a large proportion of the questions (at least  $70\%$  in CommonsenseQA). Furthermore, the incorporation of the GNN encoder is useful to improve the performance of PTMs (PTM-Only v.s. QAGNN). These results show that:
- In the joint PTM-GNN approach, PTM contributes the most to the commonsense reasoning task, which is the key to the reasoning performance.
- Commonsense KG encoder is incapable of performing effective reasoning independently, but can enhance PTM as the auxiliary role.
Effect of Node/Relation Features from KG. The
Figure 1: Performance comparison on Common-senseQA and OpenBookQA (Dev accuracy).
Figure 2: Performance examination for KG encoder on CommonsenseQA and OpenBookQA (Dev accuracy).
major aim of the KG encoder is to characterize the commonsense knowledge and provide necessary knowledge evidence for enhancing the reasoning capacity of PTMs. Generally, a CSKG consists of concept nodes and relational links. To identify the key knowledge information that is necessarily needed, we now examine the effect of node and relation features from CSKG. To eliminate the effect of PTM module, we remove it and compare the performance of only KG encoder under two experiment settings: (A) reducing the dimension of node embeddings to  $d$  (PCA (Jolliffe, 1986) is applied to select  $d$  most informative dimensions), and (B) randomly removing  $p$  percent of relational links in the KG subgraph for a question-candidate pair.
As shown in Figure 2, we surprisingly find that even after reducing the dimension of node embeddings to 1, the performance of the GNN encoder can be still improved. These results show that node features are not the key information utilized by the GNN encoder. In contrast, removing a considerable proportion of links significantly reduces the performance. From these observations, we can conclude that: The relation features from the CSKG are indeed the key knowledge information that is actually needed by the KG encoder.
The former sections show that the role of the KG encoder on CSKGs is to mainly complement PTMs in the task of commonsense reasoning. Instead of node features, relations features are the key to the KG encoder for improving PTMs. Based on these findings, we develop a simple commonsense KG encoder based on the statistical relation features from CSKGs, namely SAFE. Figure 3 presents the overview of our model.
4.1 Capturing High-Order Relation Semantics
Since relation features are shown useful to improve the performance of commonsense reasoning, we consider extracting relation features for better capturing the knowledge semantics from the CSKG. Inspired by KG reasoning studies (Lin et al., 2018; Feng et al., 2020b), we construct multi-hop relation paths that connect question nodes with answer candidate nodes on the CSKG to capture the higher-order semantic relatedness among them.
Formally, given the commonsense subgraph  $\mathcal{G}^{q,c_i}$  for the question  $q$  and the answer candidate  $c_{i}$ , we first extract a set of relation paths within  $k$  hops that connect a question concept node  $v_{q}\in \mathcal{V}_{q}$  and an answer concept node  $v_{c_i}\in \mathcal{V}_{c_i}$ , denoted as  $\mathcal{P}^{q,c_i}$ . Specifically, a path  $p\in \mathcal{P}^{q,c_i}$  can be represented as a sequence of nodes and relations as  $p = \{v_{1},r_{1},\dots ,r_{k - 1},v_{k}\}$ . Based on the empirical findings in Section 3, we consider a simplified representation for relation paths that removes node IDs but only keeps the relations on a path. To keep the role of each node, we replace a node ID by a three-valued type, indicating this node belongs to a question node (0), answer node (1) or others (2). In this way, a path  $p$  can be represented by  $p = \{t_{v_1},r_1,t_{v_2},r_2,\dots ,r_{k - 1},t_{v_k}\}$ , where  $t_v$  is the role type of node  $v$ . Since we remove explicit node IDs, our model can concentrate on more essential relation features.
Based on the above method, for a question  $q$  and an answer candidate  $c_{i}$ , we extract all the simplified relation paths and count their frequencies among all the paths. We use  $\mathcal{F}^{q,c_i} = \{\langle p_j,f_j\rangle \}$  to denote all the paths for the question  $q$  and the answer candidate  $c_{i}$ , where each entry consists of the  $j$ -th path  $p_j$  and its frequency  $f_{j}$ . Unlike prior approaches (e.g., QA-GNN), we use such very simple features of relation paths from CSKGs to improve the reasoning capacity of PTMs.
4.2 A MLP-based KG encoder
Our KG encoder is built on a full MLP architecture based on simplified relation path features, consisting of a path encoder and a feature aggregator.
Path Encoder. The path encoder is a two-layer MLP that encodes a relation path into a scalar feature value. As shown in Section 4.1, we can obtain the path feature set  $\mathcal{F}^{q,c_i} = \{\langle p_j,f_j\rangle \}$  for the question  $q$  and the answer candidate  $c_{i}$ . Different from general KGs, CSKGs usually contain much fewer
Figure 3: The illustration of our approach. We adopt an all-MLP KG encoder to model the extracted relation features from CSKG to enhance the PTM.
types of relations (e.g., 36 relations in ConceptNet), we adopt one-hot representations of these types to represent these relations. For node type (from question, candidate or others), we also adopt the similar representations. Then, we concatenate these one-hot vectors to compose the sparse representation of a relation path  $p$  in order, denoted as  $\mathbf{v}_p$ . Subsequently, the sparse path representation is encoded by a two-layer MLP (i.e., the path encoder) to produce the corresponding scalar feature value  $x_p$ :
$$
x _ {p} = \operatorname {M L P} _ {2} \left(\operatorname {M L P} _ {1} \left(\mathbf {v} _ {p}\right)\right), \tag {1}
$$
where  $x_{p}$  reflects the importance of such a relation path for commonsense reasoning.
Feature Aggregator. Based on the above path encoder, we can generate the scalar feature values for all the relation paths in the feature set  $\mathcal{F}^{q,c_i} = \{\langle p_j,f_j\rangle \}$ . The feature aggregator aims to aggregate these feature values to produce the confidence score of the answer candidate w.r.t. the question, from the KG perspective. Concretely, we sum the different feature values of relation paths weighted by their frequencies as follows:
$$
x _ {q, c _ {i}} = \sum_ {\left\langle p _ {j}, f _ {j} \right\rangle \in \mathcal {F} ^ {q, c _ {i}}} x _ {p _ {j}} \cdot f _ {j}, \tag {2}
$$
where  $x_{p_j}$  is the mapping feature value of path  $p_j$  and  $f_{j}$  is the frequency of path  $p_j$ . Here,  $x_{q,c_i}$  aims to capture the overall confidence score based on the subgraph  $\mathcal{G}^{q,c_i}$  given the question and the answer candidate. However, since the weighted sum is likely to cause extreme values (i.e., too large or too small), we add an extra two-layer MLP for scaling:
$$
S _ {K G} (q, c _ {i}) = \operatorname {M L P} _ {4} \left(\operatorname {M L P} _ {3} \left(x _ {q, c _ {i}}\right)\right), \tag {3}
$$
where  $S_{KG}$  is the prediction score indicating the confidence level that candidate  $c_i$  is the right answer to question  $q$  from the perspective of KG.
4.3 Integrating KG Encoder with PTM
In this part, we integrate the above KG encoder with the PTM for commonsense reasoning.
The PTM Encoder. Following existing works (Yasunaga et al., 2021), we utilize a PTM as the backbone of commonsense reasoning. Given a question  $q$  and an answer candidate  $c_{i}$ , we concatenate their text to compose the input of the PTM. After encoding by the multiple Transformer layers, we select the output of the [CLS] token in the last layer as the contextual representation of the question-candidate pair, denoted by  $\mathbf{h}_{cls}$ . Then, we feed  $\mathbf{h}_{cls}$  into a MLP layer to produce a scalar output  $S_{PTM}$ ,
$$
\mathbf {h} _ {c l s} = \operatorname {P T M} \left(q, c _ {i}\right), \tag {4}
$$
$$
S _ {P T M} (q, c _ {i}) = \operatorname {M L P} \left(\mathbf {h} _ {c l s}\right), \tag {5}
$$
which is the plausibility score of the answer candidate from the perspective of PTM.
Combining the Prediction Scores. We then derive the prediction score of each answer candidate for a question by leveraging both the PTM and KG encoder based on either textual or structured semantics. For each question-candidate pair  $(q, c_i)$ , we combine the prediction scores of the two modules as:
$$
S (q, c _ {i}) = S _ {P T M} (q, c _ {i}) + S _ {K G} (q, c _ {i}), \tag {6}
$$
where  $S_{PTM}(q, c_i)$  (Eq. 5) and  $S_{KG}(q, c_i)$  (Eq. 3) are the prediction scores of PTM and KG encoder, respectively. Given a set of answer candidates  $\{c_1, \dots, c_n\}$ , we further normalize  $S(q, c_i)$  into a conditional probability  $\operatorname{Pr}(c_i|q)$  via the softmax operation over the  $n$  candidates.
During the training stage, we optimize the parameters of the whole model (including both the PTM and KG encoder) with the cross entropy loss between the predictions and the ground-truth answer (based on the probability distribution  $\{\operatorname{Pr}(c_i|q)\}_{i=1}^n$ ). During inference, we first compute the probability score  $\operatorname{Pr}(c_i|q)$  for each answer candidate, and then select the highest one as the predicted answer.
4.4 Comparison with Existing KG Encoders
For the task of commonsense reasoning, it has become a common approach by integrating PTM with
RGCN	MHGRN	QAGNN	SAFE
Node emb.	✓	✓	✓	×
Relation	✓	✓	✓	✓
GNN	✓	✓	✓	×
MLP-based	×	×	×	✓
# Params	365K	547K	2845K	4.7k
Table 1: Comparisons of different KG encoders for commonsense reasoning. Instead of using node embeddings and GNN structure, we adopt relation paths as the input features and incorporate a full MLP architecture.
an external KG encoder based on CSKGs. The major difference among these methods (including our approach) lies in the design of the KG encoder. Next, we compare these variants for the KG encoder.
We summarize the comparison between our KG encoder and representative KG encoders in Table 1. We can see that, our approach no longer lies in the node embeddings and the structure of GNNs. Instead, we mainly utilize relation paths as the features of the KG encoder, which is built on a simple MLP-based architecture. Therefore, the number of the model parameters involved in our KG encoder is much smaller than those of existing KG encoders. As will be shown in Section 5, our KG encoder yields better or at least comparable performance compared with existing GNN-based encoders, based on the same configuration for PTMs.
Specifically, our approach can largely reduce the computational costs for encoding the CSKG. For our approach, we need to extract the relation paths from question nodes to all the answer candidate nodes on the CSKG, and it can be efficiently fulfilled via a  $k$ -hop constrained Depth-First Search (Tarjan, 1972), which can be pre-computed in offline processing. When the relation paths have been extracted, it is efficient to encode these paths with our MLP architecture. Such a process can be easily paralleled or accelerated by optimized matrix multiplication. In contrast, existing GNN-based encoders rely on iterative propagation and aggregation on the entire subgraph, which takes a much larger computational time cost.
In this part, we introduce the experimental setup.
Evaluation Tasks. We conduct experiments on five commonsense reasoning tasks, shown in Table 2.
Task	Train	Dev	Test
CommonsenQA	9,741	1,221	1,140
OpenBookQA	4,957	500	500
SocialIQA	33,410	1,954	-
PIQA	16,113	1,838	-
CoPA	-	500	500
Table 2: Statistics of the datasets. “-” denotes the unused or not available dataset split in our experiments.
- CommonsenseQA (Talmor et al., 2019) is a 5-way multiple-choice QA dataset. It is created based on ConceptNet (Speer et al., 2017).
- OpenBookQA (Mihaylov et al., 2018) is a 4-way multiple-choice QA dataset about elementary science questions to evaluate the science commonsense knowledge.
- SocialIQA (Sap et al., 2019b) is a 3-way multiple-choice QA dataset to evaluate the understanding of social commonsense knowledge.
PIQA (Bisk et al., 2020) is a binary-choice QA dataset about physical commonsense.
- CoPA (Roemmele et al., 2011) is a commonsense inference dataset, to select the most plausible alternative with the causal relation to the premise.
Data Preprocessing. For CommonsenseQA and OpenBookQA, we use their original train/dev/test split settings. Since the test set of CommonsenseQA is not available, we follow previous work (Lin et al., 2019) that extracts 1,241 examples from the original training set as the test set. Besides, the test sets of SocialIQA and PIQA are not available. Therefore, we report the experimental results on their development sets for a fair comparison (Shwartz et al., 2020). For CoPA that only provides development and test sets, we follow Niu et al. (2021) to train models on the development set and evaluate the performance on the test set. For commonsense KG, we adopt ConceptNet (Speer et al., 2017), a general-domain and task-agnostic CSKG, as our external knowledge source  $\mathcal{G}$  for all the above models and tasks. For each question-candidate pair  $(q, c_i)$ , we follow previous works (Lin et al., 2019; Feng et al., 2020a) to retrieve and construct the subgraph  $\mathcal{G}^{q, c_i}$  from the CSKG  $\mathcal{G}$ .
Baseline Methods. We compare our model with the following six baseline methods, including a fine-tuned PTM and five PTM+GNN models:
- Fine-tuned PTM directly fine-tunes a PTM without using any CSKG. We use RoBERTa
large (Liu et al., 2019) for all tasks. Additionally, we also use BERT-large (Devlin et al., 2019) and AristoRoBERTa (Clark et al., 2020a) for OpenBookQA to evaluate the generality of our KG-encoder.