SlowGuess/NAACL_2021_OCR huggingface dataset:
Michihiro Yasunaga Hongyu Ren Antoine Bosselut
Percy Liang Jure Leskovec
{myasu,hyren,antoineb,pliang,jure}@cs.stanford.edu
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model,  $QA-GNN$ , which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and  $\mathrm{LM} + \mathrm{KG}$  models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.
Question answering systems must be able to access relevant knowledge and reason over it. Typically, knowledge can be implicitly encoded in large language models (LMs) pre-trained on unstructured text (Petroni et al., 2019; Bosselut et al., 2019), or explicitly represented in structured knowledge graphs (KGs), such as Freebase (Bollacker et al., 2008) and ConceptNet (Speer et al., 2017), where entities are represented as nodes and relations between them as edges. Recently, pre-trained LMs have demonstrated remarkable success in many question answering tasks (Liu et al., 2019; Raffel et al., 2020). However, while LMs have a broad coverage of knowledge, they do not empirically perform well on structured reasoning (e.g., handling negation) (Kassner and Schütze, 2020). On the other hand, KGs are more suited for structured reasoning (Ren et al., 2020; Ren and Leskovec, 2020) and enable explainable predictions e.g., by providing reasoning paths (Lin et al., 2019), but may lack coverage and
Figure 1: Given the QA context (question and answer choice; purple box), we aim to derive the answer by performing joint reasoning over the language and the knowledge graph (green box).
be noisy (Bordes et al., 2013; Guu et al., 2015). How to reason effectively with both sources of knowledge remains an important open problem.
Combining LMs and KGs for reasoning (henceforth, LM+KG) presents two challenges: given a QA context (e.g., question and answer choices; Figure 1 purple box), methods need to (i) identify informative knowledge from a large KG (green box); and (ii) capture the nuance of the QA context and the structure of the KGs to perform joint reasoning over these two sources of information. Previous works (Bao et al., 2016; Sun et al., 2018; Lin et al., 2019) retrieve a subgraph from the KG by taking topic entities (KG entities mentioned in the given QA context) and their few-hop neighbors. However, this introduces many entity nodes that are semantically irrelevant to the QA context, especially when the number of topic entities or hops increases. Additionally, existing LM+KG methods for reasoning (Lin et al., 2019; Wang et al., 2019a; Feng et al., 2020; Lv et al., 2020) treat the QA context and KG as two separate modalities. They individually apply LMs to the QA context and graph
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: Overview of our approach. Given a QA context  $(z)$ , we connect it with the retrieved KG to form a joint graph (working graph; §3.1), compute the relevance of each KG node conditioned on  $z$  ( $\S 3.2$ ; node shading indicates the relevance score), and perform reasoning on the working graph ( $\S 3.3$ ).
neural networks (GNNs) to the KG, and do not mutually update or unify their representations. This separation might limit their capability to perform structured reasoning, e.g., handling negation.
Here we propose QA-GNN, an end-to-end LM+KG model for question answering that addresses the above two challenges. We first encode the QA context using an LM, and retrieve a KG subgraph following prior works (Feng et al., 2020). Our QA-GNN has two key insights: (i) Relevance scoring: Since the KG subgraph consists of all few-hop neighbors of the topic entities, some entity nodes are more relevant than others with respect to the given QA context. We hence propose KG node relevance scoring: we score each entity on the KG subgraph by concatenating the entity with the QA context and calculating the likelihood using a pretrained LM. This presents a general framework to weight information on the KG; (ii) Joint reasoning: We design a joint graph representation of the QA context and KG, where we explicitly view the QA context as an additional node (QA context node) and connect it to the topic entities in the KG subgraph as shown in Figure 1. This joint graph, which we term the working graph, unifies the two modalities into one graph. We then augment the feature of each node with the relevance score, and design a new attention-based GNN module for reasoning. Our joint reasoning algorithm on the working graph simultaneously updates the representation of both the KG entities and the QA context node, bridging the gap between the two sources of information.
We evaluate QA-GNN on two question answering datasets that require reasoning with knowledge: CommonsenseQA (Talmor et al., 2019) and OpenBookQA (Mihaylov et al., 2018), using the ConceptNet KG (Speer et al., 2017). QA-GNN outperforms strong fine-tuned LM baselines as well as the existing best LM+KG model (with the same LM) by up to  $5.7\%$  and  $3.7\%$  respectively. In particular, QA-GNN exhibits improved performance on some forms of structured reasoning (e.g., correctly han
dling negation and entity substitution in questions): it achieves  $4.6\%$  improvement over fine-tuned LMs on questions with negation, while existing LM+KG models are  $+0.6\%$  over fine-tuned LMs. We also show that one can extract reasoning processes from QA-GNN in the form of general KG subgraphs, not just paths (Lin et al., 2019), suggesting a general method for explaining model predictions.
We aim to answer natural language questions using knowledge from a pre-trained LM and a structured KG. We use the term language model broadly to be any composition of two functions,  $f_{\mathrm{head}}(f_{\mathrm{enc}}(\mathbf{x}))$ , where  $f_{\mathrm{enc}}$ , the encoder, maps a textual input  $\mathbf{x}$  to a contextualized vector representation  $\mathbf{h}^{\mathrm{LM}}$ , and  $f_{\mathrm{head}}$  uses this representation to perform a desired task (which we discuss in §3.2). In this work, we specifically use masked language models (e.g., RoBERTa) as  $f_{\mathrm{enc}}$ , and let  $\mathbf{h}^{\mathrm{LM}}$  denote the output representation of a [CLS] token that is prepended to the input sequence  $\mathbf{x}$ , unless otherwise noted. We define the knowledge graph as a multi-relational graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ . Here  $\mathcal{V}$  is the set of entity nodes in the KG;  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{R} \times \mathcal{V}$  is the set of edges that connect nodes in  $\mathcal{V}$ , where  $\mathcal{R}$  represents a set of relation types.
Given a question  $q$  and an answer choice  $a \in \mathcal{C}$ , we follow prior work (Lin et al., 2019) to link the entities mentioned in the question and answer choice to the given KG  $\mathcal{G}$ . We denote  $\mathcal{V}_q \subseteq \mathcal{V}$  and  $\mathcal{V}_a \subseteq \mathcal{V}$  as the set of KG entities mentioned in the question (question entities; blue entities in Figure1) and answer choice (answer choice entities; red entities in Figure1), respectively, and use  $\mathcal{V}_{q,a} \coloneqq \mathcal{V}_q \cup \mathcal{V}_a$  to denote all the entities that appear in either the question or answer choice, which we call topic entities. We then extract a subgraph from  $\mathcal{G}$  for a question-choice pair,  $\mathcal{G}_{\mathrm{sub}}^{q,a} = (\mathcal{V}_{\mathrm{sub}}^{q,a}, \mathcal{E}_{\mathrm{sub}}^{q,a})^1$  which comprises all nodes on the  $k$ -hop paths between nodes in  $\mathcal{V}_{q,a}$ .
<sup>1</sup>We remove the superscript  $q,a$  if there is no ambiguity.
Figure 3: Relevance scoring of the retrieved KG: we use a pre-trained LM to calculate the relevance of each KG entity node conditioned on the QA context (§3.2).
3 Approach: QA-GNN
As shown in Figure 2, given a question and an answer choice  $a$ , we concatenate them to get the QA context  $[q; a]$ . To reason over a given QA context using knowledge from both the LM and the KG, QA-GNN works as follows. First, we use the LM to obtain a representation for the QA context, and retrieve the subgraph  $\mathcal{G}_{\mathrm{sub}}$  from the KG. Then we introduce a QA context node  $z$  that represents the QA context, and connect  $z$  to the topic entities  $\nu_{q,a}$  so that we have a joint graph over the two sources of knowledge, which we term the working graph,  $\mathcal{G}_{\mathrm{W}}$  ( $\S 3.1$ ). To adaptively capture the relationship between the QA context node and each of the other nodes in  $\mathcal{G}_{\mathrm{W}}$ , we calculate a relevance score for each pair using the LM, and use this score as an additional feature for each node ( $\S 3.2$ ). We then propose an attention-based GNN module that does message passing on the  $\mathcal{G}_{\mathrm{W}}$  for multiple rounds ( $\S 3.3$ ). Finally, we make the final prediction using the LM representation, QA context node representation and a pooled working graph representation ( $\S 3.4$ ).
3.1 Joint graph representation
To design a joint reasoning space for the two sources of knowledge, we explicitly connect them in a common graph structure. We introduce a new QA context node  $z$  which represents the QA context, and connect  $z$  to each topic entity in  $\mathcal{V}_{q,a}$  on the KG subgraph  $\mathcal{G}_{\mathrm{sub}}$  using two new relation types  $r_{z,q}$  and  $r_{z,a}$ . These relation types capture the relationship between the QA context and the relevant entities in the KG, depending on whether the entity is found in the question portion or the answer portion of the QA context. Since this joint graph intuitively provides a reasoning space (working memory) over
the QA context and KG, we term it working graph  $\mathcal{G}_{\mathrm{W}} = (\mathcal{V}_{\mathrm{W}},\mathcal{E}_{\mathrm{W}})$ , where  $\mathcal{V}_{\mathrm{W}} = \mathcal{V}_{\mathrm{sub}}\cup \{z\}$  and  $\mathcal{E}_{\mathrm{W}} = \mathcal{E}_{\mathrm{sub}}\cup \{(z,r_{z,q},v)|v\in \mathcal{V}_q\} \cup \{(z,r_{z,a},v)|v\in \mathcal{V}_a\}$ .
Each node in the  $\mathcal{G}_{\mathrm{W}}$  is associated with one of the four types:  $\mathcal{T} = \{\mathbf{Z},\mathbf{Q},\mathbf{A},\mathbf{O}\}$ , each indicating the context node  $z$ , nodes in  $\nu_{q}$ , nodes in  $\nu_{a}$ , and other nodes, respectively (corresponding to the node color, purple, blue, red, gray in Figure1 and 2). We denote the text of the context node  $z$  (QA context) and KG node  $v\in \nu_{\mathrm{sub}}$  (entity name) as  $\text{text}(z)$  and  $\text{text}(v)$ .
We initialize the node embedding for  $z$  using the LM representation of the QA context  $(z^{\mathrm{LM}} = f_{\mathrm{enc}}(\mathrm{text}(z)))$ , and each node on the  $\mathcal{G}_{\mathrm{sub}}$  using the entity embedding from Feng et al. (2020). In the subsequent sections, we will reason over the working graph in order to score a given (question, answer choice) pair.
3.2 KG node relevance scoring
Many nodes on the KG subgraph  $\mathcal{G}_{\mathrm{sub}}$  (i.e., those heuristically retrieved from the KG) can be irrelevant under the current QA context. As an example shown in Figure 3, the retrieved KG subgraph  $\mathcal{G}_{\mathrm{sub}}$  with few-hop neighbors of the  $\nu_{q,a}$  may include nodes that are uninformative for the reasoning process, e.g., nodes "holiday" and "river bank" are off-topic; "human" and "place" are generic. These irrelevant nodes may result in overfitting or introduce unnecessary difficulty in reasoning, an issue especially when  $\nu_{q,a}$  is large. For instance, we empirically find that using the ConceptNet KG (Speer et al., 2017), we will retrieve a KG with  $|\mathcal{V}_{\mathrm{sub}}| > 400$  nodes on average if we consider 3-hop neighbors.
In response, we propose node relevance scoring, where we use the pre-trained language model to score the relevance of each KG node  $v \in \mathcal{V}_{\mathrm{sub}}$
conditioned on the QA context. For each node  $v$ , we concatenate the entity  $\text{text}(v)$  with the QA context  $\text{text}(z)$  and compute the relevance score:
$$
\rho_ {v} = f _ {\text {h e a d}} \left(f _ {\text {e n c}} \left(\left[ \text {t e x t} (z); \text {t e x t} (v) \right]\right)\right), \tag {1}
$$
where  $f_{\mathrm{head}} \circ f_{\mathrm{enc}}$  denotes the probability of  $\text{text}(v)$  computed by the LM. This relevance score  $\rho_v$  captures the importance of each KG node relative to the given QA context, which is used for reasoning or pruning the working graph  $\mathcal{G}_{\mathrm{W}}$ .
3.3 GNN architecture
To perform reasoning on the working graph  $\mathcal{G}_{\mathrm{W}}$ , our GNN module builds on the graph attention framework (GAT) (Velicković et al., 2018), which induces node representations via iterative message passing between neighbors on the graph. Specifically, in a  $L$ -layer QA-GNN, for each layer, we update the representation  $\pmb{h}_t^{(\ell)} \in \mathbb{R}^D$  of each node  $t \in \mathcal{V}_{\mathrm{W}}$  by
$$
\boldsymbol {h} _ {t} ^ {(\ell + 1)} = f _ {n} \left(\sum_ {s \in \mathcal {N} _ {t} \cup \{t \}} \alpha_ {s t} \boldsymbol {m} _ {s t}\right) + \boldsymbol {h} _ {t} ^ {(\ell)}, \tag {2}
$$
where  $\mathcal{N}_t$  represents the neighborhood of node  $t$ ,  $\pmb{m}_{st} \in \mathbb{R}^D$  notes the message from each neighbor node  $s$  to  $t$ , and  $\alpha_{st}$  is an attention weight that scales each message  $\pmb{m}_{st}$  from  $s$  to  $t$ . The sum of the messages is then passed through a 2-layer MLP,  $f_n: \mathbb{R}^D \to \mathbb{R}^D$ , with batch normalization (Ioffe and Szegedy, 2015). For each node  $t \in \mathcal{V}_{\mathrm{W}}$ , we set  $\pmb{h}_t^{(0)}$  using a linear transformation  $f_h$  that maps its initial node embedding (described in §3.1) to  $\mathbb{R}^D$ . Crucially, as our GNN message passing operates on the working graph, it will jointly leverage and update the representation of the QA context and KG. We further propose an expressive message  $(\pmb{m}_{st})$  and attention  $(\alpha_{st})$  computation below.
Node type & relation-aware message. As  $\mathcal{G}_{\mathrm{W}}$  is a multi-relational graph, the message passed from a source node to the target node should capture their relationship, i.e., relation type of the edge and source/target node types. To this end, we first obtain the type embedding  $\pmb{u}_t$  of each node  $t$ , as well as the relation embedding  $\pmb{r}_{st}$  from node  $s$  to node  $t$  by
$$
\boldsymbol {u} _ {t} = f _ {u} \left(u _ {t}\right), \quad \boldsymbol {r} _ {s t} = f _ {r} \left(e _ {s t}, u _ {s}, u _ {t}\right), \tag {3}
$$
where  $\mathfrak{u}_s,\mathfrak{u}_t\in \{0,1\}^{|\mathcal{T}|}$  are one-hot vectors indicating the node types of  $s$  and  $t$ ,  $\mathbf{e}_{st}\in \{0,1\}^{|\mathcal{R}|}$  is a one-hot vector indicating the relation type of edge  $(s,t)$ ,  $f_{u}\colon \mathbb{R}^{|\mathcal{T}|}\to \mathbb{R}^{D / 2}$  is a linear transformation, and  $f_{r}\colon \mathbb{R}^{|\mathcal{R}| + 2|\mathcal{T}|}\to \mathbb{R}^{D}$  is a 2-layer MLP. We then compute the message from  $s$  to  $t$  as
$$
\boldsymbol {m} _ {s t} = f _ {m} \left(\boldsymbol {h} _ {s} ^ {(\ell)}, \boldsymbol {u} _ {s}, \boldsymbol {r} _ {s t}\right), \tag {4}
$$
where  $f_{m}\colon \mathbb{R}^{2.5D}\to \mathbb{R}^{D}$  is a linear transformation.
Node type, relation, and score-aware attention. Attention captures the strength of association between two nodes, which is ideally informed by their node types, relations and node relevance scores.
We first embed the relevance score of each node  $t$  by
$$
\rho_ {t} = f _ {\rho} \left(\rho_ {t}\right), \tag {5}
$$
where  $f_{\rho}\colon \mathbb{R}\to \mathbb{R}^{D / 2}$  is an MLP. To compute the attention weight  $\alpha_{st}$  from node  $s$  to node  $t$ , we obtain the query and key vectors  $\pmb{q},\pmb{k}$  by
$$
\boldsymbol {q} _ {s} = f _ {q} \left(\boldsymbol {h} _ {s} ^ {(\ell)}, \boldsymbol {u} _ {s}, \boldsymbol {\rho} _ {s}\right), \tag {6}
$$
$$
\boldsymbol {k} _ {t} = f _ {k} \left(\boldsymbol {h} _ {t} ^ {(\ell)}, \boldsymbol {u} _ {t}, \boldsymbol {\rho} _ {t}, \boldsymbol {r} _ {s t}\right), \tag {7}
$$
where  $f_{q}\colon \mathbb{R}^{2D}\to \mathbb{R}^{D}$  and  $f_{k}\colon \mathbb{R}^{3D}\to \mathbb{R}^{D}$  are linear transformations. The attention weight is then
$$
\alpha_ {s t} = \frac {\exp (\gamma_ {s t})}{\sum_ {t ^ {\prime} \in \mathcal {N} _ {s} \cup \{s \}} \exp (\gamma_ {s t ^ {\prime}})}, \gamma_ {s t} = \frac {\boldsymbol {q} _ {s} ^ {\top} \boldsymbol {k} _ {t}}{\sqrt {D}}. \tag {8}
$$
3.4 Inference & Learning
Given a question  $q$  and an answer choice  $a$ , we use the information from both the QA context and the KG to calculate the probability of it being the answer  $p(a|q) \propto \exp(\mathrm{MLP}(z^{\mathrm{LM}}, z^{\mathrm{GNN}}, g))$ , where  $z^{\mathrm{GNN}} = h_z^{(L)}$  and  $g$  denotes the pooling of  $\{h_v^{(L)} | v \in \mathcal{V}_{\mathrm{sub}}\}$ . In the training data, each question has a set of answer choices with one correct choice. We optimize the model (both the LM and GNN components end-to-end) using the cross entropy loss.
3.5 Computation complexity
We analyze the time and space complexity of our method and compare with prior works, KagNet (Lin et al., 2019) and MHGRN (Feng et al., 2020) in Table 1. As we handle edges of different relation types using different edge embeddings instead of designing an independent graph networks for each relation as in RGCN (Schlichtkrull et al., 2018) or MHGRN, the time complexity of our method is constant with respect to the number of relations and linear with respect to the number of nodes. We achieve the same space complexity as MHGRN (Feng et al., 2020).
We evaluate QA-GNN on two question answering datasets: CommonsenseQA (Talmor et al., 2019) and OpenBookQA (Mihaylov et al., 2018). CommonsenseQA is a 5-way multiple choice QA task that requires reasoning with commonsense knowledge, containing 12,102 questions. The test set of CommonsenseQA is not publicly available, and model predictions can only be evaluated once every two weeks via the official leaderboard. Hence,
Model	Time	Space
G is a dense graph
L-hop KagNet	O(|R|L|V|L+1L)	O(|R|L|V|L+1L)
L-hop MHGRN	O(|R|2|V|2L)	O(|R||V|L)
L-layer QA-GNN	O(|V|2L)	O(|R||V|L)
G is a sparse graph with maximum node degree Δ≪|V|
L-hop KagNet	O(|R|L|V|LΔL)	O(|R|L|V|LΔL)
L-hop MHGRN	O(|R|2|V|LΔ)	O(|R||V|L)
L-layer QA-GNN	O(|V|LΔ)	O(|R||V|L)
we perform main experiments on the in-house (IH) data split used in Lin et al. (2019), and also report the score of our final system on the official test set. OpenBookQA is a 4-way multiple choice QA task that requires reasoning with elementary science knowledge, containing 5,957 questions. We use the official data split.
4.2 Knowledge graphs
We use ConceptNet (Speer et al., 2017), a general-domain knowledge graph, as our structured knowledge source  $\mathcal{G}$  for both of the above tasks. Given each QA context (question and answer choice), we retrieve the subgraph  $\mathcal{G}_{\mathrm{sub}}$  from  $\mathcal{G}$  following the pre-processing step described in Feng et al. (2020), with hop size  $k = 2$ . Henceforth, in this section ( $\S 4$ ) we use the term "KG" to refer to  $\mathcal{G}_{\mathrm{sub}}$ .
4.3 Implementation & training details
We set the dimension  $(D = 200)$  and number of layers  $(L = 5)$  of our GNN module, with dropout rate 0.2 applied to each layer (Srivastava et al., 2014). The parameters of the model are optimized by RAdam (Liu et al., 2020), with batch size 128, gradient clipping 1.0 (Pascanu et al., 2013), and learning rate 1e-5 and 1e-3 for the LM and GNN components respectively. Each model is trained using two GPUs (GTX Titan X), which takes  $\sim 20$  hours on average. The above hyperparameters were tuned on the development set.
Fine-tuned LM. To study the role of KGs, we compare with a vanilla fine-tuned LM, which does not use the KG. We use RoBERTa-large (Liu et al., 2019) for CommonsenseQA, and RoBERTa-large and AristoRoBERTa² (Clark et al., 2019) for
Table 1: Computation complexity of different  $L$ -hop reasoning models on a dense/sparse graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  with the relation set  $\mathcal{R}$ .
Methods	IHdev-Acc. (%)	IHtest-Acc. (%)
RoBERTa-large (w/o KG)	73.07 (±0.45)	68.69 (±0.56)
+ RGCN (Schlichtkrull et al., 2018)	72.69 (±0.19)	68.41 (±0.66)
+ GconAttn (Wang et al., 2019a)	72.61 (±0.39)	68.59 (±0.96)
+ KagNet (Lin et al., 2019)	73.47 (±0.22)	69.01 (±0.76)
+ RN (Santoro et al., 2017)	74.57 (±0.91)	69.08 (±0.21)
+ MHGRN (Feng et al., 2020)	74.45 (±0.10)	71.11 (±0.81)
+ QA-GNN (Ours)	76.54 (±0.21)	73.41 (±0.92)
Table 2: Performance comparison on Commonsense QA in-house split (controlled experiments). As the official test is hidden, here we report the in-house Dev (IHdev) and Test (IHtest) accuracy, following the data split of Lin et al. (2019).
Methods	Test
RoBERTa (Liu et al., 2019)	72.1
RoBERTa+FreeLB (Zhu et al., 2020) (ensemble)	73.1
RoBERTa+HyKAS (Ma et al., 2019)	73.2
RoBERTa+KE (ensemble)	73.3
RoBERTa+KEDGN (ensemble)	74.4
XLNet+GraphReason (Lv et al., 2020)	75.3
RoBERTa+MHGRN (Feng et al., 2020)	75.4
Albert+PG (Wang et al., 2020b)	75.6
Albert (Lan et al., 2020) (ensemble)	76.5
UnifiedQA* (Khashabi et al., 2020)	79.1
RoBERTa + QA-GNN (Ours)	76.1
Table 3: Test accuracy on CommonsenseQA's official leaderboard. The top system, UnifiedQA (11B parameters) is  ${30}\mathrm{x}$  larger than our model.
OpenBookQA.
Existing LM+KG models. We compare with existing LM+KG methods, which share the same high-level framework as ours but use different modules to reason on the KG in place of QA-GNN ("yellow box" in Figure2): (1) Relation Network (RN) (Santoro et al., 2017), (2) RGCN (Schlichtkrull et al., 2018), (3) GconAttn (Wang et al., 2019a), (4) KagNet (Lin et al., 2019), and (5) MHGRN (Feng et al., 2020). (1),(2),(3) are relation-aware GNNs for KGs, and (4),(5) further model paths in KGs. MHGRN is the existing top performance model under this LM+KG framework. For fair comparison, we use the same LM in all the baselines and our model. The key differences between QA-GNN and these are that they do not perform relevance scoring or joint updates with the QA context ( $\S 3$ ).
Table 2 and Table 4 show the results on CommonsenseQA and OpenBookQA, respectively. On both datasets, we observe consistent improvements over fine-tuned LMs and existing LM+KG models, e.g., on OpenBookQA,  $+5.7\%$  over RoBERTa, and  $+3.7\%$  over the prior best LM+KG system,
${}^{2}$  OpenBookQA provides an extra corpus of scientific facts in a textual form. AristoRoBERTa uses the facts corresponding to each question, prepared by Clark et al. (2019), as an
additional input to the QA context.
Methods	RoBERTa-large	AristoRoBERTa
Fine-tuned LMs (w/o KG)	64.80 (±2.37)	78.40 (±1.64)
+ RGCN	62.45 (±1.57)	74.60 (±2.53)
+ GconAtten	64.75 (±1.48)	71.80 (±1.21)
+ RN	65.20 (±1.18)	75.35 (±1.39)
+ MHGRN	66.85 (±1.19)	80.6
+ QA-GNN (Ours)	70.58 (±1.42)	82.77 (±1.56)
Table 4: Test accuracy comparison on OpenBook QA (controlled experiments). Methods with AristotleBERTa use the textual evidence by Clark et al. (2019) as an additional input to the QA context.
Methods	Test
Careful Selection (Banerjee et al., 2019)	72.0
AristoRoBERTa	77.8
KF + SIR (Banerjee and Baral, 2020)	80.0
AristoRoBERTa + PG (Wang et al., 2020b)	80.2
AristoRoBERTa + MHGRN (Feng et al., 2020)	80.6
Albert + KB	81.0
T5* (Raffel et al., 2020)	83.2
UnifiedQA* (Khashabi et al., 2020)	87.2
AristoRoBERTa + QA-GNN (Ours)	82.8