GreaseLM: Graph REASoning Enhanced Language Models
Authors: Xikun Zhang
Antoine Bosselut
Michihiro Yasunaga
Hongyu Ren
Percy Liang
Christopher D Manning
Jure Leskovec
Venue: ICLR 2022 Spotlight (2022)
Keywords: language models
commonsense
question answering
knowledge graphs
KG augmentation

Abstract: Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.

Published as a conference paper at ICLR 2022
GREASELM: GRAPH REASONING ENHANCED LANGUAGE MODELS FOR QUESTION ANSWERING
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren
Percy Liang, Christopher D. Manning, Jure Leskovec
Stanford University [xikunz2, antoineb,,masu,, hyren, pllppliii pliang, manning,
stanford.edi
ABSTRACT
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pre- trained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains open question how to effectively fuse and reason over the KG representations and the language con- text, which provides situational constraints and nuances. In this work, we propose GREASELM, new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction op- erations. Information from both modalities propagates to the other, allowing lan- guage context representations to grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks the com- monsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical ques- tion answering (i.e., MedQA- more reliably answer questions that require reasoning both situational constraints and structured knowledge, even outperforming models 8x larger!"
domains demonstrate that GREASELM
INTRODUCTION
Question answering is a challenging task that requires complex reasoning over both explicit con- straints described the textual context of the question, as well as unstated, relevant knowledge about the world (i.e., knowledge about the domain of interest). Recently, large pretrained language models fine-tuned on QA datasets have become the dominant paradigm NLP for question answer- ing tasks (Khashabi al. 2020). After pretraining on extreme-scale collection of general text corpora, these language models learn to implicitly encode broad knowledge about the world, which they are able to leverage when fine-tuned
domain-specifer downstream QA task. However, despite the strong performance this two-stage learning procedure on common benchmarks, these models struggle when given examples that are distributionally different from examples seen during fine-tuning (McCoy et al. 2019). Their learned behavior often relies on simple (at times spurious) patterns offer shortcuts to answer, rather than robust, structured reasoning that effectively fuses the explicit information provided by the context and implicit external knowledge (Marcus 2018).
On the other hand, massive knowledge graphs (KG), such as Freebase (Bollacker et 2008). Wiki- data (Vrandečić & Krötzsch 2014). ConceptNet (Speer et al. 2017). and Yago (Suchanck et al.) 2007) capture such external knowledge explicitly using triplets that capture relationships between entities. Previous research has demonstrated the significant role KGs can play in structured rea- soning and query answering (Ren et al. 2020 202T Ren & Leskovee 2020). However, extending these reasoning advantages to general QA (where questions and answers are expressed in natural language and not easily mapped to strict logical queries) requires finding the right integration of knowledge from the KG with the information and constraints provided by the QA example. Prior
All code, data and pretrained models are available at https: //gi thub. com/ snap- stanford/
Greasel.

---

Published as a conference paper at ICLR 2022
Answer Selection
ML
Laver
Japer
CrewnatM Layer X2d
Cross modal Ruser
Grespel laper
Mint
Greasct Layer
X.
Unimodil
UM Loyer
LM Larprr
GNN Layer
[1N1] if uatuaed for hair, round brusht ofample of what? (SFP) pepppppe (SPP)
IRB
Figure I: GREASELM Architecture. The textual context is appended with special interaction token and passed through N LM-based unimodal encoding layers. Simultancously. a local KG of relevant knowledge is extracted and connected to interaction node. In the later GREASELM lay- ers, the language representation continues to be updated through LM layers and the KG is processed using GNN, simulating reasoning over its knowledge. In each layer, after each modality's 's repre- sentation is updated, the representations of the interaction token and node are pulled, concatenated, and passed through modality interaction (MInt) unit mix their representations. In subsequent layers, the mixed information from the interaction elements mixes with their respective modalities, allowing knowledge from the KG to affect the representations of individual tokens, and context from language to affect fine-grained entity knowledge representations in the GNN.
methods propose various ways leverage both modalities (i.e., expressive large language models and structured KGs) for improved reasoning (Mihaylov & Frank 2018 Lin et al.) 2019 Feng al.) 2020 However, these methods typically fuse the two modalities in a shallow and non-interactive manner, encoding both separately and fusing them at the output for prediction, or using one to augment the input of the other. Consequently, previous methods demonstrate restricted capacity to exchange useful information between the two modalities. It remains open question how to effec- tively fuse the KG and LM representations can interact in non- non-shallow way to simulate structured, situational reasoning.
truly unified manner, where the two representations
In this work, we present GREASELM. a new model that enables fusion and exchange of informa- tion from both the LM and KG in multiple layers of its architecture (see Figure 1). Our proposed GREASEL.M consists of an LM that takes as input the natural language context, as well as graph neural network (GNN) that reasons over the KG. After each layer of the LM and GNN, we design an interactive scheme to bidirectionally transfer the information from each modality to the other through specially initialized interaction representations (i.e., interaction token for the LM; interac- tion node for the GNN). In such way, all the tokens in the language context receive information from the KG entities through the interaction token and the KG entities indirectly interact with the tokens through the interaction node. By such a deep integration across all layers, GREASELM en- ables joint reasoning over both the language context and the KG entities under a unified framework agnostic to the specific language model graph neural network, that both modalities can be contextualized by the other.
GREASELM demonstrates significant performance gains across different LM architectures. We perform experiments on several standard QA benchmarks: CommonsenseQA, OpenbookQA and MeddQA-UMMEE which require external knowledge across different domains (commonsense soning and medical reasoning) and use different KGs (ConceptNet and Disease Database). Across both domains, GREASELM outperforms comparably-sized prior QA models, including strong fine-
2

---

Published as a conference paper at ICLR 2022
tuned LM baselines (by 5.5%, 6.6%, and 1.3%, respectively) and state-of-the-art KG+LM models (by 0.9%. 1.8%, and 0.5%, respectively) on the three competitive benchmarks. Furthermore, with the deep fusion of both modalities, GREASELM exhibits strong performance over baselines on ques- tions that exhibit textual nuance, such as resolving multiple constraints, negation, and hedges, and which require effective reasoning over both language context and KG.
2 RELATED WORK
Integrating KG information has become a popular research area for improving neural QA systems. Some works explore using two-tower models to answer questions, where a graph representation of knowledge and language representation are fused with no interaction between them (Wang al., 2019). Other works seek to use one modality to ground the other, such as using an encoded represen- tation of a linked KG to augment the textual representation of QA example (e.g., Knowledgeable Reader, Mihaylov & Frank 2018) KagNet, [Lin et al.) 2019) KT-NET, Yang et al. (219)). Others re- verse the flow of information and use a representation of the text (e.g. final layer of LM) provide an augmentation to a graph reasoning model over an extracted KG for the example (e.g., MHGRN, Feng et al. 2020 Lv al.) 2020). In all of these settings, however, the interaction between both modalities is limited as information between them only flows one way.
More recent approaches explore deeper integrations of both modalities. Certain approaches learn to access implicit knowledge encoded in LMs (Bosselut et al. 2019 Petroni et al. 2019 Hwang et al., 202T] by training on structured KG data, and then use the LM to generate local KGs that can be used for QA (Wang et al.) 2020; Bosselut et al.) 2021). However, these approaches discard the static KG once they train the LM on its facts, losing important structure that can guide reasoning. More recently, QA-GNN (Yasunaga et al. 2021) proposed to jointly update the LM and GNN represen- tations via message passing. However, they use a single pooled representation of the LM to seed the textual component of this joint structure, limiting the updates that can be made to the textual representation. In contrast to prior works, we propose to make individual token representations in the LM and node representations in the GNN mix for multiple layers, enabling representations of both modalities to reflect particularities of the other (c.g., knowledge grounds language; language nuances specifies which knowledge is important). Simultaneously, we retain the individual structure of both modalities, which we demonstrate improves QA performance substantially ($5).
Additionally, some works explore integrating knowledge graphs with language models in the pre- training stage. However, much like for QA, the modality interaction is typically limited to knowl- edge feeding language (Zhang al., 2019 Shen et al. 2020 Yu al. 2020), rather than designing interactions across multiple layers. [Sun et al. (2020)s s work is perhaps most similar, but they do not use the same interaction bottleneck, requiring high-precision entity mention spans for linking, and they limit expressivity through shared modality parameters for the LM and KG.
PROPOSED APPROACH: GREASELM
In this work, we augment large-scale language models (Devlin et al. 2019; Liu et al. 2019 Lan al., 20200 Liu al. 202I) with graph reasoning modules over KGs. Our method, GREASELM (depicted in Figure consists of two stacked components: (1) a set of unimodal LM layers which learn an initial representation of the input tokens, and (2) a set of upper cross-modal GREASELM layers which learn to jointly represent the language sequence and linked knowledge graph, allowing textual representations formed from the underlying LM layers and a graph representation of the KG to mix with one another. We denote the number of LM layers as N, and the number of GREASELM layers as M. The total number of layers in our model is N + M.
Notation. In the task of multiple choice question answering (MCQA), generic MCQA-type dataset consists of examples with a context paragraph c, a question q and a candidate answer set A, all expressed in text. In this work, we also assume access to external knowledge graph (KG) G that provides background knowledge that is relevant to the content of the multiple choice questions.
Given QA example (c, q. A), and the KG G as input, our goal is to identify which answer E A is correct. Without loss of generality, when an operation is applied to an arbitrary answer, we refer to that answer a. We denote sequence of tokens in natural language as { {wi,...; wr), where 7' is
3

---

Published as a conference paper at ICLR 2022
the total number of tokens, and the representation of token wy from the f-th layer of the model as hy (5) We denote a set of nodes from the KG (e1,-..; est, where J the total number of nodes, (8) and the representation of a node ej in the E-th layer of the model ej
3.1 INPUT REPRESENTATION
We concatenate our context paragraph c, question q. and candidate answer a with separator tokens get our model input [c; q: a and tokenize the combined sequence into {wi..... Wr). Second, we use the input sequence to retrieve subgraph of the KG & (denoted Guub), which provides knowledge from the KG that is relevant to this QA example. We denote the set of nodes Ssub as {ei,...,e e,)
KG Retrieval. Given each QA context, we follow the procedure from Yasunaga et al. 2021 to retrieve the subgraph Gauh from G. We describe this procedure in Appendix B. Each node in Gub is assigned a type based whether its corresponding entity was linked from the context question g, answer a, or as neighbor these nodes. the rest the paper, we use "KG" to refer to Ssub-
Interaction Bottlenecks. In the cross-modke GREASELM layers, information is fused between both modalities, for which we define a special interaction token Want and a special interaction node Eint whose representations serve the bottlenecks through which the two modalities interact ($3.3). We prepend Wint to the token sequence and connect Eint to all the linked nodes Viinked in Guub-
3.2 LANGUAGE PRE-ENCODING
In the unimodal encoding component, given the sequence of tokens {Wints Wi...., WTT, we first sum the token, segment, and positional embeddings for each token to compute its (=0 input repre- sentation (h snts h ,T
.....
(0) ), and then compute an output representation for each layer
(h (0
int: (0) ....,,,()) 1 -LM-Layer((h (h((()) int for l = 1,...,N NN
(e-1) h (6-1) ....., (1111))
where LM-Layer(-) is single LM encoder layer, whose parameters are initialized using a pretrained model ($..). We refer readers to Vaswani et al. (2017) for technical details of these layers.
3.3 GREASELM
GREASELM cross-modal fusion component to inject information from the KG into language representations and information from language into KG representations. The GREASELM layer designed separately encode information from both modalities, and fuse their representations using the bottleneck of the special interaction token and node. It is comprised of three components: (1) a transformer LM encoder block which continues to encode the language context, (2) GNN layer that reasons over KG entities and relations, and (3) modality interaction layer that takes the unimodal representations the interaction token and interaction node and exchanges information through them. We discuss these three components below.
In L-th GREASELM layer, the input token embeddings (W+2-1) 1) . (N++-1) ..... h1 ((++E-1) are fed into additional transformer LM encoder blocks that continue encode the textual context based on the LM's pretrained representations:
Language Representation.
(h ing
(h (N+0) int
R (N++)
hy E(N+0) L.M-Layer(( (th (N+C-1 h (N+E-1)
int
h (N+e-11))
(2)
for f = 1.....,MM M
where h corresponds to fre-fused embeddings of the language modality. As we will discuss below, because h. h N will encode information received from the knowledge graph representation, these late language encoding layers will also allow the token representations to mix with KG knowledge.
Graph Representation. The GREASELM layers also encode representation of the local KG Gub linked from the QA example. To represent the graph, we first compute initial node embeddings (e)") ..... (0)) for the retrieved entities using pretrained KG embeddings for these nodes ( ($4. The initial embedding of the interaction node el is initialized randomly.
4

---

Published as a conference paper at ICLR 2022
Then, in each layer of the GNN, the current representation of the node embeddings (e((=") Gj (((-))
e) o fed into the layer to perform a round of information propagation between nodes in the graph and yield pre-fused node embeddings for each entity;
t int. e
e = GNNN(fe (e-1) (f-1) snd : _e1
ej 1)
(3)
for I...,,
where GNN corresponds to a variant of graph attention networks (Veličković et al. 2018) that is simplification of the method of Yasunaga et al. (202I). The GNN computes node representations (8) for each node ej (eI,...,es] ,s) via message passing between neighbors on the graph.
e = fn
E LeSENe
ej ((-1)
Maj
where Ne, represents the neighborhood of an arbitrary node ej, Maj denotes the message one of its neighbors e, passes to ej. asj is an attention weight that scales the message maj, and fn a 2-layer ML.. The messages maj between nodes allow entity information from node to affect the model's representation of neighbors, and are computed in the following manner:
Txj = (r Us Uj)
(5)
m,j = (m((! (t-1), Us rsy)
(6)
where Us U, are node type embeddings, Paj is a relation embedding for the relation connecting es and ej. fr is 2-layer MLP, and fm is linear transformation. The attention weights æ.j scale the contribution of each neighbor's message by its importance, and are computed as follows:
9s fe(e!--1) u.)
(7)
kj = fu(ee)-1) exp(Tee)
Uj. rsj)
(8)
Yay 9: kj
(9)
Caj E En,EN,, exp(1.s)
(10)
VD
where fg and f& are linear transformations and ug Uj Paj are defined the same as above.
As discussed in the following paragraph, message passing between the interaction node eint and the nodes from the retrieved subgraph will allow information from that ein receives from Wine to propagate to the other nodes in the graph.
Modality Interaction. Finally, after using transformer LM layer and GNN layer to update token embeddings and node embeddings respectively, we use a modality interaction layer (MInt) to let the two modalities fuse information through the bottleneck of the interaction token Wint and the interaction node eint- We concatenate the pre-fused embeddings of the interaction token h h and interaction node e(ع pass the joint representation through a mixing operation (MInt), and then split the output post-fused embeddings into h int and (1) e
(8) (1) [h inti end
MInt( mnt (
(11)
We two-layer MLP our MInt operation, though other fusion operators could be used mix the representation. All the tokens other than the interaction token Wint and all the nodes other than the interaction node eint are not involved in the modality interaction process: w( = (t( for w e (wi...., } and e(f) = ε(() for e E {ei,..., e. However, they receive information from the in- teraction representations ((9) and ein in the next layers of their respective modal propagation (i.e., 3) Consequently, across multiple GREASELM layers, information propagates between both modalities (see Fig. [ for visual depiction), grounding language representations to KG knowledge, and knowledge representations to contextual constraints.
Learning & Inference. For the MCQA task, given a question and an answer a from all the candidates A, compute the probability of being the correct answer p(a q, c) x e, e (N+M) as a query. We optimize the whole model end-to-end using the cross entropy loss. At inference time, we predict the most plausible answer as arg max p(a q.c).
(M) exp(MLP(h (N+M). (M) g)), where g denotes attention-based pooling of (e]M) int
"e.....,ey)) using h (
5

---

Published as a conference paper at ICLR 2022
Dataset
Example
A weasel has thin body and short legs easier burrow after prey in what?
CommonsenscQA
(A) tree (B) mulberry bush (C) chicken coop (D) viking ship (E) rabbit warren
Which of these would let the most heat travel through?
OpenbookQA
(A) new pair of jeans
(B) a steel spoon in a cafeteria (C) cotton candy at store (D) a calvin klein cotton hat
A 57-ycar-old man presents to his primary care physician with 2-month history of right upper and lower extremity weakness. He noticed the weakness when started falling far more frequently while running errands. Since then, he has had increasing difficulty with walking and lifting objects, His past medical history is significant only for well-controlled hypertension, but he says that some members his family have had musculoskeletal problems. His right upper extremity shows forcarm atrophy and depressed reflexes while his right lower extremity is hypertonic with a positive Babinski sign. Which of the following is most likely associated with cause of this patients symptoms? (A) HLA-B8 haplotype
MedQA-USMLE
(B) HLA-DR2 haplotype
(C) Mutation in SOD1
(D) Mutation in SMN1
Table 1: Examples of the MCQA task for each of the datasets evaluated in this work.
4 EXPERIMENTAL. SETUP
We evaluate GREASELM on three diverse multiple-choice question answering datasets across two domains: CommonsenseeA (Talmor et al. 2019) and OpenBookQA (Mihaylov al. 2018) com- monsense reasoning benchmarks, and MedQA- (Jin et al. 2021) as a clinical QA task.
Commonsense(AA is a 5-way multiple-choice question answering dataset of 12.102 questions that require background commonsense knowledge beyond surface language understanding. We perform our experiments using the in-house data split of Lin et al. (2019) to compare baseline methods.
Openbook QA is a 4-w multiple-choies question answering dataset that tests elementary scientific knowledge. It contains 5,957 questions along with an open book of scientific facts. We use the official data splits from Mihaylov & Frank (2018).
MedQA-USMIEE is a 4-way multiple-choice question answering dataset, which requires biomed- ical and clinical knowledge. The questions are originally from practice tests for the United States Medical License Exams (USMLE). The dataset contains 12,723 questions. We use the original data splits from Jin et al. 2021)
4.1 IMPLEMENTATION & TRAINING DETAILS
Language Models. We seed GREASELM with RoBERTa-Laan Large Liu et al. 2019) for our exper- iments on CommonsenseQA, ArERtoRoBERTa (Clark et al. 2019) for our experiments on Open- bookQA, and SapBERT (Liu et al. 202T) for our experiments on MedQA-USMIL demonstrating GREASELM'ss generality with respect to language model initializations. Hyperparameters for train- ing these models can be found in Appendix Table 7
Knowledge Graphs. We use ConceptNer (Speer et al. 2017). a general-domain knowledge graph, as external knowledge source G for both CommonsenseQA and OpenbookQA. It has 799 22273 nodes and 2,487,810 edges in total. For MedQA-USMLE, we use a self-constructed knowledge graph that integrates the Disease Database portion of the Unified Medical Language System (UMLS; Bodenreider] 2004) and DrugBank (Wishart et al.) 2018). The knowledge graph contains 9,958 nodes and 44,561 edges. Additional information about node initialization and hyperparameters for preprocessing these KGs can be found in Appendix B.2
4.2 BASELINE METHODS
Fine-tuned LMs. To study the effect of using KGs as external knowledge sources, we compare method with vanilla fine-tuned LMs, which knowledge-ggnostic. We fine-tune RoBERTa-

---

Published as a conference paper at ICLR 2022
Table Performance comparison on Commonsense QA in-house split (controlled experiments). As the official test is hidden, here we report the in-house Dev (IHdev) and Test (IHtest) accuracy, following the data split of Lin et al. (211). Experiments are controlled using same seed LMM
Methods
IHdev-Ace. (%)
IHtest-Acc. (%)
RoBERTa-LLarge (w/o KG)
73.1 (+0.5)
68.7 (+0.6)
RGCN (Schlichtkrull al. 2018)
72.7 (#0.2)
68.4 (#0.7)
GconAtun (Wang et al. 2019)
72.6 (#0.4)
68.6 (+1.0)
KagNet (Lin et 201
73.5 (#0.2)
69.0 (#0.8)
RN (Santoro et al. 2017)
74.6
69.1 (#:0.2)
MHGRN (Feng et al.
2020
74.5 (+0.1)
71.1 (+.0.)
QA-GNN (Yasunaga el al. 2021
76.5 (+.0.)
73.4 (+0.9)
GREASELM (Ours)
78.5 (+-0.5
74.2 (+.0.4)
Table 3: Test Accuracy comparison on OpenBook QA. Experiments are controlled using the same seed LM for all LM+KG methods.
Table 4: Test accuracy comparison to public Open- BookQA model implementations. *UnifieQQA (1BB
params) and T5 (3B) are 30x and 8x larger than our model.
Model
Acc. # Params
Model
Acc.
ALBERT (Lan et al.
2020) + KB
81.0
235M
HGN Yan et al. 2020)
81.4
2355M
Arist (no KG)
78.4 .4.4
AMR-SG (Xu et al.
2021
81.6
~361M
+ RGCN
74.6
ALBERT + KPG Wang
2020
81.8
2235M
+ GconAttn
71.8
QA-GNN Yasunaga al. 2021
82.8
360M
+ RN
75.4
T5" (Raffel et al. 2020)
83.2
<B
+ MHGRN
80.6
T5 KB (Pirtoaca)
85.4
>1
QA-GNN
82.8
UnifiedQA KKashabi et al.
2020
87.2
1BB
GREASELM (Ours)
84.8
GREASELM (Ours)
84.8
~559M
Large (Luu al. 2019) for CommonsenseQA, and AristoRoBERT: Clark al. 2019 for Open- BAokQA. For USMLE, we use a state-of-the-an biomedical language model, SapBERT (Liu al. 2021), which is augmentation of PubmedBERT (Gu et al.) 2022) that is trained with entity disambiguation objectives to allow the model to better understand entity knowledge.
LM+KG models. We also evaluate GREASELM's s ability to exploit its knowledge graph augmen-
tation by comparing with existing LM+KG methods: (1) Relation Network (RN; Santoro er al.,
2017). (2) RGCN (Schlichtkrull al. 2018), (3) GconAttn (Wang al.) 2019). (4) KagNet (Lin et al. 2019), (5) MHGRN (Feng et al. 2020). and (6) QA-GNN (Yasunaga et al. (021). QA-GNN
the existing top-performing model under this LM+KG paradigm. The key difference between GREASELM and these baseline methods is that they do not fuse the representations of both modal- ities across multiple interaction layers, allowing the representation of both modalities to affect the other 33.3. For fair comparison, we use the same LM to initialize these baselines as for our model.
EXPERIMENTAL RESULTS
Our results in Tables and demonstrate a consistent improvement on the Commonsense(AA and OpenbookQA datasets. On Commonsense(A. our model's test performance improves by 5.5% over fine-tuned LMs and 0.9% over existing LM+KG models. On OpenbookQA, these improvements are magnified, with 6.4% over raw LMs, and 2.0% over the prior best LM+KG system, QA-GNN, The boost over QA-GNN suggests that GREASELM's multi-layer fusion component that passes infor- mation between the text and KG representations is more expressive than LM+KG methods which do
*OpenbookQA provides extra corpus of scientific facts in textual form. AristoRoBERTa is based off RoBERTa-Large, but uses the facts corresponding each question, prepared by Clark al. (2019), additional input along with the QA context.

---

Published as a conference paper at ICLR 2022
Table 5:
Performance of GREASELM the CommonsenseQA IH-dev set on complex questions
with semantic nuance such as prepositional phrases, negation terms, and hedge terms.
Prepositional Phrases
Negation Hedge Term
Model
2
3
4
Term
210
429
316
171
59
83
167
RoBERTa- Large
66.7
72.3
76.3
74.3
69.5
63.8
70.7
QA-GNN
76.7
76.2
79.1
74.9
81.4
66.2
76.0
GREASELM (Ours)
75.7
79.3
80.4
77.2
84.7
69.9
78.4
not integrate such sustained interaction between both modalities. We also achieve competitive re- sults to other systems the leaderboard of OpenbookQA (Table posting the third highest score. However, we note that the T5 (Raffel et al.] (020) and UnifiedQA (Khashabi et al. (020) models are pretrained models with 8 and 30x more parameters, respectively, than our model. Among models with comparable parameter counts, GREASELM achieves the highest score. An ablation study on different model components and hyperparameters is reported in Appendix C.I
Quantitative Analysis. Given these overall performance improvements, we investigated whether GREASELM's improvements were reflected in questions that required more complex reasoning. Because we had no gold structures from these datasets to categorize the reasoning complexity of different questions, we defined three proxies: the number of prepositional phrases the questions, the presence of negation terms, and the presence of hedging terms. We use the number of preposi- tional phrases as proxy for the number of explicit reasoning constraints being set in the questions. For example, the Commonsense(4 question in Table [ "A weasel has thin body and short legs to casier burrow after prey in a what?" has three prepositional phrases: to easier burrow, after prey, in what, which each provide an additional search constraint for the answer (n.b., in certain cases, the prepositional phrases do not provide constraints that are needed for selecting the correct answer). The presence of negation and hedging terms stratifies evaluation to questions that have explicit negation mentions (e.g., no, never) and terms indicating uncertainty (e.g., sometimes;; maybe).
Our results in Table demonstrate that GREASELM generally outperforms RoBERTa-Large and QA-GNN for both questions with negation terms and hedge terms, indicating GREASELM handles contexts with nuanced constraints. Furthermore, we also note that GREASEL. performs better than the baselines across all questions with prepositional phrases, our measure for reasoning complexity. QA-GNN and GREASELM perform comparably on questions with no prepositional phrases, but the increasing complexity of questions requires deeper cross- moosl fusion between language and knowledge representations. While QA-GNN's end fusion approach of initializing node in the GNN from the LM's final representation of the context is an effective approach, it compresses the language context to a single vector before allowing interaction with the KG, potentially limiting the cross-relationshiiss between language and knowledge that can be captured (see example in Figure Interestingly, we note that both GREASELM and QA-GNN significantly outperform RoBERTa- Large even when prepositional phrases are in the question. We hypothesize that some of these questions may require less reasoning, but require specific commonsense knowledge that RoBERTa may not have learned during pretraining (e.g., "What is a person considered bully known for?").
Qualitative Analysis. In Figure 2 we examine GREASELM's 's node-to-node attention weights in- duced by the GNN layers of the model, and analyze whether they reflect more expressive reasoning steps compared to QA-GNN. F shows an example from the Commonsense(QA IH-dev set. In this example, GREASELM correctly predicts that the answer is "airplane" while QA-GNN makes an incorrect prediction, "motor vehicle". For both models, we perform Best First Search (BFS) on the retrieved KG subgraph Glub trace high attention weights from the interaction node (purple).
For GREASELM, we observe that the attention by the interaction node increases on the "bug" entity in the intermediate GNN layers, but drops again by the final layer, resembling a suitable intuition surrounding the hedge term "unlikely". Meanwhile, the attention on "windshield" consistently in- creases across all layers. For QA-GNN, the attention on "bug" increases over multiple layers. As "bug" is mentioned multiple times in the context, it may be well-represented in QA-GNN's context node initialization, which is never reformulated by language representations, unlike in GREASELM.
8

---

Published as a conference paper at ICLR 2022
(a) GreaseLM
What unlikely to get bugs on its windshield due to bugs' inability to reach when it is moving? airplane vehicle
Int
Int
Int
windshield
kindshield
sirplane
windshield
bag
vehicle
bug
vehicle
big
vehicle
Dar
GNN Layer
Middle Layer
Final Layer
(b) QA-GNN
What unlikely bugs its windshield duc to bugs' inability to reach when it is moving?
A, airplane E. motor vehicle
Int
Hindshteld
sirplane
windshlele
atrplane
sirplane
bug
vehicle
vehicle
Car
GNN 1st Layer
GNN Middle Layer
GNN Final Layer
Figure 2:
Qualitative analysis of GREASELM's graph attention weight changes across multiple layers of message passing compared with QA-GNN. GREASELM demonstrates attention change patterns that more closely resemble the expected change in focus on the "bug" entity.
Domain generality Our reported results thus far demonstrate the viability of our method in general commonsense reasoning domain. In this section, we explore whether GREASELM could be adapted to other domains by evaluat- ing on the BredQA-USMLLY dataset. Our results Table 6 demonstrate that GREASELM out- performs state-of-the oiiie----thh-arr fine-tuned LMs (e.g.. SapBERT; Liu et al.) 2021) and a QA-GNN augmentation of SEpBERT. Additionally, note the improved performance over all clas- sical methods and LM methods first reported in Jin et al.) (20TT). Additional results in Ap- pendix C show that our approach is also ag- nostic to the language model used with im- provements recorded by GREASELM when it seeded with other LMs, such PubmedBERT (Gu et al., (022). and BioBERT (Lee et al. 2020). While these results promising they suggest that GREASELM is effective aug- mentation pretrained LMs for different domains and KGs (i.e., the medical domain with the DDB Drugbank KG), there is still ample room for improvement this task.
Table 6: Performance MedQA -UMMLE
Methods
Acc. (%)
Baselines (Jin et al. 2021
CHANCE
25.0
PMI
31.1
IR-ES
35.5
IR-CUSTOM
36.1
CLINICALBERT-BE BASE
32.4
MOROBERTAA-BASE
36.1
BIOBERT-BASE BIOBERT-LARGE
34.1
36.7
Baselines (Our implementation)
SapBERT-Bas Base (w/o KG)
37.2
QA-GNN
38.0
GREASELM (Ours)
38.5
CONCLUSION
In this paper, we introduce GREASELM, a new model that enables interactive fusion through joint information exchange between knowledge from language models and knowledge graphs. Experi- mental results demonstrate superior performance compared to prior KG+LM and LM-only baselines across standard datasets from multiple domains (commonsense and medical). Our analysis shows improved capability modeling questions exhibiting textual nuances, such negation and hedging.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
We thank Rok Sosic, Maria Brbic, Jordan Troutman, Rajas Bansal, and our anonymous review- ers for discussions and for providing feedback on our manuscript. We thank Xiaomeng Jin for help with data preprocessing. We also gratefully acknowledge the support of DARPA under Nos. HR001 12190039 (TAMI), N66001 1924033 (MCS); ARO under Nos. W91 INF- 16- 116-1-036 1-0442 (MURT, 1911NF-16--0017 (DURIP): NSF under Nos. OAC-1 835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), IIS 2030207 (RAPID), NIH under No. R56LM013365; Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Amazon, JPMorgan Chase, Docomo, Hitachi, Intel, JD.com, KDDI, Toshiba, NEC, and UnitedHealth Group. J. L. is Chan Zuckerberg Biohub investigator. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
REFERENCES
Olivier Bodenreider. The unified medical language system (UMLS): Integrating biomedical termi- nology. Nucleic acids research, 2004.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collabo- ratively created graph database for structuring human knowledge. SIGMOD, 2008.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. In Asso- ciation for Computational Linguistics (ACL), 2019.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. Dynamic neuro-symbolic knowledge graph con- struction for zero-shot commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, al. From 'f' to on the NY Regents science exams: An overview of the Aristo project. arXiv preprint arXi:: 099900198,,,. 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scalable multi-hop relational reasoning for knowledge-awee aware question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Yuxian Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tris- tan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3:1 223, 2022.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosse- lut, and Yejin Choi. Comer-atomic 2020: On symbolic and neural commonsense knowledge graphs. AAAI, 2021.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 2021.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Ha- jishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of EMNLP, 2020.
10

---

Published as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self supervseee learning of language representations. In International Conference on Learning Representations (ICLR), 2020.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36:1234 1240, 2020.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self-alignment pretraining for biomedical entity representations. In NAACL, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 16992, 2019.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang. Guihong Cao, and Songlin Hu. Graph-base reasoning over heterogeneous external knowl- edge for commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
G. Marcus. Deep learning: A critical appraisal. ArXiv, abs//801..eo.... 1001.00311,,1..1 2018.
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL, 2019.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and A. G. Galstyan. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. ArXiv, abs/2103.11320 2021.
Todor Mihaylov and Anette Frank. Knowledgeable reader: Enhancing cloze-style reading compre- hension with external commonsense knowledge. In Association for Computational Linguistics (ACL), 2018
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Ricdel. Language models as knowledge bases? In Empirical Methods in Natural Language Processing (EMNLP), 2019.
George Sebastian Pirtoaca. Ai2 leaderboard. URL URLattes//Ledeezircdlllleed, wwe.geecccccdnddeeeeoooooo htt.:/ooobb..eeeeee.ceeeeeeeeeee
liten_book_ga/see submi as onrbrriibieeeeoo
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020.
Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations (ICLR), 2020.
Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michihiro Yasunaga, Haitian Sun, Dale Schuur- mans. Jure Leskovec, and Denny Zhou. Lego: Latent exccution guided reasoning for multi-hop question answering on knowledge graphs. In International Conference on Machine Learning (ICML), 2021.
-rciddd
11

---

Published as a conference paper at ICLR 2022
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, 2018.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. Exploiting structured knowledge text via graph-guided representation learning. In EMNLP, 2020.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards controllable biases in language generation. In the 2020 Conference Empirical Methods in Natural Language Processing EEMNL))-Findingss long, 2020.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
Fabian M. Suchanek, Giergji Kasneci, and Gerhard Weikum. Yago: A Core of Semantic Knowledge. In 16th International Conference the World Wide Web, pp. 697-706, 2007.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. Colake: Contextualized language and knowledge embedding. In COLING, 2020.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. httptttts://openeee
net. ftntt/formmmiic wttp://rrmmmm.....pppppp. d=rJXMpikC2
Denny Vrandečié and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10)788888 September 2014. ISSN 0001-0782. doi: 10. 10.1145/2629489.
URI. https:
Peifeng Wang, Nanyun Peng, Pedro Szekely, and Xiang Ren. Connecting the dots: A knowledgeable path generator for commonsense question amswering. arXiv preprint arXhv:2005.00691, 2020.
Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang. Achille Fokoue, Bassem Makni, Nicholas Mattei, et al. Improving natural language inference using external knowledge in the science questions domain. Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. Drugbank 5.0: a major update to the drugbank database for 2018. Nucleic acids research, 2018.
Weiwen Xu, Huihui Zhang, Deng Cai, and Wai Lam. Dynamic semantic graph construction and rea- soning for explainable multi-hop science question answering. arXiv preprint 1rXiv:2105. 1776, 2021.
Jun Yan, Mrigank Raman, Aaron Chan, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. Learning contextualized knowledge structures for commonsense reasoning, arXiv preprint arXiv:2010. 12873, 2020.
An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu. Qiaoqiao She, and Sujian Li. Enhancing pre-trained language representations with rich knowledge for machine reading com- prehension. In Association for Computational Linguistics (ACI.), 2019.
12

---

Published as a conference paper at ICLR 2022
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. ArXiv, abs/2104.06377, 2021.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowl- edge graph and language understanding. ArXiv, abs/2010.00796. 2020.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. In ACL, 2019.
13

---

Published as a conference paper at ICLR 2022
A ETHICS STATEMENT
We outline potential ethical issues with our work below. First, GREASELM is a method to fuse language representations and knowledge graph representations for effective reasoning about textual situations. Consequently, GREASELM could reflect many of the same biases and toxic behaviors exhibited by language models and knowledge graphs that are used initialize For example, prior large-scale language models have been shown to encode biases about race, gender, and other demographic attributes (Sheng et al., (020)) Because GREASELM seeded with pretrained lan- guage models that often learn these patterns, it is possible to reflect them in open-world settings. Second, the ConceptNet knowledge graph (Speer et al.) 2017) used this work has been shown to encode stereotypes (Mehrabi et al., 2021). rather than completely clean commonsense knowledge. If GREASELM were used outside these standard benchmarks in conjunction with ConceptNet as KG, it might rely on unethical relationships in its knowledge resource to arrive at conclusions. Con- sequently, while GREASELM could used for applications outside these standard benchmarks, would encourage implementers use the same precautions they would apply to other language models and methods that use noisy knowledge sources.
Another source of ethical concern is the of the MedQA-USMIEE evaluation. While find clinical reasoning using language models and knowledge graphs to be an interesting testbed for GREASELM and for joint language and reasoning models general, we do encourage users to use these models for real world clinical prediction, particularly at these performance levels.
B EXPERIMENTAL SETUP DETAILS
B. ENTITY LINKING
Given each QA context, we follow the procedure from Yasunaga et al. (202T) to retrieve the subgraph Gub from G. First, we perform entity linking to G to retrieve an initial set of nodes Vinked- Second, we add any bridge entities that are in a 2-hop path between any pair of linked entities in Vinked to get the set of retrieved entities Vretrieved Then prune the set of nodes Vrenieved using relevance score computed for each node. To compute the relevance score, we follow the procedure of Yasunaga et al. concatenate the node name with the context of the QA example, and pass it through pre-trained LM, using the output score of the node name as the relevance score. We only retain the top 200 scores nodes and prune the remaining ones. Finally, we retrieve all the edges that connect any two nodes in Vsuh. forming the retrieved subgraph Gauh- Each node in Gsub is assigned a type according whether its corresponding entity was linked from the context c, question q, answer a, or from a bridge path.
(021
B.2 GRAPH INITIALIZATION
To compute initial node embeddings ($3 for entities retrieved in Gub from ConceptNet, we follow the method of MHGRN (Feng al. 2020 We convert knowledge triples in the KG into sentences using pre-defined templates for each relation. Then, these sentences are fed into a BERT-large LM compute embeddings for each sentence. Finally, for all sentences containing an entity, we extract all token representations of the entity's ss mention spans in these sentences, mean pool over these representations and project this mean-pooled representation.
For MedQA-USMI node embeddings are initialized similarly using the pooled token output em- beddings of the entity name from the SapBERT model (described in $4.2 Liu al., 2021). For MedQA, 5% of examples do not yield a retrieved entity. In these cases, we represent the graph using a dummy node initialized with In essence, GreaseLM backs off to only using LM representations as the graph propagates no information.
B.3 HYPERPARAMETERS
14

---

Published as a conference paper at ICLR 2022
Table 7: Hyperparameter settings for models and experiments
Category
Hyperparameter
Dataset
Commonsense@A
OpenbookOA
MSLOA-USMLE
Number GREASEL M layers M
Number Unimodal layers
Mudel architecture
Number
Dimension enboeddings messages
200
Dimension
lagers (escege Mint opentor)
200
Number layers
Dimension of Mlnt operator hodden layer
400
200
400
Regularizaion
Drupout rite of the smbokling layer, GNN lyyers and folly-eomeetted layers
0.2
0.2
0.3
Learning rate of parameters in LM
1 1.00-445
1.008-05
5,00E:05
Learing parameters in LM
1.004433
1.00E-03
1,00E-03
Number epochs which LM's purameters frowen
Opcimization
Oytimizer
RAdam
RAdom
Leaming rate
conscant
constant
constast
Number epochs
30
20
Max gradient nonm (gradient clipping)
1.0
1.0
1.0
Data
Mas number of nodes
200
200
200
Max number of takens
100
100
512
C ADDITIONAL EXPERIMENTAL RESULTS
C. ABLATION STUDIES
In Table 8 we summarize an ablation study conducted using the CommonsenseQQA IHdev set.
Modality interaction. A key component GREASELM is the connection of the LM the GNN via the modality interaction module (Eq. T. If we remove modality interaction. the performance drops significantly, from 78.5% to 76.5% (approximately the performance of QA-GNN). Integrating the modality interaction in every other layer instead of consecutive layers also hurts performance. A possible explanation is that skipping layers could impede learning consistent representations across layers for both the LM and the GNN, property which may be desirable given we initialize the model using a pretrained LM's weights (e.g., RoBERTa). We also find that sharing parameters between modality interaction layers (Eq. [T] outperforms not sharing, possibly because datasets are not very large (e.g., 10k for Commonsense@A), and sharing parameters helps prevent overfitting.
Table 8: Ablation study of our model components, using the Commonsense@AA IH-dev set.
Ablation Type Ablation
Dev Ace.
GREASELM
78.5
No interaction
76.5
Modality Interaction
Interaction every other layer
76.3
Interaction Layer Parameter Sharing
No parameter sharing
77.1
M 4
77.7
Number of GREASELM layers (M)
78.0
M =
76.2
Interaction node connected all nodes in Vaub. not only Viikeed
Graph Connectivity
77.6
Random
60.8
Node Initialization
TransE (Bordes et 2013)
77.7
Number of GREASELM layers. We find that M GREASELM layers achieves the highest performance. However, both the results for M = and M are relatively close to the top performance, indicating method is not overly sensitive this hyperparameter.
15

---

Published as a conference paper at ICLR 2022
Graph connectivity. The interaction node Cint is key component of GREASELM that bridges the interaction between the KG and the text. Selecting which nodes in the KG are directly connected to Cint affects the rate at which information from different portions of the KG reach the text representations. We find that connecting fint KG nodes explicitly linked to the input text performs best. Connecting Cint all nodes in the subgraph (e-g., bridge entities) hurts performance (-0.9%), possibly because the interaction node is overloaded by having to attend to all nodes in the graph (up 200). By connecting the interaction node only linked entities, each linked entity serves filter for relevant information that reaches the interaction node.
KG node embedding initialization. Effectively initializing KG node representations is critical. When we initialize nodes randomly instead of using the BERT-based initialization method from Feng et al. (020), the performance drops significantly (78.5% 60.8%) While using standard KG embeddings (e.g., TransE; Bordes et al. 2013) recovers much of the performance drop (77.7%), we still find that using BERT-based entity embeddings performs best.
C.2 EFFECT OF LM INITIALIZATION ON GREASELM
Table 9: Performance on the in-house splits of Com- monsenseQA for different LM initializations of our method, GREASELM.
Table 10: Initialization MedQA-
USMLE
Methods
Acc. (%)
SAPBERT-BASE
37.2
Methods
IHdev-Ace.
IHitestAAc.
+ GREASELM (Ours)
38.5
ROBERTA-LARGE
73.1
68.7
BIOBERT-BASE
34.1
GREASELM (Ours)
78.5
74.2
+ GREASELM (Ours)
34.6
ROBERTA-BAL BASE
65.1
59.8
PUBMEDBERT-BASE
38.0
GREAAELLM (Ours)
69.3
65.0
+ GREASELM (Ours)
38.7
To evaluate whether our method is agnostic to the LMM used to seed the Greasel . layers, replace the LMs we use in previous experiments (RoBERTa-large for Commonsense(A and SapBERT for MedQA-USMLE) with RoBERTa-base for CommonsenseQA, and BioBERT and PubmedBERT for MedQA-USMLE. Across multiple LM initializations in two domains, our results demonstrate that GREASELM can provide consistent improvement for multiple LMs when used modality junction between KGs and language.
16


Reviews:
[{"review_id": "pj_9cVJxAc", "reviewer": "Reviewer_NQbJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper: \n\n1. The paper presents a novel way of combining information from text and a KB in a bidirectional way. \n2. The results presented in the paper show strong gains against baseline methods on 3 different datasets.\n3. Ablation studies show that the model achieves good performance on more complex questions.\n\nWeaknesses, suggested improvements and requested clarifications\n\n1. Regarding preposition phrases as a proxy for complexity: since the hypothesis is that the more the number of prepositional phrases in a question, the harder it is to answer. But from Table 5., the trend of performance seems to increase with the increased prepositional phrases (with 84.7 being the max for 4 PPs). It would be good if the authors could provide some discussion around this observation. Additionally, it would be good if the authors could report the number of examples (either raw number or as a fraction of the total dev set) for each of the categories: having that would help draw better conclusions. \n\n2. The paper mentions that the entity extraction was done following Yasunaga et al. Was relevance scoring was done for the entities ?\n\n3. Alongside with qualitative analysis, some quantitative analysis would be good to show what the model learns. Specifically, having the BFS analysis of the attention weights as a function of different GreaseLM layers (as done by Yasunaga et al.) can help demonstrate how having the bidirectional context information flow helps improve reasoning. Additionally, showing this for negations and / or examples which GreaseLM gets correct but QA-GNN does not (and vice-versa) can shed some light on what the model improves on (and what are the limitations). \n\n4. Having an ablation on the number of GreaseLM layers would also be quite useful to answer if performance improves with more GreaseLM layers, are there diminishing returns or do we need just a few GreaseLM layers, beyond which it is detrimental to the model's performance.\n\n5. The Graph connectivity ablation states that connecting the e_int node to all entities (instead of just the input text entities) hurts performance. It would be good if the authors could provide some intuition / insight as to why that might be the case. \n\nPresentation Suggestions:\n\n1. Page 5, line 1: should \\tilde{e}^{(l-1)} be \\tilde{e}^{l} instead ? \n\n2. Page 5, Equations (6, 7, 8): should e^{l}_{s} and e^{l}_{j} be e^{(l-1)}_{s} and e^{(l-1)}_{j} respectively ?\n\nNote to the authors:\n\nThere is a contemporaneous work submitted to ICLR 2022 [GNN is a Counter? Revisiting GNN for Question Answering](https://openreview.net/forum?id=hzmQ4wOnSb), whose hypothesis seems to be that by only using embeddings for node types and relation types, the models are able to attain good performance (86.67 acc on OpenBookQA) without needing any cross-modal information. While I understand this is contemporaneous work, but since the work is so relevant to this paper and seems to directly contradict the premise of this paper, it might be good to have a short discussion on this (just a suggestion).\n", "summary_of_the_paper": "The paper presents a novel method of fusion of information from two modalities: text (context and question) and a Knowledge Base, for the task of question answering. Concretely, they propose GreaseLM layers that combine information from a Language Model (LM) and a Graph Neural Network (GNN) by mixing representations of embeddings from two special tokens (one in the LM and one in the GNN). The embedding in the LM passes information to the text modality via self attention; while the embedding in the GNN passes information to the KB modality via connections to other nodes. The modality interaction layer (dubbed MInt) is a simple MLP for combining the embeddings.\nThe paper demonstrates superior performance compared to baseline methods on 3 different datasets. Additional ablations quantitatively show that the model performs better on questions with negations, hedging and constraints (modelled with a proxy of number of prepositional phrases). ", "summary_of_the_review": "The paper presents a novel way of combining information from text and knowledge base modalities. The results on 3 datasets show the models improved performance, and ablations show quantitatively that the model performs better on complex questions. While I believe the paper can benefit from some qualitative analysis, and there are some open questions that need clarification, I believe the contributions presented are enough to merit an acceptance.\n\nEdit: I have updated my score after reading the author response and edits made to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "haPRgiteuFE", "reviewer": "Reviewer_99q1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- A novel architecture to enable deeper interaction between LM and GCN.\n- Clear explanation of the proposed model.\n- Extensive ablation studies to demonstrate the importance of modality interaction layer \n- selection, parameter sharing, graph connectivity, and parameter initialization.\n-  Consistent performance improvement over the baseline models on all of the three QA benchmarks.\n\nWeakness:\n- paper title is misleading, not directly related to LM\n- no citation and description for the baseline method T5+KB (Table 4)\n- As shown in Table 7, the proposed method is very sensitive to many factors.  The performance drops significantly when we change any of them.\n- The data preprocessing and training steps are complex.\n\n\nQuestions:\n- According to the parameters presented in Table 8, the knowledge from LM and GNN are only fused at the last 5 layers (parameter M) when 24-layer LMs are used, and at the last 3 layers when 12-layer LMs are used. It may be useful to show how the performance changes when using different M.\n- According to [2], the CommonsenseQA IH-dev set contains 1,221 questions in total. May I know many questions are in each data split shown in Table 5?\n- Is there any case that no entities are found during KG retrieval (Sec. 3.1)? If there are any, how do you handle these cases? What’s the percentage of such cases.\n\nReference:\n[1] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA- GNN: Reasoning with language models and knowledge graphs for question answering. ArXiv, abs/2104.06378, 2021.\n[2] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Empirical Methods in Natural Language Processing (EMNLP), 2019.\n", "summary_of_the_paper": "In this paper, the authors introduce a new model for the QA tasks. They improve the existing approaches by adding the interaction components (the interaction token for the LM and interaction node for the GNN) to fuse knowledge from LM and knowledge graph in a more interactive manner. The LM can be any Transformer encoder-based pretrained LMs, where they tested RoBERTa-Large, AristoRoBERTa, and SapBERT in the experiments. The GNN is a simplification of the method of [1]. Experiments have been conducted to compare the proposed model and the prior KG+LM and LM-only baselines. Their model proved to be effective on three benchmarks, CommonsenseQA, OpenbookQA and MedQA-USMLE.\n", "summary_of_the_review": "In summary, the paper has some nice contributions, such as a novel architecture to enable deeper interaction between LM and GCN, clear writing, and extensive analysis. Although there remain some issues and questions, the paper should be able to bring to the community some new aspects. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q7zsr3-rl5e", "reviewer": "Reviewer_ZyPu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I like this paper since the method is simple and the results are good. I have only one problem: the initialization of LMs. The authors say that they initialize different LMs for different datasets to show that the method is agnostic to them. I am not convinced by this argument. A better way to do this would be to experiment with different LMs on one dataset to show that the method is agnostic to it and then use the best one for the other dataset(s) (Note: I understand that this would not apply to the biomedical dataset).\n\nOther than this, I enjoyed reading the paper and liked the analysis and the detailed ablations. I am willing to increase my rating if the above concern is resolved.\n\nEDIT: changed my rating after the author response.", "summary_of_the_paper": "This paper presents a KG augmented LM which fuses graph and contextual representations. The model, when evaluated on multi-choice QA, outperforms strong baselines.\n\nThe context, question, answer pairs are encoded with a standard LM and the nodes of a relevant subgraph which is retrieved from a KG are encoded with a GNN. The representations of (i) a special interaction token (same as CLS), and (ii) a special interaction node, are input to custom layers that mix/fuse them which potentially makes them more informative to the downstream MLP head.\n\nThe authors experiment on three MCQA datasets: commonsenseQA, openbookQA, and medQA-USMLE (from the biomedical domain). ConceptNet is used as the KG for the first two datasets and a self-constructed KG is used for the third. The evaluation and analysis show that this method outperforms strong baselines (sometimes with a much higher parameter count). especially when the questions are complex and have prepositional phrases, negation terms, etc.", "summary_of_the_review": "A good paper with a small and easily fixable issue regarding LM initialization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D_2GFBoFtOe", "reviewer": "Reviewer_RNQN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The idea of multi-layer fusion of two modality through additional interaction token and node is interesting.\n2. The model achieves good results on three multi-choice question answering datasets.\n3. The ablation study shows some interesting finds such as the importance weight sharing in Mint layers.\n\nWeakness:\n1. The technical contribution is somewhat small: except the multi-layer fusion part, others are very similar to QA-GNN including the GNN model. \n2. As the main contribution is about the modality interaction layer, more analysis should be made such as the effects of number of fusion layers $M$, why only perform interaction through one single embedding (interaction token and node) instead of directly applying cross attention through all the tokens and nodes? There are also some other ways to perform embedding interaction such as the one in ERNIE [1] Figure 2 (b). To answer these questions, both intuitive analysis and empirical results are needed.\n\nAdditional Reviews:\n1. Since this paper only focuses on question answering tasks, I suggest adding \"for question answering\" into the title similar to previous works QA-GNN and MHGRN to be more accurate.\n2. There are also some related but missing references such as GLM[1], JAKET[2], CoLAKE[3]. Although they focus on language model pre-training, the idea of combining graph reasoning and language modeling is related.\n\n[1] Zhang Z, Han X, Liu Z, et al. ERNIE: Enhanced language representation with informative entities[J]. arXiv preprint arXiv:1905.07129, 2019.\n[2] Shen T, Mao Y, He P, et al. Exploiting structured knowledge in text via graph-guided representation learning[J]. arXiv preprint arXiv:2004.14224, 2020.\n[3] Yu D, Zhu C, Yang Y, et al. Jaket: Joint pre-training of knowledge graph and language understanding[J]. arXiv preprint arXiv:2010.00796, 2020.\n[4] Sun T, Shao Y, Qiu X, et al. Colake: Contextualized language and knowledge embedding[J]. arXiv preprint arXiv:2010.00309, 2020.", "summary_of_the_paper": "This paper proposes a model which fuses representations from language models and graph neural networks through multiple interactions layers for multi-choice question answering tasks, which outperforms baseline methods over three datasets from two different domains.", "summary_of_the_review": "In summary, this paper proposes an interesting idea to combine language model and graph neural networks for question answering and shows good results. But more experiments and analysis should be conducted to justify their design on modality interaction modeling. Adding them will make it a strong paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel method of fusion of information from two modalities: text (context and question) and a Knowledge Base, for the task of question answering. The proposed method looks quite simple and clear, while the results show strong gains against baseline methods on 3 different datasets. Ablation studies show that the model achieves good performance on more complex questions. While the reviewers raise some concerns, e.g., on the sensitivity of the proposed method, the technical novelty against prior works, they see values in this paper in general. And the authors did a good job in their rebuttal. After several rounds of interactions, some reviewers were convinced to raise their scores by a little bit. As a result, we think the paper is in a good shape and ICLR audience should be interested in it.
Decision: Accept (Spotlight)