GNN is a Counter? Revisiting GNN for Question Answering
Authors: Kuan Wang
Yuyu Zhang
Diyi Yang
Le Song
Tao Qin
Venue: ICLR 2022 Poster (2022)
Keywords: GNN
Question Answering
QA
Reasoning
ML

Abstract: Question Answering (QA) has been a long-standing research topic in AI and NLP fields, and a wealth of studies has been conducted to attempt to equip QA systems with human-level reasoning capability. To approximate the complicated human reasoning process, state-of-the-art QA systems commonly use pre-trained language models (LMs) to access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs). However, many problems remain open regarding the reasoning functionality of these GNN-based modules. Can these GNN-based modules really perform a complex reasoning process? Are they under- or over-complicated for QA? To open the black box of GNN and investigate these problems, we dissect state-of-the-art GNN modules for QA and analyze their reasoning capability. We discover that even a very simple graph neural counter can outperform all the existing GNN modules on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. Our work reveals that existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting. It remains a challenging open problem to build comprehensive reasoning modules for knowledge-powered QA.

Published as a conference paper at ICLR 2022
GNN IS A COUNTER? REVISITING GNN FOR QUESTION ANSWERING
Kuan Wang*", Yuyu Zhang 1, Diyi Yang', Le Song & Tao Qin Georgia Institute of Technology Miirosoot Research Asia *BioMap *MBZUAI (kuanwang, yuyu, dyang888 @gatech. edu Isongêcc. long@cc.gatech... gatech. edu taoginemicrosoft.cm com
ABSTRACT
Question Answering (QA) has been long-standing research topic in Al and NLP fields, and wealth of studies have been conducted to attempt to equip QA systems with human-level reasoning capability. To approximate the complicated human reasoning process, statc-of-the-art QA systems commonly use pre-trained language models (LMs) access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs). However, many problems remain open regarding the reasoning functionality of these GNN-based modules. Can these GNN-based modules really perform a complex reasoning process? Are they under- or over- complicated for QA? To open the black box of GNN and investigate these problems, we dissect state-of-the-ar GNN modules for QA and analyze their reasoning capability. We discover that even very simple graph neural counter can outperform all the existing GNN modules on Conmonsense(A and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. Our work reveals that existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting. It remains a challenging open problem build comprehensive reasoning modules for wwwwledge-poweeee QA.
INTRODUCTION
Accessing and reasoning over relevant knowledge is the key to Question Answering (QA). Such knowledge can be implicitly encoded explicitly stored in structured knowledge graphs (KGs). Large pre-trained language models (Devlin et al., 2018; Radford et al., 2018; 2019; Brown et al., 2020) found be effective in learning broad and rich implicit knowledge (Petroni et al., 2019; Bosselut et al., 2019; Talmor et al., 2020) and thus demonstrate much success for QA tasks. Nevertheless, pre- trained LMs struggle lot with structured reasoning such as handling negation (Ribeiro al., 2020; Yasunaga et al., 2021). In contrast, the explicit knowledge such as knowledge graphs (KGs) (Speer al., 2017; Bollacker al.. 2008) works better for structured reasoning it explicitly maintains specific information and relations and often produces interpretable results such as reasoning chains (Jhamtani & Clark, 2020; Khot al., 2020; Clark al., 2020b).
To utilize both implicit and explicit knowledge for QA, many existing works combine large pre-trained LMs with Graph Neural Networks (GNNs: Scarselli al. (2008); Kipf & Welling (2017); Veličković et al. (2018)), which are shown to achieve prominent QA performance. These approaches commonly follow a two-step paradigm to process KGs: 1) schema graph grounding and 2) graph modeling for inference. In Step 1, schema graph is retrieved sub-graph of KG related to the QA context and grounded on concepts; such sub-graphs include nodes with concept text, edges with relation types, and their adjacency matrix. In Step 2, graph modeling is carried out via claborately designed graph- neural module. For instance, Lin et al. (2019) uses GCN-LSTM-HHP which combines graph convolutional networks (Kipf & Welling, 2017) and LSTM (Hochreiter & Schmidhuber, 1997)
Work done during an internship MSRA

---

Published as a conference paper at ICLR 2022
Blue read material outside his comfort zone because he wanted to gain what?
A. new perspective Retrieve
understanding
C. entertained
E. tired eyes D. hunger
Interpretable
Counting Edges
Efficient
Effective
We analyze state-of-the-an GNN modules for the task of KG-powered question answering. and find that the counting of edges in the graph plays an essential role in knowledge-awan reasoning. Accordingly, we design an efficient, effective and interpretable graph neural counter module for knowledge-aware QA reasoning.
Figure I:
with hierarchical path-based attention mechanism for path-based relational graph representation. Feng et al. (2020) extends the single-hop message passing of RGCN (Schlichtkrull et al., 2018) as multi-hop message passing with structured relational attention to obtain the path-level reasoning ability and intractability, while keeping the good scalability of GNN. Yasunaga et al. (2021) uses a LM to encode QA context node in the scheme graph and then utilized graph attention networks (Veličković et al., 2018) to process the joint graph.
Given that today's QA systems have become more and more complicated, would like to revisit those systems and ask several basic questions: Are those GNN-based modules under- or over- complicated for QA? What is the essential role they play in reasoning over knowledge? To answer these questions, we first analyze current state-of-the-art GNN modules for QA and their reasoning capability. Building upon our analysis, we then design simple yet effective graph- baseh neural counter that achieves even better QA performance on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning.
In the analysis part, we employ Sparse Variational Dropout (Sparse Molchanov et al. (2017)) as tool to dissect existing graph network architectures. Sparse proposed as a neural model pruning method in the literature, and its effect of model compressing serves as an indicator to figure out which part of the model can be pruned out without loss accuracy. We apply Sparse to the inner layers of GNN modules, using their sparse ratio to analyze each layer's contribution to the reasoning process. Surprisingly, we find that those GNN modules are over- parameterized: some layers GNN can be pruned very low sparse ratio, and the initial node embeddings are dispensable.
Based on our observations, we design Graph Soft Counter (GSC), a very simple graph neural model which basically serves counter over the knowledge graph. The hidden dimension of GSC layers only 1, thus each edge/node only has a single number as the hidden embedding for graph- based aggregation. As illustrated in Figure GSC not only very efficient but also interpretable, since the aggregation of I-dimensional embedding can be viewed as soft counting of edge/ node in graphs. Although GSC is designed to simplistic model, which has less than 1% trainable parameters compared to existing GNN modules for QA, it outperforms state-of-ther of-thh-rrt GNN counterparts on two popular QA benchmark datasets. Our work reveals that the existing complex GNN modules may just perform some simple reasoning such as counting in knowledge-aware reasoning.
The key contributions of our work are summarized as follows:
Analysis of existing GNN modules: We employ Sparse as diagnostic tool to analyze the importance of various parts of state-of-the-an knowledge-awar GNN modules. We find that those GNN modules are over-complicated for what they can accomplish in the QA reasoning process.
Importance of edge counting: We demonstrate that the counting of edges in the graph plays crucial role in knowledge-aware reasoning, since our experiments show that even simple hard counting model can achieve QA performance comparable to sfate-of-the- GNN-based methods.
Design of GSC module: module the replacement for existing complex GNN modules. With less than 1 1% trainable parameters compared to existing GNN modules for QA, our method even outperforms those complex GNN modules two benchmark QA datasets.
We propose Graph Soft Counter (GSC), a simple yet effective neural
2

---

Published as a conference paper at ICLR 2022
Retrieved Sub-graph
Context Node
Multi-rclational Edge
Encode One-hot Triples
Initialized with LM/TransE
Question Entity Node,
Answer Entity Node
Node Embedding
Relevance Score
Adjacency Matrix
Edge Embedding
Which ingredients are crucial?
GNN
GNN
(Pruned)
SparseVD Dissection
Sparse Ratio
KL-lloss
Figure The retrieved sub-graph of KG formulated as entity nodes representing concepts connected by edges representing relations and the central context node connected to all question entity nodes and answer entity nodes. The pre- pe-pooeessed graph data generally has the following ingredients: node embedding initialized with pre-trained KG embeddings, relevance score computed by LM, adjacency matrix representing the topological graph structure, edge embeddings to encode the node types, and edge information of relation types. We adapt Sparse diagnostic tool to dissect GNN-base reasoning modules for QA, getting the sparse ratio of each layer to indicate its importance. We find that some layers and ingredients completely dispensable, which inspires us design simple, efficient and effective GNN module as the replacement existing complex GNN modules.
ANALYSIS
2.1
PRELIMINARIES
The knowledge required by QA systems typically comes from two sources: implicit knowledge in pre-trained language models and explicit knowledge in knowledge graphs.
To use the implicit knowledge, existing works commonly use LMs the encoder to encode textual input sequence x to contextualized word embeddings, and then pool the embedding of the start token (e.g., [CLS] for BERT) the sentence embedding. In addition, MLP (we connected layer) is used to map the sentence embedding to the score for the choice.
layer fully
To process the explicit knowledge in knowledge graphs, existing works commonly follow a two-step paradigm: schema graph grounding and graph modeling for inference. The schema graph a retrieved sub- graph of KG grounding
concepts related to the QA context. We define the uubgrraph as a multi-relational graph 9 = (V&&), where V the set of entity nodes (concepts) in the KG and C V R x V is the set of triplet edges that connect nodes in V with relation types in R. Following prior works (Lin et al., 2019; Yasunaga et al., 2021), we link the entities mentioned in the question and answer choice a
the given sub-graph G.
A wealth of existing works have explored to incorporate pre-trained language models with knowledge- aware graph neural modules. These methods usually have complex architecture design process complex input knowledge. As shown in Figure 2, the processed retrieved sub-graph can be summa- rized as four main components:
Node embeddings. Many existing works GNN-based QA employ external embeddings to initialize the node embeddings in the graph. For example, Lin al. (2019) employs TransE (Wang et al., 2014) with Glo Ve embeddings (Pennington et al., 2014) the initial node embeddings. Feng et al. (2020) and Yasunaga et al. (2021) use mcan-pooled BERT embeddings of entities across the graph as the initialization of node embeddings.
Relevance score. Yasunaga et al. (2021) uses an extra embedding of relevance score in the input node features estimate the importance of KG nodes relative to the given QA context. The relevance
3

---

Published as a conference paper at ICLR 2022
GNN Layer2 Key
Concept Embedding (QA-GNN)
0.8
0.8
GNN Layer2 Query
0.5
Cancept Emilediing (MHGRN)
GNN Layer2 Value
Concept Embedding (KagNet)
Type Embriding
Relevance Score Enioedding
9.
Unonder Layer
0.2
0.2
0.2-
0.00
200
102
830
1000
1200
0.000
200
0,0 900
1000
0.0g
600
500 1000
Insining
training
Figure 3: The sparse ratio curves obtained from the Sparse VD training of GNN reasoning modules: 1) the left plot shows the curves for the embedding layers in QA-GNN, where the node score and the node type reach a low ratio, while the edge encoder preserves a large ratio; 2) the middle plot shows the curves for the layers within the QA-GNN graph attention layer, where the key and query layers converge to a fairly low ratio, while the value layer has a relatively large ratio; 3) the right plot shows the curves of the initial node embedding layers of three representative GNN-based QA methods, where all the ratios are close to 0, indicating that the initial node embeddings are dispensable.
score is computed by the masked LM loss of the stitched sequence with the QA context and the retrieved concept.
Adjacency matrix. The original adjacency matrix of the sub-graph is asymmetric. Typically, the adjacency matrix is converted to be symmetric before feeding into the GNN module: each KG relation is double recorded with opposite directions as symmetric edges in the processed adjacency matrix.
Edge embeddings. Existing works use various ways to obtain the edge embeddings. Yasunaga et al. (2021) uses concatenated one-hot vector [ug, Est, ue] encode edge triplets, where ug Ue indicates the node type and est indicates the edge type. Lin et al. (2019) uses TransE (Wang et al., 2014) initialized with GloVe embeddings to generate edge embeddings. Feng et al. (2020) designs product-based edge embeddings to approximate the multi-hop relations in the graph.
2.2 DISSECTION
To investigate the mechanism of these complex systems and how they use the complex information, we introduce neural model pruning method named Sparse Variational Dropout (Sparse VD) (Molchanov et al., 2017) as a diagnostic tool to automatically dissect the graph network architecture. Note that our dissection tool is pruning method agnostic; other pruning schemes (Han et al., 2015; He et al., 2017: Liu et al., 2017) may also be applicable. Here we choose Sparse since it prunes not only the weights with smaller scale but also the weights with higher variance, and it is theoretically supported by stochastic variational inference (Hoffman et al., 2013; Kingma & Welling, 2013; Rezende et al., 2014: Kingma et al., 2015).
Despite the fact that SparseVD was originally proposed in the field of model compression (Han et al., 2016; He et al., 2018: 8: Lin al., 2017; Zhou et al., 2018; Wang et al., 2018), we use it to investigate which parts of GNN can be pruned out (sparse ratio to zero) without loss of accuracy, which indicates that part of the model is redundant. To be specific, we keep the target model architecture unchanged, and parameterize each part of the weights in the model as Gaussian distribution. Afterwards, this probabilistic model will be trained with a cross-entropy loss jointly with a KL-divergence regularization term. So the joint loss constrains the weights to pruning prior. We implement the SparseVD with the default threshold as in Molchanov et al. (2017). Eventually, we get the pruned model with different sparsified ratios for the layers. As shown in Table we investigate three representative GNN-based QA methods, and the sparsified models could achieve the accuracy of their original counterparts. This indicates that our dissection regularization does not hurt the model performance, so we then dive into each layer to see which parts play the most important role.
w/o SparseVD
w/ Sparse VD
Methods
IHdev-Ace. (%)
IHtest-Acc. (%)
IHdev-Ace. (%)
HHtest-Ace. (%)
KagNet (Lin 2019)
73.47 (+:0.22)
69.01 (+0.76)
75.18 (=1.05)
70.48 (+0077)
MHGRN (Fang et al., 20000)
74.45 (++0.10)
71.11 (+0.81)
77.15 (+0.32)
72.66 (+0.61)
QAGNN (Yasunga al., 2021)
76.54 (+:0221)
73.41 (+:092)
77.64 (=0.50)
73.57 (=:0.88)
Table I: To preserve the reasoning ability for analysis, our Sp tool prunes the GNN-based models without loss of accuracy on CommonsenseeA dataset. As the official test is hidden, here we report the in-house dev (IHdev) and test (IHtest) accuracy, following the data split of Lin et al. (2019).
4

---

Published as a conference paper at ICLR 2022
Graph Attention Layer
Graph Soft Counter Layer,
output node
MLP
aggregate
0.5
sparsely
update node value
aggregate
SoftMax
Adjacency Matrix
0.7
K
V
update edge value
0.6
0.3
0.2
0.7
0.1
0..
0.4
Edge Embedding dimension
0.2
Node Embedding
Node Value
add in- value odge
Figure 4: Graph Soft Counter (right) extremely simplifies the architecture of conventional GNNs (left). The GSC layers are parameter-free and only keep the basic graph operations: 1) update each edge embedding with incoming node (in-node) embeddings; 2) update each node embedding by aggregating the edge embeddings. Since we reduce the hidden dimension to only GSC can be viewed as soft counter over the graph for generating the final score of the output node.
2.3 OBSERVATIONS AND HYPOTHESIS
In Figure 3, we plot the sparse ratio of some representative layers during the Sparse VD training of GNN reasoning modules. Due to the limited space, refer to Appendix A.3 for the full plots of curves. According to these plots, we summarize our key observations as follows: 1) the left plot shows that the edge encoder encoding edge triplets (edge/ / node type information) preserves relatively higher sparse ratio, while the node score embedding layer can be fully pruned; 2) the middle plot shows the layers inside GNN, and all the sparse ratios are low while the value layer has a relatively higher sparse ratio than key/query layers; 3) the right plot shows the concept embedding layers of three representative GNN methods, which process the initial node embeddings, can be completely discarded. Inspired by these findings, come up with the following design guidelines for much simpler yet effective graph neural module for knowledge-awar reasoning:
Node embeddings & relevance score. We observe that both the initial node (concept) embeddings and the relevance score embeddings are dispensable in our experiments. Therefore, we may directly remove the initial node embedding layer and the relevance score embedding layer.
Edge embeddings. The edge encoder layers are hard prune which indicates that edge/node type information is essential for reasoning, so we keep the edge embeddings.
Message passing layers. The linear lay- ers inside GNN layers (query, key, value, etc) can be pruned to a very low sparse ratio, suggesting that GNN may be over- parameterized in these systems, and we can fewer parameters for these layers.
Algorithm 1 PyPorch-style code of GSC
conecete::: question enswer pair context
adj: index with shape edge_types edge type with shape I x N_edge node_ type: node type with shape 1 x R_node
N_edge
nege_eeb odge_encoder (adj, edga_type, nodo_type) node_erb torch. (X_node) for i_aayer in range tnwm_gsc_iee propagate from node to edge emb te embemmbaa]]]]] # eggregate from edge to node aode_eab scatter (edge_emb, ad][0]) graph bontext_score score folroberta (qa_ conttxt)) qa_score context_score grapa_acore
Graph pooler. The final attention based graph pooler aggregates node representa- tions over the graph to get the graph repre- sentation. We observe that the key/ /qerry layers inside the pooler can be pruned out; as a result, the graph pooler can be reduced to linear transformation.
(num_gec_ieyers):
mode_emble
GRAPH SOFT COUNTER
Based on our findings in Section 2, we design the Graph Soft Counter (GSC), simplistic graph neural module to process the graph information. As demonstrated in Algorithm there only two
5

---

Published as a conference paper at ICLR 2022
basic components in GSC to compute the graph score: Edge encoder and Graph Soft Counter layers. We can get the QA choice score by simply summing up the graph score and context score.
The edge encoder is two-layer MLP with di- mension [46 x 32 x 1] followed by a Sigmoid float
KugNet
MHGGN
QA-GNN
GSC (ours)
function to encode the edge triplets number in the range of (0, 1). The triplets is represented as concatenated one-hot vector [ua, est, Ut]. where Ua, Ur indicates 4 node types and Est indicates 38 relation types (17 regular relations plus question/ /nsswr entity relations and their reversions).
Adjacency matrix
Edge types
Node types
Node emheddings
Relevance score
#Learnable params
2845K
Model size
819M
819M
821M
3K
Table 2: Our GSC does not use initial node em- beddings and relevance score as in previous works, making our model get rid of concept embedding pa- rameters and thus its model size is extremely small. Since our model does not have any learnable pa- rameters except the edge encoder, our model has much less chance to be wverpparameterized.
GSC layers are parameter-free and only keep the basic graph operations: Propagation and ag- gregation. To overcome ner-paraaetteizztion, one straightfowward way is to reduce the hidden size, and the extreme case is reducing it to only As shown in Figure 4, the GSC layer simply propagates and aggregates the numbers the edges and node following the two-step scheme: 1) update the edge value with in-node value, and this is done by simply indexing and adding; 2) update the node value by aggregating the edge values, and this is done by scattering edge values to out-node.
Model
Time
Space
G dense graph
L-hop KagNet
O (IR L)
O (
hop MHGRN
(
OIR VIRVLD D)
L-layer QA-GNN
O(V
OR D)
L-luyer GSC
(
O (RRIVI))
@ is a sporse graph with marvimum node degree & & [V]
Since we reduce the hidden size to only I, GSC
L-hop KagNet
o (IR "DVLLA (RRVIIAA)
O LV D)
basically propagating and aggregating num-
h-hop MHGRN
VVL. D)
bers on the graph. We can interpret these num- bers as soft counts representing the importance of the edges and nodes. The aggregation of GSC can be regarded as accumulating the soft counts, so call it Graph Soft Counter. In addition, can also formulate GSC as an extremely effi- cient variant of current mainstream GNN such as GAT and GCN. As shown in Table 3, the computation complexity of GSC is much smaller than the baselines regard to both time and space. Table 2 shows that the trainable parameters in GSC is remarkably less than previous methods since our message-passing layers are parameter-free. The model size of GSC is also extremely small since we do not use any initial node embedding.
L-layer QA-GNN
((VILA)
LLL ((IIVIL)
L-Inyer GSC
((VI))
Table 3: GSC extremely efficient compared to the computation complexity of L-hop reasoning models with hidden dimension D on a dense/ / sparse graph G = EV,E) with the relation set R.
To draw a more conclusive conclusion, we handcraft a counting-bssed feature and use a simple two-layer MLP to map it to graph score. The feature is computed by counting the occurrence of each possible edge triplet. Surprisingly, this simple model could achieve similar performance and can even outperform baselines, which further proves that Counter is a basic and crucial part in QA.
4
EXPERIMENTS
4.1 SETTINGS
Datasets and KG. We conduct extensive experiments on CommonsenseeAA (Talmor et al., 2019) and OpenBookQA (Mihaylov et al., 2018), two popular QA benchmark datasets that heavily rely on knowledge-awee aware reasoning capability. Commonsense(A is a 5-way multiple choice QA task that requires reasoning with commonsense knowledge, which contains 12, 102 questions. The test set of Commonsense QA is not publicly available, and the model predictions can only be evaluated once every two weeks via the official leaderboard. Hence, following the data split of Lin et al. (2019), we experiment and report the accuracy on the in-house dev (IHdev) and test (IHtest) splits. We also report the accuracy of our final system the official test set. OpenBookQA is a 4-way multiple choice QA task that requires reasoning with elementary science knowledge, which contains 5,957 questions. Following Clark et al. (2020a), methods with oristoRoBERTT use textual evidence external

---

Published as a conference paper at ICLR 2022
imput of the QA context. In addition, use ConceptNer (Speer et al., 2017), a general commonsense knowledge graph, as our structured knowledge source G for both of the above tasks. Given each QA context (question and answer choice), we retrieve the sub -ab-graph Gsuh from G following Feng et al. (2020). We only use the concepts that occur in the QA context, as detailed in Table 8.
Implementation and training details. We use a two-layer MLP with hidden di- mension 47 x 32 followed by Sig-
Model
IHdev-Ace. (%)
lHlest-Acc, (%)
RoBERTa-large (w/o KG)
73.07 (#0.45)
68.69 (=0.56)
moid function as our edge encoder, and the
number of GSC layers is 2. Since GSC is extremely efficient, we set the dropout (Srivastava et al., 2014) rate O. We use
RGCN (Sellichkkollen
72.69 (+0.119)
68.41 (+0.66)
GeonAlin (Wang al. 2019)
72.61( +:0.39)
68,59 (1.0.96)
KagNet (iina al., 2019)
73.47 (++0.22)
69.01 (+0.76)
(Santeen a 2017)
74.57 (++0.91)
69.08 (+0.21)
RAdam (Liu et al., 2020) as the optimizer and set the batch size to 128. The learning rate is le-5 for RoBERTa and le-2 for GSC. The maximum number of epoch set to 30 for CommonsenseeQ and 75 for Open- BookQA. On a single Quadro RTX6000 GPU, each GSC training only takes about 2 hours to converge, while other methods often take 10+ hours.
MHGRN (reng 2000)
74.45 (1:0.10)
71.11 (110.1)
QA-GNN (Yasunaga
76.54 (100.21)
73.41 (10.92)
GSC (Ours)
79.11 (#:0222)
74.48 (#0.41)
Table 4: Performance comparison on CommonsenseeA in-house split (controlled experiments). As the official test is hidden, here we report the in-house dev (IHdev) and test (IHtest) accuracy. following the data split of Lin et al. (2019).
Baselines. For GSC and all the other methods, RoBERTa-large (Liu al., 2019) is used for both CommonsenseQA and OpenBookQA, and BrEsto (Clark et al., 2020a) is used for OpenBookQA as an additional setting for fair comparison. We experiment to compare GSC with existing GNN-based QA methods, includ- ing RN (Santoro et al., 2017). GconAttn (Wang et al., 2019), RGCN (Schlichtkrull et al., 2018), KagNet (Lin et al., 2019), MHGRN (Feng et al., 2020) and QA-GNN (Yasunaga al., 2021). which only differ in the design of GNN reason- ing modules. We report the performance of base- lines referring to Yasunaga et al. (2021) and all the test results are evaluated the best model on the dev split.
Model
Test
RoBERTa (Liu et al., 2019)
72.1
RoBERTa + FreeLB (Zhu et al., 2020) (ensemble)
73.1
RoBERTa IIKAS (Ma et al., 2019)
73.2
RoBERTa KF (ensemble)
73.3
RoBERTa KEDGN (ensemble)
74.4
XLNet + GraphReason
el al., 2020)
75.3
RoBERTa MHGRN (Feng al., 2020)
75.4
ALBERT + PG (Wang et al., 2020)
75.6
RoBERTa QA-GNN (Yasunaga al., 2021)
76.1
ALBERT (Lan et al., 2019) (ensemble)
76.5
UnifiedQA (11B)* (Khashabi et al., 2020)
79.1
RoBERTa + GSC (Ours)
76.4
Table 5: Test accuracy on CommonsenseQA's of- ficial leaderboard. The existing top system, Uni- fiedQA (IIB params) is 30x larger than our model.
4.2 RESULTS
CommonsenseQA. As shown in Table 4, GSC outperforms the previous best model with 2.57% mean accuracy on In-house dev split and 1.07% mean accuracy the in- house test split. We observe that the per-
Model
RoBERTa-large
AristoRoBERTh
Fine-tuned LMs (w/o KG)
64.80 (#:2.37)
78.40 (#11.64)
RGCN
62.45 (#:1.57)
74.60 (4:2.53)
GioonAtten
64.75 1.48)
71.80 (+11221)
formance variance of GSC is smaller than the baselines, indicating that our GSC are
RN
65.20 (::1,18)
75.35 (1:1.39)
MHGRN
66.85 (:t1.19)
80.6
both effective and stable. On the official leaderboard of Commonsense@A in Table 5, GSC also outperforms all the GNN-based QA systems. Note that the previous top system UnifiedQA (IIB params) uses T5 (Raffel et al., 2020) as the pre-trained LM model, which is 30x larger than our model and uses much more pre-training data.
QA-GNN
67.80 (++2.75)
82.77 (+ 1,56)
GSC (Ours)
70.33 (.10.81)
86.67 (::0.46)
Table 6: Test accuracy on OpenBookQA. Methods with BristoRoBERTa use the textual evidence by Clark et al. (2020a) as an additional input to the QA context.
From Table our GSC outperforms the previous best model with 2.53% mean test accuracy with normal setting and 3.90% test accuracy with the AristoRoBERTa setting. More remarkably, as shown in Table 7, our GSC ranks top one on the official leaderboard of OpenBookQA, which even surpasses the performance of UnifiedQA (11B), which is 30x larger than model.
OpenBookQA.
7

---

Published as a conference paper at ICLR 2022
GSC (scad))
GSC (seddt)
GSC
GN
GSC
MIGRN
GSC
ROBERI
GSC
ALBERT
17%
14%
I6S
USA
60%
60%
6%%
30%
17%
13%
GT
GT
Figure 5: Venn diagrams for the prediction overlap of different models and ground truth (GT) of the IHtest split of CommonsenseQA. The ALBERT has the least overlap since all the left four use RoBERTa to encode the QA context. The order of the percentage of overlap of left four exactly matches the order of the performance of each models: GSC > QA-GNN > MHGRN > RoBERTa. This indicates that the reasoning capability learned by our GSC basically covers that of other GNNs.
4.3 DISCUSSION
As mentioned above, we observe that the ini- tial node embeddings are dispensable. Then start to explore how the maximum number of retrieved nodes (also related edges) affects the model. We experiment various numbers of nodes for our GSC, and summarize the results in Table 8. We find that larger maximum num- ber of node does not benefit the model, and the model achieves the best performance when we use all and only the entity nodes directly occur in question and answer, whose number of nodes is generally less than 32. This indicates that I-hop retrieval is adequate for our methods, and this could be done super efficiently than multi-hop retrieval.
Model
Test
Careful Selection (Banerjee et al., 2019)
72.0
AristoRoBERTa
77.8
KF + SIR (Banerjec & Baral, 2020)
80.0
AristoRoBERTa + PG (Wang et al., 2020)
80.2
AristoRoBERTa + MHGRN (Feng et al., 2020)
80.6
ALBERT + KB
81.0
AristoRoBERTa + QA-GNN
82.8
T5' (Raffel ct al., 2020)
83.2
UnifiedQA (11B)' (Khashabi et al., 2020)
87.2
AristoRoBERTa GSC (Ours)
87.4
Table 7:
Test accuracy on OpenBookQA leader- board. All listed methods use the provided science facts as an additional input to the language con- text. The previous top 2 systems, UnifiedQA (1IB params) and T5 (3B params) are 30x and 8x larger than our model.
To further analyze the reasoning capacity of GSC, we draw the Venn diagram for the pre- dictions of different methods and ground truth (GT) in Figure 5. We find that even for the dif- ferent runs of the same GSC model, the correct overlap is only 69, showing that the datasets are relatively noisy and there exists decent variance in the prediction results. We also observe that GSC has a larger overlap for GNN-based systems (e.g., QA-GNN and MHGRN), while at the same time having less overlap for non-GNN methods. The ALBERT model has the least overlap since all the other methods use ROBERTa as the text encoder.
We also observe that the order of the per- centage of overlap of left four exactly matches the order of the performance of
Model
IHdev-Acc. (%)
IHtest-Acc. (%)
MLP + Counter (1-hup)
78.02 (+:0.05)
73.62 (4:0.1)
each model: GSC> QA-GNN > MHGRN
MLP + Counter (2-hup)
78.30 (#:0.09)
74.13 (+:0.88
> RoBERTa. This indicates that GSC has quite similar behaviors versus other GNN- based systems, and the reasoning capability of GSC is on par with existing GNN coun- terparts. This further reveals that counting plays an essential role in knowledge-award reasoning for QA.
GSC w/ QA nodes
79.11 (1:0222)
74.48 (1:0.41)
w/ 32 nodes
78.52 (+00.58)
74.40 (++0.25)
w/ 64 nodes
78.53 (+:0.77)
73.93 (#t))
128 nodes
78.50 (1:0.96)
72.78 (1:1.15)
w/ 256 nodes
78.32 (1:0.60)
73.89 (1:0.63)
Table 8: Ablation study on the hard counter with MLP (upper) and the maximum number of retrieved nodes (bottom), showing that 1) the hard counter achieves performance on par with GNN-based methods; 2) GSC works well even when only using entities occurred in QA context, which typically contains less than 32 nodes.
To verify our observations, we handcraft a hard edge counting feature feeding into a simple two-layer MLP for ablation study. As shown in Table 8, this hard count- ing model with 2-hop edge feature could achieve a comparable performance of GSC, which even outperforms other GNN baselines. These impressive results not only show that our GSC is effective, but also prove that counting is an essential functionality in the process of the current knowledge-aware QA systems.
8

---

Published as a conference paper at ICLR 2022
Where could personal ficus live? A. front yard
B, cabin the woods
C, califorii
D. conservatory
E, tropical forest
Personal fieus could live the front yard.
Personal ficus could live in the tropical forest.
ficus
Alceettoon
yard
personality
ficus
Allzesting tropical forest
tropical
Aypcr
Tyneo
AtLocutio
Napableof
AtLocation
Related
Sub-graph (Choice A)
front yard
Sub-graph (Cholce E)
forest
2.6
2.1
0.4
K
0.0
0.2
0.2
0.7
0.5
Layer-1 output
Layer-2 output
Layer-2 output
Layer-1 output
Figure 6: Our GSC is highly interpretable. For the retrieved sub-graph of each answer choice, we can directly observe the behavior the model by print out the output edge/node values of each layer, so that we can trace back to see how the model scores the answer.
As shown in Figure 6, for the retrieved sub- sub-graph of each answer choice, we can directly observe the behavior of the GSC by printing out the output edge/ node values (soft count) of each layer. In this way, we can trace back to see why the model scores the answers like that. We list the runtime soft counts of GSC's edge encoder in Appendix A.4. A higher soft count means that the edge/ node is more important, and can contribute more to the final graph score. This demonstrates the advantage of GSC an interpretable reasoning module. In addition, GSC only reasons over the edges in the graph so that the concept embedding is not used, which makes our model better generalize to graph with new entities.
RELATED WORK
KG-powered QA. For KG-powered QA, traditional methods semantic parsers to map the question to logical form (Berant et al., 2013; Yih et al., 2015) or executable program on KG (Liang al.. 2016), which typically require them difficult to handle the noise in questions and KG. To address these challenges, various deep neural models have been applied to KG-powered QA, such as memory networks (Kumar et al., 2015; Sukhbaatar et al., 2015; Miller et al., 2016), neural programmer for knowledge table QA (Neelakantan et al., 2016), neural knowledge-aware reader (Xiong et al., 2019; Sun et al., 2019), etc.
rules and fine- anniigranedd annotations, making
GNN for KG-powered QA. To further improve the neural reasoning capability, recent studies have explored applying GNNs to KG-powered QA, where GNNs naturally fit the graph-structured knowledge and show prominent results. KagNet (Lin et al., 2019) proposes GCN-LSTM-HPA for path-based relational graph representation. MHGRN (Feng et al., 2020) extend Relation Nerworks (Santoro et al., 2017) multi-hop relation scope and unifies both path-based models and RGCN Schlichtkrull et al. (2018) to enhance the interpretability and the scalability. QA-GNN (Yasunaga al., 2021) proposes a LM+GAT framework to joint reasoning over language and KG. We do not use conventional GNNs and our GSC is extremely simple and efficient even without parameters inside the GSC layers. KG embeddings are generally used in these systems. Zhang et al. (2018 8) employs propagation of KG embeddings to perform multi-hop reasoning. KagNet pre-trains KG Embedding using TransE initialized with Glo Ve embeddings. Recently, MHGRN and QA-GNN are proposed, which leverage pre-trained LM to generate embeddings for KG. In addition, QA-GNN also scores KG node relevance using LM as extra node embedding. We follow the data pre- pre-prceessing procedure of QA-GNN and MHGRN, but our model does not use any KG embedding node score embedding.
CONCLUSION
We investigate sate-of-the-an GNN-based QA systems, and discover that they are over- werrpaaametterize and over-complex. Our diagnostic analysis using Spa shows that the initial node embeddings and some GNN layers are completely dispensable. Furthermore, our work reveals that GNN es- sentially works as a counter in the QA reasoning process. To verify this point, we design soft / hard counter models, which achieve comparable or even better experimental results than existing ENN-based methods. Our work is more of explorative and investigational research, which points QA and provides helpful insights for the QA
out how far we go in the area of community to enlighten future research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantization. In International Conference on Learning Representations, 2018.
Pratyay Banerjee and Chitta Baral. Knowledge fusion and semantic knowledge ranking for open domain question answering. arXiv preprint arXiv:2004.03//0 2020.
Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. Careful selection of knowledge to solve open book question answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6120-6129, Florence, Italy, July 2019. Association for Computational Linguistics.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answe -nnswer pairs. In EMNLP, 2013.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collabora- tively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247-1250, 2008.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, Dirk Groeneveld, Michal Guerquin, and Michael Schmitz. From 'F' to 'A' on the N.Y. regents science exams: An overview of the aristo project. Al Mag., (1():399-53 2020a.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. arXiv preprint arXiv:2002.05867 2020b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.0180004800. 2018.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. Scalable multi- hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1295-1309) Online, November 2020. Association for Computational Linguistics.
Song Han, Jeff Pool. John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv:1506.0266 2015.
Song Han, Huizi Mao, and William Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. ICLR, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
Yihui He. Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for Model Compression and Acceleration on Mobile Devices. In ECCV, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
10

---

Published as a conference paper at ICLR 2022
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(5), 2013.
Harsh Jhamtani and Peter Clark. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop creestion-answering In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord. Peter Clark, and Hannaneh Hajishirzi. UnifiedQA: Crossing format boundaries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, November 2020. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8082-8090, 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Ankit Kumar, Ozan Irsoy, and al. Ask me anything: Dynamic memory networks for natural language processing. arXiv, 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:/9000 /942, 2019.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv, 2016.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. KagNet: Knowledge-award graph networks for commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNNL-IJCNLP), pp. 1829-2839, Hong Kong, China, November 2019. Association for Computational Linguistics.
Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime Neural Pruning. NIPS, 2017.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. Proceedings of the Eighth International Conference Learning Representations (ICLR 2020), April 2020.
Yinhan Liu, Myle Out, and Naman Goyal. RoBERTa: A robustly optimized bert pretraining approach.
arXiv preprint arXiv: 1907.11107.119999 5, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.
2020.
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. Towards generalizable neuro-symbolic systems for commonsense question answering. In Proceedings of the First Workshop on Commonsense Inference Natural Language Processing, pp. 22-32, Hong Kong, China, November 2019. Association for Computational Linguistics.
11

---

Published as a conference paper at ICLR 2022
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 -3911, Brussels, Belgium, 2018. Association for Computational Linguisties.
Alexander Miller, Adam Fisch, Jesse Dodge, Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 1400-1409, Austin, Texas, November 2016. Association for Computational Linguistics.
Karimi, Antoine Bordes, and Jason
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2498-2507, 2017.
Arvind Neelakantan, Quoe V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. arXiv, 2016.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.00 010.000066. 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. OpenAl blog, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-61 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 2014.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1902249122 Online, July 2020. Association for Computational Linguistics.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1 20(1):61-80, 2008
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1999-1958 2014.
12

---

Published as a conference paper at ICLR 2022
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In NIPS, 2015.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. Open domain question answering using early fusion knowledge bases and text. EMNLP, 2018.
Haitian Sun, Tania Bedrax- Weiss, and William Cohen. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 2390, Hong Kong, China, November 2019. Association for Computational Linguisties.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-414 4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Alon Talmor. Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Seap-of-thoughh Teaching pre-trained models to systematically reason over implicit knowledge. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Lin (eds.), Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. arXiv preprint 10Xiv:8//1008800 2018.
Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro Szekely, and Xiang Ren. Connecting the dots: A knowledgeable path generator for commonsense question answering. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 4129-4140, Online, November 2020.
Association for Computational Linguistics.
Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, et al. Improving natural language inference using external knowledge in the science questions domain. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7208-7215, 2019.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Improving question answering over incomplete KBs with knowledge-aware reader. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4258-4264, Florence, Italy, July 2019. Association for Computational Linguistics.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 535-546, Online, June 2021. Association for Computational Linguistics.
Wen-tau Yih, Ming- Wei Chang, Xiaodong He, and Jianfeng Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In ACI., 2015.
13

---

Published as a conference paper at ICLR 2022
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander Smola, and Le Song. Variational reasoning for question answering with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-aware quantization for low-bit deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9426 9435, 2018.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. FreeLB: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 MULTI-HOP QA EXPERIMENTS
To apply our GSC to the task of multi-hop QA, we use a popular dataset MetaQA (Zhang et al., 2018), which contains more than 400k single and multi-hop (up to 3-hop) questions in the domain of movies. The questions were constructed using the knowledge base provided by the WikiMovies (Miller et al., 2016) dataset. We use the "vanilla" version of the QA pairs in the dataset, and we use the KG provided by the paper which consists of about 43k entities and 135k triples.
We do not encode the entities information in GSC, since our node embedding is initialized with zero. The edges in the KG are encoded with concatenated one-hot vectors according to the edge type (9 types of relation) and the node type (question/answer entity). Since the task of multi-hop QA does not provide answer choices, thus there is no context node inside the retrieved knowledge graph. We then adapt 3 units of GSC to align with this scenario:
Edge encoder We equip each GSC layer with a independent edge encoder to learn edge information for different hop. The input of the edge encoder is also adapted. Since various of questions can be asked to the same question entity, which means the retrieved graph for different question may identical, we concatenate the I.M encoded question text representation vector with edge type vector as the input of the edge encoder. So that the model could learn different edges embedding for different question text. Except the edge encoder is changed, we keep other parts of or GSC architecture the same.
Methods
1-hop
2-hop
3-hop
KV-Mem (Miller et al., 2016)
96.2
82.7
48.9
VRN (Zhing ul., 2018)
97.5
89.9
62.5
GraftNet (Sun 2018)
97.0
94.8
77.7
GSC (ours)
97.6
99.8
76.8
Table 9: Our GSC achieves comparable performance with the baselines on the MetaQA dataset, which indi- cates that our observations and hypotheses keep consis- tent multi-hop QA.
Output mechanism Since all the retrieved nodes can be the answer entities, we output all the node values as scores and the highest scored entity is our final answer. In the hit@1 setting, if our final answer is in the set of all the answer labels, we mark it correct.
Supervision Since there could be multiple correct answers in the KG given question, we process the labels for each question as a multi-hot vector and use MSE loss instead of Cross Entropy as the supervision the node scores.
We compared our GSC with three standard baselines KV-Mem (Miller et al., 2016), VRN (Zhang et al., 2018) and GraftNet (Sun et al., 2018) on MetaQA and report the hit@I accuracy in Table 9. We can see that our GSC achieves comparable performance with the baselines on the MetaQA dataset, which indicates that our observations and hypotheses keep consistent multi-hop QA.
A.2 How WE CONCLUDE THAT GNN IS A COUNTER
The process we conclude that GNN is a counter is not direct, our finding is based on sufficient experiments and obtained by a step by step process:
Methods
IHdev-Acc. (%)
IHtest-Ace, (%)
GAT w/ hidden=2
77.32 (+0062)
74.08 (=0.12)
w/ hiddenmd
77.07 (+0008)
74.30 (=0.42)
w/ hiddln
77.12 (+:0.3)
74.65 (=0.33)
w/ hidden=16
77.01 (+-0.21)
74.59 (++0668)
We find that the initial node embedding is dispensable and the layers inside the GNN pre-
w/ hidden=32
77.26 (++0.19)
74.62 (+0.43)
w/ hidden=644
77.37 (++0.43)
73.97 (+0.29)
serve relatively low sparse ratio via the Spar- se sD analysis.
w/ hidden==12
77.40 (+:038)
74.21 (+0337)
w/ hiddlen=266
77.56 (+-033)
74.27 (+0.61)
w/ hidden=512
76.80 (+0.54)
73.35 (++0.70)
So we remove the initial node embedding layers and try to set different hidden dimen- sion for GNN layers to see how the number parameters affect the performance. The exper- iments result can be found at Table 10, where we do ablation study on the hidden dimension size of the message passing layers in the GAT model, where we find that the model still works when the hidden size is extremely small, and the
w/ hidden
75.87 (+0.50)
71.34 (+1.09)
Table 10: Ablation study on the hidden dimension size of the GAT model, where we find that the model still works when the hidden size is extremely small, and the model get worse when the hidden size is extremely large due to the problem of over-fitting.
15

---

Published as a conference paper at ICLR 2022
model get worse when the hidden size is extremely large due to the problem of over-fitting. And we keep the hidden size of edge encoder with 32 since we find if the hidden size of edge encoder is extremely small the model will divveen.
Since the hidden dimension size of the message passing layers in the GAT model can not be set to even smaller, we design our model with hidden dimension equal to I and simplify the attention based message passing layers to a non-parameterized operation. We surprisingly find that this model can still work and even work better. So we further study this model and find it works like a counter, so we name it Graph Soft Counter.
4. To further verify our perspective that the GNN works like a Counter, design the hard counter setting, and find that it also works and could achieve comparable performance.
5. We may conclude that current GNNs in QA tasks work like a counter and counting for edges in graph are crucial in reasoning.
A.3 SPARSE RATIO CURVE
We attach the details of the training curve of SparseVD dissection for QA-GNN following. We find the linear query and key layers inside GNN layers and graph pooler generally can be compressed to relatively low sparse ratio and this is more obvious at the fist three GNN layers. This indicated the these layers may be wer-paaameterized and the attention mechanism (Vaswani et al., 2017;
Veličković et al., 2018 8) may degenerate to linear function in this case if we remove these layers.
As expected, the final fully connected layer (FC) preserved the highest sparse ratio, since it serves for directly generating the overall score of choice and it is very efficient (the output dimension is I). Except that, the edge encoder with two-layer MLP has very high sparse ratio, which can be explained as the edge encoding is essential for GNN reasoning in these tasks. This is also why only keep the edge encoder with relatively more learnable parameters and simplify the other components in GNN to design our Graph Soft Counter.
A.4 SOFT COUNT OF EDGES
We list the top-30 edge triplets with highest soft counts that encoded by the edge encoder of our GSC as following. These combinations of edge types and node types have relatively higher counts, which means they can contribute more the final graph score.
A.5 SPARSEVD
To take a deep scope of the graph neural networks based knowledge-awand systems, we introduce neural model pruning method Sparse Variational Dropout (Sparse VD, (Molchanov et al., 2017)) into this scenario as a dissection tool to automatically dissect the graph network architecture. We implemented the Sparse optimization based layer refer the PyToorch code" released by the author. To keep the dissection in strict accordance with the theoretical derivation of Molchanov et al. (2017), we apply regulation to all the linear layers in the GNN modules. We attach the Sparse problem formulation as well as the optimization target as following.
Consider dataset D which constructed from N pairs of objects (Nn, vn) Our goal is tune the parameters w of model p(y] x, w) that predicts given x and w. In Bayesian Learning we usually have some prior knowledge about weights w, which is expressed in terms of a prior distribution p(w). After data D arrives, this prior distribution is transformed into a posterior distribution p(w (D) p(D w) (w))(w)/p(()) This process is called Bayesian Inference. Computing posterior distribution using the Bayes rule usually involves computation of intractable multidimensional integrals, need to use approximation techniques.
One of such techniques is Variational Inference. In this approach the posterior distribution p(w] D) approximated by a parametric distribution 4s (w). The quality of this approximation is measured in terms of the Lulbbeck-Leibler divergence Dx (q)(w) p(w D)). The optimal value of variational
'https://oor research coctearrch.oogglee con/ thub http://ggghhbbbbbbggggeeeee bayesssroot/ddepppasss--aaa / blob/master/see http://dyyy/pppsvevveeeeeeee soluti on. pynb
16

---

Published as a conference paper at ICLR 2022
1.0
1.0
GNN Layerl Key
GNN Layer2 Key
0.8
GNN Layer! Query
0.8
GNN Layer2 Query
GNN Layer! Value
GNN Layer2 Valuc
0.6
GNN Layer! MLP Layerl
sporse ratio
GNN Layer2 MLP Layerl
Sparse
GNN Layerl MLP Layer2
GNN Layer2 MLP Layer2
0.4
0.4
0.2
0.2
0.0%
500
1000
1500
0.0%
500
1000
1500
training step
training stop
1.0
1.0
GNN Layer3 Key
GNN Layer4 Key
GNN Layer3 Query
GNN Layer4 Query
GNN Layer3 Value
GNN Layerd Value
0.6
Layer3 MLP Layer!
sparse ratio
0.6
GNN Layerd MLP Layer!
sparse
GNN Layer3 MLP Layer2
GNN Layer4 MIP Layer2
0.4
0.4
0.2
0.2
0.0
0.0
500
1000
1500
500
1000 training step
1500
training step
1.0
1.0
GNN LayerS Key
Pooler Key
GNN LayerS Query
Pooler Query
0.8
GNN Layer5 Value
Pooler Value
0.6-
GNN LayerS MLP Layerl
sparse ratto
0.6
GNN Vh
sparse
GNN Layer5 MLP Layer2
GNN Vx
0.4
0.4
0.2
0.2
0.0
0.0
500
1000 training step
1500
500
1000
1500
training step
1.0
1.0
0.8
0.8
Score Emb
sparse ratio
0.6
0.6
sparse
Type
0.4
Sentence to Node
0.4
Edge Encoder Layerl
FC
Edge Enceder Layer2
0.2
0.2
Concept Emb (QA-GNN)
0.0g
0.0
500
1000
1500
500
1000
1500
training step
training step
Figure The sparse ratio curve when do Sparse VD training for the QA-GNN systems.
parameters can be found by maximization the variational lower bound:
[(Φ) LD( Dr (go(w) p(w)) + max de
(1)
( = Σ Ey plogp(yn In, w)]
(2)
consists two parts, the expected lig-licld (o( and the KL-divergence DKL (wo(w) ) p(w)), which acts regularization term.
17

---

Published as a conference paper at ICLR 2022
Head Node Type
Relation Type
Head Node Type
Soft Count Value
Q entity
desires (inverse)
entity
0.9632
Q entity
desires
Q entity
0.8902
Q entity
has subevent (inverse)
Q entity
0.8775
Q entity
is
Q entity
0.8470
Q entity
has property
Q entity
0.7344
Q entity
does not desires
Q entity
0.6995
Q entity
does not desires (inverse)
entity
0.6790
Q entity
causes
Q entity
0.6252
entity
has property (inverse)
Q
entity
0.3531
A entity
is
Q entity
0.3073
Q entity
causes (inverse)
Q entity
0.2868
Q entity
is part of
entity
0.2758
Q entity
is capable of (inverse)
entity
0.2706
A entity
has property
Q entity
0.2424
A entity
desires
Q
entity
0.2207
Q entity
is the antonym of (inverse)
Q entity
0.2137
Q entity
is created by (inverse)
Q entity
0.1720
Q entity
is capable of is part of (inverse) desires (inverse) is at location of is created by is at location of (inverse)
Q entity
0.1681
Q entity
entity
0.1581
Q entity
A entity
0.1310
entity
Q entity
0.1006
Q entity
Q entity
0.0990
Q entity
Q entity
0.0914
Q entity
is has property
entity
0.0737
Q entity
entity
0.0636
Q entity
has subevent (inverse)
A entity
0.0617
O entity
is
entity
0.0606
O entity
has property
Q entity
0.0512
Q entity
desires
entity
0.0445
Q entity
is the antonym of
Q entity
0.0399
Table 11: We list the top-30 edge triplets with highest soft counts here, and it is generated by the edge encoder of GSC model. The combination of edge types and node types with higher count means can contribute more to the final graph score.
We formulate the model prunings (parsification) problem pruning problem variational inference problem and use the Kullback-Leibler Divergence approximation following (Achterhold et al., 201 8; Molchanov et al., 2017) to constrain the model parameters to converge the pruning prior: Original -xL was obtained by averaging over 10" samples of e with less than 2 x 10 variance of the estimation.
-
-DKI(q(ws) Ozj, ony) [Iw(w)
ki (A:2 + ha log as;)) - 0.5 log(1 + +
h = 0.63576 k = 1.87320
lg = 1.48695
Where o--) denotes the sigmoid function. We can see that D term increases with the growth of It means that this regularization term favors large values of The case of oi corresponds
18

---

Published as a conference paper at ICLR 2022
to Binary Dropout rate Pij - (recall - Intuitively it means that the corresponding weight is almost always dropped from the model. Therefore its value does not influence the model during the training phase and is put to zero during the testing phase.
We also look at this situation from another angle. Infinitely large aıj corresponds infinitely large multiplicative noise in Wij. It means that the value of this weight will be completely random and its magnitude will be unbounded. It will corrupt the model prediction and decrease the expected log likelihood. Therefore it is beneficial to put the corresponding weight 0z to zero in such way that 0i goes to zero as well. means that g(wn)) 0ig. a:y) effectively a delta function, centered at zero 8(wa)).
Oij - 0,
0 -
&
(4)
q(uay Ozy, 04j)
N (wiy) 10,0) 0) = 5 5(wky)
In the case of linear regression this fact can be shown analytically. We denote a data matrix as X x and 0 E RD. If is fixed, the optimal value of also be obtained in closed form.
0= 0x diag(X (X ))diag(a))- -XT
(5)
Assume that (X (XT)) * 0, so that i-th feature is not constant zero. Then from (5) it follows that 0: O(a,)) when a; + +o,, so both 0, and og0f tend to O.
In our dissection scemario, we optimize the weight 0, zero no longer for compressing the model but for figuring out which part of the model can be pruned out (sparse ratio to zero) without loss of accuracy, which indicates that part of model is not important and its output information is unused in optimization.
19


Reviews:
[{"review_id": "sXP_VNeAXsL", "reviewer": "Reviewer_nabQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is clearly written and easy to understand\n- The analysis of the results done is exhaustive and insightful\n- They get decent improvements in two multiple choice QA dataset, however it is currently unclear how their method will work for non multiple choice datasets.\n\nWeaknesses /Questions for the authors\n\n- The model seems to be designed specifically for multiple choice questions. A context node, all question entities and answer entities connected to them. Moreover the score from the graph is read off from a single node (which I believe is the context node? Line 6 of Algorithm 1). Would the same simple neural counter work for problems where answer choices are not available, for example graph classification problems? I think the paper will benefit a lot if this method is shown to be effective for a dataset which does not have multiple choice questions.\n- The context node has all incoming edges and the mechanism in which the neural counter work all the weights of each edge will be added to the context node? Is there a correlation between how dense the context node is for a question and answer choice? For example, how does a simple baseline which predicts an option based on the number of incoming edges in the context node work?\n- The output of the edge encoder (47 X 32 X 1) converts the edge into a scalar message which is essentially passed around. That part of the network does not take into account any representation of the given question. I am unsure why that is the case. Arent the edge weights supposed to be learn wrt the question. But where is the question involved in this? Is it only in the subgraph creation process. i.e. the subgraph of each question is created wrt the question and that is usually enough to learn meaningful edge weights?\n- In Fig4, the node in the bottom right has a value of 0.3 at the start. However, if I understand correctly, nodes receive score only from incoming edges and nodes are initialzied with zero values. Then how does that node have a non-zero score at any iteration?\n- In algorithm 1, is the qa_context generated for each question answer pair? If so, how does this generalize to a setting when the answer options are not available at all?\n- In figure 6, why is the personality node missing from the graph in the right hand side? The personality node is gotten as a part of the question entity and hence should be common for all graphs\n- Minor: The edge embedding is said to be 47. However, each edge is represented as [u_s, e_{st}, u_t], which makes it 2 * 4 + 38 = 46. Can you tell me where did I miss 1 dimension?", "summary_of_the_paper": "For several knowledge intensive tasks such as question answering, it is common to use a mixture of language models (that encode knowledge implicitly in its parameters) with a graph neural network based model that can encode external knowledge from an knowledge graph (KG). This paper does an analysis of such GNN based QA models that do message passing over an external knowledge graph (KG) to gather external knowledge required to answer a question. They use sparse variational dropout to diagnose the contributions of various parts of the model and find that current GNN modules are over-complicated and over parameterized. Building upon their analysis, they design a simple 1-dimensional neural counter model that counts nodes/edges in the graph and find that they outperform existing complicated GNN architectures, suggesting that those complex architectures might just be performing some simple count-based operations.\n\nIn the design of their simple model they were able to remove all of the dense node embedding layer, represent edges as simple sparse vectors and messages were reduced to single numbers, thus achieving massing improvements in storage and efficiency.", "summary_of_the_review": "The paper is definitely very interesting and is fun to read. I think it would benefit from experiments on datasets which are not multiple-choice. \nAlso, there are applications where the learned entity representations from GNNs are used for downstream tasks. Since this method is advocating for removing node embeddings, there should be a discussion added on how to handle those situations (Note: I personally think removing entity representations is a good idea as it has lot of advantages such as generalizing to new entities etc, but for completeness, I think a discussion regarding the same should be added)\n\n======Update 11/26======\nMost of my concerns have been addressed through the author rebuttal and I am changing my score to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qQat5t3WvmF", "reviewer": "Reviewer_zYDH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I feel this is a great paper that is somewhat \"too good to be true\". The proposed GSC achieves better performance than UnifiedQA on the OpenbookQA dataset but using 1/30 number of parameters (actually is this accurate? find the detailed comments below). The authors also perform several analysis on the proposed GSC, e.g., the number of parameters, ablation of GSC and so on. \n\nMy main question and concern is how/why GSC achieves such high performance with a somehow “disentangled” GNN+LM framework. By “disentangled”, I mean that the final qa_score is a sum of context_score (from LM) and graph_score (from GNN and KG). But it seems that the two terms have no interaction at all in GSC before the summation, especially for the calculation of graph_score. Besides the small number of parameters, I think this is the biggest difference between the paper and prior works such as QAGNN. QAGNN initializes the node embedding of the Context node as the Roberta embedding, and after several rounds of GNN and message passing, the information in the Roberta embedding is propagated to other nodes from the KG subgraph. However, in this work the graph_score is only calculated using the KG subgraph. Then the GSC will perform exactly the same with Roberta in theory under the following circumstances. Consider two different questions but with the same entities and same four answer candidates. Although the Roberta embedding of the question context will be different, the KG subgraph will be the same or really similar. Then the graph_score will also be similar. GSC will rely on context_score to really differentiate the reasoning process of the two questions. One such example being a question and its version with negation added. \n\nSince the proposed GNN model is super simple (less than 10 lines as shown in Algorithm 1), I would strongly suggest the author open source their implementation during the review process and also for broader use of the method later.\n\n- caption in table 2, remove “is” from “which makes our model size is extremely small”\n- caption in table 7, are you sure unifiedQA and T5 are only 30x and 8x larger than your model? I feel these models are even larger.\n- What is the difference between Learnable Param and Model size in Table 2? Why does GSC have the same param and model size?\n- Figure 5 is interesting, it seems that even for a simple model like GSC, different random seeds still have a decent effect and the results still have around 8%-9% variance. Can the author explain the reason behind this?\n- Is conceptnet the only knowledge graph the paper experimented with? I think it would make the paper much more impactful if the author can show that GSC works on another QA dataset and knowledge graph.\n- Can it go beyond multiple choice question answering and find the answers over all the nodes on the KG?\n- Can you give a detailed description on MLP+Counter in Table 8?\n- Can you compare GSC with prior works on structured reasoning, e.g., negation?\n", "summary_of_the_paper": "The paper designs a new GNN+LM model to do question answering and commonsense reasoning over knowledge graphs. The authors propose graph soft counter, a surprisingly simple and effective GNN module that counts edge mentions over the extracted subgraph for a question-answer pair. It has less than 1% trainable parameters compared with state-of-the-art GNN+LM models. \n\n", "summary_of_the_review": "I am currently a little concerned about why the method achieves such high performance, but I would definitely increase my score after my concern/questions are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "qLa1KVlS6iy", "reviewer": "Reviewer_pyaj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Interesting study about overly complex QA solutions that have emerged in recent times and hold the crown of being SOTA.\n- A simple alternative to the complex GNN module for the QA systems which even beats the SOTA numbers in two benchmark datasets.\n\nWeakness\n-  The solution seems to be too good for the simplicity and the performance that it offers. It is hard to believe that such a simple model works so well. There ought to be better explanation/reasoning for one to convincingly accept the empirical findings of the paper about existing GNN. \n- Not much novelty in the proposed solution.\n- Writing can be improved to some extent.\n- Few other concerns which I have stated below.\n\nMain Review\n- Overall, I liked the goal of the paper and line of attack. However, I am still not fully convinced with their finding that existing GNNs mostly do counting of edges, and counting the edges in the graph alone plays a crucial role in knowledge-aware reasoning. \n\n- I am also not fully clear about the diagnostic tool called SparseVD. To make the paper self-contained, it's important to elaborate a bit more on SparseVD. \n\n- Using SparseVD this paper finds that these GNN modules are over-parameterized: some layers in GNN can be pruned to a very low sparse ratio, and the initial node embeddings are dispensable. I agree with this finding but how does this imply that GNN modules are merely counting edges in the KG? This part is unclear to me. Especially, in Section 2.3, the paper looks at the plot of the sparse ratio of some representative layers during the SparseVD training of GNN reasoning modules. Based on these observations alone, the paper seems to conclude that GNN modules are merely counting edges in the KG? This seems to be purely an experiment-based hypothesis but I fail to grasp the underlying intuition. \n\n- Moreover, this appears to be a somewhat straightforward application of SparseVD and the core novelty seems to be lacking.\n\n- The description of the \"Relevance Score\" in Section 2.1 is quite unclear and confusing. A bit of rewriting is required here to bring out clarity. Particularly, the precise definition of the \"context\" here, and how it is retrieved by the masked LM. \n\n- In Table 1 column titles, what is the meaning of “IH..”?", "summary_of_the_paper": "In recent times, there has been a proliferation of QA systems that combine pre-trained Language Models (LMs) with Graph Neural Networks (GNNs) to solve QA tasks requiring human-like reasoning. The pre-trained language models facilitate access to the large knowledge which is implicitly coded into their parameters. On the other hand, GNN kind of performs reasoning over the explicit knowledge available in the form of KGs. Such systems have shown a significant performance improvement. The aim of this paper is to revisit these QA systems. The motivation behind this study is as follows. The paper starts with the premise that today’s QA systems have become more and more complicated and I fully agree with it. Therefore, this paper wants to revisit those systems and ask several basic questions: Whether GNN-based modules are under or over-complicated for QA?, What is the essential role they play in reasoning over knowledge? etc. \n\nTo answer these questions, this paper first analyzes SOTA GNN modules for QA and their reasoning capability. Further, this paper utilizes this analysis to design a simple yet effective graph-based neural counter that achieves improved QA performance on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. \n\nThese existing (LM + GNN) based approaches for QA systems follow a two-step paradigm: 1) schema graph grounding and 2) graph modeling for inference. In Step 1, a sub-graph of KG is retrieved that is related to the QA context and grounded on concepts. This graph is called a schema graph. In Step 2, graph modeling is carried out via an elaborately designed GNN module. This paper essentially analyzes SOTA GNN modules employed in Step 2 and their reasoning capability. For such an analysis, this paper employs SparseVD as a diagnostic tool. Using this tool, this paper finds that existing GNN modules used in QA systems are over-complicated for what they can accomplish in the QA reasoning process. This paper’s analysis reveals that knowledge-aware GNN modules may only carry out some simple reasoning such as counting and that counting of edges in the graph alone plays a crucial role in knowledge-aware reasoning. Based on these findings, this paper shows that even a simple counting model can achieve QA performance comparable to state-of-the-art GNN-based methods. Based on these insights, this paper proposes a Graph Soft Counter(GSC) which is a simple yet effective neural module as the replacement for existing complex GNN modules. The paper shows that with less than 1% trainable parameters compared to existing GNN modules for QA, the GSC module outperforms those complex GNN modules on two benchmark QA datasets. The hidden dimension of GSC layers is only 1, thus each edge/node only has a single number as the hidden embedding for graph-based aggregation. GSC is also claimed to be interpretable because the aggregation of 1-dimensional embedding can be viewed as soft counting of edge/node in graphs. \n", "summary_of_the_review": "See my main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E-KLqQfqQKR", "reviewer": "Reviewer_oUEP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe analysis of existing GNN modules is interesting. The insights on dissection are novel and significant for building better reasoning modules.\n2.\tThe proposed Graph Soft Counter (GSC) is simple but efficient and interpretable, achieving impressive QA performance on two popular benchmarks.\n3.\tThe ablation study in Table 8 is helpful, which shows that even a simple hard counting model can achieve QA performance comparable to state-of-the-art GNN-based methods. \n\nWeaknesses:\n1.\tThe definition of multiple-choice question answering and some terms used in related work are not briefly introduced,  making it difficult to accurately understand some of the content. For example, “answer choice a \\in C” in Section 2.1 is sudden, “relevance score” that “measure the quality of a path and prune the sub-graph”  in Section 2.1 is incomprehensible, “sparse ratio” in Section 2 is counterintuitive (intuitively, the higher sparse ratio means more weights to zero and a part of the model is less important), and “Vh” and “Vx” in Figure 7 are unknown.\n2.\tIn Algorithm 1, the PyTorch-style code may be hard to read for some people, and the variable “inputs” is not stated.\n3.\tThe statement “Our GSC do not use node embedding” in Table 2 is not rigorous. Because node values can be viewed as 1-dimensional node embeddings, and the variable “node_emb” in Algorithm 1 reflects this view.\n\nQuestions:\n1.\tWhy is the input dimension of edge encoder 47? It is inconsistent with the description (2*4+38=46) that follows.\n2.\tIn GSC, why initialize node values to 0 and scale edge values in the range of (0, 1)? Any intuitive explanation？\n3.\tThe two datasets for the experiment mainly rely on common sense, and the complexity of reasoning is relatively low. Do the observations and hypotheses in the paper apply to other complex reasoning (e.g. multi-hop) QA datasets?\n", "summary_of_the_paper": "The paper investigates several state-of-the-art GNN modules and finds that they are over-complicated: the initial node embeddings are dispensable and some layers are over-parameterized. Based on these observations, the paper designs Graph Soft Counter (GSC), a simple graph neural model which basically serves as a counter over the knowledge graph. Although GSC has less than 1% trainable parameters compared to existing GNN modules for QA, it outperforms state-of-the-art GNN counterparts on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. These experiments reveal that counting plays a basic and crucial role in reasoning and existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting.", "summary_of_the_review": "The observations are novel, the proposed model is simple but efficient, and the experimental results are impressive. However, the description of some preliminaries and implementation details is not clear enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a graph soft counter (GSC) model which is very simple and lightweight  compared to the conventional graph neural network for solving QA tasks that benefit from knowledge graphs. Compared to the conventional KG-GNN combination, the proposed method is much simpler but produces better results for QA tasks. The paper originally dealt only with multiple-choice QA tasks, but during the rebuttal process, the authors added more complex QA tasks which the reviewers appreciated. Additionally, there was (and still remains) some concern over the exact reasons and mechanisms behind this "too good to be true" result, and the authors addressed this with additional ablation studies, to be included in the appendix. With the publicly released code, others will be able to try GSC and its too-good-to-be-true performance and figure out how it actually works.
Decision: Accept (Poster)