Understanding Knowledge Integration in Language Models with Graph Convolutions
Authors: Yifan Hou
Guoji Fu
MRINMAYA SACHAN
Venue: ICLR 2022 Submitted (2022)
Keywords: knowledge integration
graph convolution
language model
interpretation
knowledge graph
mutual information

Abstract: Pretrained language models (LMs) are not very good at robustly capturing factual knowledge. This has led to the development of a number of knowledge integration (KI) methods which aim to incorporate external knowledge into pretrained LMs. Even though KI methods show some performance gains over base LMs, the efficacy and limitations of these methods are not well-understood. For instance, it is unclear how and what kind of knowledge is effectively integrated into LMs and if such integration may lead to catastrophic forgetting of already learned knowledge. In this paper, we revisit the KI process from the view of graph signal processing and show that KI could be interpreted using a graph convolution operation. We propose a simple probe model called Graph Convolution Simulator (GCS) for interpreting knowledge-enhanced LMs and exposing what kind of knowledge is integrated into these models. We conduct experiments to verify that our GCS model can indeed be used to correctly interpret the KI process, and we use it to analyze two typical knowledge-enhanced LMs: K-Adapter and ERNIE. We find that only a small amount of factual knowledge is captured in these models during integration. While K-Adapter is better at integrating simple relational knowledge, complex relational knowledge is integrated better in ERNIE. We further find that while K-Adapter struggles to integrate time-related knowledge, it successfully integrates knowledge of unpopular entities and relations. Our analysis also show some challenges in KI. In particular, we find simply increasing the size of the KI corpus may not lead to better KI and more fundamental advances may be needed.

Under review as a conference paper at ICLR 2022
UNDERSTANDING KNOWLEDGE INTEGRATION LANGUAGE MODELS WITH GRAPH CONVOLUTIONS
IN
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained language models (LMs) are not very good at robustly capturing factual knowledge. This has led to the development of number of knowledge integra- tion (KI) methods which aim to incorporate external knowledge into pretrained LMs. Even though KI methods show some performance gains over vanilla LMs, the efficacy and limitations of these methods are not well-understood. For in- stance, it is unclear how and what kind of knowledge is effectively integrated into LMS and if such integration may lead to catastrophic forgetting of already learned knowledge. In this paper, we revisit the KI process in information-theone view and show that KI could be interpreted using a graph convolution operation. We propose simple probe model called Graph Convolution Simulator (GCS) for interpreting is integrated into these models. We conduct experiments verify that our GCS model can indeed be used to correctly interpret the KI process, and we use it to analyze two typical knowledge-enhancee LMs: K-Adapter and ERNIE. We find that only a small amount of factual knowledge is captured in these models during integration. While K-Adapter is better at integrating simple relational knowledge. complex relational knowledge is integrated better in ERNIE. We further find that while K-Adapter struggles to integrate temeerelated knowledge, it successfully in- tegrates knowledge of unpopular entities and relations. Our analysis also show some challenges in KI. In particular, we find simply increasing the size the KI corpus may not lead to better KI and more fundamental advances may be needed.
soowledge-enhaneed enhanced LMs and exposing what kind of knowledge
INTRODUCTION
Pretrained language models (LMs) have achieved tate-offthe--ae performance across various natural language processing (NLP) tasks. Previous works have shown that linguistic knowledge captured quite well by LMs and it plays vital role in their success (Liu et al. However, factual knowledge is sparse and expressed in varied ways in text. Thus, LMs are much worse in capturing factual knowledge about the world (Petroni et al. 2019 Wang al. 2021b). This has led to the development of a variety of knowledge integration (KI) methods which aim to integrate external knowledge into LMs (Colon-Hernande) LMs perform better on mooweedge-relatee tasks, we lack a deep
2019a Jawahar et al. 2019).
al. 2021 Wang et al. 2222 la Zhang al., 2019).
Even though knowledge-enhancee understanding about the inner workings of these models. Better downstream performance indicates that some new knowledge has been integrated, but how much knowledge has been successfully integrated, which type of knowledge is integrated is not well-understood.
To understand what knowledge is learned in LMs, many model-agnostic methods have been pro- posed. Previous works have focused on designing simple classifiers as probe models (Hewitt & Manning 2019 Ribeiro et al. 2016). To get more reliable interpretation. information-theoree
ap- proaches have also been introduced (Guan et al. 2019 Pimentel et al. 2020 Hou Sachan 2021 However, factual knowledge is typically organized as large-scale sparse knowledge graphs (KGs). Previous interpretation methods while being suitable for small linguistic graphs, cannot provide rea- sonable interpretations for large sparse KGs. Prompting is yet another way to understand what fac- tual knowledge do these models learn. Prompts can be designed to let LMs solve fill-in-the- th-ii-thhe-llnks problems. and the prompt performance can be interpreted as probe (Petroni et al. 2019 Shin et al. 2020 Zhong et al. 202I). However, these methods rely on manually constructed templates which very time-consuming.

---

Under review as a conference paper at ICLR 2022
There are two other common challenges to understand KI. First, there are large number of ap- proaches for KI. KI in LMs can be implemented by matching sentences to entities or triples in knowledge graphs called entity-wise integration (Peters et all 2019 Zhang al. 2019) and triple- wise integration (Liu et al. 2020; Wang et al. 2021a). There are several modeling choices (Colon- Hermandez al.; 2021 for KI, including proposing some modifications to the Transformer archi- lecture (Peters et al. 2019 Zhang et al. 2019 Liu et al.) 2020), verbalizing knowledge triples and using data augmentation for finetuning [Agarwal al. [202]] ), and designing objective functions that predict the factual knowledge (Yao et al. |019 Wang et al. [2021a). How to design general method to understand KI in both entity-wise and triple-wise manner challenging. Furthermore, KI is typ- ically implemented in a continual learning setup (Parisi et al., 2019) KI is usually secondary pretraining or finetuning step (Lu et al.] (02T). As new knowledge is integrated, old knowledge could be catastrophically forgotten (CF: Kirkpatrick et al. (016). KI could also lead to situation called catastrophic remembering (CR: Kaushik et al. 2021), where the old knowledge may prevent the integration of new knowledge. Our understanding of these issues is limited.
In this paper, we first revisit the KI process ($2) We formulate KI in an information-thherrtii view (2..)) and construct transformation to approximate general KI process ($2.) Then, we prove that the KI process can be simulated by graph convolution operations ($2.3) Second, we introduce how to analyze KI, CR, and CF. Specifically, we prove that the KI process can be interpreted by graph attention mechanism ($.. Based on that, we propose Graph Convolution Simulator (GCS) model to simulate and interpret the KI process ($3.2). and introduce the way to analyze its inter- pretation results ( ($33)) We show that our designed probe model GCS can correctly simulate and interpret the KI process for two popular Knowledge-enhancee LMs: K-Adapter (Wang et al.) 2021a) and ERNIE (Zhang et al. 2019). We find that K- K-Adapter integrates simple relational knowledge (i.e., entities are leaf nodes in KGs) well, while ERNIE better at integrating complex relational knowledge (i.e.. entities are center nodes in KGs). In our qualitative study, we find that K-Adapter does not learn temporal knowledge at all. We further break down our analysis of KI in terms of type of relations and the popularity of entities. We find that catastrophic remembering often happens to simple relational knowledge (i.c. simple relational knowledge is harder to edit), while complex relational knowledge is often catastrophically forgotten. We also find that catastrophic forgetting easily happens to popular entities, while catastrophic remembering often happens when the entities not very common. Finally, KI quality. We find that there is no apparent positive relationship between them, suggesting that merely building larger KI datasets may not be enough and we may need to make more fundamental advances to build better wewwwldde-enhanneee language models.
investigate the correlation between the size of the KI corpus and
SIMULATING KNOWLEDGE INTEGRATION
In this section, we revisit KI in an information-theorett view, and prove that KI can be simulated by graph convolutions. Specifically, we first formulate the KI process. We use MI to measure the knowledge learned in LMs, and use the change of MI to define KI, catastrophic remembering (CR), and catastrophic forgetting (CF). Then, based on the definition, we construct multistep transformation to approximate the KI process with arbitrary accuracy. We show that KI can only happen in certain steps of on the transformation, which are graph convolutions.
2.1 KNOWLEDGE INTEGRATION DEFNIITION
Before presenting a formal definition of KI based MI, we introduce some basic concepts.
Knowledge graphs. We assume that factual knowledge can be formulated as a knowledge graph g= -=(V,8) E), where nodes U; E V represent entities, and edges in E represent relations between them. Let Nr, denote the set of neighbors of node v; and f; denote the entity label corresponding to the node vg. Further, let x; = Lt(t,) denote the entity (label) representations of i; given by a LM Let X E IR Vixd denote matrix formed by stacking all entity representations æ;, Rdd. In this paper, we only consider nodes and relations in the KG and ignore other kinds of KG information such as edge weights, edge directions and multi-edges (multi-relations).
We represent each entity the average of its word(-piece) embeddings given by the LM as Hewitt & Manning 2019) Hnd & Sachan 202T)
2

---

Under review as a conference paper at ICLR 2022
Information-theoretie probe. We follow theoretical settings of & Sachan (2021) that measures the knowledge captured in LMs by MI. We assume that the local graph structure (v,) contains all the factual information regarding v.. In this work, we consider factual knowledge in the form of triples (Us, 1.). Since a triple only contains entities within one-hop. it suffices to set ((v)) = Nu;- Factual knowledge that has been successfully integrated should be reflected in entity representations. Let x be a random variable that takes values ranging over all possible entity representations of LM and g be a random variable that ranges over all possible corresponding local structures ((v)). Intuitively, MI g) measures the amount of information in that is contained in x.
Definition 1 (Knowledge Integration). Given G. let entity representation matrices given by a LM before and after KI be and H. Corresponding random variables are g, x, and h. We formulate KI process fxx. g) = h as the change of MI: MI g) + MI(h; g), i.e., knowledge of is integrated into x to get h. [fM g) Rs MI(x; g), means that CR happens, i.e., there a failure to integrate new knowledge. If CF happens, we have M[(;; x) R 0.
The definition can be intuitively visualized by Figure 1 The MI change can be represented by regions and in the Venn diagram. Ideally, if we have MI(h; g) - MI g) = MI[g: g) MI[x; g). i.e., the region is small, we say most knowledge is successfully integrated. If little new knowledge has been integrated, the region 4 is very small. Then, we say that CR has happened. If CF happens, most knowledge in x is forgotten in h after KI, and region is large. Successful KI happens when much new knowledge is integrated (i.e., MT g) is large) and litrle old knowledge is forgotten (i.e., MT(h; x) is large).
3
4
5
h
Figure 1: Venn diagram for MI visualization. h, and g random variables.
2.2 APPROXIMATED TRANSFORMATION CONSTRUCTION
In this subsection, we show that we can construct a transformation to approximate the KI process with arbitrary accuracy. We begin by introducing the concept of Graph Fourier transforms.
Graph Fourier transformation. Graph Fourier transform (GFT) can be used to transform the entity representation matrix X in the Euclidean space to the graph spectral domain (i.e., KG space ). Specifically, let A E IR V|x[V| be the symmetric adjacency matrix corresponding to G. Let L, I D 1-12 AD- 1/2 denote the normalized Laplacian matrix for g. where D denotes the degree matrix of g. We do the eigendecomposstion for L,, L, UAUT, where U is the matrix of eigenvectors ordered by eigenvalues and A = ddiag Ay X2, AN) is the diagonal matrix of eigenvalues. Based on the GFT (Sandryhaila & Moura 20144,, formally, the transformation of X to the KG space can be written as CFT(X) U X, and its inverse transformation can be written as (GFT UGFT(X)= X.
It is intractable to directly simulate the KI process (with simple probe models), i.c., integrating knowledge information from g into x to get h (Definition since g is graph data that are not the Euclidean space. Fortunately, with the help of GFT, we can transform x and h into the KG space, and design transformation there to simulate the KI process. Below theorem provides a feasible way to construct a transformation that can simulate the KI process with arbitrary accuracy.
Theorem 2 (Transformation Existence). Denote the graph Fourier transformation and its inverse transformation in terms of G as GFT(-) and RGFT(-). Given a LM and its knowledge-enhame version, suppose that MI(h; g) MI(x; g) > O, and there exists a mapping that satisfies f(x, g) h. Then, for any e >> 0, there exists a neural network NN(.) such that
If g) RGFT(NN(GFT((O)) <.
(1)
The proof can be found in Appendix Theorem 2 shows that there exists an approximated transfor- mation composed of GFT and neural network that simulate the KI process (i.e., f(x, g) = h) with arbitrary accuracy. In practice, the mapping fxx, g) is realized by complex KI training pro- cess on large LMs with new datasets, objectives, or even new LM parameters. Here, we can use a simple transformation RGFT(NN(GFTT())
to generally approximate and simulate the com- plex KI process. However, KGs are normally very large, and computing the eigendecomposition of
Here, the of entity representations X R
[V] | he regarded as empirical samples from x.
3

---

Under review as a conference paper at ICLR 2022
the Laplacian matrix for GFT is prohibitively expensive. Besides, the transformation is not inter- pretable, which can not be used as a probe model directly. Thus, we take a deeper look into this transformation. We first show that it can be further simplified graph convolutions.
2.3 KNOWLEDGE INTEGRATION SIMULATION WITH GRAPH CONVOLUTIONS
In this subsection, we introduce details of the transformation in Theorem 2 We prove that even if there are multiple steps of the transformation, MI change only happens certain steps. We show that these steps are equivalent to graph convolutions.
Graph convolutions. Convolution operations on graphs are often used model relational infor- mation of KGs, where entities aggregate information from their neighbors and pass the informa- tion along based on the graph structure. Graph convolutions can be implemented by filters ge in the graph spectral domain (i.e., KG space). As the GFT of the convolution of ge and X is the pointwise product of their GFT (Bracewell & Bracewell] 1986), the convolution can be written as ge X RGFT GFT(X)) (Bruna et 2014))
In the below proposition, we show that the multistep transformation for KI simulation (Theorem 2) can be simplified graph convolutions.
Proposition 3 (Graph Convolutions for Simulation). Suppose MT(x; g) < MT(h; g). MT(h; < xI, and the mapping f(x,g) h be well approximated by the transformation (NFT(NN where the neural network NN(-) has layers. We can only use linear functions in the neural network (i.e., graph convolutions) to well simulate the MI change.
Proposition 3 indicates that if some new knowledge is integrated and some old knowledge is forgot- ten during KI, we can only use n linear functions instead of the whole transformation Theorem P to simulate the MI change (i.e., formulated KI process). These linear functions in the KG space are actually graph convolution operations. And thus, we can circumvent the expensive eigendecompo- sition in GFT with fast and well-approximated graph filters (Defferrard et al. 2016 Kipf & Welling 2017). The formal proof is in Appendix Figure briefly illustrates the overall idea.
The black dashed transformation pro- posed in Theorem P can simulate KI with arbitrary accuracy. Even if the transfor- mation is simple, it is still not very efficient with large KGs and not interpretable. The red solid lines repre- sent the simulation of KI using graph convolutions (Proposition
gairyloss
Noniinear
GFT
Linear function
activation
RGFT
Graph space
KG
Bijectivn function
Pijective function
convalution
function
Representation X
Pretrained UM
Entry Text
Knowledge- enhanced UM
Representation f
Figure Illustration of the KI simulation. Black dashed show the approx- imated transformation Theorem Red arrows show the simplified simulation
with graph convolutions in Proposition 3
3). which are much more fast and interpretable.
According to the invariance property of MI (Kraskov et al., 2004), the introduction of bijective functions does not introduce any new information MI remains unchanged upon the introduction of bijective functions. We know that GFT and RGFT are both bijective (Appendix C.T). We show that nonlinear activation functions in neural network (e.g., sigmoid(-)) are bijective well (proof in Appendix (C.T). Thus, the MI change in the KI process can only happen in the linear function (proof in Appendix C.2). Based on the convolution theorem (Bracewell & Bracewell 1986), linear functions in graph space are graph convolutions (Sandryhaila & Moura [0014] Bruna et al. 2014] Kipf & Welling 2017) (Appendix C.3) Thus, we can simply use graph convolution operations instead to simulate the KI process.
3 INTERPRETING KNOWLEDGE INTEGRATION
In the last section, we showed that KI can be simulated by graph convolutions. In this section, introduce the mechanism to interpret the KI process. Specifically, first illustrate that the
4

---

Under review as a conference paper at ICLR 2022
graph attention mechanism be used to interpret graph convolutions. Then, we introduce an implementation of the probe model proposed in our work: Graph Convolution Simulator (GCS), as well as a method to amalyze our interpretation results.
3.1 KNOWLEDGE INTEGRATION WITH GRAPH ATTENTION
Following Proposition we can select powerful graph filters to simulate and interpret the KI pro- cess. Velickovic al. (0118) and Thekumparampi al. (2018) introduce the attention mechanism to graph filters, where the contribution of each edge to the convolution can be shown explicitly. Graph attention makes filters more powerful and convolutions more interpretable (Fu et al. 2020).
Proposition 4 (Graph Attention for Interpretation), Suppose Mī(x:g) < MI g), MI(h; x) < M x), and the KI process M[(x;g) g) + MT(h; g) can be well simulated by n graph convolution operations. Then, we can use the attention coefficients on edges and self-loops to interpret the KI, CR, and CF.
The formal proof can be found in Appendix [ Note that we have n graph convolutions, and they function differently (Bruna et al. (014). In multi-layer graph convolution network, the k-th graph convolution step aggregates information from k-hop neighbors In this work, we consider integra- tion of knowledge in the form of knowledge triples. Knowledge triples link entities within 1-hop. Thus, in practice, we set n = 1 for simplicity in our work. According to Fu et al. (2020), graph attention can also be seen edge denoising. This provides alternative explanation of our GCS probe from the denoising view. More details can be found in Appendix D
3.2 GCS ARCHITECTURE.
Based on Proposition and Proposition 4 we design GCS with two bijective functions and one graph convolution function in between. To implement bijective function in practice, we show that special MLP layers can be bijective if the weight matrix square matrix (plus a small noise). The formal description and the proof are in Appendix E convolutional layer and two bijective MLP layers as:
We design our GCS model with one graph
GCS,, (.) MLP,, n (GC(MLP,
where MLP,.. (.) is the bijective MIP layer and GC(,, 9) is the graph convolutional layer on the KG which is used to simulate and interpret the KI process. Given entity v; and its set of neighbors No., we can write the graph convolutional layer as:
GC(x:)= - ( U.E.
(Wlx.) (W* x3)) x,)).
Or W. x,
where ot, = softmax
(3)
Vdu. t
Here, ar is the entity representation of v; before knowledge integration, the activation function (-) is ELU(.) function, and W is weight matrix. at,j is the attention coefficient on the edge that connects vú and Vj. w and W are two parameter matrices in the graph attention. dę is the dimension of vector W K zj, and softmax(-) is the edge-wise softmax function with respect to node ws Temperature I a hyperparameter that controls the attention distribution to be hard or soft. As the multiplicative attention mechanism (Vaswani et al. (017)) is broadly used in LMs, we also select multiplicative attention in the graph attention. We optimize GCS by letting its outputs be as close to h as possible. This can be achieved by using reconstruction loss minimization maximizing MI between the outputs of GCS and h. We use MI maximization in our implementation. More implementation details can be found in Appendix H
3.3 ANALYZING INTERPRETATION RESULTS IN PRACTICE
We implement the GCS model in Equation 2 and use attention coefficients in Equation 3 for all relations and entities in the KG for interpretation. Then we analyze these interpretation results to get conclusions for the KI process.
Note that the number of graph convolutional layers decides the receptive field of entities. layers represent that each entity can get information from its n-hop neighbors.
5

---

Under review as a conference paper at ICLR 2022
As introduced in $3.1 large edge attention coefficients mean that the triples corresponding to the edge are integrated well. To understand CR and CF, we add self-loops on entities in the KG, and use attention coefficients on the self-loops to show how much of the original information is remem- bered/forgotten for entities. In particular, large self-loop attention coefficients mean that the original entity information is kept well. Thus, we introduce thresholds analyze our interpretation results as followe We simply regard triples with attention coefficients 0z,j > 0.1 on edges as integrated triples. Entities with attention coefficients 0.4 < 04 < 0.6 on self-loops as well-learned entities. Entities with as < 0.1 on self-loops means CF has happened, where much new factual knowledge information is integrated and original entity information forgotten. Correspondingly, as,; > 0.9 means CR has happened. For interpretation, attention coefficients on edges are used for triple-wise integration, and those on self-loops are for entity-wise integration.
4 EXPERIMENTS
We first introduce two knowledge-enhancee LMs considered in this work: K-Adapter (Wang al. 202Ta) and ERNIE (Zhang et al. 2019). KG is integrated in a triple-wise manner in K-Adapter, and entity-wise manner in ERNIE.
Then, we move on to our experiments. First, we verify our GCS model. We prove that GCS can correctly interpret how much KG information is integrated, as well as which set of entities and triples are integrated. After that, we use GCS to interpret the KI process for K-Adapter and ERNIE. We present the interpretation results and find that both K-Adapter and ERNIE have only integrated few triples. but they integrate many entities. Finally, we use our probe to understand which kinds of knowledge is integrated well in these models. In order to do this, we stratify knowledge in terms of various relation type and find that K-Adapter and ERNIE integrate different kinds of factual knowledge to different extents.
4.1 KNOWLEDGE--ENHANE MODELS
Below, we describe the two knowledge-enhanced LMs considered in this work:
K-Adapter. K-Adapter takes RoBERTa (Liu et al. 2019b) as the backbone model and inserts three new layers into RoBERTa to learn new knowledge. The final output is concatenated with the output of ROBERTA During the integration, parameters of RoBERTa are frozen, only parameters of the newly inserted layers are updated. K-Adapter uses the T-REx-re (ElSahar et al. 2018) dataset for KI, which has an alignment of natural sentences with knowledge triples in Wikidata. For the KI objective, K-Adapter decides whether certain relations exist not, and classifies relation labels given the aligned sentence. As knowledge is integrated newly inserted layers, the model no longer needs to use the T-REx-re dataset for inference, and it can be finetuned like any other pretrained LMs on downstream tasks.
ERNIE. ERNIE integrates factual knowledge into BERT (Devlin et al. 2019) directly without in- troducing extra parameters. The Wikipedia corpus and Wikidata knowledge triples are selected for integration. As there is provided alignment between natural sentences and knowledge entities, ERNIE uses TAGME (Ferragina & Scaiella| 2010) to extract entity mentions in sentences and aligns them with corresponding entities in KGs. A new objective is designed for KT in addition the stan- dard MI. and NSP objectives: alignments in the input text are randomly masked, and the model asked to select aligned entities from KGs. Different from K-Adapter that stores factual knowl- edge in newly introduced parameters, when ERNIE finds the aligned entity, its embedding obtained from Bordes et al. (2013) integrated into the output representations. Thus, during inference, the KG and its embeddings are still required to be fed into ERNIE.
4.2 GCS VERIFICATION
We design a set of experiments to verify that GCS can indeed correctly interpret the KI process. We first verify that GCS can correctly interpret how much knowledge is integrated using synthetic
'Note that users may choose different thresholds. We heuristically set these thresholds for our analysis. $ Note that here only consider factual knowledge, thus, the linguistic Adapter is used.
6

---

Under review as a conference paper at ICLR 2022
experiments ($4.2.1). Then, verify that GCS can correctly interpret which type of knowledge is integrated based on the KI dataset ($4.2.2) and downstream task performance $42.3
4.2.1 SYNTHETIC EXPERIMENT
Setting. We create synthetic KI scenario where different amounts of knowledge is integrated into the LM. Specifically, we first use DeepWalk (Perozzi et al.) 2014) to obtain entity embeddings (i.e.. KG embeddings) in the KG used for KI. We regard entity embeddings as the entity repre- sentations of knowledge-enhanee entity embeddings and regard it as the entity representations of vanilla LMs. Then, we simulate the KI processes from vanilla LMs (noisy entity embeddings) to knowledge-enhance LMs (entity embeddings). Large noise ratio means much information of KG is integrated.
LMs, and we add Gaussian noise with different noise ratios on
Baselines. We select several baselines for compari- son. Representation analysis: we calculate the sim- ilarity (e.g., Cosine, Fuclidean) between two con- nected entities to estimate how much knowledge about the corresponding triple is contained in the LM. Finally, the similarity gap between vanilla LMs and seowledgeeenhnneee LMs can be used to in- terpret how much knowledge is integrated. Linear probe: we design a linear classifier to do link predic- tion in the KG based on entity representations. The performance (i.e., AUC score) can be used show how much knowledge is contained in the LM, and its gap shows how much knowledge is integrated. As for GCS, we use the mean value of self-loop atten- tion coefficients to show how much knowledge is in- tegrated: large values mean that little knowledge is integrated. For the convenience of observation, we report one minus its mean value the integration score.
Casine simlanty
Linear dlassifer- - AUC
CS FAME
LC- HRME
CS K-Acapter
KAdapter
siilartt
ES ERNIE
ERNIE
ES KAddsteer
GCS- KACapper
0.4
0.2
0.0
0.2
0.0
0.2
0.4
0.6
0.8 1.0
Ratio of noise on the input KGE
Figure 3: Interpretation results of how much
knowledge is integrated based different meth- ods. Solid lines are results of synthetic KT pro- cesses. Dotted lines and dashed lines show results of K Adaatet and ERNIE.
Implementation details are in Appendix F
Figure 3 shows the interpretation results. Solid lines show the results of synthetic KI pro- cesses. We find that GCS and representation analysis methods provide correct results: the similarity gap/score increases as the noise ratio increases. However, linear classifier probe can only differenti- ate whether new knowledge is integrated not, and fails to tell how much knowledge is integrated (its curve is not very monotone.). In practice, dotted lines and dashed lines show interpretation re- sults of K-Adapter and ERNIE. We find that only GCS provides reasonable results: they integrate little knowledge (equivalent to noise ratio = 5%). All baselines fail in interpretation: K- K-Adapter and ERNIE do not integrate any factual knowledge, or they even forget learned factual knowledge.
Results.
4.2.2 VERIFICATION USING THE KI DATASET
Setting. This experiment is composed of three steps. First, we use GCS to interpret the KI pro- cess in K-Adapter and ERNIE, and identify triples and entities that are integrated successfully. Second, we retrain BERT/ROBERTa to get K-Adapter (dropped)/ERNIE (dropped) only using the triples/entitics that are identified as successfully integrated. Third, we finetune K-Adapter/ERNII and their dropped versions on downstream tasks. If GCS correctly interpreted the KI process, the performance of their dropped versions on downstream tasks should be roughly the same as that of K-Adapter/ERNIEE
As introduced in $4.1 K-Adapter and ERNIE integrate knowledge by aligned natural sentences in- stead of using triples/entities directly. Thus, after we get the interpretation results, i.e., triples with edge attention coefficients larger than 0.1 or entities with self-loop attention coefficients smaller than 0.9, we only keep data aligned with the integrated knowledge. Specifically, we drop sentences for K- K-Adapter and entity embeddings (obtained by Bordes et al. (2013) for ERNIE. Then, we finetune them on two downstream tasks (entity typing) that K-Adapter and ERNIE outperform RoBERTa and
BERT most significantly: OpenEntity (Choi et al. (2018) and FIGER (Ling et al.) 2013). Implemen- tation details of reproduction and GCS can be found in Appendix G and Appendix
7

---

Under review as a conference paper at ICLR 2022
Results. We only keep 10.09% natural sentences that aligned with success- fully integrated triples for K-Adapter (dropped), and
Table 1: Performance of K-Adapter, ERNIE, and their dropped versions on entity typing on the OpenEntity and FIGER datasets.
Model
OpenEiitity
FIGER
R
FI-Micro
R
FI-Micro
RoBERTa
76,98
73.42
75.16
65.26
88.72
75.20
61.72% entity embed- dings (obtained by [Boddes et al. (2013)) for ERNIE
K-Adapter
76.63
75.26
75.94
67,50
88.79
76.69
K-Adapter (dropped)
75.95
75.95
75.95 1
67.29
88.88
76.59 L (6.71%)
BERT
79.68
65.70
72.02
75.59
62.32
68.32
ERNTE
78.24
68.75
73.19
77.39
65.81
71.13
ERNIE (cropped)
78.11 4
71.431
74.62
77.38
64.90
70.60 & (18.86%)
(dropped). More details
are in Appendix [ From Table [ we can find that even if we drop large amount of KI data in this way, the performance of K-Adapter (dropped) and ERNIE (dropped) on entity typing task is roughly the same as original (reproduced) versions, and they are obviously better than that of BERT and RoBERTa. We further verify GCS by introducing a random dropping strategy as comparison on the OpenEntity dataset. Detailed results can be found in Appendix
4.2.3 VERIFICATION WITH A DOWNSTREAM TASK
Settings. In this experiment, we combine the interpretation results of GCS with downstream task. Specifically, we align entities in the KI dataset and the OpenEntity dataset based on their Wikidata Q identifie For the entity typing task (OpenEntity dataset), we drop the finetuning test data samples that aligs with the integrated knowledge and non-integrated entities (called drop-IE test set and drop- UE test set), and test K- K-Adapterr and ERNIE on the two dropped test sets. If GCS correctly interpret the KI process, Knowledge-enhancee LMs should perform better on the drop-UE test set and worse on the drop-IE test set.
Results. Table 2 presents detailed results. We can find that for K-Adapter, the gap is not very obvious. We hypothesize that this may be because of the differences in the finetuning objective and the KI objec- tive, and because the knowledge integrated in K-Adapter may change during finetun- ing. As for ERNIE, the gaps are signifi- cant. The performance (FI-Micro) on the test set (drop-IE) is 20 Fl points worse than that on the complete test set. These results also verify that GCS can correctly interpret which set of knowledge is integrated.
Performance change of K-Adapter and ERNIE on the OpenEntity dataset with different test scts.
Table 2:
OpenEntity
Model (Test set)
Left test set
R
FI-Micro
K-Adapter (drop-IE)
37.44%
0.33
0.37
0.35
K-Adapter (drop-UE)
64.46%
0.18
1.12
0.47
ERNIE (drop-1E)
27.28%
18.20
25,14
22.67
ERNIE (drop-
66.87%
0.31
+ 3.08
1.57
4.3 GCS FINDINGS
After verifying GCS with three groups of experiments, we analyze the interpretation results. From macro view, we find that both K-Adapter and ERNIE integrate few knowledge triples (sx 20% 30%) and some knowledge entities (<x 60% - 70%). Detailed results can be found in Appendix K From a micro view, classify knowledge based on relation types (in terms of their topology type and Wiki data type) and analyze how K-Adapter and ERNIE integrate them respectively.
KI analysis for K-Adapter and ERNIE in terms of relation topology. We classify relations into three types based on their topology features. Specifically, relations that connect two leaf nodes (entities) in the KG the KG are N - M relations. Others are N - 1 relations. For the analysis results in terms of different types of relations, we report the percentage of successfully integrated triples and entities for K- K-Adapter and ERNIE. Besides, we also present the percentage of catastrophic remembered (CR) entities and catastrophic forgotten entities (CF).
relations, and relations that connect two center nodes (entities) in
Table 3 presents specific results. We find that for K-Adapter, triples with N - -M relations are not captured well. However, K-Adapter integrates triples with 1 relations well. This phenomenon is common for Transformer encoders, where knowledge with complex structure cannot be captured well (Petroni et al. 2019). ERNIE shows different behaviors. We find that entities connected with 1 1 and N M relations are captured well. But for entities connected to N 1 relations, they
8

---

Under review as a conference paper at ICLR 2022
Table 3: Analysis of KI interpretation results for K-Adapter and ERNIE in terms of different types of relations (topology feature). The percentages of integrated triples/entities, as well as of CR and CF entities for each type of relations are presented.
Model
K-Adapter (Wang al. 2021a| on T-REx-re
Statisties
relation
N I relatson
N
A8 relation
Total
# of triples
21,690
813,674
1.729,644
2,565,008
Integrated triple percentage
58.89%
38.39%
24.00%
28.86%
# of connected entities
21,690
406,,37
352,748
781,275
CR entity percenlage
41.11%
31.72%
26.02%
29.41%
CF entity percentage
26.40%
30.29%
40.89%
34.97%
Model
ERNIE (Zhang et al.. 2019] on Wikidata relation
Statistics
relation
relation
NV
Total
connected entities
1,799
529,186
2,744,549
3,275,534
Integrated entity percentage
70.65%
42.86%
73.33%
68.39%
CR entity percentage
29.41%
56.07%
26.67%
38.28%
CF entity percentage
23.18%
8.65%
37.10%
32.49%
are not integrated well. Since ERNIE relies on KG embedding to learn structure knowledge, KI is highly consistent with the quality of the KG embedding provided in Bordes et al. (2013). Regarding CR and CF, we find that for both K-Adapter and ERNIE, CR happens more often to entities in simple structures (i.e., connected to 1 1 or N relations), while CF is more common for entities in complex structures (i.c., connected to N M relations).
KI analysis for K-Adapter and ERNIE in terms of relation's Wiki features. We select six relations aligned with roughly the same number of sentences in the T-REx-re dataset (see Ap- pendix D for statistics) and cat- egorize them into three groups based on the Wiki Count and Wiki data rype] low-frequency (LF) relations, time-related (TR) rela-
Table 4: The interpretation of KI for K-Adapter in terms of relations.
We list 6 relations and classify them into three types based on the Wiki Count and Wiki data type. The ratio of integrated knowledge triples reported.
T-REx-re
WIKi Count
Wiki data type
Intograted triple percenlage
Place of birth (I.F)
2,850.424
Wikihase item
10.95%
(LF)
4.164.470
Wikibase
17.25%
Date of death (TR)
2,637,358
Time
<0.01%
Date hirth (TR)
5,294,649
Time
<0.01%
Located administrative
territoiial catity (HF)
10,776,120
Wikibase item
6.13%
Country (HF)
14,174.811
Wikibase
tions, and high-frequency (HF)
Tocal
10.0%%
relations. From Table 4 we find that even if LF relations has roughly the same Wiki Count as TR relations. However, since the Wiki data type of the latter set is "Time", triples with those rela- tions cannot be integrated by K-Adapter. We speculate that this is because Transformer encoders do not capture information about time well (Dhingra et al.) 2021 Zhou et al. (00T). When comparing LF relations and HF relations, find that if relations have small Wiki Count, knowledge triples are easier to be captured during KI.
Above experiments show that interpretation results of GCS are consistent with existing analysis works, which also verify GCS indirectly. Besides, we design a case study experiment in Ap- pendix M where we find that CR often happens to rare entities (small Google Ngrams), while CF of- ten happens to popular entities (large Google Ngrams). Moreover, we study the correlation between the edge attention coefficient and the number of aligned sentences for K-Adapter in Appendix N and find that the Pearson Correlation Coefficient is -0.0055. It implies that simply increasing the Kl dataset may not help LMs integrate unlearn knowledge.
LIMITATIONS AND FUTURE WORK
In this paper, we illustrate that the graph attention can be used to interpret the KI process for knowledge-enhane eetangeeeehnneddd LMs, and thus propose a Graph Convolutional Simulator (GCS) that is capable of correctly interpreting existing Knowledge-enhanced LMs. In our experiments, we verify GCS and use it to obtain interesting findings. There are some limitations of our work. We simplify the KG without considering edge direction, labels, multi-edges, entity descriptions, and timestamps. These can be considered by future works. Besides, GCS only provides way to interpret the knowledge integration. Once we have an understanding of the KI process, improving the integration quality still remains challenging.
reports/l.is .ist. of properties/all

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We will publish the code, as well as the interpretation results after the review process. The design of GCS can be found in $ The implementation details about the KI for K-Adapter and ERNIE can
be found in Appendix And details about GCS can be found in Appendix H
ETHICS STATEMENT
While our probe models are not tuned for any specific real-world application, our methods could be used in sensitive contexts such as legal or health-care settings; and it is essential that any work that builds on our approaches undertake extensive assurance and robustness testing before using in their setting.
REFERENCES
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based syn- thetic corpus generation for knowledge-enhanced language model pre-training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tüir Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), NAACL-HLZ, pp. 3554-3565 As- sociation for Computational Linguisties, 2021. doi: 10. 18653/v1/20211 .naacl-main.275 URL https: /ttps://ddo..org//// 11w..0.111311/1311
www.aaaallaaaaaaa meaec-mmii... 278
Arindam Banerjee. On bayesian bounds. In William W. Cohen and Andrew W. Moore (eds.), ICML, volume 148 of ACM International Conference Proceeding Series, pp. 81 81-88. ACM, 2006. doi: 1 145844 10.1144/114444411 143355.. URL https: //d::///oo.... orreii.orr11110
01444//114344444 1143855]
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. De- von Hjelm, and Aaron C. Courville. Mutual information neural estimation. In Jennifer G. Dy and Andreas Krause (eds.), ICML, volume of Proceedings of Machine Learning Research, pp. 530- 539. PMLR, 2018. html
URL http::/prroceednngg mlr wttpss//v//beeggaa.......
Antoine Bordes, Nicolas Usunier, Alberto Gareía-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Christopher C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Neural Information Processing Systems, pp. 2787-2795 2013. URL hts http:::e//ppoooeeiii mew.eeeiiiinnnnnninno...ee....ne neurips....p.... wttpuriissccppppppppeeepp cc/paper/ 2013/hash/ cecc 7 7792877777888 333 680.aa22600882222228 Abstract htm
Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2014. URL
http://arxiv.org abs/1312...... 6203
Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference
Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, www.iii............ www.eeeww.....a.eeeeeeeee cctppppeepp11////// pp. 11866-15876. 2019. URL https: http::::s//poceeeedieoo 1 haab://leee110000222284422133333
html
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. In Iryna Gurevych and Yusuke Miyao (eds.),.ALL. pp. 87-96. Association for Computational Linguistics, 2018. doi: 10. N..18865/v1PPP 1/133//1/PP1/// 1009. URL https: / anthol org..org/o.......... 8- 1009 9009
Pedro HelonnHernandez, Catherine Havasi, Jason B. Alonso, Matthew Huggins, and Cyn- thia Breazeal. Combining pre-trained language models and structured knowledge. CoRR, abs/2101. 2021. http::/t/:/: wwtp///////a///////eo 12294
10

---

Under review as a conference paper at ICLR 2022
George Cybenko. Approximation by superpositions of sigmoidal function. Math. Control. Sig- nals Syst. 5(4): 1992. doi: 10.1007/BBF22144016
BF BFC2134016
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Neural Information Processing Systems, pp.
3837-3845, 2016. URL http::ttp///s.. httpss:://////////////sso ings. neur iprrips... http/paper/211//mmmge 04 44df44434d44 (e1.444444811666222223 25l 1ee65- Abstracte..................t.tte html
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), NAACI-HITT, pp. 4171-4186 Association for Computational Linguistics, 2019. doi: 10.18653/v 1/m 1423. URL httpt::tpp///c.... wtt.:/..........13/1////////////
10. 11653/v1/n133 1423333
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases. CoRR, abs/2106.151 10, 2021. URL http:////xxi.oooooee.oo. haagg/bb//21in wtt.:orr/abb/.....1111111
Hady ElSahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Simperl. T-rex: large scale alignment of natural language with knowledge base triples. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Kôiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), LREC. Irec-conf. European Language Resources Association (ELRA), 2018 URL:ttp:://ww ore/pproceedings//reec/o0 htt.///sumuarreeeeeee 632.6322200 html
Paolo Ferragina and Ugo Scaiella. TAGME: on-the-fly annotation of short text fragments (by wikipedia entities). In Jimmy Huang, Nick Koudas, Gareth J. F. Jones, Xindong Wu, Kevyn Collins-Thompson, and Aijun An (eds.), CIKM, pp. 1625-1628. ACM, 2010. doi: 10.1145/
1871437.1871668
Guoji Fu, Yifan Hou, Jian Zhang, Kaili Ma, Barakeel Fanseu Kamhoua, and James Cheng. Under- standing graph neural networks from graph signal denoising perspectives. CoRR, abs/2006.04386, 2020. URL WtLhttts::2 http/arxvvv..oo abs 2006. 04386
Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. Towards deep and unified understanding of deep neural models in NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Ma- chine Learning Research, pp. 2454 PMLR, 2019. URL RtLL:ttttp//pprceeiimg .mlr. press/ v97 guanl 9a html
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word represen- tations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), NAACL-HLT, pp. 4129- 4138. Association for Computational Linguistics, 2019. doi: 10.18653/v11119--4119 URL
https://doi.orgg
Yifan Hou and Mrinmaya Sachan. Bird's eye: Probing for linguistic graph structures with
simple information-theoreti approach. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), ACI/IUNLP, pp. 1844-1859, Association for Computational Linguisties. 2021.
doi: 10.18653/v//2021.aaa aacl-long... 145. URL https: //ts:///doo..ee wttp:/..gg/.....///11111111
acl cel-iong.om 145
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), ACL, pp. 3651-3657. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1131 1356. URL hetps:
//doi.org/10 8653/v1 /p1 1356
Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan L. Yuille. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. CoRR, abs/2102. 11343, 2021.
http///axxvvv.or//a///a/a/s 02. 11343

---

Under review as a conference paper at ICLR 2022
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph ral networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d'Alché-Bel Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Anmual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 7090-7099, 2019, URL
neurips.................. absp:eeaabbbbttttattttt.t...... Abstract. pttppppee//11////aast...
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR. OpenReview.net, 2017. URL https: hettps://openzeview net http://wwwwee/ooormrr.e. formmmir
SJ04ayYg
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiaoo Ramalho, Agnieszka Baroskka--arisska,, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, abs/1612.00776, 2016. URL
/
1612.00796
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 69(6):066138 2004.
Xiao Ling, Sameer Singh, and Daniel S. Weld. Design challenges for entity linking. Trans. Assoc. Comput. Linguistics, 328, 2015. URL httphhttt///tccctiiii
col coolumbia edu
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representationss In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), NAACL-HLT, pp. 1073-1094, Association for Computational Linguistics, 2019a. doi: 111-555//11119-1112 URL (RRLhhhts://ooi.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-BERT: enabling language representation with knowledge graph. In AAAI, pp. 2901-2908. AAAI Press, 2020. URL http:ttts/aaa...r////////
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeullmoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11192. 2019b. /ttpp//ppp//aaa.......a/////....
Yinquan Lu, Haonan Lu, Guirong Fu, and Qun Liu. KELM: knowledge enhanced pre-trained language representations with message passing on hierarchical relational graphs. CoRR, abs/2109.0422 04223,, 2021.
Ilsang Ohn and Yongdai Kim. Smooth function approximation by deep neural networks with general activation functions. Entropy, 21(7):627, 2019. doi: 10.3390/e21070662. httpthttp////do
org/10. 300.33339000000
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Con- tinual lifelong learning with neural networks: A review. Newral Networks, 113:54-71, 2019. doi: 10.1016/..eunnt22111 heunet.2019.01.01. URL
neunet www.eeeeeee........... 2019.
0.12
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: online learning of social represen- tations. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani (eds.), The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA August 24 27, 2014, pp. 701-710. ACM, 2014. doi: 00.1145/22333.222277333
Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). EMNLP-IJCNLP, pp. 43-54. Association for Computational Linguistics, 2019. doi: 10.116533v 1005. URL Itt //doi. /t/////oo//////ooo/oooooo
T0.18653/v1/D19-110
12

---

Under review as a conference paper at ICLR 2022
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), EMNLP-IJCNLP, pp. 2463-2473. Association for Computational Linguistics, 2019. doi: 10. (0.18653/11/111--125) 1250. URI. https://dos .org/
10. 10.18653/v1/DD9-122
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. information-theorett probing for linguistic structure. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), ACL. Association for Computational Linguistics, 2020. doi: 18.18653/v1
URL. hittps://dan org/10. 18653/v1/
2020. acl- main. 420,
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Balaji Krishnapuram. Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144 ACM, 2016. doi: 00.1445/9339672.2937777 URL http:::: //doi.org/ 10. 111.1111////11111 29337777
Aliaksei Sandryhaila and José M. F. Moura. Discrete signal processing on graphs: Frequency anal- ysis. IEEE Trans. Signal Process. 62(1333042333554 2014, doi: 10. 10.11099TSP22014222111.1
URL httt::://doo..rgge 100 1 1109/ /Tw.. 2014. 2321121
Michael Sejr Schlichtkrull, Nicola De Cao, and Ivan Titov. Interpreting graph neural networks for
NLP with differentiable edge masking. CoRR, abs/2010.00577, 2020. URL https://aaxxiv
org abc//ba/255 alaaabaa/00000 00577
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt:
Eliciting knowledge from language models with automatically generated prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 4222-4235. Association for Computational Linguistics, 2020. doi: 10. 10.18653/v1/0020 emnlp-main.346. URL httpss/////::ttooo wtg///..orr....... wtt.oo///.......1.111.1
emnlp ma meaain... 346
Kiran Koshy Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li. Attention-based graph neural network for semi-supervised learning. CoRR, abs/1803.03335, 2018. URL http://
arxiv. www.vvorg/abs///ooo /10...1033.0000 03735
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Neural Information Processing Systems, pp. 5998-6008, 2017. URL htt.e//rrooooooaoaoaooooaaaaaaaa htt.a//rrp......o.ppp//////////s 33555222355/6e1111bbb0eecccaaas3
Abstract html
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR. OpenReview.net, 2018. URL https:/ //
openreview
www.:neeennniiiiie Deenneview..ntt/oorummit.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei. Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of ACI/IJCNLP, volume ACL/IICNLP 2021 of Findings of ACL., pp. 1405-1418. Association for Computational Linguistics, 2021a. doi: 18.18653/v 1/202/1112211122/// 1e.2222...nnnnn--aaa findings--all.111 URL https:
7/doi. o0o/:..rgg10..1 18653/
findings acl. 121
Xiaozi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representa- tion. Trans. Assoc. Comput. Linguistics, 9:17 2021b. URL htttt:://tttasaaeeeeeeeee org/
Liang Yao, Chengsheng Mao, and Yuan Luo. KG-BERT: BERT for knowledge graph completion. CoRR, abs/1909.031 2019. URL
13

---

Under review as a conference paper at ICLR 2022
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: enhanced language representation with informative entities. In Anna Korhonen, David R. Traum, and Lluis Mårquez (eds.), ACL, pp. 1441-1451. Association for Computational Linguisties, 2019. doi: 10. 10.18653/v1pp1-- 139. URL https: /tpp:://ddio /dw.. oreel.ogg/1000 11131/.1111111111 http:///wp111119/9/
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. In Kristina Toutanova, Anna Rumshisky. Luke Zettlemoyer, Dilek Hakkani-Tün, Iz Belt- agy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL-HLI 2021, Online, June 6, 2021, 5017 7- 5033. Association for Computational Linguistics, 2021. doi: 10.18653/v 1/202 maacc-main..39
URL http:::///d...rr//10...8. 163. neacl ma in. 398
Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. Temporal reasoning on implicit events from distant supervision. In Kristina Toutanova, Anna Rumshisky, Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), NAACL-IILT, pp. 1361-1371. Association for Computa- tional Linguistics, 2021. doi: 10.18653v1122211 mnacc--min. maacl// 107. URL https: /tw.. org 10. 1al.63///1222211ccc1lmaaaaa
Luke Zettlemoyer, Dilek Hakkani-Türr
14

---

Under review as a conference paper at ICLR 2022
A NOTATIONS
Table 5: Notations and their descriptions
Notation
Description
G
The knowledge graph for KI
V E
The set of entities/nodes of KG The set of relations/edges of KG The entity/node indexed as in the KG The entity text attached on i;
%
f;
LMO)
The language model, where the input is entity text, and the output representation The set neighbors (entities/nodes) connected to P2
No.
(1))
The local graph structure terms of 12 The random variahle of the entity representation
x
The entity representations of
The random variable of the local graph structure The mutual information between two random variables
M
The adjacency matrix of KG
The number of entitics/nodes KG
The set real numbers
The identity matrix
D
The degree matrix of KG
L
The normalized Laplacian matrix
diag
The diagonalization operation
U
The matrix of eigenvectors
The diagonal matrix of eigenvalues The i-th eigenvalue
X
The set entity representations terms of
The dimension entity representations; The number channels
GFT()) RGFT()
The graph Fourier transformation
The inverse graph Fouricr transformation
ge H h
The graph filter parameterized by parameter θ The entity representations given by knowledge-enhance
LM
The random variable of the entity representation given by a knowledge LM
f()
The mapping that can transform x to h
The error the approximation
(()moid(x))
The Sigmoid function sigmoid(-) = I+e-
7 W x X
The number of layers of the neural network for apporximation The weight matrix
The input vector
The bias
The minimum eigenvalue the weight matrix W
MLP (
The bijective MLP function
GO
The graph convolution function
GCSO C
The GCS model parameterized by d1
The objective of the optimization
Z sup
The output GCS, i.c., set of output entity representations
The random variable the output GCS
The supremum value A class of functions Any class of functions
T
Ω To P PIV
The domain of function
A class of functions parameterized by 62 i.e. neural networks
The probability distribution
The empirical distribution with [V] samples
(N.
The neural network with activation function a()) and parameterized by 8
U
The norm of matrix U
A,
The normalized adjacency matrix
The ground -oound-truth entity representations/node features
X'
The variable matrix
Tr())
The of matrix
61,02
The error bound of entity representationsshodee features and adjacency matrix
7
The Lagrangian multiplier
p(()
The characteristic polynomial for weight matrix W The determinant of a matrix
del())
15

---

Under review as a conference paper at ICLR 2022
B PROOF OF THEOREM 2
Proof. As aforementioned, the graph Fourier transformation GFT((-) and its inverse transformation RGF in terms of the KG & can be written as
GFT(X) = X (GFT()FT(()) -
The second equation can be derived since U is the set of eigenvectors of the normalized Laplacian matrix in terms of G. which is orthogonal.
According to the universal approximation theorem (Cybenko) 1992). in general, we can use one- layer neural networks (arbitrary width) with the sigmoid activation function fit any functions. Ohn & Kim (2099) bound the approximation with both the width and depth, and supports more activation functions. Based on the conclusion ohn Kim (2019), we know that given mapping f'()), for any é' O, there exists neural network parameterized by 0' s.t.
(-' (N-((100) <
Note that there are some constraints about the input and the model architecture, i.e., width and depth. We leave out those details for simplicity. More details can be found in Ohn & Kim (2019).
Since h is obtained by integrating g into x, we can simplify the mapping in the graph space by researching on the transformation from GFT(x) to GFT(h Assume the mapping satisfies ((FFT(()) GFT(h). Then we have
( (GNT(( <
Consider that we have f(x, g) = h RGFT( we assign d' c |00
> 0, we have
U [FFF)))) - (FFFF)))))
Since we know that
U ((FF((x) (CFT() = h f(x.g),
have
|f(x g) RGFT(NN(GPT(X))) < 101- - NN. (GFT((())))))) <6
where NN(-) is parameterized by 6' with activation function N-o0--1"). And without loss of generality, we assume is composed of n layers.
C GRAPH CONVOLUTIONS FOR KI SIMULATION (PROOF)
Proof. The basic idea of this proof can be found in Figure We first simply prove that the graph Fourier transformation is bijective. Similarly, the nonlinear activation function can be proved bijec- tive. Then, we show that information gain and loss can only happen in the linear function in graph space. After that, we briefly illustrate that linear function in graph space is graph convolution oper- ation. Finally, we prove that graph attention works as edge denoising, and we can use to interpret the KI.
'In next proof, we illustrate that the linear transformation in the graph space is graph convolution, which integrates the graph information into entities. Thus, in the graph space, we do not need to regard input. More formally description he found in Chen 2019| Keriven & Peyré 2019).
16

---

Under review as a conference paper at ICLR 2022
C.I STEP
GFT((), RGFT(-), and sigmoid(-) are bijective. Given two entity representations 2;, æj and the matrix of eigenvectors of the KG as U, suppose that GFT( GFT(z,). Then, we have
x'
Since U are of eigenvectors and are by definition nonzero, we have
2; z-j-
If æ; æj. it easy to get GFT (*)( Thus, graph Fourier transformation is bijective.
As for the nonlinear activation function, since we consider neural networks composed of MLP lay- ers, the activation function is sigmoid(.) function. It is easy to find that its inverse function is fly) = 1n(1 - ). Similarly, we can prove that it is bijective as well.
C.2 STEP
Information gain and loss can only happen in the linear function in graph space. Based on the invariance of MI (Kraskov et al. 2004), we have
MI(x, g) = MI(GFT(x), g),
MI(x. MI(RGFT(xx)..)
(4)
MI g) = M((sigmoi((x),.
Since we know that
MT(h, g) MI(x, g) >
and the neural network can well approximate the mapping, we have MT(h, g) - MT(x,g) x MITRGFF -
MI(NN(GITT()) g) ()()((()) 0.
If write NN(-) with n MLP layers as n x a (Linear( (-)). have MI(GFT(x).
MI(n x [LLinear(GFT(s///
Recursively with equations 4
it is easy to get that MI only changes in the Linear(-) functions.
C.3
STEP
The linear function in the KG space (i.e., graph spectral domain) is the graph convolution operation. Even many existing works (Sandryhaila & Moura 2014 Bruna et al. [2014] Kipf & Welling 2017) have provided clear descriptions, simply re-illustrate under the multi-channel setting. Consider the graph filter in Bruna et al. (2014) as exmaple.
For linear function f(x) W x its weight matrix W e FxC is parameterized by θ Fxc, If the parameters are not shared for all nodes, the input X R givx can be rescaled in glv xxx1 and the weight matrix is W E XXXX parameterized by θ E FxCx/VI The output of this linear function is mapped in R |
Consider the signal in graph convolution, i.e., all x in X E BIIX Since parameters are not shared (Bruna et al. 2014), for one graph filter, the parameters in ge in CR V|x|V| that is pa- rameterized by θ E c [V] with simple diagonalization. we have F different graph filters for the convolution, de is in IR FxCx CCXVVXXMV that is parameterized by E FxCx|VI Here, the graph Fourier transformation of X is GFT(X) R Vixc which can be rescaled in R² xXXxXXXXXX with simple diagonalization. The output filter is diagonalized, we rescale the output in R
IR / Note that since the parameters in the graph
we regard the weight matrix W as the parameters in the graph filter ge, the input matrix as the signal, obviously, the linear function in the graph space is the graph convolution operation. n
17

---

Proof. Graph attention works as edge denoising 9 Graph attention works as a better graph convo- lution filter, since it can adaptively learn the optimal convolution weights (i.e., attention coefficients). Consider graph signal denoising problem that we aim to extract the ground-true node features X and edge weights A, from graph g = (V. E, A.) with noise in both node features X and edge weights A, Here, A, is the normalized adjacency matrix A, D 1/2 AD 1/2. To this end, we formulate the optimization problem under the assumption that the ground-truth node features X are Under review as a conference paper at ICLR 2022
D GRAPH ATTENTION FOR KI INTERPRETATION (PROOF)
smooth w.r.t the ground truth adjacency matrix A,, and the noise in the graph can be upper-boundee
X A
argmin'Tr (xi, X
A.A.
(5)
s.t. X - x v 61,
|| A A. 11 62
where I A. EL, E R, are the level of noise in node features and edge weights, respectively. Tr(-) indicates the trace of a matrix. By Lagrange multipliers methods, we can obtain the solution as following:
X I (1- 1+7 (1-
( 1+7 A
(6)
X X-
A = A. + VC2
where y 0 the Lagrangian multiplier. Note that the attention coefficients of GAT (Velickovic itt al. 20188) and AGNN (Thekumparanpil et al. 2018) are obtained by (without less of generality,
we show the results in the first-layer) equation and equation respectively:
ai = softmax
(laakyReLU (a T [W W wX,l) JENGU(O)
(8)
(1 H HJ H₂l
0i,) softmax
H (1
Dexcurs)
where H = Rel.U(XW), a, W in equation and ,3, W in equation 9 are learnable parame- ters. The attention coefficents of GAT and AGNN are then used the weights of aggregating the neighbohood information of nodes. As we can see that equation 7 equation g and equation 9 are in a form of measuring the similarity between paired node features. Similar the denoised edge weights obtained in equation / the attention coefficents (i.e. the aggregation weights) between a node and its neighborhoods are proportional to the similarity of their node embeddings. Therefore, the attention coefficients of GAT and AGNN can be regarded as the results of denoised weights on the existing edges a graph, i.e., the graph attentions are implicitly denoising the edge weights.
In general case, graph attention functions denoising edge weights. The input is noisy representa- tions and the output is the groundtruth. Attention coefficients show how much distortion corrected during the convolution operation. For example, if the input representations are also groundtruth, there is no need to fetch information from neighbors to get output. And edge weights will be re- duced to 0, i.e., attention coefficients edges are calculated as 0. If the input representations very noisy, i.e., much noise are removed, attention coefficients on edges should be large to restore the groundtruth signal. Therefore, in the KI scenario, we can use attention coefficients in graph attention in graph convolution layer to interpret the KI process such as how much triple information integrated. As for the CR and CF, equally, we can use the attention coefficients on the self-loop edges for interpretation, such as how much original information is remembered/forgotten.
Thhe detailed proof can he found in Fu al. 2020
18

---

Under review as a conference paper at ICLR 2022
E MLP CAN BE BIJECTIVE (PROOF)
Theorem 5. Give an MLP layer denoted as ML)(x) = sigmoid W x + b). If W is a square matrix, there exist a constant Ny > O that for any 0 < E < Xb. the function below bijective:
ML.P, n (2x) sigmoid -eTa b).
(10)
Proof. We first prove that two bijective function compositions still bijective. Then, prove that adding a small noise on MLP weight matrix can make it bijective.
Give two function fi(-) and f2(-). Suppose they are injective and suppose f1(f((z)) (f)(
Since we know that (-) is injective, we have f2(x) ) = f2(y). Similarly, since fz(.) is injective, we have y. Thus (-2()) is injective. Suppose fi (:) and f2(+) surjective and C. Since
we know that (-) is surjective, there exists a set of E B with (y) = 2. Similarly, since fz(.) surjective, there exists set of æ E A with f2(2) y. Then, we have 2 fi(f(())) and so
onto / (f2())). Thus, (f2(-) is surjective. Therefore, if (-) and (2(-) are bijective, (i(((()) is also bijective.
To prove that the special MLP is bijective, consider an MLP function as MLP(x)) = o (Wa x + b),
where W E RCCC is the weight matrix and b E RC is the bias. Let
C
p()) = II (X ()
==1
be the characteristic polynomial for weight matrix W. Here X; are eigenvalues of matrix W. With- out loss of generality, let [X6 = min, [X]. Then, we know that for any constant < 126.. we have
det W - cT) = p(c) #1)) #0.
Thus, the perturbation e is small enough, the perturbed matrix W WW -ef sider the fact that the nonlinear activation function (.)) is sigmoid(() function, which is bijective. Therefore, the special MLP function MLP,..) is bijective. And there is no information loss.
W- is nonsingular. Con-
F IMPLEMENTATION DETAILS OF THE SIMULATION EXPERIMENT
Walk, We implement Deep Walk to get KG embeddings. The hyperparameters are set as the same as its default v the number of walks is set as 10, and the walk length is set as 40. The dimension of embeddings is set as 128. Note that since knowledge triples only contain neighbors within one-hop, we select the window size as
Noisy KG embeddings. We add Gaussian noisy on the KG embeddings with different ratios. For example, if we add 10% ratio of noise, it can be written as
10% noisy KGE 0.9 KGE + 0.1 noise.
Similarity. The Cosine similarity is implemented
1 - Cosine distance(,))
and the Euclidean similarity is implemented as
1 Euclidean distance(,).
Linear probe. We design a linear classifier to link prediction for the KG. Note that we cannot implement the linear classifier on the large and sparse KG to do link prediction directly. To get meaningful results, we do negative sampling with sample number as 5. Then, we select to report the AUC (Area Under the Curve) score since other metrics cannot be differentiable under this condition.
19

---

Under review as a conference paper at ICLR 2022
We randomly split the edge set of the KG into two sets for training and test. To reduce variance, we repeat the experiment for 10 times.
GCS. We train our GCS model with imput noisy KGE and output KGE. Note that here noisy KGE and KGE are in the same space, and the task is simpler compared to practical KI interpretation. Thus, set the epoch number as 10, and the reconstruction loss (i.e., mean absolute error) for efficiency. Other hyperparameters are set as the same as introduced in Appendix H
G IMPLEMENTATION DETAILS OF LMS
KI. To ensure that the experiment settings fair, set hyperparameters as the default values. For K-Adapter, the code and hyperparameters for KI that we use are from the official projects m1 published by the authors (Wang et al. 202 The only two differences are that: we use PyTorch floar 32 instead of float 16 since BERT and RoBERTa that we use are float32, and we use 4 NVIDIA Tesla V100 GPUs for KI training. For ERNIE, things are the same. All hyperparameters for KI are set as their default values [ the integration with 4 NVIDIA Tesla V100 GPUs. Note that the dataset that ERNIE used for KI is Wikipedia, since the code is to fetch latest version of it, the data that we use could be slightly different. Therefore, for both ERNIE and K-Adapter, to ensure the fairness, we reproduce their KI, and report the results of reproduced models instead of results provided in their papers.
Similarly, float 16 of PyTorch is changed to float 32, and we do
Finetuning. As for the downstream tasks, all the hyperparameters are consistent with the official project:
either they are given in the project in the README. In the same way, float 32 and NVIDIA Tesla V100 GPUs are chosen to make sure that the comparison is fair. Note that for K- Adapter and ERNIE, the best performance for different datasets is achieved in different settings. For example, the best performance for K-Adapter on the OpenEntity dataset is achieved with single GPU, but the TACRED dataset is achieved with four GPUs. Since we focus the relative performance instead of the best one, we run finetuning on 4 NVIDIA Tesla V100 GPUs for all downstream tasks and all LMs (as well BERT and RoBERTa).
Table 6: Statisties T-RExrre and Wikidata
Statistics
of entities
#o triples
of aligned sentences of entities (optimization)
# of triples (optimization)
Datasets
F-REx-Ie
781275 3,275,534
1,282,504
5,365,478
Wikidata
12,849,311
1,344,393
3,240,272
The datasets that K-Adapter and ERNIE use are T-REx-re and Wikidata, some statisties of them are given in Table
H IMPLEMENTATION DETAILS OF GCS
In this section, we introduce details of implementing GCS. In practice, GCS is composed of layers: bijective MLP layer, graph convolutional layer, and another bijective MLP layer. As for bijective MLP layers, since weight matrices them are square matrices, the dimension would remain un- changed: 1024 for K-Adapter and 768 for ERNIE. The nonlinear activation functions set as ELU(-) function, which is also bijective. The learning rate is set as le 3 and the dropout rate of the first MLP layers is 0.2.
Regarding the graph attention, to make sure interpretation results are stable, we apply multi-head attention mechanism, where the number of attention head is set as 8. Entity representations first embedded into a space with the dimension as 64. Then, the embedded representations are used to calculate the attention coefficients. Note that since the purpose is to interpret and analyze the KI process, we do not split datasets for KI. Considering that GCS model is very simple for large KGs, overfitting is unlikely happen. Thus, we optimize GCS for the whole datasets. Specifically, for K- K-Adapter, the whole KG is used for optimization, and results are used for interpretation. And
20

---

Under review as a conference paper at ICLR 2022
for ERNIE, since the KG is very large, sample small subgraph with 1, 344, 393 entities and 3, 240, 272 triples for optimization (see Table 6), and then implement the optimized GCS on the whole KG for interpretation.
The objective function of training GCS can be reconstruction loss minimization or MI maximization. In this paper, except the Simulation Experiment, we all select MI maximization the objective. For reconstruction loss minimization, we use the mean absolute error (MAE) between our GCS outputs and entity representations of the knowledge-enhaneed LMs. Regarding the MI maximization, we define the objective function by maximizing the MI as
C = - MICS, (x):
(11)
We optimize MI equation 11 by maximizing the compression lemma lower bound (Banerjec) 2006) in Belghazi et al. (2018). The inputs of GCS are X, and let the output be denoted by Z. We can regard Z and H as empirical samples of random variables and h. Thus, we have:
MI(2; h) 2 sup EP [T] log( (Ep, oF le TEF
(12)
Here, F can be any class of functions T f2 R satisfying certain integrability constraints (Belg- hazi et al. 2018). Pzh represents the joint distribution of and h, and P, &P,, represents the product of their marginal distributions. In practice, we let = {Ta } be the set of functions parameterized by neural network, and optimize it by stochastic gradient descent. Then, the objective function can be rephrased as
max 6, .2
Epiii [To.]= log (e where z = GCS,, (x).
(13)
13 plll represents the empirical distribution of z, i.e., Z. If the KG is very large, we can optimize the network by sampling small subgraph of the KG. In practice, we simply add two MLPs layers to GCS for MI maximization. The added two MLP layers may not be bijective, where the dimension would first reduced to 64, then to for MI maximization. The nonlinear activation functions are all set as ELU((- function, which is also bijective.
In equation
For interpretation, we use the attention coefficients on edges and self-loops to analyze the KI in terms of triples and entities. Different from Schlichtkrull et al. (2020) that specially designs discrete function to mask edges that are not important, we simply introduce temperature hyperparameter l and set it as 0.1 to make the attention coefficient distribution hard [3] Thus, knowledge can be well clustered into learned and unlearned.
I ADDITIONAL STATISTICS FOR INTEGRATION EXPERIMENT
Table 7: Drop statistics for the Integration Experiment.
Percmitage of integroted eniities
Pereeniage of integrated triples
of aligned sentenceskentity emboklings (integmatd knowlelgo)
T-REx- Wikidata
28.86%
561,687 out of 3,565,478 2,240,260 oui of 3.275,534
61.72%
J ADDITIONAL RESULTS FOR GCS VERIFICATION
We gradually drop out sentences aligned to knowledge triples whose attention coefficients are small for KI. For comparison, we randomly drop same number of sentences named the random strategy. We can that dropping out KI datasets based the interpretation results of GCS significantly out- performs the random strategy. It further supports that GCS can effectively interpret the KI situation for the K-Adapter.
21

---

Under review as a conference paper at ICLR 2022
0.775
0.635
0.630
P.Micro
0.770
P-Macro
0.006 625
0.620
GCS
GCS Random
Random
0.615
of Sentences for Integration
Sentences for Integration
le6
(a) Precision-Micrt score
(b) Drecision-Mact score
0.76
0.630
0.75
R-Micro
0.620
0.74
# 0.6115
GCS
0.610
Random
Random
0.605
5
of Sentences for Integration
1e6
Sentemces Integration
1e6
(c) Recall-Micro score
(d) Recall-Macro score
0.758
FlMMtrro
0.756
EL-Macro
0.600
0.754
0.600
0.752
GCS
0.615
GCS
0.750
Random
Random
Sentences for Integration
166
Sentences Integration
1e6
(c) F1-Micro score
(f) F1 F1-Macro score
Figure Detailed results K-Adapter on OpenEntity dataset, where we use different number sentences to integrate knowledge into K-Adapter. For GCS, we integrate knowledge for K-Adapter using sentences aligned with triples whose attention coefficients are larger than {0.0, 0.01, 0.1, 0.9, 1.0), where the corre- sponding number of sentences are (5565478, 1091152, 561687, 127728, 0}. Same number of sentences chosen randomly for comparison the random strategy.
0.6
Rattoo triples
0.6
0.4
0.4
0.0
0.2
0.8
0.0-0.0
0.0
0.4
Rabio of Entitles
Ratio of Enthies
0.3
Ködagten self-loops
e9NIS: self-loops
0:10
0.1
Attention Confficient
Attention Coefficient
Figure 5: The attention coefficient distributions of edges and self-loops for K-Adapter and ERNIE. The his- togram shows the empirical distributions (i.e., frreueecc), and the blue curves the Gaussian kernel density estimate. The black dashed vertical lines indicate the average values.
22

---

Under review as a conference paper at ICLR 2022
K KNOWLEDGE INTEGRATION OVERVIEW
pressetss the empirical distributions of attention coefficients for K-Adapter and ERNIE. The above two subfigures show distributions on edges, interpreting the KI from triples. While the bottom two subfigures show distributions of self-loops, illustrating the KI from entities. We can find that most knowledge triples are not integrated well for both K-Adapter and ERNIE (i.e., aij < 0.1), while K-Adapter performs slightly better. When it comes to entity-wise integration. in general, entity knowledge is also not integrated well. We find that both CR (i.e., a. > 0.9) and CF (i.c., ai,s < 0.1) happens for many entities, especially for ERNIE. K-Adapter outperforms ERNIE since some entities are integrated well (i.e., 0.4 < 0z < 0.6).
L ADDITIONAL STATISTICS FOR TABLE 4
Table 8: The number of aligned sentences for relations.
Statisties
# of triples
Relation lahel
Place of birth
134,976
Part of
134,999
Date of deuth
135,190
Date of birth
135,169
Located in the administrativv territorial entity
135,055
Country
135,147
Total
5,565,478
M ADDITIONAL EXPERIMENT: CASE STUDY
Table 9: The interpretation of KI, CR, and CF for K-Adapter in terms of relations and entities. We list 3 typical relations: CR happens to most connected entities; CF happens to most connected entities; and KI, CR, CF happens equally to connected entities. And we list 5 correspondingly aligned sentences. The Google Ngram of entities are reported to show the popularity of entities.
Ratio of triples connected connected CF entities
Ratio of triples
Ratio of triples connected WT. entities
oo aligned sentences
Relation label
entities
KI Entities (Google Ngrams)
Examples
Copyright license
85.09%
14.61%
0.16%
4,400
This article incorporates public domain material from the United States Geological Survey document "Wasson Rock" (content from Gengraphic Names Information System).
CR. (1.95, 2. 174
Office held by head of govermment
0.46%
99,34%
0.00%
14,315
CF. (0.91, x.33)
Seven people served as guverner of Colorado Territory over eight terms, appointed by the President of the United States.
Opposite of
26.30%
41.87%
28.62%
11,577
CR. (1.18, 11
is well covered with deciduous and evergreen forests.
The prison's 's north wing is filled with left-wing rehels while the south wing is filled with
CF, (3.67, 5.60
right wing government supporters and paramilitaries
WL. (6.83, 2. x14
Wildlife of Iran includes its flora and fauna and their natural babitats.
Knowledge of different entities (sorted by their popularity) has different CR and CF ratios (case study). Table 9 reports the KI for K-Adapter in terms of entities. We select three relations ("Copyright license". "Office held by head of government". "Opposite of CR happens to most connected entities for relation "Copyright license". CF happens to most connected entities for re- lation "Office held by head of government". Some connected entities are learned well for relation "Opposite of" We list 5 aligned sentences for the three relations, and report the Google Ngrams (year 2019 of entities show their popularity. We find that CF often happens to entities with large Google Ngrams such as "left" and "President". And CR often happens to entities with small Google Ngrams. Well-learned entities are also not very popular.
Note that the principle of hyperparameter selection is to maximize the MI, i.c., objective function. Users may select appropriate hyperparameters depending on the situation. wtw.:/owww..ooglc.c..ggaooaa
23

---

Under review as a conference paper at ICLR 2022
N
ADDITIONAL EXPERIMENT: CORRELATION
1000
of Aligned Sentencus
800
600
0.2
0.6
0.8
1.0
Attention Coefficient
Figure 6: The correlation between the attention coefficient of the knowledge triple and aligned sentence number. There is no correlation between them.
Can we improve the KI quality by simply increasing the size of our dataset? Above results analyze KI in LMs in different ways. However, there is key question: can we simply improve the quality of KI by increasing the amount of our aligned training corpora? We try answer this question Figure 6 which plots correlation between the attention coefficients and the number of aligned sentences for knowledge triples in the dataset. We find that the Pearson correlation between the two is -0.0055. There is no positive correlation between the two variables. This implies there no apparent positive relationship between the KI quality and the size of the KI dataset. It suggests that simply increasing the size of the aligned dataset alone may not improve KI, but, we might need more fundamental advances push the fate-of-the-ee in KI forward.
24


Reviews:
[{"review_id": "qtfamYsGMV9", "reviewer": "Reviewer_pRFd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes to measure the amount of integrated knowledge with mutual information and shows the strong connection between the KI process and the Graph Fourier Transformation and then model it as graph convolution. The analysis process is interesting and coped with interesting insights. \n\nA few comments on the paper:\n1. Could you please clearly define 1-1, N-1, and M-N relations? \n2. The authors uses attention weight on edges as measurement of relevance. It's okay to make the analysis using the attention weight in this paper, but it is arguable that attentions are not always explanations, i.e. \"Attention is not Explanation\" (Jain et al. 2019). Have you thought of performing you analysis using other measurement? Or maybe prove this is the exact measurement for your purpose?\n3. When discussing \"Can we improve the KI quality by simply increasing the size of our dataset?\", how do you measure the \"the number\nof aligned sentences for knowledge triples in the dataset\"? I might understand it wrong, but is the size of KB and text corpus in pretraining an important factor here. It maybe helpful to report the size of KB used in ERNIE and K-Adapter and modify it and see if it will change your conclusions.\n\nAnother work that may be interesting to consider:\n[1] \"Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge\" Pat Verga, Haitian Sun, Livio Baldini Soares, William W. Cohen", "summary_of_the_paper": "The paper claims that the Knowledge Integration (KI) process can be simulated with Graph convolutions and designed a Graph Convolution Simulator (GCS) to analyze the performance of two popular KI models, K-Adapter and ERNIE. They observe that K-Adapter is good at capturing simple (1-1) relations while ERNIE is good at capturing complex relations. The experiments are well conducted and the conclusions are insightful.", "summary_of_the_review": "This is an interesting analysis paper discussing the Knowledge integration process by proposing to measure their Mutual information (MI) and in turn model it as graph convolution. The claims are solid and thorough analysis are made. Good paper. Very enjoyable to read.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pMGtYPd9nDm", "reviewer": "Reviewer_eLPV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and presents a very useful method of analyzing how structured knowledge of entities may or may not be integrated into a language model through additional pretraining. The detailed analysis characterizing what kinds of knowledge are well-integrated, catastrophically remembered, or catastrophically forgotten based on relation complexity is helpful at demonstrating what kinds of probing the GCS method is capable of. Tying GCS to downstream task performance also helps to showcase the validity of the method, although this could be further strengthened by expanding on these results (see below).\n\nOne comment I had was regarding the presentation of the results in Table 1, which demonstrate that removing pretraining sentences based on knowledge identified as unlearned by GCS shows little drop in downstream task performance. While the results compare each of K-Adapter and ERNIE with their versions that drop certain pretraining examples, it would also be helpful to compare to BERT/RoBERTa results in the same table, as it is not immediately clear which differences between the two versions of each model are meaningful versus negligible without a baseline performance to compare to (though the random dropping results in the appendix somewhat help to clarify this). It would also be worth exploring downstream task performance in finer-grained detail to further support the method - for instance, how does an entity being catastrophically remembered/forgotten affect entity typing performance on examples that involve that entity? Correlating attention values found by GCS for specific entities to task performance on those entities would provide futher empirical validation of the method.", "summary_of_the_paper": "This paper proposes Graph Convolution Simulator (GCS), a probing method for investigating what knowledge is being integrated into language models that are augmented with knowledge graph information through knowledge-enhanced pretraining. The method applies recent work for estimating the mutual information between continuous random variables using neural networks to capture the mutual information between entity representations before and after a language model has undergone knowledge integration. To make the method simpler and more interpretable, the mutual information estimation is relaxed to learning a combination of graph convolution and multilayer perceptron layers that map from the original entity representations to their knowledge-integrated versions, optimized by maximizing a compression lemma lower bound to mutual information from previous work. The graph convolution is parameterized as a graph attention layer, such that the attention weights can be used to quantify how much information is catastrophically forgotten or remembered for a single entity as well as transferred between entities that participate in a triple in the knowledge graph. This method is used to analyze two recent knowledge-enhanced language models, K-Adapter and ERNIE. The analysis shows that little factual knowledge is explicitly captured by these models, each model is better at capturing different kinds of relational knowledge (e.g., simple relational knowledge for K-Adapter, complex relational knowledge for ERNIE), and that K-Adapter struggles to remember temporal information and information about popular entities, among other findings. Finally, additional experiments show that increasing the size of the training corpora does not correlate with improved knowledge integration, motivating more careful development of future knowledge-enhanced pretraining methods.", "summary_of_the_review": "While some changes to the presentation of results would further confirm the interpretability of the attention weights learned by GCS, the current set of results still seems to present a valid and useful method of probing for integrated knowledge in pretrained language models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "c1aztFWTFF", "reviewer": "Reviewer_4QcM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- The paper tries to tackle a very interesting question: how much knowledge is actually integrated by knowledge-enhancing PLMs, which knowledge and how can we best measure this.\n- As far as I’m aware, the idea to distill a KI-simulating GCN using MI is original and it could be a neat idea if some concerns are addressed better.\n- The paper is well-structured.\n\n**Weaknesses**:\n1. The paper is difficult to understand and reproduce. It is still not 100% clear to me what exactly is ultimately done in the approach, but also in the experiments, where it is not clear what exactly is being done in Table 2 (appendix F specifies the GCS architecture and interpretation principle, which have already been explained in the paper, rather than exactly explaining the experimental setup of Table 2).\n2. Relation to related work: The authors cite Hou adn Sachan's work a few times but do not further indicate how it is related. In Hou and Sachan's paper, the goal is to measure how much of an *entire* linguistic structure's information (e.g. dependency parse), as a whole, is captured in PLM's representations, and they measure this using MI between an encoding of the linguistic graph and the PLM's representations. The goal in the submitted paper is similar (except applied to KG's, and measuring the gain in knowledge after KI), the GCS objective is similar to the Bird's Eye objective, except that it also trains the GCS model. Although the submitted paper extends to quantifying knowledge gain, there are parallels with Hou and Sachan, and the relation with that work should have been discussed better.\n3. Lack of baselines: it is not clear to me (and not discussed in the paper) why some baselines have not been explored to measure the effects of KI, and relate the proposed method to. The change of factual knowledge could be measured by measuring local KG subgraph MI with PLM representations with and without KI using Hou and Sachan's method. More fine-grained information seems to have been addressed in Hou and Sachan too, in Section 2.4. Another baseline could be to see how easily every relevant KG triple can be reconstructed from the entity representations with (*h*) and without (*x*) KI. Without the authors discussing these possibilities and comparing to their approach, it is not clear why the presented approach should be chosen over more obvious possibilities.\n4. I am not sure how much sense it makes to train a different network (which might have different properties) to maximize the MI with the outputs of a knowledge-enhanced PLM (thus in a way distilling it?), and using this derived model to analyze the properties of the original model. It is not clear to me why any of the conclusions made based on the \"simulating\" model should really apply to the original model. One issue for example is that the used GAT does not seem to incorporate relation types, whereas the base models do have access to that information. How closely can GCS then approximate the knowledge actually contained in these models? I think comparing the attention weights to the results of a baseline probe with and without KI would be necessary to convince a reader that the obtained attention weights can really be interpreted like that. Without this however, I find the claim of understanding KI, CR and CF insufficiently supported.\n5. Unsatisfactory verification: GCS is only verified on two entity typing datasets. The verification is done by using GCS to identify which knowledge is \"learned\", removing the knowledge that is **not** flagged as learned and then retraining the base models (ERNIE and K-Adapter). The authors note that the performance on the tasks stays more or less the same, which is taken for proof that GCS really identifies the knowledge that is actually learned. The authors also compare this to randomly dropping knowledge. Even though the comparison to random is crucial to put the numbers in context and verify the claim, the authors put these numbers in appendix. While there indeed appears to be a higher performance penalty when dropping randomly at a certain drop rate (although the difference is only 0.006 (0.6%) micro F1 and 0.03 macro F1 points at its largest), only results for K-Adapter on OpenEntity are given (in appendix). This is only one of the four settings reported in Table 1. More complete numbers are necessary to properly judge the claim.\nIn addition, I believe that a better verification method needs to be developed where the absence of knowledge is more penalizing (in Figure 6, training without any KI doesn't appear to be much of a problem).\nAnother issue is that the baseline performance of K-Adapter in Table 1 is lower than that reported in the K-Adapter original paper, and even worse than the RoBERTa baseline used in that paper.\n\n**Questions** and **Remarks**:\n\n6. How are you taking into account the different relation types when doing GAT in GCS?\n\n7. Remark: I think it should be clarified what is simple and what is complex knowledge in the beginning.\n\n8. Remark: It should be clarified what is meant by \"entity text corresponding to the node v_i\". I assume this is the entity label. However, as it is written, it could also be an entity description, or any kind of text, that could also describe entity properties.\n\n9. Probably AutoPrompt (Shin et al. 2020) and OptiPrompt (Zhong et al. 2021) should be cited as well.\n\n10. The quality of the writing should be improved. Examples: \n- “which aim of” → “which aim to”\n- “The transformation is a simulation of the KI process, i.e., MI change, and it promise the accuracy.” ← this sentence does not make sense to me\n- “Below two” → “bottom two”\n- Some specs in Fig. 6 description are basically unreadable because of commas everywhere.\n", "summary_of_the_paper": "In this paper, the authors investigate a novel approach to improve our understanding of the knowledge integration (KI) process of knowledge-enhanced pretrained language models (PLM). While several works (Petroni et al. 2019 etc) have investigated knowledge contained in PLM, in this work, the focus is to understand how much and what kinds of knowledge are actually captured by methods that take PLM's and perform addition pretraining to incorporate factual knowledge from knowledge graphs (KG) into the language models.\nThe authors interpret the gain in mutual information (MI) between the PLM's text representations and knowledge graphs as a measurement for KI. The authors note that MI is difficult to estimate and note that instead the KI process can be simulated using a transformation that relies on forward and reverse graph Fourier transforms and a NN in that domain. However, since this can be expensive, the authors propose to instead use graph convolutions.\nThus, the proposed method attempts to \"simulate\" the knowledge integration process as graph convolution (GC) over the knowledge graph and is called Graph Convolution Simulation (GCS). The authors choose a model based on attention-based GC (graph attention network, or GAT) in order to improve interpretability. The actual model finally used by the authors consists of two MLP layers and one GAT layer in between. The GCS model is trained by maximizing the mutual information (MI) of the entity representations of GCS and the entity representaitons obtained from the PLM.\nThe authors then use the attention weights of the GAT to determine what information from the KG is used by the simulation in order to maximize its MI with the actual representations built by the knowledge-enhanced PLM. According to the authors, this allows to investigate catastrophic forgetting and remembering and which relations and types of relations are learned well during KI.\nIn experiments, GCS is first verified on entity typing. Subsequently, GCS is used to investigate what information is learned by ERNIE and K-Adapter.", "summary_of_the_review": "=== **after rebuttal** ===\n\nWhile the theoretical discussion appears sound, I do not think that contribution in itself warrants acceptance at a venue like ICLR. I have an issue with the expressivity of the proposed implementation of GCS since it does not encode relation type information, which is essential in the definition of a triple.\n\nHowever, my main concerns are with the experimental verification part, where the claimed ability of the proposed method to identify which knowledge is learned is not acceptably verified and the experiments may suffer from other issues.\n\nWhile I appreciate the efforts of the authors, and think the paper now looks better, I think it does not deserve publication at ICLR in its current form and I will not be raising my score. My concerns with some critical issues of this work were not resolved by the author's reply and the updated paper.\n\n=== **before rebuttal** ===\n\n\nWhile the paper focuses on an interesting question, due to the above issues, in my current understanding, I don't think it deserves publication in its current form at a venue like ICLR. While it's different from previous work that I'm aware of and might be a neat idea, I found the claim of using attention weights of a GAT-based network simulating the KI process to inspect what is being learned not sufficiently justified. In my opinion, the proposed simulating probe has not been properly verified, and it is not clear why it has not been compared to more basic methods and straightforward adaptation of previous work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PtpuG3lOLqH", "reviewer": "Reviewer_Lpu3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: see \"Summary Of The Paper\".\nWeaknesses: \n1. A potential issue about the method: the design of the graph attention network. While I understand most parts of the proof, I find \"Graph attention works as edge denoising\" to be difficult to understand, especially how \"graph attentions are implicitly denosing the edge weights\" lead to \"we can use attention coefficients in graph attention in graph convolution layer to interpret the KI process\". There seems to be a gap in the logic for me. I am confused about this part also because we have multiple (n) graph attention layers, right? So which layer we should use for interpretation? What is the theorical property for using any of the layers? Is it possible that multiple different edge attentions can lead to the same result?\n2. The evaluation is weak for me. Although there are interesting findings, but it is difficult for me to judge whether we can trust the results, due to lack of evaluation of the method itself, e.g., comprehensive comparison with baselines either in a qualitative or quantitative way. Although there are theoretical analysis, experiments are still needed since we still have many approximations here, right? \n3. It will be really good if there are some ways to aggregating the edge-wise attentions to compute the (decomposed) MI directly (instead of only showing edge-wise results). For example, can we consider the new pipeline with GCNs as a simplied model for which MI could be easily computed (e.g., we only need to compute the MI for the attention layers)?\n4. The authors said that \"Thus, understanding the KI process requires building probes separately for each of these KI methods.\" It is not obvious to me why we cannot using model-agnostic explanation methods for black-box models, e.g., [1][2]? This will probe all changes in a unified way, e.g., by perturbing the inputs and see how the perturbation changes the outputs. Maybe the authors could take a look at [2], which also uses MI for general and consistent model interpretation.\n5. The authors said that In this work, we consider factual knowledge in the form of triples (vi; r; vj). Thus, it suffices to set G(vi) = Nvi .\" I think there are multiple types of edges, so if we set G(vi) to N_vi, we will ignore the relation type, right?\n6. The paper presentation may be improved a little to better illustrate the meaning behind the equations. For example, this sentence is difficult for me to understand: \"The transformation is a simulation of the KI process, i.e., MI change, and it promise the accuracy.\" What does Eq. (1) actually mean? Does it verify the correctness for transforming with GFT? How does this align with MI change? Also for the sentence \"we analyze the transformation and show that MI change only happens in linear functions of the neural network (Figure 3).\" I did not understand it before seeing the proof in Appendix C. Please consider refer to Appendix C here (instead of later before Fig. 3).\n\n[1] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. https://doi.org/10.1145/1235\n[2] Guan, C., Wang, X., Zhang, Q., Chen, R., He, D., & Xie, X. (2019). Towards a Deep and Unified Understanding of Deep Neural Models in {NLP}. Proceedings of the 36th International Conference on Machine Learning, 97, 2454–2463. http://proceedings.mlr.press/v97/guan19a.html", "summary_of_the_paper": "This paper presents a probe model to analyze how well existing methods integrate knowledge into pretrianed language models. To achieve this goal, the authors first propose a mutual-information-based framework, which is reasonable and provides a unified solution for elegantly understanding whether catastrophic forgetting or catastrophic remembering happen. The authors then propose a method to quickly evaluate the mutual information by using a graph attention network. I really appreciate the theorical foundation that the authors take effort to provide, which is interesting and promising. It is also intriguing how different insights about SOTA models can be collected.", "summary_of_the_review": "1. Strengths: interesing and important problem, the MI-based framework is reasonable and elegant, most parts of the theoretical proofs are reasonable.\n2. Weaknesses: some problems related to method design and/or paper presentation, weak evaluation, insufficient discussion on (or comparison with) some related works \n\n==After rebuttal==\n\nThank the authors to carefully consider my suggestions. I appreciate the authors' great efforts in improving the paper, e.g., improving method description, adding details missing, and providing three additional numerical experiments. Most of my concerns have been addressed now. However, I can still see that some experimental results are not very good (e.g., Sec. 4.2.3). Considering all these, I would like to raise my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DzSLclpJzF", "reviewer": "Reviewer_WDUZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A clear and intuitive method to understand the knowledge integration process of LMs by estimating the MI with entity representations via applying GFT and attention.\n- The findings with GCS about the factors such as the popularity of entities and the size of KI dataset are beneficial for future research in KI.\n\nWeakness:\n- The study is quite limited to two particular KI methods and it's not convincing that the GCS method can be applied to analyze general KI methods and lead to correct conclusions. \n- The method is limited to binary relational facts (subject, predicate, object) while real-world scenarios of fact updating are mostly about complex facts that cannot be formulated this way. Say, the previous LM holds \"The US president is Obama in 2014\", and the new knowledge is \"The US president is Biden in 2021\". \n- There are missing important details. Particularly, how do you obtain the entity representations at first? And are there serious justification why entity representation should be the starting point of your formulation? It's a bit vague and unjustified. My suggestion on this is to take the representation of facts instead of the entities as the starting point in your MI formulation. \n- The key findings, although useful, are quite trivial and the analysis on the K-Adapter and ERNIE may not be particularly interesting to the audience. My suggestion on this is to either theoretically formulate the KI methods in a unified framework or study more KI methods in this paper.\n\n", "summary_of_the_paper": "This paper presents a method to analyze how knowledge integration methods perform. The proposed approach, graph convolution simulator (GCS) is to understand the KI process by simulating the change of mutual information with graph Fourier transformation as well as interpretable graph attention. \n\nThe authors focus on using LM's entity representation as the lens to understand the change of knowledge in LMs during the KI process, and formulate the catastrophic remembering and forgetting via the MI about entity representations. The key idea of the proposed GCS method is to use GFT (graph Fourier transformation) to map entity rep. to the knowledge graph space. And then the authors proposed to use graph convolution to simplify the computation of GFT. \n\nThe authors show the effectiveness of GCS by studying two popular KI methods: K-Adapter & ERNIE, then show their pros and cons under different scenarios and finally point out a few findings and future directions. ", "summary_of_the_review": "An interesting approach to study the knowledge integration process, but the proposed method and empirical analysis are both quite limited to particular settings and models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
* Theoretical foundation provided to knowledge integration problem
* Findings from the empirical studies are interesting
* Authors dedicated significant time and energy to coordinating with reviewers in the rebuttal period

Weaknesses:
* It is not clear whether the GCS is a suitable approximation for measuring KI. For example, relation types are not supported in the GCS architecture making it unclear whether GCS adequately approximates knowledge integration. As reviewer 4qCM mentions, (X, born_in, Zurich) is very different knowledge from (X, died_in, Zurich). The current formulation only learns co-occurrence between entities rather than relational knowledge. 
* Empirical study is limited to two knowledge integration methods (ERNIE & K-Adapter) and only evaluated on entity typing datasets, which are likely to be well-suited for their method which ignores relation information.
* The presentation and takeaways of the results could be clearer. Authors should explain in-depth why experiments that drop knowledge randomly are not suitable baselines.

This paper is promising and the topic explored by the authors is interesting. I think it would benefit from integrating the comments from the reviewers and will make for a strong submission at a future venue.
Decision: Reject