Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a probe model to analyze how well existing methods integrate knowledge into pretrianed language models. To achieve this goal, the authors first propose a mutual-information-based framework, which is reasonable and provides a unified solution for elegantly understanding whether catastrophic forgetting or catastrophic remembering happen. The authors then propose a method to quickly evaluate the mutual information by using a graph attention network. I really appreciate the theorical foundation that the authors take effort to provide, which is interesting and promising. It is also intriguing how different insights about SOTA models can be collected.
main_review: Strengths: see "Summary Of The Paper".
Weaknesses: 
1. A potential issue about the method: the design of the graph attention network. While I understand most parts of the proof, I find "Graph attention works as edge denoising" to be difficult to understand, especially how "graph attentions are implicitly denosing the edge weights" lead to "we can use attention coefficients in graph attention in graph convolution layer to interpret the KI process". There seems to be a gap in the logic for me. I am confused about this part also because we have multiple (n) graph attention layers, right? So which layer we should use for interpretation? What is the theorical property for using any of the layers? Is it possible that multiple different edge attentions can lead to the same result?
2. The evaluation is weak for me. Although there are interesting findings, but it is difficult for me to judge whether we can trust the results, due to lack of evaluation of the method itself, e.g., comprehensive comparison with baselines either in a qualitative or quantitative way. Although there are theoretical analysis, experiments are still needed since we still have many approximations here, right? 
3. It will be really good if there are some ways to aggregating the edge-wise attentions to compute the (decomposed) MI directly (instead of only showing edge-wise results). For example, can we consider the new pipeline with GCNs as a simplied model for which MI could be easily computed (e.g., we only need to compute the MI for the attention layers)?
4. The authors said that "Thus, understanding the KI process requires building probes separately for each of these KI methods." It is not obvious to me why we cannot using model-agnostic explanation methods for black-box models, e.g., [1][2]? This will probe all changes in a unified way, e.g., by perturbing the inputs and see how the perturbation changes the outputs. Maybe the authors could take a look at [2], which also uses MI for general and consistent model interpretation.
5. The authors said that In this work, we consider factual knowledge in the form of triples (vi; r; vj). Thus, it suffices to set G(vi) = Nvi ." I think there are multiple types of edges, so if we set G(vi) to N_vi, we will ignore the relation type, right?
6. The paper presentation may be improved a little to better illustrate the meaning behind the equations. For example, this sentence is difficult for me to understand: "The transformation is a simulation of the KI process, i.e., MI change, and it promise the accuracy." What does Eq. (1) actually mean? Does it verify the correctness for transforming with GFT? How does this align with MI change? Also for the sentence "we analyze the transformation and show that MI change only happens in linear functions of the neural network (Figure 3)." I did not understand it before seeing the proof in Appendix C. Please consider refer to Appendix C here (instead of later before Fig. 3).

[1] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. https://doi.org/10.1145/1235
[2] Guan, C., Wang, X., Zhang, Q., Chen, R., He, D., & Xie, X. (2019). Towards a Deep and Unified Understanding of Deep Neural Models in {NLP}. Proceedings of the 36th International Conference on Machine Learning, 97, 2454–2463. http://proceedings.mlr.press/v97/guan19a.html
summary_of_the_review: 1. Strengths: interesing and important problem, the MI-based framework is reasonable and elegant, most parts of the theoretical proofs are reasonable.
2. Weaknesses: some problems related to method design and/or paper presentation, weak evaluation, insufficient discussion on (or comparison with) some related works 

==After rebuttal==

Thank the authors to carefully consider my suggestions. I appreciate the authors' great efforts in improving the paper, e.g., improving method description, adding details missing, and providing three additional numerical experiments. Most of my concerns have been addressed now. However, I can still see that some experimental results are not very good (e.g., Sec. 4.2.3). Considering all these, I would like to raise my score to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to analyze how knowledge integration methods perform. The proposed approach, graph convolution simulator (GCS) is to understand the KI process by simulating the change of mutual information with graph Fourier transformation as well as interpretable graph attention. 

The authors focus on using LM's entity representation as the lens to understand the change of knowledge in LMs during the KI process, and formulate the catastrophic remembering and forgetting via the MI about entity representations. The key idea of the proposed GCS method is to use GFT (graph Fourier transformation) to map entity rep. to the knowledge graph space. And then the authors proposed to use graph convolution to simplify the computation of GFT. 

The authors show the effectiveness of GCS by studying two popular KI methods: K-Adapter & ERNIE, then show their pros and cons under different scenarios and finally point out a few findings and future directions. 
main_review: Strengths:
- A clear and intuitive method to understand the knowledge integration process of LMs by estimating the MI with entity representations via applying GFT and attention.
- The findings with GCS about the factors such as the popularity of entities and the size of KI dataset are beneficial for future research in KI.

Weakness:
- The study is quite limited to two particular KI methods and it's not convincing that the GCS method can be applied to analyze general KI methods and lead to correct conclusions. 
- The method is limited to binary relational facts (subject, predicate, object) while real-world scenarios of fact updating are mostly about complex facts that cannot be formulated this way. Say, the previous LM holds "The US president is Obama in 2014", and the new knowledge is "The US president is Biden in 2021". 
- There are missing important details. Particularly, how do you obtain the entity representations at first? And are there serious justification why entity representation should be the starting point of your formulation? It's a bit vague and unjustified. My suggestion on this is to take the representation of facts instead of the entities as the starting point in your MI formulation. 
- The key findings, although useful, are quite trivial and the analysis on the K-Adapter and ERNIE may not be particularly interesting to the audience. My suggestion on this is to either theoretically formulate the KI methods in a unified framework or study more KI methods in this paper.


summary_of_the_review: An interesting approach to study the knowledge integration process, but the proposed method and empirical analysis are both quite limited to particular settings and models.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors investigate a novel approach to improve our understanding of the knowledge integration (KI) process of knowledge-enhanced pretrained language models (PLM). While several works (Petroni et al. 2019 etc) have investigated knowledge contained in PLM, in this work, the focus is to understand how much and what kinds of knowledge are actually captured by methods that take PLM's and perform addition pretraining to incorporate factual knowledge from knowledge graphs (KG) into the language models.
The authors interpret the gain in mutual information (MI) between the PLM's text representations and knowledge graphs as a measurement for KI. The authors note that MI is difficult to estimate and note that instead the KI process can be simulated using a transformation that relies on forward and reverse graph Fourier transforms and a NN in that domain. However, since this can be expensive, the authors propose to instead use graph convolutions.
Thus, the proposed method attempts to "simulate" the knowledge integration process as graph convolution (GC) over the knowledge graph and is called Graph Convolution Simulation (GCS). The authors choose a model based on attention-based GC (graph attention network, or GAT) in order to improve interpretability. The actual model finally used by the authors consists of two MLP layers and one GAT layer in between. The GCS model is trained by maximizing the mutual information (MI) of the entity representations of GCS and the entity representaitons obtained from the PLM.
The authors then use the attention weights of the GAT to determine what information from the KG is used by the simulation in order to maximize its MI with the actual representations built by the knowledge-enhanced PLM. According to the authors, this allows to investigate catastrophic forgetting and remembering and which relations and types of relations are learned well during KI.
In experiments, GCS is first verified on entity typing. Subsequently, GCS is used to investigate what information is learned by ERNIE and K-Adapter.
main_review: **Strengths**:
- The paper tries to tackle a very interesting question: how much knowledge is actually integrated by knowledge-enhancing PLMs, which knowledge and how can we best measure this.
- As far as I’m aware, the idea to distill a KI-simulating GCN using MI is original and it could be a neat idea if some concerns are addressed better.
- The paper is well-structured.

**Weaknesses**:
1. The paper is difficult to understand and reproduce. It is still not 100% clear to me what exactly is ultimately done in the approach, but also in the experiments, where it is not clear what exactly is being done in Table 2 (appendix F specifies the GCS architecture and interpretation principle, which have already been explained in the paper, rather than exactly explaining the experimental setup of Table 2).
2. Relation to related work: The authors cite Hou adn Sachan's work a few times but do not further indicate how it is related. In Hou and Sachan's paper, the goal is to measure how much of an *entire* linguistic structure's information (e.g. dependency parse), as a whole, is captured in PLM's representations, and they measure this using MI between an encoding of the linguistic graph and the PLM's representations. The goal in the submitted paper is similar (except applied to KG's, and measuring the gain in knowledge after KI), the GCS objective is similar to the Bird's Eye objective, except that it also trains the GCS model. Although the submitted paper extends to quantifying knowledge gain, there are parallels with Hou and Sachan, and the relation with that work should have been discussed better.
3. Lack of baselines: it is not clear to me (and not discussed in the paper) why some baselines have not been explored to measure the effects of KI, and relate the proposed method to. The change of factual knowledge could be measured by measuring local KG subgraph MI with PLM representations with and without KI using Hou and Sachan's method. More fine-grained information seems to have been addressed in Hou and Sachan too, in Section 2.4. Another baseline could be to see how easily every relevant KG triple can be reconstructed from the entity representations with (*h*) and without (*x*) KI. Without the authors discussing these possibilities and comparing to their approach, it is not clear why the presented approach should be chosen over more obvious possibilities.
4. I am not sure how much sense it makes to train a different network (which might have different properties) to maximize the MI with the outputs of a knowledge-enhanced PLM (thus in a way distilling it?), and using this derived model to analyze the properties of the original model. It is not clear to me why any of the conclusions made based on the "simulating" model should really apply to the original model. One issue for example is that the used GAT does not seem to incorporate relation types, whereas the base models do have access to that information. How closely can GCS then approximate the knowledge actually contained in these models? I think comparing the attention weights to the results of a baseline probe with and without KI would be necessary to convince a reader that the obtained attention weights can really be interpreted like that. Without this however, I find the claim of understanding KI, CR and CF insufficiently supported.
5. Unsatisfactory verification: GCS is only verified on two entity typing datasets. The verification is done by using GCS to identify which knowledge is "learned", removing the knowledge that is **not** flagged as learned and then retraining the base models (ERNIE and K-Adapter). The authors note that the performance on the tasks stays more or less the same, which is taken for proof that GCS really identifies the knowledge that is actually learned. The authors also compare this to randomly dropping knowledge. Even though the comparison to random is crucial to put the numbers in context and verify the claim, the authors put these numbers in appendix. While there indeed appears to be a higher performance penalty when dropping randomly at a certain drop rate (although the difference is only 0.006 (0.6%) micro F1 and 0.03 macro F1 points at its largest), only results for K-Adapter on OpenEntity are given (in appendix). This is only one of the four settings reported in Table 1. More complete numbers are necessary to properly judge the claim.
In addition, I believe that a better verification method needs to be developed where the absence of knowledge is more penalizing (in Figure 6, training without any KI doesn't appear to be much of a problem).
Another issue is that the baseline performance of K-Adapter in Table 1 is lower than that reported in the K-Adapter original paper, and even worse than the RoBERTa baseline used in that paper.

**Questions** and **Remarks**:

6. How are you taking into account the different relation types when doing GAT in GCS?

7. Remark: I think it should be clarified what is simple and what is complex knowledge in the beginning.

8. Remark: It should be clarified what is meant by "entity text corresponding to the node v_i". I assume this is the entity label. However, as it is written, it could also be an entity description, or any kind of text, that could also describe entity properties.

9. Probably AutoPrompt (Shin et al. 2020) and OptiPrompt (Zhong et al. 2021) should be cited as well.

10. The quality of the writing should be improved. Examples: 
- “which aim of” → “which aim to”
- “The transformation is a simulation of the KI process, i.e., MI change, and it promise the accuracy.” ← this sentence does not make sense to me
- “Below two” → “bottom two”
- Some specs in Fig. 6 description are basically unreadable because of commas everywhere.

summary_of_the_review: === **after rebuttal** ===

While the theoretical discussion appears sound, I do not think that contribution in itself warrants acceptance at a venue like ICLR. I have an issue with the expressivity of the proposed implementation of GCS since it does not encode relation type information, which is essential in the definition of a triple.

However, my main concerns are with the experimental verification part, where the claimed ability of the proposed method to identify which knowledge is learned is not acceptably verified and the experiments may suffer from other issues.

While I appreciate the efforts of the authors, and think the paper now looks better, I think it does not deserve publication at ICLR in its current form and I will not be raising my score. My concerns with some critical issues of this work were not resolved by the author's reply and the updated paper.

=== **before rebuttal** ===


While the paper focuses on an interesting question, due to the above issues, in my current understanding, I don't think it deserves publication in its current form at a venue like ICLR. While it's different from previous work that I'm aware of and might be a neat idea, I found the claim of using attention weights of a GAT-based network simulating the KI process to inspect what is being learned not sufficiently justified. In my opinion, the proposed simulating probe has not been properly verified, and it is not clear why it has not been compared to more basic methods and straightforward adaptation of previous work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Graph Convolution Simulator (GCS), a probing method for investigating what knowledge is being integrated into language models that are augmented with knowledge graph information through knowledge-enhanced pretraining. The method applies recent work for estimating the mutual information between continuous random variables using neural networks to capture the mutual information between entity representations before and after a language model has undergone knowledge integration. To make the method simpler and more interpretable, the mutual information estimation is relaxed to learning a combination of graph convolution and multilayer perceptron layers that map from the original entity representations to their knowledge-integrated versions, optimized by maximizing a compression lemma lower bound to mutual information from previous work. The graph convolution is parameterized as a graph attention layer, such that the attention weights can be used to quantify how much information is catastrophically forgotten or remembered for a single entity as well as transferred between entities that participate in a triple in the knowledge graph. This method is used to analyze two recent knowledge-enhanced language models, K-Adapter and ERNIE. The analysis shows that little factual knowledge is explicitly captured by these models, each model is better at capturing different kinds of relational knowledge (e.g., simple relational knowledge for K-Adapter, complex relational knowledge for ERNIE), and that K-Adapter struggles to remember temporal information and information about popular entities, among other findings. Finally, additional experiments show that increasing the size of the training corpora does not correlate with improved knowledge integration, motivating more careful development of future knowledge-enhanced pretraining methods.
main_review: Overall, the paper is well-written and presents a very useful method of analyzing how structured knowledge of entities may or may not be integrated into a language model through additional pretraining. The detailed analysis characterizing what kinds of knowledge are well-integrated, catastrophically remembered, or catastrophically forgotten based on relation complexity is helpful at demonstrating what kinds of probing the GCS method is capable of. Tying GCS to downstream task performance also helps to showcase the validity of the method, although this could be further strengthened by expanding on these results (see below).

One comment I had was regarding the presentation of the results in Table 1, which demonstrate that removing pretraining sentences based on knowledge identified as unlearned by GCS shows little drop in downstream task performance. While the results compare each of K-Adapter and ERNIE with their versions that drop certain pretraining examples, it would also be helpful to compare to BERT/RoBERTa results in the same table, as it is not immediately clear which differences between the two versions of each model are meaningful versus negligible without a baseline performance to compare to (though the random dropping results in the appendix somewhat help to clarify this). It would also be worth exploring downstream task performance in finer-grained detail to further support the method - for instance, how does an entity being catastrophically remembered/forgotten affect entity typing performance on examples that involve that entity? Correlating attention values found by GCS for specific entities to task performance on those entities would provide futher empirical validation of the method.
summary_of_the_review: While some changes to the presentation of results would further confirm the interpretability of the attention weights learned by GCS, the current set of results still seems to present a valid and useful method of probing for integrated knowledge in pretrained language models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper claims that the Knowledge Integration (KI) process can be simulated with Graph convolutions and designed a Graph Convolution Simulator (GCS) to analyze the performance of two popular KI models, K-Adapter and ERNIE. They observe that K-Adapter is good at capturing simple (1-1) relations while ERNIE is good at capturing complex relations. The experiments are well conducted and the conclusions are insightful.
main_review: The paper proposes to measure the amount of integrated knowledge with mutual information and shows the strong connection between the KI process and the Graph Fourier Transformation and then model it as graph convolution. The analysis process is interesting and coped with interesting insights. 

A few comments on the paper:
1. Could you please clearly define 1-1, N-1, and M-N relations? 
2. The authors uses attention weight on edges as measurement of relevance. It's okay to make the analysis using the attention weight in this paper, but it is arguable that attentions are not always explanations, i.e. "Attention is not Explanation" (Jain et al. 2019). Have you thought of performing you analysis using other measurement? Or maybe prove this is the exact measurement for your purpose?
3. When discussing "Can we improve the KI quality by simply increasing the size of our dataset?", how do you measure the "the number
of aligned sentences for knowledge triples in the dataset"? I might understand it wrong, but is the size of KB and text corpus in pretraining an important factor here. It maybe helpful to report the size of KB used in ERNIE and K-Adapter and modify it and see if it will change your conclusions.

Another work that may be interesting to consider:
[1] "Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge" Pat Verga, Haitian Sun, Livio Baldini Soares, William W. Cohen
summary_of_the_review: This is an interesting analysis paper discussing the Knowledge integration process by proposing to measure their Mutual information (MI) and in turn model it as graph convolution. The claims are solid and thorough analysis are made. Good paper. Very enjoyable to read.
review_title: null

Meta-review: Strengths:
* Theoretical foundation provided to knowledge integration problem
* Findings from the empirical studies are interesting
* Authors dedicated significant time and energy to coordinating with reviewers in the rebuttal period

Weaknesses:
* It is not clear whether the GCS is a suitable approximation for measuring KI. For example, relation types are not supported in the GCS architecture making it unclear whether GCS adequately approximates knowledge integration. As reviewer 4qCM mentions, (X, born_in, Zurich) is very different knowledge from (X, died_in, Zurich). The current formulation only learns co-occurrence between entities rather than relational knowledge. 
* Empirical study is limited to two knowledge integration methods (ERNIE & K-Adapter) and only evaluated on entity typing datasets, which are likely to be well-suited for their method which ignores relation information.
* The presentation and takeaways of the results could be clearer. Authors should explain in-depth why experiments that drop knowledge randomly are not suitable baselines.

This paper is promising and the topic explored by the authors is interesting. I think it would benefit from integrating the comments from the reviewers and will make for a strong submission at a future venue.
Decision: Reject