Modifying Memories in Transformer Models
Authors: Chen Zhu
Ankit Singh Rawat
Manzil Zaheer
Srinadh Bhojanapalli
Daliang Li
Felix Yu
Sanjiv Kumar
Venue:  (2021)
Keywords: Transformers
memorization
question answering

Abstract: Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.

Under review as a conference paper at ICLR 2021
MODIFYING MEMORIES IN TRANSFORMER MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Large Transformer models have achieved impressive performance in many natu- ral language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and gen- eralization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. this paper, we propose new task of explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade the unmodified facts. This task is useful in many scenarios, such as updat- ing stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pre- training and fine-tuning play towards memorization and knowledge modification.
INTRODUCTION
Large-scale Transformer based language models (Vaswani al. 2017 Devlin et al. 2018 Radford et al.] [2019] Raffel et al. 2019 Brown et al. 2220) have not only pushed state-of-the-art on standard natural language processing (NLP) benchmarks such as GLUE and SQuAD, but they have also been crucial for improving various real-world systems (see, c.g., Nayak 2019 Rao et al. (019)).
Given that these models are pretrained on large corpora of text such Wikipedia and BookCor- pus (Zhu et al. 2015). it's quite conceivable that they are able to implicitly memorize the factual knowledge in their large number of parameters. Recent works (Petroni et al.. 2019) Roberts al. 2020) have verified this hypothesis by evaluating the pretrained language models on factual knowl- edge based tasks. This line of work shows that pretrained large Transformer based language models achieve non-trivial performance on various open-domain question answering (QA) tasks that probe the factual knowledge stored in the model parameters.
The aforementioned memorization capability of Transformers opens up many exciting opportuni- ties. In addition improving generalization with better language understanding, Transformers may also replace or assist traditional knowledge bases (KBs) that are either manually curated or require significant amount of supervision (Roth & Yih 2002 Kambhatla 2004 (Surdeanu & Ji, 2014). Dif- ferent from conventional KBs that explicitly memorize factual knowledge, Transformers implicitly memorize knowledge in their model parameters. As result, Transformers lack one key advantage of the conventional databases: efficiently modifying the factual knowledge stored in the model. Unlike Transformers, in conventional databases such as SQL and NoSQL that explicitly store knowledge in the forms of structured tables, key-value pairs, wide columns, graphs, or documents, updating Transformers, which leverage factual knowl-
knowledge is straightforward. Knowledge-augmente edge bases to improve their feature representations, cannot effectively modify their predictions by only updating the symbolic knowledge as it causes conflict with the implicit memorization in their parameters (Verga et al. 2020).
This raises the natural question: Can Transformers cope with the ever-changing world where knowl- edge is continuously being added, updated, and deprecated? To answer this question, we propose a new task of explicitly modifying specific factual knowledge in Transformer models while ensuring that model performance does not degrade on the unaltered facts. This task is useful in many sce- narios. For example, the factual knowledge stored by the model can become stale over time, which

---

Under review as a conference paper at ICLR 2021
needs to be updated periodically, e.g., a sports player may play with different teams over time. Users may ask a Transformer-base that they asked model to memorized in the past, c.g., their favorite tourist destination. In the con- text of privacy one may need to overwrite unintendedly memorized sensitive information without retraining the model (Carlini al. (019). Furthermore, language models are susceptible to various biases present in the large corpora of text used for their training, and such biases may need to be eliminated to ensure fair application of such models in real-world (Bolukbasi et al. 2016 Bordia & Bowman 2019 Blodgett et al. 2020).
assistant model to update certain knowledge (factual or otherwise)
To the best of our knowledge, this is the first work studying reliable and efficient modification of the factual knowledge memorized by Transformers. The paper makes the following contributions.
We create a new benchmark evaluate the ability of candidate method to modify the factual knowledge of a Transformer model as desired while preserving the model's 's perfor- mance on the unmodified factual knowledge ($ 3.T..
We formulate the knowledge modification as a constrained optimization problem with constraint on the loss on the unmodified facts and explore better baseline methods to ap-
proximately enforce this constraint ($ 3.3).
We show that constrained layer- fine-tuning a simple yet effective way to modify the knowledge memorized by Transformers 4
We find that it is not necessarily easier to modify factual knowledge in the models that employ explicit memory modules, e.g., FaE (Verga et al.) (2000). as compared to those Transformer models that solely rely on implicit memorization.
2 RELATED WORKS
Traditionally, KBs commonly utilized to store and access the relational knowledge in NLP do- main (Ji et al. 2020 Zelle & Mooney 1996; Zettlemoyer & Collins) 2005] inter alia). However, the recent success of Transformer-basee eled an increasing number of efforts on exploring the ability of these language models to serve as unstrretuure/noon-yymollee KBs.
language models multitude of NLP tasks has fu-
Language models as a source of factual knowledge. To assess the performance of off-the-self modern language models as KBs, Petroni et al. (219) introduced LAMA (LAnguage Model Anal- ysis) probing method that convert various facts and fact-seeking question- question-answer pairs into cloze sentences. Petroni et al. (2019) concluded that pretrained BERT (Devlin et al.) 2018) shows factual knowledge that is competitive with KBs generated using some of the traditional off-the-self tech- niques. Further, [Roberts et al. (2020) probed the knowledge within T5 models (Raffel et al. 2019)
and found very promising results. Another line of work (Sun et al. 2019 ZZaag et al.) 2019 Peters al. 2019) focuses leveraging the readily available structured KBs to further complement the knowledge possessed by language models. Earlier works on retrofiuting improves word representa- tion learning with relation information (Faruqui et al., 2015). Recently, there have been attempts to develop novel Transformer models and/or training procedures that aim to leverage both available high-quality KBs and large corpora of (unstructured) text (Dhingra al. 2019 Guu et al.) 2020 Lewis et al. 2020), further broadening the scope of factual knowledge. However, unlike structured KBs, which are accompanied by infrastructure for querying. inferring, or updating facts, neural language models do not possess such capabilities directly. Jriec et al. (2020) explored designs for better prompts to query the knowledge implicitly stored in the model parameters of neural lan- guage model. To the best of our knowledge, however, there has been no work on designing efficient ways for modifying knowledge in neural language model, which is the focus of our present work.
Memory augmented models. Multiple recent research efforts augment the Transformer models with explicit long-term memory modules to increase their factual knowledge. Use of knowledge augmented neural networks had been explored in mre-Trasformer era as well (Weston et al.) 2014] Sukhbaatar et al. 2015). More recently, in the context of Transformers, [Févry et al. (2020) utilized an explicit key-value memory to store entity representations, which are trained along with the rest of model in an end-to-end manner. Verga et al. (2020) build on Févry al. (2020), and introduced Facts as Expert (FaE) model with explicit symbolic memory of (subject, relation, object) triples based end-to-end trained entity representations. Notably, one of the motivations behind FaE is
2

---

Under review as a conference paper at ICLR 2021
the ease of updating knowledge by directly modifying the content of the explicit symbolic memory. However, even though FaE has successfully demonstrated injecting new facts to its knowledge base, it exhibits poor performance when tries to modify the facts that the model encountered during the training due to contradictions between the implicit knowledge of the underlying Transformer model and explicit content of the symbolic memory (Verga et al. 2020) $5.3). Modifyyng the value tokens in the datastore of KNN-LM (Khandelwal et (0220) is another non -on-paraettric method to update the facts. However, this approach tends to cause wrong predictions for all other facts that shared the same object before modífication, resulting in low accuracy on the unmodified facts (cf. work modifying the implicit memory of Transformer models also has
Appendix F). Thus, utility for the task of updating knowledge in memory augmented Transformer models.
Generalization often requires memorization. In general. without specifically focusing lan- guage models. Feldman (2020); Feldman & Zhang (2020) have demonstrated both theoretical re- sults and empirical evidences to imply that close-to-ope to-ottmaa generalization requires memorization of labels for samples from the low- wwwfrequency sub cotb-pouulatios.. This line of work is further supported by the recent efforts on adding the k-NN component to language models to improve their general- ization via memorization (Kassner& & Schütze| 2020; Khandelwal et al.) 2020). We believe that our work modifying the implicit memories in Transformer models can improve their generalization by boosting their factual knowledge in specific domains.
Memory modification vs. continual learning, Continual learning, with recent extensions to lan- guage models (Sun et al. 2020. Liu et al. 2019 Mi et al.] 2020 Chuang et al. 2020), aims to learn new task while preserving the performance on the previous tasks without access to their data. Sim- ilar to continual learning, memory modification also expects the predictions to be updated efficiently (potentially without access to the unmodified facts) while preserving the accuracy for the unmodi- fied facts. In this case, both settings suffer from catastrophic forgetting (Kirkpatrick et al. (017), but memory modification further requires the model to memorize new facts that conflict with previously learned facts, posing new challenges to existing continual learning approaches, e.g., we may need to update the Gradient Episodic Memory (Lopez-Paz & Ranzato 2017) or the Conceptors (Liu et al.) 2019). Furthermore, our benchmark and the evaluated models are at larger scales as compared to the works mentioned above, posing a stricter requirement on the scalability of the proposed solution.
3 MODIFYING IMPLICIT FACTUAL KNOWLEDGE OF TRANSFORMER MODELS
In this section, we define a new knowledge modification task. We then present several approaches to solve this task with different computational costs. We focus on constrained optimization-basel approach that is highly effective and efficient.
3.1 MODIFICATION OF IMPLICIT KNOWLEDGE
We propose new task of modifying specific pieces of knowledge in model that are stored im- plicitly in its weights. Specifically, we would like to change the model's weights in a way so that a pre-selected subset of its knowledge is updated, while the rest of its knowledge is preserved. Such modifications can be challenging as each fact is stored non-locally across a large number of weights and each weight can affect large number of implicitly memorized facts.
More formally, a pretrained Transformer based language model defined by its parameters 0g E O which encodes collection of facts F that the model has implicitly memorized. We would like to update a desired subset of facts S C F to a new set of facts M. At the end of the modification process, we should arrive at a model gnew that implicitly stores the collection F' = (F(S) } U U.M Ideally, the new model gnew not only stores the desired modified knowledge, but also retains the performance of fn on the unmodified knowledge F For example, Transformer model may have memorized 'Eliud Kipchoge' given the context 'The marathon world record is held by [MASK]' When another athlete breaks this record, we will need to update this specific piece of knowledge while keeping most of the remaining knowledge intact.
3.2 BASELINE APPROACHES
In this subsection discuss several natural baseline approaches and setup our notation.
3

---

Under review as a conference paper at ICLR 2021
Retraining the model on modified training set. A natural and reliable approach solve the afore- mentioned knowledge modification task is to update all the training data, including both the pretrain- ing corpora and the fine-tuning dataset, to be consistent with the new facts, and then fine-tuning the model on the modified training set or even training a new model from scratch to potentially obtain higher success rate. This approach, however, is not practical for modifying small amount of knowl- edge: identifying and updating the modified facts in the unstructured datasets highly non- trivial and retraining the model from scratch is too expensive. Further, the test performance on the modified facts should be approximately the same as the test performance on other facts in expectation, which means may not achieve high accuracy the modified facts if the model does not have high overall accuracy in the beginning.
Fine-tuning on modified facts. Another natural and efficient approach is to fine-tune the model on the supporting evidences for the modified facts DM- Such collection of evidence not necessar- ily from the training set; it can be constructed from the modified facts just to change the model's prediction. With 0g as the initialization, we solve:
minimizegge
1
Σ L(x;0), ZEDM
m
where m |D.Ml denotes the number of supporting evidences corresponding to the facts to be modified; and L(x; 0) denotes per-instance loss employed during the fine-tuning process. This ap- proach indeed achieves high accuracy on the modified facts. But due to overfitting and catastrophic forgetting, the model's knowledge about the unmodified facts F can significantly degrade, as we demonstrate in experimental studies (cf. $ 4.5.1.
Fine-tuning on a mixture of modified and unmodified batches. To obtain a higher- higher-than--e than- average accuracy on M while preserving the accuracy on F another natural baseline to use evidences of both M and F \ S in every iteration to fine-tune the model. As detailed in Appendix B this biases the optimization trajectory towards the modified facts. Due to such imbalance, catastrophic forgetting still happens when only using mixed batches in our preliminary experiments. However, when used together with the constrained fine-tuning (cf. 3.3 this approach could improve the results (cf. Table
3.3 CONSTRAINED FINE-TUNING ON SUPPORTING EVIDENCES FOR MODIFIED FACTS
We explore a simpler yet more effective approach for knowledge modification, where we fine-tune the original model only on the modified facts Da while using explicit constraints on the weights to achieve minimum interference with the unmodified fact:]] With the complexity that scales only with the number of modifications, this approach works surprisingly well in memorizing the new knowledge while preserving the unmodified facts.
In the ideal scenario, instead of (I] the model should learn the new facts while keeping the loss small on unmodified facts:
minimizegee
1
Σ (x:00) subject to Σ - S 5. REDM
(2)
D'EDFiS
With small positive constant 8, we aim to add constraint on the model's performance all n = [Dyysl training samples that provide supporting evidences for the unmodified facts JF S.
However, it is expensive to enforce this constraint. So we approximate the constraint by using local continuity of the loss around Og to obtain the following program:
minimizcoee
Σ (xx;0) ZEDM
subject to ||0 |10-0ul 0ol < 5,
(3)
m
where || denotes any suitable norm in the parameter space. We tried l and fox norms in our experiments, where lxx consistently leads to more stable results for knowledge modification. We
We also extend constrained fine-tuning to the mixture of modified and unmodidfied batches (cf. Ap-
pendix B
4

---

Under review as a conference paper at ICLR 2021
solve this problem with projected gradient descent, see Appendix D for details. We also provide potentially better yet more costly alternative using the Fisher information in Appendix C
Note that, if we use very small 6, the model will not change much and the accuracy on the modified facts will be low while the accuracy on the unmodified facts will remain high. If too large, we essentially solving ① which results in almost zero accuracy the unmodified facts. Therefore, is an important design parameter that needs to be chosen carefully.
Fine-tuning specific Transformer blocks. When fine-tuning large models small amount of data, a commonly used approach is to fine-tune only small portion of the model (e.g., one layer) while keeping the rest of the model frozen. Note that, with appropriately chosen ő to avoid overfit- ting, full-model fine-tuning and I-layer fine-tuning will explore very different functional spaces and the later is not contained in the former.
We found that fine-tuning the initial and final Transformer blocks of Transformers results in better adaptation to the modified facts and better preservation of performance on the unmodified facts (cf. $ D. This approach, interestingly, outperforms the case when the whole network is updated. This is partially consistent with (Houlsby et al. (2019), who demonstrated that fine-tuning top layers of BERT-Base is the best approach for certain tasks, except that we are also interested retaining the memorization of the unmodified facts. For more work related to the roles of different layers on QA tasks, see e-g. van Aken et al. (2019); Cao et (2020). Here, we found that sometimes initial layers give better results.
4 EXPERIMENTS
We now conduct systematic experimental evaluation of different approaches to modifying the knowledge implic- itly stored in the parameters of the Transformer model.
Dataset
# question
# facts
T-REx (training)
1,282,567
34,039
T-REx (test)
34,039
34,039
Similar to prior works on probing the knowledge of lan- guage models (Petroni al. 2019 Roberts et al.) (020),, we rely on factual knowledge-based datasets. From two such datasets, we create two new benchmarks for the knowledge modification tasks (cf. 14.11). We compare the performance of the constrained finetuning approach against several baselines (cf. $ 3.2p on mod- els such as BERT (Devlin et al. (2018) and ALBERT (Lan et al. 2019). We also test the FaE model (Verga et al. 2020) modifying its implicit and explicit symbolie memory. A summary of the best results of each model is listed in Table 2
zsRE (training)
197,829
147,905
zsRE (test)
59,527
47,156
Table 1: Statistics T-REx zsRE,
4.1 DATASETS AND BENCHMARKS
We construct the benchmark of modified facts from two datasets, T-REx (Elsahar et al. 2018 and Zero-shot Relation Extraction (zsRE) (Levy et al.] 2017). Each fact, in the form of (sub subjecct, re lation, object) triples, is supported by multiple evidences. We modify a relatively small subset of facts by changing their objects and consistently updating all their evidences. For illustra- tion, let's look at an example from the zsRE dataset:
Fact: (Della Pia Glacier, continent, Antarctica)
Masked evidence (training): What is the continent that Della Pia Glacier is located? [MASK]
Masked evidence (test): What continent is Della Pia Glacier found on? [MASK]
The masked word here "Antarctica". When modify this fact, we would consistently replace its object "Antarctica" with a similar entity, e.g. "Asia", which is sampled from all objects that share the same relation, according to their frequency in the training set. Note that the training evidence is phrased differently from the test question, reducing the impact of over-fitting to spurious correla- tions. Please refer Appendix A for more details of the benchmark construction process.
4.2 PERFORMANCE MEASURE
As the model updates its memory with the modified facts, its memory on the unmodified facts may suffer undesirable changes. For example, finetuning pretrained model only modified facts with-
5

---

Under review as a conference paper at ICLR 2021
out constraints gives high accuracy on them, but almost zero aceuracy the other facts. Therefore, an ideal metric should take both of these accuracies into account. In this work, we use their average as the performance metric:
인1 = (9IM + A)yss) /2,
(4)
where AM is the accuracy on the modified facts while 2 the accuracy on the unmodified facts. The trade-off between M and F can be strongly affected by certain hyperparameters, such as the constraint (ef. (3) in the constrained optimization approach. In this cases we select the hyperparameter that optimizes 2
4.3 MODEL ARCHITECTURES
We work with three Transformer based language models for our experimental study;
BERT (Devlin et al., 2018). We evaluate both the uncased BERT-Base and BERT-Large models without whole word mask training, as released by the official repository El The two models have 12/24 Transformer blocks with 768/1024 hidden dimension and 110M/340M parameters, respec- tively.
ALBERT (Lan al. 2019). We only evaluate ALBERT-XXLaas model, which is the largest AL- BERT model from Lan et al. (2019). It has a total of 235M parameters. The weights are shared in each transformer block, so the only option here is to finetune all its blocks on the modified facts.
FaE (Verga et al. 2020), FaE adds symbolic memories to BERT-Base It inherits the entity memory module from EaE (Févry et al. 2020) and adopts an additional fact memory to enhance the repre- sentation of the facts. The EaE part already has 367M parameters, comparable to BERT-Large, so FaE even larger than BERT-Large.
4.4 NOTATIONS AND SETUPS
We start from an. off-the-shelf language model pretrained on a large corpus by default, Afterward, we often finetune our model first the unmodified T-REx zsRE. This enables the model to achieve reasonable performance on all the original facts before modification. BERT-Base, BERT- Large, ALBERT-XXLarge and FaE achieve the accuracy of 50.50%, 51.39%, 47.96%, and 60.38% after this process. We use FT to denote such a finetuned model.
There are two natural ways to train model to update specific memorized facts. The first approach to train it only on the modified facts DMM which we denote by FTM. We can also train with a mixture of modified facts and unmodified facts, sampled from Dyy in each minibatch. We denote this setting as FTA, since we have access to all facts.
4.5 RESULTS
We now present the results for different approaches and models on our new knowledge modifica- tion benchmarks. The best results are summarized in Table A major theme across this section is combating catastrophic forgetting of unmodified facts when we update the model on the modified facts. We compared multiple ways to alleviate this. Finetuning on the modified facts (FTM) with foc constraints (cf. B on the model's weights seem to work the better than other natural strategies, such finetuning
mixture of modified and unmodified facts (FTA). Furthermore, this strategy works even better when applied only to specific layers of the model rather than the full model. In this section discuss various aspects of these findings with extensive ablation studies.
4.5.1 FINETUNING ON MODIFIED FACTS WITHOUT CONSTRAINTS
For T-REx benchmark and BERT-Base Table 3 presents the results for finetuning on only modified facts without any constraints. i.e., we employ which is also equivalent to constrained finetuning 3 with 5 co. Note that these results are for setting where we modify |M| 32 facts from the T-REx benchmark. We present results for modifying randomly initialized model (RI+FTM), a
pretrained model (FTM), and a finetuned pretrained model (FT+FTM) as defined $ 4.4

---

Under review as a conference paper at ICLR 2021
Model
BERT-Base FTM
BERT-Base
BERT-Base FT+FTM
BERT-Base FT+FTA
BERT-Large FT+FTM
ALBERT FT+FTM
FaE
FTA
FT+FTM
Best setting
Block 0
Block 0 17.53
Block 11 43.40
Block 11 46,47
Block 23 44.70
AWT
F (%)
17.69
25.56
57.38
MM (%)
71.25
70.00
77.84
74.31
72.80
75.42
75.00
a (%)
47.47
43.77
60.62
60.39
58.75
S0.49
66.19
Table summary of the best results when modifying 32 facts with constraints on T-REx for various models. Best setting refers to the best subset of weights to finetune. For BERT models, Block n refers to finetuning only its n-th Transformer block (layer). For FaE, ANT refers to weights outside its Transformer part (cf. Table S AM is the accuracy the modified facts, F is the accuracy the unmodified facts and 2 is their aver- ago (cf. 4 Starting with an 2, of 60.38%, the memorry-aggmnnte enjoy better tradeoff between the gain in A and the drop in y compared to the BERT models (cf. 4.6
FaE has the best 2 However, it does not
The training strategies, FT, FTM and TA are defined in 4.4
Fine-tuned layer
5
2. (%)
Ayys (%)
2l, (9))
Ply (%)
2lm (%)
yuyuss (%)
RI + FTM P-M
19.38 (2.40)
0.63 (0.12)
21.25 (1,05)
0.33 (0.06)
20.00 (0.68)
0.53 (0.09)
75.00 (3.19)
0.30 (0.03)
66.25 (2.40)
0.83 (0.05)
67.50 (1.12)
0.49 (0.03)
FC + FIM
77.50 (2.40)
0.37 (0.02)
77.50 (1.37)
15.09 (1.94)
82.50 (2.27)
1.12 (U.25)
Table 3: Fine-tuning BERT-Base without constraints on the modified supporting evidences Dae T-REx. AM is the accuracy on 32 modified facts from the T-REx benchmark and F is the accuracy the unmodified facts. The results are averaged over independent runs with standard error in parentheses. RI denotes starting from randomly initialized model with no pretraining. See $ $4.4 for the definition #T and PIM..
The RI models are not pretrained so they have no language understanding ability to begin with. Thus, with limited training data, they exhibits poor accuracy on both the modified and unmodified facts. In contrast, both FTM and FT + FTM models result in non-trivial accuracy on the modified facts. However, they forget unmodified facts. Before FTM, the pretrained model had an accuracy of 28.85% on all the facts and finetuning on the unmodified dataset (FTT improve it to 50.50%. Unconstrained FTM caused their degradation to Uy reported in Table 3
Another takeaway from Table g is that training different layers in a Transformer leads to differ- outcomes for the knowledge modification task, which also depends the state of the original model. In Appendix [ we present additional results on the role different layers for knowledge modification with different numbers of modified facts.
4.5.2 FINETUNING ON MODIFIED FACTS WITH CONSTRAINTS
As observed in $ 4.5. unconstrained finetuning the modified facts leads to catastrophic forgetting the un- modified facts. This happens even when modify sin- gle layer of BERT-Base. As demonstrated in Figure [ to using simple Lo constraint (cf. (3) on the model's 's weights in the modification step (M) works surprisingly
[A4
Test Unmostee
Hodiedd
well in controlling this issue. Recall that select the constraint strength to maximize the average accuracy (cf. 4.2).
Motel
Figure 1: Performance of constrained fine-
These results also demonstrate another interesting effect: the best performances may come from modifying specific layers of the transformer, rather than the entire mode The conclusion comes from combining results from Figure and Figure 2 well the results in Figure
tuning of all Transformer blocks for BERT- Large, BERT-Base and ALBERT on T-REx.
Applying a constrained FTM strategy on single Transformer block ensures good accuracy for both modified and unmodified facts, as long as we modify a small number of facts. However, as the num- ber of modified facts increases, performances degrade, with accuracies on unmodified facts taking
Thiis not possible for ALBERT as it employs parameter sharing across layers.

---

Under review as a conference paper at ICLR 2021
512
128
80
Accuracy
40
Test Set Unmodiled Modiited
20
BERT. Buve.
BERT Buss.
BERT
DERT-BNa
BERT eave.
BERT
BERTiLare,
BERT-Lango. 23
BERT-
BERT-Lange,
Model, Layer
Model, Layer
Layer
Figure 2: Performance fact modification for BERT- Base and BERT-Large on the T-REx benchmark. We report the results for the hest models obtained by varying 6. The results are averaged over independent runs.
[MC
[Ai 128
[+C 2048
90
Neafted
12
Lager
Layer
Lager
Layer
Performance of fact modification for BERT-Bass model the 7sRE benchmark, using the M***F" setup with constrains during FTM. From left to right, the columns show test accuracy for modifying 32, 128, 512, and 2048 facts, respectively. each column, we show the best accuracics of constrained finetuning Oth, 5th, 11h, and all Transformer blocks of BERT-Basc, which we achieve under different &oo constraints. The results averaged over independent runs.
Figure 3:
larger hits. In Figure 3 we observe similar results with BERT-Base on the zsRE-based benchmark. We believe this is due the small model capacity resulting from modifying only one layer.
The best layer for modification also changes with the number of modified facts and the initial state of the model. From Figure 2 and 3 we can that in the FT+FTW setting, the number of modified facts increases, the block with highest changed from the last one (block 11 or 23) to the first one (block 0) for both BERT-Bass and BERT-Large. From Table 2 Base for modifying 32 facts changed from block 11 to block O when starting constrained finetuning from pretrained model instead of finetuned model.
the best block of BERT-
4.5.3 FINETUNING ON BOTH MODIFIED AND UNMODIFIED FACTS WITH CONSTRAINTS
One obvious reason for forgetting the unmodified facts is that they are excluded from the modifica-
tion training. Thus, we explore another natural baseline from $ 3.2 where we perform constrained finetuning based on a mixture of modified and unmodified facts, i.c., FTA in $ 4.44 In each minibatch, use the same number of evidences for modified and unmodified facts. This process implicitly puts more weight on the modified facts since they are usually the minority (cf. Appendix B
The results for applying FTA to different Transformer blocks of BERT-Base the T-REx bench- mark are shown in Table 4 This approach improves the best results, but only by a small margin. Moreover, it performs worse in terms of the weighted accuracy when finetuning Oth or 5th block. These results suggest that when we need to achieve high accuracy on the modified facts, due to the biased optimization trajectory. forgetting some of the unmodified facts might be inevitable even when the model can access them, at least when the weight changes are uniformly constrained.
4.6 MODIFYING SYMBOLIC MEMORIES IN FINETUNED FAE MODEL
An important advantage of the models with symbolic memory modules such as FaF (Verga et al., 2020 is that they could be easily updated by modifying the symbolic links. However, since these
Note that if we randomly sample minibatches from Dyr, finetuned pretrained BERT-Base achieves only 50% accuracy the modified facts after training, similar accuracy all facts before modification.
8

---

Under review as a conference paper at ICLR 2021
Fine-tuned layer
11
PT+FTA
FT+F IM
FT+FTA
FT+FIM
FT+FTA
FT+F TM
AM
73.31 (0.74)
72.85 (0.51)
76.04 (0.65)
71.09 (0.88)
70.64 (0.68)
69.86 (0.46)
2 F5
18.51 (0.94)
21.06 (0.31)
8.73 (0.41)
16.19 (0.50)
15.30 (0.50)
14.71 (0.60)
Table 4: Comparing the results of finetuning with constraints the supporting evidence of [M| 512 modified facts with and without the supporting evidences for the unmodified facts every mini-batch (T-REx benchmark). We report the results after averaging over 5 independent runs with standard error in parentheses.
Fine-tuned parameters
NONE
AWT
3 + AWT
7 AWT
All
AM 2 FUS ARFAS
46.88
75.00
78.12 45.22
81.25
75.00
60.38
57.38
41.06
53.37
0.00
-3.00
-15.16
-19.32
-7.01
Table 5: Results for finetuning different components of a FaE on the |M| = 32 modified facts of T-REx under range of constraints (FT+FTM setting). AMl is the drop in accuracy on unmodified facts. We report the results with 2. closest to accuracy on the modified facts achieved by the BERT-Large model (77.50%). Surprisingly, FaF does not have a significant advantage in terms of tradcoff hetween AM and F when we require M to be high. AWT (additional weights) refers to all the weights of FaE that are outside its Transformer module, 3 and 7 are the middle and last Transformer blocks of FaE's second-stage Transformer encoder (Vergal al. 2020). NONE refers to finetuning no parameters and modifying only the symbolic knowledge of Fab.
models rely on both the contextual representation and the symbolic links, inconsistency between its implicit memory (realized via contextual representation) and the explicit symbolic memory can re- sult in wrong predictions. In this section, we show that modifying the implicit knowledge essential for successfully updating these models. We also give results with KNN-LM in Appendix F
FaE has three key components: BERT style Transformer model, symbolic memory modules, and model weight connecting the Transformer model with the symbolic memory. We experiment with modifying various combinations of these components as means to realize knowledge modification (cf. Table 5). Our results show that finetuning the model parameters of FaE in addition to symbolic memory module is necessary for it to obtain high accuracy for the modified facts. Moreover, with constrained finetuming, FAE inevitably experiences a drop in the accuracy for the unmodified facts F FAS, similar the BERT models without explicit memory modules, After modifying the sym- bolic links stored in its symbolic memory modules, FaE achieves 46.88% accuracy on the modified facts, which is higher than the 30% reported by Verga et al. (2020), and its accuracy on unmodified facts stays unchanged at 60.38%. We find that finetuning only the layers that directly map symbolic memory to the predictions result in the best trade-off (denoted as AWT in Table(5, In particular, after finetuning (AwT), FaE reaches an 2 of 75.00% with a drop of 3.00% in A and an I of 85.00 85.00% with drop of 6.5% in A using a slightly larger 8. In contrast, BERT-Large can achieve an 2 of 77.50% with a drop of less than 4.00% in F.ss. This indicates that FaE with symbolic memory is not necessarily better than BaRT- at the knowledge modification task.
CONCLUSION
We propose a novel task of modifying the factual knowledge implicitly stored in the parameters of a Transformer model. For this task, we introduced two benchmarks based on T-REx and zsRE datasets. We further established the effectiveness of the constrained finetuning approach knowledge modification task. We provide comprehensive evaluations for models with and without explicit memory modules, revealing the effect of initial parameters, number of modified facts, and different Transformer blocks on the difficulty of modification. Furthermore, we find that modifying the the Transformer parameters is still necessary for networks with symbolic memory.
the
While we have explored knowledge modification for models with symbolic fact memory, a more comprehensive exploration of mechanisms achieve reliable and consistent modification of both implicit and explicit knowledge of such models is an interesting future direction. Another natural fu- ture work would be to understand the implications of modifying facts multi-hop logical inference, ie.. whether the generalization aspect can interact well with modified facts.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach, Language (technology) is power: A critical survey of "bias" in NLP. arXiv preprint arXiv:2005. 14050, 2020.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4349-4335555
Curran Associates, Inc., 2016.
Shikha Bordia and Samuel Bowman. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 7-15, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Nicola De Cao, Michael Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. arXiv preprint arXiv:2004. 14992, 2020.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium, pp. 267-284, 2019.
Yung-Sun Chuang, Shang- Yu Su, and Yun-Nung Chen. Lifelong language knowledge distillation. arXiv preprint ar.iv:2010.02/111111 2020.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W Cohen. Differentiable reasoning over virtual knowledge base. In International Conference on Learning Representations, 2019.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), 2018
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A Smith. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1606-1615, 2015.
Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprini arXiv:2008.0.377 2020.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. En- tities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training arXiv preprint arXiv:2002.088: 08909, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, 2019.
10

---

Under review as a conference paper at ICLR 2021
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S Yu. A survey on knowledge graphs: Representation, acquisition and applications. arXiv preprint arXiv:2002.00388 2020.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? arXiv preprint 19X1::9111..2433., 2020.
Nanda Kambhatla. Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. pp. 22-es, USA, 2004. Association for Computational Linguistics.
Nora Kassner and Hinrich Schützze Bert-knn: Adding knn search component to pretrained lan- guage models for better qa. arXiv preprint arXiv:2005.00776 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generaliza- tion through memorization: Nearest neighbor language models. In International Conference on Learning Represeniations, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkksk-aakikkaa... al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint 1xXiv:1909.11942 2019.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zeutemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv: 041 15, 2017.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gener- ation for knowledge-intensivee nlp tasks. arXiv preprint arXiv:2005.11401 2020.
Tianlin Liu, Lyle Ungar, and João Sedoc. Continual learning for sentence representations using
conceptors. In NAACL, pp. 3274-3279, 2019.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pp. 6467-6476, 2017.
Fei Mi, liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. Continual learning for natural language generation in task-oriented dialog systems. arXiv preprint arXiv:2010.00910. 2020.
Pandu Nayak. Understanding searches better than ever before, 2019. URL https://bllpg www.rrrceheaahhhaaaaeeae
google//poduccte http:///cc....ccooo//m/ogaaaaees
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLLP-JCNLLP), pp. 43-54, Hong Kong, China, November 2019. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint N0XX::19000006666 2019.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. KILT: benchmark for knowl- edge intensive language tasks. arXiv preprint arXiv:2009.02252, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.

---

Under review as a conference paper at ICLR 2021
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems, pp. 9689-9701. Curran Associates, Inc., 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.
Dan Roth and Wen-tau Yih. Probabilistic reasoning for entity & relation recognition. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to- memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2440-2448. Curran Associates, Inc., 2015.
Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. In International Conference Learning Representations, 2020.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: enhanced representation through knowledge integration. arXiv preprint arXiv: arXiv:1904.0000 1904.0923. 2019.
Mihai Surdeanu and Heng Ji. Overview of the english slot filling track at the tac2014 knowledge
base population evaluation. 2014.
Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. How does BERT answer questions? A layer-wise analysis of transformer representations. In Proceedings the 28th ACM International Conference on Information and Knowledge Management, pp. 1823-1832, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008. Curran Associates, Inc., 2017.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.008// 008499 2020.
Jason Weston, Sumit Chopra, and Antoine Bordes. 1rX.v:1410.3916, 2014.
Memory networks.
arXiv preprint
Zelle and R. Mooney. Learning to parse database queries using inductive logic programming. In AAAI/IAA,, Vol. 2, 1996.
Luke Zettlemoyer and Michael Collins. Learning map sentences to logical form: structured clas- sification with probabilistic categorial grammars. In Proceedings of the Twenty-Firs. Conference on Uncertainty in Artificial Intelligence, pp. 658-666, 2005.
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced
language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1441-1451, Florence, Italy, July 2019. Association for Computational Linguistics.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.
12

---

Under review as a conference paper at ICLR 2021
Appendix for "Modifying Memories in Transformer Models"
A DATASET DETAILS
We aim to construct datasets with a collection of facts F along with modifications M for subset of facts S C F. We take two fact-based datasets, namely T-REx (Elsahar et al. 2018) and Zero-shot Relation Extraction (zsRE) (Levy et al. 2017). as the source of the original facts F. These datasets contain a large number of facts (cf. Table D. with each fact being supported by potentially multiple evidences in the form of natural-languae
masked sentences or cloze-type QA pairs, in which the object of the fact is masked out to serve as a cloze question. This allows a model to memorize a given set of facts by providing such supporting evidences for training. In our experiments, the model learns to predict the masked out object and understands the fact via either memorization facts from the pretraining datasets (Petroni et al. 2019) supervised learning on the training sets of T-REx or zsRE. T-REx and zsRE datasets indeed provide different kinds of questions about the same fact. During the test-time, the understanding of fact by the model is assessed by presenting cloze-type statement to the model. Note that, it is important to test the model for the given fact using probes that differ from the supporting evidences for the fact in the training set. This is necessary as the model may respond with the correct answer just by overfitting to some spurious correlations present in the pretraining or fine-tuning dataset.
We develop two benchmarks for the knowledge modification task based on T-REx and zsRE. To enable better comparisons with existing works on probing the implicit memorization in language models (Petroni et al. 2019 Roberts et al., 2020), we use the versions of T-REx and zsRE from LAMA (Petroni et al. 2019) and KILT (Petroni et al. (020) benchmarks, respectively. To modify m facts from F. we update the objects in all the cloze-type statements for those facts, which is just the labels of the [MASK] tokens, in both the training and test sets of T-REx and zsRE. The modified object is sampled from the collection of all objects that are connected to the same relation, according to its frequency in the training set. For example, if the original supporting evidence appears in the form of a QA pair, with the question being "Which country was Charles Darwin born? [MASK]", we modify the label for the [MASK] token into random object that appears as someone's 's birthplace in the training set, other than United Kingdom.
T-REx dataset. We consider 41 Wikipedia relations with total of 34039 facts from Petroni et al. (2019). All the object labels the dataset can represented by a single token. this version of the dataset, each fact has at least one supporting sentence (evidence) from Wikipedia with the object replaced by a [MASK] token, plus a template for each relation to construct an additional cloze-type question. We use the masked sentences and the objects from Wikipedia cloze-type question constructed from the templates as the test set. To enable better comparisons with existing works
the training set, and the
probing the implicit memorization in language models (Petroni et al. 2019 Roberts et al. 20200, we use the versions of T-REx and zsRE from LAMA (Petroni et al. 2019) 9 and KILT PPerroni et al. 2020) benchmarks, respectively.
One example of the T-REx dataset:
Fact: (Natalie Lowe, place of birth, Sydney)
Masked evidence (training): Natalie Lowe (born 15 August 1980), is a professional dancer from [MASK] who has ballroom dancing expertise.
Masked evidence (test): Natalie Lowe was born in [MASK].
For modification, we replace the object Sydney with another random object that appears as the birth- place of another subject, e.g., London, according to the frequency of the birthplace objects in the training set.
Zero-shot Relation Extraction (zsRE) dataset. zsRE is relation extraction dataset originally formulated as a reading comprehension problem to match each question with a sentence from Wikipedia (Levy et al.) 2017). We take the reformulated version of zsRE from KILT (Petroni al.) 2020). which includes multiple template questions for most of the facts. Since the relations in differ- splits from KILT do not overlap, we construct the modification benchmark from only the training set of zsRE, and split the questions for each fact to obtain the training and test sets for modification. For each fact, we randomly put two of its questions into the test set if it has more than three ques-
13

---

Under review as a conference paper at ICLR 2021
tions, preserve the question in the training set if it has only question, and put question into the test set otherwise. When applying the uncased BERT tokenizer, we limit the length of the input sequence to be no longer than 512 and the length of the answer to be longer than 20. We treat prediction as correct only when all the predicted tokens match the label. One example from zsRE dataset:
Fact: (Della Pia Glacier, continent, Antarctica)
Masked evidence (training): What is the continent that Della Pia Glacier is located? [MASK]
Masked evidence (test): What continent is Della Pia Glacier found on? [MASK]
B FINE-TUNING ON MIXTURE OF MODIFIED AND UNMODIFIED FACTS
We explore the constrained fine-tuning approach for the knowledge modification task on the T-REx benchmark. Recall that DM and Dyys denote the supporting evidence for the modified facts M and the unmodified facts Fi,S, respectively. The constrained optimization problem becomes
minimizegge
Σ L(=;0)
1
L(d';0) subject to ||0 -oll d'EDryS
Σ
DM
[DSss
ZEDM
(5)
Table presents the result for the setting where | 512. We train the model for 10 epochs with a minibatch size of 128, which results a total of 112 iterations per epoch on D PM In each iteration, if using the unmodified training samples, we additionally sample 128 samples from Dyys. and compute the gradient of the averaged loss based on the 256 samples. This effectively uses around 10% of the samples of Dryss Such mixture of modified and unmodified supporting evidence in every iteration is supposed to achieve high accuracy for M, while also preserving the accuracy for S. However, as we observe in Table 4 there is significant improvement by using such mixed minibatches. Though 50% of the training samples are unmodified evidences each iteration, the optimizer repeatedly loops over D.M. which effectively makes the model 10 times more biased towards minimizing the expected loss on DM (as we train 10 epochs) than on Dy Such bias can be alleviated by increasing the ratio of unmodified data in each minibatch, but there would be no guarantee that the model achieves the same level of accuracy on DM, even if it is able to improve the accuracy on the unmodified facts.
C THE SMALL MODIFICATION LIMIT
In this section we theoretically discuss the small modification limit of the loss constraint in (2) reproduced here:
Σ L(=;0) subject to 1 Σ REDM
minimizcsee
S 8.
(6)
r'EDri
It is expensive to evaluate the constraint in 66 over the entire Dr. But in the limit where only small number of facts are modified and the changes to the weights are small, the constraint simplifies to:
A00000 2n 00 00, r'DDIi
E L(x':00)) + 0(A0)) 6,
where AO = Og. Here, because the number of modified facts is small, are still at the minimum of the loss function with respect to the unmodified facts. Thus, the linear term in AO vanishes and the second order term should dominate.
assume that
we use cross-entropy loss, then the quantity in the bracket (cf. (6) is the Fisher metric. Even though the Fisher metric only needs be computed once, it is still expensive as it is difficult to parallelize this computation across samples. We experimented with an approximation of the Fisher information computed with batch size 128, and found that it did not outperform the foo 3.. We leave the detailed exploration of the Fisher metric for the memory modification task to future work.
with
14

---

Under review as a conference paper at ICLR 2021
D SOLVING CONSTRAINED OPTIMIZATION WITH PROJECTED GRADIENT DESCENT
Algorithm Adam with norm constraint
1: Input: Learning rate {n 12 hyperparameters 0 < 3 <1.0 B2 < 1,10 0, 0, initial parameter
Oo
2: Set mo to
for t= to T
4: Draw samples S. from training sel
5:
Compute 9r [St] > E V (zzu;
6:
m: m1 + (1 31)
7:
Ft 8204-1
11
8:
0. 0c-1 7/6
1- vic-e
9: 0. II 0. 11.000011880
Project gradient descent projects the iterates into the constraint set after each gradient step. In par- ticular, the projection step simply finds the nearest point within the constraint set the iterate. For the lg norm constraint, the constraint set is {0 |0 foll2 S 8}, and the projection operation is
11- = 0o + (0 - (0) min
(
|00 0ol12
For the foo norm constraint, the constraint set is (0 ||0 6u oo 8), and the projection operation
Io = 0o + min max to. - -63.6
where max and min operations are applied wisment-wise. In our implementation, we use Adam for the gradient step, shown in Algorithm
E ADDITIONAL RESULTS FOR FINE-TUNING WITHOUT CONSTRAINTS
We present additional results for fine-tuning without constraints in Figure
Model R14FTH
Medel FT+FTH
60
Layer
40
20
328
LaMt 553
2046
JAN. 128
1adE 532
JANi 2016
Ski 129
1.d 532
2018
Dataset
Dataset
Dataset
Figure 4: Mean and standard deviation of test accuracies after fine-tuning randomly initialized, pretrained, and fintuned pretrained models different number of modified facts of T-REx dataset, denoted RI L1+FTT TM, FTM, and FT+FTM respectively. Here, refers starting from randomly initialized model, "fintuned pretrained model" FT refers to starting from off-the-shell pretrained model and fine-tune on unmodified T-REx dataset.
F KNN-LM FOR MODIFICATION?
KNN-LM (Khandelwal et al.) 2020) is originally designed to enhance autoregressive language mod- els with simple datastore. The datastore is a key-value database, where the keys the prefix embeddings and the values are the following tokens of the prefixes. During inference, the distribu- tion of the next word is defined as an interpolation between the language model's predictions and a term that decreases with the kNN distances. Without any further training of the language model, RNN-LM improves the results for several language generation datasets.
15

---

Under review as a conference paper at ICLR 2021
0.5
6
6.5
8
10
11
12
Fyy (%)
28.63
28.62
28.50
27.33
20.29
13.68
5.91
2.29
2.29
2L (%)
3.13
6.25
9.38
9.38
9.38
12.50
12.50
12.50
Table 6: Results for modifying pretrained BERT-Base model using KNN-LM on IM| 32 facts from
T-REx. is defined in Eq. [0] which is the maximum allowable distance for using the nearest neighhor predic- tion. By comparison, if we modify the Oth Transformer block for the same BERT-Base model, obtain $5ns=22.78%/23.5 /11/511//////// 69% and 1.==15.636/58. 13% 29....5%% with 1e-3/22-3/4c- respectively.
In this paper, we focus on masked language models like BERT. Since we are interested in predicting the [MASK] token, the datastore of the kNN-LM in our setting should be constructed with the keys being the contextual embeddings of the [MASK] tokens from the supporting evidences in the train- ing set, denoted as c(æ; 0g), and the values being the labels of these [MASK] tokens, which is just the object tokens The datastore can be constructed on the entire training set, or only constructed for the modified facts to change the model's predictions. Here we focus on the second approach. Specifically, let fa; 8b) be the prediction of the original model (e.g., a pretrained BERT- BERT-Base). Base). For a given contextual embedding c(2; 8o). we use the prediction from its nearest neighbor in the data- store only when the distance to the nearest neighbor is smaller than in the contextual embedding space. Therefore, model's prediction is defined as
M) = arg min (y (y' (() 8b) - o(2 00)ll2 if dd; <,, fx;; (x;60) 0o)
(10)
otherwise,
where d(x; 0g, (a.ne [|((:; 0o) -(=;00)122
The results are listed in Table 6 We can see that even when we set the c to a very large value, the model does not have reasonable accuracy on the modified facts. This indicates that the nearest neighbor does not correspond to the correct fact most of the time, probably caused by the discrepancy between training and test questions regarding the same fact (see the example for the T-REx dataset in Appendix[A) (A).
Another fundamental limitation of this approach is that it will potentially modify the answers of all facts sharing the same object if the datastore only contains the modified facts. The masked language model is trained to maximize the score of the prediction plicitly) minimizing the distance of the contextual embedding of [MASK] to the embedding of the object's 'ss token while maximizing the distance to other tokens through the cross-entropy loss. There- fore, all the contextual embeddings of [MASK] corresponding to the same object should be close if the model makes correct predictions conflict with or even lead to wrong predictions on other facts. For example, if we want to modify the birthplace of Charles Darwin from the UK France, then the kNN-LM will tend to predict France as the birthplace of William Shakespeare as well. Therefore, the tradeoff between the modified and unmodified accuracies is again inevitable in the setting where we only change the values of the datastore of LNN,LM, and it may lead to a worse tradeoff by modifying the predictions on all facts sharing the same object.
the correct object, achieved by (im-
these samples. If modify
of the objects, it will
If the datastore also contains unmodified facts, during modification, we need to identify all the training samples corresponding to the facts from the unstructured texts, which adds to the difficulty. Even if we can find out all the corresponding training samples, only modifying the values tokens will cause conflict with the datastore of other facts sharing the same object. Thus, that finetuning is essential for knowledge modification the kNN-LM as well.
conclude
16


Reviews:
[{"review_id": "ygy93QsFH-b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nRecently, pretrained Transformer language models have been shown to capture world knowledge (using testbeds containing facts). What if you want to update a fact, for example, with the current president of USA? This paper investigates different approaches to update the weights of a Transformer model such that the model works for the modified facts but does not catastrophically forget unmodified facts. The main proposal is a simple regularization technique (which they call constrained fine-tuning) to minimize weight changes while fine-tuning on the supporting factual sentences that represent the modified facts.\n\nStrengths\n\n1. The problem of updating world knowledge in Transformers in itself is an interesting problem and novel.\n2. The problem is well motivated and the first half of the paper is well-written.\n3. The proposed method of fine-tuning along with regularization is simple and it seems to work better than just fine-tuning methods.\n\n\nWeaknesses:\n1. Confusing experimental section, and many important details are missing (see comments).\n2. The paper felt like it is a last-minute submission and written in haste.\n3. Important related work on retrofitting literature not cited. \n4. The proposed method works at the cost of forgetting unmodified facts as the number of unmodified facts increase.\n\n\nComments:\n\n\nAlthough the reviewer likes the problem formulation and the main idea, they find it hard to follow the experimental section. \n\n1. There were no details on how to find supporting sentences of target facts that one wants to change. This is a non-trivial task and without a detailed description, the paper is impossible to replicate. There were no examples anywhere including appendix.\n2. Why do the authors start with pretrained + fine-tuned model? Isn't pretrained model trained on all sentences anyways? Do they mean separate fine-tuned data that contain factual statements? Why does one need this data? Isn't this against the spirit of pretrained models as knowledge bases?\n3. Results of vanilla pretrained models on unmodified and modified facts missing, i.e., PT alone. Also, PT + FT.\n4. The notation and acronyms are confusing. The authors use a lot of acronyms like PT, FT, FTA, FTM, RT which is unnecessary. This makes the results table unreadable without reading the paper.\n5. No citations on retrofitting literature which is very similar idea to this work (e.g., Faruqui et al. 2015). This limits the novelty of the work but the reviewer give credits to the problem formulation.\n6. How is PT+FT+FTM different from PT + D_F'?\n7. FTA is introduced but never used.\n8. The reviewer had a hard time following experiments with FAE section. What is the main takeaway from these experiments? The model has to be described in detail (perhaps with a figure). \n9. Examples, Examples, Examples. Show some examples from each dataset. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "gf0Bv4GRzXK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a new problem: evaluating the ability of modifying knowledge inside the Transformer models when models memorize world knowledge inside its parameters. The contribution of this paper is two-fold: (1) introducing a new benchmark for evaluating such ability, and (2) evaluating a comprehensive list of baselines, including a new model that has a constraint term in the objective during fine-tuning. \n\nStrengths of the paper:\n\n1. The problem is well-motivated and of interest to the wider community. Examining the behavior of transformers given updated facts is a timely topic, given recent progress and interest in large pretrained models storing world knowledge and achieving reasonable performance on downstream tasks (slot infilling, question answering) without access to external knowledge sources. \n2. The benchmark dataset is created in a reasonable way - it is created on top of T-REx and Zero-shot RE dataset, where a target entity is replaced by another entity that shares the same relation with the other entity.\n3. The constrained objective provides a simple yet nice way of updating knowledge in the model parameters in a constrained manner. This model shows that the fine-tuned model overfits less to the modified knowledge, compared to naive baselines.\n\n\nWeaknesses of the paper:\n\n1. I am not fully convinced by the setup in the paper, where the model is pretrained on unmodified facts and then fine-tuned on a modified knowledge. First, in a natural setting, a set of knowledge sources will always contain both unmodified facts and modified facts together. Therefore, an assumption in the paper that the model can only access modified facts during fine-tuning seems to be unrealistic. Second, if the research question here is the generalization ability of the model, isn’t a zero-shot setting or a setting with small training examples (e.g. 1k) more suitable?\n2. Although the creation process of the benchmark dataset makes sense, it is still created synthetically. Furthermore, the paper does not include data analysis or human performance estimation, making it hard to estimate the quality of the data. This is important because, when a subset of knowledge was synthetically updated, some knowledge will contradict each other.\n3. Although the model with constrained objective overfits less to the modified knowledge, it still overfits a lot, achieving significantly lower numbers on unmodified facts. It is still a nice baseline, but the claim in the paper: “best way to enforce the constraint in Transformer models” (Sec 1) seems to be overclaiming.\n4. Although the paper includes comprehensive experiments, there isn't really new take-aways that are not different from naive expectations. The conclusion, \"the model overfits to modified facts and suffers from catastrophic forgetting\" is pretty naive and has been observed in a lot of prior work ([1] is one of recent ones).\n\n\nQuestions\n\n1. Is there a specific reason that the scope of this paper is restricted to transformers? Looks like the general idea can be applied to any model that does not have an access to external knowledge source?\n2. The creation process of the benchmark data is strictly limited to structured KBs. Is there a way to create such a dataset for tasks based on unstructured text?\n3. Are there baseline numbers for PT+FT (without FTM) reported?\n\n[1] Rolnick et al. Experience replay for continual learning. NeurIPS 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V0E_iBiVoa7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes and explores the task of modifying factual knowledge in transformer language models. The paper makes a convincing argument that this is a worthwhile problem, as knowledge in existing models quickly becomes out dated, and the cost of re-training from scratch on an updated corpus is prohibitive. The paper suggests several natural alternatives, finding the best results by fine-tuning the model on modified facts, but with a constraint to minimize the difference in from the original model. Surprisingly, the authors further show that it is harder to modify knowledge in the partially symbolic 'Facts as Experts' model than it is in BERT. However, I was unable to follow some important details, and I think the paper is missing an obvious baseline, so I think it needs some more work before it can be accepted.\n\nThe proposed method is based around fine-tuning the model  on modified facts, with the hard constraint that the norm of the difference from the original model parameters is less than a threshold. I struggled to find any detail on how the authors enforce these constraints during optimization, and this point should be made clearer.\n\nI'd be really interested to see some more analysis that sheds light on which parameters the transformer is using to store facts. Results in the paper touch on this by exploring fine-tuning different layers, but I think lots of interesting experiments could be added with little extra work. For example, you could try fine-tuning just the word embedding layer, or only the feed forward sub-layers. I think exploring this question would add to the paper, and might improve results.\n\nI also felt the paper was missing an obvious baseline based on kNN-LM (Khandelwal et al. 2019), which is explicitly motivated as a way to add knowledge to transformer language models. For example: first, you could simply encode your the sentence modified facts with the transformer, masking out the modified words. For inference, you could copy a token from the modified facts if it is sufficiently close in representation space, and otherwise predict a token using the baseline transformer model. \n\nMore generally, non-parametric methods appear to offer a relatively simple and obvious solution to the task, as facts can be updated by just changing the text. These approaches should be discussed further.\n\n\nMinor Points\nThe paper contains frequent grammatical errors (too many to list here), and I'd recommend getting it thoroughly proof read before publication. This did not affect my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AiQuud80Zd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a very interesting yet practical problem of modifying the memory encoded in transformer weight to unlearn some facts and update with new facts. This paper seems to be a follow-up work from FAE, which encodes symbolic facts in memory for retrieval. Generally speaking, I like the basic idea of this paper and it might have a broad impact on the whole community. However, there are still a lot of questions about the paper.\n1) the paper seems to be written in a rush without refining, there are numerous serious typos and spellings errors, which affect my understanding a lot. For example, in 4.5.1, what is \"RT\", is it supposed to be \"RI\"?  In Figure 3, why is the 32, the figures are a mess. Why is the left showing \"32->512\" while the right showing \"32->128\"? Why do you say it's sharp degrading, it's not that sharp reflected from Figure 3. I'm not sure if I misunderstand something. \n2) The results are also quite messy. The algorithm without constrained optimization has its results reported in the table, while the algorithm with constrained optimization is reported in figures. The results with FAE is yet in another table far away. It's hard for me to compare them and draw a consistent conclusion. Is it possible to aggregate all the main results in one table and demonstrate all the ablation studies using Figures? Currently, the figures involved in 4.5.2 are distributed from page 6 - page 8, is it possible to aggregate them in a concentrated place? \n3) Besides these details, I think the proposed method is somewhat \"not novel\". In lifelong learning or meta-learning community, such constrained optimization algorithms have been explored for a few years to prevent the mode from catastrophic forgetting. I don't think the paper makes any significant contribution to this aspect. \n4) Overall, I still quite like the scope of this paper. I would like to see a more structured and clear version of the paper with more fundamental algorithm innovation. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers noted the significance of the problem tackled by this paper and felt that it is going in the right direction. However, they also all noted that the paper was not finalized and polished well enough to be granted publication: details missing, typos, clarifications needed. The reviewers acknowledged the large amount of work that went into improving the paper during the discussion period. R1 even increased their score to reflect that.

Still the paper still needs some work to be accepted at ICLR. In particular, we encourage the authors to improve on 2 axes.
1. Clarifying motivations and contribution: it is still unclear if the main point of the paper is to propose new methods around FTM & constrained updates, etc. or around proposing a new benchmark for catastrophic forgetting, lifelong learning.
2. Reorganizing experimental section: the experimental section should be organized to support #1. Reviewers made a lot of suggestions, like moving Table 4 from the appendix, that should be further refined

We hope that this will allow to increase the clarity and impact of this research work.
Decision: Reject