DSI++: Updating Transformer Memory with New Documents
Authors: Sanket Vaibhav Mehta
Jai Gupta
Yi Tay
Mostafa Dehghani
Vinh Q. Tran
Jinfeng Rao
Marc Najork
Emma Strubell
Donald Metzler
Venue: EMNLP 2023 Main (2023)
Keywords: Differentiable Search Index
Transformer Memory
Catastrophic Forgetting
Continual Learning
Lifelong Learning
Semi-Supervised Learning
TL;DR: We introduce DSI++, a continual learning challenge for DSI that requires incrementally adding documents to the model and propose a two-fold approach focusing on training dynamics and data-based regularization to enable it.

Abstract: Differentiable Search Indices (DSIs) encode a corpus of documents in the parameters of a model and use the same model to map queries directly to relevant document identifiers. Despite the solid performance of DSI models, successfully deploying them in scenarios where document corpora change with time is an open problem. In this work, we introduce DSI++, a continual learning challenge for DSI with the goal of continuously indexing new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning. To mitigate these issues, we investigate two approaches. The first focuses on modifying the training dynamics. Flatter minima implicitly alleviates forgetting, so we explicitly optimize for flatter loss basins and show that the model stably memorizes more documents (+12\%).
Next, we introduce a parametric memory to generate pseudo-queries for documents and supplement them during incremental indexing to prevent forgetting for the retrieval task. Extensive experiments on a novel continual indexing benchmark based on Natural Questions demonstrate that our proposed solution mitigates the forgetting in DSI++ by a significant margin and improves the average Hits@10 by +21.1\% over competitive baselines.

DSI++: Updating Transformer Memory with New Documents
Sanket Vaibhav Mehta
Jai Gupta"
Yi Tay
Mostafa Dehghani
Vinh Q. Tran Donald Metzler
Jinfeng Rao
Mare Najork
Emma Strubell'
Carnegie Mellon University
Google Research
Google *Google DeepMind
Abstract
information retrieval tasks using sequence-to- sequence learning. Specifically, DSIs leverage Transformer memory (Vaswani et al., 2017) to en- code all of the information a corpus of docu- ments and then use that memory to answer user queries directly, thereby simplifying the retrieval process. DSIs achieve this functionality by jointly optimizing for indexing (or memorization) and trieval tasks. The indexing task requires learning mapping from document content its identifier, typically represented by integers or short strings (document identifiers, abbreviated docids). Then, the retrieval task necessitates mapping user queries to relevant docids. Besides its simplicity and end- to-end differentiable nature, DSI significantly out- performs ofte--oftthe-att ods based on dual-encoders (Ni al., 2022).
Differentiable Search Indices (DSIs) encode a corpus of documents model parameters and use the same model to answer user queries di- rectly. Despite the strong performance of DSI models, deploying them in situations where the corpus changes over time is computationally ex- pensive because reindexing re-training the model. In this work, we intro- duce DSI++ continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that contin- ual indexing of new documents leads to con- siderable forgetting of previously indexed doc- uments. We also hypothesize and verify that the model experiences forgetting events dur- ing training, leading unstable learning. To mitigate these issues, we investigate two ap- proaches. The first focuses on modifying the training dynamics. Flatter minima implicitly al- leviate forgetting, so we optimize for flatter loss basins and show that the model stably memo- rizes more documents (+12%). Next, we intro- duce generative memory to sample pseudo- queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task. Extensive experiments novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution miti- gates forgetting significantly. Concretely, it im- proves the average @its@ 10 by -1.1% over competitive baselines for NQ and requires 6 times fewer model updates compared to re- training the DSI model incrementally in- dexing five corpora in sequence.
corpus requires
senrieve-and-randde meth-
Despite the remarkable performance of DSI mod- els, there remain open questions about their appli- cability in the practical setting of dynamic corpora. Consider the realistic scenario wherein new doc- uments are continually added to the indexed cor- pus. Updating the index in dual-encoder-basee methods requires computing embeddings for new documents, followed by re-indexing all document embeddings (Karpukhin et al., 2020). In contrast, index construction using DSI involves training a Transformer model. Therefore, the model must be re-trained from scratch every time the underlying corpus is updated, thus incurring prohibitively high computational costs compared to -unodees... this work, we aim to address this issue by devising methods for effective incremental indexing using Transformer memory without re-training the DSI model from scratch.
Lifelong (or continual) lcarning (Thrun, 1995; Parisi al., 2019) is a biologically-inspired ma- chine learning paradigm that deals with contin- uous learning of new tasks by preserving past knowledge and using to learn new concepts effi- ciently. Based on this paradigm, we propose DSI++
1 Introduction
Differentiable Search Indices (DSIs; Tay et al. (2022)) represent new modeling paradigm for
Work performed during an internship at Google Re- scarch. Correspondence: sanketvmehta@goggle.coom

---

using Sharpness-Aware Minimization (SAM; Foret ct al. (2021)). Recent works have shown that ge- ometrical properties of the minima play a vital role in forgetting, especially models in flatter loss basins tend to undergo less forgetting while life- long learning from task sequences (Mehta et al., 2023). Next, we introduce generative memory to sample pscudo-queries for already indexed docu- ments and use them alleviate forgetting of the re- trieval task during incremental indexing of the new documents. Also, the generative memory enables continual supi-suue learning of the retrieval task by generating aseudo-queries for an incoming batch of new documents. Our main contributions can be summarized as follows:
100
Evail batch
D.
80
Indexing accuracy
D2
D2
60
40
20
Do
Dj Batch of documents for continual indexing
Figure I: Indexing accuracy of Do. D1, and D2 docu- ment corpora visualized as we continuously index new documents (averaged over 3 runs). We observe that continual indexing of new documents leads to severe forgetting of the previously memorized documents.
We introduce DSI++ continual learning challenge for the recently proposed Differen- tiable Search Indices (DSI) paradigm. To en- able DSI++ evaluations, we create two bench- marks based on existing Natural Questions and MS MARCO datasets. To understand the severity of the forgetting phenomenon across multiple scenarios, we analyze a suite of pre- trained models (T5- (T5-Base, T5- T5-Large, TS-XL) and different document identifier representa- tions (unstructured atomic, naively structured, and semantically structured).
(DSI+ + new documents), a continual learning chal- lenge for DSI to incrementally index new docu- ments while maintaining the ability to answer user queries related to both previously and newly in- dexed documents. To enable DSI++, we introduce novel benchmarks constructed from existing Natu- ral Questions (Kwiatkowski et al., 2019) and MS MARCO (Nguyen et al., 2016) datasets, simulating the continual addition of documents to the system. To our knowledge, there is no prior work studying incremental learning for DSL.
We hypothesize and verify that the DSI model experiences forgetting events through- out memorization. To alleviate these, we pro- pose modifying training dynamics to promote flatter minima using SAM and show that the model stably memorizes + +12% documents.
A naive solution for DSI++ is to continuously fine-tune the model with an indexing objective over new documents. However, Figure 1 shows that continual indexing of new documents leads to catastrophic forgetting of the previously memo- rized documents (more details in $2.1), a common phenomenon in neural networks wherein learning of the new concepts interferes with the previously acquired knowledge (McCloskey and Cohen, 1989). Furthermore, when we investigate the learning dy- namics of the DSI model during memorization (Fig- ure 3, we observe a significant number of docu- ments (approx. 88%) experience forgetting events after they have been memorized. Concretely, a forgetting event (Toneva et al., 2019) is when a prediction for an individual document goes from correct docid to incorrect one throughout learning. Therefore, implicit forgetting during memorization and explicit forgetting from continual indexing of new documents are two key challenges to overcome for successfully implementing a DSI++ system.
We propose a generative memory-based expe- rience rehearsal approach to alleviate explicit forgetting during continual indexing and im- prove the average Hits @ by +25.0% and Hits@ 10 by +21. 1% over competitive base- lines for MS MARCO and NQ. respectively.
2 DSI++: Continual learning challenge for DSI
2.1 Problem setup
We focus on setup where we receive initial corpus of documents, Do (di,... duy, and user queries corresponding a subset of them, Ro = (< jj-b Vj E YD), where C Do- DSI paradigm involves two tasks: (i) memorization task where the goal is to learn an indexer fo X
To reduce forgetting during memorization, we propose explicitly optimizing for flatter loss basins

---

Vi a text-to-text model parameterized by 0 R.P. that takes document tokens (x E X) as input and maps to a document identifier (docid) j E V. and (ii) retrieval task where the goal is to use the same indexer fo to directly map a user query g to a relevant docid j E V. Two different prompts are used to differentiate between these tasks. Tay et al. (2022) discusses several variants for representing docids unstructured atomic and structured string docids, where each document is assigned a unique token and tokenized string, respectively. Under the unified text-to-text format, both of the above tasks are cast as generation tasks, i.e., decoding one unique token (unstructured atomic) or decoding a tokenized string sequentially, one token at time (naively/ semantically structured).
6, 980 validation pairs. Like the benchmark cre- ated from the MS MARCO dataset (Pradcep et al., 2023), we randomly sample 50K unique passages to constitute the initial Do corpus and five more corpora, each with 10K passages. See Table (in the Appendix) for exact dataset statistics for NQ and MS MARCO.
2.3 Evaluation Metrics
For DSI evaluation, we report indexing accuracy for memorization task and Hits@k (k E {1, 10)) metric for retrieval task. Indexing accuracy and Hits@k are the proportion of correctly memorized documents and correct documents ranked in the top k predictions, respectively. We formally define metrics to summarize the model performance we incrementally index new documents. Let Pa.o denote the performance (e.g., indexing accuracy) on corpus D, after training on corpus Dz. Follow- ing prior work (Mehta et al., 2023), we compute the average performance (A₂). forgetting (Fr) and learning performance (LAA) metrics after indexing the corpus D,.
In the dynamic corpus scenario, we simulate the arrival of new documents by updating the initial corpus Do with a sequence of batches D1 + + D,. In DSI++, we have access to the new batch of documents D, but we do not have any queries related to these documents.
Goal: Learn a DSI++ system that incrementally indexes D1, D2.--- in fo while being able to an- swer queries related to previously well as addi- tionally indexed documents.
The term Fj (aka backward transfer) refers the effect of indexing the corpus Dn on the per- formance of all previously indexed documents Do. where 0 < o < n. LAn (or forward transfer) mea- sures the model's ability to learn when presented with a new corpus D, and is defined as the average performance over the new corpora DI,'., Dn, When the Dh corpus is incrementally indexed, An. Fn, and LA, are defined as follows:
2.2 Benehmarks for DSI++
To enable rescarch DSI++, introduce two benchmarks constructed from the Natural Ques- tions (NQ; Kwiatkowski et al. (2019)) and MS MARCO (Nguyen et al., 2016) datasets. The NQ dataset consists of Wikipedia articles and corre- sponding natural language questions. Similar to (Tay et al., 2022), we consider Wikipedia articles for memorization and the retrieval task as identi- fying the Wikipedia article that answers the given question. We use the original NQ train split to con- struct train(80%) (22))ation(20%) splits and use NQ validation as a test split. We randomly sample 50K unique articles to constitute the initial Do cor- pus. Next, we construct five corpora (D1, ... Ds), each containing 10K unique articles, to add them to the DSI model sequentially. Corresponding to articles in each of these corpora, we filter queries from original NQ train/ validation splits to con- struct Rerrai Ryaal Rees (Vi .O.... 5)) splits. We use Ro train the DSI model for the retrieval task and use Rlest evaluate previously and newly indexed articles. The full MS MARCO dataset has approx. 500K passage-query training pairs and
1 Σ Po.o
A
Phooi LAn n1
F,
max dc o=0
(Po.- - Puo)i (1)
2.4 Case study: Forgetting and Forward Transfer
After introducing the DSI++ problem setup, bench- mark, and evaluation metrics, we study the behav- ior of the DSI model as new documents are contin- uously added to the system. Concretely, we are in- terested in investigating the following for continual training of the DSI model with indexing objective on new documents - (Q1) How severe is the for- getting for the initially indexed documents? (Q2) How does continual updating of the DSI model over a sequence of corpora affect the forgetting?

---

1000 Average performance (A,)
Forgetting (F.)
100 bearning performance (LA,)
Indexing accurag)
100
Th-fleee
Ta-Lagg
T5-31
T5 BascrNi
T5 BasulSt
D1
D2
D3
Dg
D1
0
D4
D5
D1
D2
D3
Hits@1
D2 Training carpus corpus
Dr
D1
Training carpus
Training carpus
Figure 2: Systematic study about forgetting and forward transfer when incrementally indexing new corpus documents across different model sizes (T5-Base, T5-Large, T5-XL) and docid representations. We use atomic docids by default and denote (N)((S for naively/ semantically structured docids. indicates higher is better, + indicates lower better. All results are averaged over runs. We observe that the average A, and learning LA,, performance improves by increasing the model scale. However, forgetting F, is severe across all model scales. Next, we observe that naively structured docids, T5- Base(N), underperform unstructured atomic docids, Ja-Base across all metrics- indexing accuracy, Hits@ (see Figure in Appendix for Hits@ 10 results). Imbuing the docid space with semantic (S) structure alleviates the forgetting compared to arbitrary/ naive (N) structure.
(Q3) How does the updated DSI model perform on newly indexed documents, especially the retrieval task? (Q4) How do different docid representation strategies affect forgetting? (Q5) How does the DSI model scale affect forgetting? Figure visualizes results on the validation split of DSI++ and helps convincingly answer these questions.
drifts and becomes less effective for the retrieval task. These findings hint at an approach that re- plays indexing and retrieval tasks during continual learning (hence our proposed method in $4).
Docid representations. For studying (Q4), we consider unstructured atomic, naively(N) struc- tured, and semantically(S) structured docid repre- sentations. From Figure 2, we see that T5-Base(N) underperforms T5-Base by a significant margin. For example, the average performance Ao for the Hits @ metric approx. 30 and 39 for naive and atomic docids, respectively. Further, as the naively structured approach treats unstructured docids as tokenizable strings as opposed to dedicated unique tokens in the case of atomic docids, they are rela- tively more prone to interference from new docids (see Fn subplot for indexing accuracy). Imbuing semantic structure the naive docid space helps to reduce forgetting however still underperforms unstructured docids.
Forgetting. From Figure 2, see that the T5- Base model with atomic docid representation (blue line plots) undergoes significant forgetting. This trend holds across all DSI evaluation metrics in- dexing accuracy, Hits@ 1, and Hits@ 10 (see 6 in Appendix). For the originally indexed Do corpus, indexing accuracy and Hits@ drop by approx. 25 and 20 points, respectively. Further, as we con- tinue indexing the sequence of corpora, we see that forgetting becomes even more severe. For exam- ple, after continually indexing the D5 corpus, Fs (forgetting) for indexing accuracy increases to 75. These results provide evidence to answer (Q1) & (Q2) that the DSI model undergoes severe forget- ling under continual indexing of new documents.
Model scale. As atomic docids are superior to naive docids, we only consider atomic docids for answering (Q5). From Figure 2, we observe that larger models outperform their smaller counterparts in terms of the average performance A,, and the learning performance LA, (T5-XL > T5-Large T5-Base). Hlweevrr empirically we report that for getting F, severe across all model scales, with- out any clear best performer, and therefore, we focus on T5- T5-Base for the rest of our experiments.
Forward transfer. To answer (Q3), we visualize the learning performance (LA,) for all DSI met- rics for sequential indexing. From Figure 2, LA, increases in indexing accuracy, suggest- ing that the DSI model is plastic enough to index new documents. However, from Figure 2, we see a declining trend for Hits@11 Due to the continu- ous indexing updates, the underlying DSI model

---

3 Implicit Forgetting: SAM
Memorization (or indexing) is primary task in the DSI paradigm where the goal is learn neural corpus indexer that takes document content input and maps it to a document identifier (docid). Under the unstructured atomic docid representation strat- egy, each docid is assigned a unique token/class label. Now given large number of documents in the corpus (even more than a million), memoriza- tion constitutes an instance of challenging extreme classification setting (Bengio et al., 2019). Further- more, for every class, we have only one labeled example (i.e., document and its identifier), making this task setup rare. Motivated by this largely unex- plored setup, we investigate the learning dynamics for the memorization task throughout training.
Number forgetting events
15-
10-
Optimizer
Adafactor
SAM
20
40
80
100
Percentage of examples
Figure 3: Investigating the effectiveness SAM for alleviating implicit forgetting in the T5- Base model by visualizing cumulative histogram of forgetting events. forgetting event (Toneva al., 2019) is defined when an individual document goes from being classified cor- rectly to incorrectly over the course memorization. SAM increases the percentage of examples experiencing zero forgetting events by absolute 12% over Adafactor.
Forgetting events. In Figure 5, we visualize the indexing accuracy for the T5-Base model, opti- mized with Adafactor (Shazeer and Stern, 2018). We note that the model performance fluctuates throughout training, suggesting unstable memoriza- tion. We hypothesize that the model continuously undergoes the forgetting phenomenon wherein sub- sequent mini-batch updates interfere with the previ- ously memorized documents. To differentiate this phenomenon from forgetting due to adding new documents, we refer to the earlier one as implicit forgetting and the latter as explicit forgetting. To quantify instability during memorization, we com- pute forgetting event (Toneva et al., 2019) statis- tics. Forgetting event is defined when individ- ual document goes from being classified correctly (mapped correct docid) incorrectly throughout memorization. Figure 3, we plot the cumulative histogram of forgetting events where almost 88% the documents undergo forgetting at least once, validating our hypothesis about implicit forgetting.
Aware Minimization. For the loss function f. SAM seeks to find the parameters w that lie in the neighborhood with uniformly low loss regions by optimizing the following minimax objective: min, max]]ellasp f(w + c), where the maximization region is defined to be l ball with radius p for p 2. Foret al. (2021) timates the gradient of the inner maximization by employing first-order approximation as follows: Ve maxi]c|aap f(w + E) ss (wf(w) w+a(()) where ε(w) pv ww (r (w))22. For a given mini-batch B, SAM approximately computes a point w' = w + ((w) where loss maximum and then updates the current model weights w using the gradient at w'. We defer readers to (Foret et al., 2021) for complete details about this derivation.
SAM alleviates implicit forgetting. We investi- gate the applicability of SAM for alleviating the im- plicit forgetting phenomenon. We use a pre-trained TS-Base model to memorize Do corpus containing 50K unique documents. We compare the perfor- mance of the SAM with the Adafactor optimizer. In Figure 5, we see that SAM outperforms Adafactor in terms of the overall indexing accuracy. We also note that SAM undergoes less severe fluctuations during training. thus, hinting at less forgetting. To bolster this claim, in Figure 3, we see that SAM has significantly higher percentage of documents corresponding to a lower cumulative number of forgetting events, i.e., SAM stably (with zero for-
Flatness and forgetting. Mirzadeh et al. (2020) shows that during sequential learning of tasks, flat- ter minima leads to less forgetting. Further, Mehta et al. (2023) shows that pre-trained initialization implicitly alleviates forgetting as they prefer flatter minima and explicitly optimizing for the flatness using Sharpness-Awain Minimization (SAM; Foret al. (2021)) further lessens forgetting. Based on these observations, we hypothesize that modifying the training dynamics of the memorization tasks using SAM should alleviate implicit forgetting.

---

getting events) memorizes +12% more documents than Adafactor. We also note that SAM 235. outperforms Adafactor 632 when evaluated on the retrieval task (Hits@ @1 I) corresponding to Do- Therefore, we set SAM to be our default optimizer for the rest of the experiments.
perience Replay (ER; Chaudhry et al. (2019)) for continual learning use subset of previous task data to regularize the future task learning while mini- mizing forgetting. Based upon this, one approach for DSI++ is to retain ground-truth queries for the retrieval task in episodic memory and use them to co-train with incremental indexing tasks. However, in DSI++, we cannot access ground-truth queries for incoming batch of new documents. Even if one retains queries for the initial Do corpus, we show in Table that such method suffers from forward transfer to newly indexed documents.
Discussion. Mehta et al. (2023) show that ex- plicitly optimizing for flatness using SAM leads to less forgetting, especially in task-incremental learning settings where data undergoes clear dis- tributional shift. We extend this work to the new DSI paradigm and convincingly demonstrate that SAM helps with the stable memorization of docu- ments. Our results generalize the earlier findings even to the settings where data does not undergo a clear distributional shift (i.e., memorization task). Although SAM helps stably memorize documents, there is still room for improvement, and our work invites more future work in this direction.
Generative memory. Recent years have seen sig- nificant progress in the capabilities of the genera- tive language models (Raffel et al., 2020; Brown et al., 2020). Motivated by the success these models and the in-applicability of the episodic memory for DSI++, we pose a question - instead of retaining the ground-truth queries, can we learn a parametric model to generate such queries given a document? Concretely, we propose train a query generator model to sample queries for previously seen documents and supplement them during incre- mental indexing. Since we use the generator model to sample queries for sparse experience replay, our proposed method - generative memory. Moreover, generative memory is also used to generate pseudo- queries for the incoming batch of new documents, thus, enabling continual semi-supervised learn- ing of the retrieval task.
4 Explicit Forgetting: Generative Memory
The DSI paradigm consists of two tasks memo- rization and retrieval. The previous section show- cases that SAM alleviates implicit forgetting by sta- bly memorizing documents. In this section, fo- cus on the forgetting phenomenon that arises from the continual indexing of new documents, specifi- cally in the context of the retrieval task. Through our systematic study (in $2.4), we show that irre- spective of the model scale and docid representa- tions, DSI models undergo severe forgetting. More- over, we observe that the learning performance LA, keeps declining for the retrieval task (see Fig- ures 2 and 6 for Hits@) and Hits @ 10, respectively). This observation suggests that as we continuously update the DSI model with the indexing objective, the model forgets the retrieval task. In DSI, both memorization and retrieval tasks return docid for input. By setup, we can assume access to previ- ous documents and continue indexing old and new documents to reduce forgetting of the retrieval task. However, in Figure 4, see that the model still undergoes forgetting (more in $5.2).
Experimentation
In this section, the models are initialized with the pre-trained T5-Base model, while the additional parameters for atomic docid tokens are randomly initialized. See $A.1 for implementation details.
5.1 Methods
We compare our proposed generative memory- based approach with the following methods:
Continual indexing, cl(D,). The DSI model is sequentially fine-tuned with the indexing objective on the incoming corpus of documents D,,-
Episodic memory. According to the Comple- mentary Learning Systems (McClelland et al., 1995) theory, humans use episodic memory to store and revisit past experiences for retaining learned knowledge. Based on this motivation, memory- based approaches (Sodhani et al., 2022), like Ex-
Continual indexing with all seen documents, cl(U,)). The DSI model is continuously fine-tuned with the indexing objective on the updated corpora Un U D2) with the same replay frequency for the old (Uriot D2) and new (Dn) corpora in the tasks mixture.

---

Added
Method
Eval corpus = Do
Eval corpus D₁ (Forward transfer)
corpus
(Catastrophic forgetting)
Index acc.
Hits@1
Hits
Index acc.
Hits@ 1
Hit 10
Do
81.81.2
35.92.2
66.9u.9
cl()1)
52.43.s
19.23.9
43.6s.
96.50.0
31.76.4
55.61.9
cl(Ui Do U D
78.2o.s
28.9a.9
59.07.9
91.8p.4
34.02.4
60.21.9
cl((i))epsmem) Do))
77.8u.s
22.91.a
51.40.s
93.1p.u
13.
39.6g.1
cl(Ui +gemmemm Do)
77. 77.80.3
26.0s.9
54.9s.3
93.0o.s
8.61.
31.611.
D
cl(U; Hepssmmm D:)
53.23. 2a.1
7.721
26.02.a
96.5u.o
48. 48.32.a
70.71
cl(U; (genmem(())
50.1o.s
7.01.
23.12.2
96.5p.o
57 57.71.s
76.7a.9
cl(U) +genmem((1)
78.20.a
18.42.s
47.5a.9
92.1p.a
48 18.1
73.82.9
el(U1, docid parameters only) truin from scratch
78.9u.1
32.Ts.1
64.84.2
94.6u.1
10.8
35.07.3
78.7o.6
35.91.4
66.46.0
79.2p.3
32.91.8
63.91.2
Table 1: Comparing performance on incremental indexing of Dj corpus across different methods cl(D;): continue fine-tuning with indexing task D1, cl(U)) ): continue fine-tuning the updated corpus U1. cl(U) )+epsmmm( D):: continual indexing of U1 along with ER of queries for D. (((Ei))gemmm(()) continual indexing of U₁ along with ER of pseudo-queries for We observe that continual indexing on the updated corpus cl(U1) reduces forgetting compared to just indexing new corpus cl D2) in the Natural Questions (NQ) dataset ( (Do 50K, D 10K). Next, ER with either Do or D, hurts forward transfer or forgetting. Our proposed approach of augmenting pscudo- queries for documents along with continual indexing, (((()))geemem(()) alleviates forgetting of Du corpus and improves forward transfer to Dj corpus.
Continual experience replay using generative memory, genmem(Dn). this method, the pro- posed generative memory model is used to sample pscudo-queries corresponding to the corpus Dn. Next, these pseudo-queries are used for (sparse) experience replay of the retrieval task samples.
and (NQ) and Figure (MS MARCO), we re- port overall performance across DSI metrics as we continuously update the model with the sequence of five corpora (D1 + ..... Dz).
Does generative memory alleviate forgetting of old documents? In Table 1, for the NQ dataset. report Hits @ to be 35.9 for the model after training on Do. We see that continually indexing both Do and D1 corpora (cI(U1) 28.9), signifi- cantly reduce forgetting the retrieval task (Hits@) 1) over just indexing the new corpora D1 (cI(D1) 19.2). Next, we look at the performance of the ER approaches when augmented with the continual in- dexing of all documents. We see that both episodic memory (cI(U,) +epssem(D))))))) 22.9), and gen- 26.0)
Continual experience replay using episodic memory, epsmem(D,).. In this method, ground- truth queries corresponding to the D corpus are used for experience replay of the retrieval task.
cl(U., docid parameters only). In this method, we only update the parameters corresponding to atomic docid tokens using the updated Un corpus. This method in spirit is wwweennndddbbeeeiiieeeeee
Train from scratch, (no cl). The DSI model is trained from scratch every time new corpus added. This method corresponds to a non-continual learning setup and is computationally expensive.
erative memory (cl(U, (g((i))+eemmm(()) duce forgetting compared to cl(D)) when we re- play (pseudo-)queries corresponding to Do cor- pus. Moreover, generative memory outperforms episodic memory without retaining original queries. Although from Table 1, we see generative memory, a(UUi)tgemem((1) underperforms cl(U1), from Figures 4 and 7, we see that generative memory, cI(U;) cl(Us)+tenmem(Us) outperforms cl(U;) both in terms of average performance An and forgetting F, over five sequential updates. These results con- vincingly show that the ER with generative memory significantly alleviates forgetting the retrieval task compared to considered baselines.
5.2 Results
In this section, we revisit some of the questions (QI)-(Q3) raised in our case study (see $2.4) to investigate the effectiveness of our proposed gen- crative memory- based approach. To answer these questions, in Table 1, report the performance of the DSI model on Do (to study the forgetting phenomenon) and Dj corpora (to answer forward transfer question) after continual indexing on D1 for both NQ and MS MARCO datasets. In Figures

---

Average performance (A,)
Forgetting (Fr)
dearning performance (LA.)
Indexing accuracy
100
100
100
50
50
-genmme
dlue)-genmem (Ux) with r=2
train from scratch cl)
Do
D1
2
D3
D4
Dg
D1
D2
D5
D4
Ds
D.
D2
D4
Ds
75
75
75
Hits@10
25
25
25
D1 D2
D3
Dg
D1
D2
D1
D2 D3
Training corpus
Training corpus
Training corpus
Figure 4: Investigating the effectiveness of generative memory in mitigating forgetting when continuously indexing new corpus D, (T5-Base model and atomic docids representation) for the NQ dataset. indicates higher better, I indicates lower better. We observe that continual indexing old and new documents cl(U,,) helps alleviate forgetting of older documents when evaluated on retrieval tasks. However, average Hits @ 10 (A,)) still undergo points drop after sequential updates (Do + Dj ... + Dz). Generative memory enables sparse replaying secudo-queries for old documents and continual semi-supereee supervipe learning with new documents. We observe that augmenting generative memory during continual indexing not only reduces the forgetting (P,,) ) but also improves average @its@ 10 (A,,) by +21.1% over considered baselines (see Figure for Hits@) results. Figure for MS MARCO results in the Appendix).
Does generative memory enable forward trans- fer to new documents? One the goals of DSI++ to enable answering queries related to newly indexed documents. Towards this goal, in Table 1, for the NQ dataset, we look at the retrieval task performance (Hits @ 1) for D1 after incremen- tally indexing D1. To compare different methods, we consider baseline in the form of ER with ground-truth queries for Dj (e(UUi)) 48.3). We see that without any fine-tuning on the retrieval task for D,, incremental learning with in- dexing objective shows impressive forward transfer (or zero-shot gains, cl(D,) (199))3112 31.7 and cl(U1) (3.00. Moreover, ER with generative memory outper- forms supervised baseline (cl(Ui)tgenmem(D) 57.7). However, notice that replaying queries corresponding to either Do or D1 hurt forward transfer Dj (cI(U, +ggemem((D)))) 8.6) or am- plify forgetting of Do (o((Ui)+gemmm(()) 7.0). These results suggest that the memory module should include (pseudo-)queries corresponding to old and new documents. From Figure 4, we see that continual indexing method cl(U,)) has downward trend for LA,, (Hits @ 10), therefore, eventually for- getting the retrieval task. On the other hand, ER with generative memory is relatively constant, pro- viding evidence against forgetting. In summary,
with generative memory enhances retrieval task performance by reducing forgetting of indexed doc- uments and enabling forward transfer to newly indexed documents.
Does generative memory generalize to differ- ent datasets? In Table 3, for the MS MARCO dataset, we report Hits @ to be 78.2 after training on Do passages. We see that continually index- ing both Do and D1 corpora (cl(U1) 76.5 and cl(U)) +ggemmm(())) - 73.7), significantly reduce forgetting the retrieval task (Hits 1) over just in- dexing the new corpora Dj (cI(Di) 68.0). Next, we look at the retrieval task performance (Hits@) 1) for Dj after incrementally indexing D1. We that without any fine-tuning on the retrieval task for D1, incremental learning with indexing ob- jective shows impressive forward transfer (cI(D;) 36.1 and cl(U1) 35.3). Moreover, ER with generative memory, cl(U)) +genmmme)))) 80.6. performs far superior to just incremental index- ing objective. Similar to the results with the NQ dataset, show that ER with generative memory, cI(U,, (genmem(U)) improves the overall perfor- mance for the retrieval task, reducing forgetting of previously indexed documents and enables forward transfer to new documents compared to continual indexing of all documents, cl(U,). We show that

---

our results hold across two datasets, thus, show- casing the generalizability of our approach.
be regarded as a competitive baseline rather than an inherent upper bound. This also the reason behind reporting the learning accuracy (LA,) for every metric, which can be seen as an upper bound since it maintains a running average of the best per- formance across all corpora. Furthermore, one of the key objectives of continual learning is to lever- age prior knowledge to enhance the learning of new tasks. Indeed, from Tables and 3, observe that our proposed method excels in forward transfer compared to the "train from scratch" approach.
Investigating the effectiveness of the generative memory with the scale of corpus. We con- duct experiments with full MS MARCO dataset (re 8.9.M passages). We construct two corpora Do = 8M and D1 - 841, 823 passages. We train the DSI model using Do passages and incre- mental add D1 passages. In Table 3, we report results for MS MARCO. We see that continual fine- tuning with the indexing task on D1, cl(D1), com- pletely forgets the retrieval task for Do passages (Hits@I goes 0.1 from 16.3). However, the generative memory-based approach significantly reduces forgetting @Hits@I of 7.3). Moreover, gen- crative memory enables continual semi supe-suue learning by augmenting useudo-queries for D, pas- sages, thereby improving forward transfer (Hits@) I of 31.6 vs. 18.2 for cl(D))). Our proposed solution reduces forgetting in large corpus setlings.
For the NQ dataset, indexing the initial Dg cor- pus of 50K documents requires 350K training steps. If we sequentially index additional D1 to Ds cor- pora (10K each) by re-training the DSI model each time, it would require around 1.75M steps. In con- trast, our approach only requires slightly above 300K additional updates to incrementally index all corpora, which is approximately six times fewer updates. Our approach achieves superior overall performance compared to re-training from scratch, while also being more computationally efficient.
Investigating sparsity of experience replay (ER) on forgetting. ER with generative memory co- trains the indexing and pseudo-labeled retrieval tasks. Tay et al. (2022) introduces a mixing ratio to define the ratio of indexing to retrieval sam- ples. The mixing ratio is inversely related to the sparsity of ER, i.e., higher r (more indexing sam- ples) corresponds to sparse updates from pscudo- labeled retrieval samples. Following (Tay et al., 2022), we consider r = {2,32} for our analysis. From Figure 4, we see that slightly outperforms r = 2 in terms of average per- formance, forgetting, and learning accuracy. These results suggest that even sparse regularization up- dates from ER positively inftuence backward and forward transfer in DSI++.
Conclusion
DSI++ introduces new approach to address cru- cial requirement of DSI models for practical use in production setups, where continuous addition of new documents the corpus is necessary. Through experiments, we demonstrate the effectiveness of our proposed solutions: sharpness-aware minimiza- tion and generative memory, which significantly reduce catastrophic forgetting. This work estab- lishes foundation for further research, benefiting both DSI models and the broader community of continual (semi-supervised) learning.
32 (sparse replay)
Limitations
Analyzing index construction time for DSI++ DSI involves training a Transformer model for in- dex construction. DSI++ allows incremental up- dating of the indexer. In Figures 4. 7, and 8, we demonstrate that our incremental indexer updating method surpasses the "train from scratch" baseline in terms of An Note that the "train from scratch" baseline can serve as a performance upper bound for continual learning when there is no detrimental interference among tasks, and all tasks are evenly balanced. However, in the case of DSI++, there exists an initial base corpus that larger than sub- sequent corpora, leading to an imbalance among tasks. Consequently, "train from scratch" should
In this study, we explore the phenomenon of forget- ting in relation the addition of new and distinct documents into the indexer. It is important to note that when a new document refutes or modifies a previously indexed document, the model's behavior becomes unpredictable, requiring further analysis. Additionally, we examine the effectiveness of our proposed method on a larger dataset, such as the full MS MARCO dataset. However, it is worth noting that with this larger dataset, the method ex- hibits significant forgetting, As result, additional research is necessary to enhance the model's per- formance, particularly when dealing with datasets of larger scales.

---

Ethics Statement
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint urXiv:2202.0514.
Training large models is expensive and can have a detrimental impact on the environment (Strubell et al., 2019). Continual learning on top of existing models is preferable to re-training from scratch in this regard since it requires many fewer training steps. With DSI++, we aim to reduce the need to re-train DSI models from scratch whenever new set of documents is added to the corpus thereby making it cheaper and better for the environment. Concretely, in $5.2, analyze the index construc- tion time for DSI++ and show that our approach is computationally efficient in comparison to re- training the model from scratch. At the same time, we acknowledge that reduced cost can increase overall consumption (Jevons' paradox).
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, lean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International Conference on Ma- chine Learning, pages 2206-22444 PMLR.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elho- seiny, Thalaiyasingam Ajanthan, Punect K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. 2019, On tiny episodic memories in continual learning. arXiv preprint arXiv: 1902. 10486.
Acknowledgements
We thank the anonymous reviewers for their valu- able feedback and suggestions, which helped im- prove the paper. We also thank Ronak Pradeep and Kai Hui for help with the MS MARCO setup, Tal Schuster and Raghuram Mandyam Annasamy for reviewing the paper, and William W. Cohen, Aditya Gupta, Dara Bahri, and Fuzhao Xue for sharing insights and intuitions during initial discus- sions. We would like to thank COMEDY (COhorts of Maarten Sap, Emma Strubell, Daniel Fried, and Yonatan Bisk) lab members for reviewing the paper and providing valuable comments; Jeremiah Mil- bauer, Clara Na, Jared Fernandez, Nupoor Gandhi, Zhisong Zhang, and Vijay Viswanathan also gave constructive feedback on drafts and tables.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491- 6506.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning sur- vey: Defying forgetting in classification tasks. IEFF transactions on Pattern Analysis and Machine Intel- ligence, 44(7):3366-338
Cyprien de Masson D' Autume, Sebastian Ruder, Ling- peng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), page 4171-4186.
References
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona
Diab, and Marjan Ghazvininejad. 2022. A review on language models knowledge bases. arXiv preprint
arXiv:2204.060:1
Dara Bahri, Hossein Mobahi, and Yi Tay. 2022. Sharpness-aware minimization improves language model generalization. Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 7360- 7371.
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language mod- as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257- 273.
Samy Bengio, Krzysztof Dembczynski, Thorsten Joachims, Marius Kloft, and Manik Varma. 2019. Extreme classification (dagstuhl seminar 18291). In Dagstuhl Reports, volume 8. Schloss Dagstuhl- Leibniz-Zentrum für Informatik.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021. Sharpness-aware mini- mization for efficiently improving generalization. In International Conference on Learning Representa- tions.

---

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. REALM: Retrieval augmented language model pre-training. In Inter- national Conference on Machine Learning, pages 3929-3938. PMLR.
Linguistics (Volume 1: Long Papers), pages 4205- 4219. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in GPT. In Advances in Neural Information Processing Systems, volume 35.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- afion for Computational Linguistics: Main Volume, pages 874-880.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pas- canu, and Hassan Ghasemzadch. 2020, Understand- ing the role of training regimes in continual learning, Advances in Neural Information Processing Systems, 33:7308-7320.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham know what language models know. Transactions of the Association for Computational Linguistics, 8:423-438.
Neubig. 2020. How
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. In International Conference on Learning Representations.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis. Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwin Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-355
Jianmo Ni, Gustavo Hernandez. Abrego, Noah Constant, Ji Ma. Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-15: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 1864-1874.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: benchmark for question answering research. Trunsactions of the Association for Computational Linguistics, 7:453- 466.
German I Parisi, Ronald Kemker, Jose I. Part, Christo- pher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 2463-2473.
James McClelland, Bruce L McNaughton, and Ran- dall C O' Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connec- tionist models of learning and memory. Psychologi- cal Review, 102(3):419.
Ronak Pradeop. Kai Hui, Jai Gupta, Adam D Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q Tran. 2023. How does generative retrieval scale to millions of passages? arXiv preprint 118.
Michael McCloskey and Neal J Cohen. 1989 Catas- trophic interference in connectionist networks: The sequential learning problem. In Psychology of Leam- ing and Motivation, volume 24, pages 109-165. EI- sevier.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5555
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2023. An empirical investiga- tion of the role pre-training in lifelong learning. Journal of Machine Learning Research, 24(214):1 50.
Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, and Emma Strubell. 2022. Improving compositional generalization with self-training for data-to-text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. iCaRL.: In- cremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2001-2010.

---

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyzz, Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsyyashchenko, Aakanksha Chowdh- ery, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cia. Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Bren- nan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scal- ing up models and data with t5x and seqio. arXiv preprint 221iv2203..71899
Sebastian Thrun. 1995. Is learning the n-th thing any easier than learning the first? Advances in Neural Information Processing Systems, 8.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. 2019. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representa- tions.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Process- ing Systems, 30.
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. 2020. Efficient meta lifelong- learning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 535-548.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories transformer models. arXiv preprint arXi::2012.00366
Noam Shazeer and Mitchell Stern. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596 4604. PMLR.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei. Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206.1012
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. Advances Neural Information Processing Systems, 30.
A Appendix
Shagun Sodhani. Mojtaba Faramarzi, Sanket Vaib- hav Mehta, Pranshu Malviya, Mohamed Abdel- salam, Janarthanan Janarthanan, and Sarath Chan- dar. 2022. An introduction lifelong supervised learning. arXiv preprint (rXiiv2207.04335.
A.1 Implementation Details
We utilize the pre-trained T5-Base (Raffel et al., 2020) to initialize all models and randomly ini- tialize the additional parameters for atomic docid tokens. Bahri et al. (2022) demonstrates the suc- cessful applicability of SAM for language model generalization, especially in pre-trained T5 models. We mainly follow (Bahri et al., 2022) to set our hyper-parametent p 0.15, batch size=32 for the inner maximization step in SAM.
Pablo Sprechmann, Siddhant Jayakumar, Jack Rae, Alexander Pritzel, Adria Puigdomenech Badia, Be- nigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. 2018. Memory-based parameter adaptation. In International Conference on Learning Representations.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
While indexing Do corpus, we train all the mod- els for a maximum of 1M steps with a warmup of 100K steps. During continual indexing of other cor- pora, we train for a maximum of 100K steps with a warmup of 100 steps. For the rest of the hyper- parameters, we follow (Tay et al., 2022) set a learning rate to 0.001, batch size to 128, and input sequence length to 32. We evaluate models after ev- ery 5K steps and retain the checkpoint yielding the best performance. For the initial training with Do corpus, we co-train on indexing and retrieval tasks; therefore, we use the average of all DSI metrics (in- dexing accuracy, Hits@1 1, and Hits @ 10) for model
Pan-Keng Sun, Cheng-Har Ho, and Hung-Yi Yi Lee. 2020. LAMAL: LAnguage modeling is all you need for life- long language learning. In International Conference on Learning Representations.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as a differentiable search index. In Advances in Neural Information Processing Systems, volume 35.

---

strates that pre-trained T5 (Raffel al., 2020) mod- els can be employed answer open-domain ques- tions without access to any external knowledge or context. However, unlike structured KBs, it is non-trivial to update knowledge stored implicitly in the weights of these models. Therefore, Zhu et al. (2020) introduces experimentation setup where the task is to update facts stored within the pre-trained models and proposes constrained opti- mization method, similar Elastic Weight Consol- idation (Kirkpatrick al., 2017), to alleviate catas- trophic forgetting. With similar motivation, (Dhin- gra et al., 2022) introduces a diagnostic dataset to probe LMs for facts that change over time. It also suggests jointly modeling text with its times- tamp for improved memorization of seen facts. Re- cent works have been investigating efficient ways to localize and edit facts stored with the LMs (AlKhamissi et al., 2022) using finetuning (Zhu et al., 2020; Dhingra et al., 2022), hyper-networks (De Cao et al., 2021; Mitchell et al., 2022), and direct editing (Meng et al., 2022). Although a crucial line of work around updating facts in the pre-trained LMs, using prompting as our probing mechanism only provides lower bound estimate of the knowledge contained in these models (Jiang et al., 2020). On the other hand, we explicitly fo- cus on the memorization task in DSI++ This task helps us to answer questions related to catastrophic forgetting more convincingly rather than bounded by the mechanism of how we probe these models.
Indexing accuracy
75
70
Optimizer Adafccoo SAM
60
Training step ( x100.000)
Figure 5: Investigating the effectiveness of SAM for alleviating implicit forgetting in the T5-Base model by visualizing indexing accuracy during memorization. We observe serious fluctuations in the indexing accuracy in the case of the Adafactor optimizer, thereby suggesting unstable memorization. SAM leads to relatively stable memorization of documents.
selection. For the continual learning experiments, we have access to only indexing accuracy for all involved corpora, so we use for model selection. To train a parametric model for generative mem- ory, we utilize the retrieval dataset Ro, which cor- responds to the Do corpus. We set the maximum sequence length for document contents 1024, the target length for generated queries 32, batch size to 128, train for a maximum of 100K steps, and use BLUE for model selection. We use beam decoding to generate pseudo-queries. We tune the learning rate amongst {0.001, ...001,0.00 0.0005} and linear warmup amongst (1K, 10K3. For all our experiments, we use the T5X (Roberts al., 2022) framework along with 4-8 TPUv4 chips train the models.
approaches for continual learning encode the necessary inductive biases re- quired to enable continual learning by modifying the training dynamics. Flatter minima shown to alleviate forgetting (Mirzadeh et al., 2020). Further, Mchta et al. (2023) showed that explicitly optimiz- ing for flatter loss basins using Sharpness-Awar Aware Minimization (SAM; Foret et al. (2021)) reduces forgetting. Building on these works, we show that flatter minima induced by SAM reduce implicit forgetting during memorization, thereby leading to more stable memorization (see $3).
Optimization-
A.2 Related Work
We review relevant prior work along two dimen- sions:
Application setups related to DSI++ and continual learning methods to alleviate forgetting and enable forward transfer.
Language models (LMs) as knowledge bases (KBs). Petroni et al. (2019) shows that pre- trained BERT (Devlin et al., 2019) models cap- ture relational knowledge comparable to that of the KBs constructed using off-the-shelf techniques. Concretely, these models can be used to extract factual knowledge about relations between entities by providing prompt to predict missing words in a cloze-style template (c.g., "New Delhi the cap- ital _") Similarly, Roberts et al. (2020) demon-
Memory-based (aka data-based regularization) approaches for continual learning constrain the parameter updates based the previous task ex- amples sampled from memory. Sparse experience replay using episodic memory (Chaudhry et al., 2019) 9) is a prominent approach, and in $4, we dis- cuss its limitations of it for DSI++ Next, Shin et al.

---

Dataset
Natural Questions (NQ)
MS MARCO
#Train
#Validation
#Test
#Train
#Validation
#Test
Rs
50K
53.8K
13.5K
3.9K
2M
25.0K
3.6K
Ri
10K
10.7K
2.7K
809
400K
5.1K
762
R2
10K
10.6K
2.7K
400K
5.1K
770
Ra
10K
10.7K
2.7K
727
400K
4.9K
734
Ra
10K
10.9K
2.7K
772
400K
4.9K
730
Rs
10K
10.7K
2.7K
847
400K
4.9K
660
Table 2: DSI++ dataset statistics for NQ and MS MARCO: memorization and retrieval tasks.
Added corpus
Method
Eval corpus = Do (Catastrophic forgetting) Hits@1 Hits
Eval corpus Di
(Forward transfer)
Index acc.
6i 10 Index acc.
Hits
MS MARCO - Dol = 50K. [Di] 10K
99.4o.2
78.20.2
95.0o.1
c)(D))
16.718.6
68.02.0
87.31.s
99.8g.o
36.1a.5
65.8s.9
c((U1)
99.4p.0
76.50.7
94.20.s
99.8g.0
35.3a.1
64.43.3
D)
cl( ((ge)teeem)(()
99.3p.
73.70.2
93.9p.s
99.8g.o
80.61.0
95.5g.1
train from scratch
99.5p.o
75.00.2
93.9p.
99.6g.o
73.41.
93.4a.a
MS MARCO (fill) -Dol
8.M
Di]
842K
Do
99.4
16.3
46.8
clD:)) cl(U)) (gnnneen)))((U)
0.6
97.9
40.5
D1
20.4
31.3
86.6
31.6
65.8
Table 3: Comparing performance on incremental indexing of D2 corpus across different methods - cl(D1): continue fine-tuning with indexing task on D1. cl(U1): continue fine-tuning on updated corpus U1, cl(U)) (((lm))ge)mm((() continual indexing of U1 along with ER of pseudo-queries for D). We observe that continual indexing the updated corpus cl(U,) reduces forgetting compared to just indexing new corpus cl(D,) in the MS MARCO dataset. Our proposed approach of augmenting pseudo-queries for all documents along with continual indexing, cl(U, )ige))eeeemm)) (U1), alleviates forgetting of Do corpus and improves forward transfer to Dj corpus. We also show that our proposed solution reduces forgetting of Dn(= 8.)) passages while incremental indexing in a large corpus setting, MS MARCO (full) containing 8.9M passages.
Average performance (A,)
Forgetting (F,)
Learning performance (LA,)
Hits@00
60
60
40
40
20
20
20
B22051
02
D2 Dr
D2
Figure Systematic study about forgetting and forward transfer when incrementally indexing new corpus of documents across different model sizes (T5-Base, T5-Large, TS-XL) and docid representations. We use atomic docids by default and denote (N)((S) for naively/semantically structured string docids. indicates higher is better. J. indicates lower better. We observe that by increasing the model scale, the average A, and learning IAA, performance improves. However, forgetting F, is severe across all model scales. Moreover, string docids (N) underperform atomic docids across the Hits@ metric. Similar to Figure 2, imbuing the docid space with semantic (S) structure alleviates the forgetting compared to arbitrary/ naive (N) structure.
observe that naive
(2017); Sun et al. (2020) learns a parametric model to reconstruct the examples for seen tasks. How- ever, in DSI++, we do not see queries for the new documents. Therefore, use parametric mem- ory generate pseudo-queries for already indexed
(older) documents and an incoming batch of new documents, thus, enabling us to leverage unlabeled data (in the form of new documents) for continual semi-supervised learning. On the other hand, Sun et al. (2020) assumes that the incoming data are

---

Average performance (A,)
Forgetting (F,)
dearning performance (LA,)
60
40
40
Hits@1
20
20
20
d(U)))
olUn--genmmmjuu [V,l with r=2
from
Do
D1 D2 D3 D4
Dg D₂
D2
Dy D2
Dg D.
D2 D3 D4 Ds
Training corpus
Training corpus
Training corpus
Figure 7: Investigating the effectiveness of generative memory in mitigating forgetting when continuously indexing new corpus D,, (T5-Base model and atomic docids representation) for the NQ dataset. indicates higher better, + indicates lower is better. We observe that continual indexing of old and new documents cl(U,,) helps alleviate forgetting of older documents when evaluated on retrieval tasks. However, average Hits (A,,) still undergo 19 points drop after sequential updates (Du + D1 ...... + Ds). We observe that augmenting generative memory during continual indexing not only reduces the forgetting (F,) but also improves average Hits (A,)) by +17.3% over continual indexing.
Average performance (A,)
Forgetting (F,)
Learning performance (LAn).
Indexing accuracy
100
100
100
50
50
d(U))
d(Vu)-genmemluy with r=1
from db
0
0
D1
0
D1
Dg
D1
D4
Dr
Dg
Dr
75
75
75
His81
50
50
50
25
0
0
Do
D1
02
D3
D4
Dg
01
D2
D3
D2
D
D2
0
D4
D5
100
100
1000
Hits@10
50
50
Do
D1
02
D3
D4
Dg
01
D2
D3
D2
D5
D
D2
Dg D4
Dg
Training corpus
Training corpus
Training corpus
Figure 8: Investigating the effectiveness of generative memory in mitigating forgetting when continuously indexing new corpus D,, (T5-Base model and atomic docids representation) for the MS MARCO dataset. 1 indicates higher is better, I. indicates lower better. We observe that continual indexing old and new documents cl(U,, helps to alleviate forgetting of older documents when evaluated on retrieval tasks. However, average Hits 10 (A,) still undergo 25.0 points drop after sequential updates (D, + D; ... + D3). Generative memory enables sparse replaying of pseudo-queries for old documents and continual semi-supervised learning with documents. We observe that augmenting generative memory during continual indexing not only reduces the forgetting (F,,) but also improves average Hits@ (A.) by +33.0% over considered baselines.
fully labeled, which is not applicable in DSI++ (we do not get to queries for the new documents). Furthermore, Sun al. (2020) shows that using parametric model underperforms episodic mem-
ory. In our work, we do not generate example pairs (x, y) but rather generate pseudo-querics (y), sim- ilar contemporary works (Zhuang al., 2022; Bonifacio al., 2022). We show that our approach

---

outperforms episodic memory. Lastly, in the con- text of pseudo-query generation, neural models are prone hallucinate additional content not sup- ported by the input documents. Future works can study methods to filter out noisy pseudo-queries (Mehta et al., 2022) during incremental indexing.
the prediction time, these methods typically require task identity activate the corresponding subset of parameters for inference. the DSI paradigm, we are given user queries at the inference time, and the goal is to predict relevant document iden- tifiers. Now during incremental indexing, if we consider every new document corpus new task, then typical parameter isolation-ban based approach would require corpus identity for every user query at the test time, defeating the whole purpose of the DSI paradigm. Due to this, the parameter isolation- based approaches in their current form are rendered less useful for DSI++. Nevertheless, we believe that by masking the weights for the already indexed corpus, one is explicitly disabling the updates to the underlying DSI model; therefore, parameter isolation- methods would be robust to for- getting, and future works should explore them for DSI++ We believe, however, that adapting these methods for DSI++ is out of scope for this paper, and we would not be able to do both this topic and our current work justice the limited space available.
Test time adaptation approaches for contin- ual learning use episodic memory at the inference time to alter the model weights before making pre- dictions (Rebuffi et al., 2017; Sprechmann al., 2018; de Masson D' Autume et al., 2019; Wang al., 2020). Updating the DSI indexer for ev- ery user query is computationally expensive, so we focus on continual learning methods during training. Apart from continual learning-focused approaches, retrieval augmented generation (Guu et al., 2020; Izacard and Grave, 2021; Borgeaud et al., 2022) family of approaches retrieve auxil- iary passages/documents to enhance pre-trained language models. These approaches alter test-time predictions of the generative models by augmenting their input with relevant passages retrieved from ex- ternal retrievable memory. Moreover, one explicitly disables the updates to the employed pre-trained (and retrieval) model using the external retrievable memory. Such approaches do not faithfully assess the fundamental challenge of learning continually, specifically catastrophic forgetting. On the other hand, our work focuses on the recently introduced DSI paradigm (Tay et al., 2022), where information in the document corpus is encoded into the model parameters. Therefore, any updates to the under- lying corpus necessitate updating the model pa- rameters hence, undergoing severe forgetting. Our work tackles a more challenging setup for study- ing the forgetting phenomenon in detail. However, retrieval-augmeeneed generation-basee methods do not analyze the forgetting phenomenon, only look- ing at overall performance metrics. We agree that continual learning is broader than catastrophic for- getting. However, in this work, we decided to study the forgetting phenomenon in detail on one of the most challenging setups, if not the most difficult.
Parameter colation-based approaches for con- tinual learning assign different dedicated subsets of the model parameters to each task to prevent forgetting (De Lange et al., 2021). While learning a new task, these methods either freeze a subset the parameters corresponding to older tasks or dynamically add new parameters per new task. At


Reviews:
[{"review_id": "ZGP6yjXMp6", "reviewer": "Reviewer_fVVm", "summary": "", "questions": "(A) The DSI paper [Tay et al.] utilizes three different kinds of NQ datasets, NQ10k, NQ100k and NQ320k. The author only examine their approach with NQ100k. Will the model fail to mitigate forgetting if dataset scale becomes larger?\n\n(B) In standard lifelong learning paradigm, the model should not access previous datasets. In prior work about Episodic Replay (e.g., https://aclanthology.org/2020.emnlp-main.39.pdf), it is common to keep a tiny subset (1%) of previous datasets. However, the authors claim they use equal proportion of old and new corpora to obtain main results (of $cl(U_i)$). It is unclear whether SAM and ER or the high enough proportion of old examples play an role to reduce forgetting.\n\n(C) The model trained with multi-task paradigm can usually be the upper bound of lifelong learning approaches based on Episodic Replay. However, the proposed approach in this work surpasses *from-scratch approach* but authors does not provide enough analysis about what content causes such forward-transfer.\n\n(D) The model scale does affect the model's intrinsic ability to avoid forgetting. More results of larger models (like T5-XXL) are welcome.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and remarks on our work. We are pleased that the reviewer finds the paper well-written, coherent, and comprehensive and appreciates our efforts in continually updating DSI.\n\n**[Novelty]**\n\nWe list our contributions:\n\n1. We **propose DSI++**, a continual learning challenge for the recently proposed Differentiable Search Indices (DSI) paradigm. To enable research progress on DSI++, **we introduce two benchmarks** based on the existing **Natural Questions** and **MS Marco** datasets. To understand the severity of the catastrophic forgetting, **we systematically study different design choices** like – document identifier representation strategies (unstructured atomic, naively structured, and semantically structured) and the pre-trained model size (T5-base, T5-large, T5-xl).\n\n2. We are the **first to show that the DSI model undergoes a forgetting phenomenon during memorization**, leading to unstable learning dynamics. Mehta et al. (2021) show that explicitly optimizing for flatter loss basins using Sharpness-Aware Minimization (SAM) leads to lesser forgetting, especially in task-incremental learning settings where data undergoes a clear distributional shift. We extend this work to the new DSI paradigm and **convincingly demonstrate that SAM helps with the stable memorization of documents. Our results generalize the earlier findings even to the settings where data does not undergo a clear distributional shift (i.e., memorization task)**. Although SAM helps to stably memorize +12% of documents, there is still a lot of room for improvement and our work invites more future work in this direction.\n\n3. We **propose a generative memory-based approach** to alleviate forgetting during continual indexing of the new documents. Note that in DSI++, we assume no access to ground truth queries for the older documents. Furthermore, we do not get to see queries for the new documents. Therefore, **we use generative memory to generate pseudo-queries for already indexed (older) documents as well as an incoming batch of new documents, thus, enabling us to leverage unlabeled data (in the form of new documents) for continual semi-supervised learning**. On the other hand, Sun et al. (2020) **assume that the incoming data are fully labeled, which is not applicable in DSI++** (we do not get to see queries for the new documents). Next, they learn a parametric model to reconstruct the examples for all of the seen tasks. However, such an approach strictly underperforms episodic memory which retains a subset of seen examples. In our work, we do not generate example pairs (document, queries) but rather generate pseudo-queries, and we show that our approach outperforms episodic memory that retains ground truth queries.\n\nWe believe that the above list of contributions would make the paper’s novelty clearer to the reviewer. We have already discussed these contributions throughout our paper. With the extra page in the camera-ready version, we will explicitly enumerate the above list of contributions towards the end of the Introduction section for improved clarity.\n\n**[Question A: Scale of a data corpus]**\n\nTo showcase that our proposed approach generalizes to datasets other than Natural Questions [NQ; Tay et al. (2022)], **we introduce one more benchmark for DSI++, constructed from a publicly available passage retrieval dataset, namely MS Marco** [Nguyen et al. (2016)]. We show that ER with generative memory improves the overall performance of the retrieval task, reducing forgetting of previously indexed documents and enabling forward transfer to newly indexed documents. We show that our results hold across two datasets – Natural Questions and MS MARCO, thus, showcasing the generalizability of our approach.\n\nTo investigate the effectiveness of the generative memory with the scale of a corpus, we conducted experiments with a full MS MARCO passage retrieval dataset (≈ 8.9M passages). We construct two corpora – D$_0$ = 8M and D$_1$ = 841, 823 passages. We train the DSI model using D$_0$ passages and incrementally add D$_1$ passages. In Table 3 (in Appendix), we report results for MS MARCO. Overall, we show that our proposed solution convincingly reduces forgetting in large corpus settings. **In summary, we experiment with MS Marco 8.8M passages as opposed to NQ 320k (approx. 28 times larger corpus)**. We believe that the reviewer overlooked our detailed analysis of how our method interacts with the scale of a corpus, which we covered in Section 5.2 of our paper (refer to lines 584-602).\n\n**[Question B: Equal proportion of the old and new corpora]**\n\nNote that both the memorization and retrieval tasks return a document identifier corresponding to the input. So the DSI applications have access to the document contents for subsequent use. Therefore, by problem setup, we have access to the contents of the previous documents and one can continue indexing old and new documents with the hope of answering queries (retrieval task) for all seen documents. So we consider cl(U$_i$) as one of the baselines for our experimentation, where U$_i$ is all documents seen until the $i^{th}$ step. To update the DSI model with the indexing objective, **in the T5 tasks mixture**, we consider an equal proportion of the old and new documents. The hypothesis is that the model has already memorized the old documents, so aggressively memorizing the new documents 50% of the time and replaying older documents 50% of the time. **We want to clarify that the equal proportion relates to replay frequency, not a small subset of previous documents, as misinterpreted by the reviewer. We will revise the method description for clarity and to prevent future confusion**. \n\nAlso, for the cl(U$_i$) baseline, we do not retain any queries corresponding to the older documents (i.e., 0% of previous queries). So there is no episodic replay for the retrieval task and as visualized in Figures 4, 7, and 8 (Tables 1, 3), cl(U$_i$) undergoes forgetting of the retrieval task.  On the other hand, from Figures 4, 7, and 8, we see **our proposed method [cl(U$_5$)+genmem(U$_5$)] outperforms cl(U$_5$) both in terms of average performance A$_n$ and forgetting F$_n$ over five sequential updates. These results convincingly show that the SAM and ER with generative memory significantly alleviate forgetting the retrieval task compared to considered baselines**.\n\n**[Question C: train-from-scratch baseline]**\n\nMulti-task learning, referred to as “train-from-scratch”, can serve as a performance upper-bound for lifelong learning when there is no detrimental interference among tasks, and all tasks are evenly balanced. However, in the case of DSI++, there exists an initial base corpus that is larger than subsequent corpora, leading to an imbalance among tasks. Consequently, **training from scratch should be regarded as a competitive baseline rather than an inherent upper bound**. This is also the reason we report the learning accuracy for every metric, which can be seen as an upper bound since it maintains a running average of the best performance across all corpora.\n\nFurthermore, one of the key objectives of lifelong learning systems is to leverage prior knowledge to enhance the learning of new tasks. Indeed, we observe that our proposed method excels in forward transfer compared to the train-from-scratch approach. For detailed results, please refer to Tables 1 and 3, specifically in the evaluation corpus D1 (right half of the table), and compare the rows labeled cl(U$_1$)+genmem(U$_1$) with those representing the train-from-scratch approach.\n\n**[Question D: Model scale]**\n\nIn Section 2.4, we systematically study how the DSI model scale affects forgetting. (see Q5 — lines 242-243 and “Model scale.” paragraph — lines 289-299). \nFrom Figure 2, we observe that larger models outperform their smaller counterparts in terms of the average performance A$_n$ and the learning performance LA$_n$ (T5-xl > T5-large > T5-base). However, **empirically we report that forgetting F$_n$ is severe across all model scales, without any clear best performer**, and therefore given computational constraints, we focus on T5-base for the rest of our experimentation. \n\n**We express our gratitude to the reviewer for their thorough review and insightful questions. Please acknowledge if our responses adequately address your queries. We are available to provide further clarifications and eagerly await updates to the initial ratings.**\n\n[References]\n\n[Tay et al. (2022)] Tay, Yi, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, et al. \"Transformer memory as a differentiable search index.\" Advances in Neural Information Processing Systems 35 (2022): 21831-21843. \n\n[Mehta et al. (2021)] Mehta, Sanket Vaibhav, Darshan Patil, Sarath Chandar, and Emma Strubell. \"An empirical investigation of the role of pre-training in lifelong learning.\" arXiv preprint arXiv:2112.09153 (2021).\n\n[Sun et al. (2020)] Sun, Fan-Keng, Cheng-Hao Ho, and Hung-Yi Lee. \"LAMOL: LAnguage MOdeling for Lifelong Language Learning.\" In International Conference on Learning Representations. 2020.\n\n[Nguyen et al. (2016)] Nguyen, Tri, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. \"MS MARCO: A human generated machine reading comprehension dataset.\" In CoCo@ NIPs. 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Lifelong learning is a practical scenario and it is important to reduce catastrophic forgetting in lifelong learning. This work proves SAM and ER can help to reduce forgetting of indexing models.\n2. The authors propose to split forgetting into two parts, implicit and explicit forgetting. And they empirically analyze the causes of both kinds of forgetting and discuss the approaches to mitigate them.\n3. This paper is well organized and providing experimental evidence to support the effectiveness of DSI++.", "reasons_to_reject": "1. This paper is not novel enough, since SAM and (generative) episodic replay have been proven to be effective to reduce catastrophic forgetting in many NLP tasks.\n2. The authors examine the proposed approach within limited datasets and model architecture. As an improvement of DSI, the employed datasets in this work are not **the same** as DSI paper [Tay et al.]. The authors should explain why to modify dataset scale (See more in Question A).\n3. There are unclear parts of experiment settings, which can lead to misleading or even wrong conclusions (See Question B-D).", "questions_to_authors": "(A) The DSI paper [Tay et al.] utilizes three different kinds of NQ datasets, NQ10k, NQ100k and NQ320k. The author only examine their approach with NQ100k. Will the model fail to mitigate forgetting if dataset scale becomes larger?\n\n(B) In standard lifelong learning paradigm, the model should not access previous datasets. In prior work about Episodic Replay (e.g., https://aclanthology.org/2020.emnlp-main.39.pdf), it is common to keep a tiny subset (1%) of previous datasets. However, the authors claim they use equal proportion of old and new corpora to obtain main results (of $cl(U_i)$). It is unclear whether SAM and ER or the high enough proportion of old examples play an role to reduce forgetting.\n\n(C) The model trained with multi-task paradigm can usually be the upper bound of lifelong learning approaches based on Episodic Replay. However, the proposed approach in this work surpasses *from-scratch approach* but authors does not provide enough analysis about what content causes such forward-transfer.\n\n(D) The model scale does affect the model's intrinsic ability to avoid forgetting. More results of larger models (like T5-XXL) are welcome.", "paper_topic_and_main_contributions": "The authors propose an approach based on Differentiable Search Indices (DSI) for lifelong learning paradigm, enhancing it with Sharpness-Aware Minimization and generative memory replay. Experimental results show the proposed approach can bring significant improvement of Hit@1/10 and index accuracy on previously learnt corpus, when the models are trained with a sequence of corpus one by one.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XP3Spuu6a7", "reviewer": "Reviewer_mZs9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer finds: (i) our paper to be **well-written and clear in terms of the presentation**, (ii) our work is the **first to study the vital problem of updating DSI** and appreciates our efforts in this direction, and (iii) **our proposed benchmarks are valuable** to the community.\n\n**[Additional architectures]**\n\nOur primary focus in this work is to examine the performance of DSI models when continually indexing new documents. Additionally, we investigate how the scale of the model, whether it's T5-base, T5-large, or T5-xl, impacts the phenomenon of forgetting during continual indexing. **While exploring the susceptibility of DSI-BART under continual indexing is an interesting question, we believe this aspect is tangential to our current study and leave it for future investigations, following previous work in this study in terms of model choice**.\n\nThe original study by Tay et al. (2022), and subsequent research by Wang et al. (2022), employ a Transformer architecture for sequence-to-sequence learning. They initialize this Transformer architecture using pre-trained T5 checkpoints before memorizing or indexing documents. There has also been recent research by Bevilacqua et al. (2022), which uses a pre-trained BART checkpoint for initializing DSI and reported results comparable to the original DSI study. Therefore, **given considerations of computational cost, we adhere to the original work and initialize our DSI model with pre-trained T5 checkpoints**.\n\n\n**[Additional baselines]**\n\nThere are several families of approaches for continual learning – **regularization-based, memory-based, optimization-based, and parameter-isolation-based approaches** (Sodhani et al. (2022)). In the related work section of our paper (Appendix B, lines 985-1140), we discuss most of these approaches in detail pointing out relevant baselines for DSI++ setup and comparing with them in Tables 1 and 3. The reviewer suggests comparing with the regularization-based approaches, like Elastic Weight Consolidation (EWC) and Learning without Forgetting (LwF). Regularization-based approaches alleviate forgetting by regularizing the objective such that it either penalizes the feature drift on already learned tasks (LwF) or discourages change in parameters that were important to solve previous tasks (EWC). However, these approaches are shown to underperform the vanilla episodic replay-based approaches on widely used continual learning benchmarks (Prabhu et al. (2020); Hussain et al. (2021)). Therefore, we compare our proposed approach with episodic memory (epsmem). Moreover, **in the context of DSI++, LwF would *require* queries corresponding to new documents to prevent forgetting the old documents**. However, according to the DSI++ problem setup (section 2.1, lines 138-170), **we do not have any queries related to new documents, thereby rendering the LwF method not applicable**.\n\n**We thank the reviewer for their valuable time in reviewing our paper. Should there be any remaining concerns, we are readily available to address them, and we sincerely hope that the reviewer might consider revising the scores accordingly.**\n\n[References]\n\n[Tay et. al (2022)] Tay, Yi, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin et al. \"Transformer memory as a differentiable search index.\" Advances in Neural Information Processing Systems 35 (2022): 21831-21843. \n\n[Wang et. al (2022)] Wang, Yujing, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia et al. \"A neural corpus indexer for document retrieval.\" Advances in Neural Information Processing Systems 35 (2022): 25600-25614.\n\n[Bevilacqua et. al (2022)] Bevilacqua, Michele, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. \"Autoregressive search engines: Generating substrings as document identifiers.\" Advances in Neural Information Processing Systems 35 (2022): 31668-31683.\n\n[Sodhani et. al (2022)] Sodhani, Shagun, Mojtaba Faramarzi, Sanket Vaibhav Meha, Pranshu Malviya, Mohamed Abdelsalam, Janarthanan Janarthanan, and Sarath Chandar. \"An introduction to lifelong supervised learning.\" arXiv preprint arXiv:2207.04354 (2022).\n\n[Prabhu et. al (2020)] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. \"Gdumb: A simple approach that questions our progress in continual learning.\" In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 524-540. Springer International Publishing, 2020.\n\n[Hussain et. al (2021)] Hussain, Aman, Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. \"Towards a robust experimental framework and benchmark for lifelong language learning.\" In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The Differentiable Search Indices is a significant problem to work on. This paper is the first one to study the problem of updating DSI, which is an important feature of serving an LM for IR.\n\n- This paper is well-written and well-presented.\n\n- This work proposes two benchmarks built upon Natural Questions (NQ) and MS MARCO. These benchmarks could serve as good resources to foster relevant research in the community. And the authors promised to release these datasets in the Reproducibility Statement section.", "reasons_to_reject": "- The experiments are only conducted on FLAN-T5 (if I didn't miss any experiments). It would be helpful if the authors introduced more experiments with different architectures to demonstrate the approaches are generalizable.\n\n- The baseline methods seem missing. It would be helpful if the authors could add experiments with other continual learning methods, like LWF (Learning without Forgetting)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces DSI++, a continual learning challenge for Differentiable Search Indexes, along with the dataset for evaluation. The authors analyze the severity of this issue in FLAN-T5 model and locate this issue at memorization. The authors propose two techniques to tackle this problem in the context of continually indexing new documents. They propose a method for implicit forgetting based on Sharpness aware minimization and generate queries for old and new documents to use during continual indexing.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vPpfZH9pwt", "reviewer": "Reviewer_Gobp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and remarks on our work. We are pleased that the reviewer appreciates the **strong motivation, clarity, and quality of writing** in our paper! \n\n**[Implementation details and related work in appendix]**\n\nWe acknowledge the reviewer's concern regarding the inclusion of implementation details and related work in the appendix. However, **we want to emphasize that we have added relevant discussions of related work throughout the main text whenever necessary**. For instance, we delve into optimization-based approaches for continual learning in lines 337-347 and discuss episodic memory-based methods in lines 418-435. Nevertheless, in response to the reviewer's suggestion, we are committed to enhancing the comprehensiveness of our paper. **With the additional page available in the camera-ready version, we will incorporate the details currently found in the appendix into the main text for easier access and reference**.\n\n**[Source code and results]**\n\nIn our reproducibility statement (lines 62-674), we have clearly mentioned that **upon publication, we plan to release our code and dataset splits to reproduce our experiments and actively support ongoing research endeavors related to DSI++**. Also, in the appendix, we will tabulate exact numeric values for plots visualized in Figures 4, 6, 7, and 8.\n\n**We once again thank the reviewer for their positive feedback on our manuscript and look forward to answering any other questions!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivations of some designs are clear. \n2. The paper is detailed in the steps and well written.", "reasons_to_reject": "1. Implementation details and related work should be included in the content rather than the appendix.  \n2. The source code for training the proposed model is not released. It would be useful for ongoing research efforts if the paper could release its source code and predicted results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to extend the existing SDI model which can incrementally index new documents with a lower computational cost.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces DSI++, an approach using continual learning for Differentiable Search Indexes. The authors define a setting in which old and new queries are not available when updating a DSI with a new set of data, and they demonstrate that their proposed approach is able to mitigate the catastrophic forgetting happening on the old data when doing this operation. The reviewers all agree that the paper is well written and it discusses a topic worth of investigation to improve the capability of updating DSIs over time, which is a practical problem in real-world scenarios. The reviewers agree that the methodology is sound and that the provided experiments show its contribution.
Decision: Accept-Main