Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
Authors: Suyu Ge
Chenyan Xiong
Corby Rosset
Arnold Overwijk
Jiawei Han
Paul N. Bennett
Venue: EMNLP 2023 Main (2023)
Keywords: Retrieval Augmented Language Model
Zero-shot Dense Retrieval
Mixture of Memory
TL;DR: We propose a novel retrieval augmented language model to extract knowledge from diverse memory corpora for zero-shot dense retrieval.

Abstract: In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora (external memories), with the option to ''plug in'' unseen memory at inference time.
We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture.
We instantiate the model in a zero-shot dense retrieval setting by augmenting strong T5-based retrievers with MoMA.
With only T5-base, our model obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark, outperforming some systems with larger model sizes.
As a plug-in-play model, our model can efficiently generalize to any unseen corpus, meanwhile achieving comparable or even better performance than methods relying on target-specific pretraining.
Our analysis further illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of augmentation learning, and how MoMA utilizes the plug-in memory at inference time without changing its parameters.
Our code can be found at https://github.com/gesy17/MoMA.

Augmenting Zero-Shot Dense Retrievers with Plug-in Nixture-of-Memories
Suyu Ge Chenyan Xiong Corby Rosset Arnold Overwijk Jiawei Han', Paul Bennett
University of Illinois Urbana-Chane Champpaign 2 Carnegie Mellon University
3 Microsoft Research SSpotify
haijhellnooiiiss idu
cxêcs cmu. edu
[corbyrosset, jiio www.erwwww...meiiooooo.ttoooom http///iccsooottoooooooooooooooo pbennet com
Abstract
Retrieval augmented language models provide a promising alternative. They allow language mod- els to efficiently access vast resources from an ex- ternal corpus (Guu et al., 2020; Borgeaud et al., 2022) that serves as kind of "memory" they can refer to when making predictions, alleviat- ing the need to memorize much information in their network parameters (Roberts et al., 2020). This open-book approach helps language models to better generalize on token prediction tasks and machine translation (Khandelwal et al.. 2019; Borgeaud et al., 2022), and tasks which al- ready involve first-stage retrieval component, e.g., OpenQA (Borgeaud et al., 2022; Izacard et al., 2022). Existing retrieval augmentation methods usually stick to one single retrieval corpus through- out training and inference that the retrieval com- ponent can be indirectly guided by the supervision from end tasks.
In this paper improve the zero-shot general- ization ability of language models via Mixture- Of-Memory Augmentation (MoMA), mech- anism that retrieves augmentation documents from multiple information corpora ("external memories"), with the option to "plug in" un- seen memory at inference time. We develop joint learning mechanism that trains the aug- mentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We in- stantiate the model in zero-shot dense re- trieval setting by augmenting strong TS-based retrievers with MoMA. With only T5-base, our model obtains strong zero-shot retrieval accu- racy on the eighteen tasks included the stan- dard BEIR benchmark, outperforming some systems with larger model sizes. As plug-in- play model, our model can efficiently general- ize any unseen corpus, meanwhile achieving comparable or even better performance than methods relying on target-specific pretraining. Our analysis further illustrates the necessity of augmenting with mixtureoof-memory for ro- bust generalization, the benefits of augmenta- tion learning, and how MoMA utilizes the plug- memory at inference time without chang- ing its parameters. Our code can be found at https:///ithhon wtt./ooo//////ee geamm//eyy7//MMMM.
In this paper improve the zero-shot general- ization ability of language models using "mixture- of-memory" (MoMA), a new retrieval augmenta- tion mechanism. Instcad a single corpus, MoMA retrieves documents from a "mixture" multi- ple external corpora and enjoys the merits of a larger and more comprehensive source of knowl- edge. This mechanism also allows removing and/or "plugging-in in" new corpora during inference time, when more information from the target task re- vealed, or an additional way for users to con- trol the model. Specifically, we apply MoMA on the zero-shot dense retrieval task, which is the foundation of many important real-world applica- tions (Thakur et al., 202 la; Kim, 2022) and also the retrieval component of recent retrieval augmented language models (Guu et al., 2020; Izacard et al., 2022). However, it is not trivial guide trieval model leverage multiple corpora. We need to jointly train the augmentation component and dense retriever using supervised relevance sig- nals and self-mined hard negatives.
1 Introduction
Scaling up language models with more parame- ters and pretraining data data-improved model general- ization ability on downstream applications (Raffel et al., 2019; Brown et al., 2020; Smith et al., 2022), but with diminishing return: linear improvements on downstream metrics often require exponentially more parameters and computing cost (Kaplan et al., 2020; Hoffmann et al., 2022). Hence, scaling pre- trained language models in this way is economi- cally unsustainable (Strubell et al., 2020; Bender et al., 2021; Zhang et al., 2022).
Work partly done while at Microsoft.

---

We instantiate MoMA with TS-base model (Ni al., 2022) and apply to the dense retrieval task (Karpukhin et al., 2020). Our end task re- triever uses a set of augmenting documents from the mixeoresof-memories to enhance its represen- tation of the query with important context; the re- triever then uses the enhanced query representation to retrieve a final candidate set. At inference time, we plug in the target task's corpus to the memory mixture to introduce in-domain context informa- tion, without updating any parameter.
kens from an in-domain corpus as additional input to enhance the semantic representation. Despite their effectiveness, learning to retrieve useful docu- ments to augment the language model is a challeng- ing task, since human annotations on the useful- ness of augmentation documents are costly and sel- dom available. The most straightforward way is to use representations from raw pretrained language models, i.e., as unsupervised dense retrieval (Guu et al., 2020; Borgeaud al., 2022). Adapting ex- isting dense retrieval models is another common choice (Izacard and Grave, 2020b; Lewis et al., 2020; Yu et al., 2021). A more plausible solution is to jointly learn the augmentation components end-to-end using supervision from the final task, for example, treating the augmentation as latent variables and applying EM (Zhao et al., 2021), distilling the augmentation component from feed- back of the final model (Izacard and Grave, 2020a). In a parallel work, Izacard et al. (2022) found the most effective one is attention distillation method (ADist), which trains the augmentation component using soft labels derived from the end model's 's at- tention on augmentation documents.
As a plug-in-play method, MoMA provides an flexible but powerful solution to zero-shot dense re- trieval: Unlike recent state-of-the-as methods (Yu et al., 2022; Neelakantan et al., 2022), it does not require pretraining target corpus or large-scale web corpus, enabling it to generalize arbitrary unseen corpus without additional effort. can also be used as an efficient alternative for recent large language model (LLM) based generative re- trieval models (Gao et al., 2022). Given the target query, MoMA only involves the T5-base model for query encoding, which is significantly cheaper than querying an LLM to generate pseudo answers and re-encoding it.
The motivation for query augmentation coin- cides with the query expansion methods in the traditional IR community, whereby query is aug- mented by new but similar features (Carpineto and Romano, 2012). As feature selection usually quires additional semantic analysis, the efficiency and usability of traditional query expansion meth- ods remain limited when faced with a new domain. To overcome this, recent work relies on dense re- trieval results to expand the query (Yu et al., 2021). The retrieved relevant documents serve as pseudo relevance feedback signals for the model, which are concatenated with the original query as the aug- mented model input. Our work augments queries with feedback from multiple corpora and learns select important augmentation documents automat- ically.
We experimented on eighteen zero-shot dense retrieval tasks included in BEIR (Thakur et al., 2021a), the standard ZeroDR benchmark. The results demonstrate the improved zero-shot abil- ity of MoMA. MoMA achieves comparable or even stronger performance recent state-of-thee -htt---rtt---ttt dense retrieval systems with larger model scales and heavier computation costs. Our further anal- ysis reveals that large and diverse corpora in the memory leads to the best performance; while only using a single corpus during training does not im- prove performance on unseen target tasks. The learning of augmentation component is also impor- tant for MoMA to utilize the diverse information from the mixture. Our analysis and case studics il- lustrate how MoMA leverages the plug-in memory at testing time to enrich its query representations.
2.2 Zero-shot Dense Retrieval
2 Related Work
Dense retrieval models trained on a resource rich source tasks, e.g., web search, usually do not per- form as well when zero-shot transferred to other do- mains (Thakur et al., 2021b). Xin et al. (2021) ana- lyzed the challenge of shifting between training and testing domains, and leveraged Nomain-invarian learning to mitigate the gap. Another common ap- proach is to first generate domain-specine pseudo
2.1 Retrieval Augmentation
Recent research has explored the retrieval- augmented language model, which aims con- struct an external memory for the language model (Khandelwal al., 2019; Zhong et al., 2022; Guu et al., 2020; Borgcaud et al., 2022; Petroni et al., 2020). It retrieves related documents or to-

---

labels for each task, and then use them train dense retriever (Thakur al., 2021b; Wang al., 2022). Additionally, continuous pretraining the language model also improves its generalization ability in Ze (Izacard et al., 2021). Follow- ing works (Izacard et al., 2021; Yu et al., 2022) further contrastively pretrained the retriever on tar- get external corpus with sentence matching loss. One significant drawback of them is requiring the target or external corpus as part of the training corpus, which prohibits the plug-in-play feature when exposed to new data. Besides, stacking all target datasets for model pretraining also increases computation costs to a notable degree. On BEIR benchmarks which contain 18 target tasks, it en- larges the training corpus to 7 times larger.
uses dot product as the scoring function to match the embeddings q and d, which is known sup- port efficient approximate nearest neighbor search (ANN) (Johnson et al., 2019). A pretrained lan- guage model is often the encoder of choice g(). We use the ST5-EncDec variant of Sentence-T5 (Ni et al., 2022):
g(x) Dec(Enc(2)))
(2)
which feeds in the text sequence (prepended by a special [CLS] tokens) to the encoder of T5, Enc(), and uses the output representation of the [CLS] token from the decoder, Dec(). as the text represen- tation. This naturally leverages the attention from decoder to encoder at all Transformer layers (Raffel et al., 2019), fine-grained information gather- ing mechanism.
Other methods seek better generalization abil- ity in ZeroDR from various resources, for exam- ple, combining with sparse retrieval introduce exact match signals (Formal et al., 2021) or us- ing multiple vectors per documents for term-level matching (Khattab and Zaharia, 2020a). More recent work simply changes backbone models to larger language models, such T5-XXL or cpt- text-XL (Ni et al., 2021; Neelakantan al., 2022). Some rely stronger instruction--guied gener- ative language models (Gao et al., 2022), which match documents with -eneraeee query an- swers. Overall, methods relying on large language models will incur heavier costs on memory con- sumption and computation, and calling generative model API may also cause latency issues. Instead of chasing stronger backbone models, our goal in this paper is provide an efficient plug-in-play alternative for them.
The training of dense retrieval systems often applies standard ranking loss and pairs the relevant documents d' E D for each query q with hard negatives d= E D
c= Σ
Σ
(gg.d))))
q at dt d-ED
D ANNC figoo) D
(3)
Eqn. uses ANCE hard negatives, which are the top-retrieved documents from C using the retriever itself (Xiong et al., 2020). The loss function i() can be any standard ranking loss such as cross entropy. A ZeroDR model is trained on q" and documents de C' from source lask, often web search, and tested on targer tasks q' and C²; supervision signals are only present from the source.
The key idea of (document-based) retrieval augmented language models is to enrich the representation g(g) with additional contextual input for the model, i.e., aug- mentation documents d' retrieved from an external memory M. Instead of using single document corpus, MoMA uses multiple corpora to provide richer and more diverse external resources for aug- mentation. For example, M can be composed by the source corpus C a general encyclopedia, a domain specific knowledge graph, etc. Then we can retrieve the augmentation documents D"
Mixture-of-Memory.
3 Method
In this section we first describe our Mixture-of- Memory Augmentation. Then we discuss how is jointly learned with the end system and enables plug-in memory at inference time.
3.1 Mixture-of-Memory Augmentation
Before going to the details of MoMA, we first recap some preliminaries in ZeroDR.
Preliminaries. The dense retrieval (DR) task aims to find relevant documents d from corpus C for the given query & by representing them in shared embedding space. Specifically, the retrieval score in DR is often calculated as:
Da M J"(z,0)i = (C,,..,,,Ch (4)
This augmentation component uses another dense retriever f"() which also adopts the Sentence-T5 architecture. Note that instead of retrieving D
f(q, d) q d:q = g(g): g(d).
(1)

---

[CLS]+
ponent f²). Both have their own independent parameters. The two components are bridged by the augmentation documents, which are retrieved by f*() from M and used by gMoMA() to produce query representation q".
Enc
Dec
(q.0))))
BlOA2Q
Aug
d
Enc
M ()
Plagitn
Medrofis
Attention
Fusing
Aug
dk
Main Model Learning. Given the relevance labels from the source task and an augmentation model, training (M is straightforward. We can use the standard dense retrieval training to fine- tunc the enriched query encoder () and the document encoder g():
Mosro
Mixture
Memory
Figure I: Illustraion of the Mixture-of-Memory Aug- mentation.
separately from M different ANN memory sources and merging results, Eqn. 4 combines them into one ANN index. This requires the augmentation component () to be flexible enough to handle various corpora in the mixture.
C MIMA
Σ 4" adt 'd- dt D*t t D
U( [(fresmr (q , fruMa (p., )):
(7)
Using the encoder-decoder architecture for g() in Eqn. 2 enables a simple extension to incorpo- rate the augmentation documents using the fusion- in-decoder (FiD) mechanism (Izacard and Grave, 2020b):
D ANN (MaMe \ D
(8)
The training signals come from the source task, in- cluding g², its relevant documents D*+, and ANCE hard negatives D retrieved from the source cor-
MoMA (q) = Dec(Ene(g); Enc(d)), Enc(d (R)));
pus C
Augmentation Learning. Training f"() is chal- lenging as it is hard to label whether an augmen- tation document is useful. Propagating gradients from the final loss f²() is also prohibitive as the retrieval operation in Eqn. is discrete. Fortunately, recent research found the attention scores from the FiD decoder to each encoded inputs (Eqn. 5) are good approximations to the usefulness of augmen- tation documents (Izacard and Grave, 2020a):
D = { (df....,dk
(5)
It feeds in the K augmentation documents sepa- rately to the T5 encoder of g(). Then it fuses the encoded documents together with Enc(g) using one decoder that attends to all encoded vectors, as illus- trated in Figure 1.
The FiD approach in Eqn 5 is nice balance of efficiency and capacity when modeling multiple text sequences (Izacard and Grave, 2020b). It is more efficient than concatenating all text pieces to- gether, while also remaining expressive enough to model the nuances from many sequences. (Izacard and Grave, 2020a; Izacard et al., 2022).
FidAut(d)) = AttDer-End MoMA (df)).
pos head
(9)
When instantiating MoMA in the dense retrieval setting, we focus augmenting the query repre- sentation q, as queries are often short, ambiguous, and benefit more from additional contextual infor- mation (Lavrenko and Croft, 2017; Yu et al., 2021). This leads to the following definition of MoMA:
sums the attentions from g MoMA ()'s special to- ken at the decoder's [CLS] position over all layers, input positions, and attention heads. Ideally, higher FidAtt() is assigned to di that provides useful con- textual information.
Previously, FidAtt scores are often used soft labels for the augmentation model (Izacard and Grave, 2020a; Izacard et al., 2022). Doing so with memory mixtures is risky as it is too sparse and overfits memory resource that appears earlier in the training, which are the only ones available for the decoder to attend on. To improve the learning robustness, we introduce ANCE-style hard nega- tive mining to train the augmentation component as well.
MoMA (g. d) - d:
dd MoMA (g),d = g(d),
(6)
using the construction of MoMA() in Eqn. 5 upon the augmentation documents defined in Eqn. 4.
3.2 Joint Learning in MoMA and Inference with Plug In Memory
MoMA has two sets of parameters to learn, in the main model (MaMA() () and the augmentation com-
First, we formulate the positive set of augmenta-

---

tion documents as:
MUC" C which conveys in-domain information. The augmenting corpus can also be engineered by users manually to inject their preference or domain knowledge, e.g., as "memory engineering". In this work we focus on swapping out the source corpus for the target corpus; we leave other explorations for future work.
D²+ = D Top-N
(10)
which combines relevant documents D*+ and the augmenting ones that received N-highest attention scores from () Then we pair them with hard negatives formulate the training of fn() as:
Experimental Methodologies
L - E E
(f d f ));
Datasets. We choose the MS MARCO passage (11) dataset (Bajaj al., 2016) as the source domain dataset, whereas the target domains are from the datasets in BEIR (Thakur et al., 2021b) benchmark, which include including biomedical, scientific and financial texts. More details can be found in Ap- pendix A.3. The evaluation metric NDCG @ 10 is the same with BEIR benchmark, which measures Normalized Discounted Cumulative Gain (Wang et al., 2013) of top 10 prediction. The higher NDCG@ 10 value indicates better performance.
d+EDa+ E-
D ~ (P))))) Dat.
(12)
Notice the negatives for f*() have comprehensive coverage from multiple corpora.
Iterative Training. The learning of M and f"() is an iterative process that fits naturally into the training procedure of dense retrieval train- ing with hard negatives. We follow the standard iterations in ANCE and construct the f-th training episode of MoMA:
Augmenting Corpora. During training, the mixture-of-memory is composed of source train- ing corpus (MARCO), Wikipedia and medical knowledge graph. We use the Wikipedia chunk prepossessed by (Karpukhin et al., 2020) without further processing! The medical knowledge graph is extracted from the Medical Subject Headings (MeSH)², an open-source database for indexing and cataloging biomedical and health-related information. Since is hierarchical in structure, we linearize it by concatenating spans with text information. During testing, we directly replace MARCO with the corresponding document sets from BEIR. Each task from BEIR is augmented independently. More dataset and preprocessing details can be found in Appendix A.3.
1. Construct hard negatives D via Eqn. 8 us- ing weights MMMMA() () from the last episode;
2. Retrieve augmentation D via Eqn. 4 using weights f² from the last episode;
3. Train f. MoMA () as Eqn. 7;
4. Formulate new positive augmentation docu- ments D*t, using updated attention scores from fMaMA() (), and mine negative augmenta- tion documents D using ff-(0:
5. Train f2() following Eqn. 11.
Both M) and () can be initialized with BM25 warmed-up T5 retriever. Steps I and 3 above are inherited from standard dense retrieval training. The rest introduced by MoMA. The additional computation the training side mainly resides up- dating the index for the memory mixture, a standard cost in iettrivaalluugmene
Baselines and Model Choices. We compare our MoMA with standard sparse and dense retrieval models on BEIR. We also compare MoMA with advanced approaches that are specifically designed for zero-shot generalization. They involve tech- niques that not directly comparable with this pa- per, including pretraining on extra data, in-domain continuous pretraining, and generating target pairs using another pretrained generative model. Besides. some baselines use larger seale language model as their backbone. We list the details of baselines in Appendix A.4.
language models (Guu et al., 2020; Izacard et al., 2022).
Zero-Shot Retrieval with Plug in Memories. To perform zero-shot retrieval on unseen tasks, MoMA first retrieves augmented documents using f²() from M for the target query g¹, and retrieves target documents d' E Ct with the augmented model (MOAA without changing any model pa- rameters. MoMA allows f*() to attend over the target corpus as well it is plugged in: M
As plug-in-and -lay method, MoMA can be
'htps://huggingface coldatasets/wiii dpr hhtps:/wwww.nchi.nm..iiii www.w/hw..iii..mi.miooo....... gowmesh//

---

Table 1: NDCG@ 10 on the BEIR benchmark. We also include an averaged score on datasets used by Contriever for a fair comparison. The best result each task is marked bold. An denotes unfair comparison, NQ is used in training for GTR. i: GenQ generated pseudo labels to train an independent model for each task. f: Larger models
BM25
ANCE
TS-ANCE
coCondenser
GenQ'
CHRRRRT
Contriover
CTRR
GTRep'T
(TS-ANCE)
(ooComlenser) TTOM22
Parametersa
1T0M
TIM
TOMP
TROM
66MPIX
1ROM
110M
10M
3353
IT0M
TREC-UUVD
11236
0.373
0.654
01715
(1819
01677
11.9%
0,339
0.357
0.762
0.761
BioASQ
0.465
0.232
0.306
0.322
0.318
0,298
0.474
0.271
0.320
0.372
0.371
NPC3epu
0.325
0.237
0.275
0.319
0.3015
0.323
0.306
0.329
0.329
0:416
0.452
0.498
0.495
0.547
HoipurQA
0A433
0.456
0487
0.534
0.593
0L638
0.535
0.579
0.589
FIQA-2018
0.236
0.274
0.295
0.251
0.285
0.308
0.317
0.329
0.319
0.424
0.320
0.329
Signa--IM
0.330
0.238
0.249
0.246
0.274
0.281
0.274
0.261
0.265
0.258
0.264
TREC-NEWS
0.398
0.366
0.382
0.379
0.389
0.396
0.393
0.337
0.313
0.113
0.453
Robustb4
0.408
0.344
0.392
0.412
0.399
0.362
0.391
0.437
0.470
01401
0.475
ArguAm
0.414
0.414
0.415
0.455
0.11
0.493
0.233
0.446
0,511
0.525
04418
0.461
Touché-3220
0.240
0.312
012330
0,205
0.29
0.881
0,890
DiPedi--ntity
0.200
0413
0.317
SCIDOCS
0.158
0.107
0.122
0.115
0.140
0.43
0.145
0.165
0.149
0.158
0.145
Pever
0.753
0.589
0.669
0.655
0.678
0.669
0.771
0.758
0,660
0.712
0.723
0.745
Climaze-eever
0.213
0.376
0.19%
0.194
0.184
0.175
0.184
0.237
0.241
0.262
01.235
0.233
SciFact
0.665
0.475
0.507
0.566
0.600
0.644
0.671
0.677
0,600
0.639
0.632
0.630
COADupSSuck
0.299
0.281
0.296
01263
0.330
0.347
0.350
0.345
0.357
0.384
0.283
0.204
Conneevrr Sub
0.437
0.368
0.408
0.416
0.438
0.425
0.445
0.466
0,442
0.471
0.453
0.471
Avg.
0.428
0.391
0.300
0.416
0.444
combined with other techniques. We initiate MoMA on two dense retrieval models. The primi- tive MoMA (T5-ANCE) is built on the original T5 model checkpoint and optimized iteratively with ANCE--tyle (Xiong et al., 2020) hard neg- atives. By comparing it with T5-ANCE, we can clearly observe the performance gain brought by MoMA. To demonstrate it can integrate techniques from other models to achieve higher performances, we initiate MoMA on a better pretrained model. Following coCondenser (Gao and Callan, 2022), we continuously trained the original T5 model on the MARCO document corpus using sentence- level contrastive loss, combined with the original masked language modeling loss. We then per- formed the same MoMA training on top of the continuously pretrained T5 checkpoint and denoted it as MoMA (coCondenser). The only difference between MoMA (T5-ANCE) and MoMA (coCon- denser) is the initialized model start point. We compare their pretraining details with other models in Table 2. Unlike other work (Yu al., 2022), as a plug-in-plas -llayy design, we did not include target datasets and augmenting corpora in the contrastive pretraining stage. Since MARCO contains only 0.5M documents, it adds fewer computational over- head compared to other methods listed in the table, e.g., Contriever.
followed (Xiong et al., 2020) to further train it us- ing BM25 negatives for 10 epochs. After warming up, we jointly trained the two components for three episodes, each episode including three training epochs. After three joint episodes, the end retriever reaches the best performance on MSMARCO, so we select this checkpoint for evaluation. The ratio between positive and hard negative pairs is 1:7 for both models. The main hyperparameters in MoMA include the total number of grounding documents K and the attention threshold number N in Equa- tion 10. We directly set K=10 and N=5 without any parameter tuning. More details hyperpa- rameters and experimental settings can be found in Appendix A.5.
5 Evaluation Results
5.1 Zero-Shot Retrieval Accuracy and Efficiency
The retrieval accuracy of MoMA and baselines are listed in Table 1. Besides baselines of simi- lar parameter count, we also include larger models (GTRuee) or those using multiple vectors per docu- ment (ColBERT). MoMA (coCondenser) shows the strongest zero-shot accuracy against previous state- of-the-art methods that do continuous contrastive pretraining (coCondenssr)), generate pseudo la- bels (GenQ), or consume additional training sig- nals in both continuous pretraining and finetun- ing phrases (GTRusee. MoMA (T5-ANCE) also achieved nearly comparable zero-shot accuracy against larger models like GTRgage and ColBERT which scales up the number of vectors per docu- ments (one per token). This confirms that retrieval-
Implementation Details. For MoMA, we use the T5-base (Raffel et al., 2019) architecture (12- layer Transformer, 768 hidden size) by directly loading the checkpoint from HuggingFace*. To warm up the language model for dense retrieval, we
Phapss/hhuggingffecccoottt-saa

---

Table 2: Computational analysis in the pretraining stage of different models.
5.2 Performance with Different Memories
Table 4 evaluates how MoMA behaves under dif- ferent combinations of external memories. Unsur- prisingly, using a single out-of-domain memory for retrieval augmentation does not help, for example, even though MARCO is the source domain corpus, solely grounding on it reduces zero-shot accuracy. MeSH as the sole augmenting corpus also lowers performance, even on some medical retrieval tasks such as BioASQ. Interestingly, when expand the memory include MARCO, Wiki, and MeSH, but keep the target corpus excluded (w/o Target). MoMA exhibits better accuracy compared to the no- memory version. Our conclusion is that more memory sources achieves better generalization, especially when no target domain information is available.
Model
Pretraining Corpus
Batch Size
Training Steps
MoMA (TS-ANCE)
McMA (ooCondensery
MARCO
126
50k
NQ,CQA
Contriover
GENet
20448
500K
Wikipodia
2048
200k
Table Efficiency of MoMA search and training.
Operation
Offline 188h
Ouline
BM25 Index Build
BM25 Retrieval Per Query
43ms
MoMA Inference
Encoding of Corpus/Per Doc
1.5b/4,5ms
Query Encoding
55ms
ANN Retrieval (batched q)
9ms
Dense Retrieval Total
64ms
MoMA Training
Encoding of Corpus/Per Doc
1.5k/4.5ms
ANN Index Build
10s
Neg Construction Per Batch (32 queries)
45ms
Back Propagation Per Batch (32 queries)
330ms
In the Full setting, the 3-memory mixture of MARCO, Wiki, and MeSH is jointly learned with final task at training time. At test time, MARCO is swapped out for the target corpus. The Full im- proves zero-shot accuracy over both the w/o Target setting (where the target corpus is excluded at test time), and the w/o Learning setting (wherein the augmentation component is not learned). As ex- pected, plugging in the target corpus test time is the most valuable source of generalization power. It is also the most realistic, as access to the target corpus may only be available at testing time.
augmentation provides another path to improve language models' generalization ability besides scaling up. MoMA (TS-ANCE) also outperforms T5-ANCE, which MoMA (T5-ANCE) uses as subroutine for retrieval augmentation, on all but one retrieval task, showing the improved general- ization ability from plug-in mixture of memory.
We evaluate the efficiency of MoMA in two stages: offline model training and online infer- ence. In offline training from Table 2, MoMA (T5-ANCE) is significantly cheaper than other methods we do not require pretraining on large external corpora, which saves hundreds of hours training time. MoMA (condenser) additionally pre- train on MARCO for 50k steps, which is far fewer than the other compared methods. In online in- ference, similar with other retrieval enhanced lan- guage models, MoMA imposes a necessary cost of retrieval augmented model upon the baseline T5-ANCE. We further provide detailed efficiency analysis on MoMA in Table The online latency is measured on one query and 100 retrieved doc- uments. Due to the query augmentation, query encoding is more costly and takes about 55ms per query. Even with the augmentation cost, the full dense retrieval total online inference cost is 64ms, only slightly above the BM25 retrieval latency. The ANN retrieval is very efficient, only takes 9ms. In addition, the complexity of ANN retrieval is sub- linear to the corpus size, in most ANN framework such as FAISS. Thus the extra round of ANN re- trieval operation in MoMA is not the bottleneck even when the size of memory mixture scales up.
5.3 Effect of Memory Mixture Learning
To study the effect of our joint learning mechanism on the memory mixture, we compare with re- cent state-of-thh-ar Attention Distillation (ADist), which is first used in Izacard and Grave (2020a) and recently updated in a parallel work Izacard et al. (2022). It jointly trains the augmentation model us- ing attention scores from the end language model as pscudo-labels. We also enrich ADist with rel- evance labels from MARCO for more direct su- pervision (ADist + MSMARCO rel). To exclude the effect of contrastive pretraining, we choose MoMA (TS-ANCE) as our own method for com- parison. We also tried using trained ANCE re- triever without further distilling and denote it as w/o Distilling (T5-ANCE). The performances of these joint learning methods are listed in Table 5. The results show that ADist, either standalone or enriched with MARCO labels, does not improve the final accuracy compared to using a supervised dense retriever AS-ANCE. The main difference is

---

Table 4: @ 10 of MoMA under different memory compositions: no memory, single memory, and a mixture of memories. w/o Learning the end retriever select augmenting documents without of augmentation component. w/o Target excludes the target from memory, Best results are in bold.
Single Memory
Memory Mixture
No Memory
MARCO
Wiki
McSH
Target
w/o Learning
w/o Target
Full
TREC-COVID
0.653
0.576
0.592
0.669
0.731
0.759
0.664
0.761
BioASQ
0.322
0.247
0.262
0.219
0.361
0.359
0.271
0.371
NFCorpus
0.275
0,295
0.302
0.282
0.319
0.317
0.301
0,333
NQ
0.452
0.472
0.486
0.393
0.483
0,510
0.484
0.544
HotpotQA
0.487
0,481
0.519
0,462
0.538
0,539
0.520
0,589
FiQA-2018
0.294
0.296
0.286
0.280
0.320
0.304
0.285
0.329
Signal-IM
0.246
0.239
0.225
0.238
0.250
0.248
0.240
0.264
TREC-NEWS
0.379
0,381
0,391
0,372
0.416
0,410
0.398
0.453
Robust04
0.412
0,435
0.443
0.428
0,483
0.446
0,452
0.475
ArguAna
0.415
0.439
0.438
0.442
0.439
0.427
0.438
0.463
Touché-2020
0.312
0.281
0.281
0.252
0.331
0.275
0.272
0.299
Quora
0.836
0.809
0.798
0.835
0.781
0.813
0,812
0.843
DBPedia-entity
0.290
0,340
0,341
0.287
0.335
0.331
0,342
0,383
SCIDOCS
0.115
0.128
0.121
0.130
0.146
0.134
0.127
0.145
Fever
0.655
0.663
0.735
0,610
0.694
0.718
0.737
0.745
Climate-Fever
0.194
0,231
0.238
0,231
0.228
0.222
0.240
0.233
SciFact
0.566
0.583
0.587
0.585
0.624
0.618
0.598
0.630
CQaDupStack
0.283
0.207
0.218
0.203
0.283
0.235
0,215
0.294
Avg
0.399
0.395
0.403
0.384
0.431
0.426
0.411
0.453
Table Zero-shot Performances of different distillation methods. We observe consistent trend on BEIR datasets. We present results representative datasets from Wikipedia or medical domains.
Distillation Method
TREC-COVID
BIOASQ
NFCorpus
NQ
HatporQA
FEVER
Avg
Soft Attention Distill
ADist (Izacurd al., 2022)
0.609
0.185
0.227
0.351
0.387
0.615
0.396
Aist MSMARCO rel w/o Distilling (TS-ANCE)
0.664
0.220
0,255
0.397
0.394
0.624
0.426
0.741
0.361
0.301
0.472
0.513
0.684
0.512
MoMA
0.762
0.372
0.307
0.490
0.539
0.723
0.532
that ADist Iearns soft attention score distribu- tion, while the supervised retriever is trained effec- tively using hard negative sampling (Xiong al., 2020). Jointly learning using soft labels without hard negatives downgraded the augmentation accu- racy. Hence, MoMA is a simple technique to learn the end task signals via the attention scores together with hard negatives, which improves quality over a supervised retriever alone.
percentage of queries having at least one relevant document in the augmenting set increases from 46% in Eppi-O 62% in Epi-2.
This apparent discrepancy can be explained by the fact that MARCO has only one relevant label per query on average, leaving plenty of room for other types of documents to be included in the augmenting set.
Secondly, the amount of attention paid to cer- tain types of documents by MoMA is positively correlated with their representation in the augment- ing set. This confirms that the joint learning ef- fectively conveys the feedback signals from the end model to the augmentation component. For instance, in Figure 2a, MoMA pays high level of attention to MARCO Other documents, signal reflected in the composition of its augmentation set in Figure 2c. Even though MARCO Other docu- ments were not labeled relevant for the query, they can still prove to be valuable as an augmenting document because they may contain partial infor- mation that helps query understanding (Lavrenko and Croft, 2017) or it was simply not annotated
To further illustrate the joint training process, we track the attention scores of documents from different memory sources as well as their ratio in the augmentation set in Figure 2. We also split MARCO documents by whether they are labeled as Relevant (Rel) for the corresponding query. Firstly, MoMA learns to increasingly attend to, and retrieve, relevant documents from the memory mixture throughout training. In Figure 2a, more attention is paid to MARCO Relevant documents than any other type in the memory. Although the number of MARCO Relevant documents not significant as a percentage of the augmenting set in Figure 2c, a query level analysis confirms that

---

0.5
1.0
0.4
0.4
0.8
0.3
0.3
0.6
0.2
0.2
0.4
0.4
0.1
0.1
0.2
0.2
McSH
Marco Rel
0.0
Others
0.0
0.0
0.0
Egi-0
Epi-1
Epl-2
Epi-0
Epi-1
Epi-2
Epi-0
Epi-1
Epl-2
Egi-0
Epi-1
Epi-2
(a) MoMA Att. Score.
(b) ADist Att. Score.
MoMA Doc Ratio.
(d) ADist Doc Ratio.
Figure 2: Grounding component breakdown for different distillation methods in each learning iteration. We display the regularized doc and att. score ratio of documents from different augmentation sources.
100-
100-
100
100
80
60-
60-
60
80-
40
40
40
40
Target
Wiki
MeSH
NO
HotpotDA
FEVER
NFCorpus
TREC-Covid
BIDASQ
NQ
HotpotQA
FEVER
MFCorpus
TREC-Cove
BIOASO
(a) Doc Ratio. (Wiki)
(b) Doc Ratio. (Med)
(c) Att. Score Ratio. (Wiki)
(d) Att. Score Ratio, (Med)
Figure 3; The inclusion Plug-In memory during testing (grouped by the Wiki and Medical domains).
in MARCO's s sparse labels (Bajaj et al., 2016). In comparison, the correlation of the two in ADist is weak as the model seems to include 60% augment- ing documents from McSH, far greater than the fraction of medical queries in MARCO.
centric tasks like FEVER, as expected. Some tasks may need small amount of precise information from Wikipedia to answer the detailed question, e.g. in HotpotQA. Similar with the training pro- cess, there is non-trivial correspondence between attention score of memory and its membership the augmentation set.
5.4 Generalization of Plug-In Memory
In the previous section, we observed how MoMA learns to attend to, and retrieve, informative docu- ments from memories on which it was trained. In this section, examine the zero-shot behavior of MoMA (T5-ANCE) on new corpora plugged-in test time (keeping Wiki and MeSH before).
Conclusion
In this paper we propose new plug-in mixture- of-memory mechanism for the retrieval augmented language models to improve their zero-shot abil- ity on the dense retrieval task. To learn the mem- ory mixture we develop new joint learning ap- proach that trains the augmentation component using the positive signals from the end task, the language model's attention scores, and hard nega- tives retrieved from the mixture of augmentation corpora. This leads to our final model MoMA (T5- ANCE) and MoMA (coCondenser) that achieve strong zero-shot accuracy on 18 retrieval tasks in- cluded in BEIR. Our analysis shows the importance of augmenting with diverse memory sources and in- domain information for robust generalization. We hope our findings can inspire more future research in better augmenting language models, provide other alternatives to achieve generalization ability beyond solely relying on model scale.
Figure compares documents from the plugged- in target versus the remaining memory mixture in terms of membership in the augmenting set (Doc Ratio) and attention. Again, all tasks, MoMA (T5-ANCE) heavily attends and successfully retrieves in-domain documents, even those in- domain documents were only just plugged in. This confirms that the augmentation model achieves the zero-shot ability to capture relevant information from unseen corpora.
In the medical domain, the model pays more at- tention McSH documents, especially on TREC- Covid task since MeSH includes high quality up- dated information related to COVID-19, Wikipedia documents received more attention on the Wiki-

---

Limitations
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:187771101.
Although MoMA (T5-ANCE) and MoMA (coCon- denser) achieve strong zero-shot performances, we mainly verify their efficacy from the empirical per- formances on BEIR tasks, where the target cor- pora, Wiki and MARCO serve as readily avail- able retrieval sources. In a real-world scenario, the grounding corpora usually need to be customized according to query domains and user needs. Thus, how to choose effective grounding corpora and ef- ficiently evaluate their relative contribution remain an open problem. These analyses will go beyond our empirical settings and reveal a wider applica- tion scenario of MoMA.
Claudio Carpineto and Giovanni Romano. 2012. A survey of automatic query expansion in information retrieval, Acm Computing Surveys (CSUR), 44(1):1- 50.
Arman Cohan, Sergey Feldman, Iz Beltagy. Doug Downey, and Daniel Weld. 2020. SPECTER: Dacument-level representation learning using citation-informedd transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Ethics Statement
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bu- lian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER tion of real-world climate claims. arXiv preprint 2010:2012.00614.
All data in this study are publicly available and used under ethical considerations. Text and figures in the paper are used for illustration only, they do not represent the ethical attitude of the authors.
dataset for verifica-
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval. pages 2288-2292.
References
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. 2016. MS MARCO: A human generated ma- chine reading comprehension dataset. arXiv preprint 3xXiv:661.00920
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In ACL 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610-623.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. arXiv preprint arXiv:22 22. 10496.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Ming-We Chang. 2020. REALM: Retrieval- augmented language model pre-training In JCML.
Alexander Bondarenko. Maik Fröbe, Meriem Be- loucif, Lukas Gienapp. Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Pouhast, and Matthias Hagen. 2020. Overview of Touché 2020: Argument Re- trieval. In Working Notes Papers of the CLEF 2020 Evaluation Labs, volume 2696 of CEUR Workshop Proceedings.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisz- tian Balog, Svein Erik Bratsberg. Alexander Kotov. and Jamie Callan. 2017. DBpedia-Entigy v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR 177 page I 1265-1268, New York, NY, USA. Association for Computing Machinery.
Sebastian Borgeaud. Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean- -aa-ttat Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions tokens. International Conference on Ma- chine Learning, pages 2206-2240. PMLR.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Euro- pean Conference on Information Retrieval, pages 716-722. Springer.
Sebastian Hofstätter, Sheng-Chieh Lin, Jheng- Houg Yang, Jimmy Lin, and Allan Hanbury. 2021. Rf- ficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on

---

Research and Development in Information Retrieval, page 113-122. Association for Computing Machin- ery.
Yubin Kim. 2022. Applications and future of dense retrieval in industry. In Proceedings of the 45th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3373- 3374.
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A benchmark data set for community question- answering research. In Proceedings of the 20th Australasian Document Com- puting Symposium, ADCS 15, New York, NY, USA. Association for Computing Machinery.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lec, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- questions: benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 14422-466
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint wtxiv:2112.09111.
Victor Lavrenko and W Bruce Croft. 2017. Relevance- based language models. ACM SIGIR Forum, vol- ume 51, pages 1660-667-222 ACM New York, NY, USA.
Gautier Izacard and Edouard Grave. 2020a. Distilling knowledge from reader to retriever for question an- swering. arXiv preprint arXiv:2012.0450
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. -eggeeval-uggmeeteed generation for knowledge- intteggvveeeeeveeeeee nip tasks. arXiv preprint arXiv:2005.1 11401.
Gautier Izacard and Edouard Grave 2020b. Leveraging passage retrieval with generative models for open domain question answering.
Gautier Izacard, Patrick Lewis. Maria Lomeli, Lu- Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.08 03299.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv preprint arXiv: 1907.1199.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. JEEE Transactions on Big Data, 7(3):535-544 547.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence Learning Representations.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.0830
Jing Lu, Gustavo Hernandez Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2021. Multi-stage training with im- proved negative contrast for neural passage retrieval. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6091-6103, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Fduuno,, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018 WWW' open challenge: Financial opinion mining and question answering. In Companion P Proceedings of the The Web Conference 2018 WWW '18, page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Urvashi Khandelwal, Omer Levy. Dan Jurafsky, Luke Zettlemoyr,, and Mike Lewis. 2019, Generalization through memorization: Nearest neighbor language models. arXiv preprini 1rXiv:1911.000 100172.
Omar Khattab and Matei Zaharia. 2020a. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39- 48.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022. Text and code embeddings by contrastive pre- training. arXiv preprint arXiv:220i.1000
Omar Khattab and Matei Zaharia. 2020b. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, page 49-48, New York, NY, USA. Association for Computing Machinery.
Jianmo Ni, Gustavo Hernandez Abrego. Noah Constant, Ji Ma. Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-15: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022. pages 1864-1874.

---

Jianmo Ni, Chen Qu Jing Lu, Zhuyun Dai, Gus- tavo Hernández Ábrego, Ji Ma. Vincent Y Zhao, Yi Luan, Keith Hall, Ming- Wei Chang, et al. 2021. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv: 12.0799.
Nandan Thakur. Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021a. Beir: A heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint arXiv:2104.08665
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021b. BEIR: A heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint OrXiv:2104.0866
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore,, Yacine Jernite, Vladimir Karpukhin, Jean Mail- lard, et al. 2020. Kilt: a benchmark for knowl- edge intensive language tasks. arXiv preprint arXiv:2009.02252
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2027 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 5835-588 5847, On- line. Association for Computational Linguistics.
George Tsatsaronis. Georgios Balikas, Prodromos Malakasiotis, Toannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the BIOASQ large- scale biomedical semantic indexing and question swering competition. BMC bioinformatics, 16(1):1- 28.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with unitied text-to-tent trans- former. Journal of Machine Learning Research.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. TREC-COVID: Constructing a pandemic informa- tion retrieval test collection. SIGIR Forum, 54(1).
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of language model? In EMNLP.
Ellen M Voorhees al. 2004. Overview of the tree 2004 robust retrieval track. In Trec, pages 69-77.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-389
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 241 1-11, Melbourne, Australia. Association for Computational Linguistics.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. 2022. Using deep- speed and megatron to train megatron-turin nig 530b, large-scale generative language model. arXiv preprint arXiv:2201. 1990..
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics.
lan Soboroff, Shudong Huang, and Donna Harman. 2018. Trec 2018 news track overview.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2020. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 13693-13696.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022, GPL: Generative pscudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Seattle, United States. Association for Computational Linguistics.
Axel Suarez, Dyaa Albakour, David Corne Miguel Martinez, and José Esquivel. 2018. A data collection for evaluating the retrieval of related tweets to news articles. European Conference on Information Retrieval, pages 780-786. Springer.

---

Yining Wang, Liwei Wang. Yuanzhi Li, Di He, Wei Chen, and Tie Yan Liu. 2013. A theoretical analysis of ndeg ranking measures. In Proceedings of the 26th annual conference on learning theory (COLT 2013), volume 8, page 6. Citeseer.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Chen Zhao, Chenyan Xiong, Jordan Boyd-Graher, and Hal Daumé III. 2021. Distandl-suuperviee evidence retrieval enables question answering without evi- dence annotation. arXiv preprint arXiv:2110.04889
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. arXiv preprint arXiv:2205.128
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jermite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero- shot dense retrieval with momentum adversarial do- main invariant representations. In Findings of the As- sociation for Computational Linguistics: ACL. 2022, pages 4008-4020, Dublin, Ireland. Association for Computational Linguistics.
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul N Bennett. 2021. Zero-shot dense retrieval with momentum adversar- ial domain invariant representations. arXiv preprint arXiv:21 1 10.07581.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Amoll Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arxiv:2007.00808.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2369-2380
HongChien Yu, Chenyan Xiong, and Jamie Callan. 2021. Improving query representations for dense retrieval with pseudo relevance feedback. arXiv preprint arXiv:2108. 13454.
Yue Yu. Chenyan Xiong. Si Sun, Chao Zhang, and Arnold Overwijk. 2022. Coco-dr: Combating dis- tribution shifts in zero-shot dense retrieval with con- trastive distributionally robust learning. arXiv preprint arxiv:2200. 20.

---

0.40
A.3 Datasets Details
Evaluation Datasets Target domain datasets used in our experiments are collected in the BEIR benchmark (Thakur et al., 2021b)* and include the following domains:
0.38
0.36
0.34
Open-domain Question Answering (QA): Hot- potQA (Yang et al., 2018), NQ (Kwiatkowski et al., 2019), and FIQA (Maia et al., 2018).
Aug
Retriever
0.32 Epi-O
Epi-1
Epi-2
Epi-3
Figure 4: MRR 10 on MSMARCO the augmenta- tion component and end retriever during MoMA (T5- ANCE) training.
Bio-Medical Information Retrieval: TREC- COVID (Voorhees et al., 2021), NFCor- pus (Boteva al.. 2016), and BioASQ (Tsat- saronis al., 2015).
A Appendix
Argument Retrieval: Webis-Touché2022 (Bon- darenko et al., 2020) and ArguAna (Wachsmuth et al., 2018).
A.1 Performance on Source Domain
Figure 4 demonstrates the MRR@ @ 10 MS-
News Retrieval: TREC-NEWS (Soboroff et al., 2018) and Robust04 (Voorhees al., 2004).
MARCO for the end retriever and augmentation component of MoMA (coCondenser) over differ-
ent training epochs. We make the following ob-
Tweet Retrieval: Signal-Im (Suarez et al., 2018).
servations. Firstly, the augmentation component improves on the source domain even though is not directly optimized with relevance labels. Since helpful augmentation documents are usu- ally strongly related to the query, the augmentation component benefits from such indirect relevance signals. Secondly, the end retriever monotonically benefits from information collected by the augment- ing component, indicating that the two components mutually enhance each other in the joint learning process. We further compare MoMA with relevant baselines on MSMARCO in Table 6. The com- parison verifies that MoMA also achieves better performance the source domain retrieval tasks.
Duplicate Question Retrieval: Quora (Thakur al., 2021b) and CQADupStack (Hoogeveen
al., 2015).
Entity Retrieval: DBPedia (Hasibi et al., 2017)
Citation Prediction: SCIDOCS (Cohan et al.. 2020)
Fact Checking: SciFact (Wadden et al., 2020). FEVER (Thorne al., 2018), and Climate- FEVER (Diggelmann et al., 2020)
We list the statistics of the BEIR benchmark in Table 8.
Augmenting Corpora Corpus size We first in- troduce more details on how we preprocessed the Medical Subject Headings (MeSH) Database. We select text information from the Qualifier Record Set and Descriptor Record Set. Each set contains multiple <Concept> elements, which is composed of three sub-elecments, i.e., <Concept- Name>, <ScopeNote> and <TermList>. Among the sub-elecments, <ScopeNote>> is the major tex- tual information source, which is usually short description medical term or phenomenon. We directly consider each <ScopeNote> as document entry and concatenate it with corresponding <Con- ceptName>.
A.2 Case Studies
Table 7 shows examples of how augmenting doc- uments chosen by MoMA can provide valuable contextual information for the query. The first ex- ample training query from MARCO, where the augmenting documents help disambiguate the query word "rating". In the second one, documents from the official Wiki and HotpotQA's Wiki corpus are descriptions of the two entities in HotpotQA's comparison question. It illustrates how MoMA pro- vides more comprehensive augmentation by incor- porating information from different sources. The last query shows the benefit of the in-domain plug- corpus as it brings in very specific information about the query (AND-I/Cuf4) that is hard to find elsewhere.
We list the statistics of the augmenting corpora in Table 9.
hhttps://githud /

---

Table 6: Performance comparisons of different methods on MSMARCO.
DPR
T5-ANCE
coCondenser
MoMA (T5-ANCE)
MoMA (coCondenser)
MRR 10
0,3340
0.3678
0.3820
0.3866
0.4056
Table 7: MoMA retrieves augmenting documents during training (Mareo) and testing (BEIR).
ANCE constructs hard negative examples from an ANN index of the corpus. The hard nega- tive training instances are updated in parallel dur- ing fine-tuning of the model. The model is a RoBERTa (Liu et al., 2019) model trained on MS MARCO for 600k steps.
Queries
Augmentation Docs
Training
[Marco]
Marco] Why is Hotel Transylvania 2 rated PG? It rated PG for some scary images, action and rude humor. [Wiki] Another view aggregate calculated an average score of 47 out of 100, indicating "mixed av- erage reviews".
What
is
hotel
sylvanía
rated
T5-ANCE Different with default ANCE set- ting. we replace the backbone language model RoBERTa with T5-base. All the other model settings are the same with the original ANCE. We include this baseline because for MoMA, could be viewed as an ablation without memory augmentation. We can directly observe the impact of plug-in mixture of memory by comparing T5-ANCE with MoMA.
Zero-Shot Testing
[Hoipo[QA]
[Wiki] Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer. [HotpotQA] Edward Davis Wood Jr. (October 10, December 10, 1978) was an American filmmaker, actor, writer. producer, and director.
Were Scott
Derrickson and Wood of the same nationality? [BIOASQ] Is AND- 1/Ctf4 essential for prolifera-
subroutine
Ed
[BIOASQ] AND-I/CU4 bridges the CMG helicase and DNA polymerase alpha, fa- cilitating replication. [Wiki] FADD has no effect on the proliferation of B cells in- duced hy stimulation of the B cell receptor.
coCondenser a continuous pre-trained model based on BERT, with the equivalent amount parameters to BERT-base. It enhances the repre- sentation ability of [CLS] token by changing the connections between different layers of Trans- former blocks. Fine-tuning of coCondenser uses BM25 and self-mines negatives.
tion?
A.4 Baselines
We use the baselines from the current BEIR leader- board (Thakur et al., 2021b) and recent papers. These baselines can be divided into four groups: dense retrieval, dense retrieval with generated queries", lexical retrieval and late interaction.
Contriever conducts unsupervised contrastive pretraining with data augmentations and momen- tum queues Wikipedia and the larger CC- Net (Wenzek al., 2020) corpora for 500k steps.
Dense Retrieval For dense retrieval, the base- lines are the same dual-tower model as ours. We consider DPR (Karpukhin et al., 2020), ANCE (Xiong et al., 2020), T5-ANCE, coCon- denser (Gao and Callan, 2022) and one recently- proposed model GTR (Ni al., 2021) with differ- ent size configuration in this paper.
GTR initializes the dual encoders from the T5 models (Raffel et al., 2019). It is first pre-trained on Community QA with 2 billion question- answer pairs then fine-tuned on NQ and MS Marco dataset. In addition, they use the hard neg- atives released by RocketQA (Qu et al., 2021) when finetuning with MS Marco data and the hard negatives release by (Lu et al., 2021) for Nat- ural Questions. GT leverages the same T5- base model as MoMA, while GTREE is based T5-large, which is not directly comparable method it triples the parameters.
DPR uses a single BM25 retrieval example and in-batch examples as hard negative examples to train the model. Different from the original pa- per (Thakur et al., 202 lb) that train the DPR on QA datasets, we train DPR MS MARCO (Ba- jaj et al., 2016) Dataset for fair comparison. No- tice that this also lead better results according to Xin et al. (2022).
Dense Retrieval with Generated Queries GenQ first fine-tunes T5-base (Raffel et al., 2019)
*e separate them from dense retrieval since they usually rely on Seq2seq models to generate pscudo query-document pairs, and they train model for each dataset independently instead of using a single model for all datasets.
UUnfortunately, this corpus has not been released by the authors.

---

Table 8: Statistics of datasets in the BEIR benchmark. The table is taken from the original BEIR benchmark paper (Thakur et al., 2021b).
Split (-2)
Train
Dev
Test
Avg. Word I Lengths
(1)
(
Relevancy
#Query
+Query
#Corpus
Arg. D/Q
Query
Document
Passage Reuneval
Misc.
MS MARCO
X
Binary
532,761
6,960
8,841823
L.1
5.96
55.98
Bin-Medical
Bio-Medical
TREC-COVID
3-level
171,332
493.5
10.00
160.71
Information
Bio Medical
NFCorpus
/
3-level
110.575
224
323
3,633
38.2
3.30
232.26
Ketrival (IR)
Bio-Medical
BinASQ
Binary
32,916
14914,002
4.7
8.05
202.61
Question Answering
Wikipedia
NQ
Binary
132,003
3,452
2.681 4668
1.2
9.16
78.88
Wikipedia
HorporQA
Binary
170,000
5.447
7,405
5,233,329
2.0
17.61
46,30
Finance
HQQ-2018
Binary
14,166
500
57.638
2.6
10:77
132.32
Tweet-Retrieva
Twitter
Signal- IM (RT)
3-level
2.866.316
19.6
9:30
13.93
News Retrieval
News
TREC-NEWS
5-level
594,977
19.6
11:14
634,79
RobustM
15.27
Argument Retricval
Mise: Misc.
ArguAna
Rinsry
1.406
8,674
1.0
192.88
166.80
Touché-2020
3-level
382,545
19.0
6.55
242.37
Cuplicato-uusstien
StackEt. Queen
CQADupStack
/
Binary Rinsry
13,145
457, 199
1.4
8.59
129.09
Rerrieval
Queen
X
5,000
10,000
$22,991
1.6
9.53
11.44
Furity-Retrizwa Retriivl
Wikipedia
DBPadia
3-1evel
67
4,635,922
38.2
5.39
49.68
Citriio-Predistion
Solentille
SCIDOes
/
Rinsry
1,000
25,657
4.9
9:38
176.19
Wikipedia
REVER
140,085
6,666
6,66
5,416,568
1.2
8.13
84.76
Checking
Wikipedia
Climate-FEVER
Binary
5.416,593
84.26
Solentifle
SoiPaer
Rinary
920
300
5,183
1.1
12.37
213.63
Table 9: Statisties of the augmenting corpora.
AdamW (Loshchilov and Hutter, 2019) with peak learning rate 5e-6, weight decay of 0.01, and linear learning rate decay. The global batch size is set to 256. The maximum length of query and passage are set to 32 and 128 respectively, We summarize all hyperparameter settings in Table 10. The model is trained with 8 Nvidia A100 80GB GPUs and FP16 mixed-precision training. The to- tal running time is 6.6 hrs for three episodes of augmentation component training and 6.3 hrs for end retriever training. We detail the training time of each episode in Table 11.
Datasets
Corpus Size
Avg. Doc Length
MS MARCO
502,939
56.0
McSH
32.326
16.8
Wiki
21,015,324
100.0
model MS MARCO for 2 epochs and then gener- ate queries for each passage as additional training data for the target domain to continue to fine-tune the TAS-B B (Hofstätter et al., 2021) model.
Lexical Retrieval Lexical retrieval is a score function for token matching calculated berween two high-dimensional sparse vectors with token weights. BM25 (Robertson et al., 2009) the most commonly used lcxical retrieval function. We use the BM25 results reported in Thakur et al. (2021b) for comparison.
When evaluating on the BEIR benchmark, we follow the setting in GTR (Ni et al., 2021), which use sequences of 64 tokens for the questions and 512 for the documents in all datasets except Trec- News, Robust-04 and ArguAna. In particular, we set the document length to 768 for Trec-News and Robust-04. For ArguAna, set both question and document length to 128. The above length setting in accordance the average query and document lengths in these datasets.
Late Interaction We also consider a late interac- tion baseline, namely ColBERT (Khattab and Za- haria, 2020b). The model computes multiple con- textualized embeddings for each token of queries and documents, and then a maximum similar- ity function to retrieve relevant documents. This type of matching requires significantly more disk space for indexes and has a higher latency.
A.5 Detailed Experimental Settings and hyperparameters
Our implementation uses PyTorch (Paszke al., 2019) with Hugging Face Transformers (Wolf et al., 2020). We optimize the model using

---

Table 10: The hyperparameters of MoMA.
Hyperparameters
Scttings
Grounding document number
10
Attention threshold number
5
Negative mining depth
200
Global butch size (query size per batch)
256
Positive number per query
Negative number per query
7
Peak learnig rate
5e-6
Learnig rate decay
0.01
Optimizer
AdamW
Scheduler
Linear
MARCO Maximum query length
32
MARCO Maximum document length
128
Table 11: Training time for MoMA with three training episodes. We use Nvidia A A100 80GB GPUs with FP16 mixed-precision training.
Stage
Augmentation Component
End Retriever I.5h
Epi-1
0.8h
Epi-2
1.5h
Epi-3
0.8h
1.5h
Index refresh
1.4h
0.6h
Refresh number
Overall
6.6h
6.3h


Reviews:
[{"review_id": "uIwNVScVGF", "reviewer": "Reviewer_uvZi", "summary": "", "questions": "1. The author needs to clarify whether the information retrieval task in this paper is supervised or unsupervised make me confused, e.g. the model training section mentions supervised labels. \n2. The author needs to clarify the reasons behind the poor performance of the multiple knowledge enriched model on these datasets. The MoMA (coCondenser) performs worse than BM25 on certain datasets, such as Signal-1M and SciFact. However, the paper does not provide an analysis for this phenomenon. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful questions and suggestions!\n\nW1: In fact, whether the information retrieval task in this paper is supervised or unsupervised make me confused, e.g. the model training section mentions supervised labels. \n\nA1: The retrieval task is a zero-shot setting where labels are from the source task, and the model is evaluated on target tasks where no labels are available. In our setting, the model is supervised trained on MAMARCO, and zero-shot evaluated on the BEIR benchmark. It is a standard setup in zero-shot retrieval and we follow other baselines [0,1,2] to do so.\n\nW2: MoMA (coCondenser) performs worse than BM25 on certain datasets, such as Signal-1M and SciFact. However, the paper does not provide an analysis for this phenomenon. \n\nA2: The performance downgrade can be attributed to the language style (format and length) disparities between the training (MSMARCO) and evaluation (Signal-1M and SciFact) sets. In fact, results from Table 1 suggest that almost all dense retrieval models without training on external corpus experience the downgrade. For example, ANCE, coCondenser, and GTR. The training corpus for all the models is MSMARCO, a general domain QA corpus originated from the Bing search engine. In contrast, Signal-1M centered on tweet retrieval for news articles, and SciFact is for veracity of scientific claims. Both datasets are significantly different from MSMARCO in terms of length and style. In this case, a traditional lexical match such as BM25 is more robust in this scenario. \n\n[0] Yu et al. Coco-dr: Combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning. \\\n[1] Wang et al. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. \\\n[2] Ni el al. Large Dual Encoders Are Generalizable Retrievers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple, well-motivated method that works well, and the code will be publicly available.\n2. The model is evaluated extensively with good empirical results on IR benchmark (BEIR).\n3. The authors conduct sufficient ablation study and verify the effectiveness of the two main ideas in this paper.\n4. The proposed method is more efficient than the previous retrieval pretrained models.\n", "reasons_to_reject": "1. In fact, whether the information retrieval task in this paper is supervised or unsupervised make me confused, e.g. the model training section mentions supervised labels. It would be better to clarify in the paper.\n2. MoMA (coCondenser) performs worse than BM25 on certain datasets, such as Signal-1M and SciFact. However, the paper does not provide an analysis for this phenomenon. If possible, I would like to know the reasons behind the poor performance of the multiple knowledge enriched model on these datasets.\n", "questions_to_authors": "1. The author needs to clarify whether the information retrieval task in this paper is supervised or unsupervised make me confused, e.g. the model training section mentions supervised labels. \n2. The author needs to clarify the reasons behind the poor performance of the multiple knowledge enriched model on these datasets. The MoMA (coCondenser) performs worse than BM25 on certain datasets, such as Signal-1M and SciFact. However, the paper does not provide an analysis for this phenomenon. \n", "paper_topic_and_main_contributions": "In this paper, the authors propose a new retrieval augmentation mechanism for dense retriever. The proposed method is simple and effective, built upon knowledge from multiple external corpora. From what I understand, two ingredients that make this model work better than the previous models: (1) using MoMA to provide richer and more diverse external resources for augmentation and (2) jointly training the augmentation component and dense retriever using supervised relevance signals and self-mined hard negatives.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NO"}, "extra_text": {}}, {"review_id": "GqIXaRFxCm", "reviewer": "Reviewer_3HsD", "summary": "", "questions": "1. How is the in-domain result on MSMARCO?\n2. How is the trained augmentation component $f^a()$ performs alone on retrieval?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful questions and suggestions!\n\nW1: When the target corpus is excluded, the performance improvement is minimal or even reversed.\n\nA1: We will first explain the reason for the minimal performance improvement and then suggest ways to improve. \nThe marginal improvement on a few datasets could be traced to the semantic disparities between augmentation corpus and target queries. Since we use a fixed set of memory for all target tasks, it may not be as effective when the task differs significantly from the augmentation corpus (MSMARCO and Wikipedia) in terms of topics and styles. For example in Table 1, MoMA performs well on queries asking about general world knowledge (NQ, DBPedia, and Robust04), which are related to or can be answered by MSMARCO or Wikipedia in our memory mixture. However, when the queries differ a lot in topic, such as tweet retrieval for a corpus (Signal-1M), the improvement from MoMA is marginal. The observation coincides with the results in Table 4, which we summarize here as:\n\n|Task|No Memory|+MARCO|+Wiki|+Target|Mixture w/o Target|Mixture w Target|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|NQ|0.452|0.472|0.486|0.483|0.484|0.544|\n|DBPedia|0.290|0.340|0.341|0.335|0.342|0.383|\n|Robust04|0.412|0.435|0.443|0.483|0.452|0.475|\n|Signal-1M|0.246|0.239|0.225|0.250|0.240|0.264|\n\nOn the first three tasks, using each memory alone already brings a substantial improvement, and together they lead to a slightly better performance. However, on Signal-1M, MoMA is not as effective when each individual memory is not helpful enough. An intuitive explanation could be if the external corpus is very different from the usage scenario, then there is not much beneficial information from there. To conclude, finding an external corpus that includes useful information for the task is important for MoMA, and perhaps for most retrieval-augmented systems. To ensure a stable performance improvement, we suggest 1) using a relevant corpus that matches with the query forms and topics, e.g., the target corpus. In fact, the target corpus is available for the plug-in in many applications, making MoMA feasible for real-world scenarios, or 2) using an up-to-date and comprehensive corpus e.g., the real-world web search scenario. We will add the analysis and discussion in our next version.\n\nW2: Since the target corpus is retrieved once and is fused later, how does this method compare to a retrieve and re-rank method?\n\nA2: 1) The major reason of MoMA being a retrieval model not re-ranking is its design and application. Retrieval is to find top-K from the entire corpus where more efficiency restrictions are needed, e.g., dual-encoder where documents are encoded independently with the query. Reranking is to further rankthe top retrieved results. Thus more expensive models can be used, such as cross-encoders. However, reranking has to be done after an initial retrieval. MoMA belongs to the first stage retrieval as it is to find top-K documents from the entire corpus and it is a dual-encoder model. Any reranking model can be applied after MoMA.\n2) Another minor difference is how they use the augmentation documents. In MoMA, the top retrieved documents may not overlap with augmentation documents. Augmentation documents are used as input together with the original query to learn a better dense vector representation. We then use the dense vector to retrieve from the entire document set. While in re-ranking, the final results are a shuffled list of first-round retrieved documents.\n\nW3: Since the inclusion of the target corpus is crucial to the performance, the purpose method could be inefficient when the target corpus is very large.\n\nA3: From Table 3, we can decompose the computation cost of MoMA into three parts:\n1) Encoding of target corpus: Since dense retrieval models need to retrieve from the target, it is an unavoidable computation cost for any model. Moreover, it can be computed offline.\n2) Query Encoding: 55ms, which is the major online computation cost of MoMA. However, it is a cost irrelevant to the target corpus size, as we keep a fixed number of K (K=10) of augmentation documents. During query encoding, we only encode the target query along with K augmented documents, so the computation is stable for different corpus sizes.\n3) ANN Retrieval: 9ms as the sum of two retrieval rounds, which is an additional but moderate cost of MoMA. Compared with other dense retrieval models, MoMA needs to retrieve twice, bringing an overhead of ~4.5ms. This accounts for at most 4.5/(9+55)=7.03% time cost. The cost may slightly grow when the target corpus is large, but LSH-based ANN index systems in general can achieve sub-linear complexity regarding the corpus size when needed [1].\n\nQ1&2: How is the in-domain result on MSMARCO? How does the trained augmentation component fa() perform alone on retrieval?\n\nA1&2: \n||Epi-0|Epi-1|Epi-2|Epi-3|\n|:---:|:---:|:---:|:---:|:---:|\n|Retriever|0.3520|0.3818|0.3929|0.4056|\n|Augmentation Component|0.3520|0.3598|0.3554|0.3643|\n\nThe table demonstrates the MRR@10 on MSMARCO for the end retriever and augmentation component of MoMA (coCondenser) over different training epochs.We make the following observations. Firstly, the augmentation component improves on the source domain even though it is not directly optimized with relevance labels. Since helpful augmentation documents are usually strongly related to the query, the augmentation component benefits from such indirect relevance signals. Secondly, the end retriever monotonically benefits from information collected by the augmenting component, indicating that the two components mutually enhance each other in the joint learning process. We further compare MoMA with relevant baselines on MSMARCO:\n\n||DPR|T5-ANCE|coCondenser|MoMA (T5-ANCE)|MoMA (coCondenser)|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|MRR@10|0.3340|0.3678|0.3820|0.3866|0.4056|\n\nThe comparison verifies that MoMA also achieves better performance on the source domain retrieval tasks.\n\n[1] Aumüller et al. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The purpose method seems novel and the results are positive. The method generalizes well to out-of-domain data, which is a common problem for other dense retrieval methods. The paper is well-written with detailed ablation study and discussion.", "reasons_to_reject": "The ablation study shows that most performance gain comes from the inclusion of the target corpus. When the target corpus is excluded, the performance improvement is minimal or even reversed. This result brings several concerns:\n1. Since the target corpus is retrieved once and is fused later, this framework resembles a retrieve and re-rank schema. How does this method compare to a retrieve and re-rank method?\n2. Since the inclusion of the target corpus is crucial to the performance, the purpose method could be inefficient when the target corpus is very large.", "questions_to_authors": "1. How is the in-domain result on MSMARCO?\n2. How is the trained augmentation component $f^a()$ performs alone on retrieval?", "paper_topic_and_main_contributions": "This paper introduces a new method for augmenting a dense retriever. The proposed method first retrives relevant documents from an external collection and then produces the query's dense representation conditioned on the original query and the retrieved documents. The authors tested the method on BEIR. The method performs better than some other dense retrival methods and generalizes better out of distribution. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PWWT7J3lrb", "reviewer": "Reviewer_i6oZ", "summary": "", "questions": "- Is it possible to setup a dense retrieval baseline that uses ST5-EncDec variant of Sentence-T5 and retried variants are restricted to come from different sources? It would require encoder-decoder architecture to consumer information from all the sources though. \n- Are the results in Table 1 averaged over different trails assuming that they might be vary on different trials?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful questions and suggestions!\n\nQ1: Is it possible to set up a dense retrieval baseline that uses ST5-EncDec variant of Sentence-T5 and retried variants are restricted to come from different sources? It would require encoder-decoder architecture to consume information from all the sources though. \n\nA1: We agreed that such a baseline would be helpful. In the paper, we combine corpus from different sources into the same ANN index, since manually controlling the ratio of different sources may impose some inductive bias. In the next version, we will add a baseline that manually restricts the data source. We also want to mention that our T5-ANCE used the architecture of the encoder-decoder variant of Sentence-T5, as we take the decoder output as query representation. We will make this description in Section 3.1 more explicit. Overall, T5-ANCE is a stronger (improved) ST5-EncDec baseline since it is trained with negative labels on MSMARCO.\n\nQ2: Are the results in Table 1 averaged over different trails assuming that they might vary on different trials?\n\nA2: Results may vary across different runs, but the improvement of MoMA over Contriever and $GTR_{large} is statistically significant with p-value < 0.05. The improvement of MoMA over other baselines is significant with p-value < 0.01. We will add the significance test in the next version. Also, we ensure that all our runs are random and we avoid cherry-picking. From our experience, MoMA and T5-ANCE, etc. are stable as long as the hyperparameters are set in a reasonable range.\n\nMissing Reference: \n\nThanks for the suggestion. It is an insightful observation of the connection between memory networks and memory augmentation in MoMA. We will add it to the reference and discuss it in the related work as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work presents a clear and well-motivated technique to address the problem zero-shot document augmentation retrievers. \n- This method shows improved results consistently across several benchmarks. Additionally, effect of memory mixture is presented empirically which is both effective and show strong generalization performance new domains. \n- The very 'plug-n-play' nature of the proposed approach makes it widely applicable therefore is of interest for the NLP research community. ", "reasons_to_reject": "Please see questions and respond to them.", "questions_to_authors": "- Is it possible to setup a dense retrieval baseline that uses ST5-EncDec variant of Sentence-T5 and retried variants are restricted to come from different sources? It would require encoder-decoder architecture to consumer information from all the sources though. \n- Are the results in Table 1 averaged over different trails assuming that they might be vary on different trials?\n", "paper_topic_and_main_contributions": "This paper presents Mixture-Of-Memory Augmentation (MoMA), a method to improve the zero-shot generalization ability of large language models (LLMs). MoMA retrieves augmentation documents from multiple information corpora, which we refer to as \"external memories.\" This is inspired by query expansion work in the information retrieval (IR) literature. The method allows us to update the corpora, which in turn allows us to update the memory of LLMs without parameter tuning.\nThis work presents learning mechanism that trains the augmentation component. Authors evaluate the results on several tasks from the BEIR benchmark. Experiments and results suggest that the proposed MoMA approach is superior to other dense retrieval approaches", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Hierarchical Memory Networks: https://arxiv.org/pdf/1605.07427.pdf  -- this work might be worth mentioning as this is one of the earlier works that attempts to retrieve K memory specially the clustering variants proposed in this paper. "}}]

Author rebuttal:

Meta-review: This paper proposes a new retrieval augmentation mechanism to improve the zero-shot generalization ability of language models. Most of reviewers have given positive scores, indicating that they appreciate the authors' work. Although there are specific concerns raised by each reviewer that need to be addressed for further improvement, the authors have adequately addressed the concerns raised by the reviewers, demonstrating their capability to further improve the paper. The authors should carefully consider and incorporate these suggestions to further enhance the quality of the work.
Decision: Accept-Main