AUGMENTING ZERO-SHOT DENSE RETRIEVERS WITH PLUG-IN MIXTURE-OF-MEMORIES
Authors: Suyu Ge
Chenyan Xiong
Corby Louis Rosset
Arnold Overwijk
Jiawei Han
Paul N. Bennett
Venue: Submitted to ICLR 2023 (2023)
Keywords: Retrieval Augmented Language Model
Zero-shot Dense Retrieval
Mixture of Memory

Abstract: In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora (“external memories”), with the option to “plug in” new memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model in a zero-shot dense retrieval setting by augmenting a strong T5-based retriever with MoMA. Our model, MoMA-DR, obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark. It outperforms other dense retrieval models of similar scales and achieves comparable accuracy with systems that seek generalization from increased scales in encoder models or vector indices. Our analysis illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of joint learning, and how MoMA-DR utilizes the plug-in memory at inference time without changing its parameters. We plan to open source our code.

Under review as a conference paper at ICLR 2023
AUGMENTING ZERO-SHOT DENSE RETRIEVERS WITH PLUG-IN MIXTURE-OF-MEMORIE
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), mechanism that retrieves aug- mentation documents from multiple information corpora ("external memories"), with the option to "plug in" new memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model menting a strong TS-based retriever with MoMA. Our model, MoMA-DR, obtains strong zero-shot retrieval aceuracy on the eighteen tasks included in the standard BEIR benchmark. It outperforms other dense retrieval models of similar scale and achieves comparable accuracy with systems that seek generalization from increased scales in encoder models or vector indices. Our analysis illustrates the necessity of augmenting with mixture-of-mert -f-emmory for robust generalization, the benefits joint learning, and how MoMA-DR utilizes the plug-in memory inference time without changing its parameters. We plan to open source our code.
zero-shot dense retrieval setting by aug-
INTRODUCTION
Scaling up language models with more parameters, compute, and annotation data imppooes model generalization ability on downstream applications (Raffel al.] 2009 Brown et al., 2020 Smith et al. (022), but with diminishing return: linear improvements on downstream metrics often require exponentially more parameters and computing cost (Kaplan et al. 2020 Hoffmann et al. 2022).
Hence, scaling pretrained language models in this way is economically unsustainable (Strubell et al.)
2020 [ender et al. 2021 Zhang et al. 2022).
Retrieval augmented language models provide promising alternative. They allow language models to efficiently access vast resources from an external corpus (Guu et al. 2020 Borgeaud et al. 2022) that serves as a kind of "memory" they can refer to when making predictions, alleviating the need to memorize as much information in their own network parameters (Roberts et al. 2020). This open- book approach helps language models to better generalize token prediction tasks and machine translation (Khandelwal et al., 2019: Borgcaud et al.) 2022). and tasks which already involve a first-stage retrieval component, c.g., OpenQA (Borgeaud al. 2022 Izacard al. (022).
In this paper we improve the zero-shot generalization ability of language models using "mixture-of- memory" (MoMA), a new retrieval augmentation mechanism. Instead of a single corpus, MoMA retrieves documents from a "mixture" of multiple external corpora. This mechanism also allows removing and/or "plugging-in" new corpora during inference time, when more information from the target task is revealed, or as an additional way for users to control the model. It is not trivial to guide a retrieval model to leverage multiple corpora; we need to jointly train the augmentation component and dense retriever using supervised relevance signals and self-mined hard negatives.
We instantiate MoMA with T5 encoder-decoder model (Ni et al., 2022) and apply it to the dense retrieval task (Karpukhin et al. 2020). Our resulting retrieval system, MoMA-DR, uses set of augmenting documents from the ceeture-of-memmane with important context; the retriever then uses the enhanced query representation retrieve a final candidate set. At inference time, we plug in the target task's corpus to the memory mixture to introduce in-domain context information, without updating any parameter.
to enhance its representation of the query

---

Under review as a conference paper at ICLR 2023
We measure MoMA-DR zero-shot dense retrieval (ZeroDR) Thakur et al. 2021b) important real-world application. Our experiments on eighteen retrieval tasks included in BEIR (Thakur et al.) 20215), the standard ZeroDR benchmark, demonstrate the improved zero-shot ability of MoMA-DR. It outperforms baseline T5 without the MoMA augmentation component, as well as recent state- of-the-art dense retrieval systems of the same scale, by large margins. It also achieves comparable performance to ZeroDR systems that scaled their model parameters, training data, and/or number of vector representations beyond those in this study.
Our analysis reveals that large and diverse corpora in the memory leads to the best performance; only using a single corpus during training does not improve performance on unseen target tasks. The joint learning is also important for MoMA-DD DR to utilize the diverse information from the mixture. Our analysis and case studies illustrate how MoMA-DR leverages the plug-in memory at testing time to enrich its query representations with in-domain information that was not available in training.
2 RELATED WORK
Recent research has explored two common ways to construct the external memory in retrieval- augmented language models. The first is to use a token vocabulary and retrieve similar tokens for language models to copy from when predicting the next token (Khandelwal et al. 2019; Zhong et al. 2022). The second is to use a document corpus, often the pretraining corpus or the task-specific ones, and retrieve the related documents (text sequences) from the corpus as additional input (Guu et al.)
2020 Borgeaud et al. 2022). Document-based ones align well with language systems that already involve first stage retrieval component, like knowledge-intensiin tasks (Petroni et al. 2020) and
OpenQA (Chen et al. 2017). This work falls into the latter.
Learning to retrieve useful documents to augment the language model is a challenging task, since human annotations on the usefulness of augmentation documents are costly and seldom available. The most straightforward way is to use representations from raw pretrained language models to find documents similar to the task input, i.e., as unsupervised dense retrieval (Guu et al. 2020; Borgeaud et al. 2022). Adapting dense retrieval models trained for relevance matching is another common choice [Izacard & Grave 2020b] Lewis et al., 2020 Yu et al.] 2021). A more formal solution is to jointly learn the augmentation components end-to-end using supervision from the final task, for example, treating the augmentation as latent variables and applying EM (Zhao et al., 202I), or distilling the augmentation component from feedback of the final model (Izacard & Grave 2020a). In parallel work, Izacard et al. (2022) found the most effective one is attention distillation method (ADist), which trains the augmentation component using soft labels derived from the end model's attention on augmentation documents.
Recent dense retrieval systems achieve strong empirical performance in supervised settings (Lee et al., 2019 Karpukhin et al. 2020 Xiong et al. 2020). Unfortunately, dense retrieval models trained on resource rich source ks, e.g., web search, do not perform as well when zero-shot transferred to other domains (Thakur et al. 202 This is concerning since many important real-world scenarios do not have the luxury of web corpus training signals and must rely on near zero-shot transfer, especially the medical and enterprise search domains (Kim 2022).
Xin et al. (202T) analyzed the challenge of shifting between training and testing domains, and lever- aged domain-invariant learning to mitigate the gap. Another common approach is to first construct domain-specific weak supervisions for each task, and then use them to train dense retriever (Thakur) al. 2021a Wang et al., 2022). Additionally, continuous pretraining the language model also improves its generalization ability in ZeroDR (Izacard et al. 2021 Gao & Callan 2022).
Many seek better generalization ability in ZeroDR from other resources, for example, combining with sparse retrieval to introduce exact match signals (Formal et al. (021). using multiple vectors per documents for term-level matching (Khattab & Zaharia 2020b). or scaling up the retrieval model using large scale pretrained language models (Ni et al. 2021 Neclakantan et al. (022).
3
METHOD
In this section we first describe our Mixture-of Mixture-of-Memane of-Memoryy Augmentation. Then we discuss how is jointly learned with the end system and enables plug-in memory at inference time.
2

---

Under review as a conference paper at ICLR 2023
3.1 MIXTURE-OF-MEMON AUGMENTATION
Before going to the details of MoMA, we first recap some preliminaries in ZeroDR.
Preliminaries. The dense retrieval (DR) task aims to find relevant documents d from corpus C for the given query by representing them in a shared emhedding space. Specifically, the retrieval score DR is often calculated as:
fq. q q -(
uses dot product as the scoring function to match the embeddings q and d, which is known to support efficient nearest neighbor search (ANN) (Johnson et al. 22119). A pretrained language model often the encoder of choice g(). We use the ST5-EncDec variant of Sentence-T5 (Ni et al. 2022):
((x) Dec(Enc(z)),
(2)
which feeds in the text sequence (prepended by a special [CLS] tokens) to the encoder of T5, Ene(). and uses the output representation of the [CLS] token from the decoder, Dec(), as the text representation. This naturally leverages the attention from decoder to encoder at all Transformer layers (Raffel et al. (211)), as a fine-grained information gathering mechanism.
The training of dense retrieval systems often applies standard ranking loss and pairs the relevant documents d+ E D for each query q with hard negatives d= E D
[ f= <= Σ l(f(q, d+), flq. d (): )): D ANNO A
fla.o) \D+.
d d-ED
Eqn. 3 uses ANCE hard negatives, which are the top-retrieved documents from C' usingg the retriever itself (Xiong et al. 2020 The loss function (() can be any standard ranking loss such as cross entropy. A ZeroDR model is trained g" and documents d Ca from source task, often web search, and tested on target tasks of' and C"; supervision signals are only present from the source.
Mixture-of- Augmentation. The key idea of (document (aacumeet-bbaed)) retrieval augmented lan- guage models is to enrich the representation g(4) with additional contextual input for the model, ie., augmentation documents d' retrieved from external memory M.. Instead of using a single document corpus, MoMA uses multiple corpora to provide richer and more diverse external resources for augmentation. For example, M can be composed by the source corpus C general encyclopedia, a domain specific knowledge graph, etc. Then we can retrieve the augmentation documents D
D (N,)) M -C,,
This augmentation component uses another dense retriever f*() (also a Sentence T5 model), with parameters distinct from those in g(). Note that instcad retrieving D separately from M different ANN memory sources and merging results, Eqn. 1.n...combrr combines them into ANN index. This requires the augmentation component f"() to be flexible enough handle various corpora in the mixture.
Using the augmentation documents using the iisioon-nn-eccodee (FiD) mechanism (Izacard & Grave 2020b):
architecture for 9() in Equ. 2 enables simple extension incorporate the
(5) It feeds in the K augmentation documents separately to the T5 encoder of g(). Then it fuses the encoded documents together with Enc(g) using one decoder that attends to all encoded vectors, as illustrated in Figure
MoMA (q) Dec(Enn(g), Enn(( )); D = ..., dx
The FiD approach in Eqn 5 is a nice balance of efficiency and capacity when modeling multiple text sequences (Izacard & Grave (0200)) is more efficient than concatenating all pieces together, while also remaining expressive enough to model the nuances from many sequences. IIzcard &
Grave 2020al Izacard et al. (2022).
When instantiating MoMA in the dense retrieval setting, we focus on augmenting the query rep-
resentation q, as queries are often short, ambiguous, and benefit more from additional contextual information (Lavrenko & Crofi 2017: Yu et al. 202I). This leads to the following definition of
MoMA-DR:
MMOMA (q, d) d¹' d; qº MoMA d=
(6)
using the construction of M)MA() in Eqn. 5uppo the augmentation documents defined in Eqn. g
3

---

Under review as a conference paper at ICLR 2023
3.2 JOINT LEARNING IN MOMA-DA MAMAADDAAAA AND INFERENCE WITH PLUG IN MEMORY
MoMA-DR has two sets of parameters to learn, in the main model (MoMA( and the augmen- tation component f*() Both have their own T5 encoder-decoder parameters. The two com- ponents are bridged by the augmentation docu- ments, which are retrieved by f () from M and used by MMoMA() () to produce query representa- tion g"
[CLS]
Dec
q
(g,))
BlQAZQ
Aug
Enc
(M
Migg In
Attention Fusing
Enc
WInMICA
Main Model Learning. Given the relevance labels from the source task and an augmentation model, training fMaMA( () is straightforward. We can use the standard dense retrieval training to finetune the enriched query encoder g MMMMA()
Mixture Memory
Figure 1: Illustraion of the Mixture-of-Memon Augmentation.
and the document encoder g():
MuMA ΣΣ Σ T' d+ED di EDA
(A" d PMMMA )):
(7)
D ANN
(MMM .0) D
(8)
The training signals come from the source task, including q", its relevant documents D*+, and ANCE hard negatives D retrieved from the source corpus C
Augmentation Learning. Training S is challenging as it is hard to label whether an augmentation document useful. Propagating gradients from the final loss to f*() is also prohibitive the retrieval operation in Eqn. is discrete. Fortunately, recent research found the attention scores from the FiD decoder to each encoded inputs (Eqn.
is
5
are good approximations to the usefulness of augmentation
documents (Izacard & Grave 2020a):
(idAtt(d))
Σ
Σ
Attentionpe-
MoMA (d
layers positions heads
sums the attentions from g (MMA ()'s special token at the decoder's [CLS) position over all layers, input positions, and attention heads. Ideally, higher FidAtt() is assigned to de that provides useful contextual information.
Previously, FidAu scores often used soft labels for the augmentation model (Izacard & Grave 2020a Izacard et al., 2022). Doing so with memory mixtures is risky as it too sparse and overfits memory resource that appears earlier in the training. which are the only ones available for the decoder to attend on. To improve the learning robustness, we introduce ANCE-style hard negative mining to train the augmentation component as well.
First, we formulate the positive set of augmentation documents as:
Do+ = + U Top-NFaan(d().D
(10)
which combines relevant documents D and the augmenting ones that received N-highest attention scores from g MoMA(). (). Then we pair them with hard negatives to formulate the training of fo() as:
La Σ Σ Σ 10
(11)
d+E d-EDa-
Da- ~ ANN ,)) Dat
(12)
Notice the negatives for f*() have comprehensive coverage from multiple corpora.
Iterative Training. The learning of (MoMA and f²() an iterative process that fits naturally into the training procedure of dense retrieval training with hard negatives. We follow the standard iterations ANCE and construct the 1-th training episode of MoMA-DR:
1. Construct hard negatives D=- via Eqn. 8 Rusing weights MMaMA() () from the last episode;
4

---

Under review as a conference paper at ICLR 2023
2. Retrieve augmentation D via Eqn. Ausing weights fi-10 from the last episode;
3. Train fMoMMA() () as Eqn. 7
4. Formulate new positive augmentation documents D using updated attention scores from MoMMA (), and mine negative augmentation documents D using ft-10:
5. Train f2O following Equ. 11
Both MM () and Jö () can be initialized with a BM25 warmed-up T5 retriever. Steps 1 and above are inherited from standard dense retrieval training. The rest are introduced by MoMA. The additional computation in the training side mainly resides updating the index for the memory mixture, a standard cost in retrieval- -uggeette language models (Guu al.
2020 Izacard al. 2022).
Zero-Shot Retrieval with Plug in Memories. To perform zero-shot retrieval on unseen tasks, MoMA-DR first retrieves augmented documents using f*() from M for the target query q', and retrieves target documents d' C with the augmented model fMoMA(( without changing any model parameters. MoMA allows f*() to attend over the target corpus as well if it isplugged in: . = MUC which conveys in-domain information. The augmenting corpus can also be engineered by users manually to inject their preference or domain knowledge, e.g., as "memory engineering". In this work we focus on swapping out the source corpus for the target corpus; we leave other explorations for future work.
EXPERIMENTAL METHODOLOGIES
Datasets. We choose the MS MARCO passage dataset (Bajaj et al. 2016) as the source domain dataset, whereas the target domains are from the 18 datasets BEIR (Thakur et 2021a) benchmark, which include including biomedical, scientific and financial texts. More details can be found in Appendix A.I The evaluation metric NDCG@ is the same with BEIR benchmark, which measures Normalized Discounted Cumulative Gain (Wang et al. 2013) of top 10 prediction. The higher NDCG@ 10 value indicates better performance.
Augmenting Corpora. During training. the mixture-of -omeroory-mmmr is composed of source training corpus (MARCO), Wikipedia and a medical knowledge graph. We use the Wikipedia chunk pre- possessed by Karpukhin al. (2020) without further processing" [I The medical knowledge graph extracted from the Medical Subject Headings (MeSH an open-source database for indexing and cataloging of biomedical and health-related information. Since it is hierarchical in structure, we linearize it by concatenating spans with text information. During testing, we directly replace MARCO with the corresponding document sets from BEIR. Each task from BEIR is augmented independently. More dataset and preprocessing details can be found in Appendix A.
Baselines. We compare MoMA-DR with standard sparse and dense retrieval models BEIR. We also compare MoMA-DR with advanced approaches that are specifically designed for zero-shot generalization. They involve techniques that are not directly comparable with this paper, including pretraining on extra data, in-domain continuous pretraining, and generating target pairs using another pretrained generative model. Besides, some basclines use larger scale language model as their backbone. We list the details of baselines in Appendix A.2
Implementation Details. For MOMA-DR, we use the same architecture as T5-base (Raffel et al. 2019): 12- 12-lryer Transformer, 768 hidden size. Following Xiong et al. (2020), both the augmentation component and end retriever are first trained using BM25 negatives for 10 epochs. After warming up, we jointly trained the two components for three episodes, each episode including three training epochs. After three joint episodes, the end retriever reaches the best performance on MSMARCO, so select this checkpoint for evaluation. The ratio between positive and hard negative pairs is 1:7 for both models. The main hyperparameters in MoMA-DR include the total number of grounding documents K and the attention threshold number N in Equation T0 We directly set K=10 and N=5 without any parameter tuning. More details on hyperparameters and experimental settings can be found in Appendix A.3
http::/hhwggnnnnacccc.o////////s hhtps:/wwwwncchi. nlm www..ii.oov/mmshh/...
5

---

Under review as a conference paper at ICLR 2023
Table NDCG @ 10 on the BEIR benchmark. The best result each task is marked bold. The second best result each task is underlined. An denotes unfair comparison, as NQ is used in training for GTR. GenQ generated pseudo labels to train an independent model for each task. $ Larger models
Sparse BM25
Single Vecior and Bave Size Models
For Reference
DPR
ANCE
15-ANCE
coCondenser
GenQ'
GTRae:
MoMA-DR
CoIBERT
GTRan:
Parametersif
110M
110M
110M2
110M
66M
110M
1I0M-2
110M
335M-2
TREC-COVID
0.656
0.575
0.654
0.653
0.715
0.619
0.339
0.762
0.677
0.557
BioASQ
0.465
0.232
0.306
0.322
0,318
0.399
0.271
0.372
0.474
0.320
NTCorpus
0.237
0.275
0.307
00.39
0.308
NQ
0.329
0.398
0.446
0.452
0.494
0.358
0.495
0.490
0,524
0.547
HotpotQA
0.456
0.487
0.534
0.535
FiQA-2018
0.236
0.274
0.295
0.294
0,285
0.308
0.349
0,320
0.317
0.424
Signal-IM
0
TREC-NEWS
0.398
0.366
0.382
0,379
0,389
0.39%
0.337
0.413
0,393
0.343
Rohust04
0.437
ArguAna
0.414
0.414
0.415
0.415
0.411
0.493
0,511
0.438
0,233
0.525
Touch6-2020
0.240
0.312
0.182
0.205
Quora
0.789
0.842
0.852
0,836
0.863
0.830
0
0.847
0.854
0.890
DBPodia-cntity
0.281
0.290
0.328
0.347
SCIDOCS
0.158
0.107
0.122
0.115
0.140
0. 143
0.149
0.143
0.145
0.158
Fover
0.669
0.655
0.669
0.660
Clivate-Fever
0.213
0.176
0.198
0.194
0.184
0.175
0.241
0.235
0.184
0.262
SciFact
0.665
0.475
0.507
0.566
0.600
0.644
0,600
0.632
0.671
0.639
CQADupStack
0.299
0.281
0.296
0,283
0,330
0.347
0.357
0,283
0,350
0.384
Avg
0.391
0.399
0.410
5
EVALUATION RESULTS
Our experiments evaluate the zero-shot accuracy of MoMA-DR, its performance with different memory sources, the influence of memory mixture learning, and the benefits of plug-in memory.
5.1 ZERO-SHOP RETRIEVAL ACCURACY
The retrieval accuracy of MoMA-DR and baselines are listed in Table Besides baselines of similar parameter count, we also include larger models (GTRba)) or those using multiple vectors per documents (ColBERT), MoMA-DR shows stronger zero-shot accuracy against previous state-of-the- art methods that do continuous contrastive pretraining (coCondenser), generate pseudo labels (GenQ), or consume additional training signals in both continuous pretraining and finetuning phrases (GTRaase). MoMA-DR also achieved nearly comparable zero-shot accuracy against larger models like GTRaage and CoIBERT, which scales up the number of vectors per documents (one per token). This confirms that -uggevtataugmmnttiiioonn provides another path to improve language models' generalization ability besides scaling up. MoMA-DR also outperforms its direct baseline, T5-ANCE, which MoMA-DR uses as subroutine for retrieval augmentation, on all but one retrieval task, showing the robustly improved generalization ability from plug-in in mixture of memory.
5.2 PERFORMANCE WITH DIFFERENT MEMORIES
Table 2 evaluates how MoMA-DR behaves under different combinations of external memories. Unsurprisingly, using a single out-of-domain memory for retrieval augmentation does not help, for example, even though MARCO is the source domain corpus, solely grounding it reduces zero-shot accuracy. MeSH as the sole augmenting corpus also lowers performance, even on some medical retrieval tasks such as BioASQ. Interestingly, when we expand the memory to include MARCO, Wiki, and MeSH, but keep the target corpus excluded (w/o Target), MoMA-DR exhibits better accuracy compared to the no-memory TS-ANCF. Our conclusion that more memory sources achieves better generalization, especially when target domain information is available.
In the Full setting, the 3-memory mixture of MARCO, Wiki, and McSH is jointly learned with final task at training time. At test time, MARCO is swapped out for the target corpus. The Full improves zero-shot accuracy over both the w/o Target setting (where the target corpus is excluded at test time), and the w/o Learning setting (wherein the augmentation component is not learned). As expected, plugging in the target corpus at test time is the most valuable source of generalization power. It is also the most realistic, access to the target corpus may only be available at testing time.

---

Under review as a conference paper at ICLR 2023
Table 2: NDCG@ 10 of MoMA-DR under different memory compositions: no memory, single memory, and mixture of memories. w/o Learning uses the end retriever to select augmenting documents without use of an augmentation component. w/o Targer excludes the target from memory.
No Memory
Single Memory
Memory Mixture
T5-ANCE
MSMARCO
Wiki
McSH
Target
w/o Learning
w/o Target
Full
TREC-COVID
0.653
0.576
0.592
0.669
0.731
0.759
0.664
0.762
BioASQ
0.322
0.247
0.262
0.219
0.361
0.359
0.271
0.372
NFCorpus
0.275
0.295
0,302
0.282
0.319
0.317
0.307
NQ
0,452
0.472
0.486
0.393
0.483
0,510
0.484
0.490
HotpotQA
0.487
0.481
0.519
0.462
0.538
0.539
0.520
0.539
HiQA-2018
0.294
0.296
0.286
0.280
0.320
0.304
0.2885
0.320
Signal-IM
0.246
0.239
0.225
0.238
0.250
0.248
0.240
0.258
TREC-NEWS
0.379
0.381
0.391
0.372
0,416
0.410
0.398
0.413
Robust04
0.412
0,435
0.443
0,428
0.483
0,446
0,452
0.469
ArguAna
0.415
0.439
0.438
0.442
0.439
0.427
0.438
0.438
Touché-2020
0.312
0.281
0.281
0.252
0.331
0.275
0.272
0.271
Quora
0.836
0.809
0.798
0.835
0.781
0,813
0.812
0.847
DBPedia-entity
0,290
0.340
0,341
0.287
0.335
0.331
0.342
0.347
SCIDOCS
0.115
0.128
0.121
0.130
0.146
0.134
0.127
0.143
Fever
0.655
0.663
0.735
0.610
0.694
0.718
0.737
0.723
Climate-Fever
0.194
0.231
0.238
0.231
0.228
0.222
0.240
0,235
SciFact
0.566
0.583
0.587
0.585
0.624
0.618
0.598
0.632
CQADupStack
0.283
0.207
0.218
0.203
0.283
0.235
0.215
0.283
Avg
0.399
0.395
0.403
0.384
0.431
0.426
0.411
0.436
Table 3: Zero-shot Performances of different distillation methods. We observe consistent trend all BEIR datasets. We present results on 6 representative datasets from Wikipedia or medical domains.
Distillation Method
TREC-COVID
BIOASQ
NFCorpus
HotpotQA
FEVER
Avg
Soft Attention Distill
ADist (Izacard al. 2022
0.609
0.185
0.227
0.351
0.387
0.615
0.396
ADist MSMARCO
0.664
0.220
0.255
0.397
0.394
0.624
0.426
w/o Distilling (Fixed)
0.741
0.361
0.301
0.472
0.513
0.684
0.512
MoMA-DR
0.762
0.372
0.307
0.490
0.539
0.723
0.532
5.3 EFFECT OF MEMORY MIXTURE LEARNING
To study the effect our joint learning mechanism on the memory mixture, we compare it with recent state-of-the-ar Attention Distillation (ADist), which is first used in Izacard & Grave (2020a) and recently updated in parallel work Izacard et al. (2022). It jointly trains the augmentation model using attention scores from the end language model as pseudo-labels. We also enrich ADist with relevance labels from MARCO for more direct supervision, which was shown to be effective in distilling dense retriever from stronger cross-encoder ranking model (Hofstätter al. 202T).
The performances of these joint learning methods are listed in Table 3 We pick six BEIR tasks whose domains are closely related to the augmentation corpora: COVID, BIOASQ. and NFCorpus are medical search and closely related to MeSH. NQ. HotpotQA, and FEVER are all Wikipedia based. The results show that ADist, either standalone or enriched with MARCO labels, does not improve the final accuracy compared to using supervised dense retriever as the augmentation component without joint learning. The main difference that the supervised retriever has been trained effectively using hard negative sampling (Xiong et al.) (020). Jointly learning using soft labels without hard negatives downgraded the augmentation accuracy. Hence, MoMA-DR is a simple technique to learn the end task signals via the attention scores together with hard negatives, which improves quality over a supervised retriever alone.
To further illustrate the joint training process, track the attention scores of documents from different memory sources as well as their ratio in the augmentation set in Figure 2 We also split MARCO documents by whether they are labeled Relevant (Rel) for the corresponding query.
Firstly, MoMA-DR learns to increasingly attend to, and retrieve, relevant documents from the memory mixture throughout training. In Figure 2a more attention is paid to MARCO Relevant documents

---

Under review as a conference paper at ICLR 2023
0.5
0.5
1.0
1.0
0.8
0.6
0.2
0.2
0.4
0.4
MoSH
Marto Rol
0.0
0.0
0.0
0.0
Eph2
(a) MoMA-DR AIL Score.
(b) ADist Att. Score.
MoMA-DR Doc Ratio.
(d) ADist Doc Ratio.
Figure 2: Grounding component breakdown for different distillation methods in each learning iteration.
We display the regularized doc and att. score ratio of documents from different augmentation sources.
100
60
Target
20
Wikii
McSH
HotpotQA
NFCorpus TRBC Covid
FEVER
NFCorpus CoEd
(a) Doc Ratio. (Wiki)
(b) Doc Ratio. (Med)
(c) Att. Score Ratio. (Wiki) (d) Att. Score Ratio. (Med)
Figure 3: The inclusion of Plug-In memory during testing (grouped by the Wiki and Medical domains).
than to any other type in the memory. Although the number of MARCO Relevant documents is not significant as a percentage of the augmenting set in Figure 2c a query level analysis confirms that percentage of queries having at least one relevant document in the augmenting set increases from 46% in Epi-O 62% Epi-2.
This apparent discrepancy can be explained by the fact that MARCO has only one relevant label per query on average, leaving plenty room for other types of documents be included in the augmenting set.
Secondly, the amount of attention paid to certain types documents by MoMA-DR is positively correlated with their representation in augmenting set. This confirms that the joint learning effectively conveys the feedback signals from the end model to the augmen- tation component. For instance, in Figure 2a MoMA-DR pays a high level of attention to MARCO Other documents, signal flected in the composition of its augmentation set in Figure 2c Even though MARCO Other documents were labeled relevant for the query, they can still prove to be valuable as an augmenting document because they may contain partial information that helps query under- standing (Lavrenko & Crofi 2017) or it was simply not annotated MARCO's ss sparse labels (Bajaj et al. 2016). In comparison, the correlation of the two in ADist is weak as the model seems to include 60% augmenting documents from MeSH, far greater than the fraction of medical queries in MARCO.
0.40
0.36
0.34
Aug
0.32
Epi-0 Epi-1
Epi-2
Epi-3
Figure 4: MRR on MS- MARCO of the augmenta- tion component and end re- triever during training.
Figure demonstrates how the augmentation model and end retriever interact. We measure MRR on MARCO for both components during training. Firstly, the augmentation component improves the source domain even thought it is not directly optimized with relevance labels. Secondly, the end retriever monotonically benefits from information collected by the augmenting component in each iteration, indicating that the two components mutually enhance each other in the joint learning process, and the high percentage of MARCO Other documents still ultimately benefit the end-retriever.
8

---

Under review as a conference paper at ICLR 2023
Table 4: MoMA-DR retrieves augmenting documents during training (Marco) and testing (BEIR).
Queries
Augmentation Doc #1
Augmentation Doc #2
Training
[Marco] What is
[Marco] Why is Hotel Transylvania 2 rated PG? It is rated PG for some scary images, action and rude humor.
[Wiki] Another review aggregate calcu- lated an average score of 47 out of 100, indicating "mixed average reviews".
hotel transylvania
rated
Zero-Shot Testing
[HotpotQA] Were
[Wiki] Scott Derrickson (born July 16, 1966) is American director, screen- writer and producer.
[HoipotQA] Edward Davis Wood Jr. (October 10, December 10, 1978) was American filmmaker, actor, writer, pro- ducer, and director.
Scott
Derrickson
and Ed Wood of same nationality?
[BIOASQ] Is AND-
[BIOASQ] AND-I/CIf4 bridges the CMG helicase and DNA polymerase al- pha, facilitating replication.
[Wiki] FADD has no effect on the prolif- eration of cells induced by stimulation of the cell receptor.
I/Ctf4 essential for
proliferation?
5.4 GENERALIZATION OF PLUG-IN MEMORY
In the previous section, observed how MoMA-DR learns to attend to, and retrieve, informative documents from memories on which it was trained. In this section, we examine MoMA-DR's zero-shot behavior new corpora plugged-in at test time (keeping Wiki and MeSH as before).
Figure 3 compares documents from the plugged-in in target versus the remaining memory mixture in terms of membership in the augmenting set (Doe Ratio) and attention. Again, all tasks, MoMA- DR heavily attends to 2--mm and successfully retrieves in-domain documents, even if those in-domain documents were only just plugged in. This confirms that the augmentation model achieves the zero-shot ability to capture relevant information from unseen corpora.
In the medical domain, the model pays more attention to MeSH documents, especially on TREC- Covid task since MeSH includes high quality updated information related to COVID-19. Wikipedia documents received more attention on the Wiki-centric tasks like FEVER, as expected. Some tasks may need a small amount of precise information from Wikipedia to answer the detailed question, e.g. in HotpotQA. Similar with the training process, there a non-trivial correspondence between attention score of a memory and its membership in the augmentation set.
5.5
CASE STUDIES
In Table 4 we show examples of how augmenting documents chosen by MoMA-DR can provide valuable contextual information for the query. The first example a training query from MARCO, where the augmenting documents help disambiguate the query word "rating". In the second one, documents from the official Wiki Dump and HotpotQA's Wiki corpus are descriptions of the two entities in HotpotQA's comparison question. It illustrates the how MoMA-DR provides more comprehensive augmentation by incorporating information from different sources. The last query shows the benefit of the in-domain plug-in corpus brings in very specific information about the query (AND- that is hard to find elsewhere
6 CONCLUSION
In this paper we propose a new mixture-of of-memory mechanism to allow language models to leverage information across multiple disparate corpora (memories) simultaneously. This mechanism can also incorporate new corpora that are "plugged-in" test time, which improves dense retrieval models' generalization abilities to unseen corpora in zero-shot manner.
The results show that MoMA-DR achieves strong zero-shot accuracy on the eighteen retrieval tasks included in BEIR benchmark, showing that retrieval augmentation with plug-in mixture-of-memorte is another way to improve zero-shot ability of language models. Our analysis demonstrates that the most valuable memory mixtures from multiple sources with in-domain information and joint learning mechanism can utilize such diverse information. We hope our findings inspire future research in
language models to achieve generalization ability with better efficiency.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To enhance reproducibility, we provide an overall introduction on our experimental setting in Section 4 Beyond that, we present more details in Appendix. Appendix A. includes statistics on the evaluation datasets and augmenting corpora. Appendix provides the introduction and implementation of all baselines the paper. Appendix complete choices of hyperparameters. We plan to submit our code after the discussion forums are opened. We will make comment containing a link to our anonymous repository directly to the reviewers and ACS so that our code is only internally visible. We will release all code, augmentation data and model checkpoints, along with analysis scripts if this work is accepted.
the configuration of our experimental setting and the
REFERENCES
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. MS MARCO: human generated machine reading comprehension dataset. arXiv preprint arXiv:1611 0.0966, 2016.
Emily M Bender, Timnit Gebru, Angelina McMillan-Mann Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pp. 610-623, 2021.
Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen. Overview of Touché 2020: Argument Retrieval. In Linda Cappellato, Carsten Eickhoff, Nicola Ferro, and Aurélie Névéol (eds.). Working Notes Papers of the CLEF 2020 Evaluation Labs, volume 2696 of CEUR Workshop Proceedings, September 2020. URL http: ceur-ws
org /rg/Vol-2e 26666
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Tean-Baptisse Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pp.
PMLR, 2022.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning rank dataset for medical information retrieval. In European Conference on Information Retrieval, pp. 716-722. Springer, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1870-1879, 2017.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. SPECTER: Document- level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2270-2282 Online, July 2020. Association for Computational Linguistics. URL https: /tttps:/aaclaat httpa:://acaaathhoooooo org/ 2020.ac1 i 102
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. CEV::: dataset for verification of real-world climate claims. arXiv preprint arXiv:20/2.006/1 2020.
Thibault Formal, Benjamin Piwowarkki. and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2288-2292. 2021.
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre- vre-training for dense passage retrieval. In ACI. 2022, 2022.
10

---

Under review as a conference paper at ICLR 2023
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: Retrieval- augmented language model pre-training. In ICML, 2020.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. DBpedia- Entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 17, pp. 1265-1268, New York. NY, USA, 2017. Association for Computing Machinery. ISBN 9781 4783452050 doi: 10.1145/30711333008777 URL UtLhtttps://dat org/
10. 00111153377133 3080751
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimas large language models. arXiv preprint arXiv:2203. 15556, 2022.
Sehastian Hofstätter, Sheng-Chier Lin, Jheng-Hong Yang. Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of SIGIR 2021, pp. 122, 2021. URL ht //dl acm org/ abs
EAMOHRY wttpdddllggggiinnen
M1
Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 122. Association for Computing Machinery, 2021. ISBN 9781450380379. doi: 10.1 1445/34444855346222 URL https::t:: http::://doooooo///////// 1ee.e114533440 01550..0
Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. CQADupStack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium, ADCS '5, New York, NY, USA, 2015. Association for Computing Ma- chinery. ISBN 9781450340403.
28389933
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:2012.04584, 2020a. URL httpp::/aaxiv..or/////0
20112.4584
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2020b. URL
0128
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning, arXiv preprint arXiv:2112.09118 2021.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Y Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.032900 2022.
Jeff Johnson. Matthijs Douze, and Hervé Jégou. Billion-scal. similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint orXiv:2001.08361 2020.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 191.00071 2019.
11

---

Under review as a conference paper at ICLR 2023
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 39-48, New York, NY, USA, 2020a. Association for Computing Machinery. ISBN 9781450380164 URL
1145/3397271 3401075
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39 48, 2020b.
Yubin Kim. Applications and future of dense retrieval in industry. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 3373-33744 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, March 2019. URL https://
ac aclanthology www.aanoooooooooog/11----01 wwtp/or-/////////0.
Victor Lavrenko and W Bruce Croft. Relevance- based language models. In ACM SIGIR Forum, volume 51, pp. 260-267. ACM New York, NY, USA, 2017.
Kenton Lee, Ming-Wei Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of ACL 2019, pp. 6086-6096, 2019. URL
btttss://aclanthooooyy.oom
1612/
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmeente genera- tion for knowledge-intensive nip tasks. arXiv preprint arXiv:2005. //401, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv: 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL hnttps://openreeeee net/ forum? id=
Jing Lu. Gustavo Hernandez Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. Multi-stage training with improved negative contrast for neural passage retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6091-6103 Online and Punta Cana, Do- minican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2021 www.leemmmnnnneee URL (ttt::::: /tc///aaaaaa// www.annnoooooyy.. org/ 2221.0211112 emnlp ma main. 492
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. WWW' 18 open challenge: Financial opinion mining and question answer- ing. Companion Proceedings of the The Web Conference 2018, WWW '18, pp. 1941-1942, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steer- ing Committee. ISBN 9781450356404, URL https: org 1111//01111///111555113
3192301
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201. 2022.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:21/2.07889) 2021.
12

---

Under review as a conference paper at ICLR 2023
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- perforrerem deep learning library. Advances in neural information processing systems, 32. 2019.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite. Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009. 02252, 2020.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5835-5847, Online, June 2021. Association for Computational Linguistics. URL
https: ac antho ogy .rg.or/gg............o.. 2021.122 nwa.caacc...c mma i In. 466
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4)3333-389 2009.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megatron- turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 2022.
Ian Soboroff, Shudong Huang, and Donna Harman. Trec 2018 news track overview. 2018.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13693-13696, 2020.
Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, and José Esquivel. A data collection for evaluating the retrieval of related tweets news articles. In European Conference on Information Retrieval, pp. 780-786. Springer, 2018.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 2021a
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprins arXiv:2104.08663, 2021b.
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: large- scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies. Volume 1 (Long Papers). pp. 809 819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL. https:::p///// htap:///callataaac ooy.. wrg.Norr/NN N01g//N18-01 8 1074
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenbom, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. An overview of the BIOASQ large-scale biomedical semantic indexing and question 2015.
answering competition. BMC bioinformatics,
13

---

Under review as a conference paper at ICLR 2023
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fusshan Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. TREC-COVID: Constructing a pandemic information retrieval test collection. SIGIR Forum, 54(1), February 2021. ISSN 0163-5840, URL
https://doo www.odii...ggggooo www.ooog............o.oo 1145/345 964. 3451 1655
Ellen Voorhees et al. Overview of the tree 2004 robust retricval track. In Trec. pp. 69-77, 2004.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers). pp. 241-251, Melbourne, Australia, July 2018. Association for Computational Linguisties. URL httpp:: http///aacaaatooooy.ooo.o org/ P18 1023
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 7534 7550, Online, November 2020. Association for Computational Linguistics. URL https:/7
www.athllog..ogg//200. 20mp://1122.mmm1p..m.mmmmmm emnlp meteemmaii...... 609
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computtaioonal Linguistics: Human Language Technologies, Seattle. United States, July 2022. Association for Computational Linguis- tics. doi: 10. 3/653//1/2022.naacc/
168. URL https: htlLhhttt:://a///l/ /tattss//aclaattholor.ooooo
naacl-maing 168
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, Wei Chen, and Tie-Yan Liu. A theoretical analysis of ndeg ranking measures. Proceedings of the 26th annual conference on learning theory (COLT 2013), volume 8, pp. 6. Citeseer, 2013.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers:
-fate-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.11.65/11. 1/2000.cmmmppddemm.... URL
httpss//aclanttioooyy.ooo org. /2gg/2020.c
Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. Zero-shot dense retrieval with momentum adversarial domain invariant representations. arXiv preprint 1r.05:2110.07581, 2021.
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. Zero- shot dense retrieval with momentum adversarial domain invariant representations. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4008 4020, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https:ttps... / ac anthol log.. org/ 2022. f Findings-aae 6366
Lee Xiong, Chenyan Xiong, Ye Li. Kwok-Fung Tang. Jialin Liu, Paul Bennett, Junaid Ahmed, and Amold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007. 00000. 2020.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1369-2380 2018.
HongChien Yu, Chenyan Xiong, and Jamie Callan. Improving query representations for dense retrieval with pseudo relevance feedback. arXiv preprint arXiv:210S 13454, 2021.
14

---

Under review as a conference paper at ICLR 2023
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber and Hal Daumé III. Distandy-supprrvssed evidence retrieval enables questiion answering without evidence annotation. arXiv preprini arXiv:21 2021. URL
abs 04889
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. arXiv preprint arXiv:2205./2674 2022.
15

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Table 5: Statistics of datasets in the BEIR benchmark. The table is taken from the original BEIR benchmark paper Thakur al. (222la).
Truin
Dev 4Qwery
Avg. Word Lengths
(J)
Domain
Relevancy
#Paies
AQuery
#Corpus
Aug. D/Q
Query
Document
Plesage
Misc.
Binuy
552,761
8,541,823
55.98
Bio Medical
Bio-Medical
TREC-COVID
V
3-level
50
171,330
493.5
10.50
160.77
Information
Rin-Medical
NFCorpus
3-level
110,575
324
333
3,633
36.2
3.50
252.26
Resieval
Bio Medical
DioASQ
Dinary
32,96
500
14,944,022
4.7
8.05
202.61
Qsestion Answering
Wikipedia
Binary Binryy
Wikipedia
5,238,329
(UAA
PAJA-201
14,166
57,638
2.6
10.77
132.32
Tweer- Retrivval
Twiter
Signal- IM (KT)
X
3-evel
97
2,866,316
19.6
9.30
13.93
News
News
TREC-NEWS
V
S-leeel
57
$94,977
19.6
11.14
634.79
Retileval
News
Robus04
3-level
249
528.155
6919
15.27
460.40
Argument Reuieval
Misc:
ArguAna
Binary
1,406
8,74
1.0
192.88
166.80
Mise.
Touché 2020
3-level
382,545
292.37
Duplicate-quetion
CQADipSuck
Resieval
Quora
Quera
Binury
5,000
10,000
$22,931
Fncit-RRttizval
Wikipedis
DBPalia
3-level
4,635,922
49 68
Closion-Peediee
Seiemtiic
SCIDOCS
V
Binary
1,000
25.657
4.9
9.38
176. 19
Wikipedia
FEVER
V
Binary Rinary Dinary
140,085
6,666
6.666
5,416,568
1.2
X.13
84-76
Faet Checking
Wikipedis
Clinut-FEVER
1,535
5,466,90 663
3.0
20.13
84 76
Scienttic
SeiFact
920
200
5,183
12:37
213.63
A.1
DATASETS DETAILS
Evaluation Datasets Target domain datasets used in our experiments are collected in the BEIR benchmark (Thakur et al. 2021a and include the following domains:
Open-domain Question Answering (QA): HotpotQA (Yang al. 2018). NQ (Kwiatkowski et al. 2019), and FiQA (Maia et al. 2018).
Bio-Medical Information Retrieval: TREC-COVID (Voorhees et al. ct al. 2016), and BioASQ (Tsatsaronis al.
2021 NFCorpus (Boteva
2015).
Argument Retrieval: Webis-Touche22220 (Bondarenko et al. 2020) and ArguAna
(Wachsmuth
ct al. 2018).
News Retrieval: TREC-NEWS (Soboroff et al.
2018) and Robust04 (Voorhees et al. 2004).
Tweet Retrieval: Signal- Suarez et al.
2018).
Duplicate Question Retrieval: Quora (Thakur et al.
2021a) and CQADupStack (Hoogeveen et al.
2015).
Entity Retrieval: DBPedia (Hasibi et al.
2017
Citation Prediction: SCIDOCS (Cohan et al.
2020
Fact Checking: SciFact [Wadden et al.
2020
FEVER (Thome et al., 2018), and Climate-
FEVER (Diggelmann et al. 2020)
We list the statistics of the BEIR benchmark in Table
Augmenting Corpora Corpus size We first introduce more details on how preprocessed the Medical Subject Headings (MeSH) Database. We select text information from the Qualifier Record Set and Descriptor Record Set. Each set contains multiple <Concept> elements, which is composed of three sub-elecments, i.e., <ConceptName>, <ScopeNote> and <TermList>. Among the sub- elecments, <ScopeNote>> is the major textual information source, which is usually short description to medical term or phenomenon. We directly consider each <ScopeNote> as document entry and concatenate with corresponding <ConceptName>.
We list the statistics of the augmenting corpora in Table
htttp:://github.com http:///eerrreeeeeaeaaaar
16

---

Under review as a conference paper at ICLR 2023
Table 6: Statistics of the augmenting corpora.
Datasets
Corpus Size
Avg. Doc Length
MS MARCO
502,939
56.0
MeSH Wiki
32.326
16.8
21,015,324
100.0
A.2 BASELINES
We use the baselines from the current BEIR leaderboard (Thakur et al. 2021a) and recent papers. These baselines can be divided into four groups: dense retrieval, dense retrieval with generated querie lexical retrieval and late interaction.
Dense Retrieval For dense retrieval, the baselines are the same dual-tower model as ours. We consider DPR (Karpukhin al. 2020), ANCE (Xiong al. 2020), T5-ANCE, coCondenser Gao & Callan 2022) and one -econnly-proposed model GTR (Ni et al. 2021 with different size configuration in this paper.
DPR uses a single BM25 retrieval example and in-batch examples as hard negative examples to train the model. Different from the original paper Thakur et al. (2021a) that train the DPR QA datasets, we train DPR on MS MARCO Bajaj et al. (2016) Dataset for fair comparison. Notice that this also lead to better results according to Xin et al. (2022).
ANCE constructs hard negative examples from ANN index of the corpus. The hard nega- tive training instances are updated in parallel during fine-tuning of the model. The model is a RoBERTa LiiErTaLLise et al. (2019) model trained on MS MARCO for 600k steps.
T5-ANCE Different with default ANCE setting, replace the backbone language model RoBERTa with T5-base. All the other model settings are the same with the original ANCE. We include this baseline because as subroutine for MoMA-DR, it could be viewed as ablation without memory augmentation. We can directly observe the impact plug-in mixture of memory by comparing T5-ANCE with MoMA-DR.
coCondenser is a continuous pre-trained model based on BERT, with the equivalent amount of parameters to BERT-base. It enhances the representation ability of [CLS] token by changing the connections between different layers of Transformer blocks. Fine-tuning of coCondenser uses BM25 and self-mined negatives.
GTR initializes the dual encoders from the T5 models Raffel al. (2019). It is first pre-trained on Community with 2 billion question-answer pairs then fine-tuned on NQ and MS Marco dataset. In addition, they use the hard negatives released by RocketQA Qu et al. (202 when finetuning with MS Marco data and the hard negatives release by Lu et al. (2021) for Natural Questions. GoRee leverages the same ba-e model as MoMA-DR, while GTRunge is based on T5-large, which is not directly comparable to our method as it triples the parameters.
Dense Retrieval with Generated Queries GenQ first fine-tunes a T5-base (Raffel et al. 2019) model on MS MARCO for epochs and then generate 5 queries for each passage additional training data for the target domain to continue to fine-tune the TAS-B (Hofstätter et al. 2021) model.
Lexical Retrieval Lexical retrieval is score function for token matching calculated between two high-dimensional sparse vectors with token weights. BM25 (Robertson et al.] (009)) is the most commonly used lexical retrieval function. use the BM25 results reported Thakur et al. (202 for comparison.
Late Interaction We also consider a late interaction baseline, namely ColBERT (Khattab & Zaharia 2020a). The model computes multiple contextualized embeddings for each token of queries
"We separate them from dense retrieval since they usually rely on Seq2seq models to generate pseudo query-document pairs, and they train a model for each datasct independently instcad of using a single model for all datasets.
UUnfortunately, this corpus has not heen released by the authors.
17

---

Under review as a conference paper at ICLR 2023
Table 7: The hyperparameters of MoMA-DR.
Hyperparameters
Settings
Grounding document number
Attention threshold number
Negative mining depth
200
Global batch size (query size per batch)
256
Positive number per query
1
Negative number per query
7
Peak learnig rate
5e-6
Learnig rate decay
Optimizer
AdamW
Scheduler
Linear
MARCO Maximum query length
32
MARCO Maximum document length
128
Table 8: Training time for MoMA-DR with three training episodes. We use 8 Nvidia A100 80GB GPUs with FP16 mixed-precision training.
Stage
Augmentation Component
End Retriever
Epi-1
0.8h
1.5h
Epi-2
0.8h
1.5h
Epi-3
0,8h
1.5h
Index refresh
1.4h
0.6h
Refresh number
Overall
6.6h
6.3h
and documents, and then maximum similarity function to retrieve relevant documents. This type of matching requires significantly more disk space for indexes and has a higher latency.
A.3 DETAILED EXPERIMENTAL SETTINGS AND IYPERPARAMETERS
Our implementation uses PyTorch (Paszke et 2019 with Hugging Face Transformers (Wolf et al., 2020). We optimize the model using Adam W (Loshchilov & Hutter 2019) with peak learning rate at 5e-6, weight decay of 0.01, and linear learning rate decay. The global batch size is set to 256. The maximum length of query and passage are set to 32 and 1 128 respectively. We summarize all hyperparameter settings in Table 7 The model trained with 8 Nvidia A A100 80GB GPUs and FP16 mixed-preee pprecisin training. The total rumning time is 6.6 hrs for three episodes of augmentation component training and 6.3 hrs for end retriever training. We detail the training time of each episode in Table
When evaluating on the BEIR benchmark, we follow the setting in GTR (Ni et al. 2021 which use sequences of 64 tokens for the questions and 512 for the documents in all datasets except Trec-News, Robust-04 and ArguAna. In particular, we set the document length to 768 for Trec-News and Robust- 04. For ArguAna, set both question and document length 128. The above length setting is in accordance to the average query and document lengths in these datasets.
18


Reviews:
[{"review_id": "yMGNHm2aDJ", "reviewer": "Reviewer_VaeQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a mixture-of-memory augmentation framework for a zero-shot dense retrieval (MoMA-DR). The key idea is to enrich the query representation with external resources such as wiki, knowledge graphs, etc. The framework consists of two components: a retrieval model and an augmentation model. The retrieval model follows the usual dense retriever settings, and the augmentation model is used to select some relevant/useful documents from external resources for query representation enrichment. To verify the effectiveness of this idea, they conducted experiments on the BEIR benchmark datasets and showed that it could provide some performance boost.", "summary_of_the_review": "This paper is overall well-written, and the idea is also well-motivated. The proposed learning paradigm is novel, although the idea of query expansion is not original. The performance improvement is not that encouraging. Some related work and stronger baselines are not included for discussion/comparison. The expensive augmentation process is done online, which can heavily limit its use in practical/industrial environments.", "strength_and_weaknesses": "### Strength:\n- Making the query augmentation process trainable and automatic is interesting and novel in the current dense retrieval field.  The iterative joint optimization mechanism of the retrieval model and the augmentation model is natural. \n- Detailed study on the impact of different memories and the joint learning mechanism.\n-The paper is easy to follow and well-orgnaized.\n\n### Weaknesses:\n- This idea is similar to query augmentation methods [a], but it seems that this related line of research is not well explained and reviewed in the paper. \n- Preparing the external resource is not a trivial thing (the impact varies on different datasets). Important questions about this, such as what resources to include and where to find them, are not answered. Given the difficulty of finding suitable external resources and the high cost of maintaining them, the applicability/effectiveness of this framework is limited.\n- The reported performance is not that promising. Compared with a recent work contriever [b], this model performs much worse. Also, if we include hybrid models such as BM25+CE, the results of MoMA-DR are even less attractive. \n\n\n- [a] A Survey of Automatic Query Expansion in Information Retrieval, CSUR, 2012\n- [b] Unsupervised Dense Information Retrieval with Contrastive Learning, JMLR, 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mumc80H8Fbu", "reviewer": "Reviewer_DtfB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new way of dense retrieval using a retrieval augmented model. The modeling differs from previous models in the sense that it allows integrating new corpora that are never used during the training. This allows the proposed method to perform zero-shot retrieval nicely on the new domain that is just integrated at the inference time. Authors evaluated the proposed method in the BEIR benchmark.", "summary_of_the_review": "At the current state of the paper, it does not have sufficient discussion towards highly relevant recent work; however, I would like to hear what authors’ thoughts are on the discussion points.", "strength_and_weaknesses": "**Strength**\n* Authors provide various ablations that were helpful to understand the proposal.\n* Authors provide benchmarks on a large set of corpus.\n\n**Weakness**\n* Generalizable retrieval augmented language model were proposed by [1] and it does also covers zero-shot scenario (e.g. Table 6) and the idea is a bit similar. I encourage authors to discuss and compare thoroughly.\n* Since the embedding function, $f_a$, needs to be compatible with the new corpus, it is undeniable this method implicitly has some assumptions to the new corpora. For example, if the $f_a$ function performs only on the text inputs, a new corpora with product hierarchy codes will not work well. \n* Since the proposed work assumes there is a given retrieval stack, comparing to the model with a similar number of parameters in DNN is not fair. For example T5-ANCE is similar in their size of the network to MOMA-DR, but MOMA-DR is actually much larger in their space/inference cost. The size of the database should be also considered for the fair comparison.\n* Since this paper heavily relies on retrieval augmented language models, authors should also empirically compare to some popular retrieval augmented models (RETRO/REALM/FiD). All the baselines that authors considered are not retrieval augmented models.\n* I am not sure why ColBERT cannot be part of the comparison. It is actually much smaller than MoMA-DR and even without any database (but it largely outperforms on many tasks?).\n\n[1] Izacard, Gautier, et al. \"Few-shot learning with retrieval augmented language models.\" arXiv preprint arXiv:2208.03299 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "lxoRz3JVyK3", "reviewer": "Reviewer_U4cd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new retrieval algorithms by enriching the representation of the query with augmentation documents from multiple information corpora. ", "summary_of_the_review": "The proposed method showed better generalization abilities over the retrieval tasks. However, it's unclear what's the added compute cost. ", "strength_and_weaknesses": "Strengths:\nThe proposed algorithm improves retrieval models’ generalization abilities in term of zero-shot accuracy on the various retrieval tasks\n\n\nWeaknesses/questions: \n\n1. One claim made by this paper is the better efficiency. I'm curious to understand the system performance overhead (flops, training step time, serving latency etc) introduced by this mixture of memory method. For example, eq-5 required multiple runs of the same encoder with different data, I assume it would significantly increase the compute flops required for a q-d matching. \n\n2. How do the MoMA approaches compare to other baseline models with matching compute? Table 1 shows the parameter count, however, it doesn't translate to efficiency or compute cost directly. What's the compute cost for MoMa-DR comparing to that of GTR_large?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_KAEyVmbyXm", "reviewer": "Reviewer_KU9C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method called MoMA, which leverages multiple external text corpus for retrieval-augmented language models on zero-shot dense retrieval tasks. MoMA has an augmentation component for retrieving several augmentation document and a main model for measuring the relevance between the query with these augmentation document and the target document. Both models take sentence-T5 as their model architecture and trained iteratively using ANCE. Experiments on 18 datasets in BEIR demonstrate the effectiveness of MoMA.", "summary_of_the_review": "Experimental settings need further explanation. I'm curious why not just following the settings in those papers listed in the related work sections. it can be found that most dense retrieval models (including MoMA-DR w/o Target) performs worse than BM25 (a very simple baseline) and even MoMA-DR  exceeds BM25 with a slight margin. What's the cause for this phenomenon? Is the source training corpus (MS MARCO passage) and/or being unrelated to the BEIR datasets?\n\nBesides, it can be found in Table 2 that the target dataset is the core for MoMA-DR compared to external corpus (i.e., the 5th column versus the 8th column, 0.431 versus 0.436), which weakens the claim that multiple external corpus used in MoMA is helpful for zeroDR. Is it possible to make some case study to show that the external corpus does contribute some knowledge that is uncovered in the target dataset?\n", "strength_and_weaknesses": "Strength: well-motivated idea\nWeakness: Experiments can be further improved\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject