Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new retrieval algorithms by enriching the representation of the query with augmentation documents from multiple information corpora. 
strength_and_weaknesses: Strengths:
The proposed algorithm improves retrieval models’ generalization abilities in term of zero-shot accuracy on the various retrieval tasks


Weaknesses/questions: 

1. One claim made by this paper is the better efficiency. I'm curious to understand the system performance overhead (flops, training step time, serving latency etc) introduced by this mixture of memory method. For example, eq-5 required multiple runs of the same encoder with different data, I assume it would significantly increase the compute flops required for a q-d matching. 

2. How do the MoMA approaches compare to other baseline models with matching compute? Table 1 shows the parameter count, however, it doesn't translate to efficiency or compute cost directly. What's the compute cost for MoMa-DR comparing to that of GTR_large?


clarity,_quality,_novelty_and_reproducibility: The description of the proposed algorithm and the related work are well written. 
summary_of_the_review: The proposed method showed better generalization abilities over the retrieval tasks. However, it's unclear what's the added compute cost.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new way of dense retrieval using a retrieval augmented model. The modeling differs from previous models in the sense that it allows integrating new corpora that are never used during the training. This allows the proposed method to perform zero-shot retrieval nicely on the new domain that is just integrated at the inference time. Authors evaluated the proposed method in the BEIR benchmark.
strength_and_weaknesses: **Strength**
* Authors provide various ablations that were helpful to understand the proposal.
* Authors provide benchmarks on a large set of corpus.

**Weakness**
* Generalizable retrieval augmented language model were proposed by [1] and it does also covers zero-shot scenario (e.g. Table 6) and the idea is a bit similar. I encourage authors to discuss and compare thoroughly.
* Since the embedding function, $f_a$, needs to be compatible with the new corpus, it is undeniable this method implicitly has some assumptions to the new corpora. For example, if the $f_a$ function performs only on the text inputs, a new corpora with product hierarchy codes will not work well. 
* Since the proposed work assumes there is a given retrieval stack, comparing to the model with a similar number of parameters in DNN is not fair. For example T5-ANCE is similar in their size of the network to MOMA-DR, but MOMA-DR is actually much larger in their space/inference cost. The size of the database should be also considered for the fair comparison.
* Since this paper heavily relies on retrieval augmented language models, authors should also empirically compare to some popular retrieval augmented models (RETRO/REALM/FiD). All the baselines that authors considered are not retrieval augmented models.
* I am not sure why ColBERT cannot be part of the comparison. It is actually much smaller than MoMA-DR and even without any database (but it largely outperforms on many tasks?).

[1] Izacard, Gautier, et al. "Few-shot learning with retrieval augmented language models." arXiv preprint arXiv:2208.03299 (2022).

clarity,_quality,_novelty_and_reproducibility: **Clarity**
* Mostly easy to follow.
* Few suggestions: move Figure 1 and high-level description of the method at the beginning of the paper.

**Novelty**
* Please see the weakness #1 for the novelty discussion. Since [1] proposed a fairly similar generalizable retrieval augmented language model (it’s also based on joint learning), the novelty of this paper is a bit limited.

summary_of_the_review: At the current state of the paper, it does not have sufficient discussion towards highly relevant recent work; however, I would like to hear what authors’ thoughts are on the discussion points.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a mixture-of-memory augmentation framework for a zero-shot dense retrieval (MoMA-DR). The key idea is to enrich the query representation with external resources such as wiki, knowledge graphs, etc. The framework consists of two components: a retrieval model and an augmentation model. The retrieval model follows the usual dense retriever settings, and the augmentation model is used to select some relevant/useful documents from external resources for query representation enrichment. To verify the effectiveness of this idea, they conducted experiments on the BEIR benchmark datasets and showed that it could provide some performance boost.
strength_and_weaknesses: ### Strength:
- Making the query augmentation process trainable and automatic is interesting and novel in the current dense retrieval field.  The iterative joint optimization mechanism of the retrieval model and the augmentation model is natural. 
- Detailed study on the impact of different memories and the joint learning mechanism.
-The paper is easy to follow and well-orgnaized.

### Weaknesses:
- This idea is similar to query augmentation methods [a], but it seems that this related line of research is not well explained and reviewed in the paper. 
- Preparing the external resource is not a trivial thing (the impact varies on different datasets). Important questions about this, such as what resources to include and where to find them, are not answered. Given the difficulty of finding suitable external resources and the high cost of maintaining them, the applicability/effectiveness of this framework is limited.
- The reported performance is not that promising. Compared with a recent work contriever [b], this model performs much worse. Also, if we include hybrid models such as BM25+CE, the results of MoMA-DR are even less attractive. 


- [a] A Survey of Automatic Query Expansion in Information Retrieval, CSUR, 2012
- [b] Unsupervised Dense Information Retrieval with Contrastive Learning, JMLR, 2021
clarity,_quality,_novelty_and_reproducibility: ### Clarity
I do have a few questions that may need clarification from the authors:
- What is the computational cost of the proposed framework in terms of training and inference, and how is it compared with other approaches? In my understanding, if the augmentation corpus is very large (e.g., much larger than the corpus of interest), the process of retrieving augmented documents can be very costly. And this process can not be taken offline as it is query specific.
- Could you give more details on the "BM25 warmed-up T5 retriever"?
- What is the impact of K and N? Why not tune them? Does tuning them bring additional benefits or worsen the model?
- What are the training datasets of these dense retriever baselines?

### Reproducibility
The source code is not available.




summary_of_the_review: This paper is overall well-written, and the idea is also well-motivated. The proposed learning paradigm is novel, although the idea of query expansion is not original. The performance improvement is not that encouraging. Some related work and stronger baselines are not included for discussion/comparison. The expensive augmentation process is done online, which can heavily limit its use in practical/industrial environments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method called MoMA, which leverages multiple external text corpus for retrieval-augmented language models on zero-shot dense retrieval tasks. MoMA has an augmentation component for retrieving several augmentation document and a main model for measuring the relevance between the query with these augmentation document and the target document. Both models take sentence-T5 as their model architecture and trained iteratively using ANCE. Experiments on 18 datasets in BEIR demonstrate the effectiveness of MoMA.
strength_and_weaknesses: Strength: well-motivated idea
Weakness: Experiments can be further improved

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. Leveraging multiple external text corpus to improve retrieval-augmented language models on zero-shot retrieval tasks is well motivated and novel. Many details w.r.t. the model architecture and experiment implementation are discussed in the paper, which is helpful for understanding the whole paper.


summary_of_the_review: Experimental settings need further explanation. I'm curious why not just following the settings in those papers listed in the related work sections. it can be found that most dense retrieval models (including MoMA-DR w/o Target) performs worse than BM25 (a very simple baseline) and even MoMA-DR  exceeds BM25 with a slight margin. What's the cause for this phenomenon? Is the source training corpus (MS MARCO passage) and/or being unrelated to the BEIR datasets?

Besides, it can be found in Table 2 that the target dataset is the core for MoMA-DR compared to external corpus (i.e., the 5th column versus the 8th column, 0.431 versus 0.436), which weakens the claim that multiple external corpus used in MoMA is helpful for zeroDR. Is it possible to make some case study to show that the external corpus does contribute some knowledge that is uncovered in the target dataset?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a new method to improve zero-shot retrieval ability of dense retrievers. The main contribution is to enrich the query representation by using additional external resources, such as Wikipedia or knowledge bases, from which “augmentation documents” are retrieved and added to the query.

Most of the scores of the reviews for this paper are slightly below the acceptance threshold. Some of the concerns raised by the reviewers are the following. The proposed method is not well situated compared to relevant existing works, and this makes it hard for the reviewers to know what are the main contributions of the paper. Moreover, the paper feels a bit incremental, and the experimental results are not very strong compared to existing baselines. Finally, several reviewers raised concerns about claims made in the paper that should be clarified. For these reasons, I recommend to reject the paper, so that these concerns can be thoroughly addressed by the authors.
justification_for_why_not_higher_score: Here is the justification for why not a higher score:
- need better discussion of how this relates to existing work & some claims made in the paper ;
- limited technical contributions ;
- limited improvements compared to the baselines.
justification_for_why_not_lower_score: n/a
Decision: Reject