Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors benchmark 10 IR techniques on 18 publicly available datasets. Their findings conclude that BM25 is a robust baseline which is a known phenomenon from many other studies in IR evaluations. The authors define 3 criteria for selecting 18 datasets which are also very generic and nothing out of ordinary. However, the result and analysis section is a nice read. 
strengths: Extensive analysis of several IR models using both in-domain and out-of domain datasets. 
Discussed the issues of annotation selection bias
A nice framework to evaluate many IR techniques using many datasets. 
weaknesses: All these experimental findings are already knew. The authors fails to suggest any new insights. 
correctness: The authors major claim is that the framework is easily adaptable for other IR tasks and datasets but there is no discussion about it in the paper. 
clarity: Yes
relation to prior work: Not applicable
additional feedback: Nice python package. I hope you will actively maintain the repo.
review_title: This is a paper that benchmarks several IR techniques on 18 publicly available datasets

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: 
This paper presents a benchmark for text retrieval tasks -- a central task in Information Retrieval (IR). This work selects 18 already existing datasets from diverse domains and shows the effectiveness of recently-proposed (and now standard) transformer-based retrieval models. The datasets in BEIR cover a wide variety of domains and task complexities. The strongest point of the paper is the large variety of models used to showcase retrieval and ranking performance in a transfer learning setting. In fact, the task complexity is shown by the non-transferability of the models from the MSMarco task to others. Or more specifically, most of the neural retrieval models perform worse than the simple yet robust non-parameterized BM25 on zero-shot transfer.
strengths: 1) This paper deals with a wide variety of text retrieval tasks -- with varying vocabulary, domains, and annotation sparsities. This makes it a first diverse benchmark for text ranking. 

2) One of the major aspects of this benchmark is that it is easy to use and experiment with. Even if the benchmark pitches itself as targetted towards zero-shot learning, BEIR could infact be used to benchmark standard ranking models in the fine-tuning regime as well.

3) The number of approaches considered is both sufficient as well as diverse to justify "prepping" of the benchmark. Most of the sparse and dense approaches are well known and provide some insight into the task of zero-shot transfer.





weaknesses: 
1) The difference between retrieval and ranking performance is not argued in detail. In IR, both retrieval and ranking are crucial for the understanding of text-ranking models. The distributions are typically different for retrieval and ranking tasks. Specifically, while the retrieval model has to differentiate between a large set of non-relevant items and the actual relevant item/s, the ranking model has to differentiate between partially relevant items and the actual relevant item/s. Typically, most of the models are trained using the ranking data since it has better hard negatives. This also might result in overfitting the task data with very low zero-shot performance. It is not clear in which regime are the models presented in the paper trained on. The authors should clarify this point and maybe reflect on the deficiencies of training on task-specific hard negatives on the results.

2) Although the paper does a great job in curating a number of retrieval tasks, it misses out on the bread-and-butter long document retrieval task for the Web domain. The reasons for leaving this out in also not discussed. Possibly ClueWeb09 and TREC-DL datasets could be good candidates. Web datasets are often accompanied by a higher degree of query underspecification and hence are more complex than other included tasks. Granted that the news datasets also contain similar queries, but Web datasets exhibit other complexities like redundancy and high content variability.

3) The datasets considered do not come with an accompanying corpus. Since the focus is on retrieval models and re-ranking, accompanying corpora would help provide abilities for superior modeling 

correctness: The claims made in the paper are correct as much as I can say.
clarity: The paper is clearly written and is an easy read.
relation to prior work: The paper covers most of the important related works and is well contextualized.
additional feedback: The authors should re-visit the word overlap experiment to show task complexity/overlap. Rather use the real estate to show transfer performance when the models are trained on a task (apart from MSMarco). The word overlap experiment is slightly misleading and doesnt provide extra intuition about task complexity.
review_title: Well constructed IR Benchmark

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors introduce BEIR benchmark for zero-shot evaluation of information retrieval (IR) models. BEIR consists of 18 datasets covering 9 different IR tasks. The tasks include fact checking, citation prediction, duplicate question retrieval, argument retrieval, news retrieval, question answering, tweet retrieval, bio-medical IR, and entity retrieval. The authors also evaluate 10 different retrieval methods. They also provide an API to use the benchmark. They claim that they will provide a leaderboard for the models use BEIR. The leaderboard shared by the authors is a Google Spreadsheet showing ranking of IR systems. 
strengths: The benchmark they share is huge, covering several datasets and tasks. Therefore, it is useful for zero-shot evaluation.
They provide evaluation results for several IR methods and discuss their performance.
The API they developed seems to be useful. 
They share code for their evaluation, which will make it easier to use the benchmark.
The paper is well-written

weaknesses: I do not see any major weakness in the paper. The authors also provide a list of limitations. One might consider that, authors use existing datasets and pre-trained models for evaluation. So their contribution is limited. I partially agree on that, but their main contribution is easing use of existing datasets. I believe it will be useful for future research.

As another minor weakness, they report only nDCG@10 for their experimental results. It is not clear to me why they picked the cut-off value as 10. To my knowledge, nDCG@10 is not a common metric. For instance, nDCG@20 might be more common. 

As another minor issue, using a Google Spreadsheet for leaderboard will be hard to maintain. It would be great if people can just submit their outputs (i.e., output of their systems) and the system will automatically update the leaderboard.

correctness: The experimental setup is generally ok. In results for efficiency, it seems that they report execution time for running the systems only once. It would be better to run them multiple times (e.g., 3) and report average.
clarity: The paper is well-written. There are only a few typos. 

“By integrate a large number of diverse retrieval systems into BEIR, creating such diverse pools becomes significantly simplified.”  -> integrating

relation to prior work: The authors provide a discussion about related work. I think they have a solid contribution.
additional feedback: The authors convinced me that nDCG@10 is also a reasonable choice to report. Therefore, you can ignore it as one of the weaknesses I raised (I keep it there to understand their answer for that issue)
review_title: Very useful benchmark with a few minor issues

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper reports on BEIR, a new benchmark based in 18 existing public datasets for information retrieval. BEIR  harmonizes these datasets into a single unified benchmark, mostly oriented towards zero-shot evaluation. The paper provides a basic overview of the BEIR dataset and includes analyses of how the constituent datasets compare with each other. The papers also evaluates a range of different IR models, including term-based models and a variety of models based on distributed/contextual representations. The paper also includes a new analyses providing evidence that the BEIR datasets are biased toward lexical models, and that this bias persists to some extent even where the dataset annotation methods sought to address such biases.

strengths: I am confident that BEIR will prove to be a useful benchmark. My collaborators and I have already begun to use it as an evaluation tool.

Although BEIR pools existing datasets, the project is offering much more than that -- the authors have made it very easy to use these datasets to conduct experiments, and they have done a lot of great work in terms of facilitating access to different IR models.

I also really appreciate the "Selection bias" experiments in section 8, which reveal biases towards lexical items across many datasets. This is very useful to know about and it is the sort of thing that the community should be working hard to expose and address.
weaknesses: I think the model evaluations in the paper  (sections 4 and 5) are a strength on balance, as I said above. However, there are a number of aspects to the results that limit the usefulness of the findings.

* My biggest concern traces to the fact that what we have here are evaluations of artifacts not architectures. In particular, the neural models were trained on substantially different datasets. I think it is clear that MS MARCO is a very different IR resource than, for example, Natural Questions. MS MARCO looks like a reasonable chunk of the useful parts of the Web, whereas NQ is biased by design towards frequently asked questions and so covers much less of the space. Thus, when we compare, for example, DPR with TAS-B, we are not really comparing these as architectures, but rather particular instances of the architecture, and training them on the same data might reduce or remove apparent differences. That's all fair and even useful, but it isn't enough evidence for us to conclude that DPR, as a modeling idea, is different from TAS-B at that level.

* I feel unsure of the category "BM25 + CE". The current paper reports that this is based on MiniLM, but that MiniLM was chosen from a wide range of different models in this category, using MS MARCO as a dev set (if I understand correctly). However, I believe an earlier version of this paper used the model from Nogueira & Cho 2020, and the results table for "BM25 + CE" looked substantially different in that draft (https://arxiv.org/pdf/2104.08663v1.pdf) compared to the current one. So: for "BM25 + CE", it seems like we're seeing the outcome more more dev-set comparisons than for any other model evaluated. Should this affect the conclusions? I am keen to hear more from the authors about this. What is the variance among these different  "BM25 + CE" instantiations?

* I think it's great to assess models with regard to latency and index size, but what about training efficiency? This is also important, and the neural models differ very widely in this regard.
correctness: The correctness issue on my mind relate to the scope of the modeling claims, as detailed above.
clarity: The paper is very clear. I'd urge the authors to proofread it more times, and take note that, by standard English punctuation conventions, a comma cannot join two free-standing sentences unless there is a coordinator like 'and' (and adverbs like 'however' don't count for this).
relation to prior work: I think the paper does a great job connecting with existing relevant work.

additional feedback: Nothing to add here. The primary things I want to discuss are given in the 'Weaknesses' section above.
review_title: Outstanding new benchmark, with model evaluations that might be more limited than described

Meta-review: This paper presents a benchmark for text retrieval tasks, using 18 already existing datasets from diverse domains and task complexities, and covering a large variety of models used to showcase retrieval and ranking performance especially in a transfer learning setting. Overall, reviewers thought the work was strong in the diversity of included tasks, models, and usability; reviewers also praised the relevance with respect to zero-shot approaches. Concerns included overall novelty, distinction between relevance and ranking, and the fact that models were trained on different corpora at times. However, the reviewers ultimately have found this paper to be an extremely strong benchmark that will be immediately useful in the field.
Decision: Accept