InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers
Authors: Leonid Boytsov
Preksha Patel
Vivek Sourabh
Riddhi Nisar
Sayani Kundu
Ramya Ramanathan
Eric Nyberg
Venue: Accepted by TMLR ()

Abstract: We carried out a reproducibility study of InPars, which is a method for unsupervised training of neural rankers (Bonifacio et al., 2022). As a by-product, we developed InPars-light, which is a simple-yet-effective modification of InPars. Unlike InPars, InPars-light uses 7x-100x smaller ranking models and only a freely available language model BLOOM, which—as we found out—produced more accurate rankers compared to a proprietary GPT-3 model.  On all five English retrieval collections (used in the original InPars study) we obtained substantial (7%-30%) and statistically significant improvements over BM25 (in nDCG and MRR) using only a 30M parameter six-layer MiniLM-30M ranker and a single three-shot prompt. In contrast, in the InPars study only a 100x larger monoT5-3B model consistently outperformed BM25, whereas their smaller monoT5-220M model (which is still 7x larger than our MiniLM ranker) outperformed BM25 only on MS MARCO and TREC DL 2020. In the same three-shot prompting scenario, our 435M parameter DeBERTA v3 ranker was at par with the 7x larger monoT5-3B (average gain over BM25 of 1.3 vs 1.32): In fact, on three out of five datasets, DeBERTA slightly outperformed monoT5-3B. Finally, these good results were achieved by re-ranking only 100 candidate documents compared to 1000 used by Bonifacio et al. (2022). We believe that InPars-light is the first truly cost-effective prompt-based unsupervised recipe to train and deploy neural ranking models that outperform BM25. Our code and data is publicly available. https://github.com/searchivarius/inpars_light/

Published in Transactions on Machine Learning Research (02/2024)
In InPars-Light: Cost-Effective Unsupervised Training of Effi- cient Rankers
Leonid Boytsov*
leo @boytsov.sng info
Amazon AWS AT Labs
Piltsburgh
USA
Preksha Patel
Vivek Sourabh
Riddhi Nisar
Sayani Kundu
Ramya Ramanathan
Eric Nyberg
Cornegie Mellon University
Pittsburgh
USA
Reviewed on OpenReview:
https: //tp://ppppe opwnreviewwww.. net/ forum? id= #SSKFYyIO
//
Abstract
We carried out reproducibility study of TnPars, which is method for unsupervised training of neural rankers (Bonifacio et al. 2022). As a by-product, we developed InPars-light, which is a simple--ytt-eleective modification of InPars. Unlike InPars, InPars-light uses 7x-100x smaller ranking models and only a freely available language model BLOOM, which as we found out-produced more accurate rankers compared to a proprictary GPT-3 model. On all five English retrieval collections (used in the original InPars study) we obtained substantial (7%-30%) and statistically signifieant. improvements MRR) using only 30M parameter six-layer MiniLM-30M ranker and single three-shot prompt. In contrast, in the InPars study only 100x larger mono monoT5-3B model consistently outperformed BM25, whereas their smaller monoTT5-200 model (which is still 7x larger than our MiniLM ranker) outperformed BM25 only on MS MARCO and TREC DL 2020. In the same three-shot prompting scenario, our 435M parameter DeBERTA v3 ranker was at par with the 7x larger monoT5-3B (average gain over BM25 of 1.3 1.32): In fact, on three out of five datasets, DeBERTA slightly outperformed monoT5-3B. Finally, these good results achieved by re-ranking only 100 candidate documents compared to 1000 used by [Bonifacioo et al. 2022 We believe that InPars-light is the first truly cost-effective prompt-based unsupervised recipe to train and deploy neural ranking models that outperform BM25, Our code and data is publicly available.
BM25 (in nDCG and
com/ http://aacchlvrrlssinnpapaa hww.iaaarssggghh///aaa
1 Introduction
Training effective neural IR models often requires abundant in-domain training data, which can be quite costly to obtain: For a human annotator, judging a single document-query pair takes at least one minute on average (Han al.] 2020 Kwiatkowski et. al.] 2019) and a single query may need as many as of such judgements (Buckley et al. 2007) Models trained on out-of-domain data and/or fine-tuned using small number of in-domain queries often perform worse marginally better than simple non-neural BM25 rankers
WWork done outside of the scope employment. Robust04 and TREC-COVID collections used our study have about 1K judgements per query.

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Average Gains over BM25 for different Models and Training Recipes
Model name and training recipe
Avg. gain over BM25
of "wins" over RM25s (S
Unsupervised: InPars-based Training Data (three-shot prompting)
MiniLM-L6-30M (laPars-light) DeBERTA-v3-4300 (InPars-light)
1.13
1.30
mo-2215 (InPars) [Bonifacio et 2022
1.07
monoo1'-3BB (InPars) (Bomitacio et al. 2022)
1.32
Supervised transfer learning with optional unsupervised fine-tuning: transfer from MS MARCO with optional fine-tuning on consistency-checked InPars data
MiuiLM-L6-30M (MS MARCO)
1.21
MiniLM-L6-30M (MS MARCO consist. checked queries)
1.24
DeBERTA-v3-4435 (MS MARCO)
1.42
DeBERIA-v3-E35M (MS MARCO consist. checked queries)
1.36
momoT5-220M (MS MARCO) Bonifacio et al.
2022
1.46
m533TT-3B (MS MARCO) Bonilacio al. 2022
1.59
m5-ooT5-3B (MS MARCO mmrass) (Bonilacio al.
2022.
1.59
Consist. checked gueries denotes set of generated queries filtered out (via consistency chocking) using the DeBERTA-V3- 435M model trained InPars--enerated data.
Thakur et al., 2021
Mokrii et al. 2021). Good transferability requires (1) large impractical models (Rosa
ct al.
2022 Ni et al. 2021), and (2) datasots with large and diverse manually annotated query sets.
A recent brend to deal with these problems consists in gen- erating synthetic in-domain training data via prompting of Large Language Models (LLMs). This trend was spear- headed by a recent InPars study (Bonifacio et al. 2022 However, proposed solutions are not cost effective because they require either querying the costly generative models or training impractically large rankers. Although follow up studies, in particular by Dai et al. (2022). claimed improvements upon InPars, these improvements were not demonstrated under the same experimental setting. More- over, researchers used primarily proprietary LLMs whose training procedure was not controlled by the scientific communty. Thus, outcomes could have been affected by data leakage, i.e., training models on publicly available and popular IR collections whose copies could have ended up in the LLMs training data. As such, there is im- portant question of whether can obtain comparable better results using only open-source models trained by the scientific community.
1.40
1.35
1.2
Average
135
monoTS-220N (InPars)
1:10
monoTS-38 (InPars)
MiniLM-L6-30M (InPars-light)
1.55
DeBERTA V3-ERTA-3-43555 -335M (InPars-light)
500
1000
1500
2000
2500
3000
Number of parameters (in millions)
Figure Average relative improvement over BM25 for different model types/sizes and train- ing recipes. Higher and to the left is better. We compare InPars with InPars-Lighl for the unsu- pervised training scenario, where training data is generated by an LLM using a threc-shot prompt.
This study is driven by two high-level inquiries: (1) Does InPars work? (2) Can it made more accurate and cost effective? To address these inquiries, we carry out rigorous reproducibility study of InPars (Bonifacio al.] (022)). In that, we use open-source and communityy-traine 2022 Wang & Komatsuzaki] 2021), train rankers using multiple seeds, and use statistical testing when measuring improvements. Because efficiency is an important consideration, we also evaluate much smaller ranking models (sce Figure and Table [ compared to those used by Bonifacio et al. (0222.
generative LLMs (Scao al.)
More specifically, we ask the following research questions:
RQ1: Can we reproduce key findings of InPars (Bonifacio et al. 2022) using open-source and tommunityytmmneed L.LMs well as smaller ranking models?
2

---

Published in Transactions on Machine Learning Research (02/2024)
RQ2:
Are open-source models more or less useful for generation of synthetic IR. training data compared to the similar-sized GPT-3 Curie model (Brown et al. 2020 ?
RQ3: Does consistency checking proposed by Dai et al. 2022 improve the InPars recipe? Is it applicable in the purely ro-ranking sctting as opposed to the retricval sctting (as it was done by Dai (et al. (2022))?
RQ4:
Can we match performance of large monoT5 rankers-usee by Bonifacio et al. (2022 with
much smaller bi-directional Transformer (BERT) models (Devlin et al. 2018
Vaswani al. 2017
RQ5:
The smaller m5 ranker with 220M parameters used by Bonifacio et al.
2022 does not
outperform BM25 for three out of five query sets. Thus, just matching monoT5-220M performance is not enough. Can we instead substantially outperform BM25 using a small and fast ranker such as a MiniLM (Wang et al, 2020 BERT ranker with only 30 million parameters?
Our contributions and findings as follows:
We reproduced the key finding by Bonifacio et al. (2022): Generation of synthetic in-domain data using an InPars-like recipe permits training strong in-domain rankers using only a three-shot prompt and in-domain documents, which answers RQL. However, without additional effort such as all-domain pre-training and consistency checking, only sufficiently large ranking model could outperform BM25 on all datasets.
We found that open-source LLMs BLOOM (Scao et al. 2022 and GPT-J (Wang & Komatsuzaki 2021). which are trained using only next-token prediction (without further fine-tuning), could be prompted to generate effective synthetic queries. Moreover, using a community-truinndd BLOOM model produced comparable or more accurate ranking models compared to using GPT-3 Curie model (Brown el. al. 2020), which addresses RQ2.
We confirmed that consistency checking proposed by Dai et al. 2002 does work for re-rankers and always improves outcomes in the unsupervised setting, which answers RQ3.
We also discovered that in the unsupervised setting, where synthetic queries were generated using a three-shot prompt, we could match or outperform monoT5 rankers using much smaller BERT ranking models (see Figure T. which answers RQ4. More specifically:
We can replace an impractical tiree-billion parameter monoTT5-3B (Nogueira et al. 2020 model
with a 7x smaller BERT model while obtaining comparable results. The average gain over BM25
(see Table 1 was 1.32 for monoT5-3B vs. 1.3 for DeBERTA-V3-496M (ee et al. 2021 (RQ1).
Unlike Bonifacio et al. (2022) whose mono 220M model with 220 million parameters failed to outperform BM25 on three out of five datasets (unless pre-trained on MS MARCO), we show
that a much smaller MiniLM-30M model with only 30 million parameters (Wang et al.) 2020 can outperform BM25 by 7%-30% in key metrics (nDCG@K and MRR) when trained using only synthetic training data (RQ1 and RQ5).
Outperforming BM25 with small ranking model such as MiniLM-30M was possible by using: (a) better model to generate synthetic training data (BLOOM instoad of GPT-3 Curie), (b) consistency checking (Dai et al. 2022 (RQ3), and (c) all-domain pre-training, each of which helped improve outcomes.
Obtaining good results in the unsupervised setting described above required re-ranking only 100 candidate documents compared to 1000 used by BBoiffciio al. (2022). Overall, compared to InPars, our training recipe which we call InPars-light substantially more cost effective in terms of both, generation of synthetic training data and training/ happii/ggpppliiiiinnn of ranking models (see $ A.2 for detailed discussion).
Thh only exception was BEIR NQ, where BL.OM- ranker was 1.4% worse, see Table
3

---

Published in Transactions on Machine Learning Research (02/2024)
However, when pretraining MS MARCO used, the monoT3-220M model was still substantially more accurate than a 7x smaller MiniL.M-30M ranker. Moreover, this gap was not reduced by subsequent unsupervised fine-tuning of MinILM-30M using synthetically generated data. The average gain over BM25 (sce Table 1 was 1.46 for monoTT5-2000 pre-trained on MS MARCO vs. 1.24 for MiniLMM30M pre-trained on MS MARCO and fine-tumed using synthetic training data.
Our code and data are publicly available
2 Related Work
Prompting methods have gained quite a bit of popularity in NLP (see, e.g., Liu et al. (2021) for recent. survey). In particular, prior to the InPars study by [Bonifacio et al. (0022) Schick & Schütze [0021] proposed to generate synthotic training sets using in-domain data and zero-shot prompting of LLMs.
However, until recently zero-shot and few-shot prompling of LLMs was not applied lo ad hoc retrieval: We know only a few papers directly related our work. |Sachan et al. (2022) were probably the first demonstrate effectiveness of LLMs in the document ranking task. In their approach-namen UPR-they concatenate document, special prompt such as "please write question for this document" and the query itself. Then, UPR uses pre-trained LLM model to compute the likelihood of generating the query given the passage text. Unlike InPars, they do not use LLM to generate synthetic training data.
Sachan et al. [0022 evaluated their method using only QA (but not IR) datasets and their main results are for an impractically large three-billion parameter instruction finetuue model, which was used essentially as a re-ranker (in zero-shot scenario). The smallest model used by [Sachan al. (2022) had 250 million parameters (compared to our 30-million MiniLM model). It was evaluated only the Natural Questions (NQ) collection (Kwiatkowski et al. 2019) where it outperformed BM25 by about 10%. Although not directly comparable due to using different versions of NQ and model sizes, our 2x larger DeBERTA-v3-435M. model performmed BM25 by 40% while our much smaller MiniLM-30M model with 30 million parameters outperformed BM25 by 15%.
proposed an InPars method, which relied on few-shot prompting. The study had a convincing evaluation on five datasets where only one dataset, namely NQ (Kwiatkowski et. al. 2019), was
Bonifacio 2022)
typical QA collection. Unlike Sachan et al. (2022), Bonifacio et al. 2022) used few-shot prompting to generate synthetic training data for smaller ranker. For each collection Bonifacio et al. (2022) generated 100K synthetic queries and retained only 10K with the highest average log prebaabbbiiiiss.. This can be seen as distillation of an LLM into the ranker.
However, Bonifacio et al. (2022) obtained good results only for a huge mon3 parameter model. They also employed proprictary GPT-3 model, which can be quite costly to use. In a follow-up study, which is concurrent with this work, Jeronymo et al. (2023) introduced modification of InPars ddub-d InPars v2- where GPT-3 Curic (Brown et al. 2020) replaced with an open-source model GPT-J model (Wang & Komatsuzaki 2021). However, this model swap was "entangled" with at least two other modifications in the training recipe:
new query filtering condition that relied on an MS MARCO trained monoT5-3B model.
The vanilla prompt (which was used in InPars and our experiments) was replaced with the "Guided by Bad Question prompt" (introduced by Bonifacio et al. 2022
Thus, it is not possible fairly assess the impact of replacing GPT-3 Curie with GPT-J Waag & Komatsuzaki 2021
An important disadvantage of the InPars v2 recipe is that it is still not cost-effective authors use huge m-3oTT5-BB model. The filtering check uses an expensive monoT5-BB model trained on MS MARCO
4

---

Published in Transactions on Machine Learning Research (02/2024)
corpus, which is also not always possible in commercial setting due to licensing issues (MS MARCO is research-only collection).
Moreover, the m5-3 model trained on MS MARCO albeit being impractical has excellent zero-shot transferability: Fine-tuning monoT5-3B model trained on MS MARCO with InPars v2 only improves the average BETR score only by 2.4%: from 0.538 0.551. This further complicates assessment of effectiveness of GPT-J.
Dai et al. 2022 used an InPars-like method called Promptagator and created synthetic training data using huge proprictary FLAN-137B model with 137 billion parameters. Although they used modestly sized retrieval and ranking models with 110 million parameters, Dai et al. (2022) generated as many as million synthetic training queries for each dataset. In contrast, both InPare and InPars-light used only 100K queries per dataset, which was much less expensive (see a discussion in A.2).
Importantly, Dai et al. (2022) proposed use consistency chocking (Alberti et al. 2019 filter-out potentially spurious queries, which was not previously done in the IR context. They do not compare with InPars under the same conditions and it not known if consistency checking would improve the original InPars recipe.
In addition to prompt-based gemeration of training data, there are multiple proposals for self-supervised
adaptation of out-of-domain models using generative pseudo-labeling (Li & Gaussier 2022 Wang et al. 2022:
Reddy et al. 2021). To this end, questions or queries are generated using a pretrained seq2seq model (though an LLMs can be used as well) and negative examples are mined using either BM25 or an out-of-domain retriever or ranker. Unsupervised domain adaptation is complementary to the approaches considered in this work.
The disadvantage of such approaches is that they may need a reasonably effective an out-of-domain ranking model. However, such models can be hard to obtain due to licensing issues and poor transferability from other domains. For example, MS MARCO models have reasonable transferability (Thakur et al. 2021 Mokrii et al., 2021h, but MS MARCO cannot be used to train models Microsoft). In contrast, the Natural Questions (NQ) collection (Kwiatkowski et al.) 2019) has permissive license but models trained on NQ can fail to transfer to datasets that are not based on Wikipedia MMokrii
commercial context (without ext.ra licensing from
et al. 2021
Another potentially complementary approach is an LLM-assisted query expansion. In particular Gao et al. (202)) prompted a 175B InstructGPT model to generate a hypothotical answer to a question. Then this answer was encoded as vector and logether with the encoding of the original question they were compared with encoded documents. In a purely unsupervised cetting-using the Contriever bi-encoder training without. supervision (Izacard al. 2021) they were able to outperform BM25 by as much 20%.
Despite strong results, serious disadvantage of this approach is its dependence the external proprictary model that is costly and inefficient. Although we could not find any reliable benchmarks, folklore opinion is that GPT generation latency is a few seconds. To verify this, we used the OpenAl playground 3 to generate a few hypothetical answers using the prompt in Gao et al. Gao et al. (2022) and sample TREC DL 2020 queries. With a maximum generation length of 256 tokens (a default setting), the latency exceeded four seconds.
Quite interestingly, Gao et al. Gao et al. 2022 tried to replace 175B GPT-3 model with smaller open-source models on TREC DL 2019 and TREC DL 2020 (see Tables 4 and Table in their study), but failed to obtain consistent and substantial gains over BM25 with models having fewer than 50B parameters.
htte:///hhhhccoooeeeeeeeeeeeerrr https://beta.oopmat comp:/ayooooo/ppyrounnnno

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: The format of the vanilla three-shot InPars prompt (Bonifacio et al.
2022
Example 1:
Document: <text of the first example document?> Relevant. Query: <text of the first relevant query>
Example
Document: <text of the second example document>> Relevant Query: <text the second relevant query>
Example 3:
Document: <text of the third example document>>
Relevant Query: <text of the third relevant query>
Example 4:
Document: <real in-domain document text placeholder> Relevant Query:
To generate synthetic query, first insert text of a chosen real in-domain document after the prefix
Notes:
"Document:" in example four. Then, "ask" LLM to generate a completion.
3 Methods
3.1
Information Retrieval Pipeline
We use a variant of classic filte--ad-refine multi-stage retrieval pipeline (Matveeva et al. 2006 Prager 2006
Wang et al. 2011), where top-k candidate documents retrieved by fast. BM25 relriever scorer [Robersson 2004) are further re-ranked using slower neural re-ranker. For collections where documents have titles (NQ BEIR and TREC COVID BEIR), the BM25 retriever itself has two stages: In the first stage we retrieve 1K documents using a Lucene index built over a title concatenated with a main lext. the second stage, these candidates are re-ranked using equally weighted BM25 scores computed separately for the title and the main text.
Our neural rankers are cross-encoder models Nogueira & Cho 2019 Lin et al. 2021b which operate on queries concatenated with documents. Concatenated texts are passed through backbone bi-directional encoder-only Transformer model (Devlin et al. 2018) equipped with an additional ranking head (a fully- connected layer), which produces a relevance score (using the last-layer contextualized embedding of a CT.S-token (Nogueira Cho 2019)). In contrast, authors of TnPars (Bonifacio et al., 2022) use T5 (Raffel 2020 cross-encoding re-ranker (Nogueira al., (2200). which full Transformer model (Vaswani et al.
et al.
2017 It uses both the encoder and the decoder. The T5 ranking Transformer is trained to generate labels "true" and "false", which represent relevant and nolerelevant document-query pairs, respectively.
Backbone Transformer models can differ in the number of paramcters and pre-training approaches (including pre-training datasets). In this paper we evaluated the following models, all of which were pre-trained in the self-supervised fashion without using supervised IR. data:
A six-layer MiniLM-L6 model (Wang al. 2020). It tiny (by modern standards) 30-million parameter model, which was distilled (Li et al. 2014 Romero et al. 2015 Hinton et al. 2015 from
Roberta [Liu et al. 2019 We download model H6xH384 MinILMv2 from the Microsoft. website
A 24-layer (large) ERNIE v2 model from the HuggingFace hub Sun et al. 2020 Tt has 335 million
parameters.
A 24-layer (large) DeBERTA v3 model with 435 million parameters He et al. 2021 from the
HuggingFace hub

---

Published in Transactions on Machine Learning Research (02/2024)
We chose ERNIE v2 and DeBERTA v3 due to their strong performance on the MS MARCO dataset where they out performmee BERT large (Devlin et al. 2018) and several other models that we tested in the past. Both models performed comparably well in the preliminary experiments, but we chose DeBERTA for main experiments bocause it was more effective on MS MARCO and TREC-DL 2020. In the post hoc ablation study, DeBERTA outperformed ERNIE on four collections out of five (see Table J..
However, both of these models are quite large and we aspired to show that an InPars-liks training recipe be used with smaller models too. In contrast, Bonifacco et al. 2022) were able to show that only big T5-3B model with 3B parameters could outperform BM25 on all five datasets: The smaller monoT5-200M ranker with 200 million paramcters, which is still quite large, outperformed BM25 only on MS MARCO and TREC-DL 2020.
3.2
Generation of Synthetic Training Data
We generate synthetic training data using well-known few-shot prompting approach introduced by [Brown] et al. (2020). In the IR domain, it was first used by Bonifacio et al. (2022) who called it InPars. The key idea of InPars is to "prompt" large language model with few-shot textual demonstration of known relevant query-document pairs. To produce a novel query-documenl pair, Bonifacio et al. (2022) appended an in-domain document to the prompt and "asked" the model to complete the text. [Bnnffacio et al. 2022 evaluated two types of the prompts of which we use only the so-called vanilla prompt (soc Table 2p.
As in the InPars study (Bonilacio el. al., 2022), we generated 100K queries for each dataset with exception of MS MARCO and TREC D Repeating this procedure for many in-domain documents produces large training set, but it can be quite imperfect. In particular, carried out spot-checking and found quite few queries that were spurious or only tangentially relevant to the passage from which they were generated.
Many spurious queries can be filtered out automatically. To this end, Bonifacio et al. 2022) used only 10% of the queries with the highest log-probabilities (averaged over query tokens). In the Promptagator recipe, Dai et al. (2022) used a different filtering procedure, which was variant of consistency checking (Alberti et al. 2019 Dai et al.] (2022) first trained retriever model using all the generated queries. Using this retriever, they produced ranked set of documents for each query. The query passed the consistency check if the first retrieved document was the document from which the query was generated. A straight.orward modification of this approach is to check if a generated document is present in a top-k (k > 1) candidate set produced by the retriever. Dai et al.] (2022) used consistency checking with bi-encoding retricval models, but it is applicable to cross-eucoding re-ranking models as well.
3.3 InPars-light Training Recipe
The InPars-light is not a new method. Tt is a training recipe, which modification of the original InPars. Yet, it is substantially more cost effective for generation of synthetic queries, training the models, and inference. InPars-light has the following main "ingredicnts"::
Using open-source models instead of GPT-3:
Using smaller ranking BERT models instead of monoT5 rankers;
Fine-tuning models on consistency-cheeced training data;
Optional pre-training of models using all generated queries from all collections.
Re-ranking only 100 candidate documents instead of 1000: However, importantly, the training procedure still generates negatives from a top-1000 set produced by a BM25 ranker.
To obtain consistency-checked queries for a given dataset, model trained InPars -aarr-ggeeatted queries (for this dataset) used to re-rank output of all original queries (for given dataset). Then, all the queries
9 Because both datasets use the same of passages they share the same 100K generated queries.
7

---

Published in Transactions on Machine Learning Research (02/2024)
where the query-eenenatinggcocumee did not appear among top-k scored documents discarded. In our study, we experimented with & from one to three (but only on MS MARCO) Although k = 1 worked pretty well, using k = 3 lead to a small boost in accuracy. Consistency DeBERTA-v3-45 435M (He ct al. 2021). We want to emphasize that consisteney-checkod training data was used in addition to original In Para-geeeeeeeed data (but not. instead), namely, to fine-tune a model initially trained on InPars generated data.
was carried out using
Also, quite interestingly, set of consistency-checkee queries had only small (about 20-30%) overlap with the set of queries that were selected using the original InPars recipe (based on average log- probabilities). Thus, consistency-cheede increased the amount of available training data. It might seem appealing to achieve the same objective by simply picking a larger number of queries (with highest average log-probabilities) However, preliminary experiments on MS MARCO showed that a naive increase the number of queries degraded effectiveness (which consistent with findings by Bonifacio et al. (2022)) ).
Although, the original InPars recipe with open-source models and consistency checking allowed us train strong DeBERTA-v3-4955 models, performance of MiniLM models was lackluster (roughly at BM25 level for all collections).
Because bigger models performed quite well, it may be possible to distill (Li et al, 2014 Romero et al. 2015 Hinton et al. 2015) their parameters into a much smaller MiniL.M-30M model. Distillation is known to be successful in the IR domain (Hofstätter et al. 2020 Lin et al. 2020), but it failed in our case. Thus we used the following workaround instead:
First we carried out an all-domain pre-training without any filtering (i.e., using all queries from all collections);
Then, we fine-tuned all-domain pre-trained models the consistency-checked in-domain data for each collection separately.
3.4 Miscellaneous
We carried out experiments using FlexNeuART Boytsov & Nyberg (2020), which provided support for basic indexing, retrieval, and neural ranking. Both generative and ranking models were implemented using PyTorch and Huggingface (Wolf et al.] 2020). Ranking models were trained using the InfoNCE loss (Le-Khac et al., 2020). In single training epoch, we selected randomly one pair of positive and three negative examples per query (negatives were sampled from 1000 documents with highest BM25 scores). Note that, however, that during inference we re-ranked only 100 documents. In preliminary experiments on MS MARCO we used to sample from a top-100 set as well. However, the results were surprisingly poor and we switched to sampling from top-1000 set (we did not try any other sampling options though). number of negatives was not tuned: We used as much as we can while ensuring we do not run out of GPU memory during training on any collection.
We used the AdamW optimizer [Loshchilov & Hutter 2017 with a small weight decay (10-?), warm-up schedule, and a batch size of 16 16 We used different base rales for the fully-connected prediction head 10-4) and for the main Transformer layers (2 10-5). The mini-batch size was equal to one and a larger batch size was simulated using a 16-step gradient accumulation. We did not tune optimization parameters and chose the values based on our prior experience of training neural rankers for MS MARCO.
We trained each ranking model using three seeds and reported the average results (except for the best-seed analysis in Table 5) Statistical significance is computed between "seed-average" runs where query-specific metric values are first averaged over all seeds and then standard paired difference test is carried out using these seed-average values (see & A.1 for details).
We did not want optimize this parameter for all collections and, thus, commit sin of tuning hyper-parameters on the complete test set.
The learning rate grows linearly from zero for 20% of the steps until reaches the base learning rate Moshach al. 2020
Smith 2017) and then goes back zero (also linearly).
8

---

Published in Transactions on Machine Learning Research (02/2024)
Except zero-shot experiments, we trained a separate model for each dataset, which is consistent with Bonifacio et al. (002]). Moreover, we computed exactly the same accuracy metrics as Bonifacio et al. (2022) For statistical significance testing we used a paired two-sided t-test. For query sets with a large number of queries (MS MARCO development set and BEIR Natural Questions) we used a lower threshold of 0.01. For small query sets (Robust04, TREC DI,, and TREC-COVID), the statistical siguificance threshold was set to 0.05.
We implemented our query generation module using the AutoModeeloorCasuallLMM interface from HuggingFace. We used a three-shot vanilla prompt template created by [Bonifacio et al. (2022) (also shown in Table 2) The output was generated via greedy decoding. The maximum number of new tokens generated for each example was set to 32. Note that query generation was a tine-consuming process even though we used open-source models. Thus, we did it only once per dataset, i.e,, without using multiple seeds.
4 Datasets
Bocause we aimed to reproduce the main results of InPars (Bonifacio et al. 2022), we used exactly the same set of queries and datasets, which are described below. Except MS MARCO (which was processed directly using FlexNeuART Boytsov & Nyberg (2020) scripts), datasets were ingested with a help of the IR datasets package (MacAvancy et al. 2021
Some of the collections below have multiple text Gields, which were used differently between BM25 and neural ranker. All collections except Robust04 have exactly one query field. Robust04 queries have the following parts: title, description, and narrative. For the purpose of BM25 retrieval and ranking, we usod only the title field, but the neural ranker used only the description field (which is consistent with Bonifacio et al. 2022 The narrative field was not used.
Two collections have documents with both the title and the main body text fields (NQ BEIR and TREC COVID BEIR). The neural rankers operated on concatenation of these fields. If this concatenation was longer than 477 BERT tokens, the text. was truncated on the right. (queries longer than 32 BERT tokens were truncated as well). For BM25 scoring, we indexed concatenated fields as well in Lucene. However, after retrieving 1000 candidates, we re-ranked them using the sum of BM25 scores computed separately for the title and the main body text fields (using FlexNeuART Boytsov & Nyberg (2020)).
Synthetically Generated Training Queries. For cach of the datascts, Bonifacio et al. 2022 provided both the GPT-3-generated queries (using GPT-3 Curie model) and the documents that were used to generate the queries. This permits a fair comparison of the quality of training data generated using GPT-3 Curie with the quality of synthetic training data generated using open-source models GPT-J (Wang & Komatsuzaki 2021) and BLOOM (Scao et al. 2022 According to the estimates of Bonilacio et al. (2022), the Curie model has 6B parameters, which is close to the estimate made by by Gao from Fleuther Gao (202T). Thus, we used GPTTJ (Wang & Komatsuzaki] 2021) and BLOOM (Scao et al., (022)) models with and 7 billion parameters, respectively. Although other open-source models can potentially be used, generation of syuthetic queries is quite expensive and exploring other open-source options is left for future work.
MS MARCO sparse and TREC DL 2020. MS MARCO is collection of 8.8M passages extracted from approximately 3.6M Web documents, which was derived from the MS MARCO reading comprehension dataset [Bajaj et al. 2016 Craswell et al. 2020p. It "ships" with more than half million of question-like queries sampled from the Bing search engine log with subsoquent filtering. The queries are not necessarily proper English questions, e.g., "lyme disease symptoms mood", but they are answerable by short passage retrieved from set of about 3.6M Web documents (Bajaj et all 2016). Relevance judgements are quite sparse (about one relevant passage per query) and a positive label indicates that the passage can answer the respective question.
The MS MARCO collections has several development and test query sets of which we use only a development set with approximately 6.9K sparsely-judged queries and the TREC DL 2020 (Craswell et al. 2020) collection of 54 densely judged queries. Henceforth, for simplicity when we discuss the MS MARCO development set we use shortened name MS MARCO, which is also consistent with Bonifacio et al. 2022
9

---

Published in Transactions on Machine Learning Research (02/2024)
Note that the MS MARCO collection has large training set, but we do not use it in the fully unsupervised scenario. It is used only supervised transfer learning (see $ 55
Robust04 (Voorhees 2004) is a small (but commonly used) collection that has about 500K news wire documents. It comes with a small but densely judged set of 250 queries, which have about 1.2K judgements
on average.
Natural Questions (NQ) BEIR (Kwiatkowski al. 2019) is an open domain Wikipedia-based Question Answering (QA) dataset. Similar to MS MARCO, it has real user queries (submitted to Google). We use 2021 which has about 2.6M short passages from Wikipedia and 3.4K spansely-judged queries (about 1.2 relevant documents per query).
BEIR's variant of NQ Thakur et al.
TREC COVID BEIR (Roberts et al. 2020 a small corpus that has 171K scientific articles on the topic of COVID-19 and. TREC COVID BEIR comes with 50 densely-judgod queries (1.3K judged documents per query on average). It was created for NIST challenge whose objective was develop information retricval methods lailored for the COVID-19 domain (with hope to be useful tool during COVID-19 pandemic). We use the BEIR's version of this dataset (Thakur et al. 2021).
5 Results
The summary of experimental results is provided in Figure 1 and Table 1 Our detailed experimental results are presented in Table 3 Note that in addition to our own measurements, we copy key results from prior
work (Nogueira et al. 2020 Bonifacio et al. 2022), which include results for BM25 (by Bonifacio et al. (2022 re-ranking using OpenAT API, and mono rankers. In our experiments, we statistically test several hypotheses, which are explained separately at the bottom of each table.
BM25 baselines. To assess the statistical significance of the difference between BM25 and a neural ranker, we had to use our own BM25 runs. These runs were produced using FlexNeuART Boytsov & Nyberg (2020). Comparing effectiveness of FlexNeuART Boytsov & Nyberg (2020) BM25 with effectiveness of Pyserini (Lin the InPars study (Bonifacio et al. 2022) we can see that on all datasets except.
et al. 2021a) 1BM25-usse
TREC DL 2020 we closely match (within 1.5%) Pyserini numbers. On TREC DL 2020 our BM25 is 6% more effective in nNDCG@10 and 25% more effective in MAP.
Unsupervised-only training (using three-shot prompts). We consider the scenario where synthetic training data is generated using three-shot prompt to be unsupervisod. Although the prompt is basod human supervision data (three random samples from the MS MARCO training corpus), these samples are not directly used for training, but only to generate synthetic data.
In this sccnario, we reproduce the key finding by Bonifacio et al. 2022]: Generation of synthotic in-domain data using an InPars-like recipe permits training strong in-domain rankers using only a three-shol prompt and in-domain documents. However, if we use the original InPars recipe, only a large ranking model (DeBERTA- v3-435M) consistently outperforms BM25. This answers RQ1. With similar to that of monoT5-3B on four collections out of five, even though 75-3 has 7x more parameters. The average gain over BM25 is 1.3 (for DeBERTA--V3445MM vs 1.32 for mono monoT5-3B (see Table 1.
obtain accuracy
Accuracy of our smallest model MiniLM-L6-30M with all-domain pretraining and finetuning on consistency- checked data (referred to as InPars all consist. check in Table 3) roughly matches that the 7x larger monoT5-220M on MS MARCO and TREC DL 2020. Yet, substantially better than T5-20MM on the remaining datasets, where mono'T5-2000 BM25 (see Table D is 1.07 for monoTT-2000 vs. 1.13 for MiniLM-30M.. MiniLM-L6-300. outperforms BM25 all collections and all metries. In all but one case these differences are also statistically significant. In terms of nDCG and/or MRR, MinILM-30M is 7%-30% more accurate than BM25.
effectiveness is largely at BM25 level: The average gain over
In summary, we can replace monoT5 rankers with much smaller BERT models while obtaining comparable or better average gains over BM25. This answers RQ4.
Impact of consistency checking and all-domain pre-training. We found that, on its own, the InPars recipe did not produce strong Min.I.M-L.6-00M ranking model. This is in line with the findings of Bonilacio
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Model Accuracy for Various Scenarios (averaged over three seeds)
MS MARCO
TREC DL 2020
Robust04
NQ
TREC
COVID
MRR
MAP
nDCG@10
MAP
nDCG@00 nDCG@10
nDCG@10
BM25
(Bonifacio al.
2022
0.1874
0.2876 0.3612
0.4876
0.2531
0.4240
0.3290
0.6880
BM25
(this study]
0.1867
0.5159
0.2555
0.4285
0.3248
0.6767
OpenAl Ranking API: re-ranking 100 Documents (Bonifaclo et al.
2022
Curie (6B) Bonifacio et
2022
$
0.3296
0.5422
0.2785 0.2790
0.5053
0.4171 $
0.7251
Davinci (1758) [ooilaaio el al.
2022
$
0.3163
0.5366
0.5103
0.6918
Unsupervised: InPars-based Training Data (three-shot prompting)
T--20M (InPars) (Bonifacio et al. 2022
0.2585
0.3599
0.5764
0.2490
0.4268
0.3354
0.6666
monooT5-3B (InPars) [Bonilacto et al 2022
0.2967
0.4334
0.6612
0.3180
0.5181
0.5133
0.7835
MiniLMM-L4-30M (InPars)
be0.2117
$0.3482
*0.4953
be0.2263
b*0.3802
be0.2187
$0.6361
MiniLM-L6-30M (InPars consist. check)
eb20.3336
eb0.3769
cb0,5543
eb0,2556
cb0.4440
cb0.3239
4906926
MiniLM-L6-30M (InPars all consist. check)
c00.2468
c90.3929
=*0,5726
*0.2639
c*0.4599
c*0.3747
e*0.7688
DeBERTAA-V33433.. (InPars)
be0.2746
ba0.4385
*0.6649
be0.2811
ba0.4987
ba0.4476
a0.8022
DeBERTA-Y3-483M (InPars consist. check)
cbe0.2815
c640.4446
c*0.6717
cbe0.3009
cbe0.5360
cbag.4621
c*0.8183
V33-RRAA-3--33M (InPars consist. check)
co0.1957
00.3607
*0.5007
c0.2518
90.4320
*0.3267
c0.6953
Supervised transfer learning with optional unsupervised fine-tuning: transfer from MS MARCO with optional
fine-tuning chesese InPars data
MinILMM-L--300 30M (MS MARCO)
de0.3080
#0.4370
*0.6662
de de0.2295
da0.3923
da0.4646
do0.7476
MiniLM-16630M (MS MARCO consist. check)
4e0.2944
*0.4311
*0.6501
4e0.2692
da0.4730
do0.4320
da0.7898
DeBERTA-V3-IBG.. (MS MARCO)
4e0.3508
"0.4679
d=0.7269
*0.2986
*0,5304
do0.5616
*0,8304
DeBERTA--3-433 (MS MARCO consist.
490.3166
*0.4553
do0.6912
00.3011
*0.5371
de0.5075
40.8165
check)
mono T5-220M (MS MARCO)
Nogucira et
0.3810
0.4909
0.7141
0.3279
0.5298
0.5674
0.7775
2020
1'5-3B (MS
MARCO)
Nogueira
0.3980
0.5281
0.7508
0.3876
0.6091
0.6334
0.7948
2020
momo T5-3B (MS MARCO
InPars) Bonifacio
0.3894
0.5087
0.7439
0.3967
0.6227
0.6297
0.8471
2022
OpenAl API ranking results produced Bonifacio al. (2022 $ denotes experiments that were expensive
InPars denotes the original query-generation method with filtering-out 90% of queries having lowest average log- log-probabilities InPars all denotes the query-gencration method without query filtering, which was used all-domain pretraining. Consist. chocked queries denotes a set of generated queries filtered out (via consistency checking) using DeBERTA-v3. 435M model trained on imPars-gonerater data,
Best results are marked by bold font separately for each training scenario.
Super-scripted labels denote the following statistically significant differences (thresholds are given in the main text): a: between a given neural ranking model and BM25;
b: between (InPars) and (InPars consist. check) when comparing neural ranking models of same type.
between (InPars all consist. check) and (InPars consist. check) when comparing neural ranking models of same type.
d: between (MS MARCO) and (MS MARCO consist. check) when comparing neural ranking models of same type.
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Performance of InPars for Different Generating and Ranking Models.
MS MARCO MRR
TREC DL 2020
NQ MAP nDCG@20 nDCG@10
Robust04
TREC COVID
MAP nDCG@10
nDCG&10
BM25 (ours)
0.1867
0.3612
0.5159
0.2555
0.4285
0.3248
0.6767
ERNIE-v2-335M GPT-3 Curie (6B)
*0.2538
*0.4140
*0.6229
20.2357
0.4016
"0.4277
"0.7411
DRNIE-v2-335M GPT-J (6B)
000.2608
be0,4286
*0,6367
260.2691
0b40,4724
*0.4248
b.0.7750
ERNIF- v2-335M BLOOM (7B)
2b00.2605
be 0,466
*0.6407
cb 0.2852
dcbo0.5102
da0.4215
6a0.7871
DeBERTA-v3-4BR BLOOM (7B)
db.0.2746
ba 000.4385
6e0.6649
6s0.2811
dba0,4987
abc0.4476
be0.8022
Notes: Best results in bold. Super-scripted labels denote statistically significant differences (thresholds are given in the
main text):
between given neural ranking model and BM25;
b: between given neural ranking model and ERNIT-v2-335M trained using OpenAT GPT-3 Curie.
c: between two ERNIE models trained using GPT-J-generated queries and BLOOM- queries;
d: between the DeBERTA model and the ERNIE model trained using BLOOM- queries.
et al. (022), who observed that only monoT5-3B (but not a much smaller monoT3-220M outperformed BM25 on all collections. Strong performance of M-T-3MMM. in our study was due to additional training with mussstency-cheekke data and pre-training on all-domain data (all queries from all collections). To confirm the effectiveness of these procedures, we carried out ablation experiments.
Recall that the consistency-checkel training data was produced using only the DeBERTA-V3-435M model. Moreover, this data was used only to fine-tune a model that was pre-trained using data generated by the original InPars recipe. From Table 3 we can see that for both MiniLM-L6630M and DeBERTA--3-435MM fine-tunining on consistency-checkee
data improves outcomes (which answers RQ3): For 12 measurements out of 14, these improvements are statistically significant (denoted by super-script label "b").
Moreover, all-domain pretraining (instead of training on data generated by the original InPars recipe) further boosts accuracy of MiniLM-L6-30M in all cases: All these improvements are statistically significant (denoted by super-script label "c"). In contrast, all-domaín protraining substantially degrades performance of DeBERTA-v3-435M.
An in-depth investigation showed that for one seed (out of three), the model has failed to converge properly. Therefore, we also analyze the best-seed outcomes which are presented in $ A.3 Table 5 For Mi--LMM,,-30M,, the all-domain pre-training improves the best-seed accuracy in all cases. For DeBERTA-v3-435.. there is either substantial degradation or small decrease/increnee that is not icalll significant (denoted by super-script label "c"). Thus, our biggest model-uulikee a 15x smaller MiniLM-L6-30M does not benefit from all-domain pretraining. However, there is no substantial degradation either.
Supervised transfer learning with optional unsupervised fine-tuning. We found that our ranking models trained on MS MARCO (both MiniLM-L6-30M and DcBERTA-v3-43355 transferred well to other collections in almost all the cases. However, monoT55 models trained on MS MARCO are still substantially more accurate. According to Table [ the average gains over BM25 are (1) 1.21 for MiniLM-30M vs. 1.46 for mono and (2) 1.42 for DeBERTA-v3-485M vs. 1.59 for monoT5-3B. In that, this gap is not reduced by fine-tuning using synthetically generated data. This is different from the fully unsupervised scenario described above, where MiniL.M-166-30M often outperforms monoT5-2200 while DeBERTA-v3-4355 is at par with m-33B.
This is in line with prior findings that large ranking models have better zero-shot transferring effectiveness Ni et al. 2021 Rosa et al. 2022). However, using multi-billion parameter models pre-trained on MS MARCO in commercial setting is problematic from both efficiency and legal standpoints. In particular, MS MARCO
has research-only license
Model-type ablation. To assess the impact of replacing GPT-3 Curie with an open-source model, we carried out experiments using the following ranking models: ERNIE-v2 (Sun et al. 2020) and DeBERTA-v3-435M He el al. 2021). According to Table except for NQ where all generative models were equally good both
Sese terms and conditions: htttps://nicroooott. /ittatiio..obio//eeeee///
12

---

Published in Transactions on Machine Learning Research (02/2024)
GPT-J (Wang& & Komatsuzaki, 2021 and BLOOM (Scao al. 2022 outperformed GPT-3 Curie. This answers RQ2.
The difference in accuracy was particularly big for Robust04. The average relative gain over GPT-3 curie (not shown in the table) were 7.2% for BLOOM and 5.2% for GPT-J Out 14 comparisons, 10 were statistically significant (as denoted by super-seript "b").
In addiiion varying generative model, we assessed the impact of using DeBERTA-v3 instead of ERNIE-v2. This time around, both models were trained using FLO0M-generatee queries. We can see that DeBERTA-v3 was better than ERNIE-v2 except the case of Robust04.
6 Conclusion
We carried out a reproducibility study of InPars (Bonifacio al. 2022) which is a method for unsupervised training of neural rankers. As by-product of this study, developed simple-yet-eefective modification of InPars, which we called InPars-light. Unlike InPars, InPars-light uses only community-trained open-source language model BLOOM (with 7B parameters), 7x-100x smaller ranking models, and re-ranks only top-100 candidate records instead of top-1000.
Not only were we able to reproduce key findings from prior work (Bonifacio et al. 2022p: p, but, combining the original InPars recipe (Bonifacio al., 2022) with (I) fine-tuning on consistency-checkee data (Dai al.) 2022) and (2) all-domain pretraining, we trained outperforming BM25 in the unsupervised setting. In the same scenario, using larger DeBEIKTA-v3448MM model, we largely matched performance of 7x larger monoT5-3B.
efficient yet small model MiniLM-L6633M consistently
In the supervised transfer learning model was still substantially more accurate than a 7x smaller MiniLM-30M ranker and this gap was not reduced by unsupervisod fine-tuning using synthetically generated data.
when pretraining on MS MARCO was used the monoT5-220M
References
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3168-6173 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 00.18653/v1/P19-11
URL http:://aa..aattoooo...oo///////
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. MS MARCO: A human generated machine reading comprehension dataset. arXie preprint Nix:1611.09900 2016.
Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadace, and Rodrigo Nogueira. Inpars: Unsupervised datasct generation for information retrieval. In SIGIR, pp. 2387-2392. ACM, 2022.
Leonid Boytsov and Eric Nyberg. Flexible retrieval with NMSLIB and FlexNeuART. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pp. 32-43, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tla Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen M. Voorhees. Bias and the limits of pooling for large collections. Inf. Retr., 10(6) 10(6):491-508, 2007.
13ehe average gain was obtained by (1) computing relative gain separately for each datasets and key metrics (nDCG or MRR) and (2) averaging these relative gains.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the TREC 2019 deep learning track. arviv preprint or orxii:2003.0055 2020.
Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wat Wei Chang. Promptagator: Fow-shot dense retrieval from 8 examples. CoRR, abs/2209.111 2022.
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl ar urXiu:1810.04805, 2018.
Leo Gao. https://blog. htt.://e.....ttttr...a.aaa. httppppt-mmdeel-ssee
May 2021.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels, 2022. URL http://aaxiv...rgaabba/ 2212. 10496
Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, and Gianluca Demartini. Crowd worker strategies in relevance judgment tasks. In WSDM, pp. 241 249, ACM, 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: decodingg-nhancced bert with disen- tangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 9-7, 2021. OpenReview.net, 2021. URL
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, ftss/1503.02331 2015.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury, Improving efficient neural ranking models with architcrcie knowledge distillation, 2020. URL https://arxiy: org/abs/2010 02666
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. CoRR, abs/2111.091188, 2021.
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. Inpars-v2: Large language models efficient dataset generators for information retrieval, 2023. URL
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstcin, Illia Polosukhin, Jacob Devlin, Kenton Lee, al. Natural Questions: a benchmark for question answering research. Transaclions of the Association for Compulational Linguistics, 7:453 466, 2019.
Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8:193907 1939944, 2020.
Jinyn Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size DNN with outpput-diistribation-beed criteria. In INTERSPEECH, pp. 1910 1914. ISCA, 2014.
Minghan Li and Éric Gaussior. Domain adaptation for donse retrioval through sclf-supervision by pscudo- relevance labeling. CoRR, abs 22122222206552, 2022.
Jimmy Lin, Xueguang Ma, Sheng-Clie Chich Lin, Jheng- Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense represen- tations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Reirieval, pp. 2356 2362, 2021a.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures Human Language Technologies. Morgan & Claypool Publishers, 2021b
14

---

Published in Transactions on Machine Learning Research (02/2024)
Sheng-Chich Lin, Jheng-Hong Yang, and Jimmy Lin. Distilling dense representations for ranking using tightly-coupled teachers. CoRR, abs/22010.1133 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and prodict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2210.133586 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019.
Nya Loshchilov and Frank Hutter. Decoupled weight docay regularization. arXiv preprint orxi: N1111011.00100 2017.
Scan MacAvancy, Andrew Yates, Sergey Feldman, Doug Downcy, Arman Cohan, and Nazli Goharian. Simplified data wrangling with ir datasets. In SIGIR, 2021.
Trina Malveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. High accuracy retrieval with multiple nested ranker. In SIGIR, pp. 437 444. ACM, 2006.
Turii Mokrii, Leonid Boytsov, and Pavel Braslavski. A systematic evaluation of transfer learning and pscudo-labeling with bert-based ranking models. In SIGIR, pp. 2081-2085. ACM, 2021.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning BERT: misconceptions, explanations, and strong basclines. arXiv preprint Xrxv:2000 0...200060.484. 2020.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern andez 'Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wi Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. ArXin, abs/ 211//2112,7899 2021.
Rodrigo Nogucira and Kyunghyun Cho. Passage re-ranking with BERT. arXiu preprint arXiw:1901.04085, 2019.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequeeee model. arXiv preprint arXiv:2003.06% 2020.
John M. Prager. Open-domain
Found. Trends Inf. Retr., 1(2):91-20) 2311,, 2006.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer.
J. Mach. Learn. Res., 10:1400:11 2020.
Revanth Gangi Reddy, Bhavani Iyer, Md. Arafat Sultan, Rong Zhang, Avirup Sil, Vittorio Castelli, Radu Florian, and Salim Roukos. Synthetic target domain supervision for open retrieval QA. In SIGIR, pp- 1793 1797. ACM, 2021.
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman Kyle Lo, lan Soboroff, Ellen M. Voorhees, Lucy Lu Wang, and William R. Horsh. TREC-COVID, rationale and structure of an information retrieval shared lask for COVID-19. J. Am. Medical Informatics Assoc,, 27(9):1431 -1436, 2020.
Stephen Robertson. Understanding inverse document frequency: on theoretical arguments for IDF. Journal of Documentation, 60(5):503 520, 2004. doi: 10.1108/ 00022000011100505558.
Adriana Romero, Nicolas Ballas, Samira Fbrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR (Poster), 2015.
Guilherme Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzich Fadacc, Roberto Lotufo, and Rodrigo Nogucira. In defense of cross-encoders for zero-shot retrieval, 2022. URL https://arxiv.org) abs/ 0ttp/2222.01221
15

---

Published in Transactions on Machine Learning Research (02/2024)
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tan Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. CoRR, abs/ abs/2204.07496, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Llie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Bider- man, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. Coll,, abs/2211.05100 2022.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. CoRR, abs/2104.075400, 2021.
Leslic N. Smith. Cyclical learning rates for training neural networks. WACV, pp. 464-472, 2017.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE 2.0: A continual pre-training framework for language understanding. In AAAI, pp. 8968 8975. AAAI Press, 2020.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heteroge- nous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arxiv:2104. 086663 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. In NTPS, pp. 5998 6008, 2017.
Ellen Voorhees, Overview of the trec 2004 robust retrieval track. In TREC, 2004.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. May 2021.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: generative pscudo labeling for unsupervised domain adaptation of dense retrieval. In NAACL-HLT, pp. 2345-2360. Association for Computational Linguistics, 2022.
Lidan Wang, Jimmy Lin, and Donald Metzler. cascade ranking model for efficient ranked retrieval. In SIGIR, pp. 105 114. ACM, 2011.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In NewrIPS, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shlcifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, taate-of-thheart natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v v
and Alexander Rush. Transformers:
URL http::://cclanthollogg/ wrwphhhoolloyyor//2222222220 emm0220eemmm..mmmmm 1mmnp-demos.6
A Appendix
A.1 Statistical Testing with Multiple-Seed Seed Models
To compute statistical significance using a paired statistical test between results from models A and B, one first has to compute the values of accuracy metric (e.g., MRR) for each query separately. Tet mit and m
16

---

Published in Transactions on Machine Learning Research (02/2024)
be sequences of query-specific metric values for models A and B, respectively. The paired statistical test is then carried out using a sequence of differences m² - mm"
This procedure is not directly applicable when each model is presented by multiple outcomes /seeds. To overcome this issue, we (1) obtain a set of query- and seed-specific metric values, and (2) average them over seeds, thus, reducing the problem to a single-seed statistical testing. In more details, let mit and m be sets of query- and seed-specific metric values for models A and B, respectively. Recall that we have three seeds, 21). Then, obtain seed-average runs m = 1/3 > mf and m 1/3 >3> mi and compute statistical significance using paired difference test.
A.2 Cost and Efficiency
In the following sub-section, we discuss both the ranking efficiency and query-generation cost. Although one may argue that the cost of generation using open-source models is negligibly small, in reality this is true only if one owns their own hardware and generates enough queries to justify the initial investment. Thus, we make a more reasonable assessment assuming that the user can employ a cheap cloud service.
Cost of Query Generation. For the original InPars Bonifacio et al. 2022), , the cost of generation for the GPT-3 Curie model is $0.002 per one thousand lokens. The loken count includes the length of the prompt and the prompting document, [4 We estimate that (depending on the collection) a single generation involves 300 to 500 tokens: long-document collections Robust04 and TREC-COVID both have close to 500 tokens per generalion.
Taking an estimate of 500 tokens per generation, the cost of querying OpenAl GPT-3 Curie API can be up to $100 for Robust04 and TREC-COVID. Assuming that sampling from the 137-B FLAN model (used by (Dai et al. ) to be as expensive as from the largest GPT-3 model Davinci (which has a similar number of parameters), each generation in the Promptagator study (Dai et al. 2022), was 10x more expensive compared InPars study (Bonifacio al.) 2022). Moreover, because Dai et al. (2022) generated one million samples per collection, the Promptagator recipe about two orders of magnitude more expensive compared to TnPars.
In contrast, it takes only about 15 hours to generate 100K queries using RTX 3090 GPU. Extrapolating this estimate to A100, which is about 2x faster than RTX 309 and using the pricing of Lambda GPU cloud, we estimate the cost of generation in our InPars-light study to be under $10 per collection. 16
Efficiency of Re-ranking. A rather common opinion (in particular expressed by anonymous reviewers on multiple occasions) is that using cross-encoders is not practical option. This might be true for extremely constrained latency environments or very large models, but we think it is totally practical to use small models such as MiniLM-L6-33M for applications such as enterprise search. In particular, on a reasonably modern GPU (such as RTX 3090) and MinLm-LL6-3M re-ranking throughput exceeds 500 passages per second (assuming truncation to the first 477 characters). Thus re-ranking 100 documents has an acceptable sub-second latency. In fact, Cohere AI provides re-ranking with neural models as a cloud service 17
Cost of Model Training. Here, all training times are given with respect to a single RTX 3090 GPU. Training and evaluating MiniLM6-30M models had negligūble costs dominated by all-domain pretraining, which took about two hours per seed. In contrast, the all-domain pretraining of DeBERTA--3-435M took 28 hours. However, without all-domain pretraining, the training time itself was rather small, in particular, because we used only a fraction of all generated queries (10K queries in the original InPars training and about 20K queries in the follow-up fine-tuning using consistency checked data).
Aside from all-domain pre-training the two most coneuciun operations were:
Evaluation of model effectiveness on large query sets MS MARCO and NQ, which jointly have about
10K queries;
14 http::/:s/mlaaaaaaocooooooooordd
17

---

Published in Transactions on Machine Learning Research (02/2024)
Table Best-Seed Results for Unsupervised Training
MS MARCO
TREC DL
Robust04
NQ
TREC COVID
MRR
MAP nDCG@10
MAP nDCG&20 nDCG@10
nDCG&10
BM25 (ours)
0.1867
0.3612
0.5159
0.2555
0.4285
0.3248
0.6767
MinILM-LG-30M results
MiniLM (InPars)
.a0.2197
*0.3562
0.5151
+a0.2380
be be0.4029
be0.2415
00.6732
MiniLM (InPars consist. chock)
cbo0.2422
"0.3844
be0.5753
e60,2615
cb-0.4554
eb0.3297
b00.7483
MiniLM (InPars all consist. check)
ca0.2517
c0.3945
.0.5769
c0.2671
c0046991
ce0.3800
*0.7709
DeBERTA-v3-43GM results
DeBERTA (InPars)
*00.2748
"0.4437
*0.6779
hn0.2874
be0.5131
"0.4872
"0.8118
DeBERTA (InPars consist. chock)
ba 0..28847
*0.4479
00.6813
ba0.3043
ba0.5417
c00.4924
*0.8305
DeBERTA (InPars
consist. check)
*0.2804 "0.4414
90.6575 "0.3076
00.5505
co0,4746
"0.8259
Notes: Best results are marked by bold font (separately each model).
Consist. chocked queries denotes
generated queries filtered (via consistency checking) using the DeBERTA-v8-
435M model trained on InPars-gencrated data.
super-scripted labels denote the following statistically significant differences (thresholds are given main text):
a: between a given neural ranking model and BM25;
b: between (InPans) and (InPars consist. check) when comparing ranking models of same type.
c: between (InPars all consist. check) and (InPars consist. check) when comparing ranking models of same type.
Consistency checking using DeBERTA-v3 433M model.
The total effectiveness evaluation time for DeBERTA-V3-435 was about 6 hours (for all collections). The consistency checking, however, took about 48 hours. In the future, we may consider carrying out consistency checking using a much faster model, such MinILMML6-30M.
A.3 Additional Experimental Results
Our rankers were trained using three sceds. However, the case of all-domain pretraining, DeBERTA converged poorly for one seed. Therefore, in Table 5 we present best-seed results.
18


Reviews:
[{"review_id": "MKdEJ6fUL1", "reviewer": "Reviewer_TFJB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n*  It’s a very well-written, clear and well-structured paper.\n* They train rankers using multiple seeds and use statistical tests when measuring improvements.\n* They perform ablation studies of important parts, e.g. the consistency checking.\n\nWeaknesses:\n\n* Given the strong results with the small models, it would be interesting if better results would be possible with larger models. The paper leaves this open.\n* The authors use an anonymous retrieval toolkit. They do however describe experiments that compare with Pyserini and show that those closely match.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of the paper is a reproducibility study of InPars, a method for training a ranking model, using open-source models (specifically BLOOM & GPT-J). They improve the method in several ways, getting strong results with smaller models than the original paper.\n\nThe main idea behind InPars and (the new proposed InPars-Light) is using an LLM with a few-shot prompt to produce synthetic training examples.\n\nThey successfully reproduce InPars and show improved and comparable results can be achieved with a better LLM, improved techniques and smaller models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don’t have any major requests, just a suggestion based on personal preference: Posing the research questions in the introduction, and leaving them unanswered until later, may leave some readers impatient.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vr5aTtPOvs", "reviewer": "Reviewer_ECET", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths. \n\n1) Well written. It's easy to follow, with good examples and well formulated research questions.\n2) All research questions are clearly answered with extensive experiments.\n3) Cost reduction. Experiments showed that smaller models can achieve the same level of result.\n4) Open source. Showed that non-proprietary models can repeat the result, which reduces the cost and risk of data contamination.\n\nWeakness.\n\nThe reviewer didn't find any significant weakness in the paper. Please see the requested changes for smaller questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Previously, the InPars paper showed solid improvement in using LLM to help train ranking models for document retrieval. This paper reproduced and validated the InPars work independently.\n\nMore importantly, the paper noted that the original InPars method used larger proprietary models GPT-3, which is hard to verify that the training is not contaminated with training/test data. Larger models also means higher cost of use.\n\nThe paper then proposed an improved version of InPars, called InPars-light. The new approach uses open source models instead of GPT-3, the training process of which is well understood by the research community. It uses smaller BERT models than the MonoT5 used by the original InPars paper, and leverage consistency checking to help filter the generated data. Results showed that the open-source models and smaller rankers can produce comparable results than the proprietary model and larger ranker. The paper summarized it into a recipe and performed extensive experiments on MS MARCO dataset to answer the questions regarding model size reduction, effectiveness of the consistency checking, and transfer learning abilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Section 3.4 Paragraph 3. The \"average\" run setup is still not clear to the reviewer. The authors said \"we first obtained a ...\", implying there is a \"then...\" but isn't.", "broader_impact_concerns": "There's no broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jz7jFFS7hV", "reviewer": "Reviewer_KGAf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and puts itself nicely in context of past work.\n\nStrengths:\n\n-- Rigorous experimental methodology. The authors use proper statistical testing to validate effectiveness gains over baselines. This strengthens the reliability of the results.\n\n--Comprehensive evaluation on multiple standard IR datasets. The method is evaluated extensively on 5 datasets from different domains. This demonstrates broad applicability.\n\n-- Detailed cost and efficiency analysis. The paper provides a detailed breakdown of computational costs and latency measurements. This is useful for assessing real-world viability.\n\n\nWeaknesses:\n\n-- This paper is just a major replication analysis and no new insights are presented. Essentially, the paper provide sa \"recipe\" for faster InPars but provides no theoretical guarantees of any type. It is just a massive empirical exercise showing several things that \"worked.\"\n\n\n--Does not explore other open-source LLMs besides BLOOM for data generation. Only BLOOM is compared to GPT-3 for training data generation. Studying other large open-source LLMs could reveal further improvements.\n\n--Limited analysis on the impact of different consistency checking thresholds. Consistency checking is shown to help, but only a single threshold is tried. More analysis on the threshold could be beneficial.\n\n--No exploration of other prompt formats beyond the vanilla 3-shot prompt. The impact of other prompting strategies is not studied. This could potentially further improve results.\n\n\n-- Provides limited details on all-domain pretraining procedure. More implementation details on this technique that improved MiniLM results could be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "-- The paper presents a reproduction study of InPars (Bonifacio et al., 2022), a method for unsupervised training of neural rankers using synthetic training data generated via few-shot prompting of large language models (LLMs).\n\n-- As a byproduct of the reproduction study, the authors propose InPars-Light, a more cost-effective modification of InPars that uses smaller ranking models like MiniLM and DeBERTA instead of large monoT5 models, and generates training data using the open-source BLOOM LLM instead of proprietary GPT-3.\n\n--Key findings are:\n \n - InPars-Light with DeBERTA-v3 (435M params) matches effectiveness of the much larger monoT5-3B (3B params) at lower cost.\n \n - Even smaller 30M parameter MiniLM models consistently outperform BM25 when trained with InPars-Light.\n \n - Open-source BLOOM model generates better synthetic training data than GPT-3 for this task.\n \n - Additional techniques like consistency checking and all-domain pretraining further improve InPars-Light results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-- Analyze if there are benefits from using other large open-source LLMs for training data generation.\n\n-- Study the impact of different consistency checking thresholds more thoroughly.", "broader_impact_concerns": "-- The techniques presented could make prompt-based training of neural rankers more accessible due to reduced computational requirements.\n\n-- Reliance on proprietary models like GPT-3 is reduced through the use of open-source LLMs.\n\n-- No major societal concerns identified with the presented techniques.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally commend the paper for its well-written and clear presentation, thorough experimental methodology, and valuable insights into reproducibility and cost-effectiveness of neural rankers. The main contributions involve a reproducibility study of InPars and the introduction of InPars-Light, a more cost-effective modification. While reviewers lean towards acceptance, they suggest a weak opposition due to the perceived lack of novelty. There are recommendations to explore other open-source language models for data generation, delve into consistency checking thresholds, and provide more details on certain procedures. Despite these suggestions, the paper is acknowledged for its rigorous empirical exercise and insights into what works in the context of prompt-based training of neural rankers. Therefore, I recommend accepting the paper with minor revisions to address the concerns raised by reviewers and enhance the clarity of certain aspects.
Decision: Accept (poster)