Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval
Authors: Fan Jiang
Qiongkai Xu
Tom Drummond
Trevor Cohn
Venue: EMNLP 2023 Findings (2023)
Keywords: dense retrieval
iterated learning
alternating distillation
bootstrapping
TL;DR: A novel alternating bootstrapping training framework for zero-shot dense retrieval.

Abstract: Neural 'dense' retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability.
Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models.
We present $\texttt{ABEL}$, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. 
Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever.
By iterating this loop, the two components mutually enhance one another's performance. 
Experimental results demonstrate that our unsupervised $\texttt{ABEL}$ model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark.
Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training.
By either fine-tuning $\texttt{ABEL}$ on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.\footnote{Source code is available at \url{https://github.com/Fantabulous-J/BootSwitch}.}

Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval
Fan Jiang, Qiongkai Xu, Tom Drummond and Trevor Cohn'
School of Computing and Information Systems The University of Melbourne, Victoria, Australia fan. f ji 1ormg@1@steett unimelb. edu. {qiongkai. xu, tom. drummond, trevor wwwnnnonimelll.....o edu. au
Abstract
zero-shot settings while maintaining the factorised dual-encoder structure, such as pre-training models on web-scale corpus (Izacard et al., 2022) and learn- ing from cross-encoders through distillation (Qu et al., 2021). Other alternatives seek to trade effi- ciency for performance by using complex model architectures, such as fine-grained token interaction for more expressive representations (Santhanam et al., 2022) and scaling up the model size for bet- ter model capacity (Ni et al., 2022). Another line of work trains customised dense retrievers on tar- get domains through query generation (Wang et al., 2022; Dai et al., 2023). This training paradigm is generally slow and expensive, as it employs large language models to synthesise a substantial number of high-quality queries.
Neural 'dense' retrieval models are state of the art for many datasets, however these models of- len exhibit limited domain transfer ahility. Ex- isting approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, massive external mod- els. We present ABEL, a simple but effective unsupervised method to enhance passage re- trieval in zero-shot settings. Our technique fol- lows straightforward loop: a dense retriever learns from supervision signals provided by reranker, and subsequently, the reranker is up- dated based on feedback from the improved retriever. By iterating this loop, the com- ponents mutually enhance one another's per- formance. Experimental results demonstrate that unsupervised ABEL model outperforms both leading supervised and unsupervised re- trievers on the BEIR benchmark. Mcanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training. By either fine-tuning ABEL on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-ar results.
In this paper, we present ABEL, an Alternating Bootstrapping training framework for unsupervised dense rEtrievaL. Our method alternates the dis- tillation process between dense retriever and a rerunker by switching their roles as teachers and students in iterations. On the one hand, the dense retriever allows for efficient retrieval due to its fac- torised encoding, accompanied by compromised model performance. On the other hand, reranker has factorisation constraint, allowing for more fine- fine-grained and accurate scoring, but at the cost of intractable searches. Our work aims take ad- vantage of both schools by equipping the dense re- triever with accurate scoring by the reranker while maintaining search efficiency. Specifically, i) the more powerful but slower reranker is used to assist in the training of the less capable but more efficient retriever; ii) the dense retriever is employed to im- prove the performance of the reranker by providing refined training signals in later iterations. This al- temnting learning process repeated iteratively enhance both modules.
1 Introduction
Remarkable progress has been achieved in neu- ral information retrieval through the adoption of the dual-encoder paradigm (Gillick et al., 2018), which enables efficient search over vast collections of passages by factorising the model such that the encoding of queries and passages are decoupled, and calculating the query-passage similarity using dot product. However, the efficacy of training dual- encoders heavily relies on the quality of labelled data, and these models struggle to maintain com- petitive performance on retrieval tasks where dedi- cated training data is scarce (Thakur et al., 2021).
Various approaches have been proposed en- hance dense retrievers (Karpukhin et al., 2020) in
Compared with conventional bootstrapping ap- proaches (Alberti et al., 2019; Zelikman et al., 2022), wherein the well-trained model itself is
'Now at Google DeepMind
'Souree code is availahle at httpps://githh: com/ B

---

used to discover additional solutions for subse- quent training iterations, our method considers one model (i.e., teacher) as the training data genera- tor to supervise another model (i.e., student), and their roles as teacher and student are switched in every next step. This mechanism naturally cre- ates mutual-learning paradigm to enable iterative bidirectional knowledge flow between the retriever and the reranker, in contrast the typical single- step and unidirectional distillation where student learns from a fixed teacher (Miech et al., 2021). Through extensive experiments on various datasets, we observe that ABEL demonstrates out- standing performance in zero-shot settings by only using the basic BM25 model as initiation. Addi- tionally, both the retriever and reranker components involved in our approach can progressively en- hanced through the bootstrapping learning process, with the converged model outperforming more than ten prominent supervised retrievers and achieving tfte-of-the-art performance. Meanwhile, ABEL is efficient in its training process by exclusively em- ploying sentences from raw texts queries, rather than generating queries from large language mod- els. The use of the simple dual-encoder architec- ture further contributes to its efficient operation. In summary, our contributions are:
clude the necessary information. From collection of passages the corpus. P {Pi P2..... .P a retriever 'D fetches top-k passages (T()) {P1.P2.... .2. ... -Pk from P that are most relevant to a specific query q- Optionally, a reranker R. is also employed to fine-grain the relevance scores for these retrieved passages.
2.1 Dense Retrieval Model
The dense retrieval model (retriever) encodes both queries and passages into dense vectors using a dual-encoder architecture (Karpukhin et al., 2020). Two distinct encoders are applied to transform queries and passages separately, then, relevance score is calculated by dot product,
D p; = EE(g; 0g) E(p: 6p),
(1)
where E(: 0) are encoders parameterised by Op for passages and 6, for queries. The asymmetric dual- encoder works better than the shared-encoder archi- tecture in our preliminary study. For efficiency, all passages in P are encoded offline, and an efficient nearest neighbour search (Johnson et al., 2021) is employed to fetch top-k relevant passages.
2.2 Reranking Model
The reranking model (reranker) adopts a cross- encoder architecture, which computes the relevance score between a query and a passage by jointly encoding them with cross-attention. The joint en- coding mechanism is prohibitively expensive to be deployed for large-scale retrieval applications. In practice, the joint-encoding model is usually ap- plied as a reranker refine the relevance scores for the results by the retriever. The relevance score by the reranker formalised as,
1. We propose an iterative approach to bootstrap the ability of dense retriever and reranker without relying on manually-creeate data.
2. The empirical results on the BEIR benchmark show that the unsupervised ABEL outperforms a variety prominent sparse and supervised dense retrievers. After fine-tuning ABEL using supervised data or integrating it with off-the-- shelf supervised dense retrievers, our model achieves new state-of-the art performance. 3. When applying ABEL tasks that unseen in training, we observe it demonstrates re- markable generalisation capabilities in com- parison to other more sophisticated unsuper- vised dense retrieval methods.
R(g. P: FFN
where E(. ()) is a pre-trained language model parameterised by Φ. In this work, we adopt the en- coder of T5 (LncTT5) (Liu et al., 2021) E. A start- of-sequence token is appended to each sequence, with its embedding fed randomly initialised single-layer feed-forward network (FFN) caleu- late the relevance score.
4. To the best of our knowledge, we are the first to show the results that both dense retriever and cross-encoder reranker can be mutually improved in closed- ffrm learning loop, with- out the need for human-annotate labels.
3 Alternating Distillation
2 Preliminary
We propose an unsupervised alternating distilla- tion approach that iteratively boosts the ability of a retriever and a reranker, as depicted in Fig. 1.
Given a short text as a query, the passage-retrieval task aims at retrieving a set of passages that in-

---

Iterate ( T times
1. Retriever Training (Warm-up)
Passage Collection P
2. Reranker Training
3. Retriever Training
Initial Retriever
Reranker RRp
Retriever D
Sentenne Splft
Query o
D
BM25
V/
(q. P..
{g. pp 1.
Y
P 3
Initial training data
Soft labels
Updated Training data
Refresh top-k Predictions Update Soft Labels
Figure The overview of the alternating bootstrapping training approach for zero-shot dense retrieval.
Alg. I outlines the proposed method with three ma- jor steps. Our approach starts with warming up retriever by imitating BM25. Subsequently, re- cursive learning paradigm consisting of two steps
Algorithm 1: Alternating Bootstrapping
Training for Zero-Shot Dense Retricval
Input :Pre-trained language models (c.g.. T5), and Passage collection P.
Split passages into sentences as queries Q.
is conducted: (1) training a reranker based on the labels extracted from the retriever by the last itera-
2 /* Step 1: Warm-up Retriever Training */
3 Retrieve top-k predictions P (4) for each q using BM25 model.
tion; (2) refining the dense retriever using training signals derived from the reranker by the last step.
Extract initial training data Yo using Eq. and 4,
Train a warm-up retriever DJ using Do-
for < 1 T do
3.1 Retriever Warm-up
/* Step 2: Reranker R.s Training
The training starts with constructing queries from texts (line 1) and training warm-up dense retriever DJ by imitating an unsupervised BM25 model (lines 3-5).
8
Retrieve top-k predictions P Pop-: (g) for each
q E @ using the most recent retriever D -1 Extract soft lahels for cach (g. p) using D Vioft [D(g, pi D ") E -
Train reranker R2 with V using Eq. 5.
Query Construction We use a sentence split- ter to chunk all passages into multiple sentences. then we consider these sentences as cropping- sentence queries (Chen al., 2022). Compared queries synthesised from fine-tuned query gen- erators (Wang et al., 2022) or large language mod- queries i)
Step 3: Retriever D, Training
*/
Rerank (9) using R2
Extract updated training data V/ from reranking list using Eq. and Fine-tune D/g using 3/ from DS.
end for return D/P
els (Dai et al., 2023), can be cheaply scaled up without relying on super- vised data or language models and ii) have been shown effectiveness in pre-training dense retriev- ers (Gao and Callan, 2022; Izacard et al., 2022).
positive p+ p+))), while the bottom k- E Zt pas- sages are treated as hard negatives P (4), follow- ing Chen et al. (2022),
Training Data Initialisation The first challenge to our unsupervised method is how to extract effec- tive supervision signals for each cropping-sentence query to initiate training. BM25 (Robertson and Zaragoza, 2009) is an unsupervised sparse retrieve model, which has demonstrated outstanding per- formance in low-resource and out-of-domain set- tings (Thakur al., 2021). Specifically, for a given query / E Q, we use BM25 to retrieve the top- k predictions Ph (4)))))), among which the high- est ranked t+ E zt passages are considered as
P (g) = {plp E ()(g).(()) < }+
(3)
P (1) = (plp E Pb(1), r(p) 2
(4)
where the initial D uses BM25 and r(p) means the rank of a passage p. Then, we train a warm-up re- triever based on these extracted training examples.
3.2 Iterative Bootstrapping Training
We iteratively improve the capability of the re- triever and the reranker by alternating their roles as teacher and student in each iteration.

---

In the 1-th iteration, we first use the most recent retriever Dj to retrieve top-k passages pk - (q) that are most relevant to each query q E Q and gen- erate soft labels (gg, p; 0 for each (q, p) pair accordingly. Then we use all such soft labels to train a reranker R. as described in Alg. I lines 8-10 and $3.3.1. The second step is to train the t-th retriever. We employ R to rerank PA -z-1(0) (q) to obtain a refined ranking list, from which updated supervision signals are derived as Alg. lines 12- 13. We train new retriever Dÿ with these examples discussed in Alg. line 14 and $3.3.2. Train- ing iterations are repeated until no performance improvement is observed. Note that, in order to mitigate the risk of overfitting the retriever, for all iterations we refine the warm-up retriever Dj using the newest training examples but refresh top-k pre- dictions and update soft labels for reranker training with the improved retriever. Similarly, avoid the accumulation of errors in label generation, re- initialise the reranker using pre-trained language models at the start of each iteration, rather than fine- tuning the model obtained from the last iteration. Please refer to Table 3 for the empirical ablations.
T(-99, Pg. 0) and S(-1q, py: の) are the distributions from the teacher retriever and the student reranker, respectively. Our preliminary experiment shows that adding noise to the reranker's ' inputs (e.g., word deletion) can further enhance the performance. For more details, plcase refer to Table 4 in Appendix B.
3.3.2 Retriever Training For each query 9, we randomly sample one positive p and one hard negative p from P (q) (Eq. 3) and P (4) (Eq. 4), respectively. In practice, we use in-batch negatives (Karpukhin et al., 2020) for effi- cient training. The dense retriever Dn is trained by minimising the negative log-likelihood of the posi- tive passages using contrastive learning (CL) (Had- sell et al., 2006),
cxp( exp(s(g;,P) :::;;: ;0)
LeL. = log
B Σ Σ
pe(pp 27 exp(s(g.,;;)) B))
where [B] is the size of a batch B. Similarly, we find that injecting noise into the inputs of the re- triever during training results in improved perfor- mance, as demonstrated Table in Appendix B. We believe that manipulating words in queries and passages can be seen as a smoothing method to facilitate the learning of generalisable matching signals (He et al., 2020), therefore, preventing the retriever from simply replying on examining the text overlaps between the cropping-sentence query and passages. Additionally, adding noise tweaks the semantics of texts, which encourages the dense retriever to acquire matching patterns based on lex- ical overlaps in addition to semantic similarities.
3.3 Retriever and Reranker Training In each iteration, we fine-tune the reranker and then the retriever as follows.
3.3.1 Reranker Training
The reranker is trained to discriminate positive sam- ples from hard negative ones using a cross-entropy (CE) loss,
LCE = log exp(s(q,pt:d) Φ))
Epe(otjup; exp(s(1.1:
3.4 Discussion
In our preliminary experiments, we observed that using hard labels to train the rereanker yields poor results. The reason is that hard labels use binary targets by taking one query-passage pair as positive and the rest pairs as negative, failing to provide informative signals for discerning the subtle dis- tinctions among passages. To address this problem, we consider using the soft labels generated by a dense retriever Da to guide the reranker training. These labels effectively capture the nuanced seman- tic relatedness among multiple relevant passages. Specifically, we employ the KL divergence loss:
The novelty of our approach lies in two aspects: i) We introduce a separate reranker model for training label refinement. This allows the retriever training to benefit from an advanced reranker, which fea- tures more sophisticated cross- cross-encoding architec- ture. This design is effective compared to training the retriever on labels extracted from its own pre- dictions (see the blue line in Figure 5(a)); ii) We create a mutual- paradigm through iterative alternating distillation. In our case, we consider the retriever a proposal distribution generator, which selects relatively good candidates, and the reranker as reward scoring model, which mea- sures the quality of an example as a good answer. At each iteration, the improved reranker can be used to correct the predictions from the retriever,
LKL (S py. () 0)) where P, is the retrieved set of passages regard- ing to query g. which are sampled from pk. D (g).

---

Setting
Supervised
Unsupervised
ANCE
Contriever
RetroMAE
GTR -XXLX
GPL
PTRE
COCO-DR
ColBERTV2
SPLADE+
DRAGON+
ABEL-FT
BM25
REALM
SiMCSE
Contriever
ABEL
Model
Model Size
110M 110M 110M 4.8B
66M 110M 110M 110M 110M 220M 220M
110M 355M 110M 220M
QGen. Size
220M
137B
220M
Retriever Type
dense dense dense dense dense dense dense mul-vec sparse dense dense sparse dense dense dense dense
Distillation
X
X
TREC-COVID
64.9
59.6
77.2
50.1
70.0
72.7
78,9
73.8
71.1
75.9
76.5
65.6
10.0
38.6
26.2
72.7
RioASQ
30.9
38.3
42.1
32.4
44.2
42,9
50.4
43.3
45.4
46.5
23.0
30.6
41.3
NFCorpus
23.5
32.8
30.8
34.2
34.5
33.4
35.5
33.8
34.5
33.9
35.1
32.5
20.9
14.0
32.3
33.8
NQ
44.4
49.5
51.8
56.8
48.3
50.5
56.2
54.4
53.7
50.2
32.9
21.8
12.6
26.9
42.0
HotpotQA
45.1
63.8
63.5
59.9
58,2
60.4
61.6
66.7
68.6
66.2
65.7
60.3
40.5
23.3
45.5
61.9
FiQA-2018
29.5
32.9
31.6
46.7
34.4
40.4
31.7
35.6
35.1
35.6
34.3
23.6
6.1
14.8
25.0
31.1
Signal-
IM IM (RT)
24.9
19.9
26.5
27.3
27.6
27.1
29.6
30.1
28.0
33.0
13.8
21.4
23.7
30.8
TREC-NEWS
38.4
42.8
42.8
34.6
42.1
40.3
39.4
44.4
45.4
39.8
26.
25.7
39.4
48.0
Robust04
39.2
47.6
44.7
50.6
43.7
44.3
45.8
47.9
50.0
40.8
20.7
30.0
34.5
48.4
ArguAna
41.9
44.6
43.3
54.0
53.8
49.3
46.3
52.1
46.9
56.9
31.5
18.7
45.6
40.6
50.5
Touché-2022
24.0
23.0
23.7
25.5
26.6
23.8
26.3
24.4
26.3
19.5
36.7
4.9
11.6
21.7
29.5
CQADupStack
29.8
34.5
34.7
39.9
35.7
37.0
34.1
35.4
36.9
29.9
10.5
20.2
28.2
35.0
Quora
85.2
86.5
84.7
89.2
83.6
86.7
85.2
81.4
87.5
84.5
78.9
39.8
81.5
83.2
83.9
DBPedia
28.1
41.3
39.0
40.8
38.4
36.4
39.1
44.6
44.2
41.7
41.4
31.3
21.4
13.7
29.9
37.5
SCIDOCS
12.2
16.5
15.0
16.1
16.9
16.3
16.0
15.4
15.9
15.9
17.4
15.8
7.0
7.4
15.3
17.5
FEVER
66.7
75.8
77.4
74.0
75.9
76.2
75.1
78.5
79.6
78.1
74.1
75.3
42.7
20.1
61.9
74.1
ChiVate-FEVE
20.0
23.7
23.2
26.7
21.4
21.1
22.7
22.7
21.3
15.4
17.6
17.2
25.3
SciFact
51.0
67.7
65.3
66.2
67.4
62.3
70.9
69.3
69.9
67.9
72.6
66.5
46.1
38.5
64.6
73.5
Avg. PTR-11
37.0
43.8
44.5
44,9
45.5
45.5
45.7
46.2
47.1
46.5
46.9
36.1
21.0
27.9
30.3
46.1
Avg. BEIR-13
41.3
47.5
48.2
49.3
48.6
49.2
49.9
50.3
50.2
50.0
44.0
22.7
26.1
37.7
48.7
Avg. All
38.9
44.5
45.4
45.8
45.9
46.2
47.4
47.4
47.5
42.3
21.6
24.6
35,9
46.5
Table I: Zero-shot retrieval results on BEIR (nDCG@ 10). The and second-best results are marked in bold and underlined. Methods that train dedicated models for each of the datasets noted with f. Highlighted rows represent semantic relatedness tasks. QGen. means query generator. Please refer to Appendix A.2 for baseline details.
therefore reducing the number of inaccurate labels in retriever training data. Meanwhile, the improved retriever is more likely to include more relevant passages within its top-k predictions and provide nuanced soft labels, which can enhance the reranker training and thus, increase the chance of finding correct labels by the reranker in the next iteration. As the training continues, (1) the can- didates of correct answers can be narrowed down by the retriever (i.e., better proposal distributions). shown in Fig. 4 and (2) the accuracy of finding correct answers from such sets can be increased by the reranker, shown in Fig. 2(b). Overall, our framework (i.e., ABEL) facilitates synergistic advancements in both components, ultimately en- hancing the overall effectiveness of the retrieval system.
Experiments
4.1 Dataset
Our method is evaluated the BEIR bench- mark (Thakur al., 2021), which contains 18 datasets in multiple domains such as wikipedia and biomedical, and with diverse task formats includ- ing question answering, fact verification and para- phrase retrieval. We use nDCG @ 10 (Järvelin and Kekäläänen, 2002) as our primary metric and the av- erage nDCG@ 10 score over all 18 datasets is used for comprchensive comparison between models. BEIR-13 (Formal et al., 2021) and PTR-11 (Dai et al.. 2023) are two subsets of tasks, where BEIR- 13 excludes CQADupStack, Robust04, Signal- TREC-NEWS, and BioASQ from the calculation average score and PTR-1 further removes NQ and

---

Quora. We also partition the datasets into query search (e.g., natural questions) and semantic re- latedness (e.g., verification) tasks, in accordance with Santhanam et al. (2022),
TART
48
43.0
X-Large
Large
42.8
46
Avg. nDcag10
44
Base
42.4
4.2 Experimental Settings
42.2
42
We use Contriever to initialise the dense retriever. The passages from all tasks in BEIR are chunked into cropping-sentenee queries, on which a single retriever ABEL is trained. We initialise the reranker from t5-base-1m-addee train single reranker ABEL-Rerante similarly. We conduct the alternating distillation loop for three iterations and observe that the performance both the retriever and reranker converges, and we take ABEL in iteration and -BEL-Reras in iteration 3 for evaluation and model comparison. We fur- ther fine-tune ABEL on MS-MARCO (Bajaj et al., 2016) to obtain ABEL-FT, following the same train- ing recipe outlined in Izacard et al. (2022). In addition, we also evaluate ABEL and ABEL-Rerand on various datasets that are unseen during training to test the generalisation ability. More details are in Appendix A.
42.0
40
41.8
41.6
of parameters (a) Scale
Iteration (Base Model)
(Raffel et al., 2020) and
Bootstrapping Iterations
Figure 2: Reranking results on BEIR, with (a) the com- parison of models with various sizes and (b) the accu- racy of the reranker in iteration using the base model.
from the rropping-sentenee queries, and such for- mat mismatch leads to the inferior performance of ABEL. For lexical matching tasks, such as Touché- 2020, ABEL surpasses the majority of dense retriev- by a considerable margin. We attribute this success the model's ability to capture salient phrases, which facilitated by learning supervision signals from BM25 in retriever warm-up training and well preserved in the following training iter- ations. Finally, ABEL outperforms GPL and PTR, even though they have incorporated high-qualing synthetic queries and cross-encoder distillation in training. This observation demonstrates that a triever can attain promising results in zero-shot settings without relying on synthetic queries.
4.3 Experimental Results
Retriever Results Both unsupervised and super- vised versions of our model are compared with a range of corresponding baseline models in Table 1. The unsupervised ABEL outperforms various lead- ing dense models, including models that are super- vised by MS-MARCO training queries. The super- vised model, ABEL -FT, achieves 1.0% better over- all performance than ABEL. ABEL-FT also surpasses models using the target corpora for pre-traing tre-training (COCO-DR), employing fine-grained token inter- action (ColBERTv2), using sparse representations (SPLADE++)) with larger sizes (GTR-XXL), and using sophisticated training recipes with diverse supervision signals (DRAGON+).
For the supervised setting, the performance of ABEL can be improved by fine-tuning it super- vised data in dealing with natural questions. The major performance gains from query-search tasks and semantic relatedness tasks, which involve human-like queries, such as NQ (42.0 to 50.2) and DBPedia (37.5 to 41.4). On other datasets with short sentences as queries (e.g., claim verification), the performance of ABEL -FT degrades but is compa- rable to other supervised retrievers. This limitation can be alleviated by combining ABEL and ABEL-FT, thereby achieving performance improvements both types of tasks, as illustrated in the last bars Figure 3.
Considering semantic relatedness tasks, such as Signal- Climate-FEVER, and SciFact, ABEL generally achieves results superior to other super- vised dense retrievers. For query-search tasks, par- ticularly question answering tasks like NQ and Hot- potQA, ABEL underperforms many dense retrievers. We attribute such outcomes to the differences in query styles. Semantic relatedness tasks typically use short sentences queries, which aligns with the cropping-sentern
guestion-answering
Reranker Results Figure 2 shows the averaged reranking performance on subsets of BEIR, ex- cluding FEVER and HotpotQA from PTR-11. As shown in Figure 2(b), the reranker is able to achieve improvements as the bootstrapping itcration pro- gresses, providing strong evidence for the effec-
query format employed in our work. However, query query-search tasks often in- volve natural questions that deviate significantly

---

AMB
WQA
GAT
LSO
CSP
Avg.
Different Pre-trained Models
BM25
96.3
57.7
64.1
31.8
27.7
55.5
Initialisation
DRAGON
ABEL
REALM
88.4
58.2
43.0
7.2
5.9
40.5
SimCSE
92.3
48.9
41.7
9.3
13.8
41.2
BERT
46.8
47.2
Contriever
47.5
Contriever
45.2
67.2
18.3
28.4
51.0
RetroMAE
47.4
48.0
ABEL
97.4
69.6
69.3
22.3
35.4
58.8
#2. Different Training Corpus
Table 2: Zero-shot cross-task retrieval results of unsu- pervised models based nDCG@ 10. The best and second-best results are marked in bold and underlined. The tasks are AMB=AnbigQA, WQA=WikiQA, GATT=GooAQ-Tecchmiele LSO=LinkSO-Pytton, CSP=CoddSSanchNeee-Pthooo
Corpus
Avg.
A
Wikipedia & CCNet
35.9
MS-MARCO
41.0
+5.1
BEIR
45.1
+9,2
#3. Model Re-initialisation
Strategy
w/o re-init
w/ re-init
Avg.
45.7
46.5
tiveness of our iterative alternating distillation ap- proach. The final reranker model, ABEL-Rerank (i.e., l= = 3), as illustrated in Figure 2(a), enhances the performance of ABEL by 1.6%, surpassing su- pervised models of similar parameter sizes. It is noteworthy that ABEL-Rerank outperforms unsu- pervised SGPT (Muennighoff, 2022) and zero-shot UPR (Sachan et al., 2022), despite having much fewer parameters, and comparable to PTR, which creates dedicated models for each task and employs high-quality synthetic queries. With the increase in model size, wc consistently observe improvements with ABEL-Rerank and it outperforms TART (Asai et al., 2022) and MonoT5 (Nogueira et al., 2020) us- ing only fraction of the parameters. This finding demonstrates the potential of incorporating more capable rerankers to train better retrievers. We leave this exploration to future work. Please refer to Tables and Appendix for further details.
Table 3: Ablations on pre-trained models, the cor- pus used for training, the effects model re- initialisation. The average performance (nDCG @ 10) BEIR is reported. Note that for training corpus ablation, results retrievers trained with iteration / are reported. Wikipedia & CCNet are the training corpora of Contriever and A indicates performance gains over Contriever.
use more advanced pre-trained checkpoints. This demonstrates that our approach is orthogonal existing unsupervised pre-training methods, and further gains are expected when more sophisticated pre-trained models are available.
We compare models trained us- ing cropped sentences from different corpus the training data. As shown in Table #2, the method trained using MS-MARCO corpus significantly better than the vanilla Contriever (+5.1%) but is inferior to the one using diverse corpus from BEIR, and we believe that the diverse corpora we used in training is one of the factors to our success.
Training Corpus
Cross-Task Results As shown in Table 2, when directly evaluating ABEL on various tasks unseen during training, it consistently achieves significant improvements over Contriever (+7.8%) and out- performs BM25 and other advanced unsupervised retrievers. These results show that ABEL capa- ble of capturing matching patterns between queries and passages that can be effectively generalised to unseen tasks, instead of memorising training cor- pus to achieve good performance. Please refer to Appendix E for more results.
Model Re-initialisation We use re-initialisation to avoid the accumulation from biased errors in early iterations. At the start of each iteration, we re-initialise the retriever with the warm-up retriever and the reranker using pre-trained language models. respectively, rather than continuously fine-tuning the models obtained from the last iteration. Table 3 #3 shows the overall performance is increased by 0.8 % using this re- re-initialisation technology.
4.4 Analysis
Pre-trained Models Table 3 #1 compares approach with DRAGON when using different pre- trained models for initialisation. ABEL outperforms DRAGON consistently using aligned pre-trained models, with up to +0.6% gains. Morcover, our method exhibits continued improvement as we
We in- vestigate whether ABEL can advance supervised trievers. We merge the embeddings from ABEL with different supervised models through concatenation. Specifically, for given query q and passage p, and
Combination with Supervised Models

---

52
52
ABEL only
only
ABEL
ABEL
Avg. nDCG@10
50
4313
48.5
46
45
AVD. nDCG@10
50
synthetic-queries
48
46.3
47.4
47.5
44-
ABEL
42.1
46
64.5
45.4
43
syrthetic-quriies
44
42
44
41
SR
38
Reration
Task Group 2)
ANCE
Contriever
RelroMAE
DRAGON+
ABEL+
(a)
(h)
Figure 3: The comparison of combining ABEL and su- pervised dense retrievers (Model & ABEL) on BEIR.
Figure 5: Results on BEIR by removing the reranker component or using synthetic queries for training, where QS=Query- -earchh and SR=Semantic-Relateee
46.5
665
10-
Avg 1055013
15.0
45.1
44.3
A2.3
Self Supervision We investigate the necessity of taking the reranker as expert in ABEL. Specif- ically, we the top-k predictions of the latest retriever to extract training data at each iteration, instead of employing separate reranker (i.e., with- out lines 8-10 8-10 in Alg. 1). The blue line in Figure 5(a) indicates that the retriever struggles to improve when using itself as the supervisor. By investigat- ing small set of generated labels, we notice the extracted positive passages for most queries quickly converge stable set, failing to offer new signals in the new training round. This highlights the sential role of the expert reranker, which iteratively provides more advanced supervision signals.
10.0
17.5
15.9
15.0
Aadz
408-3
ABE 3
BML5
CT Http
ABG-1
4B0L
ABELL
Figure 4: The effect of bootstrapping the retrieval ability on BEIR. CT and CT (MS) indicate Contriever and Contriever (MS), respectively. Best Tasks: the number of tasks on which the models perform the best.
dense retrievers E and Ejp we compute the rel- evance score s(q, p) E J.JTT E Eg] E EN Ej. The results shown in Figure 3 indicate ABEL can be easily integrated with other models achieve significant performance improve- ment, with an average increase of up to 4.5% on RetroMAE and even a 1% gain on ABEL-FT. Be- sides, the benefit is also remarkable (+10%) when combining ABEL with weak retrievers (i.e., ANCE). Overall, we observe that the ensembling can re- sult in performance gains in both query-search and semantie-relatedness tasks, as demonstrated in Ta- ble 11 in Appendix F.
Synthetic Queries We assess the utility of synthetic queries in approach by replac- ing cropping-sentence queries with synthetic queries from a query-generator fine-tuned on MS- MARCO. The results Figure 5(a) show that us- ing synthetic queries is less effective and exhibits similar trends, with the performance improving consistently as the iterative alternating distillation progresses.? Splitting task groups, we observe syn- thetic queries yield a larger performance drop semantic-relatedness tasks than query-search tasks, in Figure 5(b). We attribute this disparity the stylistic differences between the training and test queries. Synthetic queries exhibit similarities to natural questions in terms of style, akin to those found in question-answering datasets. In contrast, semantic relatedness tasks usually involve short- sentence queries (e.g., claim) that closer to cropping-sentence queries. This finding empha-
Effect of Bootstrapping Figure 4 presents the comparison of ABEL'S performance throughout all bootstrapping iterations against the BM25 and the supervised Contriever. We observe that the accu- racy of the retriever consistently improves as the training iteration t progresses. Specifically, ABEL matches the performance of the supervised Con- triever the first iteration, and further gains achieved with more iterations / < 2. The perfor- mance converges at iteration where the results on six tasks are inferior to those achieved at iteration 2.
/https://public.. ukp. informatik. tu-darmstadt. fe//kwang/ggl/eee stwp://eeeateeddeee datbbiir
Please refer to Figure in Appendix G for results on each individual task.
UUsing large language models for query generation may yield better results. We leave this exploration to future work.

---

sises the importance of aligning the formats of training queries with test queries in zero-shot set- tings. Please refer to Figure in Appendix for results comparison in each individual task.
ity by assessing whether the solution aligns with the desired objective (Alberti et al., 2019; Zelik- man et al., 2022: Dai et al., 2023). This concept can also be extended to include multiple models. One method is the iterated expert introduced by An- thony et al. (2017), wherein an apprentice model learns to imitate an experi and the experi builds on improved apprentice to find better solutions. Our work adapts such a paradigm to retrieval tasks in zero-shot manner, where a notable distinction from previous work is the novel iterative alternating dis- tillation process. For cach iteration, the roles of the retriever and the reranker as apprentice and expert are alternated, enabling bidirectional knowledge transfer to encourage mutual learning.
5 Related Work
Neural Information Retrieval Neural retriev- ers adopt pre-trained language models and fol- low dual-encoder architecture (Karpukhin al.. 2020) to generate the semantic representations of queries and passages and then calculate their se- mantic similarities. Some effective techniques have been proposed to advance the neural retrieval models, such as hard negative mining (Xiong et al., 2021), retrieval-oriented pre-training objec- tives (Izacard et al., 2022), and multi- vector rep- resentations (Khattab and Zaharia, 2020). All of these approaches require supervised training data and suffer from performance degradation on out- of-domain datasets. Our work demonstrates the possibility that an unsupervised dense retriever can outperform a diverse range of state-of stte-offthee-an of-thheart su- pervised methods in zero-shot settings.
Conclusion
In this paper, we introduce ABEL, an unsupervised training framework that iteratively improves both retrievers and rerankers. Our method enhances a dense retriever and a cross-encoder reranker in a closed learning loop, by alternating their roles as teachers and students. The empirical results on var- ious tasks demonstrate that this simple technique can significantly improve the capability of dense retrievers without relying on any human-annotated data, surpassing wide range of competitive spanse and supervised dense retrievers. We believe that ABEL is a generic framework that could be easily combined with other retrieval augmenting tech- niques, and benefits a range of downstream tasks.
Zero-shot Dense Retrieval Recent research has demonstrated that the performance of dense retriev- ers under oft-oi-domain settings be improved through using synthetic queries (Ma et al., 2021; Gangi Reddy al., 2022: Dai et al., 2023). Inte- grating distillation from cross-encoder rerankers further advances the current ofate-of-the-art mod- els (Wang et al., 2022). However, all of these methods rely on synthetic queries, which gener- ally implies expensive inference costs on the usage of large language models. Nonetheless, the quality of the synthetic queries is worrying, although it can be improved by further efforts, such as fine-tuning the language model on high-quality supervised data (Wei et al., 2022). In contrast, our work does not have such reliance, and thus offers higher train- ing efficiency. We show that effective large-scale training examples can be derived from raw texts in the form of cropping-sentence queries (Chen et al.. 2022), and their labels can be iteratively refined by the retriever and the reranker enhance the training of the other model.
Limitations
The approach proposed in this work incurs addi- tional training costs on the refinement of training labels and the iterative distillation process when compared to standard supervised dense retriever training. The entire training pipeline requires ap- proximately one week complete on a server with 8 A 100 GPUs. This configuration is relatively mod- est according to typical academic settings.
We focus on the standard dual-encoder paradigm, and have not explored other more advanced archi- tectures, such as ColBERT, which offer more ex- pressive representations. We are interested in in- vestigating whether incorporating these techniques would yield additional benefits to our approach. Furthermore, existing research (Ni al., 2022) has demonstrated that increasing the model size can enhance the performance and generalisability of dense retrievers. Our analysis also shows that
Iterated Learning Iterated Learning refers family of algorithms that iteratively use previously learned models to update training labels for subse- quent rounds of model training. These algorithms may also involve filtering out examples of low qual-

---

scaling up the model size improves reranking per- formance. Therefore, we would like to see whether applying the scaling effect to the retriever side can result in further improvement of the performance. Moreover, we mainly examined our method on the BEIR benchmark. Although BEIR covers var- ious tasks and domains, there is still a gap to in- dustrial scenarios regarding the diversity of the re- trieval corpora, such those involving web-scale documents. We plan explore scaling up the cor- pus size in our future work. Additionally, BEIR is a monolingual corpus in English, and we are inter- ested in validating the feasibility of our method in multi-linguul settings.
Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ingion, USA, Association for Computational Linguis- tics.
Xilun Chen, Kushal Lakhotia, Barlas Oguz. Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can dense retriever imitate sparse one? Findings of the Association for Computational Linguistics: EMNLP 2022, pages 250-262. Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Zhuyun Dai, Vincent Y Zhao, Ji Ma. Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming- Wei Chang. 2023. Promptagator: Few- shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representa- tions.
A Acknowledgements
We thank the anonymous reviewers for their help- ful feedback and suggestions. The first author is supported by the Graduate Research Scholar- ships funded by the University of Melbourne. This work was funded by the Australian Research Coun- cil, Discovery grant DP230102773 This research was undertaken using the LIEF HPCGPGPU Facil- ity hosted at the University of Melbourne, which was established with the assistance of LIEF Grant LE170100000.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2022. From dis- tillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 22, page 2353-2359, New York, NY, USA. Association for Computing Machinery.
References
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168-1773, Flo- rence, Italy. Association for Computational Linguis- tics.
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2021. Splade v2: Sparse lexical and expansion model for informa- tion retrieval.
Revanth Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, and Avirup Sil. 2022. Towards robust neural retrieval with source domain synthetic pre- rre-inetuuing. In P Proceedings of the 29th International Conference on Computational Linguistics, pages 1065-1070, Gycongju, Republic of Korea. International Committee on Computational Linguistics.
Thomas Anthony, Zheng Tian, and David Barber. 2017. Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc.
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannanch Ha- jishirzi, and Wen tau Yih. 2022. Task-aware retrieval with instructions.
Luyu Gao and Jamie Callan. 2021. Condenser: a pre- training architecture for dense retrieval. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981-993 Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara. Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. Ms marco: A human gener- ated machine reading comprehension datasct.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2843-2853,
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang- 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013

---

Dublin, Ireland. Association for Computational Lin- guistics.
Daniel Khashabi, Amos Ng. Tushar Khot, Ashish Sab- harwal, Hannanch Hajishirzi, and Chris Callison- Burch. 2021. GooAQ: Open question answering with diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421-433, Punta Cana, Dominican Republic. Associa- tion for Computational Linguisties.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA. Association for Computing Machinery.
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 7:452-466.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimension- ality reduction by learning an invariant mapping. In 2006 IEEE Compuler Society Conference Com- puter Vision and Pattern Recognition (CVPR'06), volume 2, pages
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering.
Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages
Florence, Italy. Association for Computational Linguistics.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. 2019. Code- searchnet challenge: Evaluating the state of semantic code search.
Sheng- Cring-Chichhuaa Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin. Yashar Mehdad, Wen tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave, 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Frederick Liu, Terry Huang. Shihang Lyu, Siamak Shak- eri, Hongkun Yu, and Jing Li. 2021. Enct5: A frame- work for fine-tuning 15 as non-autoregressive models.
Kalervo Järvelin and Jaana Kekäläänen. 2002. Cumu- lated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422
Xueqing Liu, Chi Wang, Yue Leng, and ChengXiang Zhai. 2018. Linkso: A dataset for learning to retrieve similar question answer pairs on software develop- ment forums. In Proceedings of the 4th ACM SIG- SOFT International Workshop on NLP for Software Engineering, NLASE 2018, page 2-5, New York, NY, USA. Association for Computing Machinery.
Jeff Johnson, Matthijs Douze, and Herve Jegou. 2021. Billion-scale similarity search with gpus. JEEE Transactions on Big Data, 7(3):535-5477
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1075-1088, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Antoine Miech, Josef Sivie, and Andrew Zisserman. 2021. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 9821-9831.
Alayrac. Ivan Laptev,

---

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi. Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. 2021. End-to-end training of neural retrievers for open-domain question answer- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Lan- guage Processing (Volume 1: Long Papers), pages 6648-6662, Online. Association for Computational Linguisties.
Niklas Muennighoff. 2022. Sgpt: Gpt sentence embed- dings for semantic search.
Jianmo Ni. Chen Qu, Jing Lu. Zhuyun Dai, Gustavo Hemnande Abrego, Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Woi Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Pruceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844 9855, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3715-3734, Seat- tle, United States. Association for Computational Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835-5847, On- line. Association for Computational Linguistics.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep. and Jimmy Lin. 2020. Document ranking with a pre- model. In Findings of the Association for Computational Linguistics: T.MNIP 2020, pages 708-718, Online. Association for Computational Linguistics.
trained sequence-to-sequenee
Linguistics.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6138-6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fitth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using raasformer-basee sequential denois- ing auto-encoderfor unsupervised sentence embed- ding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 671 671-688, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to retrieve pas- sages without supervision. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 2687 -700, Seat- tle. United States. Association for Computational Linguistics.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval, In Proceedings of the 2022 Conference of the North American Chopter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2345-2366 Seattle, United States. Association for Computational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr. 3(4):333-389
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language mod- els are zero-shot learners. International Confer- ence on Learning Representations.
Devendra Sachan, Mike Lewis. Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke

---

Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-trate Pre-training strieval-oriented lan- guage models via masked auto-encoder. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 538-548, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Lee Xiong, Chenyan Xiong, Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 Con- ference Empirical Methods in Natural Language Processing, pages 2013-2011 2018, Lisbon, Portugal. As- sociation for Computational Linguistics.
Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. COCO-DR: Combating the distribution shift zero-shot dense retrieval with con- trastive and distributionally robust learning. In Pro- ceedings of the 2022 Conference Empirical Meth- ods Natural Language Processing, pages 1462- 1479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. STar: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems.

---

A Experimental Settings
Method
Avg. nDCG@10
Initial Retriever (D))
40.2
A.1 Implementation Details
+ noise
For training queries, we divide the passages in the corpus of each task in BEIR into individual sen- tences, treating them as sropping-sentence queries. datasets with a large corpus size (e.g., Climate- FEVER), we randomly select 2 million sentences for training. However, for datasets with fewer pas- sages (-5k-50k), we use all of them for training. For each query, follow Chen et al. (2022) to extract its positive and negative passages from the top-k predictions of specific retriever, where the top-10 are considered as positives while passages ranked between 46 and 50 are regarded as negatives. For the reranking process in line 12 of Algorithm 1, we always rerank the top 100 retrievals returned by a specific retriever, from which refined labels are extracted using the above rules.
Initial Reranker
41.9
noise
43.7
soft labels
45.6
noise & soft lahels
47.1
Table 4: Study on the effects of injecting input noise and using soft labels model training. Average perfor- mance on the BEIR benchmark is reported.
with a hard negative 10% of the time.
A.2 Baseline Retrievers
We compare our method with a wide range unsupervised and supervised models. Unsuper- vised models include: (1) BM25 (Robertson and Zaragoza, 2009); (2) Contriever (Izacard et al., 2022) that is pre-trained unlabelled text with contrastive learning: (3) SimCSE (Gao al., 2021) that uses contrastive learning learn unsu- pervised sentence representations by taking the en- codings of a single sentence with different dropout masks as positive pairs; (4) REALM' (Guu et al., 2020) that trains unsupervised dense retrievers us- ing the masked language modelling signals from separate reader component. Supervised models include (1) ANCE (Xiong et al., 2021) trained MS-MARCO with self-mined dynamic hard neg- atives; (2) Contriever (MS) and COCO-DR (Yu et al., 2022) that are first pretrained on unlabelled corpus with contrastive learning and then fine- tuned on MS-MARCO; (3) RetroMAE (Xiao et al., 2022) uses masked auto-encoding for model pre- training and MS-MARCO for fine- tuning: (4) GTR- XXL (Ni et al., 2022), ColBERTv2 (Santhanam et al., 2022) and SPLADE++ (Formal al., 2022) that significantly larger model size, multi- vector and sparse representations, along with dis- tillation from gross-encoder on MS-MARCO; (5) DRAGON+ (Lin et al., 2023) that learns progres- sively from diverse supervisions provided by above models on MS-MARCO with both crop-sentence and synthetic queries; (6) QGen: GPL (Wang et al.. 2022) and PTR (Dai et al., 2023) that create cus- tomised models for each target task by using syn- thetic queries and pseudo relevance labels.
We use Contriever* initialise the dense re- triever and train for 3 epochs on 8 A100 GPUs, with a per-GPU batch size 128 and learning rate 3 x 10-5 10-5_ Each query is paired with one positive and one negative passage together with in-batch negatives for efficient training. A single retriever ABEL is trained on the union of queries from all tasks BEIR. We initialise the reranker from t5-base-Im-addap (Raffel al., 2020) check- point. 5 For each query, we sample one positive and 7 negative passages. Similarly, we train single reranker
using a batch size 64 and learning rate 3 x 10 for 20k steps, with roughly 1.2k steps on each task. We conduct the iterative alternating distillation for three iterations and take ABEL in iteration 2 and ABEL-Rerank in iteration 3 for evaluation and result comparison. We the maximum query and passage lengths to 128 and 256 for training and set both input lengths to 512 for evaluation.
We further fine-tune ABEL on MS-MARCO (Ba- jaj et al., 2016) using in-batch negatives for 10k steps, with batch size 1024 and learning rate 1x10 10 The maximum query and passage lengths for training are to 32 and 128, respectively. Fol- lowing Izacard al. (2022), we first train an initial model with each query paired with one gold pos- negative. We then
itive and randomly-sampe mine hard negatives with this model and retrain a second model ABEL-FT in the same manner but
"https://huggingface http://inaaaceecoorrinttteeoeeee co/princeton-nlp/ mnsup-sinnse--roberta
"https://hugginggfass http:///caacbbooooooortieiie
aae.inn.eeecccccooooo.ooo ca/ google/ SS-base-Lm-adapp
"nttps://huggingfee co/ google/ realm-cc-nen wwwe--ccnews--eetraiii -

---

Avg. Performance PTR-11
Method
top-K
TART-9
BEIR-13
All
Supervised by MS MARCO
BM25+CE
100
39.5
46.2
49.5
47.6
Contriever+CE
100
41.3
46.6
50.2
Mono'T5 (3B)
1000
44.6
51.1
7cro-shot Domain Adaption
UPR (3B)
1000
37.8
42.7
46.2
PTR (110Mx11)
200
43.9
49.9
TART (1.24B)
100
44.8
Unsupervised
SGPT (B.IB)
100
38.5
46.7
46.2
ABEL-Rerank (110M)
100
42.8
50.7
48.3
ABEL-Rerank (335M)
100
45.9
51.3
53.7
51.3
ABEL-Rerank (1.24B)
100
46.7
52.3
54.9
52.4
Table 5: The comparison of state-of-the-art rerankers with ABEL-Rerank on BEIR benchmark using nDCGG @ 10.
The averaged results of different experiments are reported. Methods that train dedicated models for corresponding datasets are noted with TART-9 excludes FEVER and HotpotQA from PTR-11.
Model
FQ
TC
CQ
RB
Avg.
C Comparison of Unsupervised Pre-training Methods
SimCSE
26.7
55.0
68.3
29.0
37.9
43.4
ICT
27.0
58.5
69.7
31.3
37.4
44.8
MLM
30.2
60.0
30.4
38.8
45,8
We also compare ABEL with range of unsuper- vised domain-adaptiin methods that employ pre- training on the target corpus, SimCSE, ICT (Lee et al., 2019), MLL (Devlin al., 2019), TS- DAE (Wang et al., 2021), Condenser (Gao and Callan, 2021) and COCO-DR. These methods fol- low a two-stage training paradigm, which first pre- trains a dense retriever on the target corpus (i.e., BEIR) and then fine-tunes the model using MS- MARCO training data. As shown in Table 6, our ABEL model, which solely uses unlabelled text cor- pus for unsupervised training, already outperforms out of models without requiring any supervised fine-tuning. Building on ABEL, ABEL-FT achieves the best results among all models that adopt the two-stage training paradigm. These results clearly demonstrate the superiority of our method com- pared to existing ones that rely on the target corpus for unsupervised pre-training.
TSDAE
29.3
31.8
39,4
47.9
Condenser
25.0
61.7
73.2
33.4
41.1
46.9
COCO-DR
30.7
70.9
80.7
37.0
44.3
52.7
ABEL
31.1
73.5
72.7
35.0
48.4
52.1
ABEL-FF
34.3
76.5
36.9
50.0
54.1
Table 6: The comparison of unsupervised pre- training methods representative BEIR tasks fol- lowing Wang et al. (2022). The best and second- best results are marked bold and underlined. The tasks are FQ=FiQA, CF=SciFact, TC=TREC-COWE CQ=CQADupStack, RB=Robust(44. Note that ABEL has not used MS- MS-MA'CO's 's training data for fine-tuning.
B Effects of Noise Injection and Soft Labels
We find that injecting noise into the inputs for model training leads to better performance. Specif- ically, we corrupt the query and passage texts by sequentially applying random shuffling, deletion, and masking on 10% of the words. Table 4 shows the results of injecting noise into the inputs during the training of the dense retriever and the reranker, along with the use of soft labels for reranker train- ing. We observe that noise injection results in posi- tive effects on both retriever and reranekr training. Furthermore, incorporating soft labels in reranker training leads to additional benefits.
D Reranking Performance
We show the reranking results of ABEL-Rerank on different subsets of BEIR, comparing them with various state-of-the-are rerankers. Table demon- strates that ABEL-Rerank achieves comparable superior performance than both supervised and un- supervised rerankers, using similar model sizes across all subsets. In addition, Table presents the detailed results ABEL-Rerank (1M) for differ- ent iterations. The reranker models trained with

---

ABEL-Rerank
FQ
AA
CF
CQ
QU
FE
NF
T2
H1Р
NQ
RB
TN
BA
Avg.
t=1
34.7
73.4
55.2
22.7
37.0
36.1
82.8
16.5
68.5
34.4
74.3
27.9
67.5
42.0
49.4
45.6
31.5
48.6
47.1
t=
35.2
73.1
55.8
25.2
37.5
36.8
82.9
17.1
76.1
34.8
77.3
29.5
66.6
43.3
50.2
47.1
31.0
47.9
48.2
t 3
35.0
72.9
53.4
26.7
37.7
36.6
82.6
17.1
80.8
34.2
76.4
32.1
67.2
42.7
49.7
45.4
29.3
49.0
48.3
Table 7: The comparison of bootstrapping on the reranking performance of individual tasks. The tasks are FQ=FiQA, SF=SciFact. AA=ArguAna, CF=Climate DB=DBPedia, CQ=CQADupStack, QU=Quora. SD=SCIDOCS, FE=FEVER, NF=NFC,rpus, TC=TREC-COVIDD, 72-oouché-2020, HP=HotpotQA, NQ=Natural Questions, RB=Robus(04, TN-TREC-NEWS, SG=Signal-IM BA=BioASQ.
Domain
Dataset
Query
Passage
Wikipedia
AmbigQA
Question
Question
Wikipedia
WikiQA
Question
Answer Sentence
Technical
Technicl
Question
StackOverflow Answer
Technical
munKOO-Python
Question
StackOverflow Question
Code
CodeScarchNer-Pyyiee Python
Comment
Python Code
Wikipedia
Natural Questions
Question
Answer Paragraph
Wikipedia
TriviaQA
Question
Answer Paragraph
Wikipedia
WebQuestions
Question
Answer Paragraph
Wikipedia
SQuAD
Question
Answer Paragraph
Wikipedia
EntityQuestions
Question
Answer Paragraph
Table 8: Domains and task formats of cross-task evaluation datasets.
AMB
WQA
GAT
LSO
CSP
Avg.
TriviaQA (Joshi et al., 2017), WebQuestions
(WebQ) (Berant et al., 2013). SQuAD (Rajpurkar et al., 2016) and EntityQuestions (EQ) (Sciavolino et al., 2021). Since these datasets were unseen when training ABEL, their data (i.e., the text corpus) will only be accessed during the testing phase. Con- sequently, we consider this way evaluate the capacity of ABEL to generalised to unseen tasks and domains.
BM25+CE
96.8
84.3
74.7
32.7
34,9
64.7
Contriever+CE
96.9
87.0
74.7
29.1
1
43.5
66.2
BM25+MonoTS
92.9
86.2
79.9
27.8
34.8
64.3
TART
91.1
82.1
80.5
25.1
51.4
66.0
ABEL-Rerank
79.8
77.7
31.4
45.9
66.4
Table 9: The cross-task reranking result comparison of ofate-of-the-art rerankers with ABEL-Rerank (nDCG@ 10). The best and second-best results are marked in bold and underlined. AMB=AmbigQA, WQA=WikiQA, LSO-LinkSO-Pytie Python, Sug==CooeSearchNeee-Pthhoe
GAT=-GoAAQ-Tcchiccal
E.1 Cross-Tast Reranking Results
We compare our ABEL-Rerank model with various supervised rerankers on five datasets that were not encountered during training. The results, as shown in Table 9, indicate that ABEL-Rerank achieves the highest performance and outperforms supervised rerankers of comparable sizes (i.e., MonoT5 and TART). This finding provides compelling evidence that the reranker component involved in our ap- proach demonstrates the ability to generalise to unfamiliar domains and tasks, rather than simply relying on memorising the corpus of each task in the BEIR benchmark to achieve promising results.
more than one iteration (i.e., f f= 2. improve the performance significantly. ABEL-Rerank (t = 2) performs the best on more tasks (10/18), while ABEL-Rerank (L achieves a marginally higher overall score.
E Cross-Task Evaluation
To validate the generalisation ability of ABEL, we conduct evaluation on a wide range of datasets (Table 8) without any further training, including AmbigQA (Min et al., 2020), WikiQA (Yang et al., 2015), GooAi--eccnical (Khashabi et al., 2021), LinkSO-Python (Liu al., 2018), CodeSearchNet- Python (Husain al., 2019), and five open- domain question answering datasets, namely Nat- ural Questions (NQ) (Kwiatkowski et al., 2019),
E.2 Cross-Dataset Results on Open- OpeenDomainn Question Answering Datasets
We further evaluate ABEL on five open-domain datasets that were not encoun- tered during training to test its cross-dataset gen- eralisation ability. We compare ABEL with a wide

---

NQ
TriviaQA
WebQ
SQuAD
EQ
Avg.
Models
Top-20
Top-100
Top-20
Top-100
Top-20
Top-100
Top-20
Top-100
Top-20
Top-100 100
Top-20
Top-100
BM25
63.0
78.2
76.4
83.1
62.3
75.5
71.1
81.8
71.4
80.0
68.8
79.7
SimCSE
43.6
58.5
62.1
74.3
44.8
57.7
47.6
63.3
38.5
53.9
47.3
61.5
ICT
50.6
66.8
57.5
73.6
43.4
65.7
45.1
65.2
MSSI
59.8
74.9
68.2
79.4
49.2
68.4
51.3
68,4
REALM
61.4
74.8
72.2
80.1
62.4
74.1
47.3
63.5
64.0
74.1
61.5
73.3
Spider'
68.3
81.2
75.8
83.5
65.9
79.7
61.0
76.0
66.3
77.4
67.5
79.6
Contriever
67.2
81.3
74.2
83.2
65.7
79.8
63.0
78.3
63.9
75.7
66.8
79.7
ABEL
74.7
85.0
80.5
85.7
73.6
83.4
74.7
84.8
72.1
80.3
75.1
83.8
Table 10: Zero-shot cross-dataset retrieval results of unsupervised models on open-domain question answering datasets. Top-20 & Top-100 retrieval accuracy the test set of each dataset is reported. The best and second-best results are marked in bold and underlined. Unavailable results are denoted with Results reported by Ram et al. (2022) are denoted with f-
FQ SF AA CF DB CQ QU SD FE NE
TC
T2 HP
NQ
RB
TN
SG
BA
QS
SR
Avg.
ABEL
31.1 73.5 50.5 25.3 37.5 35.0 83.9 17.5 74.1 33.8 72.7 29.5 61.9 42.0 48.4 48.0 30.8 41.3
44.2 48.7 46.5
ANCE
29.5 51.0 41.9 20.0 28.1 29.8 85.2 12.2 66.7 23,5 64.9 24.0 45.1 44.4 39.2 38.4 24.9
30.9
34.0
43.8
38.9
ABEL
34.1 74.1 51.3 27.4 40.5 37.3 87.2 17.9 79.8 33.9 76.5 28.6 64.7 49.5 50.6 49.3 30.6
46.1
47.2
50.6
48.9
Contriever
32.9 67.7 44.6 23.7 41.3 34.5 86.5 16.5 75.8 32.8 59.6 23.0 63.8 49,5 47.6 42.8 19.9 38.3
43.2
45.8 44.5
+ ABEL
31.7 73.8 50.6 25.6 38.8 35.4 84.5 17.6 75.0 34.0 73.2 29.2 62.9 43.1 48.8 48.1 30.9 42.2
44,9 48.9 46.9
RetroMAE
31.6 65.3 43.3 23,2 39.0 34.7 84.7 15.0 77.4 30.8 77.2 23.7 63.5 51.8 44.7 42.8 26.5
42,1
45.8
45.2 45.4
+ ABEL
34.5 71.4 48.5 26.8 43.8 36.3 86.7 17.5 82.0 34.2 80.5 28.8 69.1 5388 51.5 47.2 30.7 47.9
49.3
49.7 49.5
DRAGON+
35.6 67.9 46.9 22.7 41.7 35.4 87.5
78.1 33.9 75.9 26.3 66.2 53.7 47.9 44.4 30.1
43.3
47.2
47.6 47.4
+ ABEL
35.5 72.3 51.3 25.4 43.6 37.5
18.2 80.2 34.8 79.0 28.8 68.5 53.7 51,2 48.4 31.4
46.9
49.1 50.3 49.7
ABEL+
34.3 72.6 56.9 21.8 41.4 36.9 84.5 17.4 74.1 35.1 76.5 19.5 65.7 50.2 50.0
45.4
28.0
45.4
46.5
48.5 47.5
ABEL
33.8 73,9 54.5 24.7 41.8 37.2 85.3 18.0 77,2 34.9 77.4 24.0 66.3 49.5 50.1 48.1 29.9 45.9
47.1 49.9 48.5
Table 11: The comparison of various supervised dense retrievers combining ABEL on BEIR benchmark using aDCG@ @ 10. QS and SR means the average performance on query-search and semantic-relatedness tasks, respectively. The tasks are FQ=FiQA. SF=SciFact, AA=ArguAna, CF=Climate DeCCiimtt-FEEEER.. DB=DBPedia, CQ=CQADupStackk, QU=Quora, SD=SCIDOCS, FE=FEVER, NF=NFCor,us, TC=TREC-COVID, 2000 Ho-Hotpo NQ=Natural Questions, RB=Robust04, IN=TREC-NEWS SG=Signal-IM, BA=BioASQ.
range of unsupervised retrievers, including BM25, REALM, SimCSE, ICT, MSS (Sachan et al., 2021),
F Combination with Supervised Models
Table 11 shows the results in each individual task when combining ABEL with supervised dense re- trievers. The findings indicate that on both types of tasks (i.e., query-search and semantic-relatedness the ensembling with ABEL leads to performance im- provements on all supervised retrievers, with the gains being generally more significant on semantic- relatedness tasks. This demonstrates the comple- mentarity between ABEL and existing supervised dense retrievers, which are commonly trained us- ing labelled data in the form of natural questions.
Spider (Ram et al., 2022) and Contriever. We use
Top-k (k = 20, 100) as the main metrics for eval- uation according to Karpukhin et al. (2020). As shown in Table 10, ABEL significantly outperforms other unsupervised retrievers that have been so- phisticatedly pre-trained, with an average accuracy improvement of +8.4% for Top-20 and +4.0% for Top- 100 over Contriever. Note that ABEL exhibits promising results and surpasses BM25 on SQuAD, a dataset with high lexical overlaps between ques- tions and answer paragraphs, and EQ, a dataset consisting of entity-centric queries. This further confirms that ABEL effectively retains and enhances the lexical- ability acquired through learn ing from BM25 and this strength generalises well to unseen datasets.
G Effects of Boostrapping
Figure 6 shows the detailed results of each individ- ual task throughout the iterative alternating distilla- tion process. We observe that ABEL-1 outperforms ABEL-0 on almost all datasets, and the performance consistently improves from ABEL-1 to ABEL-2. For

---

fiqa
scifact
arguana
climate-fever
dbpedia-enne entity
cqadupstack
12A
35-26-7
2FA
257
57.
733
35.7
C
23.8
quora
scidocs
fever
nfcorpus
trec-covid
touché-2020
30.4
15.5
723
2EA
243
33-T
31.3
3233
25.20
23.2
hotpotqa
robust04
trec-news
signal1m
bioasq
60.2
51.6
45.5
67
46.0
20.3
$2.2
42.2
53.3
315
15.5
AVEL-2
AdR
D9229
(185)
ADRL
APRLS
BM22
(963)
(960
AlMLL
APLL2 Add
ABR
ABELO
Alle
CT BMS
Figure 6: Effects of bootstrapping the retrieval ability each task of BEIR. CT and MS indicate Contriever and
MS-MARCO, respectively.
ABEL-3, it achieves improvement on 9 datasets but the results degrade on 6 datasets, with the overall performance merely competitive to ABEL-2.
H The Effects of Synthetic Queries
Figure shows that synthetic queries demonstrate greater efficacy on tasks involving natural ques- tions (e.g., FiQA), while cropping-sentente queries are more effective semantic-relatedness. tasks with short sentence queries (e.g., Climate-FEVER, Furthermore, when considering query-search tasks whose domains are substantially distinct from MS- MARCO, where the query generator fine-tuned, employing synthetic queries Icads to significantly worse results, such as touché-2020 and BioASQ.

---

ABEL
synthetic-queries
fiqa
scifact
arguana
clirmat--fewe
dbpedia-entity
cqadupstack
535.00
48.4 48.
20.5
36.3
26.3
25.7
41.2
285
52.292.1
31 31.73177
quora
scidocs
fever
nncorpus
trec-covid
touché-2020
74.133.
29.5
02.150.1
35.2
33.3g333
33.8
583
25.8
22.4
GL
31.7
hotpotqa
robust04
trec-news
signallm
bioasq
61.9
413
42-2
42.2
52.7
39.2
$8.9
50.7
24.5
32.532
212
ABEL-D
ABEL-1
ABEL-2
ABEL-D
ABEL-1
ABEL-2
ABEL-0
ABEL-1
ABEL-2
ABEL-O
A
ABEL-2
ABEL-O
ABEL-1
ABEL-2
ABEL-D
ABEL-1
ABEL-2
Figure 7: The comparison of cropping- sentence queries and synthetic queries on each task.

---

Motivation
Practical
Cognitive
Intrinsic
Fairness
Generalisation type
Compositional
Structural
Cross Task
Cross Language Cross Domain
Robustness
Shift type
Covariate
Label
Full
Assumed
Shift source
Naturally occuring
Partitioned natural
Generated shift
Fully generated
Shift locus
Teain-test
Finetune train-test
Pretrain-train
Pretrain-test
0


Reviews:
[{"review_id": "xLvsUMG7KT", "reviewer": "Reviewer_jdQL", "summary": "", "questions": "1. Does the term \"zero-shot\" in the title refer to the dataset having no manual labels (unsupervised), the dataset being constructed without reliance on synthetic queries, or the ability to perform cross-domain inference?\n\n2. Have the authors considered any methods to detect and avoid error accumulation in the loop of model training?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions!\n\n---\n\n> W1: Differences among ABEL, ABEL+ and ABEL++\n\nA: (1) ABEL+ is the model obtained by further fine-tuning ABEL on the MS-MARCO dataset and (2) ABEL++ adds a reranker to rerank the top-k predictions from ABEL, as described in Section 4.2. We will rename ABEL+ and ABEL++ to ABLE+FT (fine-tuning) and ABEL+FT+reranker respectively to highlight their difference and modify our description accordingly.\n\n---\n\n> W2 (a): “The baselines seems to be weak. Only BM25 and Contriever are involved in the unsupervised setting.”\n\nA: (1) We include BM25 [1] and Contriever [2] as baselines in Table 1 to represent competitive lexical-based and neural-based methods, respectively. For some retrieval tasks, these baselines have been proven very competitive, and BM25 can outperform the best supervised neural models [3].  (2) We have included four unsupervised approaches, i.e., BM25, REALM, SimCSE, and Contriever, as compared in Table 2. The two selected baselines used in Table 1 have the highest overall performance. (3) (Unsupervised) ABEL outperforms all these baselines as demonstrated in Tables 1 and 2. We will clarify our selection process for the unsupervised baselines in our revision.\n\n> W2 (b): Marginal Improvements over SPALDE++ and DRAGON+\n\n[Compared with the best lexical model (SPLADE++)]: Although lexical models generally outperform dense models as they map learned vectors to actual lexicons, our dense model (ABEL+) (1) is more efficient in scaling to massive documents by using GPUs; (2) manage to outperform the best lexical-based approach.\n\n[Compared with DRAGON+, a dense model learning from SPALDE++ and other models] Although DRAGON+ enjoys two advantages: (1) inheriting knowledge from a lexical-based model and (2) using a more advanced pre-trained model (RetroMAE) as backbone than ours (Contriever), our method can still outperform this baseline (DRAGON+). To have a fair comparison, we add experimental results of both methods using aligned pre-trained models as follows:\n\n | Initialisation | BERT | Contriever | RetroMAE |\n| :--: | :----: | :------: | :----: |\n| DRAGON+ | 46.8 | N/A | 47.4 |\n| ABEL+ | 47.2 | 47.5 | 48.0 |\n|Improv. | +0.4 | N/A | +0.6 |\n\nThe improvement by our approach is consistent, higher than 0.4%. We choose Contriever as our base model to align with the settings of previous papers, i.e., Contriever[2] and PTR[4]. We will add corresponding discussions in our modification.\n\n---\n\n> Q1: Definition of zero-shot setting. \n\nWhen training a retriever, the ground-truth annotations for each task are unknown - i.e., all we have are the passages, but we have no queries, nor relevance judgments.  Thus our setting is zero-shot in terms of the target task, and we assess the ability to transfer across domains. As shown in Table 2, ABEL can achieve consistent performance gains over Contriever and surpass other unsupervised baselines on unseen tasks without further training.\n\n---\n\n> Q2: Error Accumulation\n\nWe use re-initialisation to avoid the accumulation from biased errors in early iterations. At the start of each iteration, we re-initialise both the retriever and reranker using pre-trained language models rather than continuously fine-tuning the model obtained from the last iteration (more details in lines 236-241). We will highlight the setting in our revision. \n\n---\n\n[1] Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr.\n\n[2] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research. \n\n[3] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. \n\n[4] Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framework proposed in this paper is relatively simple and practical.\n\n2. The experiment is extensive.", "reasons_to_reject": "1. The overall structure of the article is clear, but some details are confusing. For example, the definition of zero-Shot dense retrieval, their differences and connections among ABEL, ABEL+, and ABEL++.\n\n2. The baselines seems to be weak. Only BM25 and Contriever are involved in the unsupervised setting. And under the supervised setting, the improvement of ABEL+ over SPLADE++ and DRAGON+ seems marginal.", "questions_to_authors": "1. Does the term \"zero-shot\" in the title refer to the dataset having no manual labels (unsupervised), the dataset being constructed without reliance on synthetic queries, or the ability to perform cross-domain inference?\n\n2. Have the authors considered any methods to detect and avoid error accumulation in the loop of model training?", "paper_topic_and_main_contributions": "This paper proposes ABEL, an Alternating Bootstrapping training framework for unsupervised dense rEtrievaL. The method alternates the distillation process between a dense retriever and a reranker by switching their roles as teachers and students in iterations. Contriever is used to initialize the dense retriever, and the reranker is initialized from t5-base-lm-adapt. The method starts with warming up the retriever by imitating BM25, then (1) training a reranker based on the labels extracted from the retriever by the last iteration; (2) refining the dense retriever using training signals derived from the reranker by the last step. Experimental results show that ABEL achieves superior performance in the unsupervised setting when compared with BM25 and Contriever.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The definition of $P$ and $P^k_D(q)$ in Line 132 to 134 is ambiguous. It seems that $P^k_D(q)$ just extracts the $k$ elements with the smallest indices from $P$.\n\n2. $Z^+$ in Line 210 and the soft labels in Line 226 are without definition. Main paper should be self-contained.\n\n3. The formula is only labeled up to 4, and the rest are not numbered.\n\n4. Line 304: Figure 5(a); Line 310: Fig. 2(b)."}}, {"review_id": "cN5DlUEZvq", "reviewer": "Reviewer_3ATb", "summary": "", "questions": "1. Have you tried using the same training data as DRAGON[1] (i.e., MS MARCO cropped sentences)? In this way, we can see a fair comparison (in terms of data) between the proposed approaches of ABEL and DRAGON.  Also, this can make reader understand how much gain from data augmentation on diverse corpus in BEIR rather than single corpus in MS MARCO. \n2. The authors have found that the sentence-cropped trained models are weaker in query-search tasks; then, why the authors choose not to include synthetic queries in the training process? Does it harms the overall performance?\n\n[1] Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023a. How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval. ArXiv:2302.07452\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your summarization of our contributions and suggestions.\n\n---\n\n> W1: The Difference from Previous Methods\n\nThanks for acknowledging our contributions on the unsupervised iterative alternating distillation methods. We would like to clarify our novelty to [1], [2], and [3]: \n1. Chen et al [1] mainly focus on how to best distill the knowledge from sparse retrievers, while our work considers crop sentences as a primary source of obtaining unsupervised training signals cheaply.\n2. Yu et al [2] primarily take the diverse corpus from BEIR for model pre-training to improve the domain-adaption ability, while our work shows that the model can outperform a wide range of supervised ones without using supervised data.\n3. Ren et al [3] jointly train a retriever and a reranker through knowledge distillation and self-mined pseudo positives. However, their work doesn’t explore the alternating distillation mechanism, which is introduced in Section 3.2 and verified in Figure 4. The significance of our work lies in that we are the first to show the effectiveness of (bi-directional) alternating distillation for unsupervised IR, while [3] only discusses one-directional knowledge flow (using reranker to improve retriever), without the reversed effect (using retriever to improve reranker).\n\n---\n\n> Q1: “Have you tried using the same training data as DRAGON[1] (i.e., MS MARCO cropped sentences)?”\n\nIn our preliminary experiments, we have tried using only cropped sentences from MS MARCO as the training data. This method on BEIR is significantly better than the vanilla Contriever (Avg. 41.0 vs Avg. 35.9) but is inferior to the one using diverse corpus from BEIR, and we believe that the diverse corpora we used in training is one of the factors to our success. \n\n---\n\n> Q2: “Why the authors choose not to include synthetic queries in the training process?”\n\nThere are two main reasons for not including synthetic queries:\n1. It incurs additional cost, especially when using large language models to generate high-quality synthetic queries. \n2. In our experiments, we did not observe the advantage of using synthetic queries, compared with those using crop sentences in both query-search and semantic-relatedness tasks, as shown in Figure 5(a). Please also refer to the corresponding comparison in Figure 7 in Appendix H.\n\n---\n\n[1] Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In Findings of the Association for Computational Linguistics: EMNLP 2022\n\n[2] Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. COCO-DR: Combating the distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \n\n[3] Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces an approach to train retriever and reranker in a completely unsupervised manner and reach best ranking effectiveness among all unsupervised models. \n2. The experiments are comprehensive, including the comparisons on cross tasks with unsupervised models, the comparisons of different training strategies. The experiments clearly illustrates the importances of introducing cross-encoder in the alternating distillation and cropped sentences as training queries.", "reasons_to_reject": "1. The proposed approach is basically the combination of the existing work. I would consider the main contribution of the paper three folds: cheap and large-scale data augmentation, diverse domain, alternating distillation with retriever and reranker. However, these two approaches are not new. For example, this sentence cropped approach has introduced by Chen et al[1]; Using training data from divers domain is introduced by Ye et al[2]; alternating distillation is also introduced by the previous work[3].  \n\n\n[1] Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen tau Yih. 2021. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In Proc. EMNLP.\n[2] Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. Coco-dr: Combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proc. EMNLP.\n[3] Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proc. EMNLP, pages 2825–2835.\n", "questions_to_authors": "1. Have you tried using the same training data as DRAGON[1] (i.e., MS MARCO cropped sentences)? In this way, we can see a fair comparison (in terms of data) between the proposed approaches of ABEL and DRAGON.  Also, this can make reader understand how much gain from data augmentation on diverse corpus in BEIR rather than single corpus in MS MARCO. \n2. The authors have found that the sentence-cropped trained models are weaker in query-search tasks; then, why the authors choose not to include synthetic queries in the training process? Does it harms the overall performance?\n\n[1] Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023a. How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval. ArXiv:2302.07452\n\n", "paper_topic_and_main_contributions": "The paper proposes to a training strategy to iteratively train retriever and reranker. The authors demonstrate that starting from unsupervised labels using BM25, both the retriever and reranker can reach competitive even better effectiveness than many supervisedly trained rankers. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YdkXoiv8z9", "reviewer": "Reviewer_ACNC", "summary": "", "questions": "1.\tIn general, bootstrapping requires some variation in the source or presentation of information, and then improved competence through self-learning. In this paper, how is the superiority of labeled data ensured during the iterative process? In addition, what are the specific trends of change between what the retriever and reranker learn and what can be instructed to each other in the process of learning data?\n2.\t“their labels can be iteratively refined by the retriever and the reranker to enhance the training of the other model.” The approach of alternating improved labels creates dependencies between the retriever and the reranker, and how the authors consider the magnitude of the ability to process the data between the two to prevent the emergence of some pernicious dependencies (singly optimized labels) as well as the disappearance of benign dependencies (co-optimized labels).\n3.\tThe authors observe that using soft labels captures subtle semantic relationships between multiple related passages and produces better results than using hard labels to train the reranker. Does the above arise from inaccuracies in hard labels information? If so, which information produces better results in the Zero-shot condition when the semantic information obtained by hard labels is closer to the real sample attributes compared to the relationship information obtained by soft labels? If not, can we understand that inter-paragraph relational information is more important than attribute information in any case?\n4.\tPlease ensure that there are corresponding notations at the indexes in the paper, e.g. line 249 lacks a notation for equation (5).\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions.\n\n---\n\n> W1: Novelty and comparison to previous work\n\nWe would like to highlight the novel aspects of our work to previous works:\n + DML [1] jointly trains multiple networks with identical model capacity through peer distillation. Our method uses models with different roles (i.e., reranker and retriever) and enforces mutual improvement between the participants. We believe our design and corresponding research are novel regarding adapting knowledge distillation to the complex setting in IR research.  \n + RocketQAv2 [2] jointly trains the retriever and the ranker using knowledge distillation, but with the effect of a single direction, namely using the reranker to improve the retriever. By contrast, we suggest two insights (1) a model with a larger capacity can be improved by a less capable model and (2) we can iteratively realise mutual improvement between two models.\n\nWe believe our proposed iterative alternating distillation algorithm is not trivial as it shows that, for the first time, two models with different modeling abilities can mutually benefit in an entirely unsupervised manner, and this can result in models that surpass sophisticated supervised retrievers. Our research is beneficial to search engines in a “cold-start” setting when no labeled data is available. The performance of ABEL can be further improved by training on accumulated user data from search logs.\n\n---\n\n> W2: “Internal reasons for the Boot and Switch between retriever and reranker”\n\nThere are two main reasons that inspire our design:\n1.  Better rerankers potentially correct the errors made by retrievers and consequently improve label qualities. By introducing an additional more capable model as a reranker, it is able to reorder the predictions from the retriever by reranking passages that are closer to gold ones higher (more likely to be regarded as positives) and sample more contrastive negatives (relevant to but not answering the query) from its predictions. \n2. An improved retriever is able to generate harder negative samples with more accurate soft labels to reflect the nuanced relatedness between passages and queries.\n\nAn alternative distillation method is always to learn from the data generated by itself. We have verified that this paradigm underperforms our approach (see the comparison in Figure 5(a), the purple line vs the blue line), with the performance saturating after one iteration. Moreover, the retriever learned from itself quickly converges to fixed labeling behavior.  We believe this results from the shortage of error corrections, which is the role the reranker played in our method, and the diverse samples and accurate labelling introduced by our alternating learning are essential\n\n---\n\n> Q1: “How is the superiority of labeled data ensured during the iterative process? In addition, what are the specific trends of change between what the retriever and reranker learn and what can be instructed to each other in the process of learning data?”\n\nThe key intuition is that (1) a better retriever provides better candidate sets (more semantically related to queries) to train a better reranker and (2) a better reranker provides more accurate positives and more contrastive negatives to train a better retriever. The two models bring very different “views” (in the sense of co-training & multi-view learning) to the problem: one imposes a factorisation with a tight bottleneck, while the other supports full cross-attention. These different architectures affect the tractability of search, but also the capacity to model the full problem, and the propensity to overfit. \n\nEmpirically, we found that the performance of both retriever and reranker keeps improving during the training iterations. This indicates a promising direction that the generated labels by both models are converging to the distributions with higher quality. \n\n---\n\n> Q2: “How the authors consider the magnitude of the ability to process the data between the two to prevent the emergence of some pernicious dependencies (singly optimized labels) as well as the disappearance of benign dependencies (co-optimized labels)?”\n\nWe can interpret the retriever as a proposal distribution generator, which selects relatively good candidates, and the reranker as a reward scoring model, which measures the quality of an example as a good answer. They can work cooperatively to generate labels of higher quality.\n\nAt each iteration, the improved reranker can be used to correct the predictions from the retriever, therefore reducing the number of inaccurate labels in retriever training data. Meanwhile, the improved retriever is more likely to include more relevant passages in its top-k predictions, which increases the chance of finding correct labels by the reranker. Overall,  as training goes on, (1) the range where correct answers appear can be narrowed down by the retriever (i.e. better proposal distributions) and (2) the accuracy of finding correct answers from such ranges can be increased by the reranker. \n\n---\n\n> Q3: “Does the inaccuracies in hard label information leads to inferior result of reranker compared to soft labels? If so, which information produces better results in the Zero-shot condition when the semantic information obtained by hard labels is closer to the real sample attributes compared to the relationship information obtained by soft labels? If not, can we understand that inter-paragraph relational information is more important than attribute information in any case?”\n\nWe show that using hard labels for reranker training is inferior to soft labels in Table 3 in Appendix B. Additionally, we found that the reranker trained on hard labels tends to converge to trivial solutions (i.e., predicting merely the passage where the crop-sentence query appears). Soft labels provide the reranker with more diverse training signals by learning from a distribution. Additionally, soft label is a quite natural choice, as soft labels were shown more effective than hard labels for distillation in [1]. \n\n---\n\n> Q4:  Presentation \n\nWe will correct the presentation issues in our modification.\n\n---\n\n[1] Mary Phuong and Christoph Lampert. 2019. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is written clearly and is well-organized. Technical details are sufficient so that I am confident the results would be reproducible.\n2.\tLots of experiments to support the methodology. The authors perform overall good discussions on the components of their work, showing the boost from each part. A lot of visualized results are illustrated, e.g., including the Zero-shot retrieval results on BEIR (nDCG@10), Reranking results on BEIR, the effect of bootstrapping the retrieval ability on BEIR, and so on.\n", "reasons_to_reject": "1.\tThe novelty of this paper is not sufficient. We found that it is not uncommon for collaborative work between teachers and students, such as those represented by DML[1], RocketQAv2[2], etc. In addition, the impact of noise on model performance in this paper is also similar to the data augmentation in the above existing work.\n[1]\tZhang Y, Xiang T, Hospedales T M, et al. Deep Mutual Learning [M]. CVPR. 2018: 4320-4328.\n[2]\tRocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. EMNLP (1) 2021: 2825-2835\n2.\tContributions are not well argued. The core contribution of the paper lies in the Boot and Switch between retriever and reranker. However, the paper introduces the working mechanism of the two, but rarely discusses the internal reasons for the Boot and Switch between retriever and reranker.\n", "questions_to_authors": "1.\tIn general, bootstrapping requires some variation in the source or presentation of information, and then improved competence through self-learning. In this paper, how is the superiority of labeled data ensured during the iterative process? In addition, what are the specific trends of change between what the retriever and reranker learn and what can be instructed to each other in the process of learning data?\n2.\t“their labels can be iteratively refined by the retriever and the reranker to enhance the training of the other model.” The approach of alternating improved labels creates dependencies between the retriever and the reranker, and how the authors consider the magnitude of the ability to process the data between the two to prevent the emergence of some pernicious dependencies (singly optimized labels) as well as the disappearance of benign dependencies (co-optimized labels).\n3.\tThe authors observe that using soft labels captures subtle semantic relationships between multiple related passages and produces better results than using hard labels to train the reranker. Does the above arise from inaccuracies in hard labels information? If so, which information produces better results in the Zero-shot condition when the semantic information obtained by hard labels is closer to the real sample attributes compared to the relationship information obtained by soft labels? If not, can we understand that inter-paragraph relational information is more important than attribute information in any case?\n4.\tPlease ensure that there are corresponding notations at the indexes in the paper, e.g. line 249 lacks a notation for equation (5).\n", "paper_topic_and_main_contributions": "The authors propose ABEL, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. Among them, the core contribution lies in the boot and switch of retriever and reranker in the iterative process. Specifically, the paper follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever. By iterating this loop, the two components mutually enhance one another’s performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes ABEL, a training strategy to iteratively train retriever and reranker for unsupervised dense retrieval in the zero-shot setting. The reviewers' opinions on this work are not unanimously positive. While the authors have addressed certain concerns raised by the reviewers, there are still some lingering apprehensions. The authors have provided appropriate responses to jdQL's concerns regarding method details and comparison with baselines, which resulted in an increased score from R1. Both 3ATb and ACNC expressed that the novelty and contributions of the proposed method were limited. The authors have provided detailed responses highlighting the differences between ABEL and previous methods, while also clarifying the contributions of ABEL. However, these concerns still persist.
Decision: Accept-Findings