Relevance-aware Diverse Query Generation for Out-of-domain Text Ranking (2024)

Abstract: Domain adaptation presents significant challenges for out-of-domain text ranking, especially when supervised data is limited. In this paper, we present ReadQG1 (Relevance-Aware Diverse Query Generation), a method to generate informative synthetic queries to facilitate the adaptation process of text ranking models. Unlike previous approaches focusing solely on relevant query generation, our ReadQG generates diverse queries with continuous relevance scores. Specifically, we propose leveraging soft-prompt tuning and diverse generation objectives to control query generation according to the given relevance. Our experiments show that integrating negative queries into the learning process enhances the effectiveness of text ranking models in out-of-domain information retrieval (IR) benchmarks. Furthermore, we measure the quality of query generation, highlighting the underlying beneficial characteristics of negative queries. Our empirical results and analysis also shed light on potential directions for more advanced data augmentation in IR. The data and code have been released.

heading: 1 Introduction
text: Many domain-specific tasks lack supervised data, posing challenges for many neuarl approaches. Recently, Thakur et al. (2021) introduce an out-ofdomain (OOD) information retrieval (IR) benchmark across diverse scenarios and domains. Their findings indicate that many neural text ranking models demonstrate limited effectiveness in such contexts. These tasks primarily struggle with adaptation (Gururangan et al., 2020), highlighting the issues of insufficient task- and domain-specific labeled data.
To address this, one line of research propose utilizing synthetic training data for adapting text
*Work partially done as an research assistnat at Academia Sinica.
1https://github.com/DylanJoo/readqg
ranking models (Ma et al., 2021; Bonifacio et al., 2022; Wang et al., 2022). These approachees employ generative models to first learn document-toquery mapping from rich-resource datasets such as MSMARCO (Bajaj et al., 2018). Subsequently, a document and its generated query can be treated as a relevant pair for fine-tuning text ranking models. Recently, these methods have been further refined with instruction-tuned large langugage models (LLMs) (Brown et al., 2020a; Chung et al., 2022). Such LLM-driven query generation can produce more informative query through incontext (Jeronymo et al., 2023) or few-shot learning (Dai et al., 2023).
Compared to these works, in this study, we introduce Relevance-aware Diverse Query Generation (ReadQG), aiming to generate more informaive synthetic queries with lightweight generative models. Specifically, we generate both positive and (hard) negative queries from the same document, as illustrated in Figure 1. Our hypothesis is that a set of diverse relevance-aware queries can enhance relevancy representation of texts in unseen domains. To achieve this, we develop the instruction prompt and relevance prompt embeddings. The instruction prompt directs LLMs to generate query, while rel-
26
evance prompt captures and conrols the relevance dynamic between docuemnt and multiple queries. Moreover, we develop two strategies to diversify our generated queries: self-contrastive loss and sequence calibration loss (Zhao et al., 2023). These strategies prevent ReadQG from degeneration (i.e., falling back to naive relevant query generation).
Finally, to exploit positive and negative queries genereted by ReadQG, we integrate the querybased objectives into the training process of text ranking models. Our experiments demonstrates that models fine-tuned on our synthetic data outperform the original model in terms of passage re-ranking efftiveness on the BEIR benchmark. In addition, we define and propose two metrics to measure the quality of generated queries. We observe that the query exhibiting both diversity and relevancy provide useful signals for passage re-ranking models to learn, emphasizing the importance of hard negative query.
To sum up, we propose a domain adaptation pipeline with ReadQG, tailored for out-of-domain text ranking. Our empirical results show that hard negative queries could provide useful signals. Further, the domain adaptation pipeline is built with lightweight generators and text ranking models, achieving improved effectiveness but more efficient in terms of inference time and computational costs. More details can be found in our results (Section 6.1) and our analysis (Section 6.2).

heading: 2 Backgrounds
text: Data augmentation in IR. Numerous IR studies have pioneered in the area of data augmentation for domain adaptation. For instance, QGen (Ma et al., 2021) used synthetic query with documents to facilitate the adaptation of bi-encoder as domainadaptive dense retrieval. This can also be combined with negative mining techniques (Xiong et al., 2020), leading to enhanced effectiveness (Wang et al., 2022). Recent data augmentation techniques in IR have further been improved by the advancements in instruction-tuned large language models (LLMs) (Brown et al., 2020b; Chung et al., 2022). InPars (Bonifacio et al., 2022) showed that specific in-context prompting can enhance the quality of generated queries. Moreover, Promptagator (Dai et al., 2023) introduces few-shot in-context learning to bridge the gap between in-domain and out-ofdomain data. Typically, all these methods center aorund augmenting synthetic queries derived from
unseen document and utilizing them as additional training data.
Query Generation. Since the documents in outof-domain corpus are usually available, we in this work focus on the query generation instead of document generation (Gao et al., 2022). Particularly, Nogueira et al. (2019) first explored the role of query generation in IR. Oguz et al. (2022) also showcased that increasing the number of synthetic queries enhances domain adaptation capability, while Lin et al. (2023) further validated diverse queries can bridge the gap between zero-shot and supervised setups. Question generation can also plays a cruicial role in improving robustness of question answering (QA) systems (Bartolo et al., 2021; Lee et al., 2020) and has broader impacts in various NLP applications such as summarization (Lyu et al., 2021) or building retrieval-intensive QA datasets (Min et al., 2020).
Diverse and controllable text generation. We further extend the concept of query generation to controllable text generation in NLP area. Similar to our goal, Cho et al. (2019) propose capturing the one-to-many relationship between texts, such as dcument-to-summaries. However, due to the discrete nature of text generation, controlling sequence diversity is challenging and often required specialized learning settings (Bowman et al., 2016) or model adjustments (See et al., 2017). Text decoding strategies also significantly influence the results (Holtzman et al., 2020). Many ongoing research focus on designing constraints and objectives, such as unlikelihood (Welleck et al., 2020) or additional constrative-like learning signals (Liu et al., 2022; Zhao et al., 2023). We hypothesize recent LLMs could transform the notion of continuous relevance and present them with divese queries, thereby improve the domain adaptation of out-of-domain text ranking.

heading: 3 Methodologies
text: In this work, we utilize synthesized out-of-domain training queries to tackle the domain-mismatch issues. We will first provide an overview of our domain adaptation pipeline in Section 3.1; it is also illustrated in Figure 2. Following this are the details of the two main stages in ths pipeline, including outof-domain data augmentation and domain-adaptive fine-tuning.

heading: 3.1 Overview
text: As illustrated in the first block of Figure 2, we introduce a novel data augmentation approach, Relevance-aware diverse Query Generation (ReadQG). We generate multiple synthetic queries for each documents in the targeted unseen2 domains. These queries enables us to construct high-quality docuemnt-centric training pairs (See example in the last block in Fig. 2). By leveraging these training data, we can efficienctly transform general text re-ranking models into domain-adaptive ones as shown in second block of the figure.

heading: 3.2 ReadQG: Relevance-aware Diverse Query Generation
text: Unlike common studies with relevant query generation, we generate a set of relevance-aware queries Q = [q(r1), q(r2), ..., q(rn)] for each unseen document d ∈ D as illustrated in the Figure 2. Specifically, we aim to generate the positive (relevant) query and the hard negative queries.3 To achieve this, we develop a controllable query generator with diversity text generation objectives.
2Here, we regard MSMARCO as the source domain; thus, the retrieval tasks in BEIR are considered as unseen domains.
3The term hard negative query refers to less-relevant queries, as oppose to the random (negative) query.

heading: 3.2.1 Controllable Query Generation
text: Here we propose to parameterize such documentto-queries generation process via soft prompttuning (Lester et al., 2021). Our focus is specifically on learning such process of relevance dynamic from a rich-resource domain (i.e., MSMARCO) with relatively smaller models instead of directly prompting LLMs. Thus, we leave the attempts of in-context prompting with larger causal LLMs as our future works. Here, we propose to parameterize such a document-to-queries generation process via soft prompt-tuning (Lester et al., 2021). Our focus is specifically on learning the process of relevance dynamic from a rich-resource domain (i.e., MSMARCO) with relatively smaller models instead of directly prompting LLMs. Thus, we leave the attempts of in-context prompting with larger causal LLMs as our future work.
Soft prompt-tuning with relevance. As depicted in Figure 3, we employ soft embedding prompts to control the relevance-aware query generation process. These prompts act as the composite input for documents and relevance scores. Simply put, the generator G is expected to generate a query conditioned on the given docuemnt d and also the specified relevance r. To achieve, we include two learnable embedding prompts: an instruction prompt Pinst and the relevance prompt Prel(r). Thus, we can formulate the relevanceaware query generation as:
q̂(r) ← G ( Pinst; Prel(r); d ) ; ∀r ∈ [0, 1], (1)
where the relevance r is a re-scaled continuous variable score ranging from 0 to 1 (See Section 4 for more details). It is worth noting that we only consider the prompts as trainable parameters, encouraging to leverage the inherent capabiltiy of instruction-tuned language models.
Prompt initialization. To inherit the merits of language model pretraining, we initialize Pinst with natural-language instructions.4 While the relevance prompt Prel(r) is a function of a continuous relevance score r:
  r 1− r ...
... r 1− r
 ×
[ P+rel ← E(true ...) P−rel ← E(false ...) ] .
This is basically a linear combination of P+rel and P−rel with respect to relevance. In addition, we initiliaize them with embeddings of “true” and “false” tokens before fine-tuning.
Encoder-decoder architecture. As our composite input and expected output are highly correlated syntactically, we choose the encoder-decoder architecture, Flan-T5 (Chung et al., 2022) as our backbone model. Formally, the inner flow of hidden states during training is as follow
H(r)enc =Genc ( Pinst; Prel(r); d ) ; H (r) dec =Gdec ( qt|q<t;H(r)enc),
where Henc and Hdec indicate the hidden states of encoder outputs and decoder outputs respectively. To optimize the parameterized embedding prompts P, we adopt the standard training recipe of teacher-forcing and maximum likelihood estimation (MLE):
Lmle = − logP ( (H (r) dec) TW ) , (2)
where W is the projection layer of LM head.

heading: 3.2.2 Learning to Diverse Generation
text: As negative relevance of document-to-queries is intricate, we impose two objectives to encourage sequence generation diversity.
In-batch self-contrastive loss. By treating the generator’s encoder Genc as an independant (document) encoder, we can leverage the similar softmax objectives with mini-batch like DPR (Karpukhin et al., 2020). We define the hidden states of encoder output H(r)enc itself as positive and the others in mini-batch as random negatives. Thus, the selfcontrastive loss of document di with relevance rj
4Among a few preliminary zero-shot tests, we cherrypicked a better one: “Generate a question for this passage with the labels:” as initialization.
is as follow
Lsc = exp
( ϕ(H (rj),i enc , H (rj),i enc )/τ )
∑ i′∈B,j′∈2m exp ( ϕ(H (rj),i enc , H (rj′ ),i′ enc )/τ ) ,
(3)
where ϕ(x, y) represents the cosine similarity scores between x and y after average pooling, and τ is the temperature. 2m refers to the indices of collected relevance-query samples: {(rj , q(rj))}2mj for each document, consisting of m positive queries and m negative ones.5 Intuitively, the semantic distance between arbitrary relevance-aware document representations Henc would propogate gradient to the relevant prompts. Therefore, this loss will guide encoder Genc to comprehend differently across different documents and relevance simultaneously.
Calibrated sequence likelihood. Since negative queries could be infinite, the models will tend to generate random trivial queries or non-scene texts (Welleck et al., 2020; Holtzman et al., 2020). Thus, we specifically control the sequence likelihoods of positive and negative query generation to avoid such degeneration. Inspired by sequence calibration (Zhao et al., 2023), which leverages multiple references to calibrate the sequence likelihood, we treat the relevance-contradicted query as a reference to calibrate the likelihood of the relevance-entailed query, as illustrated in Figure 3.
Specifically, for each composite input of document and relevance, we regard the likelihood of relevance-entailed query generation as logPfw = logP (q(r)|d, r), indicating the “forward” generation. On the contrary, we calculate the “reverse” likelihood by substituting the decoder input with the contradicted one, denoted as logPrev = logP (q(r
′)|d, r). This implies the likelihood of generating relevance-contradicted queries. Both the adjustments can be done efficiently within the batch; we simply swap the decoder inputs between the forward one and the reverse one as demonstrated in Figure 3. The calibration loss for each relevance-aware query generation is as follows:
5As we fix the number of sampled queries per document d, we here ignore the docuemnt dependency and replace the notations of rji, mi by rj , m for brevity.
Lcal = ∑
(r,r′)
max ( 0, ϵ− logPfw + logPrev ) ;
(q̂(r), q̂(r ′)) ∈ ( [R+d ;R − d ], [R − d ;R + d ] ) ,
(4)
where ϵ is a fixed margin that provides tolerances when forward-reverse gap is large enough. R+d and R−d are the available postive and negative query samples and their corresponding relevance scores (Section 4). In particular, the intuition behind this loss is to increase the discrepancy between positive and negative query generation along with the given relevance distribution.

heading: 3.3 Domain-adaptative Passage Re-ranking
text: Afterward, as depicted in the second block in Figure 2, we can generate diverse relevance-aware queries q̂(r) via ReadQG by feeding the document with different relevance scores. We then use these queries to construct special synthetic training pairs, each comprising a document d, a positive query q̂, and a negative one q̂−.6 These examples, especially the query-query pair, serve as additional domain-adaptive learning signals for downstream text ranking models.
Cross-encoder for relevance classification. We choose cross-encoder architecture and passage reranking task as the experimental testbed. And we use binary cross-entropy (BCE) loss for training cross-encoders, similar to the point-wise ranking (Nogueira and Cho, 2020), In addition, we adopt the common in-batch negative sampling (Karpukhin et al., 2020) to obtain a random negative document d− and formuate the loss Lbce as:
1
2|DB|
DB∑
d
− logPF (q̂, d) + logPF (q̂, d−), (5)
where d− is sampled from documents other than the d (i.e., the positive one) within the same minibatch DB; we choose the one with the highest predicted relevance as the negative document sample. Note that this is not a hard negative mining startegy (Xiong et al., 2020); it is only for avoiding underlying overfiting caused by imbalanced labels.
6We treat q̂(r=0) as the hard negative query and leave the exploration of other interpolated ones as our future work.
Dual learning with query-based objectives. In addition, we include query-based learning with synthetic positive and negative query pairs. The purpose is to enhance domain-specific knowledge through learning from query-query similarity. We hypothesize that the hard negative query could provide a misunderstanding comprehension of the document, offering another view of negative relevance, and thereby steering the ranking model to familiarize itself with unseen domains. Here, we adopt the margin ranking loss with query-query similarity as follows:
Lmr = DB∑
d
max ( 0, F (q̂, q̂−)− F (q̂, d) ) . (6)
The intuition is that the relevance of the hard negative query should not be greater than the querydocument relevance, providing extra gradient for relevance classification. Finally, we fine-tune domain-adaptive cross-encoders in a few-shot manner with the two objectives in Eq.(5) and Eq. (6).

heading: 4 Semi-supervised MSMARCO Document-centric Pairs
text: To fine-tune ReadQG, we collect training pairs for query generation using the MSMARCO passage ranking dataset (Bajaj et al., 2018). We utilize this dataset, along with the pseudo-relevance of BM25 hard negatives7, which are predicted by the off-the-shelf ranking model, MiniLM.8 This cross-
7https://huggingface.co/ datasets/sentence-transformers/ msmarco-hard-negatives
8https://huggingface.co/cross-encoder/ ms-marco-MiniLM-L-6-v2
encoder is fine-tuned on MS MARCO triplets. The procedure is illustrated in Figure 4.
Query-centric to document-wise aggregation. First, we sort the (query-centric) rank list by pseudo-relevance, as shown at the top of Figure 4. We also define the relevance boundary of positive and negative as 0 since MiniLM was fine-tuned with a regression-like objective. Next, we aggregate queries in a document-wise manner. For example, “Doc07” in Figure 4 appears in three ranking lists. We then re-sort the pseudo-relevance across these three lists and categorize them as positive or negative with the boundary of 0, resulting in semi-supervised document-to-queries pairs.
Re-scaling and sampling. In addition, for each pair, we rescale the relevance scores of the aggregated query set using MinMaxScaler.9 The purpose of this step is to align the scores with the relevance prompt function in Eq. (1). Finally, for each document, we collect positive queries with the top-m highest relevance scores into R+d . Conversely, queries with the bottom-m lowest scores are considered negative query samples. Documents with fewer than 2m queries are discarded, resulting in approximately 4.7M document-centric training pairs for ReadQG.

heading: 5 Experimental Setups
text: We will first report the setup of the two stages in the proposed pipeline: data augmnetation using ReadQG (Section 3.2), and domain adaptive passage re-ranking (Section 3.3) fine-tuned on the synthetic training data.

heading: 5.1 Training and Inference Setups
text: Stage I: Data augmentation. Our ReadQG is initialized with Flan-T5 base checkpoint.10 with only a few tunable parameters of embedding prompts (Section 3.2). We set the length of instruction and relevance prompt as 10 and 5, respectively, ensuring the lightweight training and inference overhead. The generator is then fine-tuned on the semisupervised training pairs (See Section 4 for details) for 20K steps with a constant learning rate 1e-2. The maximum sequence length of input (document) and target (query) are 128 and 16. We use batch size of 32, comprising 4 documents and m = 4, for each positive and negative query samples.
9https://scikit-learn.org 10https://huggingface.co/google/
flan-t5-base
During inference, to control the generation of positive and hard negative query, we specify the relevance scores as r = 1 and r = 0, respectively. We then construct the composite input for positive and negative query generation as described in Eq. (1). The maximum sequence length of input and output are 384 and 64 with top-k (k = 10) decoding strategies (Fan et al., 2018). We also analyze greedy and beam search decoding strategy in Section 6.2.
Stage II: Domain-adaptative passage re-ranking Once we have the synthetic training pairs, we use them to fine-tune domain-adaptive passage ranking models. We initialize the models with MiniLM pre-trained on MSMARCO passage ranking, the same model used for pseudo-relevance labels in Section 4. Then, we fine-tune the model with batch size 8 for 2 epoch11 with learning rate 7e-6. An epoch of training steps is defined as the corpus size divided by batch size, as we only generate one query pair per document. We set the maximum length as 384. Other training hyperparameters follow the default setups of SentenceBERT cross-encoder.12

heading: 5.2 Evaluation Setups
text: BEIR benchmark. We experiment on BEIR and select several tasks with corpus size is less than 100K for evaluation, including NFCorpus (NFC, 3.6K), FiQA (FQA, 57.6K), ArguAna (ARG, 8.7K), SCIDOCS (SCD, 25.7K), and SciFact (SCF, 5.2K). For brevity, we will use these abbreviations henceforth. We first validate the domain adaptation capability by out-of-domain text ranking effectiveness nDCG@10, the official metric in BEIR. We used the fixed candidates from BM25 top-100 retrieved results and foucsed on the re-ranking effectiveness for simpler comparison.
Performance metrics. We also investigate the unique charateristic of generated queries directly from an IR perspective. Specifically, by using the off-the-shelf bi-encoders and cross-encoders, we can analyze the useful properties of synthetic query. We introduce the following:
1. Diversity. We regard the generated queries Q from an unseen document as different texts.
11We found there is no improvement after the second. 12https://github.com/UKPLab/
sentence-transformers
Thus, we first encode nQ queries with off-theshelf bi-encoders13 E∗. Then, we compute the average pairwise angular distance (Cer et al., 2018) across n query embeddings as follow:
2
n2Q − nQ
nQ∑
i=1
∑
j=i+1
Ω (E∗(qi), E∗(qj)) ,
where we set nQ = 11 and indicate relevance scores r ∈ {0, 0.1, ..., 1.0}. Ω(u, v) indicates the angular distance between vectors u and v.
2. Relevancy. In addition, we feed the document with our generated positive and negative query into another effective cross-encoder model. We use monoT5-3B-InPars-v2 (Jeronymo et al., 2023) as we assume the predicted scores of the larger model can accurately reflect the true relevance between query and document. These metrics include
rel+ =F ∗(q̂(r=1), d);
rel− =F ∗(q̂(r=0), d);
∆rel =rel+ − rel−.
Note that all metrics will first be calculated per document and then take the average across documents in our later results.

heading: 6 Empirical Results
text: In this section, we first validate the text ranking effectiveness using the synthetic data constructed by ReadQG. Then, we explore the query generation effectiveness via the aforementioned three performance metrics.

heading: 6.1 Main Results
text: Out-of-domain text ranking. Table 1 shows that the domain-adaptive text ranking models fine-tuned with an additional negative query from ReadQG
13We use GTE encoder (Li et al., 2023) as it has been pretrained on scientific corpora.
(condition #(3)) increase the average nDCG@10 by approximately two points compared to the initial zero-shot one (i.e., #(0)). This indicates the positive query together with the hard negative query can transfer useful signals during learning. Moreover, condition #(3) outperforms our baseline #(2), the condition used only positive queries. This implies the role of negative relevance in Eq. (6) can guide models to accurately estimate the relevancy of queries and documents.
We also compare with the generated query from InPars-v2 (Jeronymo et al., 2023) by fine-tuning the synthetic pairs with identical settings. Note that we here exclude negative documents in the released data14 for a fair comparison. We also align the amounts of training pairs by random sampling. By comparing conditions #(1) and #(2), we observe the positive queries generated by ReadQG can perform on par with InPars-v2’s15 with a smaller generator (i.e., 220M parameters), demonstrating an efficient alternative to transfer knowledge from richresource MSMARCO dataset (Bajaj et al., 2018).
Generation Quality. To better understand generated queries, we compare the variants of our proposed learning objectives in Section 3.2, as shown in different rows in Table 2. We fixed all the settings of query generation, including prompt length
14https://huggingface.co/datasets/ inpars/generated-data
15For a fair comparison, we shuffle the generated queries and sample the same size as ours. And we only used the positive query-document pairs.
as 5 and greedy decoding. We observe there is no single metric solely related to the ranking effectiveness. However, one interesting finding is that the condition #(d) (i.e., MLE + two diverse generation losses) and condition #(a) (MLE only ) have similar diversity, but their relevance scores of negative queries (i.e., rel−) differ; condition #(d) has .935 but #(a) is .859. This highlights the unique characteristic of harder negative query – high diversity but also high relevance (Div. ↑; rel− ↑) – with the same document. This also shows that calibration loss with self-contrastive loss can complement each other and produce better relevanceaware diver queries. The high diversity sometimes hurts text ranking effectiveness such as condition #(c), meaning that the negative query is too trivial (i.e., random negative query).

heading: 6.2 ReadQG Analysis
text: Length of relevance prompt. In Table 3, we investigate different lengths of soft relevance prompts as many studies have claimed the impact of prompt length is significant (Li and Liang, 2021; Lester et al., 2021). We train generators with fixed learning objectives and inference with the same greedy decoding. Comparing the first two rows (lengths of 1 and 5), we observe the improvement is attributed to the better expression capability with longer prompts, enabling to parameterize more nonlinearity of query-document relevance. However, further increasing prompt length may not significantly increase the diversity of generated queries and result in lower diversity (Div. ↓). Moreover, the retrieval effectiveness would be limited when the prompt length is too long even though the relevance of the negative query is higher (rel− ↑). This
finding aligns with our observation in Table 1 that the informative hard negative query is meaningful when exhibits both high diversity and relevance.
Decoding strategies. Table 4 demonstrates the different decoding strategies. Intuitively, we consider beam search as the most effective option for negative query generation. However, the top-k sampling is the better strategy considering the efficiency. It can balance diversity and efficiency. However, since we only test the hard negative with relevance score r = 0, it required more investigation for interpolated query and the corresponding learning design of text ranking. We hypothesize the diverse generated queries can similarly benefit the dense retrieval models like Lin et al. (2023), which we leave it as our future work.

heading: 7 Conclusion
text: In this study, we present relevance-aware diverse queries generation and validate several setups for constructing more informative queries. The generation of negative query can benefit from appropriate soft-prompt tuning and diverse generation constraints, resulting in a more effective learning process of text ranking models. Thus, we consider the negative query generation as a potential research direction. There are several other avenues for future work, including (1) scaling up ReadQG or prompting larger LLMs for negative queries; (2) mining hard negative documents with hard negative query; (3) fine-tuning bi-encoders dense retrieval with an additional negative query; (4) exploreing more complicated learning tehcniques (Ren et al., 2021; Li et al., 2021) that can fully exploit interpolated negative queries. Regarding the domain adaptation, we suspect the query distribution will be another important factor, as seen in promptagator (Dai et al., 2023) boost the performance with few in-domain data. More empirical evaluation on other benchmarks like Massive Textual Evaluation Benchmark (mteb) can also provide deeper insights of the usefulness of hard negative queries.

heading: Acknowledgements
text: This research was partially supported by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https:// hybrid-intelligence-centre.nl.

heading: Future Work
text: There are several other avenues for future work, including (1) scaling up ReadQG or prompting larger LLMs for negative queries; (2) mining hard negative documents with hard negative query; (3) fine-tuning bi-encoders dense retrieval with an additional negative query; (4) exploreing more complicated learning tehcniques (Ren et al. , 2021; Li et al. , 2021) that can fully exploit interpolated negative queries. Regarding the domain adaptation, we suspect the query distribution will be another important factor, as seen in promptagator (Dai et al. , 2023) boost the performance with few in-domain data. More empirical evaluation on other benchmarks like Massive Textual Evaluation Benchmark (mteb) can also provide deeper insights of the usefulness of hard negative queries