DADA: Distribution-Aware Domain Adaptation of PLMs for Information Retrieval (2024)

Abstract: Pre-trained language models (PLMs) exhibit promising retrieval performance in various domains. However, they struggle in domains unseen during training, since the word distribution can shift significantly. To remedy this, GPL, a generative domain adaptation (DA) method, was proposed to generate pseudo queries and labels for documents in unseen domains to further train the retriever model. However, the pseudo queries often do not resemble real queries from the target domains, as they do not integrate the domain’s distributional information. we propose Distribution-Aware Domain Adaptation (DADA) to guide the model to incorporate the term distributions at both the document-level and the corpus-level, which we refer to as observation-level and domainlevel feedback, respectively. Empirical results on five distinct datasets demonstrate that our method effectively adapts the model to target domains and expands document representation to unseen gold query terms. 1

heading: 1 Introduction
text: Recent advances in pretrained language models (PLMs) (Devlin et al., 2018; Clark et al., 2020) have significantly enhanced our ability to retrieve information (Yates et al., 2021; Nogueira and Cho, 2019). These advancements are particularly pronounced when the data closely aligns with what the PLM was originally trained on, denoted as “indomain” data. However, challenges emerge when dealing with “out-of-domain” (OOD) data, characterized by substantial disparities from the training data. Studies such as Thakur et al. (2021) have highlighted these challenges.
In response to these issues, various domain adaptation (DA) methods have been proposed. A widely
∗Both authors contributed equally to this research. †Corresponding Authors
1The code and experimental details are available at https://github.com/ldilab/dada.
employed technique is Generative Pseudo-query Learning (GPL) (Wang et al., 2021), which entails generating pseudo queries linked to specific documents to enhance PLM learning. However, the efficacy of these methods hinges heavily on the quality of these pseudo queries. Poorly constructed queries can significantly diminish the effectiveness of the method.
DA techniques can be categorized by two main axes, as illustrated in Table 1, based on Chen et al. (2023). The first category examines the level of information used by DA methods, either focusing on individual document distribution (observationlevel) or broader corpus distribution (domain-level). The second category considers how these methods integrate such feedback, either by updating the training data or the loss function used for learning.
DA methods, like GPL, aim to generate pseudo queries that approximate “gold query terms” from users at test time. However, GPL fails to generate terms that are unseen from in-domain training data, when failing to capture the following signals:
• Expanded document observation: Gold query terms may not appear in the observed document, but it might be in its expanded vocabulary including synonyms of the document.
13882
• Contrasted domain distribution: Gold query terms may be infrequently seen in the training corpus but more frequent in the target corpus, as approximated from its subset.
Table 1 demonstrates that GPL concentrates on observation-level feedback by providing expanded terms through query generation. However, Figure 1 reveals that the query generated by GPL (orange) includes only a subset of the gold terms that observation-level can provide (light blue). Even when leveraging the expanded terms from observation, such as synonyms, during the query generation process (blue), denoted as GPL+, the number of generated gold terms does not exhibit a significant increase. In essence, the query generation process results in a lack of crucial input at the observation level, diminishing the effectiveness of data updating methods. This limitation motivates the importance of implementing a loss updating methodology.
In contrast, our proposed DA method: Distribution-Aware Domain Adaptation (DADA), uniquely incorporates both observationand domain-level feedbacks. While GPL leans more towards observation-level feedbacks, DADA widens the observation-level feedback and even integrates domain-level feedback, offering a broader statistical perspective. The observationlevel feedback is widened by domain-level feedback that can approximate the red bar on Figure 1. To obtain such feedback without access to entire OOD corpus, as some documents are accessible in a real-world scenario, we utilize the
2The analysis of discrepant tendency in SciDocs compare to other 3 datasets can be found on Appendix A.2
information obtained from the subset of OOD corpus. Consequently, DADA prevents the loss of essential information during query generation, enhancing the efficiency and effectiveness of the DA process, as demonstrated in experiments on five corpora from the BEIR benchmark(Thakur et al., 2021).

heading: 2 Related Work
text: This section delves into the landscape of PLMs within information retrieval (IR) and examines the evolution of DA techniques that have paved the way for our proposed method.

heading: 2.1 PLMs in IR
text: The integration of PLMs, exemplified by BERT (Devlin et al., 2018) and ELECTRA (Clark et al., 2020), has revolutionized IR. These models excel in understanding and processing complex linguistic patterns, significantly enhancing document retrieval and ranking processes. Despite their advancements, challenges arise when applying these models to OOD data, an issue highlighted in studies such as Thakur et al. (2021). This limitation has promoted research into effective DA techniques to ensure the robustness of PLMs across diverse domains.

heading: 2.2 DA for PLMs
text: Addressing the domain shift challenge has led to the development of various DA strategies. A prominent approach involves creating pseudo queries (Ma et al., 2021; Liang et al., 2020) to bridge the gap between the PLM’s training domain and new target domains. This strategy is crucial in ensuring the relevance and applicability of PLMs to diverse datasets. GPL (Wang et al., 2021) is a noteworthy example, utilizing a cross-encoder mechanism to enhance the alignment between pseudo queries and target documents. However, these techniques tend to focus on observation-level feedback, often overlooking broader domain-level insights.

heading: 2.3 Our Distinction
text: Our method marks a significant departure from traditional approaches. It uniquely integrates both observation-level and domain-level feedback, with a focus on updating the loss function. This duallevel feedback approach ensures comprehensive DA, overcoming the limitations of previous methods that mainly rely on observation-level data up-
dates. DADA’s innovative strategy enhances adaptability and accuracy in handling OOD data, setting a new precedent in DA techniques.

heading: 3 Proposed Method
text: In this section, we provide a detailed explanation of the methods employed in our study, focusing on the integration of the DADA into the GPL framework. We begin with a concise overview of GPL, followed by an in-depth description of how DADA is integrated into GPL, covering both training and inference perspectives.

heading: 3.1 Overview of GPL Training Process
text: Training process of the GPL framework is visually represented in the top of Figure 2. The GPL framework operates by calculating relevance scores between queries and documents using PLMs. Specifically, given a query q and a document d, the relevance score S(q, d) is computed as the dot product of their embedding vectors:
S(q, d;M) = M(q) ·M(d) (1)
where M denotes the PLM. To differentiate between positive (d+) and negative (d−) documents, GPL employs a Margin MSE loss, δ(q, d+, d−;M), defined as the difference between the relevance scores of each document:
δ(q, d+, d−;M) = S(q, d+;M)− S(q, d−;M) (2)
Additionally, GPL utilizes a cross-encoder3 to predict the relevance scores, resulting in the margin value δ̂(q, d). The loss function of GPL is to minimize the loss function:
LGPL(q, d;M) = |δ̂(q, d)− δ(q, d;M)|2 (3)
Our goal is to utilize multi-level feedback to update this loss function for more effective DA.

heading: 3.2 Integration of DADA into GPL
text: To enhance DA within GPL, we introduce the DADA method, which incorporates multi-level feedback and loss function update.

heading: 3.2.1 Multi-level Feedback Vector
text: To capture observation-level and domain-level feedback, each feedback is encoded as a vector. The observation-level distribution encompasses terms
3The cross-encoder parameters are frozen, which is not trainable.
appearing in a document d and augmented terms, while the domain-level distribution approximates the entire set of documents in the OOD corpus.
Observation-level Distribution As illustrated in blue in Figure 2, observation-level feedback is a process where we take information from a document, d, and turn it into a vector form. This is done using SPLADE 4, a method that allows us to expand the document’s original vocabulary, Vd, into an augmented set, V̂d. Once we have this expanded vocabulary, we then create observation-level feedback, Robs as follow:
Robs(d) = [SP(d, v1),SP(d, v2), . . . ,SP(d, vn)] (4) where vi and SP denote the i-th term of V̂d and corresponding weight from SPLADE.
Domain-level Distribution The GPL emphasizes terms that are often not important in the given document, hence having a low likelihood of being included in the gold terms. This is due to the training approach that mostly treats most documents as non-relevant. To overcome this limitation, we provide domain-level feedback. The red lines and shapes of Figure 2 shows the key role of domainlevel feedback. Domain-level feedback approximate the entire set of documents in the OOD corpus and is not limited to a specific document d. To represent the distribution Rdom of these global terms, we employ Inverse Document Frequency (IDF) in this work, as follows:
Rdom = [IDF(v1), IDF(v2), . . . , IDF(vn)] (5)
where vi denotes a term originated from the given OOD corpus and n denotes the number of terms in the corpus. Note that we use up to 1,000 of randomly selected documents to compute the IDF since we assume the oracle IDF distribution is unknown. Just as the use of SPLADE as observationlevel distribution can be straightforwardly changed, IDF can be replaced by other domain-level statistics, and key difference between the two that IDF score does not depend on observation d.
Normalization One important consideration when using both observation-level and domainlevel distributions together is that their scales can
4We selected SPLADE in this work, based on its strong performance in predicting unseen query terms in empirical results, but can straightforwardly support other observationspecific statistics, such as DeepImpact (Mallia et al., 2021).
vary. The different scales of distribution, exceeding the range of 0 to 1, can assign excessively high weights to some specific terms, which will unstabilize the model training. To ensure scale consistency between the two distributions, we first standardize the distribution with z-score normalization, denoted as Z(·), then additionally normalized with tanh, to ensure each term within a single distribution has a weight value between 0 and 1. This two-step normalization can be stated as Norm(·) = tanh (Z(·)).
Merging Two Distributions Finally, as shown in purple in Figure 2, the two distributions Rdomain and Robservation are integrated into a merged distribution Runified through dot product as follows:
Runified(d) = Norm(Rdom) · Norm(Robs(d)) (6)

heading: 3.2.2 Loss function update
text: DADA updates the loss function by injecting combined distribution Runified into GPL. This process entails extracting the term distribution from the model and incorporating it back, thereby aligning the model’s distribution with that of the data. The term distribution Rmodel represents what the model has learned and serves as the foundation for updating the loss function. Hence, updating the loss function involves two steps: extracting the term distribution from the model and injecting this distribution information back into the model.
Extraction from model One crucial consideration when extracting term distribution from a model is addressing the discrepancy between embedding space and data space. Specifically, the representations generated by the model reside within the embedding space, whereas the representations obtained from the data exist within the vocabulary space. To overcome this mismatch, we merge these two representations into a single one while respecting the information they both provide.
Taking inspiration from the Max-Sim operation of ColBERT (Khattab and Zaharia, 2020), we can compute the relevance score for each vocabulary term by selecting the maximum value from the dotproduct between the embedding vector, generated by the model, and the vocabulary vector obtained from the intermediate output of the observationlevel distribution.
Rmodel(d;M) = max i∈[|E|]
(ETi · S) (7)
E = M(d), S = SP (d) (8)
where E is the embedding matrix generated by the model M , and S is the vocabulary matrix from the intermediate output of the observation-level distribution.
Injection into model Incorporating data-driven distribution into the model involves reducing the difference between the two distributions. While data distribution interprets the importance of each
vocabulary term, considering both observationlevel and domain-level distributions, the model’s distribution highlights the relationship between the embedding and vocabulary. The highest value in the vocabulary direction shows the terms the model finds crucial. By minimizing the differences between these two distributions, the model learns the significance of each vocabulary term. Each distribution is normalized with softmax, denoted as R(d), to ensure that the distribution represents probabilities.
The final loss funciton of DADA can be defined as:
LDADA(d;M) = DKL(Runified(d) ∥ Rmodel(d;M)) (9)
where M states the model and DKL means the Kullback-Leibler (KL) divergence (Shlens, 2014).
Therefore, the overall training objective of GPL + DADA is minimizing L(d;M) which is:
L(d;M) = LGPL(q, d;M) + LDADA(d;M) (10)
where q, d, and M state the query, document, and model, respectively.

heading: 3.3 Technique for Stable Training
text: Merging two distributions can be unstable, as some documents align closely, while others diverge significantly. Therefore, it is important to introduce data in a specific sequence to prevent slow learning. To address this, we use a curriculum that guides the model from simple to complex concepts, enhancing convergence and adaptability to the new corpus. Inspired by curriculum learning (Bengio et al., 2009), we gauge learning difficulty as follows:
DKL(Rdom ∥ Robs(d)) (11) Gradually introducing increasingly challenging document distributions during training optimally improves performance, aligning with the concept of the model learning progressively from simpler to more complex data.

heading: 3.4 Inference Process of DADA
text: DADA impacts the training loss of GPL, leaving the inference process unchanged. DADA offers the advantage of enhancing DA without increasing computational costs during inference, ensuring efficient and unaltered operation in this phase. Thus, DADA facilitates optimized DA without compromising the efficiency of inference tasks.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Implementation Detail In our experiments, we employed two types of feedbacks: SPLADE (Formal et al., 2021) for observation-level and IDF 5 for domain-level feedback. Given that the target task involves DA, the model was initially trained on the MS-MARCO, as an in-domain dataset. Input document construction involves concatenating document titles and bodies, and truncating sequences longer than 256 tokens. Queries are truncated to a length of 64 tokens.
The representation of query (q) and document (d) are pooled from the output of the model as it is designed. The relevant document is retrieved using the dot product score, S = Eq ·Ed. Training was performed on four RTX 3090 GPUs with DDP setting, and any unspecified details in the paper follow the same settings as GPL.
Datasets and Evaluation Metrics Our method was evaluated using 5 datasets from the BEIR benchmark (Thakur et al., 2021). Among these, 4 out of 5 datasets (SciFact, SciDocs, FiQA, and NFCorpus) were chosen due to their smaller corpus sizes in the BEIR benchmark datasets, as we focus on scenarios requiring DA, which correlate to limited corpus size. Meanwhile, to observe whether our observation is consistent in a large corpus retrieval scenario, we contrast with Robust04, where the corpus is large and relevant documents can match a query. Detailed information on our selected subtasks is presented in Appendix A.1.
To adapt the model to the target domain, we employed training datasets pairing OOD documents with the pseudo queries and pseudo labels generated by the GPL (Wang et al., 2021) and GPL+.
The evaluation utilized the nDCG@10 metric, widely accepted for assessing the general quality of predictions on the top-10 retrieved documents.
Baselines GPL (Wang et al., 2021) generates pseudo queries and labels using the DocT5Query (Nogueira et al., 2019) query generator and retrievers, incorporating cross-encoders. Both the query generator and retrievers are pretrained on the MS-MARCO dataset (Bajaj et al., 2016). GPL+ closely resembles GPL, with the distinction of incorporating the observation-level distribution into the query generator. The resulting
5We use pyserini library to compute IDF.
pseudo queries, generated with this awareness of the observation-level distribution, are subsequently utilized for hard negative mining and pseudo labeling, following the same procedural steps as GPL.
Retrievers (PLMs) To show the robustness of applying DADA along with GPL, we have chosen three different models: coCondenser (Gao and Callan, 2022)6, COCO-DR (Yu et al., 2022)7 and TAS-B (Hofstätter et al., 2021)8. All three models were trained on the in-domain dataset, MSMARCO. The models are chosen to show the robust performance across different sizes of model parameters9.

heading: 4.2 Results and Analysis
text: Research Questions To further validate the effectiveness of our proposed method, we assess whether our goals have been achieved through the following research questions.
• RQ1: Is domain-level feedback reflected as intended?
• RQ2: Does DADA generate more gold terms as intended?
• RQ3: Does DADA adapt better to a new domain as intended?
6Luyu/co-condenser-marco 7OpenMatch/cocodr-base-msmarco 8sentence-transformers/msmarco-distilbert-base-tas-b 9coCondenser and COCO-DR have 110 million parameters
while TAS-B has 67 million parameters.

heading: 4.3 RQ1: Is domain-level feedback reflected as intended?
text: To assess the effective integration of domain-level feedback into the model, we employ the transformation technique introduced by Ram et al. (2022), which extracts vocabulary distribution vector v from the model.
This procedure utilizes the Masked Language Modeling (MLM) head of the model to carry out the projection. More precisely, an embedding vector e of a document is subjected to transformation by the MLM Head, leading to its projection onto a vector v with dimensions according to the vocabulary size.
Figure 3 illustrates the difference of modelextracted distributions from the IDF distribution of new domain (oracle, which cannot be observed during training), measured using Jensen-Shannon divergence (Nielsen, 2020). Notably, SciDocs, which shows the greatest domain difference from in-domain dataset MS-MARCO, exhibits the lowest GPL performance. Our goal is to adapt the model to perform well in OOD datasets by minimizing these differences in IDF distributions.
To verify whether DADA minimizes the disparities between the IDF distributions depicted in Figure 3, we compare the model-extracted distribution, with the oracle IDF distribution, focusing on the top 1,000 terms with high IDF values.
Figure 4 illustrates the Jaccard similarity (Jaccard, 1912) between the two IDF distributions 10. The results reveal that DADA demonstrates greater similarity than the baseline models in both the NFCorpus and SciFact datasets, with an increase of
10Jaccard similarity quantifies the resemblance between two distributions by calculating the intersection divided by the union, where a higher number of overlapping terms between the two distributions indicates a greater level of similarity
more than two times. However, GPL exhibits lower overall similarity since it does not incorporate domain-level feedback. GPL+ demonstrates greater similarity compared to GPL due to the inclusion an observation-expanded set of terms that includes more relevant phrases. In terms of similarity, the comparison between GPL and GPL+ indicates that the incorporation of domain-level feedback significantly enhances the similarity of IDF distributions.

heading: 4.4 RQ2: Does DADA generate more gold terms as intended?
text: To evaluate whether DADA generates more gold terms, we compare the term distribution from the model (as done in RQ1) and the gold terms query.
Figure 5 illustrates the gold term recall of the top 1,000 terms generated by DADA. Significantly, DADA exhibits superior recall on all OOD datasets, with an improvement ranging from 10% to 20% compared to GPL. This suggests that DADA is more proficient in identifying a larger collection of gold query terms.
Moreover, as depicted in Figure 5(c), the enhanced recall is not solely dependent on identifying seen terms, which are query terms already included in the document. To identify unseen but valuable query terms, both observation-level and domain-level feedback are utilized in DADA, to demonstrate improved ability to identify both seen and unseen terms in comparison to GPL.
For seen terms, DADA not only includes more gold query terms than GPL but also exhibits a similarity of approximately 80% with the oracle IDF distribution. This suggests that using IDF distribution as domain-specific feedback is helpful in
determining the weight for seen terms.

heading: 4.5 RQ3: Does DADA adapt better as intended?
text: In order to evaluate how effectively DADA performs in adapting to different domains, we evaluated its performance on five specific subsets of the BEIR benchmark dataset. Table 2 shows the nDCG@10 scores of GPL and DADA for each subset and each retriever. DADA demonstrates significant improvements in the BEIR subset, presenting a distinct advantage over both GPL and GPL+, on various retrievers.
When applied to GPL, DADA enhances retrieval performance in most subsets, resulting in an average nDCG@10 gain of 0.5 in coCondenser, 0.6 in COCO-DR, and 0.1 in TAS-B. Compared to the other two retrievers, where the model size may affect performance, TAS-B exhibits a relatively small performance gain overall and experiences a slight drop in performance on SciFact and FiQA. Since our approach is intended to incorporate additional distribution information through updates to the loss function, smaller models like TAS-B may encounter challenges in integrating this additional information into their parameter updates. However, despite this challenge, there is still an overall improvement.
While DADA also yields performance improvements on coCondenser with GPL+, there is a slight drop in performance on a few datasets. This discrepancy is attributed to the bias in GPL+ construction, focusing too much on expanding documentspecific synonyms while missing gold query terms, as also illustrated in Figure 1. FiQA and NFCorpus are such datasets, where the number of unseen
query terms in GPL+ is fewer than in GPL, where the adverse effect of GPL+ bias overshadows the positive gains from DADA.
We additionally performed an ablation study to assess the influence of two novel components introduced by DADA into the GPL framework: domainlevel feedback and curriculum learning. The results, presented in Table 3, showcase the performance of DADA on the SciFact, NFCorpus, FiQA and SciDocs datasets as each of these elements is selectively excluded.
The observed decrease in performance when both elements are omitted serves as compelling evidence of their beneficial impact on model training. This analysis offers valuable insights into the individual contributions of these elements and their collective effect on enhancing the overall efficiency of the model.
Unlike other datasets’ performance tendency, SciDocs shows the best performance when the Rdomain is omitted, which may caused by the nearly uniform Rdomain, which makes model suffers learning which terms are important in the target domain. The SciDocs corpus includes scientific articles and papers from various fields, which means that vari-

heading: 12.55 12.60 8.84 6.76
text: ous unique words are shown on the corpus unlike other datasets, this can be rephrased as flat term frequency distribution. Measuring the entropy of term frequency of tokenized document in each dataset, as illustrated in Table 4, SciDocs shows the lowest entropy which ensures the nearly uniform distribution of term frequency.

heading: 5 Conclusion
text: We have introduced a novel method called DADA with the specific aim of enhancing the capabilities of PLMs for addressing domain shifts in information retrieval. Unlike traditional methods that resolve domain shifts through dataset updates based on observation-level feedback, DADA takes a direct approach by updating the loss function with a multi-level feedback vector, which integrates domain-level feedback and observation-level feedback. This innovative approach enables DADA to dynamically adjust to varying data landscapes, making it particularly effective in scenarios where rapid adaptation is essential. Compared to an existing method like GPL on the BEIR benchmark, DADA demonstrates superior performance across various subtasks, highlighting its potential in handling unseen domains. While DADA shows promise, further research could investigate integration with other dataset update adaptation techniques, contributing to the development of more robust and versatile information retrieval systems.

heading: 6 Limitations
text: We use DocT5Query to generate queries in experimental setups following the GPL to align with the baseline method. As this query generator has a limitation in effectively generating OOD terms, though we provide domain-level distribution, we overcome this limitation, by directly injecting the domain-level distribution into the PLM. Another possible approach wouldbe upgrading the query to generate OOD terms effectively when such distributions are provided, potentially by utilizing PLMs. We leave this as a future work but expect similar enhancements due to the orthogonal contributions our method offers to GPL.

heading: Acknowledgements
text: This work has been financially supported by SNUNAVER Hyperscale AI Center. This work was also supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2022-0-00077, AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)].

heading: A Appendix
text: A.1 Target Datasets
In this paper, we aim to address our experiment on 5 BEIR benchmark datasets (Thakur et al., 2021): SciFact (Lo et al., 2019), SciDocs, FiQA (Maia et al., 2018), NFCorpus, and Robust04. The statistics, including domain information, total number of queries, total number of documents, average query length, average document length, and number of relevant documents for a query, of each dataset can be found on Table 5.
The main focus of our method is improving IR performance in circumstances of small corpus size, where it is difficult to adapt the model to the target domain. For this reason, we initially target 4 datasets that have a smaller corpus size, less than
60k documents: SciFact, SciDocs, FiQA, and NFCorpus. In addition to the 4 datasets, we also included 1 large corpus dataset, Robust04, which has more than 60k documents and many relevant documents that match a query, to demonstrate that DADA can also achieve improvements in a large corpus.
A.2 Discrepant Tendency of Unseen Gold Query Term Comparing GPL and GPL+
The SciDocs dataset, as depicted in Figure 1, contains a slightly larger number of gold query terms, that are not present within the corpus, in GPL (light blue bar) compare to GPL+ (blue bar). GPL+ is designed to include expanded terms during the query generation process, expecting it to capture a wider range of possible queries, which will retrieve more relevant information. However, as the query generator works as a proxy for incorporating the expanded terms, it can neglect or dilute the importance of the terms. In addition, because the other three datasets’ corpus term frequency distribution is slightly skewed, which can be found as a high entropy of term frequency in Table 4, it can be more statistically reasonable for the query generator to select seen terms from the given document. For this reason, in most cases, such as NFCorpus, Scifact, and FiQA, the GPL+ shows that the number of unseen gold query terms decreases compared to GPL. Besides the three datasets showing the decrement, the SciDocs shows an increment as the distribution is near uniform, expanding some unseen term can easily increase the value. This marginal improvement in unseen gold query terms, however, does not result in a retrieval performance improvement, in practice, as opposed to the expectation.
A.3 Experiment Results on Additional Metrics
The experiment results are evaluated by various metrics that are often used to evaluate information retrieval performance, such as MRR@10, MRR@100, and nDCG@100. The evaluation results can be found on Table 6, Table 7, and Table 8. The abbreviation of datasets is given as follows. SF: SciFact, SD: SCIDOCS, FQ: FiQA, NF: NFCorpus, RB: Robust04, TC: TREC-COVID)
A.4 Experiment with different observation feedback
The main experiment result shows the impact of using naver/splade_v2_distil for observation feed-
back. As we stated on §3.2.1, the observation feedback can be replaced by other statistics, we also experimented with naver/splade-cocondenserensembledistil and the result can be found on Table 9.

heading: Future Work
text: We leave this as a future work but expect similar enhancements due to the orthogonal contributions our method offers to GPL