On Complementarity Objectives for Hybrid Retrieval
Year: 2023

Abstract:
Dense retrieval has shown promising results in various information retrieval tasks, and hybrid retrieval, combined with the strength of sparse retrieval, has also been actively studied. A key challenge in hybrid retrieval is to make sparse and dense complementary to each other. Existing models have focused on dense models to capture “residual” features neglected in the sparse models. Our key distinction is to show how this notion of residual complementarity is limited, and propose a new objective, denoted as RoC (Ratio of Complementarity), which captures a fuller notion of complementarity. We propose a two-level orthogonality designed to improve RoC, then show that the improved RoC of our model, in turn, improves the performance of hybrid retrieval. Our method outperforms all state-of-the-art methods on three representative IR benchmarks: MSMARCOPassage, Natural Questions, and TREC Robust04, with statistical significance. Our finding is also consistent in various adversarial settings.

heading: 1 Introduction
text: Representing and matching queries and documents (or answers) is crucial for designing models for Information Retrieval (IR) and open-domain Question Answering (QA). Existing approaches have been categorized into sparse and dense retrieval.
Classic sparse (or symbolic) retrieval such as BM25 (Robertson and Zaragoza, 2009), quantifies the lexical overlaps (or exact matches) between query q and document d, weighted by term frequency (tf) and inverse document frequency (idf). Such computation can be efficiently localized to a few high-scoring q-d pairs with an inverted index, may fail to match pairs with term mismatches. For example, a text pair with identical
∗Corresponding Author †Work done before joining current affiliation.
intent—“facebook change password” and “fb modify passwd”—does not share any common word, so the pair cannot be matched by lexical retrieval.
To overcome such mismatches, dense retrieval models, such as BERT-based DPR (Karpukhin et al., 2020) or coCondenser (Gao and Callan, 2021), aim to support soft “semantic matching”, by encoding queries and documents into lowdimensional embedding vectors. Dense representation is trained so that “password” and “passwd” are located close in the space even though they have different lexical representations.
These complementary advantages of each model have naturally motivated hybrid models (Gao et al., 2020; Yadav et al., 2020; Ma et al., 2021), which we denote as BM25+DPR, extracting scores from both models and selecting documents with the highest linearly combined scores.
To illustrate how we advance BM25+DPR baseline, Figure 1(a) shows Recall@10 of BM25+DPR on Natural Questions, where a yellow circle, represents questions answerable by BM25, or S, and a blue circle, represents those answerable by DPR, or D. Desirably, two retrievers together should cover all questions in the universe U, but failure is 46.5%, which corresponds to U −D ∪ S.
To improve, there are two directions: (1) enlarging |D| and (2) making it more complementary to S. Figure 1(b) plots CLEAR (Gao et al., 2020), aiming to emphasize “residual” features neglected in sparse model, or, increase |D − S|. Though |D − S| increased from 15.2% (Figure 1a) to 20.0% (Figure 1b), as intended, failure did not decrease significantly, from 46.5% (Figure 1a) to 41.8% (Figure 1b). We argue this decrease in failure cases, is confounded by enlarging |D| from 47.6 (Figure 1a) to 54% (Figure 1b), by comparing with a hypothetical scenario keeping D fixed, but reducing failure cases significantly from 41.8% to 14.1% when the intersection is reduced.
Based on these observations, we propose a novel
13357
Semantic Lexical
Fail: 46.5% Fail: 41.8% Fail: 37.0%↓
32.4% 34.0%↑ 27.7%↓5.9% 4.2% 10.5%15.2% 20.0% 24.8%
(a) DPR baseline (b) Joint learning (CLEAR) (c) Joint learning (Ours)
Semantic Lexical Semantic Lexical
Fail: 37.0%↓
27.7%↓ 10.5%24.8%
𝐷𝐷 𝑆𝑆
complementarity metric, considering the residual complementarity |D − S|, but relatively to |D|, denoted as Ratio of Complementarity (RoC). RoC is designed to be 1 when two models are disjoint (Figure 1c), and 0 when D is subsumed to S.
RoC = |D − S| |D| = 1− |D ∩ S| |D| (1)
RoC has the following two advantages:
• RoC is backwardly compatible with existing residual complementary notions. We later derive (Section 3) that optimizing RoC can be divided into two sub-goals of increases |D− S| and |D ∩ S|, where the former is compatible with residual complementarity.
• RoC, when multiplied by |D ∪ S|, directly approximates the number of questions that can be answered by hybrid models. As a result, increasing RoC as an objective, naturally correlates to the improved performance of a hybrid retriever.
With these advantages, we use our metric in Table 1 to explain the limitation of CLEAR building on residual complementarity alone. CLEAR increases |D − S| as intended, but increases |D ∩ S| as its byproduct (see Figure 1b), due to their correlation. In contrast, we propose a simple but effective twolevel orthogonality to resolve this correlation, and achieves both sub-goals, which significantly improves RoC. Table 1 shows that the improved RoC correlates to the improved recall as well *.
We verify that enhancement in RoC leads to improvement in hybrid retrieval performance on three IR datasets: MS MARCO, Natural Questions and TREC Robust04.
*This table is a motivational preview, and detailed setting and results can be found in Section 4.2
Model Increase|D − S| Decrease |D ∩ S| RoC R@100 on NQ
CLEAR ✓ +0.05 +0.51 Ours ✓ ✓ +0.12 +1.19
Table 1: Relative increases in two sub-goals, with respect to BM25+DPR (Figure 1a)

heading: 2 Related Work

heading: 2.1 Sparse and dense retrieval
text: Sparse (or symbolic) space is generally independent such that data structures, such as inverted index or bitmap, can efficiently identify matching candidates with exact matches, and ranking can also be efficiently computed. BM25 (Robertson and Zaragoza, 2009) is a well-known lexical ranking model using bag-of-words representation.
Meanwhile, dense retrieval models (Shen et al., 2014; Guo et al., 2016; Zhai et al., 2016; Nogueira and Cho, 2019; Zhan et al., 2020) have been proposed to tackle the term mismatch problem, which can be categorized as two groups (Guo et al., 2016): (1) embedding-based and (2) interactionbased models. Our target scenario is the former, representing query q and document d into two independent dense vectors and match q-d by using the vector similarity. However, we also discuss how our idea can apply to interaction-based ranking approaches (Nogueira and Cho, 2019; McDonald et al., 2018), capturing word-by-word interactions without vectors, which we discuss as nonembedding models in Section 3.3.

heading: 2.2 Complementarity
text: To leverage complementarity, there have been approaches to either combine the two spaces, or transfer knowledge from one space to another.
First, for combining, a naive approach is aggregating the scores from two spaces (Ma et al., 2021), which is advanced to a more sophisticated model,
such as CLEAR (Gao et al., 2020) learning by a residual margin of BM25. Specifically, when q-d pair is hard to match lexically, the margin becomes larger, then the loss for semantic matching is emphasized. In other words, the model is trained
Second, for transferring, SparTerm (Bai et al., 2020) learns sparse representations by distilling contextualized knowledge of BERT into bag-ofwords space. Specifically, based on BERT encoders, SparTerm first produces a dense distribution of semantic importance for the vocabulary terms, then controls the activation of each term, ensuring the sparsity of the final representations. It means that, the representation capacity of termbased matching methods can be improved up to semantic-level matching. Ours falls into the first category that combines the two spaces, but we propose a new metric named RoC to evaluate complementarity in hybrid retrieval directly. Table 1 shows how existing complementarity metrics only partially cover RoC. Meanwhile, we also discuss how ours can be combined with the second category approaches, i.e., SparTerm, to achieve further gains (Section 4.3).

heading: 3 Proposed Method
text: We propose RoC in Section 3.1 and then discuss how S (Section 3.2) and D (Section 3.3) are implemented. Section 3.4 discusses how D and S can be combined to optimize RoC.

heading: 3.1 Ratio of Complementarity (RoC)
text: RoC is a metric that can measure complementarity and directly approximates failure cases in Figure 1, i.e., RoC ∝ |U − Fail| where U indicates all
answer documents. Based on this hypothesis, our goal is to maximize |D ∪ S| · RoC. We describe this goal into two sub-goals as follows:
|D ∪ S| ·RoC = |D ∪ S| · |D − S| / |D| = (|D − S|+ |S|) · |D − S| / |D| ≃ |D − S|2 / |D| ∗†
= |D − S| / (1 + |D ∩ S||D − S|)
(2)
The first sub-goal is optimizing |D ∩ S|, for which we separate the features captured by the sparse and dense models from each other. The second sub-goal is to maximize |D−S|, for which we propose to capture residual features of the sparse model. We describe how to achieve each sub-goal in Section 3.3.

heading: 3.2 Lexical Representation (S)
text: While any lexical retriever can be used, we describe our approach with BM25 (Robertson and Zaragoza, 2009) to construct symbolic representation (green vector qlex in Figure 2(left)), to capture lexical matches. BM25 score can be written as an inner product between bag-of-words representations of the query and document. We define q and d representation from BM25 as qbm25 and dbm25 ∈ R|V |, respectively, where the i-th element of the representations qbm25 and dbm25 can be writ-
†since |S| is a constant.
ten as follows:
qbm25(i) =
{ IDF(qi) qi ∈ q
0 qi ̸∈ q
dbm25(i) = TF(di)(k1 + 1)
TF(di) + k1 · (1− b+ b |d|avgdl) ,
where IDF(·) is inverse document frequency of term i, and TF(·) is frequency of term i in a given document. Thus, BM25(q, d) can be denoted as an inner product between qbm25 and dbm25.
Given that the lexical representations have much larger dimensionality than semantic representations, it is required to compress qbm25 and dbm25 into a low-dimensional space. For such compression, random projection (Vempala, 2004) was found effective for preserving document ranking in Luan et al. (2020), which we adopt in this work. Though compression loss exists, this loss can be bounded by changing embedding dimension k. In our experiments on NQ, we follow the protocol from Luan et al. (2020), to set k as 715, which guarantees errors to be lower than 0.038, for the 768 dimension BERT embedding. Random projection is a linear transformation via matrix A, and each element of the matrix A ∈ R768×|V | is randomly sampled from a Rademacher distribution with equal probability from the two values: {− 1√
768 , 1√ 768 }.
The final lexical representation, qlex and dlex, can be obtained as follows:
qlex = A · qbm25, dlex = A · dbm25 (3)
With Eq. (3), qlex and dlex are in the same dimensional space as semantic vectors from BERT, while preserving the ranking. In addition, our goal is to enforce complementarity with the semantic vectors in Section 3.3.
From lexical representations, the final relevance score between query q and document d is calculated by an inner product, as follows:
Scorelex(q, d) = qlex · dlex (4)
This relevance score is approximated to BM25 score, and at the same time, we can handle the two vectors, qlex and dlex, in semantic space.

heading: 3.3 Semantic Representation (D)
text: For semantic representation (pink vectors in Figure 2(left)), we adopt a state-of-the-art (coCondenser; (Gao and Callan, 2021)) for explanation
purposes, consisting of a dual-encoder structure based on BERT (Devlin et al., 2019). Thus, our dense retrieval follows BERT’s architecture, settings, and hyper-parameters. Following BERT’s input style, we apply wordpiece tokenizer to the input document and query, and then add a [CLS] token at the beginning and a [SEP] token at the end, as follows:
Input(·) = [CLS] Tokenizer(·) [SEP] (5)
Then, we take the embeddings of queries and documents, from the representation of BERT at [CLS] token. The semantic representations of q and d can be formulated as follows:
hq = BERT(Input(q)) ∈ R|q|×768 hd = BERT(Input(d)) ∈ R|d|×768
qsem = Pool(hq), dsem = Pool(hd) ∈ R768 (6)
where Pool(·) indicates [CLS] pooling extracting the first vector over the hidden states h. Their semantic relevance Scoresem is calculated by an inner product of qsem and dsem: Scoresem = qsem · dsem.
The training loss for DPR is the negative log likelihood of the positive passage:
Lrel = − log eScoresem(q,d +)
eScoresem(q,d+) + ∑
d− e Scoresem(q,d−)
,
(7)
where d+ and d− indicate positive and negative documents corresponding to given query q. For selecting the negative documents, we follow the convention in previous works (Karpukhin et al., 2020; Sachan et al., 2021; Gao et al., 2020), i.e., hard negative sampling, of selecting top-ranked documents retrieved from BM25 that do not contain the answer.

heading: 3.4 Complementarity Objective
text: We propose embedding-level and input-level orthogonality constraints, which decrease |D ∩ S| and increase |D − S|, respectively. Embedding-level Orthogonality To reduce |D∩ S|, we separate the features between the semantic and lexical representation spaces. Specifically, we enforce orthogonality between lexical and semantic representations (i.e., qlex ⊥ qsem). While training BERT by Eq. (7), we impose an additional constraint using cosine similarity, which normalizes
the features, to constrain the direction of the two vectors (lex and sem). We define the loss function of the orthogonality, as follows:
Lortho = ( ⟨qlex, qsem⟩ ∥qlex∥∥qsem∥ )2 + ( ⟨dlex, dsem⟩ ∥dlex∥∥dsem∥ )2
(8) where ⟨·, ·⟩ is an inner product. If the two vectors are perfectly perpendicular to each other, the loss is equal to 0; otherwise, it has a positive value. This is compatible with our goal of minimizing common features of the two vectors, resulting in the reduced overlap |D ∩ S| of the semantic (D) and lexical (S) model, as in Figure 1(c). Adding this orthogonality loss, the final loss function for BERT-ranker is computed as follows:
Ltotal = Lrel + Lortho (9)
While we can tune the above aggregation, we empirically found 1:1 aggregation was effective.
Input-level Orthogonality For increasing |D − S| in the input-level, we follow the convention of using residual features, neglected by the sparse model, such as the synonymy between mismatched terms, e.g., "password = passwd". For this objective, we propose a method to perturb a subset of matched tokens for learning mismatched terms, similar to a denoising autoencoder (Hill et al., 2016). In the denoising autoencoder, input text is corrupted by random noise function, then the decoder is trained to recover the original text, learning robust features on variances (Vincent et al., 2008). In contrast, while our token perturbation does not have the recovering decoder, our distinction is corrupting exact matches, focusing on soft matching. Given a q-d pair, we denote a set of exact matched tokens in d as XEM = {xi|xi ∈ q and xi ∈ d}. Through random sampling, we replace tokens in Xem with the [MASK] token and feed the new sequence d′ into BERT. By the token perturbation, we modify dsem in Eq. (6) to d′sem, computed as follows:
d′ = d\{Sample(XEM )} d′sem = Pool(BERT(Input(d ′))) (10)
where Sample(·) is random sampling of tokens.‡ This perturbation is applied for only training process and we do not use this at inference time.
‡We sample 15% tokens in Xem and this ratio was decided empirically on the dev set.
This perturbation enables to apply disentanglement ideas, not only to new models, but diverse ranges of existing models (See Section 4.1).
Final Relevance Score For aggregating the scores from the two IR models, we follow the convention of major baselines (Karpukhin et al., 2020; Gao et al., 2020; Ma et al., 2021), using a linear combination.
Scoredual(q, d) = Scorelex(q, d) + λScoresem(q, d)
= qlex · dlex + λqsem · dsem (11)
where λ is the hyper-parameter controlling the weight for the different scales.

heading: 4 Experiment
text: In this section, we describe experimental setting and formulate our research questions to guide our experiments.

heading: 4.1 Experimental Setting
text: Dataset To validate the effectiveness of our method, we conduct query-passage (or, querydocument) matching for the following three datasets, which are widely used and statistically diverse (Table 2) as well:
• MS MARCO-Passage§ (Nguyen et al., 2016): This benchmark provides 8.8 million passages, and labels are obtained from the top-10 results retrieved by the Bing search engine. As the relevance labels for the official test set are not publicly available, we evaluate the development set only. We use MRR@10 and R@100 to evaluate the performance for full-ranking retrieval.
• Natural Questions¶ (Kwiatkowski et al., 2019): In this dataset, we aim to find relevant passages that answer the given question from total 250K §https://github.com/microsoft/
MSMARCO-Passage-Ranking ¶https://ai.google.com/research/ NaturalQuestions/download
passages, and labels are minded from spans in Wikipedia articles identified by annotators. Following DPR (Karpukhin et al., 2020), we consider the passages including answers as relevant passages at evaluation regarding R@k.
• TREC Robust04|| (Voorhees et al., 2005): This dataset contains 250 topic queries and 528K documents. As there is no official train/test split published in Robust04, we follow the split setting provided in McDonald et al. (2018) using 5-fold cross-validation.
We honor the metrics used in the original work, which explains different metrics for different datasets. Robust04 is widely used but small in size, so we also follow the convention of studying MS MARCO and Natural Questions with larger sizes.
Implementation For DPR encoder, we use a base version (Uncased) of BERT (Devlin et al., 2019). For training, we set batch size 10 and use Adam (Kingma and Ba, 2015) optimizer with learning rate 0.0002. For stable training, we used gradient clipping (Pascanu et al., 2013) with norm 1.0, and we halve the learning rate for every epoch after 3 epochs of training iteration. We follow
||https://github.com/nlpaueb/ deep-relevance-ranking
DPR (Karpukhin et al., 2020) for the other training details such as hard negative sampling.
As hyper-parameters, we automatically found the best values for λ, based on MAP on development set, where we search λ in a range of [0, 2] with 0.1 step size. The best configuration for λ was 1.5, 1.3 and 2.0 on MARCO-Passage, Natural Questions and Robust04, respectively.
Evaluation Metric For task evaluation, we compute the following metrics and report average performance: Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (nDCG), and Recall at top-k ranks (R@k). For Recall, we follow the previous work (Karpukhin et al., 2020), which is computed as the proportion of questions to which the top-k retrieved passages contain answers. For MAP and nDCG, we use the latest TREC evaluation script** to compute these metrics. Results of the p-value < 0.05 on the t-test with Bonferroni correction are displayed in bold in Table 3.
Baselines We compare our model with the following baselines which are state-of-the-art retrievals. We use SPARTERM (Bai et al., 2020), COCONDENSER (Gao and Callan, 2021), and BM25
**https://trec.nist.gov/trec_eval/
as our baselines. SPARTERM is a term-based retrieval model using BERT, and gives lexical matching score. BM25 is a well-known lexical matching method using TF and IDF. For BM25, we use Pyserini†† open-source implementation. For coCondenser, we use open-source implementation‡‡ to reproduce. On the other hand, we use hybrid space baselines such as COCONDENSER+BM25 (Naive sum) and CLEAR (Gao et al., 2020). Both methods give similarity score by merging the scores of sparse and dense model. Note our implementation of CLEAR performs better than their published results, as we update its base transformer with coCondenser. For fair comparison, both ours and CLEAR build upon the same coCondenser implementation.

heading: 4.2 Experimental Results
text: Research Questions To evaluate the effectiveness of our method, we address the following research questions:
• RQ1: Does the two-level orthogonality improves the RoC?
• RQ2: Does the improved RoC contribute to better complementarity?
• RQ3: Does the improved complementarity improve hybrid retrieval?

heading: 4.2.1 RQ1: Effectiveness of Orthogonality
text: ††https://github.com/castorini/pyserini ‡‡https://github.com/luyug/Condenser
𝐷𝐷 (DPR)
Fail: 46.5% Fail: 41.8%
32.4% 34.0%↑5.9% 4.2% 38.3%15.2% 20.0% 47.6%
(a) D + S baseline (b) Hybrid Learning (CLEAR) (c) Hypothetical Scenario
𝐷𝐷+ (CLEAR) 𝑆𝑆
Semantic Lexical
Fail: 46.5% Fail: 41.8% Fail: 37.0%↓
32.4% 34.0%↑ 27.7%↓5.9% 4.2% 10.5%15.2% 20.0% 24.8%
(a) DPR baseline (b) Joint learning (CLEAR) (c) Joint learning (Ours)
Semantic Lexical Semantic Lexical
𝑆𝑆 𝐷𝐷 𝑆𝑆
We conduct an ablation study to confirm whether the two orthogonality constraints contribute to improving RoC. As Table 5 shows, embedding-level orthogonality improves RoC by 0.1 compared to naive sum and input-level orthogonality improves by 0.06. Applying both of these improves RoC by 0.15, which is a significant improvement compared to CLEAR, while CLEAR improves by only 0.05 from the naive sum.

heading: 4.2.2 RQ2: Improved Complementarity with RoC
text: In this section, we show our method improves complementarity, by using recall of hybrid retrieval, and also adversarial evaluation.
First, ours (Figure 3) shows 0.15 higher RoC than CLEAR because of the improvement in RoC and failure cases are reduced compared to CLEAR (Figure 1b). In other words, RoC is a more reliable predictor of |D ∪ S|, which directly correlates to performance.
Second, we can also observe complementarity in adverse scenarios. We categorize queries into two groups, BM25-Easy and BM25-Hard, following the convention of (Wei and Zou, 2019). Specifically, we define easy and hard set, by sorting MRR@10 scores of BM25 for all queries. Top 50% is BM25-Easy, where BM25 alone is already competitive, and the rest is BM25-Hard, which is adverse for lexical retrievers.
Desirably, we expect a hybrid model to outperform BM25 ranking in the hard set. With this expectation, on Figure 4, we compare the ratio that the hybrid model provides better ranking (with respect to MRR@10). Surprisingly, CLEAR does not improve such ratio of DPR much (+1.5%), which is consistent with the results in Figure 5. In contrast, we significantly improve the ratio by +10.2%.
Alternative way to observe adverse scenarios, is to build an adverse dataset with less matched terms. Specifically, lexically matched terms be-
tween query and document are replaced by their synonyms (as similarly done in (Wei and Zou, 2019)). Figure 5 compares how CLEAR and ours generalize to this adverse set. Both models on the original dataset improve MRR rapidly in early epochs, while not so much on adverse set until epoch 5. However, this gap decreases only in ours (Figure 5b), while stays constant in CLEAR (Figure 5a), showing CLEAR continues to focus on lexical matching, while we learn to leverage semantic matching.

heading: 4.2.3 RQ3: Effect of Complementarity in Hybrid Retrieval
text: In this experiment, we verify the effect of enhanced complementarity on various performance aspects.
We first compare the performance of the hybrid model and ours to show that our two-level orthogonality improves the hybrid retrieval performance as well as the complementarity between the two models. In Table 3, when compared with coCondenser+BM25 (Naive sum), our method improved MRR@10 by 0.78 on MSMARCO, and MAP by 1.50 & 2.21 on NQ & Robust04, respectively, showing the complementarity improves document ranking. When compared with the state-ofthe-art model, CLEAR, our method achieved 0.64 gains of MRR@10 on MSMARCO, and 1.04 & 0.88 gains of MAP on NQ & Robust04, respec-
tively. Note that our method has a statistically significant performance improvement, as indicated by superscripts in Table 3.
Ablation Study of Embedding- and Input-level Approaches To investigate the isolated effect of two-level orthogonality on hybrid retrieval performance, we conducted an ablation study in NQ and Robust04 as shown in Table 4. For this, we add each component (embedding- or input-level objective) to the baseline model: DPR+BM25 (Naive sum). In both datasets, we observe that the embedding- and input-level methods can achieve significant improvements over the baseline, showing that the enhanced complementarity improves hybrid retrieval performance. Note that the embedding-level objective is more effective than the input-level objective, which is consistent with the complementarity improvement result in Table 5. We can also see in Table 6 that the input-level objective works even for non-embedding models.
Length Generalizability Based on the wellknown weakness of BERT showing low accuracy on long documents in the NQ dataset (Luan et al., 2020), we verify the effect of improved complementarity on robustness for long documents. Our proposed model outperforms CLEAR and obtains the best scores over all the lengths except one group. This shows that complementarity plays an essential role in length generalization. Results and details are described in Section A.1.

heading: 5 Conclusion
text: We study the problem of hybrid retrieval, where existing state-of-the-arts have pursued a partial notion of complementarity. In contrast, we propose RoC, a metric that captures a fuller notion of the complementarity between sparse and dense models. We then propose a simple but effective twolevel orthogonality objective to enhance RoC and verify that optimizing RoC enhances both com-
plementarity and retrieval, leads to outperforming state-of-the-arts in three representative IR benchmarks, MSMARCO-Passage, Natural Questions, and TREC Robust04, and generalizing to adversarial settings.

heading: 6 Limitations
text: We make use of MS-MARCO, a resource that provides large-scale relevance annotations. However, as with most retrieval datasets, this dataset could contain annotation biases. Given the vast number of documents in the corpus supplied by the dataset, relevance annotations are sparsely distributed, with all other documents assumed to be non-relevant. Consequently, some relevant documents may be inaccurately labeled as non-relevant, leading to false negatives. A notable annotation bias in MSMARCO is that the relevant label correlates highly with the exact matching term (Xiong et al., 2020). This bias poses a limitation during the training or evaluation stages. To appropriately address this annotation bias, we might need to reorganize the labeling process using either a human or a neural annotator, or we could aim to design and train a model that is resilient to such bias. We reserve this task for future research efforts.

heading: Acknowledgements
text: This work was supported by the SNU-NAVER Hyperscale AI Center and MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2023-2020-0-01789) and grants [NO.20210-0268, AI Hub, SNU], [No.2022-0-00077, AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data], and [NO.2021-0-01343, AI Graduate School].

heading: A Appendices
text: A.1 Length Generalizability
As shown in Figure 6, we group test set by the length of target documents (per 40 tokens), and report MAP score per each group. From the results, we can confirm the reported weakness in long documents– Precision of DPR decreases as the document length increases, while that of BM25 stays consistent. Meanwhile, hybrid models including both CLEAR and ours show better robustness than DPR and BM25 over the longer documents. Our proposed model outperforms CLEAR and obtains the best scores over all the lengths except a group “0-40”. This shows that complementarity plays an essential role in length generalization.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We reserve this task for future research efforts