Venue: NAACL 2025

# EqualizeIR: Mitigating Linguistic Biases in Retrieval Models

Jiali Cheng Hadi Amiri

University of Massachusetts Lowell {jiali_cheng, hadi_amiri}@uml.edu

###### Abstract

This study finds that existing information retrieval (IR) models show significant biases based on the linguistic complexity of input queries, performing well on linguistically simpler (or more complex) queries while underperforming on linguistically more complex (or simpler) queries. To address this issue, we propose EqualizeIR, a framework to mitigate linguistic biases in IR models. EqualizeIR uses a _linguistically biased_ weak learner to capture linguistic biases in IR datasets and then trains a robust model by regularizing and refining its predictions using the biased weak learner. This approach effectively prevents the robust model from overfitting to specific linguistic patterns in data. We propose four approaches for developing linguistically-biased models. Extensive experiments on several datasets show that our method reduces performance disparities across linguistically simple and complex queries, while improving overall retrieval performance.

## 1 Introduction

Neural ranking models have been extensively used in information retrieval and question answering tasks Dai and Callan (2020); Zhao et al. (2021); Khattab and Zaharia (2020); Karpukhin et al. (2020); Xiong et al. (2021); Hofstatter et al. (2021). We demonstrate that these models can show strong linguistic biases, where the retrieval performance is biased with respect to the "linguistic complexity" of queries, quantified by the variability and sophistication in productive vocabulary and grammatical structures in queries using existing tools Lu (2010, 2012); Lee et al. (2021); Lee and Lee (2023).1

Footnote 1: We consider lexical and syntactic linguistic complexity indicators in this study. Details of these indicators are provided in Appendix B, Table 4.

Figure 1 shows that the average linguistic complexity of the test queries in the NFCorpus Boteva et al. (2016) and FIQA Maia et al. (2018) datasets varies significantly, where the NDCG@10 performance of the BM25 model significantly decreases on NFCorpus and improves on FIQA as the linguistic complexity of queries increase. This performance disparity across queries of different linguistic complexity leads to the focus of this paper and the following research question: _can we debias IR models to achieve equitable performance across queries of varying linguistic complexity?_

Inspired by previous debiasing works in natural language processing Utama et al. (2020); Ghaddar et al. (2021); Sanh et al. (2021); Meissner et al. (2022), we introduce a new approach, named EqualizeIR, to mitigate linguistic biases in IR models. EqualizeIR is a _weak learner_ framework; it first trains a linguistically-biased weak learner to explicitly capture linguistic biases in a dataset. This linguistically-biased weak learner is then used as a reference to inform and regularize the training of a desired (robust) IR model. It encourages the IR model to focus less on biased patterns and more on the underlying relevance signals. This is achieved by using the biased weak learner's predictions as indicators of bias intensity in inputs, and adjusting the IR model's predictions accordingly.

Figure 1: NDCG@10 of BM25 on the test set of NFCorpus Boteva et al. (2016) (left) decreases and on the test set of FIQA Maía et al. (2018) (right) increases as the average linguistic complexity Lu (2010, 2012) of queries increase. Specifically, we observe a significant drop in NDCG@10, from 0.4 to 0, and a significant increase in NDCG@10, from 0.2 to 0.3. The result shows that BM25 is significantly biased toward linguistically easy and hard examples on different datasets.



EqualizeIR does not require linguistic biases to be explicitly described for the model, and reduces the risk of overfitting to specific types of biases. Specifically, we investigate several strategies to develop a linguistically-biased weak learner: training the model using **linguistically biased data** to directly introduce and reinforce specific linguistic patterns, using a **weaker model** with fewer parameters or a simpler architecture to reduce models ability to generalize across inputs with various linguistic complexity, **shortening the training time** to prevent the model from capturing the diversity and depth of linguistic features in the data, and training on a **limited data** to emphasize the linguistic features present in a specific subset of data. Through these strategies, we aim to develop a model that effectively captures linguistic biases for developing linguistically robust IR models.

Our contribution are (a): illustrating that the performance of current IR models vary based on the linguistic complexity of input queries, (b): a novel approach that trains a linguistically robust IR model with the help of a linguistically biased IR model to mitigate such biases, and (c): four approaches to obtain linguistically biased weak learners, all effective in mitigating biases in IR models.

## 2 EqualizeIR

Linguistic Complexity:measures sophistication in productive vocabulary and grammatical structures in textual content, spanning lexical, syntactic, and discourse dimensions. In this work, we adopt existing linguistic complexity measurements (lexical complexity Lu (2012) and syntactic complexity Lu (2010)) to measure the linguistic complexity of queries in IR datasets implemented by existing tools Lu (2010, 2012; Lee et al., 2021; Lee and Lee, 2023). Specifically, given a query \(q\), a linguistic complexity score is computed by averaging scores of various linguistic complexity metrics, which includes measures such as verb sophistication and the number of T-units. The detailed list of linguistic complexity is shown in Appendix B Table 4. We column normalize linguistic complexity scores before computing average linguistic complexity for each query.

Overview:EqualizeIR mitigates linguistic biases in an IR model using a linguistically-biased weak learner, \(f_{B}\). The process begins with training \(f_{B}\) to learn linguistic biases present in a dataset. Then, a linguistically robust model, \(f_{R}\), is trained based on the confidence of \(f_{B}\) (which approximates the intensity of linguistic biases in input) and the prediction accuracy of \(f_{R}\). This approach has two purposes: firstly, \(f_{B}\) guides \(f_{R}\) to improve its robustness by learning from the identified biases of \(f_{B}\). Secondly, \(f_{B}\) can adjust the weights of training examples by prioritizing those that \(f_{R}\) fails to predict, which effectively refines the training focus of \(f_{R}\) toward more challenging examples.

Bi-Encoder Architecture:We consider a standard bi-encoder architecture with a query encoder \(f_{q}\) and a document encoder \(f_{d}\)Khattab and Zaharia (2020); Karpukhin et al. (2020); Xiong et al. (2020).

Figure 2: Architecture of EqualizeIR for mitigating linguistic biases in IR models. (a) Training process: first, a linguistically biased IR model \(f_{B}\) is trained. Then, we freeze the parameters of \(f_{B}\) to train a target, linguistically robust IR model \(f_{R}\) by taking the product of logits of \(f_{B}\) and \(f_{R}\). The biased weak learner regularizes the ranking loss of \(f_{R}\) using its learned linguistic biases. (b): Examples showing that the ensemble approach effectively moderates prediction probabilities to avoid learning biases associated with high confidence or moving too heavily toward the biased weak learner. (c): Strategies for developing linguistically biased weak learners.

 

2012; Hofstatter et al., 2021). Given the \(i\)-th batch \({\cal B}_{i}=\{q_{i},d^{+}_{i},d^{-}_{i,1},\ldots,d^{-}_{i,n}\}\), where \(q_{i}\) denotes the query, \(d^{+}_{i}\) denotes a relevant document, and \(d^{-}_{i,j},\forall j\) denote irrelevant documents, we encode them into embeddings \(h_{q_{i}},h_{d^{+}_{i}},h_{d^{-}_{i}}\), and optimize the standard contrastive loss:

\[L=-\text{log}\frac{e^{\text{sim}(h_{q},h_{d^{+}})}}{e^{\text{ sim}(h_{q},h_{d^{+}})}+\sum_{j=1}^{n}e^{\text{sim}(h_{q},h_{d^{-}_{j}})}} \tag{1}\]

### Debiasing with Biased Weak Learner

We first train a linguistically biased weak learner \(f_{B}\) using the bi-encoder architecture to model dataset biases. After training, we freeze \(f_{B}\)'s parameters and use it to train \(f_{R}\). Given an input example \(x_{i}=(q_{i},d_{i})\), we first obtain the logits from the linguistically-biased weak learner \(f_{B}\) and the target linguistically robust model \(f_{R}\):

\[z_{B}=f_{B}(x_{i}),\quad z_{R}=f_{R}(x_{i}). \tag{2}\]

As Figure 2(a) shows, to integrate the knowledge from the linguistically biased weak learner into the training of the target IR model \(f_{R}\), we compute the element-wise product of the two probabilities and normalize it with a softmax function, or more conveniently element-wise addition in log spcae:

\[\log(z_{D})=\sigma\big{(}\alpha\log(z_{B})+\log(z_{R})\big{)}, \tag{3}\]

where \(\alpha\in[0,1]\) is a scaling factor that controls the strength of the effect of the biases detected by \(f_{B}\) on the final output of \(f_{R}\). This adjusted probability \(z_{D}\) is the debiased probability (see the rationale below), which is then used to compute a standard ranking loss, where \(f_{B}\) remains frozen and only the parameters of \(f_{R}\) are updated. This approach encourages \(f_{R}\) to adopt a less linguistically biased stance under the guidance of \(f_{B}\).

We note that the effect of element-wise product can be interpreted from two perspectives: (a) dynamic curriculum: here the importance of training samples within a batch are adaptively re-weighted based on the confidence of \(f_{B}\)'s prediction; and (b): regularization function: here \(f_{B}\) act as regularizer by constraining \(f_{R}\) to avoid excessive confidence in its predictions, particularly for easy samples that it already predicts correctly. Consequently, \(f_{R}\) does not overfit to specific biased patterns within the dataset. Therefore \(f_{B}\) acts as both a guide and guard to make \(f_{R}\) a more robust model against linguistic bias.

This approach effectively refines the training of \(f_{R}\) using the weak learner \(f_{B}\). Figure 2(b) provides several examples of the functionality of \(f_{B}\). In case (1), when \(f_{B}\) confidently makes a correct prediction, \(f_{R}\) is adjusted to increase its confident in the correct label, as the input is likely an easy example. This lowers the loss (compared to \(f_{R}\)'s actual loss), reduces the weight of the example in training of \(f_{R}\), and effectively minimizes the risk of learning biases from the example by \(f_{R}\). In case (2), when \(f_{B}\) confidently makes a wrong prediction, it indicates that the input sample likely contains biases that mislead \(f_{B}\). Here, \(f_{R}\)'s confidence is adjusted to learn from the example by generating a larger than original loss, which encourages the model to adapt to these hard samples.

### Strategies for Developing Biased Learners

Previous findings show that a "weak" model learns and relies on superficial patterns for making predictions Utama et al. (2020); Ghaddar et al. (2021); Sanh et al. (2021); Meissner et al. (2022). We introduce four approaches to obtain a linguistically-biased weak learner (\(f_{B}\)) from both model and data perspectives.

* First, we obtain a biased weak learner by **repeating linguistic constructs**, such as noun phrases, in queries. This approach makes the model more sensitive to complex linguistic structures by amplifying them in queries without changing the semantics.
* Second, we train a **weaker model** with limited capacity to learn complex patterns, making it weaker in terms of predictive power but useful for exposing biases. This weaker model can be either a completely separate model (e.g. TinyBERT Turc et al. (2019)) or a subset of \(f_{R}\)Cheng and Amiri (2024).
* Third, we use the same architecture as the target IR model, but train it with significantly **fewer iterations**, which results in an "undercooked" version that is weaker.
* Finally, we train the model on **less data**, which reduces its ability to generalize and learn deeper patterns.

Each of these weak learners reveal different linguistic biases in data, and provide insights into the biases that \(f_{R}\) needs to overcome. Appendix 4, Figure 5 shows that the above approaches indeed result in linguistically biased \(f_{B}\)s 

Experiments

DatasetsWe use the _test_ sets of four IR datasets form BEIR benchmark Thakur et al. (2021):

* **MS MARCO Nguyen et al. (2016), a passage retrieval dataset with 532k training samples and 43 test queries;**
* **NFCorpus Boteva et al. (2016), a biomedical IR dataset with 110k training samples and 323 test queries,**
* **FIQA-2018 Maia et al. (2018), a question answering dataset with 14k training samples and 648 test queries, and**
* **SciFact Wadden et al. (2020), a scientific fact checking dataset with 920 training samples and 300 test queries.**

IR ModelsWe compare our approach to the following baselines:

* **BM25 Robertson et al. (2009); Manning (2009), which retrieves documents based on lexical similarity;**
* **DPR Karpukhin et al. (2020), a dense retrieval model that compute similarity in embedding space;**
* **ColBERT Khattab and Zaharia (2020), which adopts a delayed and deep interaction of token embeddings of query and document;**
* **Multiview Amiri et al. (2021), a multiview IR approach with data fusion and attention strategies;**
* **RankT5 Zhuang et al. (2023), the Seq2Seq model Raffel et al. (2023);**
* **KernelWhitening Gao et al. (2022), which learns sentence embeddings that disentangles causal and spurious features; and**
* **LC as Rev Weight, which uses linguistic complexity to reversely weight the probability.**

EvaluationFollowing previous works Thakur et al. (2021); Zhuang et al. (2023), we use NDCG@10 as the evaluation metric. We report average (\(\mu,\uparrow\)), standard deviation (\(\sigma,\downarrow\)), and coefficient of variation (\(c_{v}=\frac{\sigma}{\mu},\downarrow\)) of NDCG@10 across all test queries. In addition, we examine models' performance in terms of the linguistic complexity of test examples. A robust model should have high overall performance and low performance variation across the spectrum of linguistic complexity (e.g. easy, medium, hard). Due to the limited space, we only implement EqualizeIR to DPR.

## 4 Main Results

Existing IR models are linguistically biased Figure 3 and Table 1 show that existing IR models are linguistically biased with significant performance fluctuations as the linguistic complexity of query increases, resulting in a disparate performance across different levels of linguistic complexity. On average, BM25, DPR, ColBERT, RankT5, and Multiview have varied performance across queries, with high standard deviation of 0.32, 0.32, 0.43, 0.25 and 0.26. These results highlight the need to mitigate linguistic biases in these models.

EqualizeIR increases average performance and reduces linguistic biasEqualizeIR outperforms BM25, DPR, ColBERT, RankT5, and Multiview by 0.03, 0.15, 0.15, 0.05, and 0.05 absolute points in average NDCG@10 respectively, while also showing smaller standard deviation in NDCG@10 across all test queries. EqualizeIR outperforms baselines in terms of \(c_{v}\) (NDCG@10) by large margins of 0.30, 0.71, 1.19, 0.08, and 0.14 compared to BM25, DPR, ColBERT, RankT5, Multiview respectively.

Different IR models show different linguistic biasesOn NFCorpus, BM25 achieves 0.40 NDCG@10 on linguistically easy examples, while close to zero NDCG@10 on hard examples. Conversely, DPR perform poorly on linguistically easy examples and better on linguistically hard examples. This contrasting results can be attributed to the underlying architectures of the IR models, such as the text encoders and if late interaction is used, and the intrinsic characteristics of the datasets.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Method & \(\mu(\uparrow)\) & \(\sigma(\downarrow)\) & \(c_{v}(\downarrow)\) \\ \hline BM25 & 0.44 & 0.32 & 0.82 \\ ColBERT & 0.29 & 0.43 & 1.71 \\ DPR & 0.29 & 0.32 & 1.23 \\ RankT5 & 0.42 & 0.25 & 0.64 \\ Multiview & 0.42 & 0.26 & 0.66 \\ KernelWhitening & 0.44 & 0.25 & 0.57 \\ LC as Rev Weight & 0.27 & 0.21 & 0.78 \\ \hline EqualizeIR & **0.47** & **0.22** & **0.52** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main results. \(\mu\), \(\sigma\), and \(c_{v}\) denote average performance, standard deviation, and coefficient of various cross test queries. Best performance is in **bold** and second best is underlined. The significance test is shown in Table 3.

 



**Comparison Between Different Biased Models** Figure 5 shows that, as we hypothesized, all four types of weak learners encode substantial linguistic biases. Results in Appendix A Table 5-8 show the comparison between different methods to obtain \(f_{B}\). Overall, different \(f_{B}\) training methods have similar overall performance and performance variation in terms of NDCG@10. We notice that that the "weaker model" and "less data" approaches consistently yield higher NDCG@10 performance, which may indicate that they better capture linguistic biases for \(f_{R}\) to avoid. In contrast, the "repeating linguistic constructs" and "fewer iterations" strategies do not produce a good biased learner. This result could be attributed to the models potential overemphasis on specific linguistic features or lack of learning discriminative patterns from data, while overshadowing other aspects that may contribute to bias and resulting in a less effective bias detection. In addition, the "weaker model" and "less data" approaches may capture a broader type of biases, including implicit ones, which makes them more flexible and practical. Using a less capable model as \(f_{B}\) leads to the highest overall performance, smallest performance deviation and variation. Using less data has a slightly lower overall performance and higher performance deviation. This comparison highlights that different \(f_{B}\)s exhibit different linguistic biases and result in varying performances of \(f_{R}\).

## 5 Related Work

Information RetrievalDPR Karpukhin et al. (2020) and ColBERT Khattab and Zaharia (2020) are earlier works of dense retrieval, where similarity is computed in high-dimensional embedding space. Although effective, Faggioli et al. (2024) prove that operating in query-specific subspaces can improve the performance and efficiency of dense retrieval models. Recently, more attention has been paid to adapting Large Language Models (LLMs) to information retrieval Guo et al. (2024); Xu et al. (2024); Borges et al. (2024).

Bias MitigationLi et al. (2022) design an in-batch regularization technique to mitigate the biased performance across different subgroups. Kim et al. (2024) propose to identify semantically relevant query-document pairs to explain why documents are retrieved, and discover that existing IR models show biased performances across different brand name. Ziems et al. (2024) discover that IR models suffer from indexical bias, i.e. the bias resulted by the order of documents, and propose a new metric DUO to evaluate the amount of indexical bias an IR model has. Query performance prediction (QPP) Arabzadeh et al. (2024) studies whether we can predict the IR quality by only looking at the query itself without additional information. On other tasks, prior works have discussed how biased models or weak learners can be applied to debiasing in vision Cadene et al. (2019), natural language understanding Sanh et al. (2021); Ghaddar et al. (2021); Cheng and Amiri (2024), and speech classification tasks Cheng et al. (2024).

## 6 Conclusion

We report that IR models are biased toward linguistic complexity of queries and introduce EqualizeIR, a framework that trains a robust IR model by regularizing it with four types of linguistically-biased weak learners (by amplifying linguistic constructs in queries, using a weaker model with limited capacity, training with fewer iterations to create an underdeveloped model, and training on less data to restrict generalization), to achieve equitable performance across queries of varying linguistic complexity.

Figure 3: NDCG@10 of EqualizeIR and DPR Karpukhin et al. (2020) as linguistic complexity of queries increase. Detailed performance of all baselines is shown in Figure 4 in Appendix A.





### Limitations

Existing definitions of linguistic complexity often have a narrow focus on specific linguistic features, which can result in challenges in comprehensive quantification of linguistic biases. For example, we did not consider linguistic biases related to discourse, pragmatics, morphology and semantics. In addition, our debiasing approach slightly increases complexity of training by requiring a trained biased model. Similar to other debiasing approaches, there's a risk of model overfitting to particular biases the model is trained to address, which may limit its adaptability to generalize to new or unseen biases. Finally, although our approach can be applied to any supervised IR model, we only applied it dense retrieval models, and its performance on other IR models remained underexplored.

## Broader Impact Statement

We present an important issue in existing IR models: they show disparate and biased performance across queries with different levels of linguistic complexity-quantified by lexical and syntactic complexity. This can disproportionately disadvantage queries from users with specific writing style that result in particular types of linguistic complexity. It is important that future research and evaluation protocols in IR accounts for these biases and mitigate them.

## References

* Amiri et al. (2021) Hadi Amiri, Mitra Mohtarami, and Isaac Kohane. 2021. Attentive multiview text representation for differential diagnosis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Online, pp. 1012-1019, Online. Association for Computational Linguistics.
* Arabzadeh et al. (2024) Negar Arabzadeh, Chuan Meng, Mohammad Aliannejadi, and Ebrahim Bagheri. 2024. Query performance prediction: From fundamentals to advanced techniques. In Advances in Information Retrieval: 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24-28, 2024, Proceedings, Part V_, page 381-388, Berlin, Heidelberg. Springer-Verlag.
* Borges et al. (2024) Luis Borges, Rohan Jha, Jamie Callan, and Bruno Martins. 2024. Generalizable tip-of-the-tongue retrieval with llm re-ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, New York, NY, USA. Association for Computing Machinery.
* Boteva et al. (2016) Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval, pages 716-722, Cham. Springer International Publishing.
* Cadene et al. (2019) Remi Cadene, Corentin Dancette, Hedi Ben younes, Matthieu Cord, and Devi Parikh. 2019. Rubi: Reducing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
* Cheng and Amiri (2024) Jiali Cheng and Hadi Amiri. 2024. FairFlow: Mitigating dataset biases through undecided learning for natural language understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21960-21975, Miami, Florida, USA. Association for Computational Linguistics.
* Cheng et al. (2024) Jiali Cheng, Mohamed Elgaar, Nidhi Vakil, and Hadi Amiri. 2024. Cognivoice: Multimodal and multilingual fusion networks for mild cognitive impairment assessment from spontaneous speech. In Interspeech 2024, pages 4308-4312.
* Dai and Callan (2020) Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 1533-1536, New York, NY, USA. Association for Computing Machinery.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Faggioli et al. (2024) Guglielmo Faggioli, Nicola Ferro, Raffaele Perego, and Nicola Tonellotto. 2024. Dimension importance estimation for dense information retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, New York, NY, USA. Association for Computing Machinery.
* Gao et al. (2022) SongYang Gao, Shihan Dou, Qi Zhang, and Xuanjing Huang. 2022. Kernel-whitening: Overcome dataset bias with isotropic sentence embedding. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4112-4122, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Ghaddar et al. (2021) Abbas Ghaddar, Phillippe Langlais, Mehdi Rezagholizadeh, and Ahmad Rashid. 2021. End-to-end self-debiasing framework for robust NLU training.



In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1923-1929, Online. Association for Computational Linguistics.
* Guo et al. (2024) Ping Guo, Yubing Ren, Yue Hu, Yanan Cao, Yunpeng Li, and Heyun Huang. 2024. Steering large language models for cross-lingual information retrieval. In _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '24, page 585-596, New York, NY, USA. Association for Computing Machinery.
* Hofstatter et al. (2021) Sebastian Hofstatter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 113-122.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. _arXiv preprint arXiv:2004.04906_.
* Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_, pages 39-48.
* Kim et al. (2024) Youngwoo Kim, Razieh Rahimi, and James Allan. 2024. Discovering biases in information retrieval models using relevance measures as global explanation. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 19530-19547, Miami, Florida, USA. Association for Computational Linguistics.
* Lee et al. (2021) Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10669-10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lee and Lee (2023) Bruce W. Lee and Jason Lee. 2023. LFTK: Handcrafted features in computational linguistics. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_, pages 1-19, Toronto, Canada. Association for Computational Linguistics.
* Li et al. (2022) Yuantong Li, Xiaokai Wei, Zijian Wang, Shen Wang, Parminder Bhatia, Xiaofei Ma, and Andrew Arnold. 2022. Debiasing neural retrieval via in-batch balancing regularization. In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_, pages 58-66, Seattle, Washington. Association for Computational Linguistics.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Lu (2010) Xiaofei Lu. 2010. Automatic analysis of syntactic complexity in second language writing. _International journal of corpus linguistics_, 15(4):474-496.
* Lu (2012) Xiaofei Lu. 2012. The relationship of lexical richness to the quality of esl learners' oral narratives. _The Modern Language Journal_, 96(2):190-208.
* Maia et al. (2018) Macedo Maia, Siegfried Handschuh, Andre Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW'18 open challenge: Financial opinion mining and question answering. In _Companion Proceedings of the The Web Conference 2018_, WWV '18, page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Manning (2009) Christopher D Manning. 2009. _An introduction to information retrieval_. Cambridge university press.
* Meissner et al. (2022) Johannes Mario Meissner, Saku Sugawara, and Akiko Aizawa. 2022. Debising masks: A new framework for shortcut mitigation in NLU. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7607-7613, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. _choice_, 2640:660.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoi Steiner, Lu Fang, Junje Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Raffel et al. (2023) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends(r) in Information Retrieval_, 3(4):333-389.
* Sanh et al. (2021) Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. 2021. Learning from others' mistakes: Avoiding dataset biases without modeling them. In _International Conference on Learning Representations_ 

Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
* Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. _arXiv preprint arXiv:1908.08962_.
* Utama et al. (2020) Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Towards debiasing NLU models from unknown biases. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7597-7610, Online. Association for Computational Linguistics.
* Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7534-7550, Online. Association for Computational Linguistics.
* Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _International Conference on Learning Representations_.
* Xu et al. (2024) Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. 2024. Openp5: An open-source platform for developing, training, and evaluating llvm-based recommender systems. In _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '24, page 386-394, New York, NY, USA. Association for Computing Machinery.
* Zhao et al. (2021) Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021. SPARTA: Efficient open-domain question answering via sparse transformer matching retrieval. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 565-575, Online. Association for Computational Linguistics.
* Zhuang et al. (2023) Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '23, page 2308-2313, New York, NY, USA. Association for Computing Machinery.
* Ziems et al. (2024) Caleb Ziems, William Held, Jane Dwivedi-Yu, and Diyi Yang. 2024. Measuring and addressing indexical bias in information retrieval. In _Findings of the Association for Computational Linguistics: ACL 2024_, pages 12860-12877, Bangok, Thailand. Association for Computational Linguistics.

 

Addition Results

We present the performance with respect to linguistic complexity in Figure 4 and the performance on each dataset in Table 2. Overall, the results show that existing IR models are linguistically biased, showing significant performance fluctuations as the linguistic complexity of query changes. Table 5-8 compares the performances between different methods to obtain \(f_{B}\).

## Appendix B Linguistic Complexity

Table 4 presents the 45 linguistic complexity measurements in our study. For the full description of these metrics, see Lu (2010, 2012); Lee and Lee (2023). We provide a brief description of a few indices as an example: **Type-Token Ratio, TTR** is the ratio of unique words in the text. **D-measure** is a modification to TTR that accounts for text length. * **Variation** indicates variations in lexical words such as nouns, verbs, adjectives, and adverbs. The **Mean Length of T-Units** is the average length of T-units in text. A T-unit is defined as a minimal terminable unit, essentially an independent clause and all its subordinate clauses. It provides insight into the syntactic complexity by measuring how elaborate the clauses are on average.

## Appendix C Implementation Details

We use PyTorch Paszke et al. (2019) and BEIR Thakur et al. (2021) to implement our approach. For DPR and ColBERT, we use BERTbase Devlin et al. (2019) as the encoders. For \(f_{B}\) trained with less data, we randomly take 20% of the original training data to train \(f_{B}\). For \(f_{B}\) trained with less capable model, we use BERTTiny Turc et al. (2019) as the encoder. For 

trained with less time, we train it for 20% of the original training time. All methods are trained with AdamW Loshchilov and Hutter (2019) optimizer with a learning rate of \(1e-5\). We tune \(\alpha\) on validation sets and find choosing \(\alpha=0.1\) yields best performance consisitently across datasets.

## Appendix A Proofs of Results

In this appendix, we provide proofs of results from our experiments. The first is a simple example of the proposed method in Section 4.3. In particular, we show that the proposed method outperforms the other methods by using different strategies. For each experiment, we use the same strategy as the one used in Section 4.3.

### Comparison of Different Strategies for Developing linguistically biased models in terms of NDCG@10

\begin{table}
\begin{tabular}{l|c|c|c} \hline \hline Dataset & \(\mu(\uparrow)\) & \(\sigma(\downarrow)\) & \(c_{v}(\downarrow)\) \\ \hline Less data & 0.44 & 0.23 & 0.52 \\ Less capable model & **0.48** & **0.20** & **0.42** \\ Less trained & 0.42 & 0.26 & 0.62 \\ Linguistically biased data & 0.42 & 0.25 & 0.60 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of different strategies for developing linguistically biased models in terms of NDCG@10 on MS MARCO. Best performance is in **bold** and second post is underlined.

Figure 4: Performance in NDCG@10 as linguistic complexity of queries increase.

Figure 5: Performance of \(f_{B}\) obtained by four different strategies, which are highly linguistically biased.