Language Bias in Multilingual Information Retrieval: The Nature of the Beast and Mitigation Methods
Authors: Jinrui Yang, Fan Jiang, Timothy Baldwin


Reviewer paper summaries: The paper studies the language bias in multilingual information retrieval systems, by benchmarking methods on their performance and fairness.
Summary of strengths: 1. The paper is well-written and easy to follow.
2. The studied topic is interesting.
Summary of weaknesses: 1. The main concern from my side is the lack of analysis or interpretation on the experiment results. In section 4.1, the author simply list the comparison of scores in performance and fairness evaluation, without giving even a primitive reasoning. For example, in line 364-365, the paper state "low-resource languages (MT and GA) exhibit lower language fairness than high-resource languages", any explanation on this phenomena? 

2. The technical contribution of the paper is weak. The paper propose a metric for measuring fairness in retrieval methods and benchmark it across different languages. Since the idea is simple, more comprehensive experiments are expected. For example, more advanced DPR methods, like Contriever (Izacard et al., 2022) or ANCE (Xiong et al., 2021) can be considered. Also, deeper analysis on the results should be included. Potential examples include trade-off between fairness and performance, analyze the origin of the bias (from LMs or from the retrieval training), etc.

2. Even though in general the writing is easy to read, there are still important details that are missing, especially for readers that are not familiar with the details. For example, in line 88 of the introduction, the paper firstly mentions the method BM25, without any explanation or description. Finally in section 4.1.2, the paper briefly explain BM25. Another example is in sections of Performance Findings and Fairness Findings, can the paper better indicate which languages are more low-resource, and which are not?
Comments, suggestions and typos: Please see above
Overall assessment scores: 2.5