AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
Authors: Jianlyu Chen, Nan Wang, Chaofan Li, Bo Wang, Shitao Xiao, Han Xiao, Hao Liao, Defu Lian, Zheng Liu
Accepted at: acl25

AIR-BENCH: Automated Heterogeneous Information Retrieval Benchmark

Jianlyu Chen1,2,6  Nan Wang3  Chaofan Li2,4  Bo Wang3
Shitao Xiao2  Han Xiao3  Hao Liao5*  Defu Lian1,6*  Zheng Liu2,7*
1University of Science and Technology of China
2Beijing Academy of Artificial Intelligence  3Jina AI
4Beijing University of Posts and Telecommunications  5Shenzhen University
6State Key Laboratory of Cognitive Intelligence  7Hong Kong Polytechnic University
chenjianlv@mail.ustc.edu.cn  research@jina.ai, haoliao@szu.edu.cn
liandefu@ustc.edu.cn  zhengliu1026@gmail.com

Abstract
Evaluation plays a crucial role in the advancement of information retrieval (IR) models. However, current benchmarks, which are based on predefined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently. To address this challenge, we propose the Automated Heterogeneous Information Retrieval Benchmark (AIR-BENCH). AIR-BENCH is distinguished by three key features: 1) Automated. The testing data in AIR-BENCH is automatically generated by large language models (LLMs) without human intervention. 2) Heterogeneous. The testing data in AIR-BENCH is generated with respect to diverse tasks, domains and languages. 3) Dynamic. The domains and languages covered by AIR-BENCH are constantly augmented to provide an increasingly comprehensive evaluation benchmark for community developers. We develop a reliable and robust data generation pipeline to automatically create diverse and high-quality evaluation datasets based on real-world corpora. Our findings demonstrate that the generated testing data in AIR-BENCH aligns well with human-labeled testing data, making AIR-BENCH a dependable benchmark for evaluating IR models. The resources in AIR-BENCH are publicly available at https://github.com/AIR-Bench/AIR-Bench.

1 Introduction
As information retrieval (IR) models grow in complexity and capability, the need for sophisticated evaluation techniques becomes increasingly critical. In recent years, a series of milestone works have significantly advanced the field by introducing comprehensive evaluation datasets and benchmarks. Early contributions to IR evaluation include MS MARCO (Bajaj et al., 2016) and Natural Questions (Kwiatkowski et al., 2019), both designed for open-domain question answering (QA) tasks in English. These datasets have been crucial in driving progress in monolingual IR systems and establishing baseline performance metrics. Recognizing the importance of multilingual information retrieval, researchers developed Mr.TyDi (Zhang et al., 2021) and MIRACL (Zhang et al., 2023). These datasets cover ad hoc retrieval tasks in 11 and 18 languages, respectively, facilitating the development and evaluation of IR systems capable of handling diverse linguistic contexts. More recently, the focus has shifted towards creating general-domain, zero-shot IR benchmarks. BEIR (Thakur et al., 2021) and MTEB (Muennighoff et al., 2023) represent this trend by aggregating multiple existing datasets from diverse tasks and domains. These comprehensive benchmarks allow researchers to evaluate the generalization capabilities of IR models across various scenarios without task-specific fine-tuning.

Despite their contributions, existing benchmarks are constrained to pre-defined domains and rely heavily on human-labeled data, making it challenging to efficiently address evaluation needs in emerging domains. With the emergence of powerful large language models (LLMs), several studies have explored their application for retrieval evaluation in retrieval-augmented generation (RAG) systems (Es et al., 2024; Saad-Falcon et al., 2024; Salemi and Zamani, 2024), presenting a promising solution to this challenge. However, a comprehensive IR benchmark that addresses this limitation remains insufficiently developed.

In this work, we present the Automated Heterogeneous Information Retrieval Benchmark (AIR-BENCH), which is characterized by three features:

1. **Automated**: We develop a comprehensive data generation pipeline to automatically produce diverse and high-quality testing data with large language models (LLMs). Therefore, it
is able to instantly support the evaluation of new domains both cost-effectively and efficiently. Besides, the new testing data is almost impossible to be covered by the training sets of any existing retrievers.

2. Heterogeneous: AIR-BENCH is designed to be a heterogeneous IR benchmark including diverse tasks, domains and languages. It currently covers 2 tasks, 9 domains, and 13 languages, including a total of 69 datasets. This extensive coverage enables thorough evaluation across diverse scenarios, potentially accelerating advancements in IR technology for both established and emerging domains.

3. Dynamic: The tasks, domains and languages covered by AIR-BENCH are planed to be augmented on regular basis. There are currently two distinct versions, 24.04 and 24.05, with more anticipated in the future. We hope AIR-BENCH is able to provide an increasingly comprehensive evaluation benchmark for community developers.

These features form the foundation of our proposed benchmark and directly address the limitation in existing benchmarks for information retrieval systems. To further elucidate the impact and scope of our work, we summarize our main contributions as follows: 1) We introduce AIR-BENCH, a new information retrieval benchmark highlighted by new features: automated, heterogeneous and dynamic. 2) We demonstrate that our data generation pipeline is able to produce diverse and high-quality testing data highly consistent with human-labeled testing data, making AIR-BENCH a dependable benchmark for evaluating IR models. 3) Additionally, we develop and release software tools enabling community developers to evaluate any IR model using AIR-BENCH. To foster collaboration and progress in the field, we establish and maintain a public leaderboard1 to track and compare model performance across the community. These contributions collectively advance the field of information retrieval by providing a versatile, dynamic, and comprehensive evaluation framework.

2 Benchmark Construction

The entire data generation pipeline of AIR-BENCH consists of three stages: 1) Corpora preparation, 2) Candidate generation, and 3) Quality control.

2.1 Preliminary

AIR-BENCH focuses on the evaluation of information retrieval. The information retrieval task can be formulated as: Given a query \( q \), retrieve a ranked list of \( n \) most relevant documents \( \mathcal{L} = [d_1, d_2, \cdots, d_n] \) from the corpus \( \mathcal{D} = \{d_i\}_{i=1}^{\mathcal{D}} \).

To clarify the subsequent explanation, Table 1 lists the symbols that appear in this section along with their corresponding meanings for reference.

2.2 Corpora Preparation

As shown in Figure 1, the first stage involves preparing diverse corpora. Specifically, given a task, we collect real-world datasets from diverse domains and languages, and apply distinct pre-processing strategies to the raw datasets based on the task requirements (see Appendix A.1 for more details).

The corpus prepared in this stage is denoted as \( \mathcal{D}_0 = \{d_i\}_{i=1}^{n_0} \), including \( n_0 \) documents.

![The three-stage data generation pipeline of AIR-BENCH.](page_184_120_1077_420.png)

Figure 1: The three-stage data generation pipeline of AIR-BENCH.
<table>
  <tr>
    <th>Symbol</th>
    <th>Meaning</th>
    <th>Symbol</th>
    <th>Meaning</th>
  </tr>
  <tr>
    <td>q</td>
    <td>query</td>
    <td>Q</td>
    <td>queries set</td>
  </tr>
  <tr>
    <td>d</td>
    <td>document</td>
    <td>\( d^+ / d^- \)</td>
    <td>positive/negative document</td>
  </tr>
  <tr>
    <td>l</td>
    <td>relevance label</td>
    <td>n, m</td>
    <td>number</td>
  </tr>
  <tr>
    <td>D</td>
    <td>documents set</td>
    <td>\( D_+ / D_- \)</td>
    <td>positive/negative documents set</td>
  </tr>
  <tr>
    <td>R</td>
    <td>relevance labels set</td>
    <td>\( R_+ / R_- \)</td>
    <td>positive/negative relevance labels set</td>
  </tr>
  <tr>
    <td>L</td>
    <td>documents list</td>
    <td>\( \mathcal{M} \)</td>
    <td>re-ranking model</td>
  </tr>
</table>

Table 1: Corresponding meanings for the symbols appearing in this section.

2.3 Candidate Generation

The candidate data for a retrieval dataset consists of three components: corpus, queries and qrels. After preparing the corpus in the initial stage, the candidate generation stage produces the remaining two components of the dataset: queries and qrels.

Based on the corpus, the candidate generation process is iteratively executed in a loop. As shown in Figure 1, the generation process can be summarized as the following steps: 1) Sample one document from the raw corpus as the positive document \( d_i^+ \). 2) Prompt LLM to generate the characters who might find the document useful. 3) Prompt LLM to generate the scenarios in which the character might find the document useful. 4) Prompt LLM to generate the query \( ori_{-q_i} \) based on the specific character and scenario. To diversify the generated queries, we consider the following attributes when designing the prompt: query length, query type, information-based type, and expression style. 5) Prompt LLM to rewrite the generated query for multiple times to try to avoid the duplicated tokens as in the raw corpus, and finally get query \( q_i \). 6) Prompt LLM to generate some hard negative documents \( \{d_i^-(j)\}_{j=1}^{m_i} \) based on the generated query \( q_i \) and the positive document \( d_i^+ \). 7) Repeat Step 1-6. Considering both simplicity and the absence of examples in a new domain, the above prompting strategies are all zero-shot. For more details, please refer to Appendix A.2.

After repeating n times of the above loop, we get the queries set Q, the positive documents set \( D_+ \), the hard negative documents set \( D_- \), the corpus \( D = D_0 \cup D_+ \cup D_- \), the positive relevance labels set \( R_+ \), and the negative relevance labels set \( R_- \).

<table>
  <tr>
    <th></th>
    <th>\( l_j \) is pos</th>
    <th>\( l_j \) is neg</th>
  </tr>
  <tr>
    <td>Type 1: \( d_j \in D_+ \)</td>
    <td>-</td>
    <td>*</td>
  </tr>
  <tr>
    <td>Type 2: \( d_j \in D_- \)</td>
    <td>discard \( d_j \) from \( D_- \), remove \( (q_i, d_j, 0) \) from \( R_- \)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Type 3: \( d_j \in D_0 \setminus D_+ \)</td>
    <td>add \( d_j \) to \( D_+ \), add \( (q_i, d_j, 1) \) to \( R_+ \)</td>
    <td>-</td>
  </tr>
</table>

Table 2: Specifications of different quality control strategies based on the type of document \( d_j \) and the relevance label \( l_j \) of \( (q_i, d_j) \). Type 1 means that \( d_j \) is the original positive document, Type 2 means that \( d_j \) is the generated hard negative document, and Type 3 means that \( (q_i, d_j) \) does not have a relevance label in the second stage. "*": Skip. "*": If the type of \( d_j \) is Type 1, \( l_j \) must be positive since we have filtered low-quality queries.

2.4 Quality Control

In this stage, we design comprehensive quality control strategies to enhance the quality of the generated dataset. As shown in Figure 1, the quality control process can be summarized as two parts.

Filter low-quality queries. Since all of the queries in the candidate data are generated by LLM, there are potential low-quality queries. To improve the quality of generated queries, we utilize LLM to access the relevance between the query \( q_i \) and the positive document \( d_i^+ \). If the LLM prediction is negative, indicating that \( q_i \) is a low-quality query, we discard \( q_i \) from Q and remove the relevance labels \( \{(q_i, *, *)\} \) from \( R_+ \) and \( R_- \). For details on how we utilize LLM to label the relevance, please refer to Appendix A.3.

Correct the false relevance labels. The false relevant labels comprise two types of documents: the first type includes the generated hard negative documents, and the second type consists of relevant documents that were overlooked in the corpus. Given a query \( q_i \), we design a three-step pipeline to correct the false relevance labels. 1) Recall with embedding model. Use the embedding model to search top-1000 relevant documents \( \mathcal{L}_{recall} = [d_1, \cdots, d_{1000}] \) from the corpus for \( q_i \). 2) Pre-label with re-ranking models. Use multiple re-ranking models to re-rank \( \mathcal{L}_{recall} \). We pre-label each document \( d_j \) according to their ranking \( r_j(\mathcal{M}) \) in the re-ranked top-1000 relevant documents \( \mathcal{L}_{rerank}(\mathcal{M}) \) from the re-ranking model \( \mathcal{M} \). Specifically, if \( r_j(\mathcal{M}) \) is higher than the predetermined threshold, the label \( l_j(\mathcal{M}) \) for \( d_j \) from \( \mathcal{M} \) is positive. If more than half of re-ranking models label \( d_j \) as positive, we pre-label \( d_j \) as positive, otherwise we pre-label \( d_j \) as negative. After this step, each document \( d_j \) in \( \mathcal{L}_{recall} \) has a preliminary
label \( pre_{-}l_j \). 3) *Label with LLM*. In this step, we also utilize LLM to access the relevance between \( q_i \) and the documents \( \{d_j\}_{j=1}^{m_i} \) that are pre-labeled as positive in the last step. The prediction from LLM is denoted as \( l_j \). As shown in Table 2, we categorize \( d_j \) into three types, and take different actions by the type of \( d_j \) and \( l_j \). For details on how we select the embedding model and multiple re-ranking models, and set the predetermined threshold for pre-labeling, please refer to Appendix A.3.

After executing the above quality control process for each query, we get the new queries set \( Q' \), the new positive documents set \( D'_+ \), the new hard negative documents set \( D'_- \), the new corpus \( D' = D_0 \cup D'_+ \cup D'_- \), and the new relevance labels set \( R' = R'_+ \cup R'_- \), which form the final dataset.

2.5 Design Motivations

We elaborate the design motivations of the data generation pipeline of AIR-BENCH as follows.

*Reliance on real-world corpora.* Real-world corpora are usually diverse and available. Generating testing data based on real-world corpora not only closely aligns with real-world scenarios, but also significantly reduces the generation cost.

*Generation of characters and scenarios.* First, this step brings more transparency and interpretability on how a query is generated, compared to the naive method which directly prompts LLMs for query generation. Second, the generation of character and scenario also leads to higher diversity of queries, which contributes to the comprehensiveness of evaluation.

*Query Rewriting.* Through rewriting, queries are transformed into different forms while retaining equivalent semantics, which significantly increases the difficulty of retrieval tasks.

*Generation of hard negatives.* Similar to the introduction of query rewriting, this step increases the hardness of evaluation.

*Quality Control.* This step helps to remove low-quality queries and correct false relevance labels. Similar operations were also conducted in previous benchmark, e.g., the relevance assessment phase in MIRACL (Zhang et al., 2023).

<table>
  <tr>
    <th rowspan="2">Task →</th>
    <th colspan="2">QA</th>
    <th colspan="2">Long-Doc</th>
  </tr>
  <tr>
    <th>dev</th>
    <th>test</th>
    <th>dev</th>
    <th>test</th>
  </tr>
  <tr>
    <td>Spli t → # of datasets →</td>
    <td>54</td>
    <td>53</td>
    <td>4</td>
    <td>11</td>
  </tr>
  <tr>
    <th colspan="5">Query Type</th>
  </tr>
  <tr>
    <td>HOW</td>
    <td>16.4%</td>
    <td>17.6%</td>
    <td>17.0%</td>
    <td>19.7%</td>
  </tr>
  <tr>
    <td>WHAT</td>
    <td>34.1%</td>
    <td>30.9%</td>
    <td>28.5%</td>
    <td>33.1%</td>
  </tr>
  <tr>
    <td>WHEN</td>
    <td>4.8%</td>
    <td>5.9%</td>
    <td>1.1%</td>
    <td>1.2%</td>
  </tr>
  <tr>
    <td>WHERE</td>
    <td>3.0%</td>
    <td>3.2%</td>
    <td>0.9%</td>
    <td>0.8%</td>
  </tr>
  <tr>
    <td>WHICH</td>
    <td>4.7%</td>
    <td>5.3%</td>
    <td>4.4%</td>
    <td>4.0%</td>
  </tr>
  <tr>
    <td>WHO</td>
    <td>7.3%</td>
    <td>7.6%</td>
    <td>8.7%</td>
    <td>4.0%</td>
  </tr>
  <tr>
    <td>WHY</td>
    <td>3.2%</td>
    <td>3.2%</td>
    <td>6.4%</td>
    <td>3.8%</td>
  </tr>
  <tr>
    <td>YES/NO</td>
    <td>4.2%</td>
    <td>4.1%</td>
    <td>5.5%</td>
    <td>6.9%</td>
  </tr>
  <tr>
    <td>CLAIM</td>
    <td>22.2%</td>
    <td>22.1%</td>
    <td>27.5%</td>
    <td>26.3%</td>
  </tr>
  <tr>
    <td>OTHERS</td>
    <td>0.1%</td>
    <td>0.1%</td>
    <td>0%</td>
    <td>0.2%</td>
  </tr>
</table>

Table 3: The type distribution of queries in each split for each task in AIR-BENCH 24.05.

3 The AIR-BENCH Benchmark

3.1 Overview

LLM for Generation. We use powerful GPT-4\(^2\) (Achiam et al., 2023) as the LLM through the generation pipeline. When prompting GPT-4, we set the sampling temperature to 1.0 to encourage more diversity.

Tasks. AIR-BENCH currently covers two retrieval tasks to meet the evaluation needs in different scenarios: 1) QA. This task focuses on the classic question answering scenarios (Voorhees et al., 1999), where the corpus consists of a large collection of documents. Following BEIR (Thakur
et al., 2021), we utilize nDCG@10 as the main metric for the QA task. 2) Long-Doc. This task is closely related with today’s LLM and RAG applications (Lewis et al., 2020), where the corpus consists of chunks from a lengthy document. Given that the proportion of positive documents precedes the ranking of positive documents in the RAG scenario, we utilize Recall@10 as the main metric for the Long-Doc task. AIR-BENCH will be extended to cover more retrieval tasks in the future.

Datasets. As shown in Figure 2, AIR-BENCH currently has two distinct versions, 24.04 and 24.05, where the latest version 24.05 consists of a total of 69 datasets, covering 9 domains3 and 13 languages4 on two retrieval tasks. We hope to incorporate more domains and languages in the future version to provide an increasingly comprehensive evaluation benchmark for community developers. The specifications of all datasets in AIR-BENCH 24.05 are available in Table 17, Table 18, Table 19, Table 20, and Table 21. More details are available in Appendix B.1.

Software. We develop the AIR-BENCH software5 to facilitate the evaluation of any information retrieval methods. Besides, we maintain a Hugging Face leaderboard6 with all datasets and models. For more details, please refer to Appendix C.

3.2 Diversity Analysis

To analyze the query type diversity of AIR-BENCH, we utilize GPT-4o7 (Achiam et al., 2023) as labeler to label the type of the generated queries. Specifically, given a query, we prompt GPT-4o to select the most suitable type for the query from the optional types. The statistics are grouped by tasks and splits in Table 3. Based on the results, we can make the following observations. Firstly, both the QA and Long-Doc tasks have the highest frequency of WHAT queries, followed by CLAIM queries as the second most common, and HOW queries as the third. Additionally, the QA task exhibits a more balanced distribution of the other query types, whereas the Long-Doc task shows a lower frequency of WHEN queries and WHERE queries. Lastly, a small number of queries are classified as OTHERS, reflecting the diverse types of queries present in AIR-BENCH to some extent. Further diversity analysis of AIR-BENCH is presented in Appendix B.3.

<table>
  <tr>
    <th></th>
    <th>#corpus</th>
    <th>#queries</th>
    <th>#positives</th>
  </tr>
  <tr>
    <td>R-MSMARCO</td>
    <td>8,841,823</td>
    <td>6,980</td>
    <td>7,437</td>
  </tr>
  <tr>
    <td>G-MSMARCO</td>
    <td>8,872,840</td>
    <td>6,319</td>
    <td>31,447</td>
  </tr>
  <tr>
    <td>w/o quality control</td>
    <td>8,878,865</td>
    <td>7,429</td>
    <td>7,429</td>
  </tr>
</table>

Table 4: Comparison of R-MSMARCO and G-MSMARCO. R-MSMARCO is the raw MS MARCO passage ranking dataset (Bajaj et al., 2016), and G-MSMARCO is the generated MS MARCO passage ranking dataset in AIR-BENCH. #corpus represents the number of documents in the corpus, #queries represents the number of queries, and #positives represents the number of positive relevance labels. Since there are some generated hard negative documents in the corpus of G-MSMARCO, it is slightly larger than the corpus of R-MSMARCO.

3.3 Positioning of AIR-BENCH

We analyze the positioning of AIR-BENCH in this section to highlight extra values of AIR-BENCH over existing benchmarks. Firstly, as a diverse and continually evolving benchmark, AIR-BENCH enables comprehensive evaluation of existing retrievers while addressing the saturation issue that many popular benchmarks (e.g., MTEB (Muennighoff et al., 2023) / C-MTEB (Xiao et al., 2024)) face due to intensive in-domain fine-tuning. Furthermore, as an automated evaluation toolkit, AIR-BENCH supports ad-hoc evaluations for emerging domain-specific retrieval applications. We also provide experimental results in Section 4.2.

4 Experiment

In this section, we aim to address the following research questions:

RQ1: How well does the LLM-generated testing data in AIR-BENCH align with the human-labeled testing data?

RQ2: What additional evaluation functionalities does AIR-BENCH offer compared to MTEB/BEIR?

RQ3: How effectively can AIR-BENCH distinguish the capabilities of distinct IR models?

4.1 Consistency Analysis (RQ1)

Thomas et al. (2024) have demonstrated that LLMs like OpenAI’s GPT-4 are as accurate as human labelers when generating high-quality golden labels
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="2">R-MSMARCO</th>
    <th colspan="2">G-MSMARCO</th>
  </tr>
  <tr>
    <th>nDCG@10</th>
    <th>Rank</th>
    <th>w/ quality control nDCG@10</th>
    <th>Rank</th>
    <th>w/o quality control nDCG@10</th>
    <th>Rank</th>
  </tr>
  <tr>
    <td>repllama-v1-7b-lora-passage (Ma et al., 2023)</td>
    <td>6.74B</td>
    <td>48.000</td>
    <td>1</td>
    <td>59.625</td>
    <td>1</td>
    <td>33.434</td>
    <td>2</td>
  </tr>
  <tr>
    <td>e5-large-v2 (Wang et al., 2022b)</td>
    <td>335M</td>
    <td>45.232</td>
    <td>2</td>
    <td>55.260</td>
    <td>4</td>
    <td>32.581</td>
    <td>5</td>
  </tr>
  <tr>
    <td>multilingual-e5-large (Wang et al., 2024)</td>
    <td>560M</td>
    <td>45.119</td>
    <td>3</td>
    <td>54.431</td>
    <td>5</td>
    <td>32.099</td>
    <td>6</td>
  </tr>
  <tr>
    <td>multilingual-e5-base (Wang et al., 2024)</td>
    <td>278M</td>
    <td>44.130</td>
    <td>4</td>
    <td>52.581</td>
    <td>8</td>
    <td>30.870</td>
    <td>8</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5 (Wang et al., 2024)</td>
    <td>335M</td>
    <td>44.122</td>
    <td>5</td>
    <td>55.513</td>
    <td>3</td>
    <td>33.119</td>
    <td>4</td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct (Wang et al., 2023)</td>
    <td>7.11B</td>
    <td>43.787</td>
    <td>6</td>
    <td>59.015</td>
    <td>2</td>
    <td>36.186</td>
    <td>1</td>
  </tr>
  <tr>
    <td>e5-small-v2 (Wang et al., 2022b)</td>
    <td>33.4M</td>
    <td>43.104</td>
    <td>7</td>
    <td>51.456</td>
    <td>10</td>
    <td>30.471</td>
    <td>10</td>
  </tr>
  <tr>
    <td>e5-base-v2 (Wang et al., 2022b)</td>
    <td>109M</td>
    <td>43.056</td>
    <td>8</td>
    <td>51.438</td>
    <td>11</td>
    <td>30.411</td>
    <td>11</td>
  </tr>
  <tr>
    <td>bge-small-en-v1.5 (Xiao et al., 2024)</td>
    <td>33.4M</td>
    <td>42.553</td>
    <td>9</td>
    <td>51.528</td>
    <td>9</td>
    <td>30.155</td>
    <td>13</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5 (Xiao et al., 2024)</td>
    <td>109M</td>
    <td>42.388</td>
    <td>10</td>
    <td>54.292</td>
    <td>7</td>
    <td>32.067</td>
    <td>7</td>
  </tr>
  <tr>
    <td>multilingual-e5-small (Wang et al., 2024)</td>
    <td>118M</td>
    <td>42.253</td>
    <td>11</td>
    <td>47.989</td>
    <td>14</td>
    <td>28.579</td>
    <td>15</td>
  </tr>
  <tr>
    <td>simlm-base-msmarco-finetuned (Wang et al., 2022a)</td>
    <td>110M</td>
    <td>41.675</td>
    <td>12</td>
    <td>48.102</td>
    <td>13</td>
    <td>30.548</td>
    <td>9</td>
  </tr>
  <tr>
    <td>jina-embeddings-v3 (Sturua et al., 2024)</td>
    <td>572M</td>
    <td>39.787</td>
    <td>13</td>
    <td>51.098</td>
    <td>12</td>
    <td>30.297</td>
    <td>12</td>
  </tr>
  <tr>
    <td>bge-m3 (Chen et al., 2024b)</td>
    <td>568M</td>
    <td>39.565</td>
    <td>14</td>
    <td>54.404</td>
    <td>6</td>
    <td>33.286</td>
    <td>3</td>
  </tr>
  <tr>
    <td>contriever-msmarco (Izacard et al., 2022)</td>
    <td>109M</td>
    <td>36.570</td>
    <td>15</td>
    <td>47.127</td>
    <td>15</td>
    <td>29.231</td>
    <td>14</td>
  </tr>
  <tr>
    <td>msmarco-roberta-base-ance-firstp (Xiong et al., 2021)</td>
    <td>125M</td>
    <td>33.637</td>
    <td>16</td>
    <td>42.107</td>
    <td>16</td>
    <td>24.798</td>
    <td>16</td>
  </tr>
  <tr>
    <td>BM25 (Robertson and Zaragoza, 2009)</td>
    <td>-</td>
    <td>26.211</td>
    <td>17</td>
    <td>34.155</td>
    <td>17</td>
    <td>22.582</td>
    <td>17</td>
  </tr>
  <tr>
    <th colspan="2">Spearman Rank Correlation Coefficient (P-value)</th>
    <th colspan="2"></th>
    <th colspan="2">0.8211 (5e-5)</th>
    <th colspan="2">0.6912 (2e-3)</th>
  </tr>
</table>

Table 5: The consistency between the testing data generated by the pipeline of AIR-BENCH and the human-labeled testing data. We use the MS MARCO passage ranking dataset (Bajaj et al., 2016) to evaluate the consistency. For the public link of the models appearing in the table, please refer to Table 15.

for search system. Based on this conclusion, we attempt to examine how well the LLM-generated testing data aligns with human-labeled testing data.

Setup. We utilize MS MARCO passage ranking dataset (Bajaj et al., 2016) to access the consistency between the LLM-generated testing data in AIR-BENCH and human-labeled testing data. Specifically, we use the positive passages in the raw MS MARCO dev split as the candidate positives (\( d_i^+ \) in Stage 2, refer to Section 2.3), and finally generate a new MS MARCO passage ranking dataset. The raw MS MARCO passage ranking dataset (dev split) is denoted as \( R\text{-}MSMARCO \), and the new generated MS MARCO passage ranking dataset is denoted as \( G\text{-}MSMARCO \). Table 4 shows the comparison of R-MSMARCO and G-MSMARCO.

To examine how well G-MSMARCO aligns with R-MSMARCO, we evaluate 17 IR models on R-MSMARCO and G-MSMARCO using nDCG@10, and compute the Spearman rank correlation coefficient (Spearman, 1961) between their rankings on R-MSMARCO and G-MSMARCO as the consistency metric.

Main Results. As shown in Table 5, the Spearman rank correlation coefficient is 0.8211 with a p-value of 5e-5, indicating that the LLM-generated testing data aligns well with the human-labeled testing data. Overall, each model achieves higher nDCG@10 on G-MSMARCO than on R-MSMARCO. This can be largely attributed to more comprehensive quality control strategy of AIR-BENCH, which results in more positives for each query (see Table 4).

Ablation of Quality Control. To demonstrate the necessity of the quality control stage in the data generation pipeline of AIR-BENCH, we also evaluate the consistency between R-MSMARCO and G-MSMARCO generated without quality control. As shown in Table 5, the correlation coefficient shows a significant degradation (0.8211 → 0.6912). Besides, the nDCG@10 of each model on G-MSMARCO without quality control also has a huge drop, due to some low-quality queries and very limited positives (see Table 4, there are 1,110 low-quality queries and only 7,429 positives). Therefore, quality control stage is necessary to ensure the data generation pipeline a reliable data generation pipeline.

Robustness of Consistency. To investigate the robustness of consistency, we simulate 30 generation processes by randomly sampling 2,000 generated queries from G-MSMARCO on each occasion. After each sampling, we access the consistency between the sampled G-MSMARCO and R-MSMARCO. As illustrated in Figure 3, the LLM-generated testing data exhibits stable and strong
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="4">MTEB (English)</th>
    <th colspan="4">AIR-BENCH 24.05 (English, test)</th>
  </tr>
  <tr>
    <th colspan="2">Overall 56 datasets</th>
    <th colspan="2">Retrieval (BEIR) 15 datasets</th>
    <th colspan="2">QA 7 datasets</th>
    <th colspan="2">Long-Doc 11 datasets</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Avg.</th>
    <th>Rank</th>
    <th>nDCG@10</th>
    <th>Rank</th>
    <th>nDCG@10</th>
    <th>Rank</th>
    <th>Recall@10</th>
    <th>Rank</th>
  </tr>
  <tr><th colspan="10"><i>LLM-based Embedding Models</i></th></tr>
  <tr>
    <td>NV-Embed-v2</td>
    <td>7.85B</td>
    <td>72.31</td>
    <td>1</td>
    <td>62.65</td>
    <td>1</td>
    <td>53.35</td>
    <td>3</td>
    <td>73.45</td>
    <td>1</td>
  </tr>
  <tr>
    <td>bge-en-ic1 (zero-shot)</td>
    <td>7.11B</td>
    <td>71.24</td>
    <td>2</td>
    <td>61.67</td>
    <td>2</td>
    <td>53.60</td>
    <td>2</td>
    <td>72.62</td>
    <td>3</td>
  </tr>
  <tr>
    <td>bge-en-ic1-e5data (zero-shot)</td>
    <td>7.11B</td>
    <td>64.67</td>
    <td>11</td>
    <td>59.59</td>
    <td>6</td>
    <td>54.46</td>
    <td>1</td>
    <td>73.43</td>
    <td>2</td>
  </tr>
  <tr>
    <td>SFR-Embedding-2_R</td>
    <td>7.11B</td>
    <td>70.31</td>
    <td>3</td>
    <td>60.18</td>
    <td>5</td>
    <td>50.80</td>
    <td>7</td>
    <td>65.83</td>
    <td>9</td>
  </tr>
  <tr>
    <td>gte-Qwen2-7B-instruct</td>
    <td>7.61B</td>
    <td>70.24</td>
    <td>4</td>
    <td>60.25</td>
    <td>3</td>
    <td>51.87</td>
    <td>5</td>
    <td>63.97</td>
    <td>10</td>
  </tr>
  <tr>
    <td>NV-Embed-v1</td>
    <td>7.85B</td>
    <td>69.32</td>
    <td>5</td>
    <td>59.36</td>
    <td>7</td>
    <td>50.97</td>
    <td>6</td>
    <td>72.08</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Linq-Embed-Mistral</td>
    <td>7.11B</td>
    <td>68.17</td>
    <td>6</td>
    <td>60.19</td>
    <td>4</td>
    <td>49.76</td>
    <td>9</td>
    <td>70.02</td>
    <td>5</td>
  </tr>
  <tr>
    <td>SFR-Embedding-Mistral</td>
    <td>7.11B</td>
    <td>67.56</td>
    <td>7</td>
    <td>59.00</td>
    <td>8</td>
    <td>52.78</td>
    <td>4</td>
    <td>68.10</td>
    <td>6</td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>7.11B</td>
    <td>66.40</td>
    <td>8</td>
    <td>56.87</td>
    <td>10</td>
    <td>49.88</td>
    <td>8</td>
    <td>66.91</td>
    <td>7</td>
  </tr>
  <tr><th colspan="10"><i>Large-size Embedding Models</i></th></tr>
  <tr>
    <td>jina-embeddings-v3</td>
    <td>572M</td>
    <td>65.51</td>
    <td>9</td>
    <td>53.88</td>
    <td>12</td>
    <td>45.07</td>
    <td>13</td>
    <td>61.50</td>
    <td>13</td>
  </tr>
  <tr>
    <td>gte-large-en-v1.5</td>
    <td>434M</td>
    <td>65.39</td>
    <td>10</td>
    <td>57.91</td>
    <td>9</td>
    <td>46.251</td>
    <td>11</td>
    <td>60.71</td>
    <td>14</td>
  </tr>
  <tr>
    <td>multilingual-e5-large-instruct</td>
    <td>560M</td>
    <td>64.41</td>
    <td>12</td>
    <td>52.47</td>
    <td>13</td>
    <td>45.39</td>
    <td>12</td>
    <td>63.96</td>
    <td>11</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5</td>
    <td>335M</td>
    <td>64.23</td>
    <td>13</td>
    <td>54.29</td>
    <td>11</td>
    <td>44.91</td>
    <td>14</td>
    <td>61.86</td>
    <td>12</td>
  </tr>
  <tr>
    <td>e5-large-v2</td>
    <td>335M</td>
    <td>62.20</td>
    <td>14</td>
    <td>50.56</td>
    <td>14</td>
    <td>46.253</td>
    <td>10</td>
    <td>66.16</td>
    <td>8</td>
  </tr>
  <tr><th colspan="10"><i>Lexical Method</i></th></tr>
  <tr>
    <td>BM25</td>
    <td>.</td>
    <td>.</td>
    <td>.</td>
    <td>40.76</td>
    <td>15</td>
    <td>39.16</td>
    <td>15</td>
    <td>53.09</td>
    <td>15</td>
  </tr>
</table>

Table 6: Comparison of the performance of 15 IR models on AIR-BENCH and MTEB/BEIR. The results on MTEB/BEIR are directly taken from the MTEB leaderboard. For detailed information of the models appearing in the table, please refer to Table 15. The detailed results for each dataset in AIR-BENCH are available in Appendix F.2.

![Robustness analysis of the consistency between the LLM-generated testing data and the human-labeled testing data. The mean correlation coefficient is 0.8031 with a mean p-value of 1e-4 across 30 simulated generation processes.](page_1012_1342_377_246.png)

Figure 3: Robustness analysis of the consistency between the LLM-generated testing data and the human-labeled testing data. The mean correlation coefficient is 0.8031 with a mean p-value of 1e-4 across 30 simulated generation processes.

consistency with the human-labeled testing data, highlighting the robustness of this consistency.

4.2 Comparison with MTEB/BEIR (RQ2)

To investigate what additional evaluation functionalities AIR-BENCH can offer compared to MTEB (Muennighoff et al., 2023) and BEIR (Thakur et al., 2021), we compare the performance of 15 IR models on AIR-BENCH and MTEB/BEIR.

Setup. In addition to 14 large-size and LLM-based embedding models exhibiting superior performances on MTEB/BEIR, we also evaluate the performance of lexical method BM25 (Robertson and Zaragoza, 2009).

Main Results. As presented in Table 6, we can make the following observations based on the comparison results. 1) LLM-based embedding models generally outperform large-size embedding models on both AIR-BENCH and MTEB/BEIR, largely due to the superior generalization ability of LLMs. Besides, BM25 performs worse than all embedding models on both AIR-BENCH and BEIR. 2) The QA task and the Long-Doc task in AIR-BENCH exhibit a level of heterogeneity. The Spearman rank correlation coefficient between the rankings of the nine LLM-based embedding models across the two tasks is only 0.6, with a p-value of 0.0876. Moreover, as a large-size embedding model, e5-large-v2 even outperforms some LLM-based embedding models on the Long-Doc task. 3) By comparing the results on AIR-BENCH and MTEB/BEIR, we observe that better performance on MTEB/BEIR may not indicate better
<table>
  <tr>
    <th rowspan="2">Dataset (L)</th>
    <th colspan="2">mContriever</th>
    <th colspan="2">mContriever-finetuned</th>
  </tr>
  <tr>
    <th>nDCG@10</th>
    <th>nDCG@10 / Training Data</th>
    <th>nDCG@10</th>
    <th>nDCG@10 / Training Data</th>
  </tr>
  <tr>
    <td>finance_en</td>
    <td>39.452</td>
    <td></td>
    <td>41.281 (\( \uparrow 1.829 \))<br>FiQA (Maia et al., 2018)</td>
    <td></td>
  </tr>
  <tr>
    <td>healthcare_zh</td>
    <td>14.557</td>
    <td></td>
    <td>17.351 (\( \uparrow 2.794 \))<br>cMedQAv2 (Zhang et al., 2018)</td>
    <td></td>
  </tr>
  <tr>
    <td>law_de</td>
    <td>5.614</td>
    <td></td>
    <td>6.687 (\( \uparrow 1.073 \))<br>Hoppe et al. (2021)</td>
    <td></td>
  </tr>
  <tr>
    <td>law_fr</td>
    <td>3.102</td>
    <td></td>
    <td>4.325 (\( \uparrow 1.223 \))<br>BSARD (Louis and Spanakis, 2022)</td>
    <td></td>
  </tr>
  <tr>
    <td>web_hi</td>
    <td>19.067</td>
    <td></td>
    <td>30.103 (\( \uparrow 11.036 \))<br>mMARCO (Bonifacio et al., 2021)</td>
    <td></td>
  </tr>
  <tr>
    <td>wiki_ar</td>
    <td>38.159</td>
    <td></td>
    <td>43.470 (\( \uparrow 5.311 \))<br>MIRACL (Zhang et al., 2023)</td>
    <td></td>
  </tr>
</table>

Table 7: AIR-BENCH can showcase models’ performance enhancement in specific domains. The training process takes 100 steps for cMedQAv2, and 50 steps for the other datasets.

performance on AIR-BENCH. For example, according to Li et al. (2024), bge-en-icl utilizes more in-domain training data in MTEB/BEIR than bge-en-icl-e5data and achieves more superior performance on MTEB/BEIR. However, compared to bge-en-icl-e5data, bge-en-icl shows performance degradation on AIR-BENCH, including both the QA task (54.46 \( \rightarrow \) 53.60) and the Long-Doc task (73.43 \( \rightarrow \) 72.62). This suggests that increased in-domain training data in MTEB/BEIR may lead to over-fitting, thereby reducing the generalization ability of embedding models.

In conclusion, as a new benchmark, AIR-BENCH can offer additional evaluation functionalities for community developers compared to MTEB/BEIR.

4.3 Distinguishing Models (RQ3)

To examine how effectively AIR-BENCH can distinguish the capabilities of distinct IR models, we evaluate the performance of a single model before and after fine-tuning to illustrate that AIR-BENCH can reflect the performance enhancement of IR models in specific domains.

Setup. We fine-tune mContriever8 (Izard et al., 2021) using domain-specific training datasets, and compare the model’s performance on the corresponding datasets in AIR-BENCH before and after fine-tuning. Specifically, we fine-tune9 mCon-

8https://huggingface.co/facebook/mcontriever-msmarco

9The learning rate is \( 2 \times 10^{-4} \), the warmup ratio is 0.1, and the weight decay is 0.01. The training process takes around a hundred steps with a total batch size of 64 on 8 A800 GPUs.

triever with FlagEmbedding tool10 to enhance its domain-specific capabilities. The domain-specific training data used for fine-tuning is independent of the corresponding testing data in AIR-BENCH.

Main Results. Table 7 presents the detailed information about each domain-specific training dataset and compares the model’s performance on the corresponding dataset in AIR-BENCH before and after fine-tuning. For example, after fine-tuning with the Hindi training data from mMARCO (Bonifacio et al., 2021), the performance of mContriever on the web_hi dataset in AIR-BENCH improves from 19.067 to 30.103. This trend is also observed in other domains, such as finance, healthcare, law and wiki. Therefore, AIR-BENCH effectively reflects the performance enhancement of IR models in specific domains following fine-tuning with domain-specific training datasets.

We also evaluate a diverse set of IR models on AIR-BENCH to further demonstrate its capability of distinguishing different models across multiple dimensions, including model type, domain, and language. Refer to Appendix F.1 for the details.

5 Related Work

The related works are reviewed from two aspects: evaluation datasets for IR, and synthetic data generation for IR.

5.1 Evaluation Datasets for IR

Evaluation datasets are critically important for the development of IR models.

In recent years, a series of milestone works have been introduced to the community. As the earlier contributions, MS MARCO (Bajaj et al., 2016) includes Bing search questions paired with human-labeled relevant passages from Web documents. Natural Questions (NQ) (Kwiatkowski et al., 2019) consists of Google search queries with human-labeled relevant Wikipedia pages. Both MS MARCO and NQ are designed for open-domain question answering tasks in English. Recent works like Mr.TyDi (Zhang et al., 2021) and MIRACL (Zhang et al., 2023) focus on multilingual retrieval in non-English languages. Mr.TyDi covers 11 languages and MIRACL encompasses an extended 18 languages. BEIR (Thakur et al., 2021) and MTEB (Muennighoff et al., 2023) are introduced to benchmark IR models in a general-

10https://github.com/FlagOpen/FlagEmbedding
domain zero-shot setting, including multiple existing datasets from diverse tasks and domains.

However, all of these benchmarks, which rely on pre-defined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently. Recently, several studies have explored the application of large language models for retrieval evaluation in retrieval-augmented generation (RAG) systems (Es et al., 2024; Saad-Falcon et al., 2024; Salemi and Zamani, 2024), offering a promising solution to this challenge. Nonetheless, a comprehensive IR benchmark that addresses this limitation remains insufficiently developed.

5.2 Synthetic Data Generation for IR

The tasks and domains in IR applications are often diverse and dynamic, meaning that the the training and evaluation data are frequently unavailable for new tasks and domains. As a result, it becomes challenging to fine-tune and evaluate IR models in these contexts.

Several recent works (Bonifacio et al., 2022; Dai et al., 2023; Jeronymo et al., 2023; Khramtsova et al., 2024; Thakur et al., 2024) have focused on addressing the scarcity of domain-specific training data by prompting LLMs to generate synthetic training data. Wang et al. (2023) and Chen et al. (2024a) employ LLMs to generate synthetic task and training data. Lee et al. (2024b) further refines the synthetic training data by using LLMs to select more relevant positives and negatives.

However, there is currently limited research addressing the scarcity of domain-specific evaluation datasets. Thomas et al. (2024) have demonstrated that powerful LLMs can generate high-quality golden labels for search system with accuracy comparable to human labelers, laying a solid foundation for our work. Our experiment results also demonstrate that the LLM-generated testing data aligns well with the human-labeled testing data. To our knowledge, AIR-BENCH is the first comprehensive IR benchmark that utilizes the LLM-generated datasets to perform evaluation.

6 Conclusion

In this paper, we introduce a new IR benchmark AIR-BENCH, which is highlighted for three main features: 1) Automated, 2) Heterogeneous, and 3) Dynamic. We demonstrate that the generated testing data in AIR-BENCH is highly consistent with the human-labeled testing data, which makes AIR-BENCH a dependable benchmark for evaluating IR models. Additionally, we demonstrate that AIR-BENCH can offer additional evaluation functionalities compared to MTEB/BEIR. Last but not least, we demonstrate that AIR-BENCH can effectively distinguish the capabilities of distinct IR models from multiple dimensions.

AIR-BENCH currently covers 2 tasks, 9 domains and 13 languages, including a total of 69 datasets. In the future, AIR-BENCH will be extended to cover more tasks, domains and languages to provide an increasingly comprehensive evaluation benchmark for community developers. We welcome datasets contributions to AIR-BENCH\(^{11}\) as well as the model submissions to our leaderboard\(^{12}\).

Limitations

While AIR-BENCH aims to be a comprehensive IR benchmark by introducing new features to address the limitations of existing benchmarks, it still has several inherent constraints: 1) Dependence on real-world corpora. The dataset generation process in AIR-BENCH begins with corpus preparation. As a result, access to real-world corpora is essential for constructing evaluation datasets. Fortunately, this requirement is typically both feasible and practical in real-world scenarios. 2) Reliance on capabilities of LLM. The quality of the generated testing data in AIR-BENCH largely depends on the LLM’s capabilities. However, This limitation can be mitigated by the rapid advancement of LLMs. 3) Potential biases from quality control models. In addition to the LLM, we incorporate several existing IR models during the quality control stage. This reliance may introduce potential biases into the final evaluation datasets. However, as these models continue to improve, the impact of such biases can be progressively reduced.

Ethics Consideration

Since AIR-BENCH is built on testing data generated by LLM, it may inherit potential biases, toxicity, and other issues present in the LLM used during the generation process. Additionally, considering that the corpora utilized in the generation process are derived from the real-world sources,

\(^{11}\)https://github.com/AIR-Bench/AIR-Bench
\(^{12}\)https://huggingface.co/spaces/AIR-Bench/leaderboard
they may contain sensitive content. Therefore, the testing data in AIR-BENCH may only be used for evaluation purposes.

Acknowledgements

This work is supported by National Science and Technology Major Project (2023ZD0121504), National Natural Science Foundation of China (No. U24A20253, 62276171), Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515011938), Shenzhen Fundamental Research-General Project, China under Grant (No. JCYJ20240813141503005). We appreciate the valuable feedback from Tom Aarsen, Niklas Muennighoff, Jiajun Wang, and Linpeng Tang.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2387–2392, New York, NY, USA. Association for Computing Machinery.

Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Nogueira. 2021. mmrco: A multilingual version of ms marco passage ranking dataset. corr abs/2108.13897 (2021). arXiv preprint arXiv:2108.13897.

Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz, and Anders Søgaard. 2023. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. 2024a. Little giants: Synthesizing high-quality embedding data at scale. arXiv preprint arXiv:2410.18634.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2318–2335, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.

Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representations.

Tobias Daudert and Sina Ahmadi. 2019. CoFiF: A corpus of financial reports in French language. In Proceedings of the First Workshop on Financial Technology and Natural Language Processing, pages 21–26, Macao, China.

Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.

Felix Hamburg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223.

Peter Henderson*, Mark S. Krass*, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. arXiv preprint.

Christoph Hoppe, David Pelkmann, Nico Migenda, Daniel Hötte, and Wolfram Schenck. 2021. Towards intelligent legal advisors for document retrieval and question-answering in german legal documents. In 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pages 29–32. IEEE.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.

Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577.

Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Bakhtashmotlagh, and Guido Zuccon. 2024. Leveraging llms for unsupervised dense retriever ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 1307–1317, New York, NY, USA. Association for Computing Machinery.

Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy-yeong Sohn, and Chanyeol Choi. 2024. Ling-embed-mistral-elevating text retrieval with improved gpt data through task-specific control and quality refinement. Linq AI Research Blog.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenneth Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024a. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv:2405.17428.

Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Botratko, Rajiv Kapadia, Wen Ding, et al. 2024b. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327.

David Lewis. 1997. Reuters-21578 Text Categorization Collection. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C52G6M.

Patrick Lewis, Ethan Perez, Aleksandra Pikuts, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700.

Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023a. Huatuo-26m, a large-scale chinese medical qa dataset. Preprint, arXiv:2305.01526.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356–2362.

Jerry Liu. 2022. LlamaIndex.

Antoine Louis and Gerasimos Spanakis. 2022. A statutory article retrieval dataset in french. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, page 6789–6803, Dublin, Ireland. Association for Computational Linguistics.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval. arXiv:2310.08319.

Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942.

Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

Inc. NetEase Youdao. 2023. Bcembedding: Bilin-gual and crosslingual embedding for rag. https://github.com/netease-youdao/BCEmbedding.

Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E. Ho. 2023. Multilegalpile: A 689gb multilingual legal corpus. Preprint, arXiv:2306.02069.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1).

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 338–354, Mexico City, Mexico. Association for Computational Linguistics.

Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2395–2400, New York, NY, USA. Association for Computing Machinery.

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational Linguistics.

Charles Spearman. 1961. The proof and measurement of association between two things.

Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapapa, Andreas Koukounas, Nan Wang, et al. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173.

Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, and Daniel Cer. 2024. Leveraging LLMs for synthesizing training data across many languages in multilingual dense retrieval. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7699–7724, Mexico City, Mexico. Association for Computational Linguistics.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 1930–1940, New York, NY, USA. Association for Computing Machinery.

Fabian Villena. 2019. Multilingual medical corpora.

Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77–82.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022a. Simlm: Pre-training with representation bottleneck for dense passage retrieval. ArXiv, abs/2207.02578.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022b. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 641–649, New York, NY, USA. Association for Computing Machinery.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and
Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.

Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061–74071.

Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval.

Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 127–137, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114–1131.

Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot LLM rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 358–370, Mexico City, Mexico. Association for Computational Linguistics.

Overview of Appendix

• Appendix A: Details on Benchmark Construction.

• Appendix B: AIR-BENCH Datasets.

• Appendix C: AIR-BENCH Software.

• Appendix D: AIR-BENCH Data Examples.

• Appendix E: Evaluation Details.

• Appendix F: More Experiment Results.

A Details on Benchmark Construction

In this section, we provide more details on the construction of datasets in AIR-BENCH.

A.1 Corpora Preparation

AIR-BENCH currently covers two different tasks: QA and Long-Doc. For QA task, we directly use the real-world dataset as the corpus, such as Wikipedia, mc4(Raffel et al., 2020), CC-News(Hamborg et al., 2017), etc. We filter out text that is either too short or too long and make a straightforward attempt to remove any information that names or uniquely identifies individuals, as well as any offensive content. For Long-Doc task, we first select one long document for each dataset, such as book, ArXiv paper, legal document, etc., and remove table of contents and references. Then we use the node parser13 tool from LlamaIndex(Liu, 2022) to split the long document into fixed-size chunks14 as the corpus. All corpora used in AIR-BENCH are available in Appendix B.1.

A.2 Candidate Generation

A.2.1 Query Generation

To diversify the generated queries, we consider the following attributes when designing the prompt.

Query Length. This refers to the length of the query. We consider four different categories based on word count: less than 5 words, less than 10 words, 10 to 20 words, and at least 20 words. The ratio of the number of queries in these categories is 1:4:2:1.

Query Type. This refers to the type of the query. We consider three different types: question, problem, and claim. Based on our observation, the “problem” type is usually more difficult than the

13SimpleNodeParser: https://github.com/run-llama/llama_index
14chunk_size=200, chunk_overlap=50
“question” type. The ratio of the number of queries in these three types is 3:1:1. For Long-Doc task, considering that the chunks in the corpus are derived from the same long document, the topics of these chunks are highly related. Therefore, we only utilized two types for Long-Doc task: question and claim. For the “claim” type, we observe that when the claim is too short, it will become too ambiguous to be a high-quality query. Therefore, the query length for the “claim” type is only sampled from “between 10 and 20 words” and “at least 20 words”.

Information-based Type. This refers to the type of the information used when formulating queries. We consider two different types: queries based on the overall information in the document, and queries based on the partial information beyond the main topic of the document. The ratio of the number of queries in these two types is 1:1.

Expression Style. This refers to the style of query formulation. The three attributes mentioned above are used in Step 4. In Step 5, we consider different types of expression styles, allowing the LLM to rewrite the queries using various styles, thereby enhancing the diversity of query formulations. There are seven different styles in total: concise, casual, informal, formal, professional, complicated, and academic. During the rewriting process, the sampling probabilities for these styles are in the ratio of 5:3:3:1:1:1:1.

A.2.2 Hard Negative Generation
To improve the difficulty of the generated datasets, we prompt LLM to generate 3-7 hard negative documents based on the rewritten query and the original positive document. For Long-Doc task, considering the chunks are extracted from the same long document and some of them have been hard enough, we do not generate additional hard negatives. The statistics of the number of hard negatives in each dataset are available in Appendix B.1.

A.3 Quality Control
We present more details on how we use LLM as labeler to label the relevance, select the embedding model and multiple re-ranking models, and set the predetermined threshold for pre-labeling.

Use LLM as labeler. Thomas et al. (2024) demonstrated that LLMs like OpenAI’s GPT-4 are as accurate as human labelers when generating high-quality golden labels for search system. Zhuang et al. (2024) showed that incorporating fine-grained relevance labels into the prompt for LLM rerankers significantly improves their performance on zero-shot reranking. In our paper, we use GPT-4 as labeler with a 4-level relevance generation strategy. The prompt we used is shown in Table 8.

For the following query and document, judge whether the document is relevant to the query.

Query:
"{query}"

Document:
"{doc}"

Your output must be one of the following:
- 0: The document is not relevant to the query.
- 1: The document is superficially relevant but actually not relevant to the query.
- 2: The document is somewhat relevant to the query.
- 3: The document is relevant to the query.

Do not explain your answer in the output. Your output must be a single number.

Table 8: Prompt used for LLM to label the relevance. {query} and {doc} are placeholders of query and document, respectively.

Embedding Model. Considering that the corpora in AIR-BENCH are multilingual, we use bge-m3\(^{15}\) as the embedding model to recall the top-1000 relevant documents.

Multiple Re-ranking Models. For the datasets in English and Chinese, we use the following three re-ranking models: bge-ranker-large\(^{16}\), bce-ranker-base_v1\(^{17}\), mmarco-mMiniLMv2-L12-H384-v1\(^{18}\). For the datasets in the other languages, we use the following three re-ranking models: bge-ranker-v2-m3\(^{19}\), mmarco-mMiniLMv2-L12-H384-v1, bge-ranker-v2-gemma\(^{20}\).

Predetermined Threshold. For the hard negative documents, we set the threshold to 20. For the other documents, we set the threshold to 10.

\footnotetext{
15https://huggingface.co/BAAI/bge-m3
16https://huggingface.co/BAAI/bge-ranker-large
17https://huggingface.co/maida1un1020/bce-ranker-base_v1
18https://huggingface.co/nreimers/mmarco-mMiniLMv2-L12-H384-v1
19https://huggingface.co/BAAI/bge-ranker-v2-m3
20https://huggingface.co/BAAI/bge-ranker-v2-gemma
}
A.4 Queries Split

After the quality control stage, we split the generated queries into different sets. For QA task, we split the queries in each dataset into dev set and test set in a 1:4 ratio. For Long-Doc task, we select one dataset as the dev set for each domain, and remain other datasets as the test set. Refer to Appendix B.1 for more details.

B AIR-BENCH Datasets

B.1 Specifications

The available versions of AIR-BENCH are listed in Table 9.

<table>
  <tr>
    <th>Version</th>
    <th>Release Date</th>
    <th>#domains</th>
    <th>#languages</th>
    <th>#datasets</th>
    <th>Statistics</th>
  </tr>
  <tr>
    <td>24.04</td>
    <td>May 21, 2024</td>
    <td>8</td>
    <td>2</td>
    <td>28</td>
    <td>Table 16</td>
  </tr>
  <tr>
    <td>24.05</td>
    <td>Oct 17, 2024</td>
    <td>9</td>
    <td>13</td>
    <td>69</td>
    <td>Table 17-21</td>
  </tr>
</table>

Table 9: Available versions of AIR-BENCH.

For each dataset, we use the same format as BEIR, i.e. corpus, queries and qrels21, which are all available in the Hugging Face Hub22 of AIR-BENCH. To avoid the possible data leakage, we keep the qrels in test splits private. For the qrels in dev splits, we make them public to enable the developers to perform evaluation by themselves.

As the initial version, AIR-BENCH 24.04 only covered 2 languages, English and Chinese. Additionally, each dataset in AIR-BENCH 24.04 only contains the test set, which means that the developers could not know the evaluation results until they submit their model’s search results to the leaderboard. As for the latest version AIR-BENCH 24.05, we have covered 13 languages, and included dev set and test set. The golden labels of dev set are made public, and the golden labels of test set remain private. Furthermore, the corpus size of some datasets in AIR-BENCH 24.04 is too large (such as 6.7M for wiki_en dataset and 2.4M for finance_zh dataset in QA task), which makes the download of datasets and the evaluation of models relatively inefficient. Therefore, in AIR-BENCH 24.05, we trimmed the large corpora to maintain a corpus size of around 1M for each dataset.

For the detailed statistics of all datasets in AIR-BENCH 24.04 and 24.05, please refer to Table 16 and Table 17-21, respectively. Note that we use the tokenizer23 of OpenAI’s GPT-4o24 to count the token number for every language.

<table>
  <tr>
    <th rowspan="2">split → # of datasets →</th>
    <th colspan="2">QA</th>
    <th colspan="2">Long-Doc</th>
  </tr>
  <tr>
    <th>dev</th>
    <th>test</th>
    <th>dev</th>
    <th>test</th>
  </tr>
  <tr>
    <th colspan="5">Query Style</th>
  </tr>
  <tr>
    <td>FORMAL</td>
    <td>31.3%</td>
    <td>35.3%</td>
    <td>17.7%</td>
    <td>17.9%</td>
  </tr>
  <tr>
    <td>INFORMAL</td>
    <td>44.3%</td>
    <td>44.7%</td>
    <td>28.3%</td>
    <td>27.4%</td>
  </tr>
  <tr>
    <td>PROFESSIONAL</td>
    <td>7.0%</td>
    <td>6.8%</td>
    <td>8.2%</td>
    <td>10.3%</td>
  </tr>
  <tr>
    <td>CASUAL</td>
    <td>0.8%</td>
    <td>0.7%</td>
    <td>0.5%</td>
    <td>0.6%</td>
  </tr>
  <tr>
    <td>COMPLICATED</td>
    <td>0.5%</td>
    <td>0.3%</td>
    <td>0.7%</td>
    <td>1.4%</td>
  </tr>
  <tr>
    <td>CONCISE</td>
    <td>8.3%</td>
    <td>5.4%</td>
    <td>12.3%</td>
    <td>12.5%</td>
  </tr>
  <tr>
    <td>ACADEMIC</td>
    <td>7.8%</td>
    <td>6.8%</td>
    <td>32.2%</td>
    <td>29.8%</td>
  </tr>
  <tr>
    <td>OTHERS</td>
    <td><0.1%</td>
    <td><0.1%</td>
    <td>0.1%</td>
    <td>0.1%</td>
  </tr>
</table>

Table 10: The style distribution of queries in each split for each task in AIR-BENCH 24.05.

B.2 Licenses

In Table 16-21, we also list the licenses of the source corpora used for the dataset generation in AIR-BENCH. All generated testing data in AIR-BENCH is licensed under CC BY-NC-SA-4.025. The testing data in AIR-BENCH may only be used for evaluation purposes.

B.3 Additional Diversity Analysis

We provide more analysis of diversity to better characterize AIR-BENCH.

B.3.1 Query Diversity

We also analyze the style diversity of the generated queries in AIR-BENCH. We still utilize GPT-4o26 as labeler to label the style of the queries in AIR-BENCH. The optional query styles include: FORMAL, INFORMAL, PROFESSIONAL, CASUAL, COMPLICATED, CONCISE, ACADEMIC, and OTHERS. The statistics are grouped by tasks and splits in Table 10.

We can make the following observations according to the results. First of all, since the optional styles given to GPT-4o are not mutually exclusive, the ratio of the number of different styles is not consistent with the ratio we set in the generation stage (Step 5 of the Candidate Generation stage). Secondly, the QA task tends to have more INFORMAL queries, and Long-Doc task tends to have more ACADEMIC queries, which may be due to

21qrels are the relevance labels for queries. The relevance label is 1 for the positive document, and 0 for the negative document.
22https://huggingface.co/AIR-Bench
23https://github.com/openai/tiktoken
24https://openai.com/index/hello-gpt-4o/
25https://creativecommons.org/licenses/by-nc-sa/4.0
26gpt-4o-2024-08-06: https://platform.openai.com/docs/models/gpt-4o
Figure 4: Pairwise weighted Jaccard similarity scores between AIR-BENCH English datasets. We use the tokenizer of GPT-4o to tokenize the corpus of each dataset.

the fact that the long documents in the Long-Doc task are more academic related, such as ArXiv papers, books, etc. Finally, PROFESSIONAL queries and COMPLICATED queries account for a certain portion, which means that some queries in AIR-BENCH are probably challenging for IR models.

B.3.2 Corpus Diversity

Following the work of BEIR (Thakur et al., 2021), we compute the pairwise weighted Jaccard similarity scores between the datasets in AIR-BENCH. Considering that there are 69 datasets in total, we only present the results of datasets in English here. As shown in Figure 4, we can observe that the corpora from different domains have a low weighted Jaccard similarity word overlap, indicating that AIR-BENCH is a challenging benchmark where the IR methods must generalize well to diverse out-of-distribution domains.

C AIR-BENCH Software

The AIR-BENCH software\(^{27}\) makes it convenient for the evaluation of any information retrieval methods. With the provided Python framework, in order to evaluate a retrieval method, users only need to implement a Retriever that takes the queries and the corpus as the inputs, and returns the top-\(k\) relevant documents for each query as the outputs. If the users want to evaluate the performance of retrieval-then-reranking method, they only need to additionally implement a Reranker, which takes the queries, the corpus, and the top-\(k\) search results from Retriever as the inputs, and returns the re-ranked top-\(k'\) (\(k' \leq k\)) relevant documents as the outputs.

We also maintain a Hugging Face leaderboard\(^{28}\) with all datasets and models. To make the leaderboard more readable, we classify the submissions into three categories: 1) Retrieval Only. It means

\footnotetext{27https://github.com/AIR-Bench/AIR-Bench}
\footnotetext{28https://huggingface.co/spaces/AIR-Bench/leaderboard}
that this submission only uses a specific retrieval method to generate the top-\( k \) search results. 2) Reranking Only. It means that this submission uses BM25 as the retrieval method and then uses a specific reranking method to re-rank the search results from BM25 to generate the re-ranked top-\( k \) search results. 3) Retrieval+Reranking. It means that this submission first uses a specific retrieval method to generate the top-\( k \) search results, and then uses a specific reranking method to re-rank to get the final search results. It should be noted that our leaderboard only maintain the evaluation results for the test splits, and the evaluations results for the dev splits will be available on the MTEB leaderboard\(^{29}\).

To facilitate the evaluation of existing IR models, we also develop the evaluation scripts based on two mainstream architectures: HuggingFace Transformers\(^{30}\) (Wolf et al., 2020) and Sentence Transformers\(^{31}\) (Reimers and Gurevych, 2019). These scripts are all available in our repository\(^{32}\).

D AIR-BENCH Data Examples

We list some examples of the generated testing data in Table 12-14.

E Evaluation Details

E.1 Models

For detailed information of the models appearing in this paper, please refer to Table 15. For the BM25 method, we employ the implementation from Pyserini\(^{33}\) (Lin et al., 2021). For the evaluation of BM25-based re-ranking models, we evaluate the performance by re-ranking the top-100 search results from BM25 with the re-ranking models.

The models used in this paper are all publicly available (see Table 15 for the public link). We confirm that we did not violate the license of any model used in our paper.

E.2 Parameters

When performing evaluation, we set the max length of both query and passage to 512 tokens. If the embedding models need task specific instruction, such as e5-mistral-7b-instruct (Wang et al., 2023), SFR-Embedding-Mistral, etc., we use the same instruction for all datasets: “Given a question, retrieve passages that answer the question”, which is denoted as Instr-1. Considering that the queries in AIR-BENCH include both questions and claims, we also evaluate the performance of e5-mistral-7b-instruct with a more reasonable but more complex instruction: “Given a question or claim, retrieve passages that answer the question or support the claim”, which is denoted as Instr-2. However, as shown in Table 11, the performance of e5-mistral-7b-instruct using Instr-2 is slightly worse than that using Instr-1, which may indicate that current models are not yet able to adapt well to more complex instruction.

For the total computational budget, we did not perform detailed statistics. However, based on our estimates, all evaluations in this paper required approximately 2000 GPU hours using 24 A800 (80GB) GPUs.

F More Experiment Results

F.1 Distinguishing Models

We evaluate a diverse set of IR models on AIR-BENCH to demonstrate its capability of distinguishing different models from multiple dimensions: model type, domain, language.

Model Type. As shown in Figure 5a and Figure 5b, we can observe the following three points on both QA task and Long-Doc task, regardless of whether the datasets are only in English or multilingual: 1) BM25 performs worse than all embedding models. 2) BM25 + bge-ranker-v2-m3 achieves more excellent performance than all of the embedding models. 3) The performance of embedding models from the same series scales with model size.

Domain. We evaluate three kinds of embedding models with the same model size (\( large-size \)), and compare their performances in each domain on AIR-BENCH. As shown in Figure 5c and Figure 5d, regardless of whether the task is QA or Long-Doc and whether the datasets are only in English or multilingual, no model is able to achieve the best performance on all domains.

Language. We evaluate three kinds of embedding models with the same model size (\( large-size \)) on the multilingual datasets of AIR-BENCH, and compare their performance on the datasets of each language. As shown in Figure 5e, we also observe that no model is able to achieve the best perfor-
<table>
  <tr>
    <th># of datasets →</th>
    <th>QA (English, test)<br>7 datasets</th>
    <th>QA (Multilingual, test)<br>53 datasets</th>
    <th>Long-Doc (English, test)<br>11 datasets</th>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct (Instr-1)</td>
    <td>49.880</td>
    <td>48.077</td>
    <td>66.908</td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct (Instr-2)</td>
    <td>49.252</td>
    <td>47.772</td>
    <td>66.766</td>
  </tr>
</table>

Table 11: Comparison of performances when using different evaluation parameters on AIR-BENCH. The metric for QA task is nDCG@10, and the metric for Long-Doc task is Recall@10.

mance on all languages.

Apart from the results of large-size embedding models in Figure 5, we also perform investigation with base-size embedding models and LLM-based embedding models. The additional results are shown in Figure 6.

F.2 Detailed Evaluation Results

In this section, we provide the detailed evaluation results of each model on AIR-BENCH 24.05. Table 22 presents the detailed evaluation results of English IR models on AIR-BENCH 24.05. Table 23 presents the detailed evaluation results of multilingual IR models on AIR-BENCH. For detailed information of the models appearing in these tables, please refer to Table 15.
Domain: news; Language: English
Original Positive:
“It’s hard to think of a part of the world that hasn’t been touched by robotic advances this year.\n In 2016, strides were taken in the areas of robotic home delivery, cooking, tough terrain navigation and even attempts to conquer the beautiful game of football.\n Here are the top five robots of the year.\n While we’re not quite at the singularity yet, more sophisticated automation is an inevitability of the future.\n The strides in Artificial Intelligence (AI) over the past decade have been huge, so expect to see a lot more in this area in the coming years.\n We just hope the tech guys making super AI fit it with an “off” switch so it can be unplugged when it wants to, you know, take over the world and destroy everything.”

Character: Robotics Engineer

Scenario: Preparing a presentation on the yearly advancements in robotics technology.

Original Query: Which industries implemented robotic home delivery?
Rewritten Query: In which sectors has the implementation of autonomous delivery robots for residential services been observed?

Hard Negative 1:
“Autonomous technologies have been expanding rapidly across various industries, with drones making headway in aerial inspections and surveillance. Companies are investing in autonomous flight for package delivery, but primarily in commercial settings. The convenience and efficiency improvements in logistics are undeniable, but residential use isn’t widespread yet.”

Hard Negative 2:
“Residential sectors are increasingly relying on technology, with smart homes integrating systems for automated cleaning, energy management and advanced security. These innovations in domestic tech have redefined the way we live, promising a future where household chores are managed seamlessly through digital interfaces and remote controls.”

Hard Negative 3:
“Recent developments in the robotics industry have witnessed significant progress in various sectors, such as industrial manufacturing, precision agriculture, and automated warehousing solutions. These robots have revolutionized production efficiency, crop management, and inventory control, enhancing economic output.”

Hard Negative 4:
“In recent years, residential areas have seen an uptick in smart home innovations that include automated climate control, security systems with facial recognition, and voice-activated appliances. The integration of AI in household management has significantly enhanced the convenience and efficiency of daily living.”

Hard Negative 5:
“Experts predict an expansion in the use of unmanned vehicles for military logistics and combat support missions. The autonomous systems being developed are designed for supply transport, surveillance, and even tactical offense, set to revolutionize battlefield strategies in the near future.”

Table 12: Random sampled examples for the generated testing data. Domain: news, Language: English.
Domain: healthcare; Language: English
Original Positive:
“Only two patients, 5 and 12 years old, with primary gastric NHL were found. Upper gastroduodenal endoscopy detected an ulcer in the lesser curvature of the body of the stomach, in both cases. Endoscopy revealed a moderate chronic gastritis in the antrum of both patients that was H. pylori associated in one of them who also suffered from chronic gastritis. Biopsy specimens demonstrated infiltration by Burkitt lymphoma (BL). The two patients received chemotherapy for 6 months. Additionally, one of the two patients received a triple therapy regimen with bismuth, amoxicillin, and metronidazole for H. pylori. Fifteen and six years later they are in complete remission, free of symptoms.”

Character: College student

Scenario: Creating a presentation on the clinical manifestations and treatment outcomes of primary gastric non-Hodgkin’s lymphoma in pediatric patients.

Original Query: How long did the pediatric patients receive chemotherapy for primary gastric NHL?
Rewritten Query: How long were the kids treated with chemo for their stomach lymphoma?

Hard Negative 1:
“In a recent clinical review, five pediatric cases of gastrointestinal complaints were assessed. The patients, ranging in age from 3 to 14 years, presented with various symptoms including abdominal pain, vomiting, and weight loss. In-depth medical evaluations, including blood tests, abdominal ultrasonography, and, for three patients, an upper gastroduodenal endoscopy, were conducted. The endoscopic examination in these three patients showed mild inflammation in the stomach lining and superficial gastric erosions in the antrum and the lesser curvature. None of the patients had a history of gastric malignancies, and there were no indications of Non-Hodgkin Lymphoma (NHL) or any other types of cancer. Helicobacter pylori infection was not detected in any of the cases. The patients’ symptoms were managed with dietary modifications and antacid medications. Symptom relief was noted in all cases, and follow-up visits over the course of six months revealed significant improvement and no further gastrointestinal issues. The clinical team concluded that the symptoms were likely due to functional dyspepsia and emphasized the importance of considering less severe diagnoses when pediatric patients present with gastrointestinal symptoms.”

Hard Negative 2:
“Two young individuals, aged 6 and 11, presented with abdominal discomfort and were subsequently screened for gastrointestinal disorders. Initial evaluation through pediatric upper gastrointestinal series indicated irregularities in the stomach lining, prompting further investigation. Comprehensive upper gastrointestinal endoscopies were performed, illuminating significant gastroesophageal reflux disease (GERD) in both patients, characterized by distinctive erosions in the esophagus and transient lower esophageal sphincter relaxations. GERD was particularly pronounced along the greater curvature of the stomach. Their evaluations also included biopsies of the gastric tissue, which fortunately ruled out malignancy, including lymphomas and other gastric cancers. To manage the GERD, both patients were placed on a rigorous treatment regimen including lifestyle modifications and proton-pump inhibitors (PPIs). Each was monitored regularly via follow-up endoscopies which demonstrated gradual improvements in esophageal tissue integrity. Concurrently, both were tested for H. pylori, with one testing positive. The H. pylori-positive patient underwent an eradication protocol with a combination therapy of clarithromycin, amoxicillin, and a PPI, resulting in successful elimination of the infection. Years later, through diligent management and follow-up, both individuals have achieved excellent control over their symptoms and maintain a good quality of life.”

Hard Negative 3:
“Numerous pediatric cases have been reviewed to understand the duration and efficacy of chemotherapy in treating various forms of juvenile cancer. One study outlines the treatment plan for a pair of siblings, aged 7 and 14, diagnosed with acute lymphoblastic leukemia (ALL). The treatment protocol involved a comprehensive induction regimen followed by a consolidation phase. During the induction phase, which lasted for about a month, the patients were administered a combination of vincristine, prednisone, asparaginase, and an anthracycline. The consolidation phase incorporated methotrexate and 6-mercaptopurine and extended over several months. Intrathecal chemotherapy was included to prevent CNS disease. Maintenance therapy was subsequently initiated, which is scheduled to continue for a period of three years, with regular follow-ups to monitor remission status. It was observed that the older child had to face additional challenges due to the emergence of several therapy-related side effects. Despite the intensive treatment, both patients are currently responding positively with substantial remission observed in follow-up examinations. The study emphasizes the importance of a tailored approach to pediatric chemotherapy, taking into account not only the type of cancer but also individual patient factors and potential long-term outcomes.”
Domain: wiki; Language: English
Original Positive:
“Caffeine/ergotamine (trade name Cafergot) is the proprietary name of a medication consisting of ergotamine tartrate and caffeine. This combination is used for the treatment of headaches, such as migraine headache.\n\nUse\n\n Correct timing of use is important. Cafergot is an abortive headache treatment, which prevents the development of the headache, rather than a treatment for an established headache. The medication should be administered at the first sign of headache.\n\n There exist some limitations as to the maximum number of tablets that can be taken per day per week. Different sources of drug information may carry different information, and patients are encouraged to ask their pharmacist or prescriber about such details.\n\nCafergot is currently available as a generic drug (ergotamine tartrate/caffeine)\n\nMechanism of action\n\n According to a topic review on UpToDate, ergotamine and dihydroergotamine (DHE 45) bind to 5HT 1b/d receptors, just as triptans do.This along with binding to other serotonergic and dopaminergic receptors is their presumed mechanism of action in treating migraine.\n\nAdverse effects\n\nBecause the vasoconstrictive effects of ergotamine and caffeine are not selective for the brain, adverse effects due to systemic vasoconstriction can occur. Cold feet or hands, angina pectoris, myocardial infarction, or dizziness are some examples.\n\nIt has also been shown to be associated with mitral valve stenosis.\n\nReferences \n\nAntimigraine drugs\nCombination drugs”

Character: Pharmacist

Scenario: Advising a patient on the proper usage of Cafergot, including timing and dosage limits.

Original Query: What is the optimal timing for administering Cafergot to treat migraine headaches?
Rewritten Query: At which temporal juncture is it considered most optimal to commence administration of Cafergot for the alleviation of cephalalgic discomfort characteristic of a migraine?

Hard Negative 1:
“The importance of adherence to a prescribed treatment regimen cannot be overstated, especially when managing chronic conditions such as hypertension and diabetes. Medications for these diseases, while different in function and timing from migraine treatments like Cafergot, require consistent and timely dosing to maintain health and prevent complications. For example, antihypertensive drugs must be taken daily to effectively control blood pressure and reduce the risk of heart attack and stroke. Similarly, diabetic patients must monitor their blood sugar levels regularly and administer insulin or oral hypoglycemic agents as directed to avoid hyperglycemic or hypoglycemic episodes. Although the precise timing may differ from abortive headache therapies, the principle of timing in medication administration is universally critical. Patients are advised to follow the specific instructions provided by their healthcare provider or pharmacist to achieve the best outcomes from their medication regimen. Furthermore, lifestyle modifications, such as diet and exercise, also play a vital role in the management of these conditions and should be initiated in conjunction with pharmacotherapy for an integrated approach to treatment.”

Hard Negative 2:
“Caffeine and its Role in Pain Relief: An Overview\n\nCaffeine, a central nervous system stimulant, has been widely recognized for its ability to increase alertness and alleviate fatigue. Commonly found in various beverages such as coffee, tea, and energy drinks, caffeine is also included in certain pain relief medications. Its application in pain management is based on its pharmacological properties that enhance the efficacy of other analgesic compounds.\n\nAlthough not a primary treatment for migraine pain, caffeine is sometimes combined with analgesics like acetaminophen or aspirin to increase their effectiveness. The precise timing for administration of such combination therapies is generally flexible and tailored to individual patient needs. Unlike migraine-specific treatments, these over-the-counter remedies aim to reduce the severity of pain after onset of symptoms.\n\nResearch into caffeine’s role in pain relief extends beyond headaches to muscle soreness and other types of pain. While it possesses some anti-inflammatory properties, the exact mechanism through which caffeine exerts its effect on pain pathways is still being investigated. However, it is thought to involve adenosine receptor antagonism.\n\nKnowing the right amount of caffeine consumption for pain relief is crucial since excessive intake can cause side effects such as jitteriness, insomnia, and an increased heart rate. As with any medication or supplement, users should consult healthcare professionals to determine the appropriate dosage for their condition.”
(a) Model dimension comparison results (English).
(b) Model dimension comparison results (Multilingual).
(c) Domain dimension comparison results (English, large-size embedding models).
(d) Domain dimension comparison results (Multilingual, large-size embedding models).
(e) Language dimension comparison results (Multilingual, large-size embedding models).

Figure 5: AIR-BENCH can distinguish models in different dimensions, including model dimension, domain dimension, and language dimension. For detailed information of the models appearing in this figure, please refer to Table 15. The detailed metric value and additional results on other model size are all available in Appendix F.2.
(a) Domain dimension comparison results (English, base-size embedding models).

(b) Domain dimension comparison results (Multilingual, base-size embedding models).

(c) Domain dimension comparison results (English, LLM-based embedding models).

(d) Domain dimension comparison results (Multilingual, LLM-based embedding models).

(e) Language dimension comparison results (Multilingual, base-size embedding models).

(f) Language dimension comparison results in multilingual datasets (LLM-based embedding models).

Figure 6: Additional results indicating that AIR-BENCH can distinguish models in different dimensions. For detailed information of the models appearing in this figure, please refer to Table 15.
<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Model Link</th>
  </tr>
  <tr>
    <td colspan="3"><b>Lexical Method</b></td>
  </tr>
  <tr>
    <td>BM25 (Robertson and Zaragoza, 2009)</td>
    <td>-</td>
    <td><a href="https://github.com/castorini/pyserini">https://github.com/castorini/pyserini</a></td>
  </tr>
  <tr>
    <td colspan="3"><b>English Embedding Models</b></td>
  </tr>
  <tr>
    <td>bge-small-en-v1.5 (Xiao et al., 2024)</td>
    <td>33.4M</td>
    <td>https://huggingface.co/BAAI/bge-small-en-v1.5</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5 (Xiao et al., 2024)</td>
    <td>109M</td>
    <td>https://huggingface.co/BAAI/bge-base-en-v1.5</td>
  </tr>
  <tr>
    <td>bge-large-en-v1.5 (Xiao et al., 2024)</td>
    <td>335M</td>
    <td>https://huggingface.co/BAAI/bge-large-en-v1.5</td>
  </tr>
  <tr>
    <td>bge-en-icl (Li et al., 2024)</td>
    <td>7.11B</td>
    <td>https://huggingface.co/BAAI/bge-en-icl</td>
  </tr>
  <tr>
    <td>bge-en-icl-e5data (Li et al., 2024)</td>
    <td>7.11B</td>
    <td>https://huggingface.co/BAAI/bge-en-icl-e5data</td>
  </tr>
  <tr>
    <td>e5-small-v2 (Wang et al., 2022b)</td>
    <td>33.4M</td>
    <td>https://huggingface.co/intfloat/e5-small-v2</td>
  </tr>
  <tr>
    <td>e5-small-v2 (Wang et al., 2022b)</td>
    <td>109M</td>
    <td>https://huggingface.co/intfloat/e5-small-v2</td>
  </tr>
  <tr>
    <td>e5-large-v2 (Wang et al., 2022b)</td>
    <td>335M</td>
    <td>https://huggingface.co/intfloat/e5-large-v2</td>
  </tr>
  <tr>
    <td>gte-small (Li et al., 2023b)</td>
    <td>33.4M</td>
    <td>https://huggingface.co/thenlper/gte-small</td>
  </tr>
  <tr>
    <td>gte-base (Li et al., 2023b)</td>
    <td>109M</td>
    <td>https://huggingface.co/thenlper/gte-base</td>
  </tr>
  <tr>
    <td>gte-large (Li et al., 2023b)</td>
    <td>335M</td>
    <td>https://huggingface.co/thenlper/gte-large</td>
  </tr>
  <tr>
    <td>gte-large-en-icl (Li et al., 2023b)</td>
    <td>438M</td>
    <td>https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5</td>
  </tr>
  <tr>
    <td>replicate-7B-hf-embedding (Ma et al., 2023)</td>
    <td>6.74B</td>
    <td>https://huggingface.co/replicate/replicate-7B-hf-embedding</td>
  </tr>
  <tr>
    <td>SFR-Embedding-Mistral</td>
    <td>7.11B</td>
    <td>https://huggingface.co/Salesforce/SFR-Embedding-Mistral</td>
  </tr>
  <tr>
    <td>SFR-Embedding-2_R</td>
    <td>7.11B</td>
    <td>https://huggingface.co/Salesforce/SFR-Embedding-2_R</td>
  </tr>
  <tr>
    <td>NV-Embed-v1 (Lee et al., 2024a)</td>
    <td>7.85B</td>
    <td>https://huggingface.co/nvidia/NV-Embed-v1</td>
  </tr>
  <tr>
    <td>NV-Embed-v2 (Lee et al., 2024a)</td>
    <td>7.85B</td>
    <td>https://huggingface.co/nvidia/NV-Embed-v2</td>
  </tr>
  <tr>
    <td>Mistral-Embed-v1 (Kim et al., 2024)</td>
    <td>7.11B</td>
    <td>https://huggingface.co/Salesforce/Mistral-Embed-v1</td>
  </tr>
  <tr>
    <td>simlm-base-msmarco-finetuned (Wang et al., 2022a)</td>
    <td>110M</td>
    <td>https://huggingface.co/intfloat/simlm-base-msmarco-finetuned</td>
  </tr>
  <tr>
    <td>msmarco-roberta-base-ance-firstp (Xiong et al., 2021)</td>
    <td>125M</td>
    <td>https://huggingface.co/sentence-transformers/msmarco-roberta-base-ance-firstp</td>
  </tr>
  <tr>
    <td>contriever-msmarco (Izacard et al., 2022)</td>
    <td>109M</td>
    <td>https://huggingface.co/facebook/contriever-msmarco</td>
  </tr>
  <tr>
    <td colspan="3"><b>Multilingual Embedding Models</b></td>
  </tr>
  <tr>
    <td>bge-m3 (Chen et al., 2024b)</td>
    <td>568M</td>
    <td>https://huggingface.co/BAAI/bge-m3</td>
  </tr>
  <tr>
    <td>bge-multilingual-gemma2 (Li et al., 2024)</td>
    <td>9.24B</td>
    <td>https://huggingface.co/BAAI/bge-multilingual-gemma2</td>
  </tr>
  <tr>
    <td>jina-embeddings-v3 (Sturna et al., 2024)</td>
    <td>572M</td>
    <td>https://huggingface.co/jinaai/jina-embeddings-v3</td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct (Wang et al., 2023)</td>
    <td>7.11B</td>
    <td>https://huggingface.co/intfloat/e5-mistral-7b-instruct</td>
  </tr>
  <tr>
    <td>multilingual-e5-small (Wang et al., 2024)</td>
    <td>118M</td>
    <td>https://huggingface.co/intfloat/multilingual-e5-small</td>
  </tr>
  <tr>
    <td>multilingual-e5-base (Wang et al., 2024)</td>
    <td>278M</td>
    <td>https://huggingface.co/intfloat/multilingual-e5-base</td>
  </tr>
  <tr>
    <td>multilingual-e5-large (Wang et al., 2024)</td>
    <td>560M</td>
    <td>https://huggingface.co/intfloat/multilingual-e5-large</td>
  </tr>
  <tr>
    <td>multilingual-e5-large-instruct (Wang et al., 2024)</td>
    <td>560M</td>
    <td>https://huggingface.co/intfloat/multilingual-e5-large-instruct</td>
  </tr>
  <tr>
    <td>gte-multilingual-base (Zhang et al., 2024)</td>
    <td>305M</td>
    <td>https://huggingface.co/Alibaba-NLP/gte-multilingual-base</td>
  </tr>
  <tr>
    <td>bce-embedding-base_v1 (NetEase Youdao, 2023)</td>
    <td>278M</td>
    <td>https://huggingface.co/maidalun102/bce-embedding-base_v1</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-instruct (Li et al., 2023b)</td>
    <td>1.78B</td>
    <td>https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct</td>
  </tr>
  <tr>
    <td>gte-Qwen2-7B-instruct (Li et al., 2023b)</td>
    <td>7.61B</td>
    <td>https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct</td>
  </tr>
  <tr>
    <td colspan="3"><b>Re-ranking Models</b></td>
  </tr>
  <tr>
    <td>bge-ranker-large (Xiao et al., 2024)</td>
    <td>560M</td>
    <td>https://huggingface.co/BAAI/bge-ranker-large</td>
  </tr>
  <tr>
    <td>bge-ranker-v2-m3</td>
    <td>568M</td>
    <td>https://huggingface.co/BAAI/bge-ranker-v2-m3</td>
  </tr>
  <tr>
    <td>bge-ranker-v2-gemma</td>
    <td>2.51B</td>
    <td>https://huggingface.co/BAAI/bge-ranker-v2-gemma</td>
  </tr>
  <tr>
    <td>bce-ranker-base_v1 (NetEase Youdao, 2023)</td>
    <td>278M</td>
    <td>https://huggingface.co/maidalun102/bce-ranker-base_v1</td>
  </tr>
  <tr>
    <td>msmarco-mMiniLMv2-L12-H384-v1</td>
    <td>118M</td>
    <td>https://huggingface.co/msmarco/msmarco-mMiniLMv2-L12-H384-v1</td>
  </tr>
</table>

Table 15: Detailed information on all of the models appearing in our paper.
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th rowspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="13">qa</td>
    <td>arxiv</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>222,877</td>
    <td>334</td>
    <td>test</td>
    <td>1,731</td>
    <td>19</td>
    <td>5,340</td>
    <td>6,288</td>
  </tr>
  <tr>
    <td>finance</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>Reuters/21578 (Lewis, 1997)</td>
    <td>CC BY 4.0</td>
    <td>26,226</td>
    <td>202</td>
    <td>test</td>
    <td>1,585</td>
    <td>17</td>
    <td>3,357</td>
    <td>5,595</td>
  </tr>
  <tr>
    <td></td>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>Duxiaoman-DIFinCorpus</td>
    <td>Apache 2.0</td>
    <td>2,398,095</td>
    <td>1,616</td>
    <td>test</td>
    <td>1,805</td>
    <td>29</td>
    <td>7,836</td>
    <td>7,211</td>
  </tr>
  <tr>
    <td>healthcare</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>PubMedQA (Jin et al., 2019)</td>
    <td>MIT</td>
    <td>847,395</td>
    <td>103</td>
    <td>test</td>
    <td>1,707</td>
    <td>19</td>
    <td>5,052</td>
    <td>7,023</td>
  </tr>
  <tr>
    <td></td>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>Huatou-26M (Li et al., 2023a)</td>
    <td>Apache 2.0</td>
    <td>360,218</td>
    <td>751</td>
    <td>test</td>
    <td>1,874</td>
    <td>31</td>
    <td>10,029</td>
    <td>7,336</td>
  </tr>
  <tr>
    <td>law</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>Pile of Law (Henderson* et al., 2022)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>141,678</td>
    <td>1,509</td>
    <td>test</td>
    <td>1,801</td>
    <td>19</td>
    <td>5,372</td>
    <td>6,574</td>
  </tr>
  <tr>
    <td>news</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>CC-News (Hamborg et al., 2017)</td>
    <td>Unknown</td>
    <td>574,417</td>
    <td>531</td>
    <td>test</td>
    <td>1,614</td>
    <td>16</td>
    <td>5,798</td>
    <td>6,784</td>
  </tr>
  <tr>
    <td></td>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>infotext/multilingual_cc_news</td>
    <td>Unknown</td>
    <td>935,162</td>
    <td>1,263</td>
    <td>test</td>
    <td>1,697</td>
    <td>31</td>
    <td>7,381</td>
    <td>6,618</td>
  </tr>
  <tr>
    <td>web</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>mC4 (Raffel et al., 2020)</td>
    <td>ODC-BY</td>
    <td>2,459,587</td>
    <td>840</td>
    <td>test</td>
    <td>1,707</td>
    <td>16</td>
    <td>5,543</td>
    <td>7,439</td>
  </tr>
  <tr>
    <td></td>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>mC4 (Raffel et al., 2020)</td>
    <td>ODC-BY</td>
    <td>956,699</td>
    <td>1,208</td>
    <td>test</td>
    <td>1,683</td>
    <td>29</td>
    <td>6,250</td>
    <td>6,721</td>
  </tr>
  <tr>
    <td>wiki</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>Wikipedia 20240401</td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>6,738,498</td>
    <td>667</td>
    <td>test</td>
    <td>1,727</td>
    <td>17</td>
    <td>4,260</td>
    <td>7,882</td>
  </tr>
  <tr>
    <td></td>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>Wikipedia 20240401</td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,161,226</td>
    <td>557</td>
    <td>test</td>
    <td>1,679</td>
    <td>30</td>
    <td>4,745</td>
    <td>6,963</td>
  </tr>
  <tr>
    <td>web (msmarco)</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>MS MARCO (Bajaj et al., 2016)</td>
    <td>MIT</td>
    <td>8,872,840</td>
    <td>81</td>
    <td>test</td>
    <td>6,319</td>
    <td>16</td>
    <td>31,447</td>
    <td>26,828</td>
  </tr>
  <tr>
    <td rowspan="4">long-doc</td>
    <td>arxiv</td>
    <td>English</td>
    <td>en</td>
    <td>gemini</td>
    <td>Paper of Gemini</td>
    <td>CC BY 4.0</td>
    <td>276</td>
    <td>136</td>
    <td>test</td>
    <td>249</td>
    <td>18</td>
    <td>249</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>English</td>
    <td>en</td>
    <td>gpt3</td>
    <td>Paper of GPT-3</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>515</td>
    <td>137</td>
    <td>test</td>
    <td>337</td>
    <td>16</td>
    <td>496</td>
    <td>0</td>
  </tr>
  <tr>
    <td>healthcare</td>
    <td>English</td>
    <td>en</td>
    <td>llama2</td>
    <td>Paper of Llama 2</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>566</td>
    <td>136</td>
    <td>test</td>
    <td>326</td>
    <td>18</td>
    <td>635</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>English</td>
    <td>en</td>
    <td>ilm-survey</td>
    <td>Survey of LLM</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>1,144</td>
    <td>135</td>
    <td>test</td>
    <td>357</td>
    <td>17</td>
    <td>924</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="5">book</td>
    <td>book</td>
    <td>English</td>
    <td>en</td>
    <td>a-brief-history-of-time_stephen-hawking</td>
    <td><i>A Brief History of Time</i></td>
    <td>Unknown</td>
    <td>778</td>
    <td>127</td>
    <td>test</td>
    <td>370</td>
    <td>16</td>
    <td>876</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>origin-of-species_darwin</td>
    <td><i>On the Origin of Species</i></td>
    <td>Unknown</td>
    <td>1,758</td>
    <td>126</td>
    <td>test</td>
    <td>366</td>
    <td>16</td>
    <td>1,145</td>
    <td>0</td>
  </tr>
  <tr>
    <td>healthcare</td>
    <td>English</td>
    <td>en</td>
    <td>pubmed_100K-200K_1</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>899</td>
    <td>133</td>
    <td>test</td>
    <td>372</td>
    <td>20</td>
    <td>1,008</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>pubmed_100K-200K_2</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>872</td>
    <td>136</td>
    <td>test</td>
    <td>355</td>
    <td>18</td>
    <td>980</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>pubmed_100K-200K_3</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>873</td>
    <td>133</td>
    <td>test</td>
    <td>357</td>
    <td>19</td>
    <td>978</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>pubmed_30K-40K_10-merged</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>2,154</td>
    <td>133</td>
    <td>test</td>
    <td>368</td>
    <td>18</td>
    <td>1,485</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>pubmed_40K-50K_5-merged</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>1,731</td>
    <td>136</td>
    <td>test</td>
    <td>336</td>
    <td>21</td>
    <td>1,046</td>
    <td>0</td>
  </tr>
  <tr>
    <td>law</td>
    <td>English</td>
    <td>en</td>
    <td>lex_files_300K-400K</td>
    <td>LexFiles (Chatkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>2,797</td>
    <td>137</td>
    <td>test</td>
    <td>339</td>
    <td>15</td>
    <td>1,307</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>lex_files_400K-500K</td>
    <td>LexFiles (Chatkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>3,320</td>
    <td>137</td>
    <td>test</td>
    <td>333</td>
    <td>17</td>
    <td>1,427</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>lex_files_500K-600K</td>
    <td>LexFiles (Chatkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>4,087</td>
    <td>136</td>
    <td>test</td>
    <td>346</td>
    <td>17</td>
    <td>1,324</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td>lex_files_600K-700K</td>
    <td>LexFiles (Chatkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>5,049</td>
    <td>138</td>
    <td>test</td>
    <td>338</td>
    <td>18</td>
    <td>1,442</td>
    <td>0</td>
  </tr>
</table>

Table 16: Statistics of all datasets in AIR-BENCH 24.04.
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th colspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>ISO Code</th>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="5">arxiv</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>222,877</td>
    <td>334</td>
    <td>dev</td>
    <td>346</td>
    <td>19</td>
    <td>1,091</td>
    <td>1,230</td>
  </tr>
  <tr>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>Reuters-21578 (Lewis, 1997)</td>
    <td>CC BY 4.0</td>
    <td>26,226</td>
    <td>202</td>
    <td>dev</td>
    <td>317</td>
    <td>17</td>
    <td>677</td>
    <td>1,122</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>default</td>
    <td>asas-ar/financial_news</td>
    <td>Apache 2.0</td>
    <td>11,235</td>
    <td>397</td>
    <td>dev</td>
    <td>293</td>
    <td>49</td>
    <td>635</td>
    <td>727</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td>CoFiF (Daudert and Ahmadi, 2019)</td>
    <td>CC BY-NC 4.0</td>
    <td>1,006,801</td>
    <td>92</td>
    <td>dev</td>
    <td>310</td>
    <td>21</td>
    <td>1,841</td>
    <td>1,071</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>Duxiaoman-DL/FinCorpus</td>
    <td>Apache 2.0</td>
    <td>1,014,974</td>
    <td>1,613</td>
    <td>dev</td>
    <td>361</td>
    <td>29</td>
    <td>1,516</td>
    <td>1,471</td>
  </tr>
  <tr>
    <td rowspan="5">finance</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>PubMedQA (Jin et al., 2019)</td>
    <td>MIT</td>
    <td>847,395</td>
    <td>103</td>
    <td>dev</td>
    <td>341</td>
    <td>20</td>
    <td>1,008</td>
    <td>1,382</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>default</td>
    <td>MLSUM (Scialom et al., 2020)</td>
    <td>MIT</td>
    <td>27,934</td>
    <td>909</td>
    <td>dev</td>
    <td>1,366</td>
    <td>19</td>
    <td>4,044</td>
    <td>5,641</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>default</td>
    <td>Multilingual Medical Corpora (Villena, 2019)</td>
    <td>Unknown</td>
    <td>1,006,093</td>
    <td>60</td>
    <td>dev</td>
    <td>300</td>
    <td>21</td>
    <td>1,210</td>
    <td>930</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td>Multilingual Medical Corpora (Villena, 2019)</td>
    <td>Unknown</td>
    <td>972,938</td>
    <td>202</td>
    <td>dev</td>
    <td>331</td>
    <td>23</td>
    <td>1,885</td>
    <td>1,261</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td>Huatuo-26M (Li et al., 2023a)</td>
    <td>Apache 2.0</td>
    <td>360,218</td>
    <td>751</td>
    <td>dev</td>
    <td>374</td>
    <td>31</td>
    <td>2,030</td>
    <td>1,490</td>
  </tr>
  <tr>
    <td rowspan="3">qa</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>Pile of Law (Henderson* et al., 2022)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>141,678</td>
    <td>1,509</td>
    <td>dev</td>
    <td>360</td>
    <td>20</td>
    <td>1,080</td>
    <td>1,341</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>default</td>
    <td>MultiLegalPile (Niklaus et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>752,913</td>
    <td>3,361</td>
    <td>dev</td>
    <td>345</td>
    <td>24</td>
    <td>1,373</td>
    <td>1,099</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td>MultiLegalPile (Niklaus et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>649,017</td>
    <td>2,540</td>
    <td>dev</td>
    <td>348</td>
    <td>23</td>
    <td>1,371</td>
    <td>1,260</td>
  </tr>
  <tr>
    <td>science</td>
    <td>Russian</td>
    <td>ru</td>
    <td>default</td>
    <td>mksa-iai-msu-lab/ru_sci_bench</td>
    <td>MIT</td>
    <td>200,532</td>
    <td>347</td>
    <td>dev</td>
    <td>345</td>
    <td>34</td>
    <td>1,577</td>
    <td>1,160</td>
  </tr>
  <tr>
    <td>web (msmarco)</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td>MS MARCO (Bajaj et al., 2016)</td>
    <td>MIT</td>
    <td>8,872,840</td>
    <td>81</td>
    <td>dev</td>
    <td>6,319</td>
    <td>16</td>
    <td>31,447</td>
    <td>26,828</td>
  </tr>
</table>

Table 17: Statistics of all datasets in AIR-BENCH 24.05 (Part 1).
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th colspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>ISO Code</th>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="13">qa news</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td><b>CC-News (Hamborg et al., 2017)</b></td>
    <td>Unknown</td>
    <td>574,417</td>
    <td>531</td>
    <td>dev</td>
    <td>322</td>
    <td>16</td>
    <td>1,206</td>
    <td>1,375</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,006,308</td>
    <td>992</td>
    <td>dev</td>
    <td>349</td>
    <td>42</td>
    <td>1,810</td>
    <td>1,307</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>bn</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>20,681</td>
    <td>912</td>
    <td>test</td>
    <td>289</td>
    <td>84</td>
    <td>562</td>
    <td>741</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,006,876</td>
    <td>659</td>
    <td>dev</td>
    <td>336</td>
    <td>23</td>
    <td>1,448</td>
    <td>1,234</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,007,104</td>
    <td>615</td>
    <td>dev</td>
    <td>337</td>
    <td>23</td>
    <td>1,541</td>
    <td>1,240</td>
  </tr>
  <tr>
    <td>Persian</td>
    <td>fa</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,002,797</td>
    <td>1,351</td>
    <td>dev</td>
    <td>346</td>
    <td>50</td>
    <td>1,952</td>
    <td>1,341</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,007,592</td>
    <td>641</td>
    <td>dev</td>
    <td>345</td>
    <td>23</td>
    <td>1,548</td>
    <td>1,424</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>hi</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,006,218</td>
    <td>1,465</td>
    <td>dev</td>
    <td>349</td>
    <td>66</td>
    <td>1,716</td>
    <td>1,264</td>
  </tr>
  <tr>
    <td>Indonesian</td>
    <td>id</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,007,724</td>
    <td>548</td>
    <td>dev</td>
    <td>338</td>
    <td>24</td>
    <td>1,799</td>
    <td>1,397</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>ja</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>834,364</td>
    <td>1,559</td>
    <td>dev</td>
    <td>344</td>
    <td>35</td>
    <td>1,817</td>
    <td>1,334</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>ko</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,006,798</td>
    <td>1,072</td>
    <td>dev</td>
    <td>361</td>
    <td>34</td>
    <td>1,967</td>
    <td>1,413</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>ru</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>1,004,550</td>
    <td>776</td>
    <td>dev</td>
    <td>337</td>
    <td>34</td>
    <td>1,676</td>
    <td>1,301</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td><a href="http://intfloat/multilingual_cc_news">intfloat/multilingual_cc_news</a></td>
    <td>Unknown</td>
    <td>935,162</td>
    <td>1,263</td>
    <td>dev</td>
    <td>339</td>
    <td>32</td>
    <td>1,477</td>
    <td>1,354</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>test</td>
    <td>1,358</td>
    <td>30</td>
    <td>5,904</td>
    <td>5,264</td>
  </tr>
</table>

Table 18: Statistics of all datasets in AIR-BENCH 24.05 (Part 2).
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th colspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>ISO Code</th>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="13">qa</td>
    <td rowspan="13">web</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>1,012,910</td>
    <td>838</td>
    <td>dev</td>
    <td>341</td>
    <td>16</td>
    <td>1,087</td>
    <td>1,511</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>165,902</td>
    <td>1,686</td>
    <td>test</td>
    <td>1,366</td>
    <td>16</td>
    <td>4,456</td>
    <td>5,928</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>bn</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>45,375</td>
    <td>2,161</td>
    <td>dev</td>
    <td>334</td>
    <td>42</td>
    <td>1,133</td>
    <td>1,119</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>441,182</td>
    <td>1,025</td>
    <td>test</td>
    <td>362</td>
    <td>73</td>
    <td>1,142</td>
    <td>1,073</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>403,020</td>
    <td>912</td>
    <td>dev</td>
    <td>341</td>
    <td>23</td>
    <td>1,317</td>
    <td>1,281</td>
  </tr>
  <tr>
    <td>Persian</td>
    <td>fa</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>181,463</td>
    <td>2,114</td>
    <td>test</td>
    <td>338</td>
    <td>49</td>
    <td>1,389</td>
    <td>1,160</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>387,210</td>
    <td>1,076</td>
    <td>dev</td>
    <td>364</td>
    <td>20</td>
    <td>1,444</td>
    <td>1,451</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>hi</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>50,501</td>
    <td>2,396</td>
    <td>test</td>
    <td>1,457</td>
    <td>20</td>
    <td>5,572</td>
    <td>5,552</td>
  </tr>
  <tr>
    <td>Indonesian</td>
    <td>id</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>245,878</td>
    <td>1,059</td>
    <td>dev</td>
    <td>355</td>
    <td>68</td>
    <td>1,180</td>
    <td>1,180</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>ja</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>547,419</td>
    <td>1,026</td>
    <td>test</td>
    <td>1,293</td>
    <td>36</td>
    <td>4,473</td>
    <td>4,976</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>ko</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>250,605</td>
    <td>1,137</td>
    <td>dev</td>
    <td>327</td>
    <td>34</td>
    <td>1,156</td>
    <td>1,083</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>ru</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>490,581</td>
    <td>1,266</td>
    <td>test</td>
    <td>1,309</td>
    <td>36</td>
    <td>4,239</td>
    <td>4,457</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>956,699</td>
    <td>1,208</td>
    <td>dev</td>
    <td>324</td>
    <td>32</td>
    <td>1,330</td>
    <td>1,277</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td><a href="https://github.com/google-research/mc4">mC4 (Raffel et al., 2020)</a></td>
    <td>ODC-BY</td>
    <td>956,699</td>
    <td>1,208</td>
    <td>test</td>
    <td>1,347</td>
    <td>29</td>
    <td>5,020</td>
    <td>5,355</td>
  </tr>
</table>

Table 19: Statistics of all datasets in AIR-BENCH 24.05 (Part 3).
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th colspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>ISO Code</th>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="13">qa</td>
    <td rowspan="13">wiki</td>
    <td>English</td>
    <td>en</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,012,092</td>
    <td>665</td>
    <td>dev</td>
    <td>345</td>
    <td>18</td>
    <td>863</td>
    <td>1,576</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,232</td>
    <td>787</td>
    <td>test</td>
    <td>1,382</td>
    <td>17</td>
    <td>3,397</td>
    <td>6,306</td>
  </tr>
  <tr>
    <td>Bengali</td>
    <td>bn</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>152,064</td>
    <td>2,129</td>
    <td>dev</td>
    <td>364</td>
    <td>69</td>
    <td>1,016</td>
    <td>1,542</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,186</td>
    <td>891</td>
    <td>dev</td>
    <td>350</td>
    <td>23</td>
    <td>817</td>
    <td>1,481</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,147</td>
    <td>801</td>
    <td>dev</td>
    <td>345</td>
    <td>22</td>
    <td>879</td>
    <td>1,451</td>
  </tr>
  <tr>
    <td>Persian</td>
    <td>fa</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>999,223</td>
    <td>627</td>
    <td>dev</td>
    <td>332</td>
    <td>41</td>
    <td>1,179</td>
    <td>1,444</td>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,270</td>
    <td>779</td>
    <td>dev</td>
    <td>356</td>
    <td>20</td>
    <td>768</td>
    <td>1,496</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>hi</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>162,188</td>
    <td>997</td>
    <td>dev</td>
    <td>340</td>
    <td>59</td>
    <td>995</td>
    <td>1,324</td>
  </tr>
  <tr>
    <td>Indonesian</td>
    <td>id</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>687,513</td>
    <td>451</td>
    <td>dev</td>
    <td>343</td>
    <td>22</td>
    <td>1,003</td>
    <td>1,365</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>ja</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,365</td>
    <td>1,470</td>
    <td>dev</td>
    <td>358</td>
    <td>30</td>
    <td>1,099</td>
    <td>1,537</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>ko</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>665,227</td>
    <td>775</td>
    <td>dev</td>
    <td>346</td>
    <td>39</td>
    <td>1,109</td>
    <td>1,423</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>ru</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,008,405</td>
    <td>1,211</td>
    <td>dev</td>
    <td>365</td>
    <td>30</td>
    <td>1,154</td>
    <td>1,505</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>default</td>
    <td><a href="https://www.aclweb.org/anthology/W20-401/">Wikipedia 20240401</a></td>
    <td>CC BY-SA 3.0, GFDL</td>
    <td>1,011,604</td>
    <td>557</td>
    <td>dev</td>
    <td>335</td>
    <td>30</td>
    <td>952</td>
    <td>1,301</td>
  </tr>
  <tr>
    <td>test</td>
    <td>1,344</td>
    <td>30</td>
    <td>3,793</td>
    <td>5,662</td>
  </tr>
</table>

Table 20: Statistics of all datasets in AIR-BENCH 24.05 (Part 4).
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Domain</th>
    <th rowspan="2">Language</th>
    <th rowspan="2">Dataset Name</th>
    <th colspan="2">Source of Corpus</th>
    <th rowspan="2">#corpus</th>
    <th rowspan="2">Avg #token of corpus</th>
    <th rowspan="2">Split</th>
    <th rowspan="2"># of queries</th>
    <th rowspan="2">Avg #token of queries</th>
    <th rowspan="2"># of positives</th>
    <th rowspan="2"># of hard negatives</th>
  </tr>
  <tr>
    <th>Link & Citation</th>
    <th>License</th>
  </tr>
  <tr>
    <td rowspan="4">long-doc</td>
    <td rowspan="4">arxiv</td>
    <td rowspan="4">English</td>
    <td>gemini</td>
    <td>Paper of Gemini</td>
    <td>CC BY 4.0</td>
    <td>276</td>
    <td>126</td>
    <td>test</td>
    <td>249</td>
    <td>18</td>
    <td>249</td>
    <td>0</td>
  </tr>
  <tr>
    <td>gpt3</td>
    <td>Paper of GPT-3</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>515</td>
    <td>137</td>
    <td>test</td>
    <td>337</td>
    <td>16</td>
    <td>496</td>
    <td>0</td>
  </tr>
  <tr>
    <td>llama2</td>
    <td>Paper of Llama 2</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>566</td>
    <td>136</td>
    <td>dev</td>
    <td>326</td>
    <td>18</td>
    <td>635</td>
    <td>0</td>
  </tr>
  <tr>
    <td>llm-survey</td>
    <td>Survey of LLM</td>
    <td>arXiv.org perpetual, non-exclusive license 1.0</td>
    <td>1,144</td>
    <td>135</td>
    <td>test</td>
    <td>357</td>
    <td>17</td>
    <td>924</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="2">book</td>
    <td rowspan="2">English</td>
    <td rowspan="2">en</td>
    <td>a-brief-history-of-time_stephen-hawking</td>
    <td><i>A Brief History of Time</i></td>
    <td>Unknown</td>
    <td>778</td>
    <td>127</td>
    <td>dev</td>
    <td>370</td>
    <td>16</td>
    <td>876</td>
    <td>0</td>
  </tr>
  <tr>
    <td>origin-of-species_darwin</td>
    <td><i>On the Origin of Species</i></td>
    <td>Unknown</td>
    <td>1,758</td>
    <td>126</td>
    <td>test</td>
    <td>366</td>
    <td>16</td>
    <td>1,145</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="5">healthcare</td>
    <td rowspan="5">English</td>
    <td rowspan="5">en</td>
    <td>pubmed_100K-200K_1</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>899</td>
    <td>133</td>
    <td>test</td>
    <td>372</td>
    <td>20</td>
    <td>1,008</td>
    <td>0</td>
  </tr>
  <tr>
    <td>pubmed_100K-200K_2</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>872</td>
    <td>136</td>
    <td>test</td>
    <td>355</td>
    <td>18</td>
    <td>980</td>
    <td>0</td>
  </tr>
  <tr>
    <td>pubmed_100K-200K_3</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>873</td>
    <td>133</td>
    <td>dev</td>
    <td>357</td>
    <td>19</td>
    <td>978</td>
    <td>0</td>
  </tr>
  <tr>
    <td>pubmed_30K-40K_10-merged</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>2,154</td>
    <td>133</td>
    <td>test</td>
    <td>368</td>
    <td>18</td>
    <td>1,485</td>
    <td>0</td>
  </tr>
  <tr>
    <td>pubmed_40K-50K_5-merged</td>
    <td>long-summarization (Cohan et al., 2018)</td>
    <td>Apache 2.0</td>
    <td>1,731</td>
    <td>136</td>
    <td>test</td>
    <td>336</td>
    <td>21</td>
    <td>1,046</td>
    <td>0</td>
  </tr>
  <tr>
    <td rowspan="3">law</td>
    <td rowspan="3">English</td>
    <td rowspan="3">en</td>
    <td>lex_files_300K-400K</td>
    <td>LexFiles (Chalkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>2,797</td>
    <td>137</td>
    <td>dev</td>
    <td>339</td>
    <td>15</td>
    <td>1,307</td>
    <td>0</td>
  </tr>
  <tr>
    <td>lex_files_400K-500K</td>
    <td>LexFiles (Chalkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>3,320</td>
    <td>137</td>
    <td>test</td>
    <td>333</td>
    <td>17</td>
    <td>1,427</td>
    <td>0</td>
  </tr>
  <tr>
    <td>lex_files_500K-600K</td>
    <td>LexFiles (Chalkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>4,087</td>
    <td>136</td>
    <td>test</td>
    <td>346</td>
    <td>17</td>
    <td>1,324</td>
    <td>0</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>lex_files_600K-700K</td>
    <td>LexFiles (Chalkidis et al., 2023)</td>
    <td>CC BY-NC-SA 4.0</td>
    <td>5,049</td>
    <td>138</td>
    <td>test</td>
    <td>338</td>
    <td>18</td>
    <td>1,442</td>
    <td>0</td>
  </tr>
</table>

Table 21: Statistics of all datasets in AIR-BENCH 24.05 (Part 5).
<table>
  <tr>
    <th rowspan="2">Dataset (.)</th>
    <th colspan="3">bge-en-icl (zero-shot)</th>
    <th colspan="3">bge-en-icl-e5data (zero-shot)</th>
    <th colspan="3">bge-*-en-v1.5 small</th>
    <th colspan="3">bge-*-en-v1.5 base</th>
    <th colspan="3">bge-*-en-v1.5 large</th>
    <th colspan="3">NV-Embed-* small</th>
    <th colspan="3">NV-Embed-* base</th>
    <th colspan="3">NV-Embed-* large</th>
    <th colspan="3">gte-large-* v1</th>
    <th colspan="3">gte-large-* v2</th>
    <th colspan="3">SFR-Embedding-Mistral en-v1.5</th>
    <th colspan="3">SFR-Embedding-2_R lora-passage</th>
  </tr>
  <tr>
    <th>bge-en-icl</th>
    <th>bge-en-icl</th>
    <th>bge-en-icl</th>
    <th>small</th>
    <th>base</th>
    <th>large</th>
    <th>small</th>
    <th>base</th>
    <th>large</th>
    <th>small</th>
    <th>base</th>
    <th>large</th>
    <th>small</th>
    <th>base</th>
    <th>large</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
    <th>v1</th>
    <th>v2</th>
  </tr>
  <tr>
    <th colspan="30">QA Task (English, 7 Datasets)</th>
  </tr>
  <tr>
    <td>arxiv_en</td><td>50.43</td><td>50.43</td><td>35.56</td><td>36.44</td><td>38.68</td><td>35.44</td><td>37.56</td><td>38.31</td><td>36.25</td><td>38.09</td><td>39.20</td><td>47.39</td><td>49.88</td><td>40.00</td><td>48.10</td><td>43.58</td><td>47.20</td>
  </tr>
  <tr>
    <td>finance_en</td><td>55.48</td><td>55.48</td><td>44.68</td><td>47.35</td><td>45.59</td><td>49.44</td><td>47.33</td><td>51.34</td><td>47.60</td><td>49.08</td><td>47.91</td><td>51.21</td><td>54.09</td><td>54.33</td><td>58.34</td><td>56.02</td><td>54.14</td>
  </tr>
  <tr>
    <td>healthcare_cn</td><td>57.20</td><td>57.20</td><td>47.23</td><td>49.50</td><td>50.93</td><td>46.48</td><td>48.93</td><td>52.45</td><td>47.48</td><td>49.20</td><td>49.02</td><td>58.98</td><td>59.85</td><td>48.34</td><td>59.09</td><td>55.83</td><td>56.25</td>
  </tr>
  <tr>
    <td>law_en</td><td>28.92</td><td>28.92</td><td>17.12</td><td>16.71</td><td>23.53</td><td>15.67</td><td>16.34</td><td>21.28</td><td>15.89</td><td>15.63</td><td>17.57</td><td>20.77</td><td>25.43</td><td>18.73</td><td>22.52</td><td>21.22</td><td>17.98</td>
  </tr>
  <tr>
    <td>news_en</td><td>54.92</td><td>54.92</td><td>41.38</td><td>40.59</td><td>43.70</td><td>43.37</td><td>42.00</td><td>45.26</td><td>40.70</td><td>43.64</td><td>43.70</td><td>50.93</td><td>52.35</td><td>45.74</td><td>51.20</td><td>50.57</td><td>51.58</td>
  </tr>
  <tr>
    <td>web_en</td><td>60.80</td><td>60.80</td><td>47.04</td><td>46.77</td><td>48.92</td><td>45.27</td><td>46.12</td><td>48.60</td><td>49.55</td><td>50.41</td><td>51.47</td><td>56.46</td><td>58.76</td><td>50.02</td><td>57.44</td><td>55.57</td><td>58.06</td>
  </tr>
  <tr>
    <td>wiki_en</td><td>73.44</td><td>73.44</td><td>62.15</td><td>62.53</td><td>63.05</td><td>65.94</td><td>66.61</td><td>66.54</td><td>65.07</td><td>65.92</td><td>65.96</td><td>71.08</td><td>73.06</td><td>66.59</td><td>72.78</td><td>72.77</td><td>70.84</td>
  </tr>
  <tr>
    <th colspan="30">Long-Doc Task (English, 11 Datasets)</th>
  </tr>
  <tr>
    <td>arxiv_en_gemini</td><td>87.55</td><td>87.55</td><td>69.08</td><td>69.88</td><td>75.50</td><td>72.29</td><td>76.31</td><td>72.69</td><td>71.89</td><td>74.70</td><td>83.94</td><td>85.54</td><td>72.29</td><td>79.92</td><td>78.72</td><td>82.73</td>
  </tr>
  <tr>
    <td>arxiv_en_gpt3</td><td>81.65</td><td>81.65</td><td>65.23</td><td>64.91</td><td>64.37</td><td>65.83</td><td>69.44</td><td>75.07</td><td>66.79</td><td>67.41</td><td>70.62</td><td>83.31</td><td>83.28</td><td>68.30</td><td>76.88</td><td>72.28</td><td>76.81</td>
  </tr>
  <tr>
    <td>arxiv_en_llm_survey</td><td>71.05</td><td>71.05</td><td>46.25</td><td>49.80</td><td>51.32</td><td>52.71</td><td>54.30</td><td>58.07</td><td>49.54</td><td>55.85</td><td>56.58</td><td>67.62</td><td>69.33</td><td>55.94</td><td>63.51</td><td>60.15</td><td>64.71</td>
  </tr>
  <tr>
    <td>book_en_origin_of_species_darwin</td><td>71.07</td><td>71.07</td><td>52.50</td><td>56.09</td><td>60.22</td><td>53.03</td><td>54.07</td><td>61.51</td><td>58.37</td><td>59.76</td><td>63.32</td><td>68.21</td><td>70.77</td><td>51.35</td><td>66.18</td><td>62.49</td><td>66.64</td>
  </tr>
  <tr>
    <td>healthcare_en_pubmed_100k_200k_1</td><td>71.23</td><td>71.23</td><td>57.24</td><td>60.50</td><td>61.68</td><td>58.76</td><td>62.33</td><td>64.62</td><td>56.94</td><td>57.59</td><td>58.76</td><td>68.51</td><td>69.17</td><td>57.78</td><td>63.77</td><td>65.13</td><td>68.17</td>
  </tr>
  <tr>
    <td>healthcare_en_pubmed_100k_200k_2</td><td>75.59</td><td>75.59</td><td>57.23</td><td>59.39</td><td>62.54</td><td>62.14</td><td>66.00</td><td>69.10</td><td>58.30</td><td>61.70</td><td>64.49</td><td>75.53</td><td>77.96</td><td>63.91</td><td>73.15</td><td>70.78</td><td>69.78</td>
  </tr>
  <tr>
    <td>healthcare_en_pubmed_30k_40k_10_merged</td><td>79.00</td><td>79.00</td><td>65.76</td><td>67.33</td><td>69.36</td><td>68.32</td><td>69.74</td><td>74.16</td><td>67.03</td><td>68.50</td><td>68.34</td><td>78.45</td><td>79.98</td><td>64.79</td><td>74.45</td><td>71.97</td><td>77.24</td>
  </tr>
  <tr>
    <td>healthcare_en_pubmed_40k_50k_5_merged</td><td>65.57</td><td>65.57</td><td>52.03</td><td>57.15</td><td>57.28</td><td>57.94</td><td>57.83</td><td>61.35</td><td>50.79</td><td>54.53</td><td>53.57</td><td>64.89</td><td>64.57</td><td>55.14</td><td>61.95</td><td>61.19</td><td>60.55</td>
  </tr>
  <tr>
    <td>law_en_lex_files_400k_500k</td><td>67.73</td><td>67.73</td><td>54.46</td><td>58.07</td><td>60.25</td><td>53.20</td><td>58.06</td><td>62.17</td><td>55.56</td><td>59.03</td><td>62.66</td><td>66.54</td><td>66.06</td><td>58.80</td><td>61.76</td><td>60.35</td><td>64.77</td>
  </tr>
  <tr>
    <td>law_en_lex_files_500k_600k</td><td>68.52</td><td>68.52</td><td>53.03</td><td>60.23</td><td>61.68</td><td>57.98</td><td>60.08</td><td>65.89</td><td>58.75</td><td>61.63</td><td>63.54</td><td>68.14</td><td>71.56</td><td>63.36</td><td>66.31</td><td>63.16</td><td>67.55</td>
  </tr>
  <tr>
    <td>law_en_lex_files_600k_700k</td><td>68.73</td><td>68.73</td><td>49.98</td><td>53.58</td><td>56.30</td><td>51.79</td><td>57.61</td><td>59.56</td><td>53.84</td><td>55.29</td><td>56.53</td><td>67.76</td><td>69.68</td><td>56.18</td><td>61.18</td><td>57.90</td><td>63.97</td>
  </tr>
</table>

Table 22: Detailed evaluation results of English IR models on QA (English, test) datasets and Long-Doc (English, test) datasets of AIR-BENCH 24.05.
<table>
  <tr>
    <th rowspan="2">Dataset (j.)</th>
    <th colspan="11">QA Task (Multilingual, 53 Datasets)</th>
    <th colspan="5">Long-Doc Task (English, 11 Datasets)</th>
  </tr>
  <tr>
    <th>BM25</th><th>BM25 + hge-retoken-d=1 nm</th><th>multilingual-cfs</th><th>bce-QwnetL1+classifier</th><th>bcg-m3</th><th>multilingual-cfs-lr-flarge-unmix</th><th>cfs-mixual-bf-largeunmix</th><th>bce-multilingual-emgemm1</th><th>htt-multilingual-7bnmixualGemmi1</th><th>bce-embedding</th><th>multilingual+cfs-up</th>
    <th>arxiv_zh_div2_m</th><th>arks_csl_csf_div2_m</th><th>setsen_querocks_survey_it</th><th>bock_ONeorganization_sp_species_darwin</th><th>both_uk_nlu_human_6094_1</th><th>br02_t contacting_7shXXX_60K</th><th>entr_knn_urls_60kAcc_600k</th><th>bce군사적_t,d_3mixual.GenMmi1</th>
  </tr>
  <tr><td>arxiv_en_jwrvp_base</td><td>8.16</td><td>#NA</td><td>12.44</td><td>12.41</td><td>43.19</td><td>#NA</td><td>52.72</td><td>55.82</td><td>66.06</td><td>24.01</td><td>32.16</td><td>20.60</td><td>8.68</td></tr>
  <tr><td>bce_rte-mixual(mut)/ermul cancers</td><td>49.32</td><td>46.12</td><td>42.75</td><td>11.16</td><td>30.94</td><td>53.02</td><td>50.71</td><td>#NA</td><td>50.55</td><td>32.16</td><td>33.40</td><td>35.52</td><td>66.80</td></tr>
  <tr><td>nfafvia_fastrc:unsTOKENS Rue/H/t ONpcaHET/bm ALSODE去除2rt.abile</td><td>73.87</td><td>73.37</td><td>81.15</td><td>43.85</td><td>42.33</td><td>22.18</td><td>35.24</td><td>41.89</td><td>55.90</td><td>58.68</td><td>53.24</td><td>41.67</td><td>51.70</td></tr>
  <tr><td>finance_cz</td><td>43.19</td><td>40.04</td><td>34.72</td><td>20.06</td><td>25.21</td><td>45.35</td><td>47.86</td><td>44.39</td><td>53.25</td><td>55.08</td><td>52.24</td><td>52.71</td><td>56.32</td></tr>
  <tr><td>finance_en</td><td>28.61</td><td>34.68</td><td>20.89</td><td>16.52</td><td>30.57</td><td>41.44</td><td>24.73</td><td>38.38</td><td>51.10</td><td>35.47</td><td>39.27</td><td>37.14</td><td>35.10</td></tr>
  <tr><td>finance_de</td><td>27.63</td><td>2.08</td><td>2.59</td><td>3.43</td><td>38.44</td><td>6.32</td><td>20.95</td><td>27.70</td><td>29.52</td><td>32.75</td><td>32.22</td><td>26.36</td><td>30.96</td></tr>
  <tr><td>healthcare_en</td><td>50.16</td><td>32.15</td><td>53.45</td><td>17.54</td><td>32.45</td><td>47.34</td><td>35.74</td><td>40.26</td><td>47.30</td><td>48.02</td><td>47.23</td><td>46.88</td><td>39.43</td></tr>
  <tr><td>healthcare_de</td><td>52.97</td><td>35.28</td><td>55.81</td><td>12.40</td><td>20.98</td><td>36.21</td><td>54.22</td><td>53.12</td><td>54.40</td><td>56.28</td><td>55.05</td><td>49.86</td><td>49.36</td></tr>
  <tr><td>healthcare_zh</td><td>34.84</td><td>38.26</td><td>41.98</td><td>40.78</td><td>32.11</td><td>54.46</td><td>49.02</td><td>54.82</td><td>45.34</td><td>37.54</td><td>47.48</td><td>39.85</td><td>46.42</td></tr>
  <tr><td>intelligence_en</td><td>32.15</td><td>25.07</td><td>36.14</td><td>9.42</td><td>8.42</td><td>19.89</td><td>34.56</td><td>53.63</td><td>27.67</td><td>20.13</td><td>33.76</td><td>18.45</td><td>27.23</td></tr>
  <tr><td>healthcare_de</td><td>20.82</td><td>20.59</td><td>15.78</td><td>18.45</td><td>20.85</td><td>24.00</td><td>20.99</td><td>32.99</td><td>37.26</td><td>48.15</td><td>29.42</td><td>36.39</td><td>38.20</td></tr>
  <tr><td>healthcare_zh</td><td>18.11</td><td>24.38</td><td>29.87</td><td>20.86</td><td>32.62</td><td>39.11</td><td>42.31</td><td>46.30</td><td>37.96</td><td>20.37</td><td>29.34</td><td>25.84</td><td>93.21</td></tr>
  <tr><td>laws.sto</td><td>12.36</td><td>2:2:1:2:</td><td>9.68</td><td>16.95</td><td>9.35</td><td>32.00</td><td>12.43</td><td>18.11</td><td>15.39</td><td>15.72</td><td>15.53</td><td>14.77</td><td>11.15</td></tr>
  <tr><td>law_en</td><td>13.65</td><td>22.05</td><td>9.95</td><td>27.46</td><td>17.63</td><td>20.47</td><td>17.83</td><td>13.00</td><td>12.06</td><td>10.82</td><td>2.64</td><td>48.76</td><td>120.24</td></tr>
  <tr><td>rte_zh</td><td>19.56</td><td>34.17</td><td>-3.33</td><td>47.57</td><td>-2.39</td><td>52.65</td><td>40.61</td><td>52.44</td><td>56.82</td><td>51.03</td><td>51.92</td><td>48.94</td><td>59.50</td></tr>
  <tr><td>arf8_zh</td><td>13.16</td><td>23.19</td><td>7.14</td><td>33.09</td><td>7.49</td><td>42.72</td><td>17.15</td><td>25.11</td><td>15.12</td><td>13.48</td><td>14.91</td><td>11.68</td><td>10.26</td></tr>
  <tr><td>de8_zh</td><td>26.54</td><td>51.87</td><td>36.89</td><td>42.64</td><td>35.93</td><td>47.83</td><td>46.93</td><td>48.20</td><td>58.30</td><td>48.41</td><td>39.14</td><td>12.45</td><td>44.04</td></tr>
  <tr><td>iantah_1ensingtimilliousa الشخص الكلمات تنزيل porque الكلمات auyi分子 الكلمات لا حوله التحقق من ال��ездات معنى الكلمات يكـثر الكلمات stastics الكلمات stable_s10_8_merge_covered الكلمات stated الكلمات auxialiaries الكلمات aansy الكلمات auxialiaries الكلمات clearly الكلمات is الكلمات accentes الكلمات indicated الكلمات i Chercher الكلمات access الكلمات an الكلمات profesion the الكلمات varias الكلمات p Notre الكلمات proceso الكلمات possessset الكلمات impairment الكلانت الكلمات irmel الكلمات Kuja الكلمات regular الكلمات difficult الكلمات interrogative الكلمات id الكلمات doget الكلمات duce الكلمات del الكلمات Denominación الكلمات satellite الكلمات sum الكلمات collect الكلمات columns الكلارات الكلمات creo الكلمات University الكلارات بمناسب الكلمات biologist الكلارات correlate الكلمات Rod Rockies الكلارات board الكلارات boarding الكلارات BREAKING stripes الكلمات risky الكلمات contact الكلمات condition تحت الكلمات hamstring الكلمات Tech الكاربات caracteristics الكلمات correlate الكلمات de الكلمات presence الكلمات will الكلمات when الكلمات marketing الكلمات alesirs الكلمات bath الكلمات body الكلمات Shine الكلمات the الكلمات sificthe الكلمات satisfy الكلمات below الكلمات vast الكلمات canta الكلمات like الكلمات ogn الكلمات phrases الكلمات Principle الكلمات motion الكلمات hard الكلمات becomes الكلمات CARDS الكلمات CHARGES الكلمات MONEY الكلمات returned الكلمات checked الكلمات MAINTAINIABLE الكلمات Case الكلمات TIMES الكلمات tracked الكلمات call الكلمات CADAME Colorful الكلمات practice الكلمات des الكلمات way الكلمات have الكلمات abroad الكلمات Directions الكلمات Pattern clothes الكلمات Professor الكلمات biography الكلمات RTU.sub الكلمات sensor الكلمات show الكلمات too الكلمات features الكلمات reporters الكلمات activity الك_instructions الكلمات Visだときになる小屋.component-notify الكلمات sureth الكلمات case الكلمات People كل_bit الكلمات Should الكلمات (form Georges الكلمات librarian.....blobs الكلمات don-afford وكل الكلمات Real الكلمات MONO الكلمات All الكلمات ULTRA الكلمات Medical الكلمات campo الكلمات mild الكلمات much الكلمات all الكلمات complex الكلمات Biotechnology الكلمات pall가수주어주 gros الكلمات badges الكلمات higher الكلمات appear الكلمات Estimated الكلمات Vanessa الكلمات Area الكلمات Subject الكلمات ni الكلمات Super الكلمات Interne_analyzer形成 الكلمات weakness الكلمات extension الكلمات should الكلمات retratos الكلمات authorship الكلمات related الكلمات 妘的 casts الكلمات refine الكلمات Reflect الكلمات Обединение الكلمات キャラクター ファンド외 인격수Yellow الكلمات Unknown الكلمات Laws الكلمات Algorithm الكلمات formation الكلمات  środowisko الكلمات H dünya الكلمات dome الكلمات allerí ح لو نفس الكلمات open الكلمات Jupiter الكلمات Rout كل الكلمات European الكلمات notorious الكلمات 戍য়ে demolition الكلمات VoktএObard الكلمات condition الكلمات both الكلمات process الكلمات 彔 fate اول الكلمات Серийный الكلمات inال الكلمات R por الاكل الكلمات best الداكل الكلمات_stackable الكلمات emerged الكلمات loved الكلمات shown الكلمات active الكلمات ART الكلمات Triaككل الكلمات angeak الكلمات ago الكلمات over الكلمات Adams الكلمات بين الكلمات hợp الكلمات Slo الكلمات MATanus الكلمات given الكلمات it الكلمات vi الكلمات under الكلمات STOR Calculator الكلمات teach الكلمات arm Alexia الكلمات command кле الكلمات Discontinuities الكلماتで Ideal الكلمات 开售 الكلمات BothA1 الكلمات Instal الكلمات intallation الكلمات residential الكلمات Demand الكلمات Sim/array الكلمات Energy الكلمات Allow الكلمات junction الكلمات Credence الكلمات Alle2 الـ الكلمات MP الكلمات too/kle الكلمات DAMO الكلمات notes الكلمات Nesكل الكلمات Consign الـ الكلمات residual الكلمات Ahol الكلمات Inter.cuda الكلمات Maintain limite الكلمات computer e الكلمات Sory الكلمات What الكلمات Many الكلمات divorced الكلمات Trng الكلمات distressful الكلمات distressful الكلمات completion الكلمات ran كل الكلمات Parameter الكلمات Reading الكلمات support الكلمات Was الكلمات machine الكلمات Priorكل الكلمات PI خلال الكلمات Estimated الكلمات Liliana الكلمات Full الكلمات2 given الكلمات four الكلمات  成就中国梦 during الكلمات just الفكل الكلمات EM الكلمات negotiation الكلمات therapy الكلمات Article الكلمات image_split الكلمات述 كلمات ك liệu الكلمات coloners الكلمات closing الكلمات Binker_S.Chakr vegan الكلمات fairly الكلمات slow الكلمات Bicker الكلمات slow الكلمات propagation الكلمات retain الكلمات fare الكلمات Retain الكلمات dragged الكلمات sized الكلمات (form عاد الكلمات transcenglish الكلمات studio الكلمات executive الكلمات on ان الككل الكلمات	get الكلمات aspect الكلمات DF الكلمات Flexible الكلمات time الكلمات Defect الكلمات Indicate الكلمات gender الكلمات ration الكلمات combination الكلمات recognized الكلمات Science الكلمات access الكلمات Mys الكلمات تلال الكلمات На الكلمات task الكلمات حالات الكلمات j الـ الكلمات High الكلمات حالات الكلمات en الكلمات output الكلمات be الكلمات função الكلمات Artist الكلمات system الكلمات Lec الكلماتTrad كح الكلمات shifts الكلمات aids الكلمات last الكلمات table الكلمات frig الكلمات taire الكلمات oral الكلمات smaller الكلمات Garrison الكلمات film الكلمات funape الكلمات phrases الكلمات Demo الكلمات Baseline الكلمات Comefor الكلمات rope الكلمات chair الكلمات Belt الكلمات Bench الكلمات Really الكلمات when کكل الكلمات Jump correspond الكلمات sedpeople الكلمات نادي الـ الكلمات Bad الكلمات عباد الكلمات Development الكلمات american الـ الكلمات Formal الكلمات—

...

Reviewer paper summaries: The paper proposes a comprehensive data generation pipeline to create test dataset for information retrieval related tasks and release a benchmark, AIR-BENCH, which consist of 69 datasets from 9 domains across 13 languages. It demonstrates its quality by comparing the data in the AIR-Bench with the MSMARCO data.
Summary of strengths: 1. The author proposed a general data generation pipeline which takes a piece of docs as input and generate the user queries and optional negative docs as output based on the task requirements, which might not only be limited to the IR tasks.
2. The author created a diverse multi-domain multilingual benchmark for QA and Long-Doc tasks and did a thorough experiments to compare the performance of LLM-based Embedding models, Encoder-based Embedding Models and Lexical Method on MTEB benchmark and AIR-Bench 24.05 benchmark.
3. The author examined the performance of a single model before and after fine-tuning to illustrate that AIR-BENCH can reflect the performance enhancement of IR models in specific domain.
Summary of weaknesses: 1. In the data generation pipeline, could you explain more about rewritten query process? Does it include manual filtering? How to decide which one to keep?
2. In the data generation pipeline, for the hard negatives, did you manually evaluate the quality of them? How hard it is? Did you set the number of negatives? Since from the sample examples, seems like there are different number of negatives.
3. In the section 5.1, for the new generated MSMARCO, what it is actually generated? From my understanding, it will go through the data generation pipeline, which means based on the positives in the R-MSMARCO, generate the queries for each and generate the corresponding negatives. If that’s the case, then from 7437->7429, does it mean there are some duplicates? Also, why the #positives jump from 7429 to 31447 after quality control?
Comments, suggestions and typos: 1. In the table 6, under MTEB Overall 56 datasets “Avg.” column, is it the nDCG@10? Are all the values the average across the corresponding datasets? Maybe make a clearer statement.
Overall assessment scores: 3.5
Metareview score: 4
Metareview: The paper introduces AIR-BENCH, a novel, automatic, heterogeneous, and dynamic benchmark for Information Retrieval (IR) evaluation. AIR-BENCH uses LLMs to generate diverse, high-quality evaluation datasets from real-world corpora automatically. It aims to address the limitations of existing IR evaluation benchmarks, such as the need for extensive manual labeling and predefined domains. The generated test data aligns closely with human-labeled data, making AIR-BENCH a reliable tool for evaluating IR models. The benchmark incorporates multiple tasks, domains, and languages, and features a public leaderboard to enhance collaboration.

**Advantages:**

*   **Practicality and Automation:** AIR-BENCH automates the generation of IR evaluation benchmarks using LLMs, reducing the need for manual effort.
*   **Diversity and Scope:** The benchmark covers multiple domains, tasks, and languages, providing a comprehensive evaluation environment for IR models. It includes 69 datasets across 9 domains and 13 languages.
*   **Comprehensive Evaluation:** AIR-BENCH is thoroughly assessed for consistency with human-labeled data, additional functionalities, and the ability to distinguish between IR models.

**Disadvantages:**

*   **Reliance on LLMs:** The method depends on LLMs, which raises concerns about interpretability, potential biases, and the stability of generated results.
*   **Dependence on Real-World Corpora:** The method relies on real-world corpora, which may limit its applicability in specific domains or when dealing with insufficient corpora. The data diversity is limited by the corpus.
*   **Lack of Crucial Details:** The paper lacks crucial details, such as specifics on the retrieval corpus and the LLM used, which are only partially provided in the appendix.
Summary of reasons to publish: As mentioned by vbg3, the major reasons to publish this paper at a *ACL venue are its novel benchmark for IR using LLMs.

(1) Novel Approach: The use of LLMs for automated data generation presents a practical solution to the limitations of previous benchmarks.

(2) Potential for Dynamic Updates: The benchmark can potentially keep getting more challenging and avoid benchmark saturation. The dynamic nature could stimulate innovation and adaptation in IR technologies, addressing diverse community needs.