Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration (2024)

Abstract: The proliferation of Large Language Models (LLMs) has led to an influx of AI-generated content (AIGC) on the internet, transforming the corpus of Information Retrieval (IR) systems from solely human-written to a coexistence with LLM-generated content. The impact of this surge in AIGC on IR systems remains an open question, with the primary challenge being the lack of a dedicated benchmark for researchers. In this paper, we introduce Cocktail, a comprehensive benchmark tailored for evaluating IR models in this mixed-sourced data landscape of the LLM era. Cocktail consists of 16 diverse datasets with mixed human-written and LLM-generated corpora across various text retrieval tasks and domains. Additionally, to avoid the potential bias from previously included dataset information in LLMs, we also introduce an up-to-date dataset, named NQUTD, with queries derived from recent events. Through conducting over 1,000 experiments to assess state-of-the-art retrieval models against the benchmarked datasets in Cocktail, we uncover a clear trade-off between ranking performance and source bias in neural retrieval models, highlighting the necessity for a balanced approach in designing future IR systems. We hope Cocktail can serve as a foundational resource for IR research in the LLM era, with all data and code publicly available at https://github.com/KID-22/Cocktail.

heading: 1 Introduction
text: Information retrieval (IR) systems, as the keystone in overcoming information overload, have seen widespread application across various domains, including search engines (Li et al., 2014), question answering (Karpukhin et al., 2020), dialog systems (Chen et al., 2017), etc. A typical IR system aims at finding relevant documents or passages from a specific corpus in response to user’s
∗Corresponding author. Work partially done at Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education.
queries (Li, 2022; Zhao et al., 2023a). Traditionally, IR benchmarks, notably MS MARCO (Nguyen et al., 2016), TREC (Craswell et al., 2020) and BEIR (Thakur et al., 2021), have exclusively utilized human-written content. However, the recent surge in Artificial Intelligence Generated Content (AIGC) facilitated by advanced Large Language Models (LLMs) has revolutionized the IR landscape (Dai et al., 2024a). This evolution has broadened the scope of IR systems, which now encompass a hybrid corpus composed of both humanwritten and LLM-generated content (Ai et al., 2023; Zhu et al., 2023; Dai et al., 2024a), presenting new challenges and opportunities of IR in the LLM era.
For instance, recent studies (Dai et al., 2024b; Xu et al., 2024) have shed light on a critical issue within neural retrieval models in the LLM era: a pronounced “source bias”. This bias, characterized by the preferential ranking of LLM-generated content over semantically equivalent human-written content, poses a significant threat to the IR ecosystem. Hence, the need to comprehensively understand such impact of LLM-generated content across different IR models and diverse IR domains and tasks has become more pressing, especially with the escalating prevalence of LLM-generated content (Hanley and Durumeric, 2023; Bengio et al., 2023). However, existing IR benchmarks either fail to reflect the real-world IR scenarios of the LLM era, as they solely contain human-written texts in their corpus, or provide limited datasets for exploring source bias. These shortcomings highlight the need for a comprehensive benchmark that accurately mirrors the current IR landscape, characterized by the integration of both human-written and LLM-generated texts within the corpus, to facilitate new research questions in this LLM era.
To fill this gap, we present a comprehensive benchmark tailored for IR in the LLM era, namely Cocktail, where the corpus contains both humanwritten and LLM-generated texts. Cocktail en-
7052
compasses 16 retrieval datasets spanning different domains and tasks, enabling both in-domain and out-of-domain evaluation settings. To construct these datasets, we first select 15 widely used public human-written corpora from MS MARCO (Nguyen et al., 2016), TREC (Craswell et al., 2020), and BEIR (Thakur et al., 2021). Then, based on these human-written corpora, we use Llama2 (Touvron et al., 2023) to rewrite each text to preserve semantic equivalence while introducing LLM-generated corpora. Finally, we mix the original human-written corpora and the LLM-generated corpora to get the final Cocktail corpora and assign the same relevancy label for the corresponding query-document pairs. Furthermore, to address the potential biases introduced by the inherent knowledge of LLMs during the rewriting process, we collect an additional new dataset, NQ-UTD. This new dataset comprises 80 queries and 800 documents from recent events. It serves as a critical component of Cocktail, offering an essential perspective for assessing the performance of IR systems in the context of both pre- and post-LLM era datasets.
With the benchmarked diverse datasets in Cocktail, we then conduct comprehensive evaluations of over ten state-of-the-art (SOTA) retrieval models through more than 1,000 experiments. Our analysis firstly reinforces previous findings by Dai et al. (2024b), highlighting a pervasive bias towards LLM-generated content across nearly all 16 datasets in Cocktail among all neural retrieval models. Furthermore, the results illustrated in Figure 1 reveal a distinct trade-off between ranking performance and source bias within these SOTA neural models. This observation suggests that while striving for high performance, these models may rely on inherent shortcuts, failing to grasp true semantic relevance and resulting in severe source bias. Hence, future work requires better considering a balance between performance and bias mitigation in the design of next-generation IR models.
In summary, our contributions are as follows:
(1) To the best of our knowledge, Cocktail is the first comprehensive benchmark with 16 datasets from a variety of domains and tasks tailed for IR research in the LLM era, where the corpus of each dataset contains both human-written texts and corresponding LLM-generated counterparts.
(2) We conduct extensive evaluations of state-ofthe-art retrieval models using the Cocktail benchmark, assessing both retrieval accuracy and source
bias. The evaluation tool, along with the codes, is open-sourced, facilitating ease of adaptation for evaluating new models and datasets.
(3) Extensive empirical studies reveal a clear trade-off between ranking performance and source bias in neural retrieval models. This finding underscores the importance of achieving a suitable balance between performance improvement and bias mitigation in future IR model designs.

heading: 2 Related Work
text: IR meets Large Language Models. Information retrieval (IR), the keystone of information access, has now significantly been reshaped by the emergence of large language models (LLMs) (Zhao et al., 2023b; Ai et al., 2023; Dai et al., 2024a). This intersection has manifested in two pivotal ways. On the one hand, much effort has been made to utilize the advanced capabilities of LLMs to refine the whole retrieval pipeline (Zhu et al., 2023), including the integration of LLMs across various IR components, such as query rewriters (Srinivasan et al., 2022), retrievers (Wang et al., 2023), re-rankers (Sun et al., 2023b), and readers (Shi et al., 2023). On the other hand, the capacity of LLMs for generating human-like text at scale has shifted the landscape of searchable corpora, which now includes both human-written and LLMgenerated texts. This evolution has introduced new challenges, most notably the emergence of source bias (Dai et al., 2024b; Xu et al., 2024; Dai et al., 2024a), where neural retrievers exhibit a preference
for LLM-generated content, potentially compromising the fairness and accuracy of search results. Moreover, the inclusion of LLM-generated content also raises concerns about privacy (Yao et al., 2023) and the dissemination of misinformation (Pan et al., 2023). In this paper, we focus on the second line and aim to establish a comprehensive benchmark for evaluating IR models in the LLM era, which can help understand the impact of LLM-generated content on IR systems.
Related Benchmarks. Historically, before the proliferation of LLM-generated content on the internet, several benchmarks were established for the evaluation of IR models, primarily utilizing corpora composed of human-written documents or passages. Notably, MS MARCOO (Nguyen et al., 2016) and TREC (Craswell et al., 2020) are widely used for supervised evaluation in IR research. Similarly, BEIR (Thakur et al., 2021) presents a diverse benchmark incorporating 18 datasets from various IR domains and tasks, tailored to zero-shot evaluation. Despite their contributions to advancing IR systems, these benchmarks fall short of reflecting the real-world scenarios of the current LLM era due to the absence of LLM-generated content in their corpora (Dai et al., 2024b; Xu et al., 2024; Dai et al., 2024a). This gap underscores the necessity for new benchmarks that include both human-written and LLM-generated texts, offering a more comprehensive and realistic evaluation environment to navigate the challenges and opportunities presented by
the integration of LLMs into IR.

heading: 3 Benchmarking Retrieval Datasets
text: Cocktail establishes a comprehensive benchmark tailored for the evaluation of IR models in the LLM era, characterized by corpora containing both human-written and LLM-generated content. This benchmark aims to access the performance and biases of existing retrieval models while encouraging the creation of future IR systems that excel in robustness and generalization across diverse scenarios in the LLM era. To achieve this, we collect and construct 16 IR datasets, incorporating 15 datasets from the pre-LLM era and one newly developed dataset to ensure a wide representation of domains and tasks. The statistics for the 16 datasets benchmarked in Cocktail are summarized in Table 1. We also list the dataset website links and the corresponding licenses in Appendix Table 5.

heading: 3.1 Dataset Construction
text: A key concern in constructing datasets with LLMs is the potential for LLMs to have prior knowledge about queries, which could lead to an unfair evaluation. To address this, following previous works (Dai et al., 2024b), we choose rewriting documents without incorporating query-related information rather than a full generation from LLMs with given queries, ensuring that any detected source bias is a genuine reflection of model preferences. Moreover, by avoiding full document gen-
IR System Online Serving
Offline Training Corpus Query Interaction Log Interaction
IR System Corpus
Offline Training Interaction Log Online Serving Query Interaction … IR System Corpus
Interaction Log Online Serving Query Interaction …… … IR in Pre-LLM Era IR in Early LLM Era IR in LLM Era
eration based on queries, this approach also simplifies controlling the text length across different sources, ensuring consistency and further reducing potential biases in the evaluation. Specifically, we construct each dataset in Cocktail following a three-step process: 1) collecting seed datasets with human-written corpus and alongside relevancy labels for given queries; 2) leveraging LLMs to generate corresponding LLM-generated corpus with the human-written corpus as inputs; and 3) assigning relevancy labels to the LLM-generated content and given queries, ensuring seamless integration into the benchmark. An overview of our construction pipeline is shown in Figure 2.
Seed Human Datasets. To cover as diverse IR domains and tasks as possible, we select the widelyused 15 datasets from MS MARCO (Nguyen et al., 2016), TREC (Craswell et al., 2020), and BEIR (Thakur et al., 2021) benchmark as our human-written corpus. These datasets span six domains and eight distinct tasks, facilitating comprehensive evaluations under both in-domain and outof-domain settings. Moreover, as suggested in Sun et al. (2023b), we also construct a new test dataset NQ-UTD to ensure that relevance annotations have not been learned by LLMs. NQ-UTD comprises 80 queries distributed across eight domains, sourcing from recent events (from Nov. 2023 to Jan. 2024). We verify that neither GPT-4 nor Gemini, two of the most advanced LLMs to date, can not answer the questions within NQ-UTD, validating no prior knowledge of these questions in LLM. For details about the collection and annotation processes of NQ-UTD, along with detailed descriptions of all 16 datasets, please refer to Appendix A.1.
LLM-Generated Corpus. To ensure a fair evaluation of the impact of incorporating LLMgeneration content, we utilize the widely used llama-2-7b-chat to rewrite each human-written text without changing its semantic information, with the following instructions: “Original Text: {{human-written text}} Please rewrite the above given text. Your answer must be formatted as
follows: Rewritten Text: <your rewritten text>.” This post-instruction strengthens the instructionfollowing capabilities of LLMs, proving particularly effective for processing and generating responses to long input texts (Liu et al., 2023).
Relevancy Label Assignment. Upon the creation of the LLM-generated corpus for each dataset, we assign the relevancy labels from the original <query, human-written doc> pairs to the new <query, LLMgenerated doc> pairs. This process is underpinned by the premise that both sources of the content preserve nearly identical semantic information, which will be further verified in the following section.
All the datasets in Cocktail are organized in a standard format (corpus, queries, qrels) akin to the BEIR benchmark (Thakur et al., 2021), facilitating ease of use and comparison. Examples for each dataset are showcased in Appendix Table 12. More details about the data processing and quality control are provided in Appendix A.2.

heading: 3.2 Dataset Statistics and Analysis
text: As shown in Table 1, there are minimal differences in average word length between human-written and LLM-generated documents, with the latter being marginally shorter. The detailed text length distribution, depicted in Appendix Figure 5, further confirms the negligible variance in text length. Additionally, term-based Jaccard similarity and overlap distributions between LLM-generated and humanwritten documents, visualized in Appendix Figure 6, show a noticeable distinction in terms despite similar semantic content.
Following practices in Dai et al. (2024b), our dataset construction process involved feeding only the original human-written text to the LLM for rewriting, without inputting any query-specific information to avoid introducing additional queryrelated bias. Furthermore, we employed the OpenAI embedding model1 to obtain semantic embeddings for both text sources, comparing them
1text-embedding-ada-002: https://platform.openai. com/docs/guides/embeddings
through cosine similarity. For calibrating the results, we also compare the semantic similarity between randomly selected <human-written doc, LLM-generated doc> pairs and matching pairs <human-written doc, LLM-generated doc>. These comparisons, presented in Appendix Figure 7 and Table 9, demonstrate a high similarity between the LLM-generated document and the corresponding original human-written counterpart, indicating successful semantic retention of the original humanwritten text in the LLM-generated texts. Additionally, evaluations of several retrieval models on sole human-written or LLM-generated corpora showed consistent performance, as seen in Appendix Table 8. Moreover, we also conduct human evaluations for quality verification for each dataset in Appendix Table 10. These observations reinforce confidence in the quality of our newly constructed datasets, suggesting that the LLM-generated content maintains semantics comparable to humanwritten texts for IR tasks. The detailed dataset analysis is provided in Appendix A.3.

heading: 4 Benchmarking Evaluation Protocol
text: Evaluation Framework. To standardize the assessment of IR models within the LLM era, we also develop a Python framework that combines userfriendliness with comprehensive evaluation capabilities. Built on the foundation of the BEIR (Thakur et al., 2021), our framework inherits its best features, including the ability to easily replicate experiments from open-sourced repositories and incorporate new models and datasets. A key innovation of our framework is its ability to conduct evaluations using either individual or mixed corpora, accommodating the mixture of humanwritten and LLM-generated content characteristic of the LLM era. These features make our framework an invaluable tool for advancing IR research and application in both academia and industry. A detailed description and all the scripts of our evaluation framework are provided in the link https://github.com/KID-22/Cocktail.
Evaluation Metrics. Aligned with the BEIR benchmark (Thakur et al., 2021), we select Normalized Discounted Cumulative Gain (NDCG@K) as our primary metric to assess retrieval accuracy, given its robustness in capturing the effectiveness of IR systems across tasks with binary and graded relevance judgments. Following previous studies (Dai et al., 2024b; Xu et al., 2024), we choose
K = 1 since the top-1 item in the retrieved list is most likely to be viewed and clicked by users. To provide a more comprehensive evaluation, we also report results on K = 3 and K = 5 in Appendix C.
Moreover, in the multi-sourced corpus shaped by LLMs, the evaluation of IR systems necessitates not only mere accuracy metrics but also a critical assessment of source bias (Dai et al., 2024b). This bias, as evidenced by ranking LLM-generated content higher than human-written content, poses a significant challenge in today’s IR ecosystem. To quantify and normalize source bias in different datasets, we follow previous works (Dai et al., 2024b; Xu et al., 2024) and adopt the Relative ∆. This metric captures the relative percentage difference in NDCG scores between human-written and LLM-generated content, which is defined as:
Relative ∆ = NDCGHuman−NDCGLLM(NDCGHuman+NDCGLLM)/2 × 100%, where the NDCGHuman and NDCGLLM denote the NDCG scores attributed to human-written and LLM-generated content, respectively. Note that Relative ∆ > 0 indicates that IR models rank human-written content higher than LLM-generated content, and conversely, Relative ∆ < 0 indicates the opposite tendency. The absolute value of Relative ∆ reflects the extent of source bias.

heading: 5 Benchmarking Retrieval Models
text: In this section, we delve into the evaluation and analysis of various retrieval models utilizing the constructed Cocktail benchmarked datasets.

heading: 5.1 Retrieval Models
text: Following the BEIR benchmark (Thakur et al., 2021), we focus on evaluating the advanced stateof-the-art transformer-based neural retrieval models. Besides the widely used lexical retriever BM25 (Robertson et al., 2009), our experiments include two main types of neural retrieval models:
Neural Retriever. We utilize and fine-tune the following two most commonly used pre-trained language models on the MS MARCO dataset (Nguyen et al., 2016) using the official training script2 from BEIR: 1) BERT (Devlin et al., 2019); 2) RoBERTa (Liu et al., 2019). Additionally, we evaluate the performance of state-of-the-art models also trained on MS MARCO, employing officially released checkpoints: 3) ANCE (Xiong et al.,
2https://github.com/beir-cellar/beir/blob/ main/examples/retrieval/training/train_msmarco_ v3.py
2020); 4) TAS-B (Hofstätter et al., 2021); 5) Contriever (Izacard et al., 2022); 6) coCondenser (Gao and Callan, 2022); 7) RetroMAE (Xiao et al., 2022) ; 8) DRAGON (Lin et al., 2023).
Neural Re-ranker. For re-ranking, we employ two state-of-the-art models with their publicly available official pre-trained checkpoints: 1) CE (Wang et al., 2020); 2) monoT5 (Raffel et al., 2020).
In our experiments, unless specified otherwise, neural re-rankers re-rank the top-100 documents retrieved by BM25. Detailed information on the benchmarked models, including the publicly available official pre-trained checkpoints and implementation details, can be found in Appendix B.

heading: 5.2 Benchmarked Results
text: We conduct extensive evaluations with more than 1,000 experiments on the Cocktail benchmarked datasets. The results of retrieval accuracy and source bias across all benchmarked retrieval models and datasets are reported in Table 2 and Table 33, respectively. Figure 1 shows average results of neural retrieval models across all datasets. From the results, we have the following key observations:
3Note that SciFact was regenerated with our prompt, leading to slight result discrepancies from Dai et al. (2024b)
Neural models consistently exhibit source bias towards LLM-generated content. This bias is evident across neural retrieval models, spanning both in-domain and, more significantly, out-ofdistribution datasets. This trend persists in data from both pre-LLM and LLM eras. Remarkably, the average Relative ∆ across all neural models on the datasets surpasses −25%. These findings further support the findings of Dai et al. (2024b) and verify the widespread source bias in neural retrieval models, regardless of the domain or task, highlighting an urgent need to address this bias.
Stronger neural retrieval models exhibit more severe source bias. The results illustrated in Figure 1 underscore a significant trade-off faced by neural retrieval models: advancements in ranking performance often come with an increase in source bias. This trend suggests that these SOTA neural models may not fully understand semantic relevance. Instead, these models tend to leverage inherent shortcuts to enhance performance, inadvertently leading to increased bias. This phenomenon suggests that attempts to boost model performance could unintentionally magnify source bias issues, underlining the challenge of advancing model capabilities without leading to severe source bias.
The numbers (i.e., Relative ∆ > 0) suggest that retrieval models generally prefer human-written content while the numbers (i.e., Relative ∆ ≤ 0) indicate retrieval models prefer LLM-generated content.
Neural re-rankers generalize better but are still biased towards LLM-generated content. Neural re-rankers, while achieving superior ranking performance on most datasets and showing enhanced generalization capabilities compared to neural retrievers, are not exempt from pervasive source bias. Specifically, re-ranking models like CE and monoT5 exhibit a significant bias towards LLM-generated content, sometimes even more pronounced than that observed in neural retrievers. This is particularly evident in datasets such as NQ and Touché-2020, further emphasizing the widespread nature of source bias in PLM-based neural retrieval models.
Bias in the first retrieval stage tends to propagate and even amplify during the re-ranking stage. This trend is particularly notable in datasets like NFCorpus and Touché-2020, where the source bias observed in the first retrieval stage persists
and tends to intensify during the second-stage reranking. Moreover, as detailed in Table 4, enhancing the efficiency of the first-stage retrieval by replacing BM25 with DRAGON does not necessarily improve performance in the subsequent re-ranking phase. However, the source bias inherent in the first-stage retriever significantly impacts and may even magnify in subsequent re-ranking. This observation underscores the critical need for developing holistic approaches to mitigate bias throughout the retrieval pipeline, thereby ensuring fairness and accuracy in the whole IR system.

heading: 5.3 Further Analysis
text: Impact of Different Pooling Strategies. Pooling strategies in PLMs are critical for aggregating information from the token embeddings for downstream semantic matching. We explore the ranking performance and source bias on BERT and RoBERTa w.r.t. different pooling strategies, including CLS token, max token, last token (Muennighoff, 2022), mean (Reimers and Gurevych, 2019), and weighted mean pooling (Muennighoff, 2022). The averaged results of all benchmarked datasets in Cocktail are shown in Figure 3. As we can see, the ranking performance and degree of source bias vary signifi-
cantly with the specific pooling strategy. Weighted mean pooling demonstrates the most effectiveness for ranking, which can be attributed to the nuanced semantic understanding by positionally weighting tokens, thus enhancing document relevance matching. Yet, this strategy also incurs the most severe source bias, possibly because LLM-generated texts have distinctive structural or stylistic features that become more pronounced and amplified under weighted aggregation.
Conversely, max pooling, which selects the maximum value across each dimension from the token embeddings, appears to be the least effective. This could be due to its focus on the most dominant features within the text, potentially overlooking the broader contextual nuances captured by other strategies. The dominance of specific features might not always align with the relevance signals needed for accurate document ranking, explaining the lower performance and less bias.
Other strategies, such as mean, CLS token, and last token pooling, strike a balance by capturing overall text representations while still highlighting key tokens. However, the results show that while they are capable of supporting effective document ranking, they still suffer from severe source bias with Relative ∆ far below −10%.
Overall, the variance in ranking performance and source bias across pooling strategies underscores the critical role of model architecture in retrieval model design. This analysis suggests that while weighted mean pooling offers enhanced ranking capabilities, it comes with a trade-off of increased source bias. Future work can explore hybrid or innovative pooling methods to balance performance with bias mitigation.
Impact of PLM Model Size. Our investigation extends to the influence of model size on ranking
performance and source bias, utilizing BERT models of varying sizes: mini, small, base, and large. As illustrated in Figure 4, an increase in model size leads to a notable enhancement in ranking performance. However, this improvement is paralleled by a rise in source bias (i.e., more negative).
The trend indicates that larger models, with their enhanced semantic understanding and processing capabilities, are more effective in judging document relevance, boosting their ranking performance. However, the increased capabilities may also make them more sensitive to the nuanced distinctions between human-written and LLM-generated texts, amplifying the source bias. Consequently, while the advanced performance of larger models is promising, it is coupled with a more severe bias risk, posing a significant challenge that merits more exploration in future research.

heading: 6 Conclusion and Future Work
text: This study proposes Cocktail, the first comprehensive benchmark consisting of 16 diverse datasets with mixed human-written and LLM-generated corpora. Alongside this, we present an evaluation tool equipped with standardized data formats and easily adaptable evaluation codes for a wide array of retrieval models. This tool is designed to systematically evaluate ranking performance and source bias in IR systems, both in supervised and zero-shot settings, paving the way for the development of more robust next-generation IR models.
Utilizing Cocktail, we conduct an extensive evaluation of over ten state-of-the-art neural retrieval models through more than 1,000 experiments. Our findings reveal a clear trade-off between ranking performance and source bias in neural retrieval models, underscoring the difficulty of enhancing model performance without increasing bias to-
wards LLM-generated content. This challenge emphasizes the need for a suitable balance between performance enhancement and bias mitigation in the design of future IR models.
The newly collected and annotated NQ-UTD dataset comprises queries derived from recent events, featuring all content not yet incorporated into the pre-training knowledge of most LLMs. This characteristic renders it a valuable resource for fairly evaluating the effectiveness of LLMs in processing new, unseen data, especially for LLMbased retrieval or question-answering systems.

heading: Limitations
text: We believe our proposed Cocktail benchmark is a foundational step toward advancing IR research in the LLM era. Nonetheless, our work still has several limitations for future research efforts. First, while our benchmarked 16 datasets encompass a broad range of IR domains and tasks, the IR field is continually evolving, with new areas of interest emerging regularly. Future updates to the Cocktail benchmark could benefit from including datasets from other IR domains, such as legal information retrieval (Sansone and Sperlí, 2022). This expansion would not only diversify but also enhance the benchmark’s utility across more specialized areas. Second, the scale of our NQ-UTD dataset is currently limited to 800 query-passage pairs, primarily due to the high costs of human annotation. This encompasses financial costs, the time required to develop annotation guidelines, train annotators, and perform manual audits and validations. Future initiatives could focus on expanding the NQ-UTD dataset, possibly by employing LLMs to support and streamline the annotation process (Zhang et al., 2023). Such an approach could facilitate broader coverage and richer labeled query-passage pairs. Third, the construction of LLM-generated corpus in our benchmarked 16 datasets was significantly influenced by the inference costs of LLM, leading us to rely exclusively on Llama2 (Touvron et al., 2023) for generating LLM-based content. However, the real-world IR landscape is shaped by a variety of LLMs. Future research might include content generated by an array of LLMs, such as OpenAI ChatGPT and Google Gemini, to mirror the diversity of LLM-generated content more accurately. Despite these limitations, we envision the Cocktail benchmark as a valuable resource for IR research in the LLM era, offering a foundation
upon which next-generation IR models can be built.

heading: Ethics Statement
text: We commit to maintaining the highest ethical standards in our research, strictly adhering to ethical guidelines, applicable licenses, and laws. The 15 datasets utilized in this study are sourced from public repositories and have been employed within the bounds of their respective licenses and legal constraints. For the collected NQ-UTD dataset, we followed stringent ethical protocols, ensuring the corpus collected from online news sources was carefully screened and double-checked to remove any personally identifiable or sensitive information. Participants involved in data annotation were thoroughly informed about the process and provided informed consent before their participation.
We recognize the potential risks posed by large language models (LLMs), such as generating harmful, biased, or incorrect content (Pan et al., 2023; Chen and Shu, 2023). The Llama2 model, used in this study for corpus generation, is not immune to these issues and may inadvertently produce misleading content. We have taken measures to minimize these risks, ensuring our benchmark aligns with ethical standards. Despite the acknowledged risks associated with LLMs, we assert that the scientific benefits and insights derived from our benchmark far outweigh potential concerns. Our resources are intended solely for scientific research, designed to foster advancements in information retrieval within this LLM era.

heading: Acknowledgements
text: This work was funded by the National Key R&D Program of China (2023YFA1008704), the National Natural Science Foundation of China (No. 62377044, No.62276248), Beijing Key Laboratory of Big Data Management and Analysis Methods, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, PCC@RUC, funds for building world-class universities (disciplines) of Renmin University of China, and the Youth Innovation Promotion Association CAS under Grants No.2023111. This work was supported by the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (RUC24QSDL013).

heading: A Dataset Details
text: In this section, we provide details about the datasets in Cocktail. Table 5 lists the dataset website links and the corresponding licenses.

heading: A.1 Detailed Description of Datasets
text: A.1.1 NQ-UTD
The Natural Question-Up To Date (NQ-UTD) dataset comprises 80 questions focusing on recent hotspot events from November 2023 to January 2024. These questions span eight domains: Sports, News, Science, Technology, Autos, Music, Movies, and Games, with approximately 10 queries per domain related to the latest events, questioning about their times, locations, or other specifics. To gather relevant passages for each query, we searched these questions on platforms like Google, X (formerly Twitter), Reddit, Wikipedia, Quora, and Facebook. Following the practice in Sun et al. (2023b), we also conducted searches using keywords to retrieve passages that are partially relevant but do not directly answer the questions. We also queried these questions with the latest state-of-the-art LLMs, gpt-4-1106-preview and gemini-pro, which have their knowledge bases updated only until April 2023. The results show their 0% accuracy in answering questions from our test set, confirming that these LLMs had no preknowledge about these questions. The collected 800 documents were then manually annotated for relevance by the authors and their highly educated colleagues. Each document received a relevance score: 0 for not relevant, 1 for partially relevant (mentioning some aspects of the query but not fully answering it), and 2 for relevant (providing a complete answer to the query). To ensure consistent and high-quality annotations, each document was reviewed by five individuals, with a majority vote determining the final label.
The statistics and examples of the NQ-UTD dataset are presented in Table 6 and Table 7, respectively. Note that this dataset also offers a fair evaluation of the current capabilities of the advanced LLM-based IR models. As NQ-UTD contains content not previously included in LLM training data, it can serve as a valuable resource for evaluating LLMs’ ability to process and answer queries on recent events.

heading: A.1.2 MS MARCO
text: MS MARCO (Nguyen et al., 2016), developed by Microsoft Research, is a cornerstone dataset in the fields of natural language processing (NLP) and information retrieval (IR). This dataset comprises over a million user-generated questions derived from Bing search logs, tailored to advance three primary NLP tasks: document ranking, passage retrieval, and question answering. Within our Cocktail benchmark, we specifically utilize the passage retrieval subset, which contains 532K labeled query-passage pairs. A majority of the open-sourced pre-trained transformer checkpoints have been trained using this dataset. Following previous studies (Thakur et al., 2021; Lin et al., 2023), we employ the MS MARCO Dev set as the supervised evaluation test set.
A.1.3 TREC
DL19. The DL19 (TREC Deep Learning Track 2019) dataset (Craswell et al., 2020), is a key resource for exploring ad hoc ranking across extensive datasets from various domains. Our study focuses on the passage retrieval task utilizing the DL19 dataset, which comprises 43 queries and uses the MS MARCO passage corpus. This allows for its application in supervised evaluation settings.
DL20. The DL20 dataset (Craswell et al., 2021) comes from the second year of the TREC Deep Learning Track, and also focuses on ad hoc ranking through human-annotated training sets across diverse domains. We also focus on the passage ranking task featured in the DL20 dataset, which includes 54 queries and also leverages the MS MARCO passage corpus. This also allows for its application in supervised evaluation settings.
A.1.4 BEIR If not specified, we utilize the preprocessed version of the following open-sourced datasets as included in the BEIR benchmark (Thakur et al., 2021) to serve as the human-written seed datasets in our Cocktail benchmark, ensuring consistency and comparability in our evaluations.
TREC-COVID. The TREC-COVID dataset is developed by the Text REtrieval Conference (TREC) (Voorhees et al., 2021), specifically designed to address the challenges of biomedical information retrieval in the context of the COVID-19 pandemic. This dataset represents a concerted effort to harness the rapidly expanding collection of
scholarly articles related to COVID-19, providing a focused resource for biomedical IR.
NFCorpus. The NFCorpus dataset (Boteva et al., 2016) serves as a detailed resource tailored for ranking tasks within the biomedical field. A distinctive aspect of NFCorpus is its meticulously curated relevance links, connecting queries directly to pertinent research articles.
NQ. The NQ (Natural Questions) dataset aims to propel advancements in natural language understanding, particularly focusing on the questionanswering task (Kwiatkowski et al., 2019). This dataset is compiled from real questions submitted by users on Google search, each annotated with answers extracted from Wikipedia articles.
HotpotQA. HotpotQA (Yang et al., 2018) is a large-scale dataset that includes diverse questions posed by human participants. These questions necessitate a multi-step reasoning process, often requiring the extraction of answers from extensive Wikipedia texts and the identification of supporting evidence within them. HotpotQA is specifically
crafted to evaluate the effectiveness of models in deciphering complex, multi-hop questions and delivering precise answers.
FiQA-2018. FiQA-2018 (Financial Opinion Mining and Question Answering-2018) (Maia et al., 2018) presents a specialized question-answering challenge within the financial domain. The dataset benefits from a comprehensive corpus gathered from esteemed financial news platforms, analyst reports, and influential financial blogs.
Touché-2020. Touché-2020 dataset (Bondarenko et al., 2020) was created with the specific aim of advancing the field of argument retrieval. Its purpose is to identify argumentative content relevant to contentious queries. It contains argumentative passages from a broad spectrum of domains, which are carefully selected and annotated.
CQADupStack. CQADupStack (Hoogeveen et al., 2015) is a popular dataset for duplicate question retrieval, which aims to identify duplicate questions in community question answering (cQA) forums. This dataset includes carefully annotated
and categorized questions and answers across various domains. To maintain focus without losing the essence of generalizability, our study incorporates a subset from the English domains as our selected CQADupStack dataset.
DBPedia. The DBPedia dataset (Hasibi et al., 2017) includes structured information about entities, concepts, categories, and their relationships, gathered from Wikipedia entries. This dataset focuses on the entity retrieval task, where queries aim to retrieve relevant entities from the English DBpedia corpus dated October 2015.
SCIDOCS. The SCIDOCS dataset (Cohan et al., 2020) is a vast collection of scholarly articles from the Semantic Scholar database, enhanced with detailed metadata extraction and annotations. It serves as a robust dataset for the citation prediction task, where the objective is to identify papers cited by a given query paper title.
FEVER. The FEVER (Fact Extraction and VERification) dataset (Thorne et al., 2018) comes from the fact-checking domain, offering a curated collection of human-labeled claims sourced from Wikipedia. Each claim is meticulously classified as Supported, Refuted, or NotEnoughInfo, setting the stage for a nuanced fact-checking task.
Climate-FEVER. Climate-FEVER (Diggelmann et al., 2020) is a specialized dataset focused on the area of climate science, comprising real-world claims accompanied by evidence sentences from Wikipedia. Similar to the FEVER dataset, the primary task involves evaluating whether the provided evidence supports, refutes, or lacks sufficient infor-
mation to adjudicate each claim.
SciFact. SciFact (Wadden et al., 2020) is designed for the verification of scientific claims sourced from peer-reviewed literature, with each claim meticulously matched with corroborative or refuting evidence from related studies. The dataset comprises expert-crafted scientific claims alongside abstracts containing evidence, each annotated with labels indicating support or contradiction, and rationales explaining the basis of the evidence.

heading: A.2 Dataset Processing and Quality Control
text: Given the substantial computational cost associated with rewriting documents for the 16 datasets using Large Language Models (LLMs), we adopted sampling methods in line with practices from Zhou et al. (2022); Sun et al. (2023a). Specifically, for datasets with an original corpus size smaller than 200,000 documents, we retained the entire corpus without filtering. For datasets exceeding 100,000 documents, we retain all candidate documents that appear in the labeled data (including train, valid, and test set). If the number of documents postfiltering fell below 100,000, we supplemented the corpus with a random selection of up to 100,000 additional documents from the remaining corpus to ensure a challenging dataset size.
Documents with text lengths between 10 and 2,000 words are retained for each corpus. This criterion was set because texts shorter than 10 words often consisted of empty texts or symbols, compromising data quality—this was particularly evident in the TREC-COVID dataset, which contained over 40,000 documents with empty text. Con-
versely, texts over 2,000 words were filtered out to satisfy the context length limitation of 4,096 tokens for rewriting with Llama2, impacting only 0.05% of the data. These filters did not significantly alter the overall data distribution, with a total of less than 1% data removed, and our evaluation results align closely with those reported in the BEIR benchmark (Thakur et al., 2021) for the original datasets, as shown in Table 2. For most datasets, we generated LLM Corpus once. However, for MS MARCO, HotpotQA, DBPedia, and NQ-UTD, to construct a more challenging benchmark for evaluating source bias in IR models, we generated multiple versions and used BERT to filter and merge the corpus, aiming to increase the task difficulty.
When LLM is used for rewriting, LLM will inevitably refuse to rewrite due to safety constraints, especially for datasets like Touché-2020, which focus on retrieving contentious viewpoints. For such LLM-generated data, we keep the same content as the human-written counterparts. This approach allows for the easy removal of refused rewrites if necessary, ensuring both the integrity and the quality of the dataset processing and control measures implemented. The statistics of the 16 benchmarked datasets in Cocktails are summarized in Table 1.

heading: A.3 Dataset Statistics and Analysis
text: Term-based Statistics. Figure 5 illustrates the distribution of text lengths, revealing minimal variation between the lengths of texts. We then calculated the Jaccard similarity and overlap between each LLM-generated document dG and the corresponding human-written document dH , using the following two formulas:
Jaccard similarity = |dG⋂ dH | |dG⋃ dH | ,
Overlap = |dG⋂ dH |
|dH | .
The results presented in Figure 6 highlight significant differences in terms despite their ostensibly similar semantic information.
Semantic Analysis. Cosine similarity between LLM-generated and human-written documents, calculated using the OpenAI embedding model4, is displayed in Figure 7. The results, predominantly above 0.9, signify a high level of semantic preservation in the LLM-generated texts compared to
4text-embedding-ada-002
the original human-written content. We also compare the semantics between randomly selected <human doc, LLM doc> pairs and matching pairs <human doc, LLM doc>, as shown in Table 9. The average similarity score for matching pairs was 0.9816, significantly higher than the 0.7135 average for random pairs. This significant statistical difference supports the close semantic alignment between LLM-generated and human-authored documents, further validating the quality of our constructed dataset. Further, evaluations of various retrieval models on sole human-written or LLMgenerated corpora, as detailed in Table 8, demonstrate consistent performance across all datasets between two types of corpus. These observations across all datasets reinforce confidence in the quality of our newly constructed datasets, suggesting that the LLM-generated content maintains semantics comparable to human-written texts for IR tasks.
Quality Verification with Human Evaluation. To further validate the assigned relevancy label, we also conduct a human evaluation study. Due to cost constraints of human evaluation, we sample 20 <query, human doc, LLM doc> triples from each of the 16 datasets included in Cocktail. These triples were annotated by graduate students and our colleagues, who were asked to assess which document is more semantically relevant to the given query, with options being “human doc”, “LLM doc”, or “equal”. At least three different annotators evaluated each triple, with the majority vote deciding the final label. We summarized the results in Table 10, with numbers in parentheses indicating the percentage of agreement among all three evaluators for each option. The results show a comparable level of semantic relevance between human-written and LLM-generated texts for the given queries, ensuring the fairness of our analysis of source bias.

heading: A.4 Dataset Examples
text: For a better understanding of the datasets used in our Cocktail benchmark, we offer examples for each dataset in Table 12, showcasing a given query along with the corresponding relevant humanwritten document and LLM-generated document.

heading: B Model Details

heading: B.1 Detailed Description of Models
text: We select the widely used approach BM25 (Robertson et al., 2009) as the representation of lexical retrieval models in our benchmark.
For neural retrieval models that leverage pretrained language models to acquire semantic relationships between queries and documents, we select the following representative and state-of-theart models:
BERT (Devlin et al., 2019), a foundational pretrained language model, is commonly used as an encoder in dense retrieval systems. It was fine-
tuned on the MSMARCO dataset using the official BEIR benchmark training script.
RoBERTa (Liu et al., 2019) builds on BERT’s success with more data and refined training techniques to enhance performance. RoBERTa was fine-tuned in the same manner to BERT, using the MSMARCO dataset.
ANCE (Xiong et al., 2020) improves dense re-
trieval by selecting challenging negatives across the corpus and updating the Approximate Nearest Neighbor (ANN) index asynchronously with each training iteration.
TAS-B (Hofstätter et al., 2021), a bi-encoder model, leverages balanced margin sampling for efficient query selection and dual supervision from a cross-encoder and a ColBERT model for enhanced learning.
Contriever (Izacard et al., 2022) employs contrastive learning with positive samples generated through cropping and token sampling, using MoCo to incorporate negatives from a queue of previous batches for self-supervised training.
coCondenser (Gao and Callan, 2022) undertakes a two-stage process beginning with pretraining and unsupervised contrastive loss for embedding generation, followed by supervised training with the pre-trained encoder.
RetroMAE (Xiao et al., 2022) utilizes a Masked
Auto-Encoder approach for pretraining, enhancing sentence reconstruction from masked inputs to refine language modeling.
DRAGON (Lin et al., 2023) applies progressive supervision and query augmentation for effective dense retrieval, suitable for both supervised and zero-shot settings.
Additionally, the following two widely used neural re-ranking models are also adopted in our evaluation framework:
CE (Wang et al., 2020) is a cross-encoder model pre-trained on MS MARCO dataset (Nguyen et al., 2016) through knowledge distillation from three teacher models: BERT-base (Devlin et al., 2019), BERT-large (Devlin et al., 2019), and ALBERTlarge (Lan et al., 2019).
monoT5 (Raffel et al., 2020) is a sequenceto-sequence re-ranker based on T5 (Raffel et al., 2020), which is also pre-trained on the MS MARCO dataset.

heading: B.2 Checkpoints and Implementation Details
text: For the fine-tuning of BERT and RoBERTa models, we utilized the official training script5 in BEIR on the MSMARCO training dataset. Unless otherwise noted, mean pooling was employed as the pooling strategy, alongside cosine similarity as the scoring function. Training parameters for each model included a duration of 10 epochs, utilizing a batch size of 75, a learning rate set at 2e-5, and the AdamW optimizer for efficient optimization. Our software framework utilizes PyTorch version 2.0.0 and the HuggingFace Transformers library version 4.31.0. All experiments were performed using approximately 100 hours on four NVIDIA RTX A6000 (48G) GPUs.
5https://github.com/beir-cellar/beir/blob/ main/examples/retrieval/training/train_msmarco_ v3.py
For the other benchmarked neural retrieval models, we use the publicly available official pretrained checkpoints, which are listed in Table 11. For all the neural models, only the first 512 word tokens of all documents are inputted.

heading: C More Experimental Results
text: In addition to the findings presented through NDCG@1 in Table 2 and Table 3, we further extend our analysis to include the results for ranking performance and source bias for NDCG@3 and NDCG@5. These results are detailed in Table 13 and Table 14 for NDCG@3, and Table 15 and Table 16 for NDCG@5, respectively. Consistent with our earlier observations in Section 5.2, results on these metrics also reveal a similar trend, underscoring the robustness of our findings.
The numbers (i.e., Relative ∆ > 0) suggest that retrieval models generally prefer human-written content while the numbers (i.e., Relative ∆ ≤ 0) indicate retrieval models prefer LLM-generated content.
The numbers (i.e., Relative ∆ > 0) suggest that retrieval models generally prefer human-written content while the numbers (i.e., Relative ∆ ≤ 0) indicate retrieval models prefer LLM-generated content.

heading: Future Work
text: Nonetheless, our work still has several limitations for future research efforts. First, while our benchmarked 16 datasets encompass a broad range of IR domains and tasks, the IR field is continually evolving, with new areas of interest emerging regularly. Future updates to the Cocktail benchmark could benefit from including datasets from other IR domains, such as legal information retrieval (Sansone and Sperlí, 2022). This expansion would not only diversify but also enhance the benchmark’s utility across more specialized areas. Second, the scale of our NQ-UTD dataset is currently limited to 800 query-passage pairs, primarily due to the high costs of human annotation. This encompasses financial costs, the time required to develop annotation guidelines, train annotators, and perform manual audits and validations. Future initiatives could focus on expanding the NQ-UTD dataset, possibly by employing LLMs to support and streamline the annotation process (Zhang et al. , 2023). Such an approach could facilitate broader coverage and richer labeled query-passage pairs. Third, the construction of LLM-generated corpus in our benchmarked 16 datasets was significantly influenced by the inference costs of LLM, leading us to rely exclusively on Llama2 (Touvron et al. , 2023) for generating LLM-based content. However, the real-world IR landscape is shaped by a variety of LLMs. Future research might include content generated by an array of LLMs, such as OpenAI ChatGPT and Google Gemini, to mirror the diversity of LLM-generated content more accurately. Despite these limitations, we envision the Cocktail benchmark as a valuable resource for IR research in the LLM era, offering a foundation
upon which next-generation IR models can be built