OpenRaiser/Intern-Atlas huggingface dataset:
FairDiverse: A Comprehensive Toolkit for Fairness- and Diversity-aware Information Retrieval
Authors: Chen Xu, Zhirui Deng, Clara Rus, Xiaopeng Ye, Yuanna Liu, Jun Xu, Zhicheng Dou, Ji-Rong Wen, Maarten de Rijke
Venue: SIGIR
Year: 2025

In modern information retrieval (IR), going beyond accuracy is crucial for maintaining a healthy ecosystem, particularly in meeting fairness and diversity requirements. To address these needs, various datasets, algorithms, and evaluation methods have been developed. These algorithms are often tested with different metrics, datasets, and experimental settings, making comparisons inconsistent and challenging. Consequently, there is an urgent need for a comprehensive IR toolkit, enabling standardized assessments of fairness- and diversity-aware algorithms across IR tasks. To address these issues, we introduce an open-source standardized toolkit called FairDiverse. First, FairDiverse provides a comprehensive framework for incorporating fairness- and diversity-aware approaches, including pre-processing, in-processing, and post-processing methods, into different pipeline stages of IR. Second, FairDiverse enables the evaluation of 29 fairness, and diversity algorithms across 16 base models for two fundamental IR tasks-search and recommendation-facilitating the establishment of a comprehensive benchmark. Finally, FairDiverse is highly extensible, offering multiple APIs to enable IR researchers to quickly develop their own fairness- and diversity-aware IR models, and allows for fair comparisons with existing baselines. The project is open-sourced on GitHub:~ https://github.com/XuChen0427/FairDiverse.
Fairness in Information Retrieval from an Economic Perspective
Authors: Chen Xu, Clara Rus, Yuanna Liu, Marleen de Jonge, Jun Xu, Maarten de Rijke
Venue: SIGIR
Year: 2025

Recently, fairness-aware information retrieval (IR) systems have been receiving much attention. Numerous fairness metrics and algorithms have been proposed. The complexity of fairness and IR systems makes it challenging to provide a systematic summary of the progress that has been made. This complexity calls for a more structured framework to navigate future fairness-aware IR research directions. The field of economics has long explored fairness, offering a strong theoretical and empirical foundation. Its system-oriented perspective enables the integration of IR fairness into a broader framework that considers societal and intertemporal trade-offs. In this tutorial, we first highlight that IR systems can be understood as a specialized economic market. Then, we re-organize fairness algorithms through three key economic dimensions-macro vs. micro, demand vs supply, and short-term vs. long-term. We effectively view most fairness categories in IR from an economic perspective. Finally, we illustrate how this economic framework can be applied to various real-world IR applications and we demonstrate its benefits in industrial scenarios. Different from other fairness-aware tutorials, our tutorial not only provides a new and clear perspective to re-frame fairness-aware IR but also inspires the use of economic tools to solve fairness problems in IR. We hope this tutorial provides a fresh, broad perspective on fairness in IR, highlighting open problems and future research directions.
Understanding Accuracy-Fairness Trade-offs in Re-ranking through Elasticity in Economics
Authors: Chen Xu, Jujia Zhao, Wenjie Wang, Liang Pang, Jun Xu, Tat-Seng Chua, Maarten de Rijke
Venue: SIGIR
Year: 2025

Fairness is an increasingly important factor in re-ranking tasks. Prior work has identified a trade-off between ranking accuracy and item fairness. However, the underlying mechanisms are still not fully understood. An analogy can be drawn between re-ranking and the dynamics of economic transactions. The accuracy-fairness trade-off parallels the coupling of the commodity tax transfer process. Fairness considerations in re-ranking, similar to a commodity tax on suppliers, ultimately translate into a cost passed on to consumers. Analogously, item-side fairness constraints result in a decline in user-side accuracy. In economics, the extent to which commodity tax on the supplier (item fairness) transfers to commodity tax on users (accuracy loss) is formalized using the notion of elasticity. The re-ranking fairness-accuracy trade-off is similarly governed by the elasticity of utility between item groups. This insight underscores the limitations of current fair re-ranking evaluations, which often rely solely on a single fairness metric, hindering comprehensive assessment of fair re-ranking algorithms. Centered around the concept of elasticity, this work presents two significant contributions. We introduce the Elastic Fairness Curve (EF-Curve) as an evaluation framework. This framework enables a comparative analysis of algorithm performance across different elasticity levels, facilitating the selection of the most suitable approach. Furthermore, we propose ElasticRank, a fair re-ranking algorithm that employs elasticity calculations to adjust inter-item distances within a curved space. Experiments on three widely used ranking datasets
Rethinking Pseudo Word Learning in Zero-Shot Composed Image Retrieval: From an Object-Aware Perspective
Authors: Zhe Li, Lei Zhang, Kun Zhang, Weidong Chen, Yongdong Zhang, Zhendong Mao
Venue: SIGIR
Year: 2025

Composed Image Retrieval (CIR) takes a composed query of a reference image and a text describing the user's intention, with the aim to retrieve the target image under both conditions. Conventional CIR approaches heavily rely on massive annotated triplets, which often comes at a considerable cost. Zero-Shot CIR (ZS-CIR) offers a new solution that can perform diverse CIR tasks without training on the triplet datasets. The key to the ZS-CIR task is to make specified changes to specific objects in the reference image based on the text. Previous works utilize a projection module to map the reference image into single or multiple pseudo words. However, they are either only applicable to single-object scenarios, or naively convert entire image features into multiple pseudo words and fail to focus on the desired target objects specified by the text description. In this work, we rethink how to learn pseudo words based on the objects attended by the text and propose a Multi-Object Aware ZS-CIR framework (MOA). Specifically, a multi-object recognizer first recognizes valid objects in the reference image guided by a set of learnable object queries. Then, we devise an object filtering strategy, which utilizes contextual prompts comprised of noun categories to guide the model in precisely screening out the objects that need to be modified. Finally, the pseudo word learning branch adaptively converts the screened objects into multiple pseudo words for accurate ZS-CIR. Although simple, our MOA consistently outperforms previous state-of-the-art methods across diverse benchmarks and even achieves competitive results with many supervised methods.
Reassessing Large Language Model Boolean Query Generation for Systematic Reviews
Authors: Shuai Wang, Harrisen Scells, Bevan Koopman, Guido Zuccon
Venue: SIGIR
Year: 2025

Systematic reviews are comprehensive literature reviews that address highly focused research questions and represent the highest form of evidence in medicine. A critical step in this process is the development of complex Boolean queries to retrieve relevant literature. Given the difficulty of manually constructing these queries, recent efforts have explored Large Language Models (LLMs) to assist in their formulation. One of the first studies, Wang et al. [21], investigated ChatGPT for this task, followed by Staudinger et al. [14], which evaluated multiple LLMs in a reproducibility study. However, the latter overlooked several key aspects of the original work, including (i) validation of generated queries (ii) output formatting constraints, and (iii) selection of examples for chainof- thought (Guided) prompting. As a result, its findings diverged significantly from the original study. In this work, we systematically reproduce both studies while addressing these overlooked factors. Our results show that query effectiveness varies significantly across models and prompt designs, with guided query formulation benefiting from well-chosen seed studies. Overall, prompt design and model selection are key drivers of successful query formulation. Our findings provide a clearer understanding of LLMs' potential in Boolean query generation and highlight the importance of modeland prompt-specific optimisations. The complex nature of systematic reviews adds to challenges in both developing and reproducing methods but also highlights the importance of reproducibility studies in this domain.
2D Matryoshka Training for Information Retrieval
Authors: Shuai Wang, Shengyao Zhuang, Bevan Koopman, Guido Zuccon
Venue: SIGIR
Year: 2025

2D Matryoshka Training is an advanced embedding representation training approach designed to train an encoder model simultaneously across various layer-dimension setups. This method has demonstrated higher effectiveness in Semantic Text Similarity (STS) tasks over traditional training approaches when using sub-layers for embeddings. Despite its success, discrepancies exist between two published implementations, leading to varied comparative results with baseline models. In this reproducibility study, we implement and evaluate both versions of 2D Matryoshka Training on STS tasks and extend our analysis to retrieval tasks. Our findings indicate that while both versions achieve higher effectiveness than traditional Matryoshka training on sub-dimensions, and traditional full-sized model training approaches, they do not outperform models trained separately on specific sub-layer and sub-dimension setups. Moreover, these results generalize well to retrieval tasks, both in supervised (MSMARCO) and zero-shot (BEIR) settings. Further explorations of different loss computations reveals more suitable implementations for retrieval tasks, such as incorporating full-dimension loss and training on a broader range of target dimensions. Conversely, some intuitive approaches, such as fixing document encoders to full model outputs, do not yield improvements. Our reproduction code is available at https://github.com/ielab/2DMSE-Reproduce.
Retrieving Tables via Inter- and Intra-Content Contrastive Representation Learning
Authors: Jie Wu, Mengshu Hou
Venue: SIGIR
Year: 2025

Contrastive learning has emerged as a highly effective and versatile technique in information retrieval. However, its application within table retrieval remains limited, and often neglecting a typical phenomenon in table retrieval: one table can be associated with multiple, distinct queries. Directly applying traditional contrastive learning strategies may lead to semantic contrast conflicts during training, potentially impairing the quality of learned representations. Additionally, many current table retrieval methods still operate on query-table joint encoding, which introduces notable inefficiencies during both the training and retrieval processes. For this issue, this paper proposes ConTR, a tabular semantic contrastive learning method that simultaneously considers both inter-table and intra-table differences. By segmenting table into multiple vector representations and enabling the matching of diverse queries through differentiated vectors, thereby facilitating a more focused contrastive learning process. Retrieval experiments based on two typical table-related tasks validate the feasibility and effectiveness of proposed method.
Towards Distribution Matching between Collaborative and Language Spaces for Generative Recommendation
Authors: Yi Zhang, Yiwen Zhang, Yu Wang, Tong Chen, Hongzhi Yin
Venue: SIGIR
Year: 2025

Generative recommendation aims to learn the underlying generative process over the entire item set to produce recommendations for users. Although it leverages non-linear probabilistic models to surpass the limited modeling capacity of linear factor models, it is often constrained by a trade-off between representation ability and tractability. With the rise of a new generation of generative methods based on pre-trained language models (LMs), incorporating LMs into general recommendation with implicit feedback has gained considerable attention. However, adapting them to generative recommendation remains challenging. The core reason lies in the mismatch between the input-output formats and semantics of generative models and LMs, making it challenging to achieve optimal alignment in the feature space. This work addresses this issue by proposing a model-agnostic generative recommendation framework called DMRec, which introduces a probabilistic meta-network to bridge the outputs of LMs with user interactions, thereby enabling an equivalent probabilistic modeling process. Subsequently, we design three cross-space distribution matching processes aimed at maximizing shared information while preserving the unique semantics of each space and filtering out irrelevant information. We apply DMRec to three different types of generative recommendation methods and conduct extensive experiments on three public datasets. The experimental results demonstrate that DMRec can effectively enhance the recommendation performance of these generative models, and it shows significant advantages over mainstream LM-enhanced recommendation methods.
DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward
Authors: Yi Zhang, Ruihong Qiu, Xuwei Xu, Jiajun Liu, Sen Wang
Venue: SIGIR
Year: 2025

Model-based offline reinforcement learning (RL) has emerged as a promising approach for recommender systems, enabling effective policy learning by interacting with frozen world models. However, the reward functions in these world models, trained on sparse offline logs, often suffer from inaccuracies. Specifically, existing methods face two major limitations in addressing this challenge: (1) deterministic use of reward functions as static look-up tables, which propagates inaccuracies during policy learning, and (2) static uncertainty designs that fail to effectively capture decision risks and mitigate the impact of these inaccuracies. In this work, a dual-agent framework, DARLR, is proposed to dynamically update world models to enhance recommendation policies. To achieve this, a selector is introduced to identify reference users by balancing similarity and diversity so that the recommender can aggregate information from these users and iteratively refine reward estimations for dynamic reward shaping. Further, the statistical features of the selected users guide the dynamic adaptation of an uncertainty penalty to better align with evolving recommendation requirements. Extensive experiments on four benchmark datasets demonstrate the superior performance of DARLR, validating its effectiveness. The code is available at this address.
Triplet Contrastive Learning with Learnable Sequence Augmentation for Sequential Recommendation
Authors: Wei Wang, Yujie Lin, Moyan Zhang, Hongyu Lu, Jianli Zhao, Jie Sun, Xianye Ben, Pengjie Ren, Yujun Li
Venue: SIGIR
Year: 2025

The quality of augmented data directly affects the performance of contrastive learning. Low-quality augmentation offers limited benefits for model optimization. Existing contrastive learning-based sequential recommendation works primarily utilize heuristic data augmentation methods, which often exhibit excessive randomness and struggle to generate positive samples that align with users' true intentions. To address this limitation, we propose Triplet Contrastive learning with Learnable sequence Augmentation for sequential Recommendation (TCLARec). Unlike heuristic or rule-based augmentation methods, we train a learnable sequence augmentation module that can automatically leverage the self-supervised information from global context to select appropriate modification positions and augmentation operations, thereby generating positive samples that more accurately reflect user preferences. Furthermore, we design a ranking-based triplet contrastive loss to enhance the positive feedback from augmented sequence generated by the augmentation module, providing more nuanced contrastive signals for model optimization. Extensive experiments on three real-world datasets demonstrate that TCLARec outperforms state-of-the-art sequential recommendation baselines. Our in-depth analyses confirm that both the learnable augmentation and triplet contrastive learning contribute to improving recommendation accuracy. We have released the code at https://github.com/anonymityww/TCLA.
Conversational Search: Towards Personalization and Evaluation
Authors: Zahra Abbasiantaeb
Venue: SIGIR
Year: 2025

Conversational information seeking (CIS) systems aim to understand a user's evolving information needs in the context of a conversation and respond effectively. These systems are especially valuable for users who may struggle with traditional interfaces, offering a more natural and accessible mode of interaction. To ensure the groundedness and accuracy of responses, existing methods break the task into several subtasks, namely, dialogue context modeling, retrieval, and answer generation. The user's information need is often represented by either a single rewritten query or a single representation in the query embedding space. This leads to several limitations, especially in cases where the information need cannot be answered using a single passage and requires complex reasoning over multiple facts from different sources. We address this limitation by proposing the MQ4CS model, which breaks the information need of the user into multiple queries with different aspects. In MQ4CS, the retrieval is done for each query and rank list fusion is done over the list of documents retrieved for each query. Evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an open problem. The quality of the response is mainly assessed using surface-based QA metrics, human evaluation, or instructing the Large Language Models (LLMs). Since RAG-generated responses integrate both retrieved documents and the LLMs's internal knowledge, traditional surface-based QA metrics are not effective for assessment. Furthermore, existing RAG benchmarks address more complex information needs compared to ad-hoc retrieval and QA, highlighting the necessity of measuring the completeness of generated responses. To better evaluate completeness and correctness, we propose a nugget-based evaluation pipeline called CONE-RAG that measures the precision and recall of key information nuggets in generated answers. Personalization is an emerging key challenge in Conversational Search (CS). A personalized CS system must adapt its response based on the user's personal information and search history. Given the same user question, the response of a personalized CS system must be different for different users. To facilitate the research on the development and evaluation of personalized CS systems, I have co-organized the iKAT track at TREC, where we released the iKAT 2023 and 2024 datasets.
Conversational Gold: Evaluating Personalized Conversational Search System Using Gold Nuggets
Authors: Zahra Abbasiantaeb, Simon Lupart, Leif Azzopardi, Jeffrey Dalton, Mohammad Aliannejadi
Venue: SIGIR
Year: 2025

The rise of personalized conversational search systems has been driven by advancements in Large Language Models (LLMs), enabling these systems to retrieve and generate answers for complex information needs. However, the automatic evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an understudied challenge. In this paper, we introduce a new resource for assessing the retrieval effectiveness and relevance of responses generated by RAG systems, using a nugget-based evaluation framework. Built upon the foundation of TREC iKAT 2023, our dataset extends to the TREC iKAT 2024 collection, which includes 17 conversations and 20,575 relevance passage assessments, together with 2,279 extracted gold nuggets and 62 manually written gold answers from NIST assessors. While maintaining the core structure of its predecessor, this new collection enables a deeper exploration of generation tasks in conversational settings. Key improvements in iKAT 2024 include: (1) ''gold nuggets'' - concise, essential pieces of information extracted from relevant passages of the collection - which serve as a foundation for automatic response evaluation; (2) manually written answers to provide a gold standard for response evaluation; (3) expanded user personas, providing richer contextual grounding; and (4) a transition from Personal Text Knowledge Base (PTKB) ranking to PTKB classification and selection. Built on this resource, we provide a framework for long-form answer generation evaluation, involving nugget extraction and nugget matching, linked to retrieval. This establishes a solid resource for advancing research in personalized conversational search and long-form answer generation. Our resources are publicly available at https://github.com/irlabamsterdam/CONE-RAG.
LLM-based Query Expansion Fails for Unfamiliar and Ambiguous Queries
Authors: Kenya Abe, Kunihiro Takeoka, Makoto P. Kato, Masafumi Oyamada
Venue: SIGIR
Year: 2025

Query expansion (QE) enhances retrieval by incorporating relevant terms, with large language models (LLMs) offering an effective alternative to traditional rule-based and statistical methods. However, LLM-based QE suffers from a fundamental limitation: it often fails to generate relevant knowledge, degrading search performance. Prior studies have focused on hallucination, yet its underlying cause-LLM knowledge deficiencies-remains underexplored. This paper systematically examines two failure cases in LLM-based QE: (1) when the LLM lacks query knowledge, leading to incorrect expansions, and (2) when the query is ambiguous, causing biased refinements that narrow search coverage. We conduct controlled experiments across multiple datasets, evaluating the effects of knowledge and query ambiguity on retrieval performance using sparse and dense retrieval models. Our results reveal that LLM-based QE can significantly degrade the retrieval effectiveness when knowledge in the LLM is insufficient or query ambiguity is high. We introduce a framework for evaluating QE under these conditions, providing insights into the limitations of LLM-based retrieval augmentation.
CoSRec: A Joint Conversational Search and Recommendation Dataset
Authors: Marco Alessio, Simone Merlo, Tommaso Di Noia, Guglielmo Faggioli, Marco Ferrante, Nicola Ferro, Cristina Ioana Muntean, Franco Maria Nardini, Fedelucio Narducci, Raffaele Perego, Giuseppe Santucci, Nicola Viterbo
Venue: SIGIR
Year: 2025

Conversational Information Access systems have experienced widespread diffusion thanks to the natural and effortless interactions they enable with the user. In particular, they represent an effective interaction interface for conversational search (CS) and conversational recommendation (CR) scenarios. Despite their commonalities, CR and CS systems are often devised, developed, and evaluated as isolated components. Integrating these two elements would allow for handling complex information access scenarios, such as exploring unfamiliar recommended product aspects, enabling richer dialogues, and improving user satisfaction. As of today, the scarce availability of integrated datasets - focused exclusively on either of the tasks - limits the possibilities for evaluating by-design integrated CS and CR systems. To address this gap, we propose CoSRec, the first dataset for joint Conversational Search and Recommendation (CSR) evaluation. The CoSRec test set includes 20 high-quality conversations, with human-made annotations for the quality of conversations, and manually crafted relevance judgments for products and documents. Additionally, we provide supplementary training data comprising partially annotated dialogues and raw conversations to support diverse learning paradigms. CoSRec is the first resource to model CR and CS tasks in a unified framework, enabling the training and evaluation of systems that must shift between answering queries and making suggestions dynamically.