OpenRaiser/Intern-Atlas huggingface dataset:
Efficient In-Memory Inverted Indexes: Theory and Practice
Authors: Joel Mackenzie, Sean MacAvaney, Antonio Mallia, Michal Siedlaczek
Venue: SIGIR
Year: 2025

Inverted indexes are the backbone of most large-scale information retrieval systems. Although conceptually simple, high-performance inverted indexes require a deep understanding of low-level system optimizations, compression techniques, and traversal strategies. With the widespread adoption of in-memory search engines, the rise of learned sparse retrieval (LSR), and the increasing complexity of ranking pipelines, the design space for efficient indexing and retrieval systems has expanded significantly. This tutorial addresses a critical knowledge gap between textbook-style explanations and advanced techniques required for efficient and optimized retrieval. It aims to equip researchers and practitioners with a comprehensive understanding of how modern in-memory search systems are designed, built, and optimized for high-performance retrieval across large-scale document collections.
DataRec: A Python Library for Standardized and Reproducible Data Management in Recommender Systems
Authors: Alberto Carlo Maria Mancino, Salvatore Bufi, Angela Di Fazio, Antonio Ferrara, Daniele Malitesta, Claudio Pomo, Tommaso Di Noia
Venue: SIGIR
Year: 2025

Recommender systems have demonstrated a significant impact across diverse domains, yet ensuring the reproducibility of experimental findings remains a persistent challenge. A primary obstacle lies in the fragmented and often opaque data management strategies employed during the preprocessing stage, where decisions about dataset selection, filtering, and splitting can substantially influence outcomes. To address these limitations, we introduce DataRec, an open-source Python-based library specifically designed to unify and streamline data handling in recommender system research. By providing reproducible routines for dataset preparation, data versioning, and seamless integration with other frameworks, DataRec promotes methodological standardization, interoperability, and comparability across different experimental setups. Our design is informed by an in-depth review of 55 state-of-the-art recommendation studies, ensuring that DataRec adopts best practices while addressing common pitfalls in data management. Ultimately, our contribution facilitates fair benchmarking, enhances reproducibility, and fosters greater trust in experimental results within the broader recommender systems community. The DataRec library, documentation, and examples are freely available at https://github.com/sisinflab/DataRec.
AiReview: An Open Platform for Accelerating Systematic Reviews with LLMs
Authors: Xinyu Mao, Teerapong Leelanupab, Martin Potthast, Harrisen Scells, Guido Zuccon
Venue: SIGIR
Year: 2025

Systematic reviews are fundamental to evidence-based medicine. Creating one is time-consuming and labour-intensive, mainly due to the need to screen, or assess, many studies for inclusion in the review. Existing tools help streamline this process, mostly using traditional machine learning. Large language models (LLMs) offer new opportunities to speed up screening, yet no tool currently enables users to directly apply LLMs or ensures systematic and transparent use of these methods. This paper presents (i) a flexible framework for using LLMs in systematic review tasks, especially title and abstract screening, and (ii) a web-based interface for LLM-assisted screening. Together, they form AiReview-a novel platform that connects cutting-edge LLM-assisted screening methods with real-world systematic review practice. The live tool is available at https://aireview.ielab.io. We also release the code publicly at https://github.com/ielab/ai-review.
Efficient Approximate Nearest Neighbor Search on a Raspberry Pi
Authors: Silvio Martinico, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini
Venue: SIGIR
Year: 2025

Approximate Nearest Neighbors (ANN) search is a core task in Information Retrieval. However, the high computational demands and reliance on expensive infrastructures limit broader contributions to ANN research. Enabling efficient and effective ANN search on low-resource devices would allow researchers in low-income countries to participate in the ANN community, thereby democratizing the field. Despite its potential, the IR literature offers little work on the feasibility of ANN search under resource constraints. In this proposal, we explore efficient solutions for large-scale ANN search on low-resource devices. We report a preliminary experimentation highlighting current limitations and outlining future challenges.
2nd Workshop on Information Retrieval for Understudied Users (IR4U2) - Bridging User-centered AI with IR: Making Information Retrieval Accessible for All
Authors: Noemi Mauro, Angelo Geninatti Cossatin, Maria Soledad Pera, Federica Cena, Monica Landoni, Theo Huibers, Emiliana Murgia
Venue: SIGIR
Year: 2025

The Workshop on Information Retrieval for Understudied Users (IR4U2) serves as a platform to highlight information retrieval (IR) research that directly impacts often understudied user groups. The second (IR4U2) workshop focuses on a user-centred AI perspective, which is vital for informing the design, development, and assessment of information retrieval systems that thoughtfully address the diverse needs of understudied populations, ensuring genuine accessibility and inclusivity. The objectives of IR4U2 are: (1) to build community and awareness by sharing AI and IR developments that serve underrepresented user groups in this research area; (2) to identify challenges and open issues along with lessons learned and challenges inherent to this area of research; and (3) to spark discussions that establish common frameworks for future research.
Towards Brain Passage Retrieval: An Investigation of EEG Query Representations
Authors: Niall McGuire, Yashar Moshfeghi
Venue: SIGIR
Year: 2025

Information Retrieval (IR) systems primarily rely on users' ability to translate their internal information needs into (text) queries. However, this translation process is often uncertain and cognitively demanding, leading to queries that incompletely or inaccurately represent users' true needs. This challenge is particularly acute for users with ill-defined information needs or physical impairments that limit traditional text input, where the gap between cognitive intent and query expression becomes even more pronounced. Recent neuroscientific studies have explored Brain-Machine Interfaces (BMIs) as a potential solution, aiming to bridge the gap between users' cognitive semantics and their search intentions. However, current approaches attempting to decode explicit text queries from brain signals have shown limited effectiveness in learning robust brain-to-text representations, often failing to capture the nuanced semantic information present in brain patterns. To address these limitations, we propose BPR (Brain Passage Retrieval), a novel framework that eliminates the need for intermediate query translation by enabling direct retrieval of relevant passages from users' brain signals. Our approach leverages dense retrieval architectures to map EEG signals and text passages into a shared semantic space. Through comprehensive experiments on the ZuCo dataset, we demonstrate that BPR achieves up to 8.81% improvement in precision@5 over existing EEG-to-text baselines, while maintaining effectiveness across 30 participants. Our ablation studies reveal the critical role of hard negative sampling and specialised brain encoders in achieving robust cross-modal alignment. These results establish the viability of direct brain-to-passage retrieval and provide a foundation for developing more natural interfaces between users' cognitive states and IR systems.
Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems
Authors: Jack McKechnie, Graham McDonald, Craig Macdonald
Venue: SIGIR
Year: 2025

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.
Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
Authors: Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke
Venue: SIGIR
Year: 2025

Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a unified global retrieval objective. However, existing GenIR models suffer from token-level misalignment, where models trained to predict the next token often fail to capture document-level relevance effectively. While reinforcement learning-based methods, such as reinforcement learning from relevance feedback (RLRF), aim to address this misalignment through reward modeling, they introduce significant complexity, requiring the optimization of an auxiliary reward function followed by reinforcement fine-tuning, which is computationally expensive and often unstable. To address these challenges, we propose direct document relevance optimization (DDRO), which aligns token-level docid generation with document-level relevance estimation through direct optimization via pairwise ranking, eliminating the need for explicit reward modeling and reinforcement learning. Experimental results on benchmark datasets, including MS MARCO document and Natural Questions, show that DDRO outperforms reinforcement learning-based methods, achieving a 7.4% improvement in MRR@10 for MS MARCO and a 19.9% improvement for Natural Questions. These findings highlight DDRO's potential to enhance retrieval effectiveness with a simplified optimization approach. By framing alignment as a direct optimization problem, DDRO simplifies the ranking optimization pipeline of GenIR models while offering a viable alternative to reinforcement learning-based methods
A Generative Re-ranking Model for List-level Multi-objective Optimization at Taobao
Authors: Yue Meng, Cheng Guo, Yi Cao, Tong Liu, Bo Zheng
Venue: SIGIR
Year: 2025

E-commerce recommendation systems aim to generate ordered lists of items for customers, optimizing multiple business objectives, such as clicks, conversions and Gross Merchandise Volume (GMV). Traditional multi-objective optimization methods like formulas or Learning-to-rank (LTR) models take effect at item-level, neglecting dynamic user intent and contextual item interactions. List-level multi-objective optimization in the re-ranking stage can overcome this limitation, but most current re-ranking models focus more on accuracy improvement with context. In addition, re-ranking is faced with the challenges of time complexity and diversity. In light of this, we propose a novel end-to-end generative re-ranking model named Sequential Ordered Regression Transformer-Generator (SORT-Gen) for the less-studied list-level multi-objective optimization problem. Specifically, SORT-Gen is divided into two parts: 1)Sequential Ordered Regression Transformer innovatively uses Transformer and ordered regression to accurately estimate multi-objective values for variable-length sub-lists. 2)Mask-Driven Fast Generation Algorithm combines multi-objective candidate queues, efficient item selection and diversity mechanism into model inference, providing a fast online list generation method. Comprehensive online experiments demonstrate that SORT-Gen brings +4.13% CLCK and +8.10% GMV for Baiyibutie, a notable Mini-app of Taobao. Currently, SORT-Gen has been successfully deployed in multiple scenarios of Taobao App, serving for a vast number of users.
Fairness-Aware Classification over Incomplete Data
Authors: Xiaoye Miao, Lei Qiang, Guilin Huang, Yangyang Wu, Jianwei Yin
Venue: SIGIR
Year: 2025

The missing values widely existed in tabular data hinder the effective analysis of algorithmic fairness. Existing fairness intervention algorithms incorporate constraints or regularizers to reduce discrimination which rely on the complete information. They cannot effectively handle common tabular data with missing values in both sensitive and non-sensitive attributes without imputation. In this paper, we propose a novel Transformer-based fairness-aware prediction model FATE that mitigates the bias introduced by missing values to achieve algorithmic fairness without imputation. FATE consists of two modules, i.e., an incomplete data encoding (IDE) module and a debiased representation learning (DRL) module. IDE designs an incomplete tabular data embedding strategy and a missingness-aware Transformer block to effectively learn the observed data distribution and the missing state information. DRL converts fairness into attention parity when the sensitive attributes are completely missing. It offers a debiased attention mechanism to normalize attention weights in the attention score calculation process. We theoretically prove that, the differences in attention scores can represent the demographic disparities among sensitive groups which in FATE are bounded by a constant, substantially minimizing the group discrimination. Extensive experiments on three public real-world datasets demonstrate that, FATE, with the competitive fairness, yields more than 22% accuracy gain, compared to the state of the arts.
Conversational Search: From Fundamentals to Frontiers in the LLM Era
Authors: Fengran Mo, Chuan Meng, Mohammad Aliannejadi, Jian-Yun Nie
Venue: SIGIR
Year: 2025

Conversational search enables multi-turn interactions between users and systems to fulfill users' complex information needs. During this interaction, the system should understand the users' search intent within the conversational context and then return the relevant information through a flexible, dialogue-based interface. The recent powerful large language models (LLMs) with capacities of instruction following, content generation, and reasoning, attract significant attention and advancements, providing new opportunities and challenges for building up intelligent conversational search systems. This tutorial aims to introduce the connection between fundamentals and the emerging topics revolutionized by LLMs in the context of conversational search. It is designed for students, researchers, and practitioners from both academia and industry. Participants will gain a comprehensive understanding of both the core principles and cutting-edge developments driven by LLMs in conversational search, equipping them with the knowledge needed to contribute to the development of next-generation conversational search systems.
A Flexible Resource for Top-Weighted Comparisons Between Sets and Rankings
Authors: Alistair Moffat, Joel Mackenzie, Antonio Mallia, Matthias Petri
Venue: SIGIR
Year: 2025

We describe rbstar a toolkit of software for carrying out measurements when the goal is to determine how similar a system observation is to a gold-standard reference output. The resource covers all four combinations that arise when each of observation and reference can be either an unordered finite set in which element ordering is unimportant, or a finite prefix of an arbitrarily long ranking in which early elements are more important than later ones. Specifically, the package realizes four ''rank-biased'' measurement approaches that have been presented in a sequence of papers over a 15-year span, bringing them together into a single location with a uniform interface and efficient reference implementations. The provision of all of rank-biased precision, rank-biased overlap, rank-biased recall, and rank-biased alignment, with the latter two recent additions to the family, allows a wide range of measurement scenarios to be handled in a consistent manner.
The Agent Perspective In LLM-Based Strategic Information Retrieval Ecosystems
Authors: Tommy Mordo
Venue: SIGIR
Year: 2025

The rise of generative AI and the emergence of Large Language Models (LLMs) have sparked a new line of research on the evolution of strategic information retrieval ecosystems. My research adopts an agent-based perspective to analyze sponsored search and competitive search ecosystems. The primary motivation of my work is to examine how agent strategies influence social welfare and the existence of equilibrium. To that end, I leverage frameworks from game theory and mechanism design.
LEMSS: LLM-Based Platform for Multi-Agent Competitive Search Simulation
Authors: Tommy Mordo, Tomer Kordonsky, Haya Nachimovsky, Moshe Tennenholtz, Oren Kurland
Venue: SIGIR
Year: 2025

In competitive search settings, document publishers (authors) respond to rankings induced for queries of interest: they modify the documents to improve their future ranking. Hence, for some queries there is an on-going ranking competition. Prior empirical studies of competitive search were based on controlled ranking competitions between humans. Large Language Models (LLMs), capable of generating high quality content, provide new opportunities for studying ranking competitions. Furthermore, there is a significant amount of content on the Web, which is a canonical example of a competitive search setting, generated by LLMs. In this paper, we introduce LEMSS: a multi-agent platform that leverages LLMs as publishers in competitive search settings. In addition to enabling the execution of large-scale and highly configurable ranking competitions, LEMSS includes tools to analyze and compare the competitions using a wide range of measures. We use these tools to analyze examples of datasets that result from ranking competitions executed using LEMSS. The analysis reveals, for example, that using LLMs as publishers reduced content diversity in the corpus to a larger extent than having human publishers.
Brain-Machine Interfaces & Information Retrieval Challenges and Opportunities
Authors: Yashar Moshfeghi, Niall McGuire
Venue: SIGIR
Year: 2025

The fundamental goal of Information Retrieval (IR) systems lies in their capacity to effectively satisfy human information needs - a challenge that encompasses not just the technical delivery of information, but the nuanced understanding of human cognition during information seeking. Contemporary IR platforms rely primarily on observable interaction signals, creating a fundamental gap between system capabilities and users' cognitive processes. Brain-Machine Interface (BMI) technologies now offer unprecedented potential to bridge this gap through direct measurement of previously inaccessible aspects of information-seeking behaviour. This perspective paper offers a broad examination of the IR landscape, providing a comprehensive analysis of how BMI technology could transform IR systems, drawing from advances at the intersection of both neuroscience and IR research. We present our analysis through three identified fundamental vertices: (1) understanding the neural correlates of core IR concepts to advance theoretical models of search behaviour (2) enhancing existing IR systems through contextual integration of neurophysiological signals, and (3) developing proactive IR capabilities through direct neurophysiological measurement. For each vertex, we identify specific research opportunities and propose concrete directions for developing BMI-enhanced IR systems. We conclude by examining critical technical and ethical challenges in implementing these advances, providing a structured roadmap for future research at the intersection of neuroscience and IR.
Revisiting Algorithmic Audits of TikTok: Poor Reproducibility and Short-term Validity of Findings
Authors: Matej Mosnar, Adam Skurla, Branislav Pecher, Matus Tibensky, Jan Jakubcik, Adrian Bindas, Peter Sakalik, Ivan Srba
Venue: SIGIR
Year: 2025

Social media platforms are constantly shifting towards algorithmically curated content based on implicit or explicit user feedback. Regulators, as well as researchers, are calling for systematic social media algorithmic audits as this shift leads to enclosing users in filter bubbles and leading them to more problematic content. An important aspect of such audits is the reproducibility and generalisability of their findings, as it allows to draw verifiable conclusions and audit potential changes in algorithms over time. In this work, we study the reproducibility of the existing sockpuppeting audits of TikTok recommender systems, and the generalizability of their findings. In our efforts to reproduce the previous works, we find multiple challenges stemming from social media platform changes and content evolution, but also the research works themselves. These drawbacks limit the audit reproducibility and require an extensive effort altogether with inevitable adjustments to the auditing methodology. Our experiments also reveal that these one-shot audit findings often hold only in the short term, implying that the reproducibility and generalizability of the audits heavily depend on the methodological choices and the state of algorithms and content on the platform. This highlights the importance of reproducible audits that allow us to determine how the situation changes in time.
Zero-Shot Reranking with Large Language Models and Precomputed Ranking Features: Opportunities and Limitations
Authors: Maria Movin, Claudia Hauff
Venue: SIGIR
Year: 2025

LLMs have been explored for their use in IR as end-to-end rankers, rerankers and assessors. Recently, the exploration of the prompt-and-predict paradigm for reranking in combination with highly performant LLMs have drawn the attention of researchers. Instead of training or fine-tuning a reranker, LLMs are prompted in a zero-shot manner to produce relevance scores, pairwise preferences, or reranked lists. Existing research, though, has been confined to unstructured text corpora, leaving a gap in our understanding: to what extent do the findings of zero-shot LLM rerankers established on plain text corpora hold for datasets containing predominantly precomputed ranking features as is common in industrial settings? We explore this question via an empirical study on one public learning-to-rank dataset (MSLR-WEB10K) and two datasets collected from an audio streaming platform's search logs. Our results paint a differentiated picture: On average, there remains a significant performance gap: prompting the high-capacity LLM GPT-4 results in up to 16% lower NDCG@10 compared to the traditional supervised learning-to-rank (LTR) approach LambdaMART on the public MSLR-WEB10K dataset. However, when focusing only on a subset of hard queries-i.e. queries where the LTR approach ranks a non-relevant document at the top-the zero-shot LLM reranking outperforms the LTR baseline. We confirm the same trends on two proprietary audio search datasets. We also provide insights into prompt design choices and their impact on LLM reranking. We show that LLMs remain brittle, with the same strategies sometimes helping or hurting depending on the model size and dataset.