OpenRaiser/Intern-Atlas huggingface dataset:
Variations in Relevance Judgments and the Shelf Life of Test Collections
Authors: Andrew Parry, Maik Fröbe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Faggioli, Saber Zerhoudi, Sean MacAvaney, Eugene Yang
Venue: SIGIR
Year: 2025

The fundamental property of Cranfield-style evaluations, that system rankings are stable even when assessors disagree on individual relevance decisions, was validated on traditional test collections. However, the paradigm shift towards neural retrieval models affected the characteristics of modern test collections, e.g., documents are short, judged with four grades of relevance, and information needs have no descriptions or narratives. Under these changes, it is unclear whether assessor disagreement remains negligible for system comparisons. We investigate this aspect under the additional condition that the few modern test collections are heavily re-used. Given more possible query interpretations due to less formalized information needs, an ''expiration date'' for test collections might be needed if top-effectiveness requires overfitting to a single interpretation of relevance. We run a reproducibility study and re-annotate the relevance judgments of the 2019~TREC Deep Learning track. We can reproduce prior work in the neural retrieval setting, showing that assessor disagreement does not affect system rankings. However, we observe that some models substantially degrade with our new relevance judgments, and some have already reached the effectiveness of humans as rankers, providing evidence that test collections can expire.
KIMERA: From Evaluation-as-a-Service to Evaluation-in-the-Cloud
Authors: Andrea Pasin, Nicola Ferro
Venue: SIGIR
Year: 2025

Experimental evaluation steers the development of Information Retrieval (IR) systems, and large-scale evaluation campaigns provide the field with a common infrastructure to conduct comparable evaluation exercises. Over the years, tools and platforms have been developed to manage and automate these activities, enhance the reproducibility of conducted experiments and facilitate data sharing. In this context, Evaluation-as-a-Service (EaaS) emerged as an approach to avoid distributing experimental collections, which may contain copyrighted or sensitive data, and instead execute containerised code on that data on remote servers. We propose Kubernetes Infrastructure for Managed Evaluation and Resource Access (KIMERA) as the next step from EaaS into Evaluation-in-the-Cloud (EitC), allowing researchers to directly code and execute their systems through their browsers, requiring only an internet connection. Moreover, recent advancements, such as Large Language Models, or new computing paradigms, such as quantum computers, require external third party services and computational resources. In this respect, KIMERA streamlines and simplifies access to such services on-demand via their APIs. More in detail, KIMERA relies on state-of-the-art containerization and orchestration tools, such as Docker and Kubernetes, to provide a robust, scalable, secure, and fault-tolerant IR evaluation platform. KIMERA monitors and stores all the participants' submissions, accurately keeping track of the resource usage, allowing for evaluating both the efficiency and the effectiveness of the deployed methods. Moreover, all participants can be assigned workspaces sharing the same resources (i.e., CPU and RAM), thus enhancing reproducibility and comparability among systems. Finally, KIMERA has been designed with modularity and extensibility in mind, allowing it to be easily adapted to new use cases and usage scenarios. KIMERA has been developed and adopted in the context of the QuantumCLEF lab, to allow for mixed experiments, comparing approaches running on traditional hardware and on real quantum annealers provided by external companies. KIMERA has also been used as a learning resource to provide Quantum Computing tutorials for IR at major conferences, such as ECIR and SIGIR. The source code of KIMERA is openly available at https://github.com/MjPaxter/KIMERA.
The Effects of Demographic Instructions on LLM Personas
Authors: Angel Felipe Magnossão de Paula, J. Shane Culpepper, Alistair Moffat, Sachin Pathiyan Cherumanal, Falk Scholer, Johanne R. Trippas
Venue: SIGIR
Year: 2025

Social media platforms must filter sexist content in compliance with governmental regulations. Current machine learning approaches can reliably detect sexism based on standardized definitions, but often neglect the subjective nature of sexist language and fail to consider individual users' perspectives. To address this gap, we adopt a perspectivist approach, retaining diverse annotations rather than enforcing gold-standard labels or their aggregations, allowing models to account for personal or group-specific views of sexism. Using demographic data from Twitter, we employ large language models (LLMs) to personalize the identification of sexism. Our empirical results show that OpenAI's LLMs (GPT-3.5, GPT-4, and GPT-4o) and two open-source LLMs (Mistral and Qwen) exhibit higher Krippendorff's alpha label agreement with female annotators than with male annotators. As well, each LLM presents higher Krippendorff's alpha agreement with a specific annotator age group. We then sought to counter these trends by providing ''persona'' instructions as part of the LLM prompt, with somewhat surprising outcomes, highlighting the potential of user-centered perspectivist methods to improve content moderation systems.
Some Things Never Change: Overcoming Persistent Challenges in Children IR
Authors: Maria Soledad Pera, Theo Huibers, Emiliana Murgia, Monica Landoni
Venue: SIGIR
Year: 2025

There is a lack of a steady and solid influx of information retrieval (IR) research that has children (as the user group) as the protagonist. Existing work is scattered, conducted by only a few research groups, and often based on small-scale user studies or data that cannot be widely shared. Moreover, much of the current research focuses on specific age ranges and abilities, neglecting the broader spectrum of children's needs. Consequently, the paucity of IR research on how search and recommender systems serve and/or ultimately affect children translates into one of many 'Low-resource environments' in IR. Drawing from the literature and our experience in this area, we highlight key challenges and encourage greater attention from the IR community to address this critical gap.
Efficient Recommendation with Millions of Items by Dynamic Pruning of Sub-Item Embeddings
Authors: Aleksandr V. Petrov, Craig Macdonald, Nicola Tonellotto
Venue: SIGIR
Year: 2025

A large item catalogue is a major challenge for deploying modern sequential recommender models, since it makes the memory footprint of the model large and increases inference latency. One promising approach to address this is RecJPQ, which replaces item embeddings with sub-item embeddings. However, slow inference remains problematic because finding the top highest-scored items usually requires scoring all items in the catalogue, which may not be feasible for large catalogues. By adapting dynamic pruning concepts from document retrieval, we propose the RecJPQPrune dynamic pruning algorithm to efficiently find the top highest-scored items without computing the scores of all items in the catalogue. Our RecJPQPrune algorithm is safe-up-to-rank K since it theoretically guarantees that no potentially high-scored item is excluded from the final top K recommendation list, thereby ensuring no impact on effectiveness. Our experiments on two large datasets and three recommendation models demonstrate the efficiency achievable using RecJPQPrune: for instance, on the Tmall dataset with 2.2M items, we can reduce the median model scoring time by 64× compared to the Transformer Default baseline, and 5.3× compared to a recent scoring approach called PQTopK. Overall, this paper demonstrates the effective and efficient inference of Transformer-based recommendation models at catalogue scales not previously reported in the literature. Indeed, our RecJPQPrune algorithm can score 2 million items in under 10 milliseconds without GPUs, and without relying on Approximate Nearest Neighbour (ANN) techniques.
Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion
Authors: Jakub Podolak, Leon Peric, Mina Janicijevic, Roxana Petcu
Venue: SIGIR
Year: 2025

This study presents a comprehensive reproducibility analysis and extension of the Setwise prompting method for zero-shot ranking with Large Language Models (LLMs), as proposed by Zhuang et al. We evaluate the method's effectiveness and efficiency compared to traditional Pointwise, Pairwise, and Listwise approaches in document ranking tasks. Our reproduction confirms the findings of Zhuang et al., highlighting the trade-offs between computational efficiency and ranking effectiveness in Setwise methods. Building on these insights, we introduce Setwise Insertion, a novel approach that leverages the initial document ranking as prior knowledge, reducing unnecessary comparisons and uncertainty by prioritizing candidates more likely to improve the ranking results. Experimental results across multiple LLM architectures - Flan-T5, Vicuna, and Llama - show that Setwise Insertion yields a 31% reduction in query time, a 23% reduction in model inferences, and a slight improvement in reranking effectiveness compared to the original Setwise method. These findings highlight the practical advantage of incorporating prior ranking knowledge into Setwise prompting for efficient and accurate zero-shot document reranking.
RelEx: An XAI-Enhanced Relevance Feedback Model for User-Adaptive Explanations
Authors: Sayantan Polley, Govind Shukla, Pritha Ghosal, Andreas Nürnberger
Venue: SIGIR
Year: 2025

The rise of Gen-AI and LLMs often makes it difficult for users to trust retrieved results. The risk of IR systems using LLMs and being susceptible to misinformation can be tackled under the lens of explainability in AI. The topic of explainability in AI, machine learning (XAI) and information retrieval (IR) has been explored through various methods, yet few incorporate user feedback to adapt explanations. In this work, we present an XAI-driven extension to the classic relevance feedback model in IR, incorporating user feedback in the process of explaining the model behavior to the user. Our proposed model, RelEx, introduces XAI-specific elements, including key phrase vectors, text summaries, and contextual phrases combined with a neural ranker. RelEx interactively gathers user feedback, adapting search results based on the modified query and contextual vectors. We further introduce a novel additive similarity scheme that combines document similarity with key-phrase overlap. Retrieval performance is empirically evaluated on multiple benchmark datasets. In the absence of ground truth explanations, we assess explainability and assessability via user studies, where RelEx exhibit promising results.
Towards Best Practices of Axiomatic Activation Patching in Information Retrieval
Authors: Gregory Polyakov, Catherine Chen, Carsten Eickhoff
Venue: SIGIR
Year: 2025

Mechanistic interpretability research, which aims to uncover the internal processes of machine learning models, has gained significant attention. One state-of-the-art technique, activation patching, has been applied to analyzing neural ranker behavior in relation to information retrieval (IR) axioms. To date, however, this remains a rapidly evolving topic in IR, with no established methodology for measuring results or constructing datasets to ensure pronounced, robust, and consistent patching effects. In this study, based on experimental results, we provide recommendations on measuring patching effects and designing diagnostic datasets for investigating term frequency. We identify the rareness and informativeness of injected terms as a key factor influencing the magnitude of patching effects. Additionally, we find that low score differences between baseline and perturbed documents introduce significant noise, which can be mitigated by filtering or applying penalty scores to the metric. More generally, we provide practical recommendations for the reliable application of activation patching in IR, advancing future interpretability research of neural ranking models. Our code is available at https://github.com/polgrisha/best-practices-ir-patching.
An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc
Authors: Aldo Porco, Dhruv Mehra, Igor Malioutov, Karthik Radhakrishnan, Moniba Keymanesh, Daniel Preotiuc-Pietro, Sean MacAvaney, Pengxiang Cheng
Venue: SIGIR
Year: 2025

Learned Sparse Retrieval (LSR) models encode text as weighted term vectors, which need to be sparse to leverage inverted index structures during retrieval. SPLADE, the most popular LSR model, uses FLOPS regularization to encourage vector sparsity during training. However, FLOPS regularization does not ensure sparsity among terms-only within a given query or document. Terms with very high Document Frequencies (DFs) substantially increase latency in production retrieval engines, such as Apache Solr, due to their lengthy posting lists. To address the issue of high DFs, we present a new variant of FLOPS regularization: DF-FLOPS. This new regularization technique penalizes the usage of high-DF terms, thereby shortening posting lists and reducing retrieval latency. Unlike other inference-time sparsification methods, such as stopword removal, DF-FLOPS regularization allows for the selective inclusion of high-frequency terms in cases where the terms are truly salient. We find that DF-FLOPS successfully reduces the prevalence of high-DF terms and lowers retrieval latency (around 10x faster) in a production-grade engine while maintaining effectiveness both in-domain (only a 2.2-point drop in MRR@10) and cross-domain (improved performance in 12 out of 13 tasks on which we tested). With retrieval latencies on par with BM25, this work provides an important step towards making LSR practical for deployment in production-grade search engines.
Large Scale Deployment of BERT Based Cross Encoder Model for Re-Ranking in Walmart Search Engine
Authors: Ajit Puthenputhussery, Changsung Kang, Alessandro Magnani, Tian Zhang, Hongwei Shang, Nitin Yadav, Prijith Chandran, Bhavin Madhani, Yuan-Tai Fu, He Wang, Zbigniew Gasiorek, Salvatore Tornatore, Srikanth Dasaka, Vivek Agrawal, Michael Bowersox, Cun Mu, Ciya Liao
Venue: SIGIR
Year: 2025

Re-ranking plays a crucial role in product search by reassessing products from the primary retrieval system based on specific engagement and relevance criteria. While transformer-based models like the cross encoder have advanced the relevance of ranking models in recent years, a significant challenge arises from the high latency cost associated with running a cross encoder model at runtime. This challenge becomes more pronounced in the long-tail segment, where conventional techniques like caching prove ineffective. To tackle these issues, our paper introduces a scalable framework featuring a BERT-based cross encoder model for re-ranking, deployed in the Walmart search engine. We employ strategies such as intermediate representations, operator fusion, and vectorization to improve the inference latency of the cross encoder model. Furthermore, we provide a detailed discussion on the runtime implementation, highlighting key learnings and practical tricks that ensured minimal impact on response latency during production. Finally, we present the results of online experiments, including manual evaluation and interleaving test conducted on real-world e-commerce search traffic.
Towards Accurate Social User Geolocation: Mean Shift, Incremental Learning and Graph Convolutional Networks
Authors: Yaqiong Qiao, Aobo Jiao, Xiangyang Luo, Chenliang Li, Jiangtao Ma, Chenkai Guo
Venue: SIGIR
Year: 2025

The geolocation of social users is crucial for understanding user behavior, optimizing advertisement placement, and enhancing public safety. However, existing methods tend to show some deficiencies when handling sparse datasets and may not fully capture the natural clustering characteristics of user locations, thereby resulting in inadequate geolocation accuracy. This paper proposes a novel social user geolocation method (MILGCN) that innovatively integrates Mean Shift Clustering, Incremental Learning, and Graph Convolutional Networks. Specifically, Mean Shift performs fine-grained clustering of user locations based on density peak characteristics, ensuring that geographically close users are grouped into the same cluster. Introducing an incremental learning mechanism into graph convolutional networks enables MILGCN to have progressive learning ability. As a result, the problem of incomplete feature extraction from sparse data is alleviated, resulting in more comprehensive user features and improved geolocation accuracy. Extensive experiments proved that the proposed method significantly outperforms the state-of-the-art baselines on the real Twitter datasets, demonstrating a substantial improvement in geolocation performance.
Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
Authors: Jingfen Qiao, Jia-Huei Ju, Xinyu Ma, Evangelos Kanoulas, Andrew Yates
Venue: SIGIR
Year: 2025

Visual Document Retrieval (VDR) is an emerging research area that focuses on encoding and retrieving document images directly, bypassing the dependence on Optical Character Recognition (OCR) for document search. A recent advance in VDR was introduced by ColPali, which significantly improved retrieval effectiveness through a late interaction mechanism. ColPali's approach demonstrated substantial performance gains over existing baselines that do not use late interaction on an established benchmark. In this study, we investigate the reproducibility and replicability of VDR methods with and without late interaction mechanisms by systematically evaluating their performance across multiple pre-trained vision-language models. Our findings confirm that late interaction yields considerable improvements in retrieval effectiveness; however, it also introduces computational inefficiencies during inference. Additionally, we examine the adaptability of VDR models to textual inputs and assess their robustness across text-intensive datasets within the proposed benchmark, particularly when scaling the indexing mechanism. Furthermore, our research investigates the specific contributions of late interaction by looking into query-patch matching in the context of visual document retrieval. We find that although query tokens cannot explicitly match image patches as in the text retrieval scenario, they tend to match the patch contains visually similar tokens or their surrounding patches.
Question-Answering Dense Video Events
Authors: Hangyu Qin, Junbin Xiao, Angela Yao
Venue: SIGIR
Year: 2025

This paper presents question-answering on dense video events, a novel task that answers and grounds dense-event questions in long videos, thus challenging MLLMs to faithfully comprehend and reason about multiple events over extended periods of time. To facilitate the study, we construct DeVE-QA -- a dataset featuring 78K questions about 26K events on 10.6K long videos. Our benchmarking shows that state-of-the-art MLLMs struggle on DeVE-QA. For improvement, we propose DeVi, a novel training-free MLLM approach that highlights a hierarchical captioning module, a temporal event memory module, and a self-consistency checking module to respectively detect, contextualize and memorize, and ground dense-events in long videos for question answering. Extensive experiments show that DeVi is superior at answering dense-event questions and grounding relevant video moments. Compared with existing MLLMs, it achieves a notable increase of 4.8% and 2.1% for G(round)QA accuracy on DeVE-QA and NExT-GQA, respectively. Data and code are available at https://github.com/QHUni/DeVE-QA.
Interpretable Knowledge Tracing with Difficulty-Aware Attention and Selective State Space Model
Authors: Yang Qin, Xinning Zhu, Xiaosheng Tang, Chunhong Zhang, Kunbao Wu, Fengjie Chang, Jianzhou Diao, Zheng Hu
Venue: SIGIR
Year: 2025

Knowledge Tracing (KT) aims to model students' knowledge states based on their historical learning sequence, playing a critical role in online education platforms. As the performance of sequence-based KT methods continues to improve, their increasing model complexity and lack of transparency have become significant limitations. In contrast, educational theory-driven KT methods incorporate educationally meaningful features (such as question difficulty or time spent on questions) to enhance interpretability and performance. However, these models typically adopt simpler structures to reduce complexity and avoid overfitting, which limits their ability to effectively capture the sequential characteristics of learning compared to sequence-based methods. To address these limitations, this paper aims to integrate the strengths of both types of methods by proposing an Interpretable KT approach with Difficulty-Aware Attention and Selective State Space Model (ASIKT). Specifically, leveraging educational context, we design a difficulty-enhanced attention mechanism to model students' knowledge retrieval process at a fine-grained level without introducing additional parameters. Moreover, we employ selective state space model to capture the dynamic evolution of students' knowledge states based on their historical performance, ensuring both efficiency and simplicity in the model. Experimental results on four real-world datasets demonstrate that ASIKT outperforms state-of-the-art KT methods, achieving superior predictive performance. Additionally, ASIKT provides explanations for its predictions from multiple perspectives, showcasing strong interpretability. The code can be found at https://github.com/mrsser/ASIKT.
Intent-aware Diffusion with Contrastive Learning for Sequential Recommendation
Authors: Yuanpeng Qu, Hajime Nobuhara
Venue: SIGIR
Year: 2025

Contrastive learning has proven effective in training sequential recommendation models by incorporating self-supervised signals from augmented views. Most existing methods generate multiple views from the same interaction sequence through stochastic data augmentation, aiming to align their representations in the embedding space. However, users typically have specific intents when purchasing items (e.g., buying clothes as gifts or cosmetics for beauty). Random data augmentation used in existing methods may introduce noise, disrupting the latent intent information implicit in the original interaction sequence. Moreover, using noisy augmented sequences in contrastive learning may mislead the model to focus on irrelevant features, distorting the embedding space and failing to capture users' true behavior patterns and intents. To address these issues, we propose Intent-aware Diffusion with contrastive learning for sequential Recommendation (InDiRec). The core idea is to generate item sequences aligned with users' purchasing intents, thus providing more reliable augmented views for contrastive learning. Specifically, InDiRec first performs intent clustering on sequence representations using K-means to build intent-guided signals. Next, it retrieves the intent representation of the target interaction sequence to guide a conditional diffusion model, generating positive views that share the same underlying intent. Finally, contrastive learning is applied to maximize representation consistency between these intent-aligned views and the original sequence. Extensive experiments on five public datasets demonstrate that InDiRec achieves superior performance compared to existing baselines, learning more robust representations even under noisy and sparse data conditions.