OpenRaiser/Intern-Atlas huggingface dataset:
My System Is As Effective As Yours: Reproducibility, Sustainability, and More
Authors: Tetsuya Sakai
Venue: SIGIR
Year: 2025

When a paired or two-sample t-test shows that a difference is not statistically significant (i.e., the null hypothesis H is "accepted"), all we obtain as a "conclusion" is: " we cannot conclude from the data whether the difference is real or not. " In general, statistical significance tests with a significance level of α give us a proper conclusion with 100(1-α)% confidence only if the H is rejected. Hence, for example, in a reproducibility study in IR, traditional significance tests like the t-test are not adequate if one wishes to claim that the reproduced run is as good as the target run. In this paper, we show that the equivalence test, which has been gaining popularity in the medical domain for comparing new drugs with standard drugs, is applicable to IR research in situations such as above. We also discuss the less ambitious noninferiority test, which can be used, for example, when an IR researcher wants to claim that their system performs no worse than a state-of-the-art system that is computationally much more expensive and therefore earth-unfriendly. The equivalence and noninferiority tests require researchers to pre-define an indifference zone, which defines how much difference is "practically negligible"; hence we also discuss one possible approach to establishing an indifference zone for a given IR effectiveness measure. In addition, we provide recommendations for IR researchers who wish to conduct equivalence/noninferiority tests involving more than two systems. It is hoped that equivalence and noninferiority tests will become standard practices in IR, so that we can avoid making claims based on lack of statistical significance.
BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
Authors: Andrey Sakhovskiy, Elena Tutubalina
Venue: SIGIR
Year: 2025

In recent years, there has been substantial progress in using pretrained Language Models (LMs) on a range of tasks aimed at improving the understanding of biomedical texts. Nonetheless, existing biomedical LLMs show limited comprehension of complex, domain-specific concept structures and the factual information encoded in biomedical Knowledge Graphs (KGs). In this work, we propose BALI (Biomedical Knowledge Graph and Language Model Ali gnment), a novel joint LM and KG pre-training method that augments an LM with external knowledge by the simultaneous learning of a dedicated KG encoder and aligning the representations of both the LM and the graph. For a given textual sequence, we link biomedical concept mentions to the Unified Medical Language System (UMLS) KG and utilize local KG subgraphs as cross-modal positive samples for these mentions. Our empirical findings indicate that implementing our method on several leading biomedical LMs, such as PubMedBERT and BioLinkBERT, improves their performance on a range of language understanding tasks and the quality of entity representations, even with minimal pre-training on a small alignment dataset sourced from PubMed scientific abstracts.
MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion
Authors: Saron Samuel, Dan DeGenaro, Jimena Guallar-Blasco, Kate Sanders, Seun Eisape, Arun V. Reddy, Alexander Martin, Andrew Yates, Eugene Yang, Cameron Carpenter, David Etter, Efsun Selin Kayi, Matthew Wiesner, Kenton Murray, Reno Kriz
Venue: SIGIR
Year: 2025

Videos inherently contain multiple modalities, including visual events, text overlays, sounds, and speech, all of which are important for retrieval. However, state-of-the-art multimodal language models like VAST and LanguageBind are built on vision-language models (VLMs), and thus overly prioritize visual signals. Retrieval benchmarks further reinforce this bias by focusing on visual queries and neglecting other modalities. We create a search system MMMORRF that extracts text and features from both visual and audio modalities and integrates them with a novel modality-aware weighted reciprocal rank fusion. MMMORRF is both effective and efficient, demonstrating practicality in searching videos based on users' information needs instead of visual descriptive queries. We evaluate MMMORRF on MultiVENT 2.0 and TVR, two multimodal benchmarks designed for more targeted information needs, and find that it improves nDCG@20 by 81% over leading multimodal encoders and 37% over single-modality retrieval.
Post-event Modeling via Causal Optimal Transport for CTR Prediction
Authors: Yizhou Sang, Congcong Liu, Yuying Chen, Zhiwei Fang, Xue Jiang, Changping Peng, Zhangang Lin, Ching Law, Jingping Shao
Venue: SIGIR
Year: 2025

Accurate click-through rate (CTR) prediction is critical for online advertising, relying on regular features like browsing history and demographics and post-event features such as exposed position and detailed page behaviors. However, post-event features, unavailable during inference, often face training-inference inconsistency and low coverage issues, especially post-click features like dwell time that are available only for clicked items. To address these challenges, we propose Causal Optimal Transport (COT), a novel framework that (1) generates pseudo post-click features via semi-supervised pseudo-labeling (2) causally generates accurate feature distributions using a Causal Distribution Shaper (CDS), and (3) refines generated features through optimal transport to minimize distributional divergence, facilitating further knowledge transfer. Experiments on real-world data confirm COT's superiority and practical efficacy in enhancing CTR prediction via improved user interest modeling and bias mitigation. Theoretical guarantees underpin the framework's robustness.
Rational Retrieval Acts: Leveraging Pragmatic Reasoning to Improve Sparse Retrieval
Authors: Arthur Satouf, Gabriel Ben Zenou, Benjamin Piwowarski, Habiboulaye Amadou Boubacar, Pablo Piantanida
Venue: SIGIR
Year: 2025

Current sparse neural information retrieval (IR) methods, and to a lesser extent more traditional models such as BM25, do not take into account the document collection and the complex interplay between different term weights when representing a single document. In this paper, we show how the Rational Speech Acts (RSA), a linguistics framework used to minimize the number of features to be communicated when identifying an object in a set, can be adapted to the IR case - and in particular to the high number of potential features (here, tokens). RSA dynamically modulates token-document interactions by considering the influence of other documents in the dataset, better contrasting document representations. Experiments show that incorporating RSA consistently improves multiple sparse retrieval models and achieves state-of-the-art performance on out-of-domain datasets from the BEIR benchmark.
Interpreting Multilingual and Document-Length Sensitive Relevance Computations in Neural Retrieval Models through Axiomatic Causal Interventions
Authors: Oliver Savolainen, Dur e Najaf Amjad, Roxana Petcu
Venue: SIGIR
Year: 2025

This reproducibility study analyzes and extends the paper ''Ax- iomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models,'' which investigates how neural retrieval models encode task-relevant properties such as term frequency. We reproduce key experiments from the original paper, confirming that information on query terms is captured in the model encoding. We extend this work by applying activa- tion patching to Spanish and Chinese datasets and by exploring whether document-length information is encoded in the model as well. Our results confirm that the designed activation patching method can isolate the behavior to specific components and to- kens in neural retrieval models. Moreover, our findings indicate that the location of term frequency generalizes across languages and that in later layers, the information for sequence-level tasks is represented in the CLS token. The results highlight the need for further research into interpretability in information retrieval and reproducibility in machine learning research. Our code is available at https://github.com/OliverSavolainen/axiomatic-ir-reproduce.
Robust Inverse Retrieval in Online Advertising with Contrastive Learning
Authors: Phaniram Sayapaneni, Konstantin Shmakov, Sunil Goda
Venue: SIGIR
Year: 2025

Sponsored product plays a key role at advertising business which attracts an increasing number of advertisers seeking better platforms to promote their products. Connecting advertiser's products (demand) to the customer interests (supply) becomes a crucial factor in forecasting advertising campaign success. In this work, we propose a general framework for learning the inverse product retrieval process, a more complex procedure compared to the conventional forward approach. The first set of challenges arise due to a significant level of noise in the retrieval data caused by users who deviate from the intended platform usage. To address this, we propose a contrastive learning approach that minimizes the effect of such noise in data, ensuring robustness against false positives and false negatives. In addition, we propose a calibration procedure for the inverse retrieval that handles dynamic size of a set of queries that may retrieve any particular product of interest. Our framework is universally applicable across various contexts of an advertising platform, including but not limited to search pages, item pages, and browsing pages. We showcase the efficacy of the proposed approach using Walmart's advertising data in ten product domains.
Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)
Authors: Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer, Andreas Konstantin Kruff
Venue: SIGIR
Year: 2025

Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central IA research and evaluation questions, especially around interactive settings when real users are unavailable, or their participation is impossible due to ethical reasons. In addition, simulations in IA can help contribute to a better understanding of users, reduce complexity of evaluation experiments, and improve reproducibility. Building on recent developments in methods and toolkits, the second iteration of our Sim4IA workshop aims to again bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.
Psychological Aspects in Retrieval and Recommendation
Authors: Markus Schedl, Elisabeth Lex, Marko Tkalcic
Venue: SIGIR
Year: 2025

Psychological processes play a critical role in shaping users' interactions with information retrieval (IR) and recommender systems (RS). Therefore, understanding human cognition, decision-making, and emotions is vital to enable user-centric retrieval and recommendation systems. Vice versa, understanding whether these aspects are also present in the systems themselves (e.g., in training data, ranking models, or outputs), or even injecting them on purpose, can inform the development of psychology-inspired systems. The purpose of this tutorial is to provide its attendees with an introduction to psychological concepts that are important in the ecosystem of search, retrieval, and recommendation, in particular, cognitive architectures, cognitive effects and biases, as well as personality and affect. Leveraging corresponding models allows its audience to build or refine psychology-informed IR and RS technology. The interdisciplinary tutorial requires intermediate expertise in terms of IR and RS, while we do not assume knowledge in psychology.
WARP: An Efficient Engine for Multi-Vector Retrieval
Authors: Jan Luca Scheerer, Matei Zaharia, Christopher Potts, Gustavo Alonso, Omar Khattab
Venue: SIGIR
Year: 2025

Multi-vector retrieval methods such as ColBERT and its recent variant, the ConteXtualized Token Retriever (XTR), offer high accuracy but face efficiency challenges at scale. To address this, we present WARP, a retrieval engine that substantially improves the efficiency of retrievers trained with the XTR objective through three key innovations: (1) WARPSELECT for dynamic similarity imputation; (2) implicit decompression, avoiding costly vector reconstruction during retrieval; and (3) a two-stage reduction process for efficient score aggregation. Combined with highly-optimized C++ kernels, our system reduces end-to-end latency compared to XTR's reference implementation by 41x, and achieves a 3x speedup over the ColBERTv2/PLAID engine, while preserving retrieval quality. WARP also reduces index sizes by a factor of 2x-4x compared to XTR, enabling deployment on memory-constrained devices.
TITE: Token-Independent Text Encoder for Information Retrieval
Authors: Ferdinand Schlatt, Tim Hagen, Martin Potthast, Matthias Hagen
Venue: SIGIR
Year: 2025

Transformer-based retrieval approaches typically use the contextualized embedding of the first input token as a dense vector representation for queries and documents. The embeddings of all other tokens are also computed but then discarded, wasting resources. In this paper, we propose the Token-Independent Text Encoder (TITE) as a more efficient modification of the backbone encoder model. Using an attention-based pooling technique, TITE iteratively reduces the sequence length of hidden states layer by layer so that the final output is already a single sequence representation vector. Our empirical analyses on the TREC 2019 and 2020 Deep Learning tracks and the BEIR benchmark show that TITE is on par in terms of effectiveness compared to standard bi-encoder retrieval models while being up to 3.3 times faster at encoding queries and documents. Our code is available at: https://github.com/webis-de/SIGIR-25.
Permutation-Invariant Transformers for Attribute Embeddings in Information Retrieval
Authors: Ernest Kirubakaran Selvaraj, Akshay Shah, Shubham Agrawal, Riya Hedaoo, Samira Golsefid
Venue: SIGIR
Year: 2025

Generating robust embeddings for unordered sets of phrases is a challenging problem in NLP, particularly when embeddings must exhibit permutation invariance at the set level while preserving the inherent order of tokens within each phrase. This challenge is especially relevant in e-commerce, where product attributes such as size, color, and dimensions are often unstructured and vary across products. Embedding these unordered attribute sets is critical for query-product understanding, search ranking, and recommenda- tion systems. Existing set embedding methods have primarily focused on domains such as graphs, with limited applicability to NLP tasks. To address this gap, we propose a Phrase-Localized Attention Network (PLAN), a Transformer-based model that ensures intra-attribute order preservation while enabling permutation-invariant representations at the set level. Experimental results on the Amazon Shopping Queries dataset demonstrate that PLAN outperforms existing models, highlighting its effectiveness in search applications.
Bias-Aware Curriculum Sampling For Fair Ranking
Authors: Shirin Seyedsalehi, Hai Son Le, Morteza Zihayat, Ebrahim Bagheri
Venue: SIGIR
Year: 2025

Neural ranking models are widely used to retrieve and rank relevant documents. However, these models may inherit and amplify biases present in the training data, posing challenges for fairness and relevance in ranking outputs. In this paper, we propose a novel curriculum-based training approach that manages bias exposure throughout the training process. We design a bias-aware curriculum that stages the exposure of the model to biased samples during the training stages, allowing the model to establish a fair relevance baseline. We conduct extensive experiments across different LLMs and datasets to evaluate the effectiveness of our approach. Our results demonstrate that our proposed strategy outperforms other bias reduction methods in terms of both fairness and relevance, without sacrificing retrieval effectiveness.
Multimodal Search in Chemical Documents and Reactions
Authors: Ayush Kumar Shah, Abhisek Dey, Leo Luo, Bryan Amador, Patrick Philippy, Ming Zhong, Siru Ouyang, David Mark Friday, David Bianchi, Nick Jackson, Richard Zanibbi, Jiawei Han
Venue: SIGIR
Year: 2025

We present a multimodal search tool for retrieval of chemical reactions, molecular structures, and associated text from scientific literature. Queries may combine molecular diagrams, textual descriptions, and reaction data, allowing users to connect different chemical information representations. Indexing includes chemical diagram extraction and parsing, extraction of reaction data from text in tabular form, and cross-modal linking of diagrams with their mentions in text. We describe the system's architecture and retrieval features, along with expert assessments of the system. Our demo highlights the workflow and search components. Online demo: https://www.cs.rit.edu/~dprl/reactionminer-demo-landing
MultiConAD: A Unified Multilingual Conversational Dataset for Early Alzheimer's Detection
Authors: Arezo Shakeri, Mina Farmanbar, Krisztian Balog
Venue: SIGIR
Year: 2025

Dementia is a progressive cognitive syndrome with Alzheimer's disease (AD) as the leading cause. Conversation-based AD detection offers a cost-effective alternative to clinical methods, as language dysfunction is an early biomarker of AD. However, most prior research has framed AD detection as a binary classification problem, limiting the ability to identify Mild Cognitive Impairment (MCI)-a crucial stage for early intervention. Also, studies primarily rely on single-language datasets, mainly in English, restricting crosslanguage generalizability. To address this gap, we make three key contributions. First, we introduce a novel, multilingual dataset for AD detection by unifying 16 publicly available dementia-related conversational datasets. This corpus spans English, Spanish, Chinese, and Greek, and incorporates both audio and text data derived from a variety of cognitive assessment tasks. Second, we perform finer-grained classification, including MCI, and evaluate various classifiers using sparse and dense text representations. Third, we conduct experiments in monolingual and multilingual settings, finding that some languages benefit from multilingual training while others perform better independently. This study highlights the challenges in multilingual AD detection and enables future research on both language-specific approaches and techniques aimed at improving model generalization and robustness.
Fact Verification in Knowledge Graphs Using LLMs
Authors: Farzad Shami, Stefano Marchesin, Gianmaria Silvello
Venue: SIGIR
Year: 2025

Automated fact-checking systems often struggle with trustworthiness, as they lack transparency in their reasoning processes and fail to handle relationships in data. This work presents FactCheck, a fact verification system topped by a web platform that shows how Large Language Models (LLMs) can be collectively used to verify facts within Knowledge Graphs (KGs). While the underlying verification engine implements a system that combines Retrieval Augmented Generation (RAG) with an ensemble of LLMs to validate KG facts, the platform focuses on making the results of this complex process as transparent and accessible as possible. Users can explore how different models interpret the same evidence, compare their reasoning patterns, and understand the factors that lead to the final verification result. The platform supports technical users who want to analyze the model behavior and general users who need to verify whether the facts in the dataset are correct.
RankLLM: A Python Package for Reranking with LLMs
Authors: Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, Jimmy Lin
Venue: SIGIR
Year: 2025

The adoption of large language models (LLMs) as rerankers in multi-stage retrieval systems has gained significant traction in academia and industry. These models refine a candidate list of retrieved documents, often through carefully designed prompts, and are typically used in applications built on retrieval-augmented generation (RAG). This paper introduces RankLLM, an open-source Python package for reranking that is modular, highly configurable, and supports both proprietary and open-source LLMs in customized reranking workflows. To improve usability, RankLLM features optional integration with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. Additionally, RankLLM includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. This paper presents the architecture of RankLLM, along with a detailed step-by-step guide and sample code. We reproduce results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models. RankLLM integrates with common inference frameworks and a wide range of LLMs. This compatibility allows for quick reproduction of reported results, helping to speed up both research and real-world applications. The complete repository is available at rankllm.ai, and the package can be installed via PyPI.
A Worrying Reproducibility Study of Intent-Aware Recommendation Models
Authors: Faisal Shehzad, Maurizio Ferrari Dacrema, Dietmar Jannach
Venue: SIGIR
Year: 2025

Lately, we have observed a growing interest in intent-aware recommender systems (IARS). The promise of such systems is that they are capable of generating better recommendations by predicting and considering the underlying motivations and short-term goals of consumers. From a technical perspective, various sophisticated neural models were recently proposed in this emerging and promising area. In the broader context of complex neural recommendation models, a growing number of research works unfortunately indicates that (i) reproducing such works is often difficult and (ii) that the true benefits of such models may be limited in reality, e.g., because the reported improvements were obtained through comparisons with untuned or weak baselines. In this work, we investigate if recent research in IARS is similarly affected by such problems. Specifically, we tried to reproduce five contemporary IARS models that were published in top-level outlets, and we benchmarked them against a number of traditional non-neural recommendation models. In two of the cases, running the provided code with the optimal hyperparameters reported in the paper did not yield the results reported in the paper. Worryingly, we find that all examined IARS approaches are consistently outperformed by at least one traditional model. These findings point to sustained methodological issues and to a pressing need for more rigorous scholarly practices.