OpenRaiser/Intern-Atlas huggingface dataset:
RankingSHAP - Faithful Listwise Feature Attribution Explanations for Ranking Models
Authors: Maria Heuss, Maarten de Rijke, Avishek Anand
Venue: SIGIR
Year: 2025

While SHAP (SHapley Additive exPlanations) and other feature attribution methods are commonly employed to explain model predictions, their application within information retrieval (IR), particularly for complex outputs such as ranked lists, remains limited. Existing attribution methods typically provide pointwise explanations, focusing on why a single document received a high-ranking score, rather than considering the relationships between documents in a ranked list. We present three key contributions to address this gap. First, we rigorously define listwise feature attribution for ranking models. Secondly, we introduce RankingSHAP, extending the popular SHAP framework to accommodate listwise ranking attribution, addressing a significant methodological gap in the field. Third, we propose two novel evaluation paradigms for assessing the faithfulness of attributions in learning-to-rank models, measuring the correctness and completeness of the explanation with respect to different aspects. Through experiments on standard learning-to-rank datasets, we demonstrate RankingSHAP's practical application while identifying the constraints of selection-based explanations. We further employ a simulated study with an interpretable model to showcase how listwise ranking attributions can be used to examine model decisions and conduct a qualitative evaluation of explanations. Due to the contrastive nature of the ranking task, our understanding of ranking model decisions can substantially benefit from feature attribution explanations like RankingSHAP.
A Generalised and Adaptable Reinforcement Learning Stopping Method
Authors: Reem Bin Hezam, Mark Stevenson
Venue: SIGIR
Year: 2025

This paper presents a Technology Assisted Review (TAR) stopping approach based on Reinforcement Learning (RL). Previous such approaches offered limited control over stopping behaviour, such as fixing the target recall and tradeoff between preferring to maximise recall or cost. These limitations are overcome by introducing a novel RL environment, GRLStop, that allows a single model to be applied to multiple target recalls, balances the recall/cost tradeoff and integrates a classifier. Experiments were carried out on six benchmark datasets (CLEF e-Health datasets 2017-9, TREC Total Recall, TREC Legal and Reuters RCV1) at multiple target recall levels. Results showed that the proposed approach to be effective compared to multiple baselines in addition to offering greater flexibility.
Score-Fitted Indexes and Constant Length Indexes for Information Retrieval
Authors: Djoerd Hiemstra
Venue: SIGIR
Year: 2025

We present two novel inverted index approaches and corresponding query processing strategies for information retrieval: 1) score-fitted indexes and 2) constant length indexes. These indexes do not store document lengths and document priors, but nevertheless support popular rankers like BM25 and language models by approximating their results. We answer the question: What is the effect of score-fitted indexes and constant length indexes, and the combination of both approaches, on the search quality? We show on three diverse datasets that the two indexes perform on par with approaches that use a standard inverted index in almost all cases. Our work suggests that it is possible to develop search engines that are more efficient than engines that store document lengths and/or document priors, such as Lucene and Terrier.
Defining & Optimizing Quality of LinkedIn's Content Search
Authors: Ali Hooshmand, Rupesh Gupta
Venue: SIGIR
Year: 2025

Most search engines optimize for quality metrics in addition to engagement metrics. Quality metrics require an explicit definition of the relevance of a document for a query. Defining relevance is a non-trivial problem. Simple definitions such as on-topicness of a document to a query may not adequately capture all aspects of quality, such as document originality, depth and recency. These aspects can have an impact on the value provided by a search engine to searchers, which in-turn drives engagement metrics in the long-term. So, a more thoughtful definition can be very beneficial. However, a complex definition necessitates the need for a search system that has a deep understanding of language. In this paper we present our definition of relevance for content search, and the design of our content search system that enables optimization of Precision based on that definition.
Optimize Visual Shopping Journey with Embedding-based Retrieval in Pinterest Closeup
Authors: Junpeng Hou, Wei-Ting Lin, Arkin Dharawat, Jiaxing Qu, Qi Wang, Sai Xiao, Xianxing Zhang, Weiran Li
Venue: SIGIR
Year: 2025

Pinterest is the visual discovery platform where people find inspiration, curate ideas, and shop products for all life's moments. An intentful journey can start when Pinners (users) click on Pins and arrive at the Closeup surface, where they can continue to explore or refine their intent by browsing related content powered by our visual search and related recommendation engines. Product Pins, or contents that are linked to merchants and are buyable in general, are critical to realizable fulfillment in Pinners' exploratory journey. This paper focuses on optimizing embedding-based retrieval (EBR) to retrieve relevant and personalized product Pins to drive actionable engagement. In contrast to conventional EBR systems, we introduce complementary Shopping Priority Corpora that are prepared by probability models from a dynamic inventory with billions of candidates and highly skewed distributions. This novel design enabled us to significantly improve the retrieval efficiency, scale up the systems, and meet different business requirements. On top of that, we build retrieval models that infuse multimodal information with multi-task contrastive learning to balance relevance and engagement. We evaluate the EBR system on random off-policy traffic with thorough baseline comparisons and rigorous online A/B experiments. This work leads to significant metric gains in our production systems and provides practical lessons on improving early retrieval for multiple business objectives at large scales.
FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval
Authors: Bohan Hou, Haoqiang Lin, Xuemeng Song, Haokun Wen, Meng Liu, Yupeng Hu, Xiangyu Zhao
Venue: SIGIR
Year: 2025

Due to their strong generalizable multimodal processing and reasoning capabilities, Multimodal Large Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs' retrieval performance, particularly for complex tasks such as long-text-to-image retrieval, visual dialog retrieval, and composed image retrieval (CIR). Therefore, in this work, we propose an automated fine-grained multimodal quintuple dataset construction pipeline and a novel two-stage fine-grained multimodal fine-tuning strategy. The dataset generation pipeline produces a comprehensive CIR dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context reasoning-oriented fine-tuning and (2) fine-grained retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target alignment capabilities, thereby improving retrieval performance. Extensive experiments across five datasets encompassing diverse and complex image retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot retrieval settings, even with a more lightweight MLLM backbone compared to those methods.
Adaptive Orchestration of Modular Generative Information Access Systems
Authors: Mohanna Hoveyda, Harrie Oosterhuis, Arjen P. de Vries, Maarten de Rijke, Faegheh Hasibi
Venue: SIGIR
Year: 2025

Advancements in large language models (LLMs) have driven the emergence of complex new systems to provide access to information, that we will collectively refer to as modular generative information access (GenIA) systems. They integrate a broad and evolving range of specialized components, including LLMs, retrieval models, and a heterogeneous set of sources and tools. While modularity offers flexibility, it also raises critical challenges: How can we systematically characterize the space of possible modules and their interactions? How can we automate and optimize interactions among these heterogeneous components? And, how do we enable this modular system to dynamically adapt to varying user query requirements and evolving module capabilities? In this perspective paper, we argue that the architecture of future modular generative information access systems will not just assemble powerful components, but enable a self-organizing system through real-time adaptive orchestration - where components' interactions are dynamically configured for each user input, maximizing information relevance while minimizing computational overhead. We give provisional answers to the questions raised above with a roadmap that depicts the key principles and methods for designing such an adaptive modular system. We identify pressing challenges, and propose avenues for addressing them in the years ahead. This perspective urges the IR community to rethink modular system designs for developing adaptive, self-optimizing, and future-ready architectures that evolve alongside their rapidly advancing underlying technologies.
FairWork: A Generic Framework For Evaluating Fairness In LLM-Based Job Recommender System
Authors: Yuhan Hu, Ziyu Lyu, Lu Bai, Lixin Cui
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) have revolutionized recommender systems by offering highly personalized and context-aware suggestions. However, their inherent biases pose significant challenges in sensitive scenarios like job recommendation, potentially compromising fairness and resulting in harmful effects on both users and platforms. While previous studies have explored fairness issues in LLM-based job recommendations, they often focus on limited dimensions. We introduce FairWork, a comprehensive fairness evaluation framework to examine LLM-based recommender system from both the user's and recruiter's perspectives, employing fairness metrics to assess how sensitive user attributes influence job recommendations. The system allows stakeholders such as recruitment platforms and job seekers to upload personalized profiles and job descriptions for fairness analysis. By integrating specific job requirements and user-driven data inputs, FairWork captures the relationship between candidate qualifications and job demands. This framework provides a robust foundation for evaluating fairness in LLM-based job recommender systems and supports future research on bias mitigation strategies. The demo is available at https://github.com/chenzhouli/FairWork.
CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs
Authors: Yuntong Hu, Zhihan Lei, Zhongjie Dai, Allen Zhang, Abhinav Angirekula, Zheng Zhang, Liang Zhao
Venue: SIGIR
Year: 2025

Research question answering requires accurate retrieval and contextual understanding of scientific literature. However, current Retrieval-Augmented Generation (RAG) methods often struggle to balance complex document relationships with precise information retrieval. In this paper, we introduce Contextualized Graph Retrieval-Augmented Generation (CG-RAG), a novel framework that integrates sparse and dense retrieval signals within graph structures to enhance retrieval efficiency and subsequently improve generation quality for research question answering. First, we propose a contextual graph representation for citation graphs, effectively capturing both explicit and implicit connections within and across documents. Next, we introduce Lexical-Semantic Graph Retrieval (LeSeGR), which seamlessly integrates sparse and dense retrieval signals with graph encoding. It bridges the gap between lexical precision and semantic understanding in citation graph retrieval, demonstrating generalizability to existing graph retrieval and hybrid retrieval methods. Finally, we present a context-aware generation strategy that utilizes the retrieved graph-structured information to generate precise and contextually enriched responses using large language models (LLMs). Extensive experiments on research question answering benchmarks across multiple domains demonstrate that our CG-RAG framework significantly outperforms RAG methods combined with various state-of-the-art retrieval approaches, delivering superior retrieval accuracy and generation quality.
Understanding Large Language Model Performance in Software Engineering: A Large-scale Question Answering Benchmark
Authors: Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, Cuiyun Gao
Venue: SIGIR
Year: 2025

In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering capabilities in the field of software engineering. CodeRepoQA encompasses five programming languages and covers a wide range of scenarios, enabling comprehensive evaluation of language models.To construct this dataset, we crawl data from 30 well-known repositories in GitHub, the largest platform for hosting and collaborating on code, and carefully filter the raw data.In total, CodeRepoQA is a multi-turn question-answering benchmark with 585,687 entries. It covers a diverse array of software engineering scenarios, with an average of 6.62 dialogue turns per entry. We evaluate ten popular large language models on our dataset and provide in-depth analysis. We find that LLMs still have limitations in question-answering capabilities in the field of software engineering, and medium-length contexts are more conducive to their performance. The entire benchmark and details are publicly available at https://github.com/kinesiatricssxilm14/CodeRepoQA.
HyperG: Hypergraph-Enhanced LLMs for Structured Knowledge
Authors: Sirui Huang, Hanqian Li, Yanggan Gu, Xuming Hu, Qing Li, Guandong Xu
Venue: SIGIR
Year: 2025

Given that substantial amounts of domain-specific knowledge are stored in structured formats, such as web data organized through HTML, Large Language Models (LLMs) are expected to fully comprehend this structured information to broaden their applications in various real-world downstream tasks. Current approaches for applying LLMs to structured data fall into two main categories: serialization-based and operation-based methods. Both approaches, whether relying on serialization or using SQL-like operations as an intermediary, encounter difficulties in fully capturing structural relationships and effectively handling sparse data. To address these unique characteristics of structured data, we propose HyperG, a hypergraph-based generation framework aimed at enhancing LLMs' ability to process structured knowledge. Specifically, HyperG first augment sparse data with contextual information, leveraging the generative power of LLMs, and incorporate a prompt-attentive hypergraph learning (PHL) network to encode both the augmented information and the intricate structural relationships within the data. To validate the effectiveness and generalization of HyperG, we conduct extensive experiments across two different downstream tasks requiring structured knowledge. Our code is publicly available at: https://github.com/s1ruihuang/HyperG.
Training-free Periodic Interest Augmentation in Incremental Recommendation
Authors: Heyuan Huang, Xingyu Lou, Changwang Zhang, Chaochao Chen, Kuiyao Dong, Feng Lu, Han Lei, Yihao Wang, Wangchunshu Zhou, Jun Wang
Venue: SIGIR
Year: 2025

Industrial recommender systems usually train models incrementally to grasp recent interests of users. However, a fundamental issue of these incremental updated models is their tendency to overfit current data while neglecting past information. Specifically, we have observed that the data distribution of real systems exhibits periodic drifts, leading to periodic fluctuations of prediction bias. To alleviate the above bias fluctuations while minimizing the loss of recent interests, we propose TPIA, a Training-free approach for Periodic Interest Augmentation in incremental recommendation. Specifically, after the latest model is trained, we first calculate the importance score of each model in the previous period. Then, we merge these models based on the importance scores. To minimize information loss due to interference of parameters during model merging, we further develop a method for trimming redundant and abnormal parameters. Offline experiments on both public and private datasets demonstrate the effectiveness of TPIA. It has also been deployed on a large-scale industrial recommender system, and has shown a notable 1.61% increase in CVR and a 1.97% increase in CPM, along with enhanced stability in prediction bias.
Predicting RAG Performance for Text Completion
Authors: Oz Huly, David Carmel, Oren Kurland
Venue: SIGIR
Year: 2025

We address the challenge of predicting the performance of using retrieval augmented generation (RAG) in large language models (LLMs) for the task of text completion; specifically, we predict the perplexity gain attained by applying RAG. We present novel supervised post-retrieval prediction methods that utilize the specific characteristics of the text completion setting. Our predictors substantially outperform a wide variety of prediction methods originally proposed for ad hoc document retrieval. We then show that integrating our post-retrieval predictors with recently proposed post-generation predictors - i.e., those analyzing the next-token distribution - is of much merit: the resultant prediction quality is statistically significantly better than that of using the post-generation predictors alone. Finally, we show that our post-retrieval predictors are as effective as post-generation predictors for selective application of RAG. This finding is of utmost importance in terms of efficiency of selective RAG.
MANILA25: SIGIR 2025 Workshop on Information Retrieval for Climate Impact
Authors: Bart van den Hurk, Maarten de Rijke, Flora Salim
Venue: SIGIR
Year: 2025

The MANILA25 workshop is aimed at collaborative agenda setting around the general area of information retrieval for climate impact in general and around adaptation tracking in particular. To this end, the workshop starts by creating a shared understanding of the problem space through invited talks around information needs in climate impact, search and analysis of climate impact literature, adaptation tracking, and resources. It then brings in different perspectives on the four topics through a number of brief ''flash'' presentations by participants in the workshop. Then, the participants will work to co-develop a research agenda in a small-scale, highly interactive setting.
Evaluating the Impact of Automated Labeling on Retrieval Instability in Neural IR
Authors: William A. Ingram
Venue: SIGIR
Year: 2025

Effective information retrieval (IR) depends on accurate relevance classification. But when the criteria are subjective or underspecified, small variations in classification can cause consequential shifts in retrieval results. The potential for such variability becomes critical for institutions when they use IR for research assessment. Retrieval instability can lead to relevant literature being overlooked, hindering a comprehensive understanding of the research landscape, and potentially undermining the validity of subsequent analyses and decisions.
Doctron: A Web-based Collaborative Annotation Tool for Ground Truth Creation in IR
Authors: Ornella Irrera, Stefano Marchesin, Farzad Shami, Gianmaria Silvello
Venue: SIGIR
Year: 2025

In Information Retrieval (IR), ground truth creation is a crucial yet resource-intensive task that relies on human experts to build test collections - essential for training and evaluating retrieval models. Large-scale evaluation campaigns, such as TREC and CLEF, demand significant human effort to produce reliable, high-quality annotations. To ease this process, tailored annotation tools are pivotal to supporting assessors and streamlining their workload. To this end, we introduce Doctron, a web-based, dockerized annotation tool designed to streamline ground truth creation for IR tasks. Doctron enables the annotation of both textual documents and images. It supports annotating textual passages, identifying relationships, tagging and linking entities, evaluating document relevance to a topic with graded labels, and performing object detection. It offers a collaborative environment where teams can work with defined user roles and permissions. The integration of Inter Annotator Agreement (IAA) measures helps to identify inconsistencies between annotators, thereby ensuring the reliability and high quality of the annotated ground truth data.
Counterfactual Model Selection in Contextual Bandits
Authors: Shion Ishikawa, Young-joo Chung, Yun-Ching Liu, Yu Hirate
Venue: SIGIR
Year: 2025

Contextual bandit algorithms are crucial in various decision-making applications, such as personalized content recommendation, online advertising, and e-commerce banner placement. Despite their successful applications in various domains, contextual bandit algorithms still face significant challenges with exploration efficiency compared to non-contextual bandit algorithms due to exploration in feature spaces. To overcome this issue, model selection policies such as MetaEXP and MetaCORRAL have been proposed to interactively explore base policies. In this paper, we introduce a novel counterfactual approach to address the model selection problem in contextual bandits. Unlike previous methods, our approach leverages unbiased Off-Policy Evaluation (OPE) to dynamically select base policies, making it more robust to model misspecification. We present two new algorithms, MetaEXP-OPE and MetaGreedy-OPE, which utilize OPE for model selection policy. We also provide theoretical analysis on regret bounds and evaluate the impact of different OPE estimators. We evaluated our model on synthetic data and a semi-synthetic simulator using a real-world dataset, and the results show that MetaEXP-OPE and MetaGreedy-OPE significantly outperform existing policies, including MetaEXP and MetaCORRAL.
MELON: Learning Multi-Aspect Modality Preferences for Accurate Multimedia Recommendation
Authors: Dongho Jeong, Taeri Kim, Donghyeon Cho, Sang-Wook Kim
Venue: SIGIR
Year: 2025

Existing multimedia recommender systems have made the best efforts to predict user preferences for items by utilizing behavioral similarities between users and the modality features of items a user has interacted with. However, we identify two key limitations in existing methods regarding preferences for modality features: (L1) although preferences for modality features is an important aspect of users' preferences, existing methods only leverage neighbors with similar interactions and do not consider the neighbors who may have similar preferences for modality features while having different interactions; (L2) although modality features of a user and an item may have a complex geometric relationship in the latent space, existing methods overlook and face challenges in precisely capturing this relationship. To address these two limitations, we propose a novel multimedia recommendation framework, named MELON, which is based on two core ideas: (Idea 1) Modality-cEntered embedding extraction; (Idea 2) reLatiOnship-ceNtered embedding extraction. We validate the effectiveness and validity of MELON through extensive experiments with four real-world datasets, showing 10.51% higher accuracy compared to the best competitor in terms of recall@10. The code and dataset of MELON is available at https://github.com/Bigdasgit/MELON.