OpenRaiser/Intern-Atlas huggingface dataset:
Reason-to-Rank: Distilling Direct and Comparative Reasoning from Large Language Models for Document Reranking
Authors: Yuelyu Ji, Zhuochun Li, Rui Meng, Daqing He
Venue: SIGIR
Year: 2025

Reranking documents in information retrieval often relies on black-box models that improve effectiveness but lack explainability. We introduce Reason-to-Rank (R2R), a novel framework that separates direct relevance reasoning from comparison reasoning to provide both direct and comparitive explanations. We first prompt a large language model to produce comprehensive rationales and a ranking order; then we distill both the ranking decisions and textual explanations into a smaller, open-source student model. Our approach not only improves retrieval performance, as demonstrated in MSMARCO, BEIR, and BRIGHT, but also provides interpretable justifications for why one document outranks another. We report NDCG@5 (and NDCG@10) for direct comparisons with prior work, and show that the distilled student model achieves competitive results while significantly reducing computational overhead. By unifying direct and comparative reasoning in a single pipeline, R2R bridges the gap between transparency and effectiveness in modern reranking systems.
AgentIR: 2nd Workshop on Agent-based Information Retrieval
Authors: Pengyue Jia, Qingpeng Cai, Xiangyu Zhao, Ling Pan, Xin Xin, Jin Huang, Weinan Zhang, Li Zhao, Dawei Yin, Grace Hui Yang
Venue: SIGIR
Year: 2025

Information retrieval (IR) systems are essential in modern society, aiding users to efficiently locate relevant information through query expansion, document retrieval, ranking, and re-ranking. User feedback from ranked outputs forms a dynamic interaction loop with IR systems, which can be modeled as either one-time or sequential decision-making problems. Over the past decade, deep reinforcement learning (DRL) has emerged as a promising approach to decision-making, leveraging the high model capacity of deep learning for complex tasks. While significant research has explored the application of DRL to IR tasks, several fundamental challenges remain underexplored, including the underlying information theory in DRL settings, the limitations of reinforcement learning methods for industrial IR applications, and the simulation of DRL-based IR systems. Concurrently, the advent of large language models (LLMs) has introduced new opportunities for optimizing and simulating IR systems. Building on the success of the Agent-based IR Workshop at SIGIR 2024, we propose hosting the second Agent-based IR Workshop at SIGIR 2025. This workshop will continue to provide a platform for researchers and practitioners from academia and industry to present cutting-edge advances in DRL-based and LLM-based IR systems from an agent-based perspective. By building on the foundation laid in the first workshop, the 2025 edition aims to delve deeper into emerging research challenges, foster collaborations, and explore innovative applications. Through engaging discussions and insightful presentations, the workshop seeks to further expand the boundaries of IR research and solidify its role as a premier venue for advancing agent-based IR systems.
PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization
Authors: Yang Jiao, Xiaodong Wang, Kai Yang
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of applications, e.g., medical question-answering, mathematical sciences, and code generation. However, they also exhibit inherent limitations, such as outdated knowledge and susceptibility to hallucinations. Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm to address these issues, but it also introduces new vulnerabilities. Recent efforts have focused on the security of RAG-based LLMs, yet existing attack methods face three critical challenges: (1) their effectiveness declines sharply when only a limited number of poisoned texts can be injected into the knowledge database (2) they lack sufficient stealth, as the attacks are often detectable by anomaly detection systems, which compromises their effectiveness, and (3) they rely on heuristic approaches to generate poisoned texts, lacking formal optimization frameworks and theoretic guarantees, which limits their effectiveness and applicability. To address these issues, we propose coordinated Prompt-RAG attack (PR-attack), a novel optimization-driven attack that introduces a small number of poisoned texts into the knowledge database while embedding a backdoor trigger within the prompt. When activated, the trigger causes the LLM to generate pre-designed responses to targeted queries, while maintaining normal behavior in other contexts. This ensures both high effectiveness and stealth. We formulate the attack generation process as a bilevel optimization problem leveraging a principled optimization framework to develop optimal poisoned texts and triggers. Extensive experiments across diverse LLMs and datasets demonstrate the effectiveness of PR-Attack, achieving a high attack success rate even with a limited number of poisoned texts and significantly improved stealth compared to existing methods. These results highlight the potential risks posed by PR-Attack and emphasize the importance of securing RAG-based LLMs against such threats.
An Instruction-Response Perspective on Large Language Models in Information Retrieval Tasks
Authors: Hideo Joho, Joemon M. Jose
Venue: SIGIR
Year: 2025

The increasing use of retrieval-augmented applications, where large language models (LLMs) are instructed to generate queries, assess relevance, and synthesise responses, has introduced new challenges in Information Retrieval (IR). The lack of transparency in LLMs means that even subtle variations in instructions can significantly impact the quality, consistency, and reliability of their responses. To address this issue, we propose Instruction-Response Study, an experimental framework for systematically analysing how task instructions influence LLM-generated responses in IR tasks. This paper presents the core components of the framework and demonstrates its utility through four case studies, examining 1) the effect of IR tasks on query formulation, 2) the impact of topic information size on retrieval effectiveness, 3) the reproducibility of LLM-generated queries, and 4) the role of meta-instructions in diversifying instruction design. The findings highlight how the proposed framework enables controlled experimentation on instruction design and its effects, offering a foundation for optimising prompt engineering and enhancing retrieval-augmented applications.
Learning Universal User Representations Leveraging Cross-domain User Intent at Snapchat
Authors: Clark Mingxuan Ju, Leonardo Neves, Bhuvesh Kumar, Liam Collins, Tong Zhao, Yuwei Qiu, Qing Dou, Yang Zhou, Sohail Nizam, Rengim Aykan Ozturk, Yvette Liu, Sen Yang, Manish Malik, Neil Shah
Venue: SIGIR
Year: 2025

The development of powerful user representations is a key factor in the success of recommender systems (RecSys). Online platforms employ a range of RecSys techniques to personalize user experience across diverse in-app surfaces. User representations are often learned individually through user's historical interactions within each surface and user representations across different surfaces can be shared post-hoc as auxiliary features or additional retrieval sources. While effective, such schemes cannot directly encode collaborative filtering signals across different surfaces, hindering its capacity to discover complex relationships between user behaviors and preferences across the whole platform. To bridge this gap at Snapchat, we seek to conduct universal user modeling (UUM) across different in-app surfaces, learning general-purpose user representations which encode behaviors across surfaces. Instead of replacing domain-specific representations, UUM representations capture cross-domain trends, enriching existing representations with complementary information. This work discusses our efforts in developing initial UUM versions, practical challenges, technical choices and modeling and research directions with promising offline performance. Following successful A/B testing, UUM representations have been launched in production, powering multiple use cases and demonstrating their value. UUM embedding has been incorporated into (i) Long-form Video embedding-based retrieval, leading to 2.78% increase in Long-form Video Open Rate, (ii) Long-form Video L2 ranking, with 19.2% increase in Long-form Video View Time sum, (iii) Lens L2 ranking, leading to 1.76% increase in Lens play time, and (iv) Notification L2 ranking, with 0.87% increase in Notification Open Rate.
HAETAE: In-domain Table Pretraining with Header Anchoring
Authors: Woojun Jung, Susik Yoon
Venue: SIGIR
Year: 2025

Understanding structured table data with language models is crucial for various downstream tasks in information retrieval. However, transformer-based table embedding models struggle to consistently represent headers across varying entity contexts. This inconsistency undermines the generalizability of embeddings across in-domain tables that share universal semantics. To address this gap, we propose a novel pretraining method for in-domain tables, HAETAE, that explicitly separates header embeddings from contextual entity embeddings. Our method introduces a dedicated header encoder and learnable alignment mechanisms, built upon header-aware serialization. Experimental results demonstrate that HAETAE enhances generalization and stability in predicting headers and values of in-domain tables, achieving higher accuracy than baselines while showing the notable potential of knowledge transfer in cross-domain tables. The source code of HAETAE is available at https://github.com/woojoonjung/HAETAE.
When Less is Enough: Optimizations for Low-Cost Recommendation Systems
Authors: Md. Danish Kalim, Deepanshu Hardaha, Shivansh Joshi
Venue: SIGIR
Year: 2025

Social media platforms rely heavily on ad revenue, making cost-efficient recommendation systems essential for sustaining profitability while preserving user satisfaction, especially in regions with low Average Revenue Per User (ARPU). We present a comprehensive approach that combines model-side optimizations (e.g., fused operations, reducing system responsiveness) with system-side enhancements (e.g., improved load balancing, half precision) to lower infrastructure costs without significantly compromising user engagement. Some of the optimizations below were deployed in production to serve 180M users, these optimizations deliver 57% cloud cost savings with less than 0.4% drop in retention and increase net profitability. This work provides a scalable blueprint for cost-efficient recommendation systems and offers actionable insights for deploying such systems in resource-constrained environments.
OnSET: Ontology and Semantic Exploration Toolkit
Authors: Benedikt Kantz, Kevin Innerebner, Peter Waldert, Stefan Lengauer, Elisabeth Lex, Tobias Schreck
Venue: SIGIR
Year: 2025

Retrieval over knowledge graphs is typically performed using specialized, complex query languages such as SPARQL. We propose a novel system, Ontology and Semantic Exploration Toolkit (OnSET), that allows novice users to quickly build queries with visual user guidance provided by topic modeling and semantic search throughout the application. OnSET enables users without prior knowledge of the ontology or networked knowledge to start exploring topics of interest over knowledge graphs, including the retrieval and detailed exploration of prototypical sub-graphs and their instances. Existing systems either focus on direct graph exploration or do not foster further exploration of the result set. We, however, provide a node-based editor that can extend these missing properties of existing systems to support search over large ontologies with sub-graph instances. Furthermore, OnSET combines efficient and open platforms to deploy the system on commodity hardware.
Exploring the Role of Diversity in Example Selection for In-Context Learning
Authors: Janak Kapuriya, Manit Kaushik, Debasis Ganguly, Sumit Bhatia
Venue: SIGIR
Year: 2025

In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized examples relevant to a given input using sparse or dense embedding-based similarity functions. However, relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance. We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance. To achieve this, we leverage maximum marginal relevance (MMR) which balances topical similarity with inter-example diversity. Our experimental results demonstrate that diversifying the selected examples leads to consistent improvements in downstream performance across various context sizes and similarity functions. The implementation of our approach is made available at https://github.com/janak11111/Diverse-ICL.
A System for Triggering Sports Instant Answers on Search Engines
Authors: Ankith Karat, Atishay Tibrewal, Nishka Kotian, Manan Dang, Ravindra Valluri, Antony Ravi Teja Marineni, Sarthak Sahni, Rhea Sundaresan, Ankit Kumar, Aditya Mehndiratta, Sunil Shah, Arun D. Poondi, Chandra Bhushan, Subhasis Panigrahi, Manish Gupta
Venue: SIGIR
Year: 2025

Bing Sports serves Instant Answers to sports-related queries from hundreds of millions of users every day with a latency of <100ms. Answering sports-related factual questions is challenging because of the dynamic nature of the domain, contextual nature of queries, and a large variety of possible intents. In this paper, we discuss various blocks of our scalable multilingual sports answer triggering pipeline. The pipeline mainly comprises of two main stages: Query Understanding (QU) and Ranking. QU leverages blocks like domain classifier, named entity recognizer, intent classifier, and entity linker to detect queries for which a sports answer should get triggered and also accurately identify user intent and entities in queries. The ranking stage is driven using blocks like pre-web ranker, answer lookup, and post-web suppression and reranking for ranking various candidates and choose the best one to be shown as the final triggered sports answer. We leverage various heuristics, deep learning models like UniLM and large language models like GPT-4o for both QU and ranking. Lastly, we present evaluation results for various blocks in our pipeline on a set of ~80K Bing queries from May 2024.
TAFSIL: Taxonomy Adaptable Fine-grained Entity Recognition through Distant Supervision for Indian Languages
Authors: Prachuryya Kaushik, Shivansh Mishra, Ashish Anand
Venue: SIGIR
Year: 2025

Several studies have used distant supervision to create resources for fine-grained entity recognition (FgER) to mitigate the challenges of manual annotation. However, most of these methods are primarily developed for English and cannot be efficiently adapted to many other languages, including Indian languages. Moreover, the emergence of new and unseen entity types deteriorates the performance of the supervised models trained on FgER datasets with different predefined sets of entity types. This work introduces TAFSIL, a taxonomy-adaptable FgER framework to create FgER datasets in six Indian languages. The chosen languages are spoken by more than a billion speakers across various countries. TAFSIL utilizes the high interlink between the knowledge base WikiData and linked corpora Wikipedia through multi-stage heuristics and improves annotation through fuzzy match and quality sentence selection. TAFSIL enables us to create datasets of a total size of around three million samples for six languages Hindi (Hi), Marathi (Mr), Sanskrit (Sa), Tamil (Ta), Telugu (Te), and Urdu (Ur) belonging to two language families Indo-European and Dravidian. We evaluate the robustness of TAFSIL by creating various datasets in four taxonomies FIGER, OntoNotes, HAnDS, and MultiCoNER2. Our extensive experiments suggest the sound quality of the datasets as there is a relative improvement of 83% in average F1 score over zero-shot performance across various FgER state-of-the-art models. The resource is publicly available at https://huggingface.co/datasets/prachuryyaIITG/TAFSIL.
Evaluating LLMs' (In)ability to Follow Prompts in QA Tasks
Authors: Aparup Khatua, Tobias Kalmbach, Prasenjit Mitra, Sandipan Sikdar
Venue: SIGIR
Year: 2025

While LLMs have achieved impressive performance across various tasks, one under-explored area is evaluating their ability to follow instructions provided in the prompt when generating responses. In the context of question-answering (QA) tasks, a crucial research gap is whether LLMs prioritize their own parametric knowledge or the context provided in the prompt when generating an answer. Ignoring prompts, even when explicitly instructed to follow them, may adversely affect performance and potentially lead to unintended consequences. Additionally, LLMs should be self-reflective (i.e., LLMs should recognize when their knowledge is inadequate) and avoid hallucinations in such scenarios. To address our research question, we propose Oedipus, an evaluation framework to evaluate LLMs' ability to follow prompts. We further note that such abilities could also be influenced by contamination (i.e., exposure to datasets during training) and parametric knowledge. Consequently, we develop a novel QA dataset with four types of contexts- correct, masked, noisy, and absurd contexts with recent questions that LLMs are unlikely to have encountered in pre-training data or corpus and cannot be answered from parametric knowledge. We evaluate eight LLMs through our proposed evaluation framework and observe that LLMs often fail to follow instructions correctly and are not self-reflective.
Hypencoder: Hypernetworks for Information Retrieval
Authors: Julian Killingback, Hansi Zeng, Hamed Zamani
Venue: SIGIR
Year: 2025

Existing information retrieval systems are largely constrained by their reliance on vector inner products to assess query-document relevance, which naturally limits the expressiveness of the relevance score they can produce. We propose a new paradigm; instead of representing a query as a vector, we use a small neural network that acts as a learned query-specific relevance function. This small neural network takes a document representation as input (in this work we use a single vector) and produces a scalar relevance score. To produce the small neural network we use a hypernetwork, a network that produces the weights of other networks, as our query encoder. We name this category of encoder models Hypencoders. Experiments on in-domain search tasks show that Hypencoders significantly outperform strong dense retrieval models and even surpass reranking models and retrieval models with an order of magnitude more parameters. To assess the extent of Hypencoders' capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue and instruction-following retrieval tasks. On harder tasks, we find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method, we implement an approximate search algorithm and show that our model is able to retrieve from a corpus of 8.8M documents in under 60 milliseconds.
DIFF: Dual Side-Information Filtering and Fusion for Sequential Recommendation
Authors: Hye-young Kim, Minjin Choi, Sunkyung Lee, Ilwoong Baek, Jongwuk Lee
Venue: SIGIR
Year: 2025

Side-information Integrated Sequential Recommendation (SISR) benefits from auxiliary item information to infer hidden user preferences, which is particularly effective for sparse interactions and cold-start scenarios. However, existing studies face two main challenges. (i) They fail to remove noisy signals in item sequence and (ii) they underutilize the potential of side-information integration. To tackle these issues, we propose a novel SISR model, Dual Side-Information Filtering and Fusion (DIFF), which employs frequency-based noise filtering and dual multi-sequence fusion. Specifically, we convert the item sequence to the frequency domain to filter out noisy short-term fluctuations in user interests. We then combine early and intermediate fusion to capture diverse relationships across item IDs and attributes. Thanks to our innovative filtering and fusion strategy, DIFF is more robust in learning subtle and complex item correlations in the sequence. DIFF outperforms state-of-the-art SISR models, achieving improvements of up to 14.1% and 12.5% in Recall@20 and NDCG@20 across four benchmark datasets.
Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation
Authors: Jieyong Kim, Hyunseo Kim, Hyunjin Cho, SeongKu Kang, Buru Chang, Jinyoung Yeo, Dongha Lee
Venue: SIGIR
Year: 2025

Recent advancements in Large Language Models (LLMs) have demonstrated exceptional performance across a wide range of tasks, generating significant interest in their application to recommendation systems. However, existing methods have not fully harnessed the potential of LLMs, often constrained by limited input information or failing to fully utilize their advanced reasoning capabilities. To address these limitations, we introduce EXP3RT, a novel LLM-based recommender designed to leverage rich preference information contained in user and item reviews. EXP3RT is basically fine-tuned through distillation from a teacher LLM to perform three key steps in order: (1) preference extraction (2) profile construction, and (3) textual reasoning for rating prediction. EXP3RT first extracts and encapsulates essential subjective preferences from raw reviews, next aggregates and summarizes them according to specific criteria to create user and item profiles. It then generates detailed step-by-step reasoning followed by predicted rating, i.e., reasoning-enhanced rating prediction, by considering both subjective and objective information from user/item profiles and item descriptions. This personalized preference reasoning from EXP3RT enhances rating prediction accuracy and also provides faithful and reasonable explanations for recommendation. Extensive experiments show that EXP3RT outperforms existing methods on both rating prediction and candidate item reranking for top-k recommendation, while significantly enhancing the explainability of recommendation systems.
ReCDAP: Relation-based Conditional Diffusion with Attention Pooling for Few-Shot Knowledge Graph Completion
Authors: Jeongho Kim, Chanyeong Heo, Jaehee Jung
Venue: SIGIR
Year: 2025

Knowledge Graphs (KGs), composed of triples in the form of (head, relation, tail) and consisting of entities and relations, play a key role in information retrieval systems such as question answering, entity search, and recommendation. In real-world KGs, although many entities exist, the relations exhibit a long-tail distribution, which can hinder information retrieval performance. Previous few-shot knowledge graph completion studies focused exclusively on the positive triple information that exists in the graph or, when negative triples were incorporated, used them merely as a signal to indicate incorrect triples. To overcome this limitation, we propose Relation-Based Conditional Diffusion with Attention Pooling (ReCDAP). First, negative triples are generated by randomly replacing the tail entity in the support set. By conditionally incorporating positive information in the KG and non-existent negative information into the diffusion process, the model separately estimates the latent distributions for positive and negative relations. Moreover, including an attention pooler enables the model to leverage the differences between positive and negative cases explicitly. Experiments on two widely used datasets demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance. The code is available at https://github.com/hou27/ReCDAP-FKGC.