OpenRaiser/Intern-Atlas huggingface dataset:
In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification
Authors: Daria Alexander, Arjen P. de Vries
Venue: SIGIR
Year: 2025

User intent classification is an important task in information retrieval. Previously, user intents were classified manually and automatically; the latter helped to avoid hand labelling of large datasets. Recent studies explored whether LLMs can reliably determine user intent. However, researchers have recognized the limitations of using generative LLMs for classification tasks. In this study, we empirically compare user intent classification into informational, navigational, and transactional categories, using weak supervision and LLMs. Specifically, we evaluate LLaMA-3.1-8B-Instruct and LLaMA-3.1-70B-Instruct for in-context learning and LLaMA-3.1-8B-Instruct for fine-tuning, comparing their performance to an established baseline classifier trained using weak supervision (ORCAS-I). Our results indicate that while LLMs outperform weak supervision in recall, they continue to struggle with precision, which shows the need for improved methods to balance both metrics effectively.
Techie: Tackling Video Prefetching at Edge Networks as POMDP Via an Intrinsically Motivated RL Agent
Authors: Nawras Alkassab, Chin-Tser Huang, Tania Lorido-Botran
Venue: SIGIR
Year: 2025

Video prefetching techniques play a critical role in hiding the I/O latency of video delivery from cloud networks to access networks by optimizing edge networks. Content Delivery Networks rely on edge computing to improve Quality of Experience at the client-side, and improve resource utilization at edge and cloud networks simultaneously by utilizing prefetching optimization techniques. Recently, we have witnessed a trend in shifting intelligence to edge networks by building video prefetchers that utilize deep reinforcement learning to make online prefetching decisions. Unfortunately, these methods often lack generalization to different workloads when dealing with long sequences, rendering them incapable of adapting to distribution shifts and various changes in users' requests which represent a non i.i.d distribution. In this work, we tackle video prefetching at edge networks as a Partially Observable Markov Decision Process, and propose Techie, an intrinsically motivated policy-gradient reinforcement learning agent that differentiates intrinsic rewards from extrinsic rewards based on their availability to edge networks. Techie is adaptive to unseen user requests by prefetching aggressively to handle randomization in workloads, achieving 52.27% of prefetching accuracy and 34.34% of prefetching coverage. Our results show that Techie improves prefetching accuracy and coverage by at least 16.12% and 6.57%, respectively, compared to baseline approaches that utilize deep learning, deep reinforcement learning, or video popularity to build prefetching algorithms. Consequently, Techie minimizes end-to-end latency by at least 7.5% and reduces cache pollution with an improvement of at least 13.1% on unseen user requests compared to baselines.
FACTors: A New Dataset for Studying the Fact-checking Ecosystem
Authors: Enes Altuncu, Can Baskent, Sanjay Bhattacherjee, Shujun Li, Dwaipayan Roy
Venue: SIGIR
Year: 2025

Our fight against false information is spearheaded by fact-checkers. They investigate the veracity of claims and document their findings as fact-checking reports. With the rapid increase in the amount of false information circulating online, the use of automation in fact-checking processes aims to strengthen this ecosystem by enhancing scalability. Datasets containing fact-checked claims play a key role in developing such automated solutions. However, to the best of our knowledge, there is no fact-checking dataset at the ecosystem level, covering claims from a sufficiently long period of time and sourced from a wide range of actors reflecting the entire ecosystem that admittedly follows widely-accepted codes and principles of fact-checking. We present a new dataset FACTors, the first to fill this gap by presenting ecosystem-level data on fact-checking. It contains 118,112 claims from 117,993 fact-checking reports in English (co-)authored by 1,953 individuals and published during the period of 1995-2025 by 39 fact-checking organisations that are active signatories of the IFCN (International Fact-Checking Network) and/or EFCSN (European Fact-Checking Standards Network). It contains 7,327 overlapping claims investigated by multiple fact-checking organisations, corresponding to 2,977 unique claims. It allows to conduct new ecosystem-level studies of the fact-checkers (organisations and individuals). To demonstrate the usefulness of our dataset, we present three example applications. They include a first-of-its-kind statistical analysis of the fact-checking ecosystem, examining the political inclinations of the fact-checking organisations, and attempting to assign a credibility score to each organisation based on the findings of the statistical analysis and political leanings. Our methods for constructing FACTors are generic and can be used to maintain a live dataset that can be updated dynamically.
TINK: Text Information Navigation Kit
Authors: Dean E. Alvarez, ChengXiang Zhai
Venue: SIGIR
Year: 2025

In many cases it is desirable to navigate an arbitrary text collection. While current technologies allow for fast and efficient text collection search, comparable technologies do not yet exist for navigation. Our prior work has proposed Hypergraph of Text (HoT) as a structure that could enable such navigation, but there is not yet a general system that allows users to interact with such a structure. To address this challenge, we introduce Text Information Navigation Kit (TINK) a novel tool-kit that allows users to turn any collection of plain text documents into a navigable collection. We discuss the modular and extensible two-part structure of TINK. In this demonstration paper we discuss this novel system, its strengths and its weaknesses. The code for TINK can be found in our GitHub repo here:https://github.com/TIMAN-group/TINK
Enhancing AI Agents with Human Theory of Mind (ToM) for Context-Aware Actions
Authors: Maryam Amirizaniani
Venue: SIGIR
Year: 2025

In the domain of artificial intelligence, the role of agents is rapidly expanding. An agent is an autonomous entity that perceives its environment, makes informed decisions, and takes actions to complete tasks. However, enabling agents to accurately interpret and respond to human requests-particularly through understanding human Theory of Mind (ToM)-remains a complex and pivotal challenge. ToM encompasses the ability to recognize that individuals possess distinct mental states, such as intentions, beliefs, and desires, which may differ from one's own. This cognitive faculty is fundamental to human interaction, enabling empathy and understanding across diverse contexts. As AI agents are increasingly deployed across new domains, approximating ToM becomes vital for producing appropriate and context-sensitive responses. Recent studies have explored methodologies to enhance ToM capabilities in large language models (LLMs), aiming to refine their ability to infer and track human mental states. By leveraging ToM, LLMs have been shown to approximate key aspects of human-like social cognition. However, most existing research has focused primarily on evaluating ToM within LLMs, overlooking the broader challenge of enabling AI agents to actively understand and respond to human cognitive and mental states. While previous studies have developed agents focused on extracting human mental states, further research is needed to design agents capable of performing diverse tasks while integrating ToM reasoning. AI agents must not only infer mental states but also leverage them to enhance reasoning, planning, decision-making, and task execution. For example, an educational tutor agent assisting a student with a math problem must recognize whether the student seeks step-by-step guidance or conceptual clarification, and whether their broader intent is to master the material or to prepare for an exam. Without accurately inferring and leveraging such mental states, agents risk producing ineffective interactions, leading to user frustration and disengagement. Incorporating ToM information is therefore crucial for improving decision-making and enabling more contextually appropriate responses. To address this gap,ToM capabilities must be incorporated into AI agents by modeling core mental states such as Belief, Desire, and Intention (BDI). Building on BDI as a structured foundation for representing human cognition, ToM-Act is proposed as a framework to enhance agent decision-making and action-taking through ToM-informed reasoning. This motivation stems from the increasing deployment of AI agents, highlighting the need for methods that optimize their decision-making.
Beyond Whole Dialogue Modeling: Contextual Disentanglement for Conversational Recommendation
Authors: Guojia An, Jie Zou, Jiwei Wei, Chaoning Zhang, Fuming Sun, Yang Yang
Venue: SIGIR
Year: 2025

Conversational recommender systems aim to provide personalized recommendations by analyzing and utilizing contextual information related to dialogue. However, existing methods typically model the dialogue context as a whole, neglecting the inherent complexity and entanglement within the dialogue. Specifically, a dialogue comprises both focus information and background information, which mutually influence each other. Current methods tend to model these two types of information mixedly, leading to misinterpretation of users' actual needs, thereby lowering the accuracy of recommendations. To address this issue, this paper proposes a novel model to introduce contextual disentanglement for improving conversational recommender systems, named DisenCRS. The proposed model DisenCRS employs a dual disentanglement framework, including self-supervised contrastive disentanglement and counterfactual inference disentanglement, to effectively distinguish focus information and background information from the dialogue context under unsupervised conditions. Moreover, we design an adaptive prompt learning module to automatically select the most suitable prompt based on the specific dialogue context, fully leveraging the power of large language models. Experimental results on two widely used public datasets demonstrate that DisenCRS significantly outperforms existing conversational recommendation models, achieving superior performance on both item recommendation and response generation tasks.
VAP3: Variation-Aware Prompt Performance Prediction
Authors: Negar Arabzadeh, Ebrahim Bagheri
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) exhibit strong capabilities across various Information Retrieval (IR) and natural language processing tasks. However, they are highly sensitive to prompt variations, where slight rephrasings can significantly alter responses, leading to inconsistent or incorrect outputs. This variability poses challenges for response reliability in real-world applications. Inspired by Query Performance Prediction (QPP) in IR, we focus on Prompt Performance Prediction (PPP), which estimates whether an LLM will generate a correct response for a given prompt before execution. We propose VAP3 (Variation-Aware Prompt Performance Prediction), a novel pre-generation PPP approach that integrates prompt variations with adversarial training to enhance robustness against trivial modifications and better capture prompt sensitivity. Evaluating VAP3 against LLM-based self-evaluation, QPP-inspired baselines, and supervised classification models on the PromptSET-HotpotQA and PromptSET-TriviaQA datasets, we demonstrate that VAP3 consistently outperforms all baselines, achieving stable and reliable performance across datasets.
A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
Authors: Negar Arabzadeh, Charles L. A. Clarke
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks-binary, graded, and pairwise-yielding 90 prompts in total. We compare LLM-generated labels with TREC official human labels using Cohen's κ and pairwise agreement measures. In addition, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.
IR-RAG @SIGIR25: The Second Edition of the Workshop on Information Retrieval's Role in RAG Systems
Authors: Negar Arabzadeh, Ziheng Chen, Fabio Petroni, Federico Siciliano, Fabrizio Silvestri, Giovanni Trappolini
Venue: SIGIR
Year: 2025

In recent years, Retrieval-Augmented Generation (RAG) systems have become a cornerstone of artificial intelligence, attracting considerable attention in a variety of fields. By integrating the strengths of information retrieval and generative models, these systems have shown immense potential to push the boundaries of machine learning applications. Nevertheless, RAG systems still face significant challenges and offer ample room for advancement and innovation. This workshop aims to highlight the central role of information retrieval within RAG frameworks, which we believe has often been overshadowed by the emphasis on the generative components. While the generative models are integral to these systems, the quality and effectiveness of the retrieval mechanism is equally critical, as it has a direct impact on the overall system performance and outcomes. We invite papers that rethink and prioritise the fundamental aspects of RAG systems, particularly in strengthening the information retrieval component. Through this workshop, we aim to gain deeper insights into how improved retrieval methods can enhance the performance and reliability of RAG systems. The event will bring together leading experts, researchers and practitioners to provide a collaborative platform for exchanging ideas, sharing results and fostering innovation. Our aim is to stimulate research and discussion that reaffirms the essential role of information retrieval in shaping the next generation of generative systems.
From Professional Search to Generative Deep Research Systems: How Can Expert Oversight Improve Search Outcomes?
Authors: Samy Ateia
Venue: SIGIR
Year: 2025

Generative AI (GenAI) models are increasingly employed for professional search tasks, from enterprise copilots to systematic reviews. Recently introduced ''deep research'' products (OpenAI, Google), marketed as automated professional search agents, promise comprehensive query generation and synthesis from large document sets. However, this drive toward automation risks minimizing user involvement, potentially leading to misaligned results and inappropriate reliance on AI-generated outputs. Professional search, especially in domains such as biomedical research and systematic reviews, necessitates expert input, transparency, and user control, requirements often not met by current generative search tools. Interactive Information Retrieval (IIR) literature has long emphasized the value of user interactions and iterative search refinement. The rise of generative search introduces innovative methods of accessing information through natural language interactions, which show promise in addressing complex information needs. However, professional search tasks differ substantially from general consumer searches, involving transparent criteria for relevance judgments and systematic data extraction processes. Despite early evidence suggesting increased productivity among knowledge workers using GenAI tools. these systems also exhibit substantial shortcomings. Studies highlight issues such as unsupported claims, inaccurate citations, and overreliance on generated content, questioning the reliability and suitability of current generative search systems for high-stakes professional use. Given these challenges, this work explores systematically integrating structured expert feedback at multiple stages of a generative search process, specifically query formulation, relevance judgments, and information extraction. We aim to shift the focus from full automation toward increased expert control and quality assurance. Our key research questions are:RQ1: How can expert feedback be systematically integrated into GenAI-driven retrieval systems to improve result quality? RQ2: How does a feedback-driven generative search system compare to fully automated approaches in terms of retrieval effectiveness and task performance? RQ3: Does the integration of human expert feedback improve professional search outcomes compared to GenAI-simulated expert feedback? Building on our earlier experiments that assessed the performance of GenAI models in a fully automated setting for biomedical question answering and introduced an interactive system for refining generated queries, we now seek to evaluate how a structured, human-in-the-loop feedback approach might improve retrieval and task effectiveness over fully automated pipelines. By evaluating our proposed framework on biomedical Q&A datasets such as CLEF BioASQ and TREC BioGen as well as systematic review datasets, we want to highlight the value of human guidance for professional generative search.
Measuring the Fairness Gap Between Retrieval and Generation in RAG Systems using a Cognitive Complexity Framework
Authors: Sandeep Avula, Chia-Jung Lee, Rongting Zhang, Vanessa Murdock
Venue: SIGIR
Year: 2025

In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.
Leveraging Artificial Intelligence-Powered Virtual Assistant for Information Retrieval in Indigenous Agriculture: Insights from Nigeria
Authors: Olusesan Michael Awoleye, Oluwamayowa Adebisi, Idowu Ademola Atoloye, Atanda Samuel Oladejo, Abiodun T. Atoloye, Cornelius Talade Atere, Victoria Tanimonure
Venue: SIGIR
Year: 2025

This study examines the use of an AI-powered Virtual Assistant (VA) to aid information retrieval in the cultivation of indigenous vegetables in Nigeria. A two-phase approach involved a needs assessment with 240 youth and the deployment of a bilingual web-based VA using a Large Language Model. The results show a strong youth interest in the tool, despite infrastructure and financial challenges. The VA achieved a high Precision@3 score (0.87), indicating its potential to improve agricultural decision-making in low-resource areas. The use of the VA can improve access to timely and context-specific agricultural knowledge, thereby supporting climate change adaptation and contributing to global food security.
Question-Answer Extraction from Scientific Articles Using Knowledge Graphs and Large Language Models
Authors: Hosein Azarbonyad, Zi Long Zhu, Georgios Cheirmpos, Zubair Afzal, Vikrant Yadav, Georgios Tsatsaronis
Venue: SIGIR
Year: 2025

When deciding to read an article or incorporate it into their research, scholars often seek to quickly identify and understand its main ideas. In this paper, we aim to extract these key concepts and contributions from scientific articles in the form of Question and Answer (QA) pairs. We propose two distinct approaches for generating QAs. The first approach involves selecting salient paragraphs, using a Large Language Model (LLM) to generate questions, ranking these questions by the likelihood of obtaining meaningful answers, and subsequently generating answers. This method relies exclusively on the content of the articles. However, assessing an article's novelty typically requires comparison with the existing literature. Therefore, our second approach leverages a Knowledge Graph (KG) for QA generation. We construct a KG by fine-tuning an Entity Relationship (ER) extraction model on scientific articles and using it to build the graph. We then employ a salient triplet extraction method to select the most pertinent ERs per article, utilizing metrics such as the centrality of entities based on a triplet TF-IDF-like measure. This measure assesses the saliency of a triplet based on its importance within the article compared to its prevalence in the literature. For evaluation, we generate QAs using both approaches and have them assessed by Subject Matter Experts (SMEs) through a set of predefined metrics to evaluate the quality of both questions and answers. Our evaluations demonstrate that the KG-based approach effectively captures the main ideas discussed in the articles. Furthermore, our findings indicate that fine-tuning the ER extraction model on our scientific corpus is crucial for extracting high-quality triplets from such documents.
Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems
Authors: Yimeng Bai, Shunyu Zhang, Yang Zhang, Hu Liu, Wentian Bao, Enyun Yu, Fuli Feng, Wenwu Ou
Venue: SIGIR
Year: 2025

Ranking models primarily focus on modeling the relative order of predictions while often neglecting the significance of the accuracy of their absolute values. However, accurate absolute values are essential for certain downstream tasks, necessitating the calibration of the original predictions. To address this, existing calibration approaches typically employ predefined transformation functions with order-preserving properties to adjust the original predictions. Unfortunately, these functions often adhere to fixed forms, such as piece-wise linear functions, which exhibit limited expressiveness and flexibility, thereby constraining their effectiveness in complex calibration scenarios. To mitigate this issue, we propose implementing a calibrator using an Unconstrained Monotonic Neural Network (UMNN), which can learn arbitrary monotonic functions with great modeling power. This approach significantly relaxes the constraints on the calibrator, improving its flexibility and expressiveness while avoiding excessively distorting the original predictions by requiring monotonicity. Furthermore, to optimize this highly flexible network for calibration, we introduce a novel additional loss function termed Smooth Calibration Loss (SCLoss), which aims to fulfill a necessary condition for achieving the ideal calibration state. Extensive offline experiments confirm the effectiveness of our method in achieving superior calibration performance. Moreover, deployment in Kuaishou's large-scale online video ranking system demonstrates that the method's calibration improvements translate into enhanced business metrics. The source code is available at https://github.com/baiyimeng/UMC.