OpenRaiser/Intern-Atlas huggingface dataset:
Bridging Short Videos and Streamers with Multi-Graph Contrastive Learning for Live Streaming Recommendation
Authors: Changle Qu, Liqin Zhao, Yanan Niu, Xiao Zhang, Jun Xu
Venue: SIGIR
Year: 2025

Recently, live streaming services have seen a surge in popularity, prompting many platforms to offer both short video and live streaming services to meet the diverse needs of users and streamers. This has resulted in a close connection between short videos and live streaming within these platforms. Incorporating short video data into live streaming recommendation through cross-domain approaches can effectively mitigate the sparsity of live streaming gifting data. However, existing cross-domain recommendation methods primarily focus on transferring information across domains through overlapping users or items, while overlooking the strong connection between non-overlapping short videos and streamers. In this paper, we propose MGCCDR, a Multi-Graph Contrastive learning framework for Cross-Domain Recommendation, which leverages both overlapping users and non-overlapping items to enhance information transfer. Specifically, we first learn global representations from a global graph to establish connections between streamers and short videos. Subsequently, we construct three bipartite graphs among users, authors, and videos and introduce multi-graph learning to capture preferences within the target domain view, the source domain view, and the cross-domain view. Additionally, to address the varying contributions of each graph to the final recommendation task, we design an attention-based method to effectively integrate these representations, facilitating the information aggregation across domains. Extensive experiments on both commercial and public datasets demonstrate that our MGCCDR significantly outperforms the state-of-the-art methods.
ROKSANA: An Open-Source Toolkit for Robust Graph-Based Keyword Search
Authors: Radin Hamidi Rad, Amir Khosrojerdi, Ebrahim Bagheri
Venue: SIGIR
Year: 2025

We introduce ROKSANA, an open-source Python toolkit designed to support research in graph-based keyword search under adversarial settings. ROKSANA provides a modular environment for dataset handling, graph neural network (GNN)-based retrieval, and adversarial attack modeling, enabling systematic evaluation of search robustness. The framework integrates built-in retrieval and attack methods while allowing seamless customization of search algorithms and perturbation strategies. Users can benchmark performance on a centralized leaderboard, generate reproducible evaluation reports, and explore ranking behaviors through an interactive web-based visualization interface. By centering around reproducibility, extensibility, and collaborative benchmarking, ROKSANA serves as a comprehensive platform for advancing robust and interpretable keyword search in graphs. This demonstration will showcase ROKSANA's capabilities in real-time, illustrating its impact on experimental workflows and adversarial robustness analysis in graph IR research.
Reassessing the Effectiveness of Reinforcement Learning based Recommender Systems for Sequential Recommendation
Authors: Dilina Chandika Rajapakse, Dietmar Jannach
Venue: SIGIR
Year: 2025

Over the past few years, researchers have explored the use of reinforcement learning (RL) for sequential recommendation problems. However, since RL techniques commonly target at optimizing long-term rewards, it is surprising that RL-based models are reported to be competitive with traditional supervised models when evaluated under the myopic next-item prediction protocol. A recent study suggests that reported performance gains of combining RL with supervised learning techniques, as done in the Self-Supervised Q-Learning (SQN) framework, may actually not come from learning an optimal policy, but that the RL component helps to learn embeddings that encode the users' past interactions. Given these observations, we aimed to reassess the performance of RL-enhanced sequential recommendations in the SQN framework. While we were able to reproduce the results reported in the respective papers, we found that properly-tuned supervised learning models like GRU4Rec substantially outperform the proposed RL-models from the literature. Our analyses furthermore revealed that there is a significant inconsistency in terms of evaluation protocols in the literature, and that the use of third-party implementations of existing models may lead to unreliable conclusions. Overall, still more research and alternative evaluation schemes seem required to fully leverage the power of RL for sequential recommendation tasks.
Private Preferences, Public Rankings: A Privacy-Preserving Framework for Marketplace Recommendations
Authors: Guilherme Ramos, Ludovico Boratto, Mirko Marras
Venue: SIGIR
Year: 2025

Protecting user privacy in recommender systems is crucial for fostering trust in marketplaces. In this paper, we propose a privacy-preserving framework that integrates public seller rankings into personalized recommendations without exposing sensitive user preferences. By utilizing ''seller representative users'' (encoding seller item rankings) and a novel recommendation mechanism, the framework preserves privacy while ensuring robust ranking accuracy. Our approach is validated on multiple use cases extracted from real-world datasets, showing its effectiveness across varying marketplace configurations. This framework is suited for real-world applications, such as e-commerce platforms, where it can enhance user trust, protect sensitive data, and improve engagement by transparently balancing personalization and privacy.
Text Obsoleteness Detection using Large Language Models
Authors: Rishav Ranaut, Sriparna Saha, Adam Jatowt, Manish Gupta
Venue: SIGIR
Year: 2025

Maintaining accurate and up-to-date information is a persistent challenge for large-scale knowledge repositories, where outdated content can compromise their value. In this paper, we present a Multitask learning framework that uses Large Language Models (LLMs) for two tasks: semantic update detection and semantic update necessity prediction. The update detection task identifies obsoleteness by comparing older and newer text versions, while the update necessity prediction task determines whether an update is required based on a given context. To support these tasks, we curate a specialized dataset from Wikipedia called SEMUPDATES, focusing on frequently updated articles. Our experiments with five LLMs across four datasets in zero-shot, few-shot, and fine-tuned settings demonstrate that fine-tuning significantly enhances performance. In the multitask learning setup, Qwen delivers the best overall performance, while Mistral achieves the highest accuracy on individual tasks when fine-tuned separately. However, the performance differences across models are not substantial, suggesting that multiple LLMs can be effectively adapted for content update automation. These findings highlight the potential of LLMs in detecting and predicting obsolescence, providing a scalable solution for maintaining the timeliness of digital knowledge repositories.
Breaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets
Authors: Mandeep Rathee, Venktesh V, Sean MacAvaney, Avishek Anand
Venue: SIGIR
Year: 2025

Advanced relevance models, such as those that use large language models (LLMs), provide highly accurate relevance estimations. However, their computational costs make them infeasible for processing large document corpora. To address this, retrieval systems often employ a telescoping approach, where computationally efficient but less precise lexical and semantic retrievers filter potential candidates for further ranking. However, this approach heavily depends on the quality of early-stage retrieval, which can potentially exclude relevant documents early in the process. In this work, we propose a novel paradigm for re-ranking called online relevance estimation that continuously updates relevance estimates for a query throughout the ranking process. Instead of re-ranking a fixed set of top-k documents in a single step, online relevance estimation iteratively re-scores smaller subsets of the most promising documents while adjusting relevance scores for the remaining pool based on the estimations from the final model using an online bandit-based algorithm. This dynamic process mitigates the recall limitations of telescoping systems by re-prioritizing documents initially deemed less relevant by earlier stages-including those completely excluded by earlier-stage retrievers. We validate our approach on TREC benchmarks under two scenarios: hybrid retrieval and adaptive retrieval. Experimental results demonstrate that our method is sample-efficient and significantly improves recall, highlighting the effectiveness of our online relevance estimation framework for modern search systems.
A Large-Scale Study of Reranker Relevance Feedback at Inference
Authors: Revanth Gangi Reddy, Pradeep Dasigi, Md. Arafat Sultan, Arman Cohan, Avirup Sil, Heng Ji, Hannaneh Hajishirzi
Venue: SIGIR
Year: 2025

Neural IR systems often employ a retrieve-and-rerank framework: a bi-encoder retrieves a fixed number of candidates (e.g., K=100), which a cross-encoder then reranks. Recent studies have indicated that relevance feedback from the reranker at inference time can improve the recall of the retriever. The approach works by updating the retriever's query representations via a distillation process that aligns it with the reranker's predictions. While a powerful idea, the arguably narrow scope of past studies focusing on a small number of specific domains such as english question answering and entity retrieval has left a gap in our understanding of how well it generalizes. In this paper, we study inference-time reranker relevance feedback extensively across multiple retrieval domains, languages, and modalities, while also investigating aspects such as the performance and latency implications of the number of distillation updates and feedback candidates.
LLM-based Search Assistant with Holistically Guided MCTS for Intricate Information Seeking
Authors: Ruiyang Ren, Yuhao Wang, Junyi Li, Jinhao Jiang, Wayne Xin Zhao, Wenjie Wang, Tat-Seng Chua
Venue: SIGIR
Year: 2025

In the era of vast digital information, the sheer volume and heterogeneity of available information present significant challenges for intricate information seeking. Users frequently face multistep web search tasks that involve navigating vast and varied data sources. This complexity demands every step remains comprehensive, accurate, and relevant. However, traditional search methods often struggle to balance the need for localized precision with the broader context required for holistic understanding, leaving critical facets of intricate queries underexplored. In this paper, we introduce an LLM-based search assistant that adopts a new information seeking paradigm with holistically guided Monte Carlo tree search (HG-MCTS). We reformulate the task as a progressive information collection process with a knowledge memory and unite an adaptive checklist with multi-perspective reward modeling in MCTS. The adaptive checklist provides explicit sub-goals to guide the MCTS process toward comprehensive coverage of complex user queries. Simultaneously, our multi-perspective reward modeling offers both exploration and retrieval rewards, along with progress feedback that tracks completed and remaining sub-goals, refining the checklist as the tree search progresses. By striking a balance between localized tree expansion and global guidance, HG-MCTS reduces redundancy in search paths and ensures that all crucial aspects of an intricate query are properly addressed. Extensive experiments on real-world intricate information seeking tasks demonstrate that HG-MCTS acquires thorough knowledge collections and delivers more accurate final responses compared with existing baselines.
OBELLA: Open the Book for Evaluating Long-Form Large Language Model Answers in Open-Domain Question Answering
Authors: Tianyu Ren, Zhaoyu Zhang, Hui Wang, Karen Rafferty
Venue: SIGIR
Year: 2025

Reliable factuality evaluation is critical for the iterative development of open-domain question answering (ODQA) systems, especially given the rise of large language models (LLMs) and their propensity for hallucination. However, state-of-the-art (SOTA) automatic metrics, which are mostly supervised, remain notably less reliable than humans. In this paper, we find two key challenges behind this gap: (1) length distribution mismatch between lengthy LLM answers and shorter training answers used by current metrics; and (2) reference incompleteness, where current metrics often misjudge valid system answers absent from given references-a challenge worsened by the diversity of LLM outputs. To address these issues, we present a new ODQA factuality evaluation dataset called OBELLA (Open-Book Evaluation for Long-form LLM Answers). OBELLA narrows the length distribution mismatch by significantly increasing the candidate answer length to align with LLM outputs. Moreover, it introduces a neutral class for plausible yet under-supported candidate answers to differentiate reference incompleteness from outright incorrectness, thus enabling flexible reevaluation by consulting external knowledge for more references. Based on OBELLA, we propose a novel metric named OBELLAM (OBELLA Metric). OBELLAM integrates a cross-attention mechanism to enhance long-form candidate answer representations and employs a dynamic closed-open book evaluation strategy to tackle reference incompleteness. Our OBELLAM sets a new SOTA in aligning with human judgments across two ODQA evaluation benchmarks, marking a promising step toward more robust ODQA factuality evaluation.
Reverse-Engineering the Retrieval Process in GenIR Models
Authors: Anja Reusch, Yonatan Belinkov
Venue: SIGIR
Year: 2025

Generative Information Retrieval (GenIR) is a novel paradigm in which a transformer encoder-decoder model predicts document rankings based on a query in an end-to-end fashion. These GenIR models have received significant attention due to their simple retrieval architecture while maintaining high retrieval effectiveness. However, in contrast to established retrieval architectures like cross-encoders or bi-encoders, their internal computations remain largely unknown. In this work, we investigate this retrieval mechanism and uncover the roles played by different model components (self-attention, cross-attention, MLPs) and their interaction to generate the document identifier. First, we show that the pre-trained encoder, which was not fine-tuned for retrieval, is sufficient for the retrieval process. Then, we find that the pass through the decoder can be divided into three stages: (I) the priming stage in which no component contributes query-specific information (II) the bridging stage where cross-attention transfers query information from the encoder to the decoder, and (III) the interaction stage where MLPs process this transferred information to predict the document identifier in the last layer. Our results indicate that document-specific information is only stored in a few components in the final stage of the retrieval process. We hope that our findings will motivate the development of more effective GenIR models and facilitate their improvements.
BM25 and all that - a look back
Authors: Stephen Robertson
Venue: SIGIR
Year: 2025

It is 30 years since the weighting-and-ranking function BM25 was published, and more than 55 years since I started work in the field we know as information retrieval. I will be talking about my experiences as an IR researcher over the period from 1968 to the early 2000s, including the development of the probabilistic model which led to BM25, and also some of the work on IR evaluation in the years since the Cranfield experiment. More generally, I will talk about some of the ways in which the field has changed and developed over that time, and about some of the characters who helped to shape the field, including my own interactions with them.
Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
Authors: Kevin Roitero, Dustin Wright, Michael Soprano, Isabelle Augenstein, Stefano Mizzaro
Venue: SIGIR
Year: 2025

Evaluating the truthfulness of online content is critical for combating misinformation. This study examines the efficiency and effectiveness of crowdsourced truthfulness assessments through a comparative analysis of two approaches: one involving full-length webpages as evidence for each claim, and another using summaries for each evidence document generated with an LLM. Using an A/B testing setting, we engage a diverse pool of participants tasked with evaluating the truthfulness of statements under these conditions. Our analysis explores both the quality of assessments and the behavioral patterns of participants. The results reveal that relying on summarized evidence offers comparable accuracy and error metrics to the standard modality while significantly improving efficiency. Workers in the Summary setting complete a significantly higher number of assessments, reducing task duration and costs. Additionally, the Summary modality maximizes internal agreement and maintains consistent reliance on and perceived usefulness of evidence, demonstrating its potential to streamline large-scale truthfulness evaluations.
InstInfo: A Just-in-Time Literature Recommendation System for Presentations
Authors: Kevin Ros, Rahul Suresh, ChengXiang Zhai
Venue: SIGIR
Year: 2025

The efficient discovery of academic literature is critical for research progress, yet many researchers have difficulties in finding literature. This work proposes InstInfo: a novel just-in-time literature recommendation system for presentations. InstInfo transcribes audio in real-time and recommends literature according to the ideas being discussed, thereby helping researchers ground presentations in academic literature while saving them the time of having to manually search. Informal usability studies show that InstInfo is easy to use and that researchers find value in the recommendations. InstInfo can be accessed at https://instinfo.com.
RARR Unraveled: Component-Level Insights into Hallucination Detection and Mitigation
Authors: Jonathan J. Ross, Ekaterina Khramtsova, Anton van der Vegt, Bevan Koopman, Guido Zuccon
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) often exhibit hallucinations, which makes detecting and mitigating these errors a critical challenge. The Retrofit Attribution using Research and Revision (RARR) framework addresses this challenge by extracting key aspects of an LLM response, verifying them against retrieved evidence, and resolving errors through re-prompting. In this work, we critically examine RARR and adapt its framework to incorporate publicly available evidence retrieval systems and generative models, thereby operationalizing the approach. We focus on hallucination detection, analyzing how each pipeline component contributes to this task. We also conduct a sentence-level analysis of hallucinations to provide a more granular assessment of RARR's performance. A key finding is that while query generation and retrieval are effective, the agreement module emerges as the weakest link in the RARR pipeline. We offer deeper insights into RARR's strengths, limitations, and potential areas for improvement, thereby broadening our understanding of hallucination detection in LLMs.
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for Deep Research
Authors: Corbin Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, Nikhil Rao
Venue: SIGIR
Year: 2025

Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ''known unknowns'' with clear indications of both what information is missing, and how to find it to answer the question. A yet unmet need of the NLP community is a bank of non-factoid, multi-perspective questions involving a great deal of unclear information needs, i.e. ''unknown unknowns''. We claim we can find such questions in search engine logs, which is surprising because most question-intent queries are indeed factoid. Furthermore, recent products like Google's DeepResearch (announced a year after this resource was released publicly) specifically address such queries, retrieving hundreds of documents to synthesize report-style responses. We present Researchy Questions, the world's first, only and largest public dataset of ''Deep Research'' questions filtered from real search engine logs to be non-factoid, ''decompositional'' and multi-perspective. We show that users spend substantial ''effort'' on these questions in terms of signals like clicks and session length. We also show that ''slow thinking'' answering techniques, like decomposition into sub-questions shows benefit over answering directly. We release (at https://huggingface.co/datasets/corbyrosset/researchy_questions) about 100k Researchy Questions with a permissive CDLA-2.0 license, along with click histograms on over 350k Clueweb22 URLs that were clicked for each question.
Action First: Leveraging Preference-Aware Actions for More Effective Decision-Making in Interactive Recommender Systems
Authors: Renting Rui, Yunjia Xi, Weiwen Liu, Jianghao Lin, Bo Chen, Ruiming Tang, Weinan Zhang, Yong Yu
Venue: SIGIR
Year: 2025

Interactive recommender systems (IRSs) aim to meet user needs through natural language dialogues, optimizing recommendations with minimal interactions. Typically, IRSs are based on large language models (LLMs). Existing methods generally consist of two stages: decision-making (deciding whether to recommend or ask clarification questions) and action execution (generating recommendations or clarification questions). These methods usually follow a decision-first paradigm, where the model first decides on the action based on past conversations, and then executes the corresponding action. Since LLMs struggle to process a large number of candidate items, the recommendation process is often carried out in collaboration with external recommendation tools, which provide a small candidate set for LLMs to refine. Existing methods face two key information gaps: (1) In the decision-making stage, the decision-first paradigm relies solely on past conversations, leading to incomplete decisions due to the uncertainty of subsequent actions' outcomes. (2) In the action execution stage, there is a unidirectional flow from external recommendation tools to LLMs, where these tools fail to interpret user preferences effectively, thus reducing the recommendation accuracy. To address these challenges, we introduce Action-First Interactive Recommender System(AF-IRS), a novel model that uses preference-aware actions to guide decision-making. Our Action-First paradigm informs decisions with future recommendations, ensuring more accurate decisions. Additionally, we establish a bidirectional interaction loop between LLMs and external recommendation tools, enabling LLMs to interpret and transmit session preferences for more precise recommendations. Experimental results on three benchmark datasets demonstrate that AF-IRS significantly improves both recommendation accuracy and efficiency, addressing the information gaps in both stages.