OpenRaiser/Intern-Atlas huggingface dataset:
ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions
Authors: Zhiyuan Peng, Jinming Nian, Alexandre V. Evfimievski, Yi Fang
Venue: SIGIR
Year: 2025

Retrieval-augmented generation (RAG) has become integral to large language models (LLMs), particularly for conversational AI systems where user questions may reference knowledge beyond the LLMs' training cutoff. However, many natural user questions lack well-defined answers, either due to limited domain knowledge or because the retrieval system returns documents that are relevant in appearance but uninformative in content. In such cases, LLMs often produce hallucinated answers without flagging them. While recent work has largely focused on questions with false premises, we study out-of-scope questions, where the retrieved document appears semantically similar to the question but lacks the necessary information to answer it. In this paper, we propose a guided hallucination-based approach ELOQ . https://github.com/zhiyuanpeng/ELOQ.git, for automatically generating a diverse set of out-of-scope questions from post-cutoff documents, followed by human verification to ensure quality. We use this dataset to evaluate several LLMs on their ability to detect out-of-scope questions and generate appropriate responses. Finally, we introduce an improved detection method that enhances the reliability of LLM-based question-answering systems in handling out-of-scope questions.
Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation
Authors: David Otero, Javier Parapar, Álvaro Barreiro
Venue: SIGIR
Year: 2025

Offline evaluation of search systems depends on test collections. These benchmarks provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. While test collections are an integral part of Information Retrieval (IR) research, their creation involves significant efforts in manual annotation. Large language models (LLMs) are gaining much attention as tools for automatic relevance assessment. Recent research has shown that LLM-based assessments yield high systems ranking correlation with human-made judgements. These correlations are helpful in large-scale experiments but less informative if we want to focus on top-performing systems. Moreover, these correlations ignore whether and how LLM-based judgements impact the statistically significant differences among systems with respect to human assessments. In this work, we look at how LLM-generated judgements preserve ranking differences among top-performing systems and also how they preserve pairwise significance evaluation as human judgements. Our results show that LLM-based judgements are unfair at ranking top-performing systems. Moreover, we observe an exceedingly high rate of false positives regarding statistical differences.
Can LLMs Enhance Fairness in Recommendation Systems? A Data Augmentation Approach
Authors: Hanzhe Li, Dazhong Shen, Chao Wang, Yuting Liu, Jingjing Gu
Venue: SIGIR
Year: 2025

Despite the vital role of recommendation systems (RS) in delivering personalized services tailored to users' needs, user fairness issues have increasingly emerged in recent years, especially differentiated treatments caused by user sensitive attributes. This not only undermines both user experience and platform revenues, but also leads to potential social unfairness. Although many fairness-aware methods have been developed and achieved some success, many of them filter out sensitive attribute information while ignoring the potential loss of personalized information, leading to suboptimal results. Large language models (LLMs) have demonstrated remarkable capabilities across various tasks, while their potential in fairness-aware recommendation remains further unexplored. In this paper, we propose a new exploration of fairness-aware RS by prompting LLMs with the user's personalized fairness degrees to augment fair user-item interaction for training. Specifically, to estimate the fairness degree of each user, we first design a personalized unfairness modelling module, consisting of a replaceable fairness-aware representation learning model. Moreover, to enable LLMs to perceive fairness from semantic information and adapt to various scenarios, we propose a prompt tuning mechanism to optimize user-shared prompt templates with the objective of maximizing the consistency with users' preferences and the diversity of augmented data. Finally, we utilize LLMs to augment fair interaction data with the optimal prompts and integrate it with the raw data to re-train the recommendation model. Extensive experiments on two real-world datasets demonstrate the superiority of our approach in terms of recommendation performance, fairness, and robustness.
From To-Do to Ta-Da: Transforming Task-Focused IR with Generative AI
Authors: Chirag Shah, Ryen W. White
Venue: SIGIR
Year: 2025

For decades, scholars have emphasized that tasks should be the central focus in Information Retrieval (IR). This point of view holds even more significance with the advent of Generative Artificial Intelligence (GenAI) models, which can, among other capabilities, understand natural language, engage in dialog with users, generate bespoke user interfaces, and power agents to help complete tasks. GenAI presents an unprecedented opportunity to finally realize the potential of tasks in IR, enhance task-focused retrieval and interaction, and create ''magical'' task completion moments for users. In this paper, we explore the rationale and methodology behind this argument. Traditional IR systems support mostly simple tasks. The emergence of GenAI creates an opportunity for IR systems to help users achieve complex tasks and for the IR community to rekindle its interest and demonstrate leadership in this sizable and significant problem space. We underscore the pivotal role of tasks in IR and introduce new evidence supporting the notion that task-centric approaches, abstracted from specific modalities, represent the future of IR. Building on this foundation, we envision the development, utilization, and evaluation of next-generation IR systems. We propose a promising future where IR agents prioritize users, their tasks, and their situations. However, despite their potential to address task-focused and modality-independent IR, agents alone are insufficient. We propose a robust ecosystem around these agents that transcends traditional queries, questions, prompts, and modalities to address users' fundamental needs, tasks, and goals.
Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation
Authors: Qidong Liu, Xiangyu Zhao, Yejing Wang, Zijian Zhang, Howard Zhong, Chong Chen, Xiang Li, Wei Huang, Feng Tian
Venue: SIGIR
Year: 2025

Cross-domain Sequential Recommendation (CDSR) aims to extract the preference from the user's historical interactions across various domains. Despite some progress in CDSR, two problems set the barrier for further advancements, i.e., overlap dilemma and transition complexity. The former means existing CDSR methods severely rely on users who own interactions on all domains to learn cross-domain item relationships, compromising the practicability. The latter refers to the difficulties in learning the complex transition patterns from the mixed behavior sequences. With powerful representation and reasoning abilities, Large Language Models (LLMs) are promising to address these two problems by bridging the items and capturing the user's preferences from a semantic view. Therefore, we propose an LLMs Enhanced Cross-domain Sequential Recommendation model (LLM4CDSR). To obtain the semantic item relationships, we first propose an LLM-based unified representation module to represent items. Then, a trainable adapter with contrastive regularization is designed to adapt the CDSR task. Besides, a hierarchical LLMs profiling module is designed to summarize user cross-domain preferences. Finally, these two modules are integrated into the proposed tri-thread framework to derive recommendations. We have conducted extensive experiments on three public cross-domain datasets, validating the effectiveness of LLM4CDSR. We have released the code online.
ID-Free Not Risk-Free: LLM-Powered Agents Unveil Risks in ID-Free Recommender Systems
Authors: Zongwei Wang, Min Gao, Junliang Yu, Xinyi Gao, Quoc Viet Hung Nguyen, Shazia Sadiq, Hongzhi Yin
Venue: SIGIR
Year: 2025

Recent advances in ID-free recommender systems have attracted significant attention for effectively addressing the cold start problem. However, their vulnerability to malicious attacks remains largely unexplored. In this paper, we unveil a critical yet overlooked risk: LLM-powered agents can be strategically deployed to attack ID-free recommenders, stealthily promoting low-quality items in black-box settings. This attack exploits a novel rewriting-based deception strategy, where malicious agents synthesize deceptive textual descriptions by simulating the characteristics of popular items. To achieve this, the attack mechanism integrates two primary components: (1) a popularity extraction component that captures essential characteristics of popular items and (2) a multi-agent collaboration mechanism that enables iterative refinement of promotional textual descriptions through independent thinking and team discussion. To counter this risk, we further introduce a detection method to identify suspicious text generated by our discovered attack. By unveiling this risk, our work aims to underscore the urgent need to enhance the security of ID-free recommender systems.
Query Understanding in LLM-based Conversational Information Seeking
Authors: Yifei Yuan, Zahra Abbasiantaeb, Mohammad Aliannejadi, Yang Deng
Venue: SIGIR
Year: 2025

Query understanding in CIS involves accurately interpreting user intent through context-aware interactions. This includes resolving ambiguities, refining queries, and adapting to evolving information needs. LLM enhance this process by interpreting nuanced language and adapting dynamically, improving the relevance and precision of search results in real-time. In this tutorial, we explore advanced techniques to enhance query understanding in LLM-based CIS systems. We delve into LLM-driven methods for developing robust evaluation metrics to assess query understanding quality in multi-turn interactions, strategies for building more interactive systems, and applications like proactive query management and query reformulation. We also discuss key challenges in integrating LLM for query understanding in conversational search systems and outline future research directions. Our goal is to deepen the audience's understanding of LLM-based conversational query understanding and inspire discussions to drive ongoing advancements in this field.
Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrieval
Authors: Yongkang Li, Panagiotis Eustratiadis, Simon Lupart, Evangelos Kanoulas
Venue: SIGIR
Year: 2025

This paper concerns corpus poisoning attacks in dense information retrieval, where an adversary attempts to compromise the ranking performance of a search algorithm by injecting a small number of maliciously generated documents into the corpus. Our work addresses two limitations in the current literature. First, attacks that perform adversarial gradient-based word substitution search do so in the discrete lexical space, while retrieval itself happens in the continuous embedding space. We thus propose an optimization method that operates in the embedding space directly. Specifically, we train a perturbation model with the objective of maintaining the geometric distance between the original and adversarial document embeddings, while also maximizing the token-level dissimilarity between the original and adversarial documents. Second, it is common for related work to have a strong assumption that the adversary has prior knowledge about the queries. In this paper, we focus on a more challenging variant of the problem where the adversary assumes no prior knowledge about the query distribution (hence, unsupervised). Our core contribution is an adversarial corpus attack that is fast and effective. We present comprehensive experimental results on both in- and out-of-domain datasets, focusing on two related tasks: a top-1 attack and a corpus poisoning attack. We consider attacks under both a white-box and a black-box setting. Notably, our method can generate successful adversarial examples in under two minutes per target document; four times faster compared to the fastest gradient-based word substitution methods in the literature with the same hardware. Furthermore, our adversarial generation method generates text that is more likely to occur under the distribution of natural text (low perplexity), and is therefore more difficult to detect.
REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources
Authors: Björn Engelmann, Fabian Haak, Philipp Schaer, Mani Erfanian Abdoust, Linus Netze, Meik Bittkowski
Venue: SIGIR
Year: 2025

Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources.
The 1st NIP@IR Workshop on New Interaction Paradigms for Information Retrieval in the Era of Generative AI
Authors: Yujia Zhou, Wei Ji, Xuri Ge, Qingyao Ai, Joemon M. Jose, Yiqun Liu
Venue: SIGIR
Year: 2025

The advent of generative artificial intelligence (AI), driven by advancements in large language models (LLMs), has unlocked transformative possibilities for information retrieval (IR), giving rise to a new wave of interactive and conversational paradigms. This workshop, titled New Interaction Paradigms for Information Retrieval in the Era of Generative AI, aims to serve as a collaborative platform for researchers and practitioners to explore the challenges and opportunities of integrating generative AI into IR systems. By focusing on tasks such as multi-turn conversational search, adaptive retrieval interfaces, and context-aware response generation, this workshop will address key areas including system design, user engagement, and evaluation methodologies. The workshop will also delve into broader concerns such as trust, transparency, and fairness, emphasizing the ethical implications of deploying generative AI in IR systems. Through panel discussions, poster sessions, and interactive roundtables, this workshop will foster critical dialogue and innovation, paving the way for a new era of user-centric, generative AI-powered IR systems.
Unveiling Knowledge Boundary of Large Language Models for Trustworthy Information Access
Authors: Yang Deng, Moxin Li, Liang Pang, Wenxuan Zhang, Wai Lam
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) have emerged as powerful tools for generating content and facilitating information seeking across diverse domains. While their integration into conversational systems opens new avenues for interactive information-seeking experiences, their effectiveness is constrained by their knowledge boundaries-the limits of what they know and their ability to provide reliable, truthful, and contextually appropriate information. Understanding these boundaries is essential for maximizing the utility of LLMs for real-time information seeking while ensuring their reliability and trustworthiness. In this tutorial, we will explore the taxonomy of knowledge boundary in LLMs, addressing their handling of uncertainty, response calibration, and mitigation of unintended behaviors that can arise during interaction with users. We will also present advanced techniques for optimizing LLM behavior in generative information-seeking tasks, ensuring that models align with user expectations of accuracy and transparency. Attendees will gain insights into research trends and practical methods for enhancing the reliability and utility of LLMs for trustworthy information access.
Generative Meta-Learning for Zero-Shot Relation Triplet Extraction
Authors: Wanli Li, Tieyun Qian, Yi Song, Zeyu Zhang, Jiawei Li, Zhuang Chen, Lixin Zou
Venue: SIGIR
Year: 2025

Zero-shot Relation Triplet Extraction (ZeroRTE) aims to extract relation triplets from texts containing unseen relation types. This capability benefits various downstream information retrieval (IR) tasks. The primary challenge lies in enabling models to generalize effectively to unseen relation categories. Existing approaches typically leverage the knowledge embedded in pre-trained language models to accomplish the generalization process. However, these methods focus solely on fitting the training data during training, without specifically improving the model's generalization performance, resulting in limited generalization capability. For this reason, we explore the integration of bi-level optimization (BLO) with pre-trained language models for learning generalized knowledge directly from the training data, and propose a generative meta-learning framework which exploits the 'learning-to-learn' ability of meta-learning to boost the generalization capability of generative models. Specifically, we introduce a BLO approach that simultaneously addresses data fitting and generalization. This is achieved by constructing an upper-level loss to focus on generalization and a lower-level loss to ensure accurate data fitting. Building on this, we subsequently develop three generative meta-learning methods, each tailored to a distinct category of meta-learning. Extensive experimental results demonstrate that our framework performs well on the ZeroRTE task. Our code is available at https://github.com/leeworry/TGM-MetaLearning.
LIGHT: Enhancing Learning Path Recommendation via Knowledge Topology-Aware Sequence Optimization
Authors: Xiaoshan Yu, Shangshang Yang, Ziwen Wang, Siyu Song, Haiping Ma, Zhiguang Cao, Xingyi Zhang
Venue: SIGIR
Year: 2025

Learning path recommendation (LPR) aims to provide individualized and effective learning item routes by modeling learners' learning histories and goals, which has been widely considered a essential task in the field of personalized education. Indeed, considerable research efforts have been dedicated to this direction in recent years, focusing on step-based and sequence-based modeling approaches. However, most of existing studies overlook the complementarity between explicit and implicit relationships among knowledge concepts, while failing to harmonize static knowledge structures with dynamic path generation. To this end, in this paper, we propose LIGHT, a knowLedge topology-aware sequence optImization model for enhancing learninG patH recommendaTion. Specifically, we first construct a composite concept graph that incorporates explicit prerequisite relationships and implicit collaborative relationships, achieved by mining interaction statistics and collaborative signals from learners' learning processes. Next, we design a complementary contrastive fusion module to fully capture the interplay between the two relational views of concepts through graph structure learning and contrastive constraints, which enhances the effectiveness of the learned representations. Following this, we introduce a knowledge topology-aware modeling module that integrates structural semantics clustering with candidate path sampling. Finally, we develop a bidirectional sensing path optimization network to deeply model and optimize the sampled paths from a sequential perspective, thereby enhancing modeling efficiency while preserving structural semantics. Extensive experiments on three real-world educational datasets clearly demonstrate the effectiveness of the proposed LIGHT model in the LPR task.
Why is Normalization Necessary for Linear Recommenders?
Authors: Seongmin Park, Mincheol Yoon, Hye-young Kim, Jongwuk Lee
Venue: SIGIR
Year: 2025

Despite their simplicity, linear autoencoder (LAE)-based models have shown comparable or even better performance with faster inference speed than neural recommender models. However, LAEs face two critical challenges: (i) popularity bias, which tends to recommend popular items, and (ii) neighborhood bias, which overly focuses on capturing local item correlations. To address these issues, this paper first analyzes the effects of two existing normalization methods for LAEs, i.e., random-walk and symmetric normalization. Our theoretical analysis reveals that normalization highly affects the degree of popularity and neighborhood biases among items. Inspired by this analysis, we propose a versatile normalization solution, called Data-Adaptive Normalization (DAN), which flexibly controls the popularity and neighborhood biases by adjusting item- and user-side normalization to align with unique dataset characteristics. Owing to its model-agnostic property, DAN can be easily applied to various LAE-based models. Experimental results show that DAN-equipped LAEs consistently improve existing LAE-based models across six benchmark datasets, with significant gains of up to 128.57% and 12.36% for long-tail items and unbiased evaluations, respectively. Refer to our code in https://github.com/psm1206/DAN.
COHESION: Composite Graph Convolutional Network with Dual-Stage Fusion for Multimodal Recommendation
Authors: Jinfeng Xu, Zheyu Chen, Wei Wang, Xiping Hu, Sang-Wook Kim, Edith C. H. Ngai
Venue: SIGIR
Year: 2025

Recent works in multimodal recommendations, which leverage diverse modal information to address data sparsity and enhance recommendation accuracy, have garnered considerable interest. Two key processes in multimodal recommendations are modality fusion and representation learning. Previous approaches in modality fusion often employ simplistic attentive or pre-defined strategies at early or late stages, failing to effectively handle irrelevant information among modalities. In representation learning, prior research has constructed heterogeneous and homogeneous graph structures encapsulating user-item, user-user, and item-item relationships to better capture user interests and item profiles. Modality fusion and representation learning were considered as two independent processes in previous work. This paper reveals that these two processes are complementary and can support each other. Specifically, powerful representation learning enhances modality fusion, while effective fusion improves representation quality. Stemming from these two processes, we introduce a COmposite grapH convolutional nEtwork with dual-stage fuSION for the multimodal recommendation, named COHESION. Specifically, it introduces a dual-stage fusion strategy to reduce the impact of irrelevant information, refining all modalities using behavior modality in the early stage and fusing their representations at the late stage. It also proposes a composite graph convolutional network that utilizes user-item, user-user, and item-item graphs to extract heterogeneous and homogeneous latent relationships within users and items. Besides, it introduces a novel adaptive optimization to ensure balanced and reasonable representations across modalities. Extensive experiments on three public datasets demonstrate the significant superiority of COHESION over various competitive baselines.