OpenRaiser/Intern-Atlas huggingface dataset:
Dynamic and Parametric Retrieval-Augmented Generation
Authors: Weihang Su, Qingyao Ai, Jingtao Zhan, Qian Dong, Yiqun Liu
Venue: SIGIR
Year: 2025

Retrieval-Augmented Generation (RAG) has become a foundational paradigm for enhancing large language models (LLMs) with external knowledge, playing an important role in modern information retrieval and knowledge-intensive NLP applications. Standard RAG systems typically adopt a static retrieve-then-generate pipeline and rely on in-context knowledge injection, which can be suboptimal for complex tasks that require multihop reasoning, adaptive information access, and deeper integration of external knowledge. Motivated by these limitations, the research community has moved beyond static retrieval and in-context knowledge injection. Among the emerging directions, this tutorial delves into two rapidly growing and complementary research directions on RAG: Dynamic RAG and Parametric RAG. Dynamic RAG explores how LLMs can actively decide when and what to retrieve during generation, enabling real-time adaptation to evolving information needs. Parametric RAG rethinks how the retrieved knowledge should be incorporated, moving from input-level to parameter-level knowledge injection for improved efficiency and effectiveness. This tutorial offers a comprehensive overview of recent advances in both directions. It provides participants with the theoretical foundations and actionable insights needed to build flexible and scalable RAG systems.
MIDI-Zero: A MIDI-driven Self-Supervised Learning Approach for Music Retrieval
Authors: Yuhang Su, Wei Hu, Hongfeng Gao, Fan Zhang
Venue: SIGIR
Year: 2025

Content-based Music Retrieval (CBMR) is a fundamental task in music information retrieval, encompassing sub-tasks including Audio Identification, Audio Matching, and Version Identification. Traditional methods typically analyze audio signals or spectrograms to extract features related to rhythm, melody, harmony, and timbre. However, with the rapid development of Music Transcription and digital music technologies, MIDI representation has emerged as a powerful alternative fo r music analysis. In this paper, we propose MIDI-Zero, a novel self-supervisedlearning framework for CBMR that operates entirely on MIDI representations. Unlike existing approaches, MIDI-Zero requires no external training data; all training data is automatically generated based on predefined task rules, eliminating the need for labeled datasets or external music collections. MIDI-Zero is designed to handle both symbolic music data and audio-based tasks by leveraging Music Transcription models. Its strong robustness ensures effectiveness even with low-quality transcriptions. Extensive experiments demonstrate that MIDI-Zero achieves competitive performance across various CBMR sub-tasks, particularly excelling in Audio Matching. Our approach simplifies the feature extraction process, bridges the gap between audio and symbolic music representations, and offers a versatile and scalable solution for music retrieval.
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Authors: Taoyu Su, Jiawei Sheng, Duohe Ma, Xiaodong Li, Juwei Yue, Mengxiao Song, Yingkai Tang, Tingwen Liu
Venue: SIGIR
Year: 2025

Multi-Modal Entity Alignment (MMEA) aims to retrieve equivalent entities from different Multi-Modal Knowledge Graphs (MMKGs), a critical information retrieval task. Existing studies have explored various fusion paradigms and consistency constraints to improve the alignment of equivalent entities, while overlooking that the visual modality may not always contribute positively. Empirically, entities with low-similarity images usually generate unsatisfactory performance, highlighting the limitation of overly relying on visual features. We believe the model can be biased toward the visual modality, leading to a shortcut image-matching task. To address this, we propose a counterfactual debiasing framework for MMEA, termed CDMEA, which investigates visual modality bias from a causal perspective. Our approach aims to leverage both visual and graph modalities to enhance MMEA while suppressing the direct causal effect of the visual modality on model predictions. By estimating the Total Effect (TE) of both modalities and excluding the Natural Direct Effect (NDE) of the visual modality, we ensure that the model predicts based on the Total Indirect Effect (TIE), effectively utilizing both modalities and reducing visual modality bias. Extensive experiments on 9 benchmark datasets show that CDMEA outperforms 14 state-of-the-art methods, especially in low-similarity, high-noise, and low-resource data scenarios.
JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System
Authors: Weihang Su, Baoqing Yue, Qingyao Ai, Yiran Hu, Jiaqi Li, Changyue Wang, Kaiyuan Zhang, Yueyue Wu, Yiqun Liu
Venue: SIGIR
Year: 2025

This paper introduces JuDGE (Judgment Document Generation Evaluation), a novel benchmark for evaluating the performance of judgment document generation in the Chinese legal system. We define the task as generating a complete legal judgment document from the given factual description of the case. To facilitate this benchmark, we construct a comprehensive dataset consisting of factual descriptions from real legal cases, paired with their corresponding full judgment documents, which serve as the ground truth for evaluating the quality of generated documents. This dataset is further augmented by two external legal corpora that provide additional legal knowledge for the task: one comprising statutes and regulations, and the other consisting of a large collection of past judgment documents. In collaboration with legal professionals, we establish a comprehensive automated evaluation framework to assess the quality of generated judgment documents across various dimensions. We evaluate various baseline approaches, including few-shot in-context learning, fine-tuning, and a multi-source retrieval-augmented generation (RAG) approach, using both general and legal-domain LLMs. The experimental results demonstrate that, while RAG approaches can effectively improve performance in this task, there is still substantial room for further improvement. All the codes and datasets are available at: https://github.com/oneal2000/JuDGE
Are Generative AI Agents Effective Personalized Financial Advisors?
Authors: Takehiro Takayanagi, Kiyoshi Izumi, Javier Sanz-Cruzado, Richard McCreadie, Iadh Ounis
Venue: SIGIR
Year: 2025

Large language model-based agents are becoming increasingly popular as a low-cost mechanism to provide personalized, conversational advice, and have demonstrated impressive capabilities in relatively simple scenarios, such as movie recommendations. But how do these agents perform in complex high-stakes domains, where domain expertise is essential and mistakes carry substantial risk? This paper investigates the effectiveness of LLM-advisors in the finance domain, focusing on three distinct challenges: (1) eliciting user preferences when users themselves may be unsure of their needs (2) providing personalized guidance for diverse investment preferences, and (3) leveraging advisor personality to build relationships and foster trust. Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs. When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Our results show that accurate preference elicitation is key, otherwise, the LLM-advisor has little impact, or can even direct the investor toward unsuitable assets. More worryingly, users appear insensitive to the quality of advice being given, or worse these can have an inverse relationship. Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice.
Bias in Language Models: Interplay of Architecture and Data?
Authors: Mozhgan Talebpour, Yunfei Long, Alba Garcia Seco de Herrera, Shoaib Jameel
Venue: SIGIR
Year: 2025

Pre-trained language models (PLMs), despite showing strong performance, can carry and increase biases, which can limit the development of fair NLP and IR systems. This research investigates the foundational origins of bias within PLMs, moving beyond detection to a detailed analysis of its formation and propagation across diverse architectures. Through a novel attention weight analysis, we reveal distinct attention patterns for biased versus neutral content, offering insights into the internal representations learned by PLMs. Our findings demonstrate a complex interplay between training data and model architecture, revealing that while the transformer's self-attention mechanism amplifies existing biases, the training data plays a crucial role in the initial encoding of bias within the model's representations.
A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization
Authors: Shiyin Tan, Jaeeon Park, Dongyuan Li, Renhe Jiang, Manabu Okumura
Venue: SIGIR
Year: 2025

In the field of multi-document summarization (MDS), transformer-based models have demonstrated remarkable success, yet they suffer an input length limitation. Current methods apply truncation after the retrieval process to fit the context length; however, they heavily depend on manually well-crafted queries, which are impractical to create for each document set for MDS. Additionally, these methods retrieve information at a coarse granularity, leading to the inclusion of irrelevant content. To address these issues, we propose a novel retrieval-based framework that integrates query selection and document ranking and shortening into a unified process. Our approach identifies the most salient elementary discourse units (EDUs) from input documents and utilizes them as latent queries. These queries guide the document ranking by calculating relevance scores. Instead of traditional truncation, our approach filters out irrelevant EDUs to fit the context length, ensuring that only critical information is preserved for summarization. We evaluate our framework on multiple MDS datasets, demonstrating consistent improvements in ROUGE metrics while confirming its scalability and flexibility across diverse model architectures. Additionally, we validate its effectiveness through an in-depth analysis, emphasizing its ability to dynamically select appropriate queries and accurately rank documents based on their relevance scores. These results demonstrate that our framework effectively addresses context-length constraints, establishing it as a robust and reliable solution for MDS.
Boosting Retrieval-Augmented Generation with Generation-Augmented Retrieval: A Co-Training Approach
Authors: Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng
Venue: SIGIR
Year: 2025

Large language models (LLMs) have shown success in knowledge-intensive tasks, including closed-book question answering and entity linking. However, their susceptibility to hallucination undermines their reliability. Retrieval-augmented generation (RAG) partially addresses this issue by combining a retriever to locate relevant documents and a generator to produce responses grounded in the retrieved evidence. Despite its advantages, RAG faces challenges: (i) the structural gap between traditional dense retrievers and autoregressive generators, and (ii) limited generation performance due to insufficient contextual guidance returned by the retriever. To tackle these limitations, we propose MINT, a novel framework that enhances RAG by co-training Retrieval-augMented generatIon and geNeration-augmented reTrieval (GAR). MINT (i) bridges the gap between the retriever and generator using a unified encoder-decoder structure (ii) incorporates an iterative co-training strategy between RAG and GAR, enabling mutual enhancement through pseudo-samples generation, and (iii) introduces three heuristic inference strategies to generate relevant document identifiers and answers. We conduct an empirical study on the KILT benchmark, and MINT is found to yield significant improvements in both retrieval and generation tasks compared with prevailing baselines.
Clarifying Ambiguities: on the Role of Ambiguity Types in Prompting Methods for Clarification Generation
Authors: Anfu Tang, Laure Soulier, Vincent Guigue
Venue: SIGIR
Year: 2025

In information retrieval (IR), providing appropriate clarifications to better understand users' information needs is crucial for building a proactive search-oriented dialogue system. Due to the strong in-context learning ability of large language models (LLMs), recent studies investigate prompting methods to generate clarifications using few-shot or Chain of Thought (CoT) prompts. However, vanilla CoT prompting does not distinguish the characteristics of different information needs, making it difficult to understand how LLMs resolve ambiguities in user queries. In this work, we focus on the concept of ambiguity for clarification, seeking to model and integrate ambiguities in the clarification process. Following the reasoning and acting paradigm, we propose a new prompting scheme Ambiguity Type-Chain of Thought (AT-CoT), which enhances the reasoning abilities of LLMs by limiting CoT to first predict ambiguity types that can be interpreted as actions, then generate clarifications correspondingly. Experiments are conducted on various datasets containing human-annotated clarifying questions to compare AT-CoT with multiple baselines. We also perform user simulation to implicitly measure the quality of generated clarifications under various IR scenarios. Our codes are available at: https://github.com/anfutang/ClarifyingAmbiguities/.
Unlearning for Federated Online Learning to Rank: A Reproducibility Study
Authors: Yiling Tao, Shuyi Wang, Jiaxi Yang, Guido Zuccon
Venue: SIGIR
Year: 2025

This paper reports on findings from a comparative study on the effectiveness and efficiency of federated unlearning strategies within Federated Online Learning to Rank (FOLTR), with specific attention to systematically analysing the unlearning capabilities of methods in a verifiable manner. Federated approaches to ranking of search results have recently garnered attention to address users privacy concerns. In FOLTR, privacy is safeguarded by collaboratively training ranking models across decentralized data sources, preserving individual user data while optimizing search results based on implicit feedback, such as clicks. Recent legislation introduced across numerous countries is establishing the so called ''the right to be forgotten '', according to which services based on machine learning models like those in FOLTR should provide capabilities that allow users to remove their own data from those used to train models. This has sparked the development of unlearning methods, along with evaluation practices to measure whether unlearning of a user data successfully occurred. Current evaluation practices are however often controversial, necessitating the use of multiple metrics for a more comprehensive assessment - but previous proposals of unlearning methods only used single evaluation metrics. This paper addresses this limitation: our study rigorously assesses the effectiveness of unlearning strategies in managing both under-unlearning and over-unlearning scenarios using adapted, and newly proposed evaluation metrics. Thanks to our detailed analysis, we uncover the strengths and limitations of five unlearning strategies, offering valuable insights into optimizing federated unlearning to balance data privacy and system performance within FOLTR. We publicly release our code and complete results at https://github.com/Iris1026/Unlearning-for-FOLTR.git.
Towards Enhanced Agricultural Information Access in Kiswahili: Integrating Knowledge Graphs and Retrieval-Augmented Generation
Authors: Joseph P. Telemala, Neema Nicodemus Lyimo, Anna R. Kimaro, Camilius A. Sanga
Venue: SIGIR
Year: 2025

Access to and consumption of agricultural research findings remains a challenge for Kiswahili-speaking farmers and extension officers in Tanzania due to the predominance of English in agriculture scholarly publications. To address this challenge, the Mkulima repository, a digital collection of over 600 Swahili agricultural publications, was developed at the Sokoine University of Agriculture to provide agriculture knowledge in Kiswahili. However, its current structure limits effective retrieval and accessibility, given the type of its intended audience, smallholder farmers. This work-in-progress aims to improve access to agricultural knowledge in Kiswahili through a hybrid model that integrates a domain-specific Knowledge Graph (KG) with Retrieval-Augmented Generation (RAG), an approach that combines traditional retrieval with generative language models for producing informed answers. The project's findings are aimed to contribute to AI-driven retrieval systems for low-resource languages, with results targeted for submission as a paper to SIGIR 2026.
Assessing Support for the TREC 2024 RAG Track: A Large-Scale Comparative Study of LLM and Human Evaluations
Authors: Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
Venue: SIGIR
Year: 2025

Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing ''ground truth''. A crucial factor in RAG evaluation is ''support'', or whether the information in the cited documents supports the answer. We conducted a comparative study of submissions to the TREC 2024 RAG Track, evaluating an automatic LLM judge (GPT-4o) against human judges for support assessment. We considered two conditions: (1) fully manual assessments from scratch and (2) manual assessments with post-editing of LLM predictions. Our results indicate good agreement between human and GPT-4o predictions. Further analysis of the disagreements shows that an independent human judge correlates better with GPT-4o than a human judge, suggesting that LLM judges can be a reliable alternative for support assessment. We provide a qualitative analysis of human and GPT-4o errors to help guide future evaluations.
Translative Neural Team Recommendation: From Multilabel Classification to Sequence Prediction
Authors: Kap Thang, Hawre Hosseini, Hossein Fani
Venue: SIGIR
Year: 2025

Neural team recommendation has achieved state-of-the-art performance in forming teams of experts whose success in completing complex tasks is almost surely guaranteed. The proposed models frame the problem as a Boolean multilabel classification, mapping the dense vector representations of required skills to the sparse occurrence (multi-hot) vector representation of an optimum subset of experts using multilayer feedforward neural networks. Such approaches, however, suffer from the curse of sparsity in the high-dimensional vector of optimum experts in the output layer. In this paper, we propose to reformulate the team recommendation problem into a sequence prediction task and leverage seq-to-seq models, including transformers, to map an input sequence of the required subset of skills onto an output sequence of the optimum subset of experts. Our experiments on four large-scale datasets from various domains, with distinct distributions of skills in teams, show that the seq-to-seq approach is consistently superior overall in a host of classification and information retrieval metrics. Our codebase is available at https://github.com/fani-lab/OpeNTF/tree/nmt.
System Comparison Using Automated Generation of Relevance Judgements in Multiple Languages
Authors: Paul Thomas, Douglas W. Oard, Eugene Yang, Dawn J. Lawrie, James Mayfield
Venue: SIGIR
Year: 2025

Recent work has shown that Large Language Models (LLMs) can produce relevance judgements for English retrieval that are useful as a basis for system comparison, and they do so at vastly reduced cost compared to human assessors. Using relevance judgements and ranked retrieval runs from the TREC NeuCLIR track, this paper shows that LLMs can also produce reliable assessments in other languages, even when the topic description or the prompt are in a language different from the documents. Results with Chinese, Persian and Russian documents show that although document language affects both agreement with human assessors on graded relevance and on preference ordering among systems, prompt-language and topic-language effects are negligible. This has implications for the design of multilingual test collections, suggesting that prompts and topic descriptions can be developed in any convenient language.
Template-Based Financial Report Generation in Agentic and Decomposed Information Retrieval
Authors: Yong-En Tian, Yu-Chien Tang, Kuang-Da Wang, An-Zi Yen, Wen-Chih Peng
Venue: SIGIR
Year: 2025

Tailoring structured financial reports from companies' earnings releases is crucial for understanding financial performance and has been widely adopted in real-world analytics. However, existing summarization methods often generate broad, high-level summaries, which may lack the precision and detail required for financial reports that typically focus on specific, structured sections. While Large Language Models (LLMs) hold promise, generating reports adhering to predefined multi-section templates remains challenging. This paper investigates two LLM-based approaches popular in industry for generating templated financial reports: an agentic information retrieval (IR) framework and a decomposed IR approach, namely AgenticIR and DecomposedIR. The AgenticIR utilizes collaborative agents prompted with the full template. In contrast, the DecomposedIR approach applies a prompt chaining workflow to break down the template and reframe each section as a query answered by the LLM using the earnings release. To quantitatively assess the generated reports, we evaluated both methods in two scenarios: one using a financial dataset without direct human references, and another with a weather-domain dataset featuring expert-written reports. Experimental results show that while AgenticIR may excel in orchestrating tasks and generating concise reports through agent collaboration, DecomposedIR statistically significantly outperforms AgenticIR approach in providing broader and more detailed coverage in both scenarios, offering reflection on the utilization of the agentic framework in real-world applications.
CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge
Authors: Armin Toroghi, Willis Guo, Scott Sanner
Venue: SIGIR
Year: 2025

The rise of Large Language Models (LLMs) has redefined the AI landscape, particularly due to their ability to encode factual and commonsense knowledge, and their outstanding performance in tasks requiring reasoning. Despite these advances, hallucinations and reasoning errors remain a significant barrier to their deployment in high-stakes settings. In this work, we observe that even the most prominent LLMs, such as OpenAI-o1, suffer from high rates of reasoning errors and hallucinations on tasks requiring commonsense reasoning over obscure, long-tail entities. To investigate this limitation, we present a new dataset for Commonsense reasoning over Long-Tail entities (CoLoTa), that consists of 3,300 queries from question answering and claim verification tasks and covers a diverse range of commonsense reasoning skills. We remark that CoLoTa can also serve as a Knowledge Graph Question Answering (KGQA) dataset since the support of knowledge required to answer its queries is present in the Wikidata knowledge graph. However, as opposed to existing KGQA benchmarks that merely focus on factoid questions, our CoLoTa queries also require commonsense reasoning. Our experiments with strong LLM-based KGQA methodologies indicate their severe inability to answer queries involving commonsense reasoning. Hence, we propose CoLoTa as a novel benchmark for assessing both (i) LLM commonsense reasoning capabilities and their robustness to hallucinations on long-tail entities and (ii) the commonsense reasoning capabilities of KGQA methods.