OpenRaiser/Intern-Atlas huggingface dataset:
Insight Agents: An LLM-Based Multi-Agent System for Data Insights
Authors: Jincheng Bai, Zhenyu Zhang, Jennifer Zhang, Jason Zhu
Venue: SIGIR
Year: 2025

Today, E-commerce sellers face several key challenges, including difficulties in discovering and effectively utilizing available programs and tools, and struggling to understand and utilize rich data from various tools. We therefore aim to develop Insight Agents (IA), a conversational multi-agent Data Insight system, to provide E-commerce sellers with personalized data and business insights through automated information retrieval. Our hypothesis is that IA will serve as a force multiplier for sellers, thereby driving incremental seller adoption by reducing the effort required and increase speed at which sellers make good business decisions. In this paper, we introduce this new LLM-backed end-to-end agentic workflow designed for comprehensive coverage, high accuracy, and low latency. It features a hierarchical multi-agent structure, consisting of manager agent and two worker agents: data presentation and insight generation, for efficient information retrieval and problem-solving. We design a simple yet effective ML solution for manager agent that combines Out-of-Domain (OOD) detection using a lightweight encoder-decoder model and agent routing through a BERT-based classifier, optimizing both accuracy and latency. Within the two worker agents, a strategic planning is designed for API-based data model that breaks down queries into granular components to generate more accurate responses, and domain knowledge is dynamically injected to to enhance the insight generator. IA has been launched for Amazon sellers in US, which has achieved high accuracy of 89.5% based on human evaluation, with latency of P90 below 15s.
Refining Fidelity Metrics for Explainable Recommendations
Authors: Mikhail Baklanov, Veronika Bogina, Yehonatan Elisha, Yahlly Schein, Liron I. Allerhand, Oren Barkan, Noam Koenigstein
Venue: SIGIR
Year: 2025

Counterfactual evaluation provides a promising framework for assessing explanation fidelity in recommender systems, but per- turbation metrics adapted from computer vision suffer three key limitations: (1) they conflate explaining and contradictory features (2) they average over entire user histories instead of prioritizing concise, high-impact explanations, and (3) they use fixed-percentage perturbations, leading to inconsistencies across users. We introduce refined counterfactual metrics that focus on the most relevant explaining features, exclude contradictory elements, and assess fidelity at a fixed explanation length, ensuring a more consistent and interpretable evaluation. Our code is at: https:// github.com/DeltaLabTLV/FidelityMetrics4XRec
Theory and Toolkits for User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation
Authors: Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, ChengXiang Zhai
Venue: SIGIR
Year: 2025

Interactive AI systems, including search engines, recommender systems, conversational agents, and generative AI applications, are increasingly central to user experiences. However, rigorously evaluating their performance, training them effectively with interaction data, and modeling user behavior for personalization remain significant challenges, often difficult to address reproducibly and at scale. User simulation, which employs intelligent agents to mimic human interaction patterns, offers a powerful and versatile methodology to tackle these interconnected issues. This half-day tutorial provides a comprehensive overview of modern user simulation techniques for interactive AI systems. We will explore the theoretical foundations and practical applications of simulation for system evaluation, algorithm training, and user modeling, emphasizing the crucial connections between these uses. The tutorial covers key simulation methodologies, with a particular focus on recent advancements leveraging large language models, discussing both the opportunities they present and the open challenges they entail. Crucially, we will also provide practical guidance, highlighting relevant toolkits, libraries, and datasets available to researchers and practitioners.
Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
Authors: Krisztian Balog, Don Metzler, Zhen Qin
Venue: SIGIR
Year: 2025

Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation. This widespread adoption necessitates a critical examination of potential biases arising from the interplay between these LLM-based components. This paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. Furthermore, we observe limitations in LLM judges' ability to discern subtle system performance differences. Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content. These results highlight the need for a more holistic view of the LLM-driven information ecosystem. To this end, we offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.
SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Data Generation for Personalized Tourism Recommenders
Authors: Ashmi Banerjee, Adithi Satish, Fitri Nur Aisyah, Wolfgang Wörndl, Yashar Deldjoo
Venue: SIGIR
Year: 2025

Tourism Recommender Systems (TRS) are crucial in personalizing travel experiences by tailoring recommendations to users' preferences, constraints, and contextual factors. However, publicly available travel datasets often lack sufficient breadth and depth, limiting their ability to support advanced personalization strategies - particularly for sustainable travel and off-peak tourism. In this work, we explore using Large Language Models (LLMs) to generate synthetic travel queries that emulate diverse user personas and incorporate structured filters such as budget constraints and sustainability preferences. This paper introduces a novel SynthTRIPs framework for generating synthetic travel queries using LLMs grounded in a curated knowledge base (KB). Our approach combines persona-based preferences (e.g., budget, travel style) with explicit sustainability filters (e.g., walkability, air quality) to produce realistic and diverse queries. We mitigate hallucination and ensure factual correctness by grounding the LLM responses in the KB. We formalize the query generation process and introduce evaluation metrics for assessing realism and alignment. Both human expert evaluations and automatic LLM-based assessments demonstrate the effectiveness of our synthetic dataset in capturing complex personalization aspects underrepresented in existing datasets. While our framework was developed and tested for personalized city trip recommendations, the methodology applies to other recommender system domains. Code and dataset are made public at https://bit.ly/synthTRIPs
GRAIN: Group-Reinforced Adaptive Interaction Network for Cold-Start CTR Prediction in E-commerce Search
Authors: Wei Bao, Hao Chen, Bang Lin, Tao Zhang, Chengfu Huo
Venue: SIGIR
Year: 2025

Accurate prediction of click-through rates (CTR) for cold-start entities (CSEs) within search engine ecosystems presents significant challenges. Notably, CSEs encompass novel users/items and new session search queries, each characterized by their limited interaction data and poor-quality embeddings, which collectively contribute to the complexity of CTR estimation.Existing studies predominantly address cold-start challenges in isolation, such as focusing separately on new users or new items, and lack a comprehensive framework to effectively integrate atomic ID features with group-level representations. To address these limitations, we propose GRAIN (Group Reinforced Adaptive Interaction Network), a novel framework that enhances CTR prediction across all maturity phases, namely Cold-Start, Warm-Up, and Common. GRAIN consists of three key components: 1) a Graph-based Id-to-Cluster (GIC) module that aggregates atomic ID features into cluster-level representations; 2) an ID-Cluster Cross (ICC) module that aligns ID-level and cluster-level features through contrastive learning and cross-grained interaction mechanism; 3) a lightweight auxiliary task that classifies entities into different maturity stages using a data-driven phase partitioning algorithm. Extensive experiments demonstrate GRAIN's effectiveness in improving CTR prediction accuracy across multiple maturity phases. GRAIN has been successfully deployed on the 1688 App, handling billions of daily requests.
Automatic Document Editing for Improved Ranking
Authors: Niv Bardas, Tommy Mordo, Oren Kurland, Moshe Tennenholtz
Venue: SIGIR
Year: 2025

We present a study of using large language models (LLMs) to modify a document so as to have it highly ranked for a query by an undisclosed ranking function. We present different prompting methods inspired by work on using LLMs to induce ranking. Empirical evaluation attests to the merits of the best performing methods with respect to human modifications and a highly effective feature-based modification method.
Combining Evidence and Reasoning for Biomedical Fact-Checking
Authors: Mariano Barone, Antonio Romano, Giuseppe Riccio, Marco Postiglione, Vincenzo Moscato
Venue: SIGIR
Year: 2025

Misinformation in healthcare, from vaccine hesitancy to unproven treatments, poses risks to public health and trust in medical systems. While machine learning and natural language processing have advanced automated fact-checking, validating biomedical claims remains uniquely challenging due to complex terminology, the need for domain expertise, and the critical importance of grounding in scientific evidence. We introduce CER (Combining Evidence and Reasoning), a novel framework for biomedical fact-checking that integrates scientific evidence retrieval, reasoning via large language models, and supervised veracity prediction. By integrating the text-generation capabilities of large language models with advanced retrieval techniques for high-quality biomedical scientific evidence, CER effectively mitigates the risk of hallucinations, ensuring that generated outputs are grounded in verifiable, evidence-based sources. Evaluations on expert-annotated datasets (HealthFC, BioASQ-7b, SciFact) demonstrate state-of-the-art performance and promising cross-dataset generalization. Code and data are released for transparency and reproducibility: https://github.com/PRAISELab-PicusLab/CER
Combating Biomedical Misinformation through Multi-modal Claim Detection and Evidence-based Verification
Authors: Mariano Barone, Antonio Romano, Giuseppe Riccio, Marco Postiglione, Vincenzo Moscato
Venue: SIGIR
Year: 2025

Misinformation in healthcare, from vaccine hesitancy to unproven treatments, poses risks to public health and trust in medical systems. While machine learning and natural language processing have advanced automated fact-checking, validating biomedical claims remains uniquely challenging due to complex terminology, the need for domain expertise, and the critical importance of grounding in scientific evidence. We introduce CER (Combining Evidence and Reasoning), a novel framework for biomedical fact-checking that integrates scientific evidence retrieval, reasoning via large language models, and supervised veracity prediction. By integrating the text-generation capabilities of large language models with advanced retrieval techniques for high-quality biomedical scientific evidence, CER effectively mitigates the risk of hallucinations, ensuring that generated outputs are grounded in verifiable, evidence-based sources. Evaluations on expert-annotated datasets (HealthFC, BioASQ-7b, SciFact) demonstrate state-of-the-art performance and promising cross-dataset generalization. Code and data are released for transparency and reproducibility: https://github.com/PRAISELab-PicusLab/CER
Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs
Authors: Julia Belikova, Konstantin Polev, Rauf Parchiev, Dmitry Simakov
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly deployed in industry applications, yet their reliability remains hampered by challenges in detecting hallucinations. While supervised state-of-the-art (SOTA) methods that leverage LLM hidden states-such as activation tracing and representation analysis-show promise, their dependence on extensively annotated datasets limits scalability in real-world applications. This paper addresses the critical bottleneck of data annotation by investigating the feasibility of reducing training data requirements for two SOTA hallucination detection frameworks: Lookback Lens, which analyzes attention head dynamics, and probing-based approaches, which decode internal model representations. We propose a methodology combining efficient classification algorithms with dimensionality reduction techniques to minimize sample size demands while maintaining competitive performance. Evaluations on standardized question-answering RAG benchmarks show that our approach achieves performance comparable to strong proprietary LLM-based baselines with only 250 training samples. These results highlight the potential of lightweight, data-efficient paradigms for industrial deployment, particularly in annotation-constrained scenarios.
Augmenting Vision-Language Retrieval: The Role of Multimodal LLMs as Synthetic Data Generators
Authors: Aidan Bell, James Gore, Behrooz Mansouri
Venue: SIGIR
Year: 2025

Multimodal Large Language Models (MLLMs) connect and interpret different data types, making them suitable for various vision-language tasks. Despite the rapid advancements in MLLMs, their effectiveness for specialized cross-modal retrieval tasks remains underexplored. A challenging example is art retrieval, where the task is to find visually and conceptually relevant artwork corresponding to a textual description. This paper investigates the effects of fine-tuning cross-modal retrieval models using both human-annotated and MLLM-generated captions for artistic paintings. To this end, two cross-modal retrieval models, Long-CLIP and BLIP, are studied. Experimental results show that models fine-tuned on MLLM-generated captions achieve search effectiveness comparable to those fine-tuned on human-annotated captions.
International Workshop on Algorithmic Bias in Search and Recommendation (BIAS 2025)
Authors: Alejandro Bellogín, Ludovico Boratto, Styliani Kleanthous, Elisabeth Lex, Francesca Maridina Malloci, Mirko Marras
Venue: SIGIR
Year: 2025

Designing search and recommendation models that are both efficient and effective has long been a central objective for both industry professionals and academic researchers. Yet, growing evidence highlights how models trained on historical data can reinforce pre-existing biases, potentially leading to harmful outcomes. Addressing these challenges by defining, evaluating, and mitigating bias across development workflows is a crucial step toward the responsible deployment of search and recommendation models in practice. The BIAS 2025 workshop seeks to gather innovative research and foster a shared space for dialogue among researchers and practitioners committed to advancing this fundamental direction. Workshop website: https://biasinrecsys.github.io/sigir2025/.
LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data
Authors: Grigor Bezirganyan, Sana Sellami, Laure Berti-Équille, Sébastien Fournier
Venue: SIGIR
Year: 2025

Multimodal Deep Learning enhances decision-making by integrating diverse information sources, such as texts, images, audio, and videos. To develop trustworthy multimodal approaches, it is essential to understand how uncertainty impacts these models. We propose LUMA, a unique multimodal dataset, featuring audio, image, and textual data from 50 classes, specifically designed for learning from uncertain data. It extends the well-known CIFAR 10/100 dataset with audio samples extracted from three audio corpora, and text data generated using the Gemma-7B Large Language Model (LLM). The LUMA dataset enables the controlled injection of varying types and degrees of uncertainty to achieve and tailor specific experiments and benchmarking initiatives. LUMA is also available as a Python package including the functions for generating multiple variants of the dataset with controlling the diversity of the data, the amount of noise for each modality, and adding out-of-distribution samples. A baseline pre-trained model is also provided alongside three uncertainty quantification methods: Monte-Carlo Dropout, Deep Ensemble, and Reliable Conflictive Multi-View Learning. This comprehensive dataset and its tools are intended to promote and support the development, evaluation, and benchmarking of trustworthy and robust multimodal deep learning approaches. We anticipate that the LUMA dataset will help the research community to design more trustworthy and robust machine learning approaches for safety critical applications. The code and instructions for downloading and processing the dataset can be found at: https://github.com/bezirganyan/LUMA.
IGP: Efficient Multi-Vector Retrieval via Proximity Graph Index
Authors: Zheng Bian, Man Lung Yiu, Bo Tang
Venue: SIGIR
Year: 2025

Neural embedding models are extensively employed in retrieval applications, including passage retrieval, question answering, and web search. In particular, multi-vector models (e.g., ColBERTv2), which represent a document as multiple embedding vectors, have been demonstrated to achieve superior retrieval quality. Nevertheless, these models incur significant overhead at the retrieval time due to the massive amount of embedding vectors. Several promising proposals (e.g., PLAID, DESSERT, EMVB, and MUVERA) have been made to optimize the query latency. To yield high recall, these methods need to generate a considerable amount (e.g., ten thousands) of document candidates, rendering both the candidate generation phase and the refinement phase inefficient. In this paper, we propose a high-quality candidate generation technique that produces only hundreds of candidates yet achieves high recall. Specifically, we develop an incremental next-similar retrieval technique for a proximity graph index in order to facilitate high-quality candidate generation. Our experiments on real datasets show that our proposed method IGP achieves 2x-3x query throughput compared to existing methods at the same accuracy level.
Fair Access to Food Data in Africa: An Approach Based on Retrieval-Augmented Generation
Authors: Jean Petit Bikim, Charles Loic Njiosseu, Emmanuel Leuna Fienkak, Azanzi Jiomekong, Sören Auer
Venue: SIGIR
Year: 2025

In this paper, we propose a Retrieval-Augmented Generation-based (RAG) approach for fair access to food data in developing countries. This work also contributes to achieving sustainable development goals, specifically goal 2: Zero hunger and goal 3: Ensure healthy lives and promote well-being for all at all ages. Actually, given that a lot of African food data is accessible only in PDF format, we firstly extracted and organized these data using the Open Research Knowledge Graph (ORKG) and the image data using the Firebase database, with the link to the corresponding food description in the ORKG. Currently, more than 1000 foods eaten in around 50 African countries are already documented. Given that Large Language Models necessitate a lot of data and resources to be train/fine-tuned, which we do not have in our university, our idea is to use the food data stored in the ORKG to build a Retrieval-Augmented Generation system (RAG) for improving access to food data in Africa. Its implementation involved the use of multi-embedding models, vector databases (Pinecone), Falcon3 and LangChain. Because we do not have enough resources, this work is made possible thanks to a collaboration with TIB-Hannover who provide access to remote computers.
Small Data, Big Impact: Navigating Resource Limitations in Point-of-Interest Recommendation for Individuals with Autism
Authors: Ludovico Boratto, Federica Cena, Mirko Marras, Noemi Mauro, Giacomo Medda
Venue: SIGIR
Year: 2025

Autism Spectrum Disorder (ASD) affects sensory perception, making spatial exploration difficult. Recommender systems can assist ASD users by suggesting Points of Interest (POIs) aligned with their sensory preferences. However, demographic constraints, difficulties in engaging ASD users, and the complexity of obtaining sensory data position POI recommendation for ASD people as a low-resource problem. In this paper, we identify key challenges in developing such systems and present our ongoing efforts. Using a local ASD center as a use case, we are developing a structured user involvement protocol. From the limited data, we are deriving knowledge graphs (KGs) to model preferences and sensory aspects. We are then exploring KG-based techniques to generate paths from users to POIs to suggest. With psychologists, we are refining the paths structure to match varying complexity levels and translate them into natural language accessible for people with ASD.
Multilingual Evaluation of Main Content Extractors for Web Pages
Authors: Aurélien Bournonville, Gaël Dias, Thomas Largillier, Emmanuel Marchand, Fabrice Maurel, Guillaume Pitel, François Rioult
Venue: SIGIR
Year: 2025

Tools designed to extract main content from web pages require thorough evaluation, yet existing benchmarks disproportionately focus on English-language datasets. Consequently, previous studies have shown that while these extractors are well-optimized for English, their effectiveness partially or entirely diminishes in other languages. This study reproduces and extends recent benchmarks by incorporating multilingual datasets as a key factor. We analyze extractor performance across five languages-Greek, English, Polish, Russian, and Chinese-highlighting the need to adapt extraction models to linguistic variations. Our results show that while some extractors maintain stable performance, others suffer significant drops in precision and recall on non-English or structurally irregular pages.
A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking
Authors: Henrik Brådland, Morten Goodwin, Per-Arne Andersen, Alexander Salveson Nossum, Aditya Gupta
Venue: SIGIR
Year: 2025

Document chunking fundamentally impacts Retrieval-Augmented Generation (RAG) by determining how source materials are segmented before indexing. Despite evidence that Large Language Models (LLMs) are sensitive to the layout and structure of retrieved data, there is currently no framework to analyze the impact of different chunking methods. In this paper, we introduce a novel methodology that defines essential characteristics of the chunking process at three levels: intrinsic passage properties, extrinsic passage properties, and passages-document coherence. We propose HOPE (Holistic Passage Evaluation), a domain-agnostic, automatic evaluation metric that quantifies and aggregates these characteristics. Our empirical evaluations across seven domains demonstrate that the HOPE metric correlates significantly (p > 0.13) with various RAG performance indicators, revealing contrasts between the importance of extrinsic and intrinsic properties of passages. Semantic independence between passages proves essential for system performance with a performance gain of up to 56.2% in factual correctness and 21.1% in answer correctness. On the contrary, traditional assumptions about maintaining concept unity within passages show minimal impact. These findings provide actionable insights for optimizing chunking strategies, thus improving RAG system design to produce more factually correct responses.