OpenRaiser/Intern-Atlas huggingface dataset:
Understanding the Effect of Opinion Polarization in Short Video Browsing
Authors: Bangde Du, Ziyi Ye, Zhijing Wu, Monika Jankowska, Qingyao Ai, Yiqun Liu
Venue: SIGIR
Year: 2025

This paper explores the impact of Opinion Polarization (OP) in the increasingly prevalent context of short video browsing, a dominant medium in the contemporary digital landscape with significant influence on public opinion and social dynamics. We investigate the effects of OP on user perceptions and behaviors in short video consumption, and find that traditional user feedback signals, such as like and browsing duration, are not effective in detecting and measuring OP. Recognizing this problem, our study employs Electroencephalogram (EEG) signals as a novel, noninvasive approach to assess the neural processing of perception and cognition related to OP. Our user study reveals that OP notably affects users' sentiments, resulting in measurable changes in brain signals. Furthermore, we demonstrate the potential of using EEG signals to predict users' exposure to polarized short video content. By exploring the relationships between OP, brain signals, and user behavior, our research offers a novel perspective in understanding the dynamics of short video browsing and proposes an innovative method for quantifying the impact of OP in this context.
MINTT: Memory Inductive Transfer for Temporal Graph Neural Networks
Authors: Tanishq Dubey, Sidharth Agarwal, Shubham Gupta, Srikanta Bedathur
Venue: SIGIR
Year: 2025

Interactions between entities are often time-dependent in real-world systems such as e-commerce, social networks, streaming platforms, finance, and healthcare, and are best modeled as temporal interaction graphs. The temporal dimension plays a crucial role in modern recommendation systems, which rely on future link predictions. Temporal Graph Neural Networks (TGNN) have demonstrated state-of-the-art performance in future link prediction tasks for temporal interaction graphs. However, these models often require substantial training data unavailable in real-world settings. A potential solution to data scarcity is model pre-training on semantically related datasets. Unfortunately, transferring the TGNN model from one dataset to another is not trivial, as it contains node-specific memory modules vital for performance, resulting in them being inherently non-transferable. To overcome this limitation, we propose a novel transfer method that effectively utilizes common attributes between source and target datasets by decoupling graph nodes and corresponding attributes via bipartite encoding. This decoupling facilitates the transfer of memories and other inductive biases from source datasets to a target dataset. We evaluate the proposed transfer technique on real-world datasets and establish that it improves the performance of TGNN on the target dataset by 56% compared to the no-transfer methods and 36% over the state-of-the-art baselines in data-scarce settings.
Reproducing NevIR: Negation in Neural Information Retrieval
Authors: Coen van den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, Mohammad Aliannejadi
Venue: SIGIR
Year: 2025

Negation is a fundamental aspect of human communication, yet it remains a challenge for Language Models (LMs) in Information Retrieval (IR). Despite the heavy reliance of modern neural IR systems on LMs, little attention has been given to their handling of negation. In this study, we reproduce and extend the findings of NevIR, a benchmark study that revealed most IR models perform at or below the level of random ranking when dealing with negation. We replicate NevIR's original experiments and evaluate newly developed state-of-the-art IR models. Our findings show that a recently emerging category-listwise Large Language Model (LLM) re-rankers-outperforms other models but still underperforms human performance. Additionally, we leverage ExcluIR, a benchmark dataset designed for exclusionary queries with extensive negation, to assess the generalisability of negation understanding. Our findings suggest that fine-tuning on one dataset does not reliably improve performance on the other, indicating notable differences in their data distributions. Furthermore, we observe that only cross-encoders and listwise LLM re-rankers achieve reasonable performance across both negation tasks.
Query Smarter, Trust Better? Exploring Search Behaviours for Verifying News Accuracy
Authors: David Elsweiler, Samy Ateia, Markus Bink, Gregor Donabauer, Marcos Fernández-Pichel, Alexander Frummet, Udo Kruschwitz, David E. Losada, Bernd Ludwig, Selina Meyer, Noel Pascual-Presa
Venue: SIGIR
Year: 2025

While it is often assumed that searching for information to evaluate misinformation will help identify false claims, recent work suggests that search behaviours can instead reinforce belief in misleading news, particularly when users generate queries using vocabulary from the source articles. Our research explores how different query generation strategies affect news verification and whether the way people search influences the accuracy of their information evaluation. A mixed-methods approach was used, consisting of three parts: (1) an analysis of existing data to understand how search behaviour influences trust in fake news (2) a simulation of query generation strategies using a Large Language Model (LLM) to assess the impact of different query formulations on search result quality, and (3) a user study to examine how 'Boost' interventions in interface design can guide users to adopt more effective query strategies. The results show that search behaviour significantly affects trust in news, with successful searches involving multiple queries and yielding higher-quality results. Queries inspired by different parts of a news article produced search results of varying quality, and weak initial queries improved when reformulated using full SERP information. Although 'Boost' interventions had limited impact, the study suggests that interface design encouraging users to thoroughly review search results can enhance query formulation. This study highlights the importance of query strategies in evaluating news and proposes that interface design can play a key role in promoting more effective search practices, serving as one component of a broader set of interventions to combat misinformation.
CoDIME: A Counterfactual Approach for Dimension Importance Estimation through Click Logs
Authors: Guglielmo Faggioli, Nicola Ferro, Raffaele Perego, Nicola Tonellotto
Venue: SIGIR
Year: 2025

Contextual dense representation models for text marked a shift in text processing, enabling a richer semantic understanding of the text and more effective Information Retrieval. These models project pieces of text into a latent space, describing them in terms of shared latent concepts, which are not explicitly tied to the text's content. Previous work has shown that certain dimensions of such dense text representations can be irrelevant and detrimental to retrieval effectiveness depending on the information need specified in the query. Higher effectiveness can be achieved by performing retrieval within a linear subspace that excludes these dimensions. Dimension IMportance Estimators (DIMEs) are models designed to identify such harmful dimensions, refining the representations of queries and documents to retain only the useful ones. Current DIMEs rely either on pseudo-relevance feedback, which often delivers inconsistent effectiveness, or on explicit relevance feedback, which is challenging to collect. Inspired by counterfactual modelling, we introduce Counterfactual DIMEs (CoDIMEs), designed to leverage noisy implicit feedback to assess the importance of each dimension. The CoDIME framework presented here approximates the relationship between a document's click frequency and its interaction with a given query dimension through a linear model. Empirical evaluations demonstrate that CoDIME outperforms traditional pseudo-relevance feedback-based DIMEs and surpasses other unsupervised counterfactual methods that utilize implicit feedback.
ADORE: Autonomous Domain-Oriented Relevance Engine for E-commerce
Authors: Zheng Fang, Donghao Xie, Ming Pang, Chunyuan Yuan, Xue Jiang, Changping Peng, Zhangang Lin, Zheng Luo
Venue: SIGIR
Year: 2025

Relevance modeling in e-commerce search remains challenged by semantic gaps in term-matching methods (e.g., BM25) and neural models' reliance on the scarcity of domain-specific hard samples. We propose ADORE, a self-sustaining framework that synergizes three innovations: (1) A Rule-aware Relevance Discrimination module, where a Chain-of-Thought LLM generates intent-aligned training data, refined via Kahneman-Tversky Optimization (KTO) to align with user behavior; (2) An Error-type-aware Data Synthesis module that auto-generates adversarial examples to harden robustness; and (3) A Key-attribute-enhanced Knowledge Distillation module that injects domain-specific attribute hierarchies into a deployable student model. ADORE automates annotation, adversarial generation, and distillation, overcoming data scarcity while enhancing reasoning. Large-scale experiments and online A/B testing verify the effectiveness of ADORE. The framework establishes a new paradigm for resource-efficient, cognitively aligned relevance modeling in industrial applications.
Does UMBRELA work on other LLMs?
Authors: Naghmeh Farzi, Laura Dietz
Venue: SIGIR
Year: 2025

We reproduce the UMBRELA LLM Judge evaluation framework across a range of large language models (LLMs) to assess its generalizability beyond the original study. Our investigation evaluates how LLM choice affects relevance assessment accuracy, focusing on leaderboard rank correlation and per-label agreement metrics. Results demonstrate that UMBRELA with DeepSeek V3 obtains very comparable performance to GPT-4o (used in original work). For LLaMA-3.3-70B we obtain slightly lower performance, which further degrades with smaller LLMs.
A Comparative Study of Large Language Models and Traditional Privacy Measures to Evaluate Query Obfuscation Approaches
Authors: Francesco Luigi De Faveri, Guglielmo Faggioli, Nicola Ferro
Venue: SIGIR
Year: 2025

When interacting with an Information Retrieval (IR) system, users might disclose personal information, such as medical details, through their queries. Thus, assessing the level of privacy granted to users when querying an IR system is essential to determine the confidentiality of submitted sensitive data. Query obfuscation protocols have traditionally been employed to obscure a user's real information need when retrieving documents. In these protocols, the query is modified employing ε-Differential Privacy (DP) obfuscation mechanisms, which alter query terms according to a predefined privacy budget ε. While this budget ensures formal mathematical guarantees, it provides only limited guarantees of the privacy experienced by the user and calls for empirical privacy evaluation to be carried out. Such privacy assessments employ lexical and semantic similarity measures between the original and obfuscated queries. In this study, we explore the role of Large Language Models (LLMs) in privacy evaluation, simulating a scenario where users employ such models to determine whether their input has been effectively privatized. Our primary research objective is to determine whether LLMs provide a novel perspective on privacy estimation and if their assessments serve as a proxy for traditional similarity metrics, such as the Jaccard and cosine similarity derived from Transformer-based sentence embeddings. Our findings reveal a positive correlation between LLMs-generated privacy scores and cosine similarity computed using different Transformer architectures. This suggests that LLM assessments act as a proxy for similarity-based measures.
Evaluating Multi-Dimensional Cumulated Utility in Information Retrieval
Authors: Francesco Luigi De Faveri, Guglielmo Faggioli, Nicola Ferro, Kalervo Järvelin
Venue: SIGIR
Year: 2025

Traditional Information Retrieval (IR) effectiveness metrics assume that a relevant document satisfies the information need as a whole. Nevertheless, if the information need is faceted or contains subtopics, this notion of relevance cannot model documents relevant only to one or a few subtopics. Furthermore, faceted documents in a ranked list may focus on the same subtopics, and their content may overlap while neglecting other subtopics. Hence, a search result, where topranked documents deal with different subtopics should be preferred over a result where documents are thematically limited and provide overlapping information. The Multi-Dimensional Cumulated Utility (MDCU) metric, recently formulated theoretically by Järvelin and Sormunen, extends the evaluation of novelty and diversity by considering content overlapping among documents. While Järvelin and Sormunen described the theory of MDCU and illustrated its application on a toy example, they did not investigate its empirical use. In this paper, we show the practical feasibility and validity of the MDCU by applying it to publicly available TREC test collections. Furthermore, we analyse its relation with the well-established α-nDCG, and finally, we provide a Python implementation of the MDCU, fostering its adoption as an evaluation framework. Our results indicate a positive correlation between α-nDCG and MDCU, suggesting that both measures correctly identify similar trends when evaluating the IR systems. Finally, compared to α-nDCG, MDCU exhibits a stronger statistical power and identifies up to 9 times more statistically significantly different pairs of systems.
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
Authors: Naihe Feng, Yi Sui, Shiyi Hou, Jesse C. Cresswell, Ga Wu
Venue: SIGIR
Year: 2025

Existing research on Retrieval-Augmented Generation (RAG) primarily focuses on improving overall question-answering accuracy, often overlooking the quality of sub-claims within generated responses. Recent methods that attempt to improve RAG trustworthiness, such as through auto-evaluation metrics, lack probabilistic guarantees or require ground truth answers. To address these limitations, we propose Conformal-RAG, a novel framework inspired by recent applications of conformal prediction (CP) on large language models (LLMs). Conformal-RAG leverages CP and internal information from the RAG mechanism to offer statistical guarantees on response quality. It ensures group-conditional coverage spanning multiple sub-domains without requiring manual labelling of conformal sets, making it suitable for complex RAG applications. Compared to existing RAG auto-evaluation methods, Conformal-RAG offers statistical guarantees on the quality of refined sub-claims, ensuring response reliability without the need for ground truth answers. Additionally, our experiments demonstrate that by leveraging information from the RAG system, Conformal-RAG retains up to 60% more high-quality sub-claims from the response compared to direct applications of CP to LLMs, while maintaining the same reliability guarantee.
MGIPF: Multi-Granularity Interest Prediction Framework for Personalized Recommendation
Authors: Ruoxuan Feng, Zhen Tian, Qiushi Peng, Jiaxin Mao, Wayne Xin Zhao, Di Hu, Changwang Zhang
Venue: SIGIR
Year: 2025

Personalized recommender systems, which focus on predicting users' interests, have significantly enhanced user experiences across diverse applications. However, existing approaches implicitly model users' preferences through fitting the fine-grained labels (e.g., click labels), but often neglecting the coarse-grained interest information inherent in the inputs themselves. Relying solely on the fine-grained labels could bring negative impact on interest modeling and limit the performance, as the labels may carry inevitable noise in real-world scenarios. In addition, it is considerably demanding in terms of data for most existing approaches to effectively model users' multi-granularity interests with limited or no supporting examples, resulting in subpar performance due to the significant long-tail phenomenon. To tackle these issues, we propose a novel learning framework named the Multi-Granularity Interest Prediction Framework (MGIPF), for better modeling users' diverse interests. Unlike prior work, our key idea is to utilize both the coarse-grained and fine-grained interests for supervising the training of models. Specifically, we introduce a pseudo-labeling approach explicitly mining users' potential multi-granularity interests from the raw data, and propose coarse-grained interest prediction modules that collaborate to utilize the multi-granularity supervision signals to enhance the learning of low-frequency items. The corresponding coarse-grained losses are softly weighted, taking into account the varying confidence of potential multi-granularity preferences on positive and negative samples. Importantly, our framework is lightweight and adaptable, capable of being applied effectively to mainstream recommendation models, establishing a comprehensive end-to-end training process. Extensive experiments conducted on three publicly available datasets have demonstrated the efficacy of our approach. The code is available at https://github.com/GeWu-Lab/MGIPF.
RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptation
Authors: William Fleshman, Benjamin Van Durme
Venue: SIGIR
Year: 2025

Large language models (LLMs) fine-tuned for text-retrieval have demonstrated state-of-the-art results across several information retrieval (IR) benchmarks. However, supervised training for improving these models requires numerous labeled examples, which are generally unavailable or expensive to acquire. In this work, we explore the effectiveness of extending reverse engineered adaptation to the context of information retrieval (RE-AdaptIR). We use RE-AdaptIR to improve LLM-based IR models using only unlabeled data. We demonstrate improved performance in both training domains and in zero-shot domains where the models have seen no queries. We analyze performance changes in various fine-tuning scenarios and offer findings of immediate use to IR practitioners.
Optimizing Tail-Head Trade-off for Extreme Multi-Label Text Classification (XMTC) with RAG-Labels and a Dynamic Two-Stage Retrieval and Fusion Pipeline
Authors: Celso França, Gestefane Rabbi, Thiago Salles, Washington Cunha, Leonardo Rocha, Marcos André Gonçalves
Venue: SIGIR
Year: 2025

We tackle Extreme Multi-Label Text Classification (XMTC), which involves assigning relevant labels to texts from a huge label space. Attempting to optimize the underexplored tail-head trade-off, we address the XMTC task through its core challenges of volume, skewness, and quality by proposing xCoRetriev, a novel two-stage retrieving and fusing ranking pipeline. Our pipeline addresses the volume challenge by dynamically slicing the large label space; it also tackles the skewness challenge by favoring the tail labels while fusing sparse and dense retrievers. Finally, xCoRetriev faces the quality challenge by enhancing the label space with Retrieval-Augmented Generated (RAG)-labels. Our experiments with four XMTC benchmarks with hundreds of thousands of text documents and labels against six state-of-the-art XMTC baselines demonstrate xCoRetriev's strengths in terms of: (i)~scalability for large label spaces, being among the most efficient methods at training and prediction; (ii)~effectiveness in the face of high skewness, with gains of up to 48% in propensity-scored metrics against the best state-of-the-art baselines; and (iii)~capability of handling very noisy datasets by exploiting RAG-labels.
Digital Health
Authors: Ophir Frieder
Venue: SIGIR
Year: 2025

The origins of digital health are open to interpretation. If visualization of internal human anatomy constitutes digitization, then the dawn of digitized health can be traced to Röntgen's X-ray discovery in 1895. Another interpretation relies on electronic medical records, where the origin rests with home-grown systems developed at various medical centers in the 1960's. Much has changed since those early days. Technology has advanced tremendously, as has its role in healthcare. Today, digital health extends far beyond imaging and electronic medical records. It now involves collecting data from a diverse array of sensors and sources, applying privacy-preserving methods for gathering and sharing data and knowledge for both clinical and research purposes, and integrating, presenting, and interpreting health information. Digital health also encompasses treatment recommendation systems and much more. Succinctly stated, technology from our community is improving daily patient care. In the United States, and similarly in other parts of the world, patients interact with the medical community via interconnected entities, namely: medical call centers, nursing staff, and medical doctors of various specialty. Using search, recommendation, and language technology, we address critical resource (staffing and/or equipment) shortages in each of the aforementioned entities either via staff augmentation or simply providing a ''second opinion'' to overburdened clinicians.
Large Language Model Relevance Assessors Agree With One Another More Than With Human Assessors
Authors: Maik Fröbe, Andrew Parry, Ferdinand Schlatt, Sean MacAvaney, Benno Stein, Martin Potthast, Matthias Hagen
Venue: SIGIR
Year: 2025

Relevance judgments can differ between assessors, but previous work has shown that such disagreements have little impact on the effectiveness rankings of retrieval systems. This applies to disagreements between humans as well as between human and large language model (LLM) assessors. However, the agreement between different LLM~assessors has not yet been systematically investigated. To close this gap, we compare eight LLM~assessors on the TREC DL tracks and the retrieval task of the RAG track with each other and with human assessors. We find that the agreement between LLM~assessors is higher than between LLMs and humans and, importantly, that LLM~assessors favor retrieval systems that use LLMs in their ranking decisions: our analyses with 30-50 retrieval systems show that the system rankings obtained by LLM~assessors overestimate LLM-based re-rankers by 9~to 17~positions on average.