OpenRaiser/Intern-Atlas huggingface dataset:
Wrong Answers Can Also Be Useful: PlausibleQA - A Large-Scale QA Dataset with Answer Plausibility Scores
Authors: Jamshid Mozafari, Abdelrahman Abdallah, Bhawna Piryani, Adam Jatowt
Venue: SIGIR
Year: 2025

Large Language Models (LLMs) are revolutionizing information retrieval, with chatbots becoming an important source for answering user queries. As by their design, LLMs prioritize generating correct answers, the value of highly plausible yet incorrect answers (candidate answers) tends to be overlooked. However, such answers can still prove useful, for example, they can play a crucial role in tasks like Multiple-Choice Question Answering (MCQA) and QA Robustness Assessment (QARA). Existing QA datasets primarily focus on correct answers without explicit consideration of the plausibility of other candidate answers, limiting opportunity for more nuanced evaluations of models. To address this gap, we introduce PlausibleQA, a large-scale dataset comprising 10,000 questions and 100,000 candidate answers, each annotated with plausibility scores and justifications for their selection. Additionally, the dataset includes 900,000 justifications for pairwise comparisons between candidate answers, further refining plausibility assessments. We evaluate PlausibleQA through human assessments and empirical experiments, demonstrating its utility in MCQA and QARA analysis. Our findings show that plausibility-aware approaches are effective for MCQA distractor generation and QARA. We release PlausibleQA as a resource for advancing QA research and enhancing LLM performance in distinguishing plausible distractors from correct answers.
WikiHint: A Human-Annotated Dataset for Hint Ranking and Generation
Authors: Jamshid Mozafari, Florian Gerhold, Adam Jatowt
Venue: SIGIR
Year: 2025

The use of Large Language Models (LLMs) has increased significantly with users frequently asking questions to chatbots. In the time when information is readily accessible, it is crucial to stimulate and preserve human cognitive abilities and maintain strong reasoning skills. This paper addresses such challenges by promoting the use of hints as an alternative or a supplement to direct answers. We first introduce a manually constructed hint dataset, WikiHint, which is based on Wikipedia and includes 5,000 hints created for 1,000 questions. We then finetune open-source LLMs for hint generation in answer-aware and answer-agnostic contexts. We assess the effectiveness of the hints with human participants who answer questions with and without the aid of hints. Additionally, we introduce a lightweight evaluation method, HintRank, to evaluate and rank hints in both answer-aware and answer-agnostic settings. Our findings show that (a) the dataset helps generate more effective hints (b) including answer information along with questions generally improves the quality of generated hints, and (c) encoder-based models perform better than decoder-based models in hint ranking.
Efficient Conversational Search via Topical Locality in Dense Retrieval
Authors: Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Guido Rocchietti, Cosimo Rulli
Venue: SIGIR
Year: 2025

Pre-trained language models have been widely exploited to learn dense representations of documents and queries for information retrieval. While previous efforts have primarily focused on improving effectiveness and user satisfaction, response time remains a critical bottleneck of conversational search systems. To address this, we exploit the topical locality inherent in conversational queries, i.e., the tendency of queries within a conversation to focus on related topics. By leveraging query embedding similarities, we dynamically restrict the search space to semantically relevant document clusters, reducing computational complexity without compromising retrieval quality. We evaluate our approach on the TREC CAsT, 2019 and 2020 datasets using multiple embedding models and vector indexes, achieving improvements in processing speed of up to 10.3X with little loss in performance (4.3X without any loss). Our results show that the proposed system effectively handles complex, multi-turn queries with high precision and efficiency, offering a practical solution for real-time conversational search.
Bridging Personalization and Control in Scientific Personalized Search
Authors: Sheshera Mysore, Garima Dhanania, Kishor Patil, Surya Kallumadi, Andrew McCallum, Hamed Zamani
Venue: SIGIR
Year: 2025

Personalized search is a problem where models benefit from learning user preferences from per-user historical interaction data. The inferred preferences enable personalized ranking models to improve the relevance of documents to users. However, personalization is also seen as opaque in its use of historical interactions and is not amenable to users' control. Further, personalization limits the diversity of information users are exposed to. While search results may be automatically diversified this does little to address the lack of control over personalization. In response, we introduce a model for personalized search that enables users to control personalized rankings proactively. Our model, CtrlCE, is a novel cross-encoder model augmented with an editable memory built from users' historical interactions. The editable memory allows cross-encoders to be personalized efficiently and enables users to control personalized ranking. Next, because all queries do not require personalization, we introduce a calibrated mixing model which determines when personalization is necessary. This enables users to control personalization via their editable memory only when necessary. To thoroughly evaluate CtrlCE, we demonstrate its empirical performance in four domains of science, its ability to selectively request user control in a calibration evaluation of the mixing model, and the control provided by its editable memory in a user study.
Strategic Multi-Agent Dynamics in Competitive IR
Authors: Haya Nachimovsky
Venue: SIGIR
Year: 2025

In the Web ecosystem, document authors often strategically modify content to maximize visibility in search rankings. These ranking-incentivized modifications pose challenges to retrieval effectiveness, content diversity, and long-term stability. At the same time, the rise of large language models (LLMs) is transforming IR: queries and documents that were once assumed to be generated exclusively by humans can now also be created by automated agents [5]. These agents can formulate queries, generate documents, and perform ranking. This introduces a variety of new challenges and opportunities in competitive search environments, where interactions between strategic agents can substantially influence outcomes. This research pursues two primary goals. First, it seeks to understand strategic interactions, particularly how different configurations of users (query), authors (document) and systems (ranker) influence competition outcomes. Second, it explores methods for designing systems that foster beneficial outcomes, such as stability and fairness. To this end, this research employs a combination of game-theoretic analysis, empirical ranking competitions between humans, and simulation-based experiments to investigate competitive dynamics in information retrieval systems [1-4]. Our theoretical analysis begins by modeling settings where authors compete to promote their documents across multiple queries as a continuous game in which each author allocates emphasis across different queries. We fully characterize when pure Nash equilibria exist and show that, unlike single-query settings, a stable state almost never exists [4]. To address this instability, we propose a novel method of strategic corpus enrichment. Specifically, we show using theoretical analysis how the mediator, by strategically planting documents in the corpus, can achieve stability in the system [2].
Effective Inference-Free Retrieval for Learned Sparse Representations
Authors: Franco Maria Nardini, Thong Nguyen, Cosimo Rulli, Rossano Venturini, Andrew Yates
Venue: SIGIR
Year: 2025

Learned Sparse Retrieval (LSR) is an effective IR approach that exploits pre-trained language models for encoding text into a learned bag of words. Several efforts in the literature have shown that sparsity is key to enabling a good trade-off between the efficiency and effectiveness of the query processor. To induce the right degree of sparsity, researchers typically use regularization techniques when training LSR models. Recently, new efficient-inverted index-based-retrieval engines have been proposed, leading to a natural question: has the role of regularization changed in training LSR models? In this paper, we conduct an extended evaluation of regularization approaches for LSR where we discuss their effectiveness, efficiency, and out-of-domain generalization capabilities. We first show that regularization can be relaxed to produce more effective LSR en- coders. We also show that query encoding is now the bottleneck limiting the overall query processor performance. To remove this bottleneck, we advance the state-of-the-art of inference-free LSR by proposing Learned Inference-free Retrieval (Li-Lsr). At training time, Li-Lsr learns a score for each token, casting the query encoding step into a seamless table lookup. Our approach yields state-of-the-art effectiveness for both in-domain and out-of-domain evaluation,surpassing Splade-v3-Doc by 1 point of mRR@10 on MsMarco and 1.8 points of nDCG@10 on Beir.
Advancing Chichewa IR
Authors: Stanley Ndebvu, Reuben Moyo, Catherine Chavula
Venue: SIGIR
Year: 2025

Malawi is home to over ten local languages, including Chichewa, yet many of these languages lack both printed and digital resources. Consequentially, access to information in these languages is limited, and this hinders knowledge sharing, which may potentially impact socio-economic development. In this paper, we discuss our work on developing language resources and tools for Chichewa. We begin by providing an overview of the Chichewa language, and highlight its inherent complexities that require new approaches to informa tion retrieval (IR) and natural language processing (NLP). We then present our past, current, and ongoing research and conclude with future directions. Our goal is to engage with the IR community to discuss how we can advance IR for low resource languages (LRLs) like Chichewa.
Distributionally Robust Optimization for Unbiased Learning to Rank
Authors: Zechun Niu, Lang Mei, Chong Chen, Jiaxin Mao
Venue: SIGIR
Year: 2025

Unbiased learning to rank (ULTR), which utilizes historical click logs to train ranking models, has attracted much attention in the IR community. Previous studies on ULTR have focused on mitigating a variety of biases in click logs, such as position bias, trust bias, and presentation bias, to recover the true relevance of the query-document pairs. However, they overlooked the intrinsic distribution shifts between the training data and test data. In this paper, we first validate and analyze the distribution shift problem with a real-world ULTR dataset. To solve this problem, we propose distributionally robust unbiased learning to rank (DRO-ULTR) methods. Specifically, we design two kinds of group distributionally robust optimization (group-DRO) frameworks for the existing ULTR methods, one using the pointwise click prediction loss and the other using the listwise counterfactual ranking loss. Finally, we empirically verify the effectiveness of our DRO-ULTR methods by conducting extensive experiments on the real-world dataset.
Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study
Authors: Zechun Niu, Zhilin Zhang, Jiaxin Mao, Qingyao Ai, Ji-Rong Wen
Venue: SIGIR
Year: 2025

Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models. While the CLTR models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate, their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely available, large-scale, real click logs. However, many previous simulation-based experiments are somewhat limited because they may have one or more of the following deficiencies: 1) using a weak production ranker to generate initial ranked lists, 2) relying on a simplified user simulation model to simulate user clicks, and 3) generating a fixed number of synthetic click logs. As a result, the robustness of CLTR models in complex and diverse situations is largely unknown and needs further investigation. To address this problem, in this paper, we aim to investigate the robustness of existing CLTR models in a reproducibility study with extensive simulation-based experiments that (1) use production rankers with different ranking performance (2) leverage multiple user simulation models with different user behavior assumptions, and (3) generate different numbers of synthetic sessions for the training queries. We find that the IPS-DCM, DLA-PBM, and UPE models show better robustness under various simulation settings than other CLTR models. Moreover, existing CLTR models often fail to outperform naive click baselines when the production ranker is strong and the number of training sessions is limited, indicating a pressing need for new CLTR algorithms tailored to these conditions.
NodeRec+: A Lightweight Framework for Federated Recommender Systems
Authors: Diarmuid O&apos, Reilly-Morgan, Erika Duriakova, Elias Z. Tragos, Neil Hurley, Aonghus Lawlor
Venue: SIGIR
Year: 2025

Data privacy is a critical concern in today's data-driven world. To this end, Federated Learning (FL) has been researched extensively, as it allows sensitive data to be kept secure on local devices while training global Machine Learning (ML) models across multiple devices. Several FL topologies have been introduced to address different users' needs. FL fits well within the Recommender Systems (RS) domain, with decentralised large-scale datasets and user privacy issues, as it improves personalised recommendations and accuracy while keeping users' sensitive data secure. In this paper, we introduce NodeRec+, the prototype of an FL framework, geared towards enabling the design and evaluation of decentralised RS. NodeRec+ allows quick and easy composition of various FL topologies in simple Python code and comes ready integrated with a number of RS models and evaluation protocols, enabling rapid benchmarking of models in various FL topologies. In contrast to most similar frameworks that are either used for simulations or focused solely on a single FL topology, NodeRec+ provides a framework to build real decentralised networks of nodes that can work jointly using several FL topologies in RS settings. A demonstration video showcasing the NodeRec+ framework can be found at https://www.youtube.com/watch?v=IIo3Cr56YyI. Code is provided at https://github.com/SEDIMARK-UCD/noderec.
Understanding Audio-Text Retrieval Through Singular Value Decomposition
Authors: Yoori Oh, Yoseob Han, Joonhyeon Bae, Jaeheon Sim, Kyogu Lee
Venue: SIGIR
Year: 2025

Audio-language retrieval faces a fundamental challenge due to many-to-one mappings, where multiple audio instances correspond to the same or similar textual descriptions. This ambiguity leads to overlapping representations, making it difficult for models to learn discriminative features between samples. Queue-based contrastive learning helps mitigate this ambiguity by maintaining a large pool of negatives, improving retrieval performance. However, selecting appropriate hyperparameters, such as the queue size for optimal performance and the number of epochs to prevent overfitting, remains a challenge. In this study, we propose a LAtent Space Embedding Rank (LASER) analysis. Specifically, we analyze that a low-rank property of a queue indicates the presence of informative negative samples, and demonstrate that the best performance is achieved when the queue has the lowest rank. Furthermore, the convergence of rank variation can be interpreted as the model parameters reaching an optimal state during training. Thanks to the proposed rank-based analysis, the model hyperparameters can be selected analytically rather than through trial and error. Experimental results confirm that retrieval performance depends on queue rank, with lower ranks yielding better performance. These findings suggest that our LASER analysis can provide insight into how rank-based analysis can improve retrieval performance in audio-language tasks.
Classifying Term Variants in Query Formulation
Authors: Nuha Abu Onq, Mark Sanderson, Falk Scholer
Venue: SIGIR
Year: 2025

Formulating queries is a challenging stage of the search process. This study investigates how crowd workers formulate an initial query for a common information need described in a backstory, resulting in diverse query variations. Using the UQV100 dataset of information need backstories and corresponding queries, we analyze the variations. Our findings show that 70% of the query terms used in crowd worker queries did not appear in the backstory text. Examining such terms we developed a taxonomy of search strategies, with the most common being semantic variations of backstory terms, followed by information type specifications. Additionally, we categorized the backstories by cognitive complexity, showing that higher complexity led to greater diversity in query variations and a wider range of term variant categories. This study highlights the importance of accounting for query variations, term variants, user strategies, and cognitive complexity in designing search systems and test collections to better align with users' information needs, influenced by the cognitive demands of a task, and enhance system performance and usability.
Optimizing Compound Retrieval Systems
Authors: Harrie Oosterhuis, Rolf Jagerman, Zhen Qin, Xuanhui Wang
Venue: SIGIR
Year: 2025

Modern retrieval systems do not rely on a single ranking model to construct their rankings. Instead, they generally take a cascading approach where a sequence of ranking models are applied in multiple re-ranking stages. Thereby, they balance the quality of the top-K ranking with computational costs by limiting the number of documents each model re-ranks. However, the cascading approach is not the only way models can interact to form a retrieval system. We propose the concept of compound retrieval systems as a broader class of retrieval systems that apply multiple prediction models. This encapsulates cascading models but also allows other types of interactions than top-K re-ranking. In particular, we enable interactions with large language models (LLMs) which can provide relative relevance comparisons. We focus on the optimization of compound retrieval system design which uniquely involves learning where to apply the component models and how to aggregate their predictions into a final ranking. This work shows how our compound approach can combine the classic BM25 retrieval model with state-of-the-art (pairwise) LLM relevance predictions, while optimizing a given ranking metric and efficiency target. Our experimental results show optimized compound retrieval systems provide better trade-offs between effectiveness and efficiency than cascading approaches, even when applied in a self-supervised manner. With the introduction of compound retrieval systems, we hope to inspire the information retrieval field to more out-of-the-box thinking on how prediction models can interact to form rankings.
Hierarchical User Long-term Behavior Modeling for Click-Through Rate Prediction
Authors: Mao Pan, Xuanhua Yang, Nan Qiao, Dongyue Wang, Feng Mei, Xiwei Zhao, Sulong Xu
Venue: SIGIR
Year: 2025

State-of-the-art approaches for click-through rate (CTR) prediction in industry predominantly rely on transformer-based networks or their variants. However, as user behavior sequences become longer, employing self-attention networks for CTR prediction within a constrained inference time presents a significant challenge. To address this, mainstream methods adopt a classical two-stage paradigm: a General Search Unit (GSU) for quickly retrieving relevant items from long-term behaviors, and an Exact Search Unit (ESU) for applying effective Multi-Head Target Attention (MHTA) over the items selected by the GSU. These two-stage algorithms have certain limitations. Firstly, the GSU needs to retrieve different target subsequences for different target items, restricting the ESU to a suboptimal MHTA network rather than a more effective transformer-based network. Secondly, the GSU retrieves only a subset of items from the user's behavior sequence, ignoring the evolution of user interests and the interrelationships between different points of interest. To overcome these challenges, we propose a novel end-to-end hierarchical user long-term behavior modeling network for CTR prediction (HBM). Specifically, we employ the multi-interest routing layer to channel the user's long-term behavior to several aggregated interest clusters. Furthermore, we introduce a fine interest learning network that selects the top-k interests from the initial aggregated representations. Subsequently, we employ a transformer network to model the user's behavior sequence associated with these top-k interests in a detailed manner, while also capturing the inherent correlations between different user interests at a coarse level. Finally, we integrate the coarse and fine interests. Extensive experiments on two real-world datasets demonstrate the effectiveness of our proposed methods. In addition, an online A/B test on the JD recommendation platform shows promising improvements, with a 2.15% increase in CTR and a 0.98% increase in CVR, accompanied by lower online inference latency.
LLM as User Simulator: Towards Training News Recommender without Real User Interactions
Authors: Choongwon Park
Venue: SIGIR
Year: 2025

News recommendation systems traditionally rely on extensive real user interaction data to personalize content, which is often inaccessible and raises privacy concerns, particularly in regions lacking such data. To address these challenges, we propose LAUS (LLM As User Simulator), a novel framework that leverages LLM to simulate user interactions for training a news recommender without real user data. Our framework consists of two stages. First, we simulate user clicks by using an LLM to generate click probabilities for candidate news articles based on simulated user reading histories, with candidate sets generated through random sampling and embedding-based retrieval. Second, we train a lightweight news recommender using these simulated user interactions. Experiments on three datasets demonstrate that a news recommender trained with simulated data outperforms models using LLM prompting, while significantly reducing recommendation time per user. Our findings highlight the potential of using LLMs as user simulators to address data scarcity in news recommendation systems.