OpenRaiser/Intern-Atlas huggingface dataset:
Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets
Authors: Preetam Prabhu Srikar Dammu, Himanshu Naidu, Chirag Shah
Venue: SIGIR
Year: 2025

As question answering (QA) systems advance alongside the rapid evolution of foundation models, the need for robust, adaptable, and large-scale evaluation benchmarks becomes increasingly critical. Traditional QA benchmarks are often static and publicly available, making them susceptible to data contamination and memorization by large language models (LLMs). Consequently, static benchmarks may overestimate model generalization and hinder a reliable assessment of real-world performance. In this work, we introduce Dynamic-KGQA, a scalable framework for generating adaptive QA datasets from knowledge graphs (KGs), designed to mitigate memorization risks while maintaining statistical consistency across iterations. Unlike fixed benchmarks, Dynamic-KGQA generates a new dataset variant on every run while preserving the underlying distribution, enabling fair and reproducible evaluations. Furthermore, our framework provides fine-grained control over dataset characteristics, supporting domain-specific and topic-focused QA dataset generation. Additionally, Dynamic-KGQA produces compact, semantically coherent subgraphs that facilitate both training and evaluation of KGQA models, enhancing their ability to leverage structured knowledge effectively. To align with existing evaluation protocols, we also provide static large-scale train/test/validation splits, ensuring comparability with prior methods. By introducing a dynamic, customizable benchmarking paradigm, Dynamic-KGQA enables a more rigorous and adaptable evaluation of QA systems.
Data Augmentation as Free Lunch: Exploring the Test-Time Augmentation for Sequential Recommendation
Authors: Yizhou Dang, Yuting Liu, Enneng Yang, Minhan Huang, Guibing Guo, Jianzhe Zhao, Xingwei Wang
Venue: SIGIR
Year: 2025

Data augmentation has become a promising method of mitigating data sparsity in sequential recommendation. Existing methods generate new yet effective data during model training to improve performance. However, deploying them requires retraining, architecture modification, or introducing additional learnable parameters. The above steps are time-consuming and costly for well-trained models, especially when the model scale becomes large. In this work, we explore the test-time augmentation (TTA) for sequential recommendation, which augments the inputs during the model inference and then aggregates the model's predictions for augmented data to improve final accuracy. It avoids significant time and cost overhead from loss calculation and backward propagation. We first experimentally disclose the potential of existing augmentation operators for TTA and find that the Mask and Substitute consistently achieve better performance. Further analysis reveals that these two operators are effective because they retain the original sequential pattern while adding appropriate perturbations. Meanwhile, we argue that these two operators still face time-consuming item selection or interference information from mask tokens. Based on the analysis and limitations, we present TNoise and TMask. The former injects uniform noise into the original representation, avoiding the computational overhead of item selection. The latter blocks mask token from participating in model calculations or directly removes interactions that should have been replaced with mask tokens. Comprehensive experiments demonstrate the effectiveness, efficiency, and generalizability of our method. We provide an anonymous implementation at https://github.com/KingGugu/TTA4SR.
In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code
Authors: Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, Gul Calikli
Venue: SIGIR
Year: 2025

When applying LLM-based code generation to software development projects that follow a feature-driven or rapid application development approach, it becomes necessary to estimate the functional correctness of the generated code in the absence of test cases. Just as a user selects a relevant document from a ranked list of retrieved ones, a software generation workflow requires a developer to choose (and potentially refine) a generated solution from a ranked list of alternative solutions, ordered by their posterior likelihoods. This implies that estimating the quality of a ranked list - akin to estimating ''relevance'' for query performance prediction (QPP) in IR - is also crucial for generative software development, where quality is defined in terms of ''functional correctness''. In this paper, we propose an in-context learning (ICL) based approach for code quality estimation. Our findings demonstrate that providing few-shot examples of functionally correct code from a training set enhances the performance of existing QPP approaches as well as a zero-shot-based approach for code quality estimation.
Toward Holistic Evaluation of Recommender Systems Powered by Generative Models
Authors: Yashar Deldjoo, Nikhil Mehta, Maheswaran Sathiamoorthy, Shuai Zhang, Pablo Castells, Julian J. McAuley
Venue: SIGIR
Year: 2025

Recommender systems powered by generative models (Gen-RecSys) extend beyond classical item-ranking by producing open-ended content, which simultaneously unlocks richer user experiences and introduces new risks. On one hand, these systems can enhance personalization and appeal through dynamic explanations and multi-turn dialogues. On the other hand, they might venture into unknown territory-hallucinating nonexistent items, amplifying bias, or leaking private information. Traditional accuracy metrics cannot fully capture these challenges, as they fail to measure factual correctness, content safety, or alignment with user intent. This paper makes two main contributions. First, we categorize the evaluation challenges of Gen-RecSys into two groups: (i) existing concerns that are exacerbated by generative outputs (e.g., bias, privacy) and (ii) entirely new risks (e.g., item hallucinations, contradictory explanations). Second, we propose a holistic evaluation approach that includes scenario-based assessments and multi-metric checks-incorporating relevance, factual grounding, bias detection, and policy compliance. Our goal is to provide a guiding framework so that researchers and practitioners can thoroughly assess Gen-RecSys, ensuring both effective personalization and responsible deployment.
GENNEXT: The Next Generation of IR and Recommender Systems with Language Agents, Generative Models, and Conversational AI
Authors: Yashar Deldjoo, Scott Sanner, Enrico Palumbo, Hugues Bouchard, Shuai Zhang, Pablo Castells, Julian J. McAuley
Venue: SIGIR
Year: 2025

We present GENNEXT, a workshop dedicated to exploring the integration of language agents, generative models, and conversational AI within information retrieval (IR) and recommender systems (RS). Building on the success of our recent RecSys'24 workshop, GENNEXT aims to advance discussions on the applications of language agents powered by Large Language Models (LLMs). The workshop will focus on enhancing interactivity between users and systems through multi-turn dialogues, improving creative content generation, advancing personalization, and enabling multifaceted, context-aware decision-making. For example, a language agent could respond to a query like ''Suggest an eco-friendly food tour for a weekend in my city'' by using a recommendation API to identify eateries specializing in sustainable or organic cuisine and a pollution API to ensure the selected routes have low air pollution levels. GENNEXT will bring together leading researchers and practitioners through keynotes, paper presentations, and a panel discussion. We invite full papers, short papers, and extended abstracts covering theoretical advancements, practical applications, and evaluation strategies for generative technologies in IR and RS. The workshop will address key themes such as conversational adaptation, generative content creation, and agentic tool usage, while tackling challenges like bias, data privacy, and hallucination risks. Overall, our main ambition is to foster dialogue on creating ethical, sustainable, and innovative systems while addressing emerging opportunities and risks in modern IR and RS.
HeterRec: Heterogeneous Information Transformer for Scalable Sequential Recommendation
Authors: Hao Deng, Haibo Xing, Kanefumi Matsuyama, Yulei Huang, Jinxin Hu, Hong Wen, Jia Xu, Zulong Chen, Yu Zhang, Xiaoyi Zeng, Jing Zhang
Venue: SIGIR
Year: 2025

Transformer-based sequential recommendation (TSR) models have shown superior performance in recommendation systems, where the quality of item representations plays a crucial role. Classical representation methods integrate item features using concatenation or neural networks to generate homogeneous representation sequences. While straightforward, these methods overlook the heterogeneity of item features, limiting the transformer's ability to capture fine-grained patterns and restricting scalability. Recent studies have attempted to integrate user-side heterogeneous features into item representation sequences, but item-side heterogeneous features, which are vital for performance, remain excluded. To address these challenges, we propose a Heterogeneous Information Transformer model for Sequential Recommendation (HeterRec), which incorporates Heterogeneous Token Flatten Layer (HTFL) and Hierarchical Causal Transformer Layer (HCT). Our HTFL is a novel item tokenization method that converts items into a heterogeneous token set and organizes these tokens into heterogeneous sequences, effectively enhancing performance gains when scaling up the model. Moreover, HCT introduces token-level and item-level causal transformers to extract fine-grained patterns from the heterogeneous sequences. Experiments on offline and online datasets show that the HeterRec model achieves superior performance.
CSMF: Cascaded Selective Mask Fine-Tuning for Multi-Objective Embedding-Based Retrieval
Authors: Hao Deng, Haibo Xing, Kanefumi Matsuyama, Moyu Zhang, Jinxin Hu, Hong Wen, Yu Zhang, Xiaoyi Zeng, Jing Zhang
Venue: SIGIR
Year: 2025

Multi-objective embedding-based retrieval (EBR) has become increasingly critical due to the growing complexity of user behaviors and commercial objectives. While traditional approaches often suffer from data sparsity and limited information sharing between objectives, recent methods utilizing a shared network alongside dedicated sub-networks for each objective partially address these limitations. However, such methods significantly increase the model parameters, leading to an increased retrieval latency and a limited ability to model causal relationships between objectives. To address these challenges, we propose the Cascaded Selective Mask Fine-Tuning (CSMF), a novel method that enhances both retrieval efficiency and serving performance for multi-objective EBR. The CSMF framework selectively masks model parameters to free up independent learning space for each objective, leveraging the cascading relationships between objectives during the sequential fine-tuning. Without increasing network parameters or online retrieval overhead, CSMF computes a linearly weighted fusion score for multiple objective probabilities while supporting flexible adjustment of each objective's weight across various recommendation scenarios. Experimental results on real-world datasets demonstrate the superior performance of CSMF, and online experiments validate its significant practical value.
LLM-Driven Usefulness Labeling for IR Evaluation
Authors: Mouly Dewan, Jiqun Liu, Chirag Shah
Venue: SIGIR
Year: 2025

In the information retrieval (IR) domain, evaluation plays a crucial role in optimizing search experiences and supporting diverse user intents. In the recent LLM era, research has been conducted to automate document relevance labels. These labels have traditionally been assigned by crowd-sourced workers, a process that is both time consuming and costly. This study focuses on LLM-generated usefulness labels, a crucial evaluation metric that considers the user's search intents and task objectives, an aspect where relevance falls short. Our experiment utilizes task-level, query-level, and document-level features along with user search behavior signals, which are essential in defining the usefulness of a document. Our research finds that (i) pre-trained LLMs can generate moderate usefulness labels by understanding the comprehensive search task session, and (ii) pre-trained LLMs perform better judgment in short search sessions when provided with search session contexts. Furthermore, we investigate whether LLMs can capture the unique divergence between relevance and usefulness, along with conducting an ablation study to identify the most critical metrics for accurate usefulness label generation. In conclusion, this work explores LLM-generated usefulness labels by evaluating critical metrics and optimizing for practicality in real-world settings.
Targeted Multi-Modal Passage Search for Molecules and their Synthesis Pathways
Authors: Abhisek Dey, Nathaniel H. Stanley, Richard Zanibbi
Venue: SIGIR
Year: 2025

We present a chemical extraction and search pipeline intended to support information tasks related to drug discovery. Commonly used search tools for drug discovery such as Reaxys and SciFinder do not allow users to obtain retrieval results at the passage level. To address this, we present a passage retrieval tool for chemical patents that supports queries combining text and molecule diagrams expressed in SMILES. When SMILES is provided as a part of a query, the system refines text retrieval results through matching both textual names and drawn figures based on extracted SMILES representations. Molecule matches are obtained through substructure matching and structural similarity. This functionality was motivated by a chemist's need to find synthesis pathways for specific molecules containing a substructure of interest that binds and thus inhibits specific human genes. For this demonstration, we index a collection of 131 PDF patents categorized into 12 specific genes enabling a user to search on them. There are 32,301 document pages in the collection. Our user interface can be accessed at https://unichemfinder.gccis.rit.edu/. Our source code and data is available at https://gitlab.com/dprl/unichemfinder.
RSGEA: Relationship Structure Line Graph for Semi-supervised Entity Alignment based on Edge Weight Adjustment
Authors: Linlin Ding, Mengjunyao Si, Mo Li
Venue: SIGIR
Year: 2025

Entity alignment (EA) aims to identify equivalent entities across different knowledge graphs (KGs). While existing approaches leverage KG neighborhood structures for alignment, they often fail to effectively distinguish relevant from irrelevant neighbors due to insufficient handling of neighbor heterogeneity. Additionally, entity enhancement strategies remain underutilized. To address these issues, we propose a novel Relationship Structure Line Graph for Semi-supervised Entity Alignment Based on Edge Weight Adjustment, named RSGEA. It first enhances entity representations by deeply analyzing relational connectivity structures in KGs, capturing key relational information from second-order and triangular-ring structures. It then employs an attention mechanism to dynamically adjust edge weights, mitigating the impact of noisy edges during information propagation. Finally, we employ the Sinkhorn algorithm to refine the similarity matrix, improving alignment accuracy. Furthermore, we introduce an unsupervised version to accommodate diverse scenarios. Extensive experiments on five cross-lingual datasets validate the effectiveness and robustness of the RSGEA, demonstrating significant performance improvements.
WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval
Authors: Michael Dinzinger, Laura Caspari, Kanishka Ghosh Dastidar, Jelena Mitrovic, Michael Granitzer
Venue: SIGIR
Year: 2025

We present WebFAQ, a large-scale collection of open-domain question answering datasets derived from FAQ-style schema.org annotations. In total, the data collection consists of 96 million natural question-answer (QA) pairs across 75 languages, including 47 million (49%) non-English samples. WebFAQ further serves as the foundation for 49 monolingual retrieval benchmarks with a total size of 11.2 million QA pairs (5.9 million non-English). These datasets are carefully curated through refined filtering and near-duplicate detection, yielding high-quality resources for training and evaluating multilingual dense retrieval models. To empirically confirm WebFAQ's efficacy, we use the collected QAs to fine-tune an in-domain pretrained XLM-RoBERTa model. Through this process of dataset-specific fine-tuning, the model achieves significant retrieval performance gains, which generalize - beyond WebFAQ - to other multilingual retrieval benchmarks evaluated in zero-shot setting. Last but not least, we utilize WebFAQ to construct a set of QA-aligned bilingual corpora spanning over 1000 language pairs using state-of-the-art bitext mining and automated LLM-assessed translation evaluation. Due to our advanced, automated method of bitext dataset generation, the resulting bilingual corpora demonstrate higher translation quality compared to similar datasets. WebFAQ and all associated resources are publicly available on GitHub and HuggingFace.
A Reproducibility Study of Graph-Based Legal Case Retrieval
Authors: Gregor Donabauer, Udo Kruschwitz
Venue: SIGIR
Year: 2025

Legal retrieval is a widely studied area in Information Retrieval (IR) and a key task in this domain is retrieving relevant cases based on a given query case, often done by applying language models as encoders to model case similarity. Recently, Tang et al. proposed CaseLink, a novel graph-based method for legal case retrieval, which models both cases and legal charges as nodes in a network, with edges representing relationships such as references and shared semantics. This approach offers a new perspective on the task by capturing higher-order relationships of cases going beyond the stand-alone level of documents. However, while this shift in approaching legal case retrieval is a promising direction in an understudied area of graph-based legal IR, challenges in reproducing novel results have recently been highlighted, with multiple studies reporting difficulties in reproducing previous findings. Thus, in this work we reproduce CaseLink, a graph-based legal case retrieval method, to support future research in this area of IR. In particular, we aim to assess its reliability and generalizability by (i) first reproducing the original study setup and (ii) applying the approach to an additional dataset. We then build upon the original implementations by (iii) evaluating the approach's performance when using a more sophisticated graph data representation and (iv) using an open large language model (LLM) in the pipeline to address limitations that are known to result from using closed models accessed via an API. Our findings aim to improve the understanding of graph-based approaches in legal IR and contribute to improving reproducibility in the field. To achieve this, we share all our implementations and experimental artifacts with the community.
MO-LightGBM: A Library for Multi-objective Learning to Rank with LightGBM
Authors: Chaosheng Dong, Michinari Momma
Venue: SIGIR
Year: 2025

This paper introduces MO-LightGBM, an open-source library built upon LightGBM, specifically designed to offer an integrated, versatile, and easily adaptable framework for Multi-objective Learning to Rank (MOLTR). MO-LightGBM supports diverse Multi-objective optimization (MOO) settings and incorporates 12 state-of-the-art optimization strategies. Its modular architecture enhances usability and flexibility, allowing researchers and practitioners to easily develop new MOO methodologies, perform rigorous comparisons with existing techniques, and effectively deploy MOO algorithms in practical ranking applications. We illustrate the utility of MO-LightGBM through a Bi-objective Learning to Rank example and present visualizations of the results. MO-LightGBM is available at https://github.com/amazon-science/MO-LightGBM.
ELEC: Efficient Large Language Model-Empowered Click-Through Rate Prediction
Authors: Rui Dong, Wentao Ouyang, Xiangzheng Liu
Venue: SIGIR
Year: 2025

Click-through rate (CTR) prediction plays an important role in online advertising systems. On the one hand, traditional CTR prediction models capture the collaborative signals in tabular data via feature interaction modeling, but they lose semantics in text. On the other hand, Large Language Models (LLMs) excel in understanding the context and meaning behind text, but they face challenges in capturing collaborative signals and they have long inference latency. In this paper, we aim to leverage the benefits of both types of models and pursue collaboration, semantics and efficiency. We present ELEC, which is an Efficient LLM-Empowered CTR prediction framework. We first adapt an LLM for the CTR prediction task. In order to leverage the ability of the LLM but simultaneously keep efficiency, we utilize the pseudo-siamese network which contains a gain network and a vanilla network. We inject the high-level representation vector generated by the LLM into a collaborative CTR model to form the gain network such that it can take advantage of both tabular modeling and textual modeling. However, its reliance on the LLM limits its efficiency. We then distill the knowledge from the gain network to the vanilla network on both the score level and the representation level, such that the vanilla network takes only tabular data as input, but can still generate comparable performance as the gain network. Our approach is model-agnostic. It allows for the integration with various existing LLMs and collaborative CTR models. Experiments on real-world datasets demonstrate the effectiveness and efficiency of ELEC for CTR prediction.
Open-World Fine-Grained Fashion Retrieval with LLM-based Commonsense Knowledge Infusion
Authors: Jianfeng Dong, Junwei Zhu, Daizong Liu, Xiaoye Qu, Cuizhu Bao, Zhike Han, Jixiang Zhu, Xun Wang
Venue: SIGIR
Year: 2025

Attribute-Specific Fashion Retrieval (ASFR) focuses on retrieving images based on fine-grained, attribute-specific criteria rather than naive global visual similarity, enabling more precise and interpretable search results. Existing ASFR methods ideally assume that all attribute semantics are in-domain distributions of the training datasets. However, realistic scenarios are generally more complex and naturally contain unseen attribute information, often resulting in ungeneralizable retrieval outcomes. In this paper, we take the first step to address the new and challenging open-world ASFR setting, which involves handling diverse and practical attributes instead of relying solely on predefined attribute sets in closed-world scenarios. Specifically, to comprehend unseen attributes, we propose a novel LLM-based Commonsense Knowledge Infusion (CoKi) framework that integrates commonsense knowledge as complementary context into attribute representations using a Large Language Model (LLM). By infusing such LLM-based commonsense knowledge through descriptive contexts, our method enables robust semantic enrichment and effective generalization to unseen attributes. Additionally, we introduce a modality-switchable prompt and an imputation mechanism to ensure model robustness across diverse input configurations by dynamically adapting to missing modalities. Extensive experiments demonstrate that our approach not only achieves state-of-the-art in-domain retrieval performance but also significantly enhances adaptability to unseen attributes and cross-domain generalization, establishing a new benchmark for fine-grained fashion retrieval in open-world scenarios. Our source code is publicly available at https://github.com/HuiGuanLab/CoKi.