OpenRaiser/Intern-Atlas huggingface dataset:
An Empirical Study of Evaluating Long-form Question Answering
Authors: Ning Xian, Yixing Fan, Ruqing Zhang, Maarten de Rijke, Jiafeng Guo
Venue: SIGIR
Year: 2025

Long-form question answering (LFQA) aims to generate lengthy answers to complex questions. This scenario presents great flexibility as well as significant challenges for evaluation. Most evaluations rely on deterministic metrics that depend on string or n-gram matching, while the reliability of large language model-based evaluations for long-form answers remains relatively unexplored. We address this gap by conducting an in-depth study of long-form answer evaluation with the following research questions: (i) To what extent do existing automatic evaluation metrics serve as a substitute for human evaluations? (ii) What are the limitations of existing evaluation metrics compared to human evaluations? (iii) How can the effectiveness and robustness of existing evaluation methods be improved? We collect 5,236 factoid and non-factoid long-form answers generated by different large language models and conduct a human evaluation on 2,079 of them, focusing on correctness and informativeness. Subsequently, we investigated the performance of automatic evaluation metrics by evaluating these answers, analyzing the consistency between these metrics and human evaluations. We find that the style, length of the answers, and the category of questions can bias the automatic evaluation metrics. However, fine-grained evaluation helps mitigate this issue on some metrics. Our findings have important implications for the use of large language models for evaluating long-form question answering. All code and datasets are available at https://github.com/bugtig6351/lfqa_evaluation.
AdaRPT: An Adaptive Rule Pattern Transfer Model for Fully Inductive Knowledge Graph Reasoning
Authors: Zhiwen Xie, Zhuo Zhao, Jinjin Ma, Guangyou Zhou, Jimmy Xiangji Huang
Venue: SIGIR
Year: 2025

Knowledge graph reasoning (KGR) is a key technology that infers missing facts in knowledge graphs (KGs). Given that real-world scenarios typically encounter unseen KGs with new entities and new relations, researchers have begun to explore fully inductive KGR methods. This setting presents greater challenges and has not been fully explored. Current methods primarily construct relation graphs based on the original KG to facilitate message passing between relations. These models have made significant progress in achieving fully inductive reasoning. However, as relation graphs focus solely on the co-occurrence patterns between relations, they often fail to capture reasoning patterns in KGs, which causes the model to struggle in effectively distinguishing between different relations and entities. This limitation severely restrict the reasoning capabilities of existing methods. In light of this, we propose the Adaptive Rule Pattern Transfer model (AdaRPT) for KGR. It aims to leverage logical rules for each relation in the KG to learn more comprehensive and transferable knowledge representations for entities and relations. For entities, we design a non-parameter message passing model that aggregates path information from the query entity to other entities. The path information for each entity is then matched with rules to obtain the transferable feature of each entity. And for relations, we extract both reasoning and co-occurrence patterns from KGs as transferable relation features. Finally, a path-based graph neural network (GNN) is employed on the transferable features of entities and relations to perform reasoning on KGs. Extensive experimental evaluations on 43 datasets for both inductive and transductive reasoning demonstrate the effectiveness and generalization capability of AdaRPT.
Enhancing the Patent Matching Capability of Large Language Models via the Memory Graph
Authors: Qiushi Xiong, Zhipeng Xu, Zhenghao Liu, Mengjia Wang, Zulong Chen, Yue Sun, Yu Gu, Xiaohua Li, Ge Yu
Venue: SIGIR
Year: 2025

Intellectual Property (IP) management involves strategically protecting and utilizing intellectual assets to enhance organizational innovation, competitiveness, and value creation. Patent matching is a crucial task in intellectual property management, which facilitates the organization and utilization of patents. Existing models often rely on the emergent capabilities of Large Language Models (LLMs) and leverage them to identify related patents directly. However, these methods usually depend on matching keywords and overlook the hierarchical classification and categorical relationships of patents. In this paper, we propose MemGraph, a method that augments the patent matching capabilities of LLMs by incorporating a memory graph derived from their parametric memory. Specifically, MemGraph prompts LLMs to traverse their memory to identify relevant entities within patents, followed by attributing these entities to corresponding ontologies. After traversing the memory graph, we utilize extracted entities and ontologies to improve the capability of LLM in comprehending the semantics of patents. Experimental results on the PatentMatch dataset demonstrate the effectiveness of MemGraph, achieving a 17.68% performance improvement over baseline LLMs. The further analysis highlights the generalization ability of MemGraph across various LLMs, both in-domain and out-of-domain, and its capacity to enhance the internal reasoning processes of LLMs during patent matching. All data and codes are available at https://github.com/NEUIR/MemGraph.
A Learnable Fully Interacted Two-Tower Model for Pre-Ranking System
Authors: Chao Xiong, Xianwen Yu, Wei Xu, Lei Cheng, Chuan Yuan, Linjian Mo
Venue: SIGIR
Year: 2025

Pre-ranking plays a crucial role in large-scale recommender systems by significantly improving the efficiency and scalability within the constraints of providing high-quality candidate sets in real time. The two-tower model is widely used in pre-ranking systems due to a good balance between efficiency and effectiveness with decoupled architecture, which independently processes user and item inputs before calculating their interaction (e.g. dot product or similarity measure). However, this independence also leads to the lack of information interaction between the two towers, resulting in less effectiveness. In this paper, a novel architecture named learnable Fully Interacted Two-tower Model (FIT) is proposed, which enables rich information interactions while ensuring inference efficiency. FIT mainly consists of two parts: Meta Query Module (MQM) and Lightweight Similarity Scorer (LSS). Specifically, MQM introduces a learnable item meta matrix to achieve expressive early interaction between user and item features. Moreover, LSS is designed to further obtain effective late interaction between the user and item towers. Finally, experimental results on several public datasets show that our proposed FIT significantly outperforms the state-of-the-art baseline pre-ranking models.
DeepReport: An AI-assisted Idea Generation System for Scientific Research
Authors: Yi Xu, Luoyi Fu, Shuqian Sheng, Bo Xue, Jiaxin Ding, Lei Zhou, Xinbing Wang, Chenghu Zhou
Venue: SIGIR
Year: 2025

Nowadays, the explosive growth of academic literature has been going far beyond scientists' limited capability to read through, making it increasingly difficult for them to absorb disciplinary insights and extract intellectual essences critical for generating novel research ideas in interdisciplinary studies. To address this, we develop DeepReport, an AI-assisted scientific idea generation system to alleviate the research burden. Technically, DeepReport maintains evolving concept co-occurrence graphs to extract core insights from over 260 million publications across all disciplines. These concepts are periodically collected and updated, enabling the automatic extraction of hidden cross-domain connections. Combining temporal link prediction and analysis techniques with large language models, DeepReport is able to further transform these patterns of insights into actionable ideas. With the function of integrating up-to-date academic databases, visualizing dynamic relationships of concepts, and automatically generating new ideas, DeepReport empowers researchers to navigate complex knowledge landscapes, reduce cognitive burdens, and accelerate the generation of groundbreaking concepts. This work provides an in-depth exploration of DeepReport's architecture, functionalities, and applications, highlighting its transformative potential for advancing interdisciplinary research and fostering innovation. DeepReport is available at https://idea.acemap.cn/.
Towards More Relevant Product Search Ranking with Fulfillment Intent Understanding
Authors: Jingxu Xu, Xinyi Liu, Yang Yu, Semih Yagli, Jingbo Liu, Cun Mu
Venue: SIGIR
Year: 2025

E-commerce retailers increasingly offer diverse fulfillment options (e.g., in-store pickup, same-day delivery from store, standard shipping from fulfillment centers and marketplace sellers), creating a need for search systems that understand and cater to individual customer preferences. This paper addresses the challenge of incorporating fulfillment intent into product search ranking. We propose a model that predicts customer fulfillment intent based on the search query, past user interactions, and other contextual information. A fulfillment match signal is introduced to quantify the alignment between a product's available fulfillment methods and the predicted customer intent. Integrating this signal into the ranking process ensures that search results prioritize products matching the user's preferred fulfillment type. Offline and online experiments demonstrate the efficacy of our approach and highlight the importance of fulfillment-aware ranking in omnichannel e-commerce.
HiLTV: Hierarchical Multi-Distribution Modeling for Lifetime Value Prediction in Online Games
Authors: Junwei Xu, Aisi Zheng, Ling Ding, Huangbin Zhang, Zhengwei Deng, Qun Yu, Xiao-Ping Zhang
Venue: SIGIR
Year: 2025

Customer Lifetime Value (LTV) prediction is a critical task in online games, as it helps with the formulation of refined game operation strategies, resource allocation, and personalized recommendation. Accurate LTV values enable to identify and target high-value users, enhance user retention and further long-term revenue growth. However, LTV prediction in online games faces unique challenges. Most in-app purchases (IAP) games have various fixed recharge levels, and users with different payment preferences show distinct LTV distributions. Moreover, existing methods fail to capture the multi-modal distribution of LTV values, and suffer from bias when predicting LTV values of games that users have not registered, i.e., new users. To address these challenges, we propose HiLTV, a novel hierarchical framework for LTV prediction in online games. We devise hierarchical modules to align with real-world user recharge behaviors, and a Zero-Inflated Mixture-of-Logistic (ZIMoL) loss instead of a unimodal distribution loss is adopted to better model various segments of users. We also introduce a calibration module that enables more robust predictions for new users. Both offline evaluation on real-world industrial datasets over state-of-the-art baselines and online A/B test from a leading game platform demonstrate the superior performance of our method.
From Knowledge Forgetting to Accumulation: Evolutionary Relation Path Passing for Lifelong Knowledge Graph Embedding
Authors: Jing Yang, Xinfa Jiang, Xiaowen Jiang, Yuan Gao, Laurence T. Yang, Shaojun Zou, Shundong Yang
Venue: SIGIR
Year: 2025

The continual emergence of new entities and relations drives the dynamic expansion of knowledge graphs (KG). In the face of such growing KG, relearning from scratch wastes acquired knowledge, while learning solely from new snapshots leads to model forgetting of old knowledge. Existing methods focus on lifelong learning in growing KG through transfer and regularize embeddings. However, extensive entity updates to adapt to new snapshots introduce conflicts between old and new knowledge, thereby resulting in the inevitable occurrence of knowledge forgetting. To address these challenges, we propose the Evolutionary Relation Path Passing (ERPP) model for lifelong knowledge graph embedding, aiming to shift from knowledge forgetting to knowledge accumulation, thereby achieving accurate long-term prediction. Specifically, we propose a snapshot conditional relation path passing strategy to generate expressive representations that better adapt to snapshots compared to the transferred embeddings in existing methods. Subsequently, we propose a relation inheritance and evolution mechanism across snapshots and continue relation path passing in next snapshots. This allows ERPP to avoid inevitable catastrophic forgetting from frequent entity embedding updates. ERPP outperforms SOTA models in 35 scenarios, with average improvements of 11.1% in long-term prediction and 12.9% in knowledge transfer. Moreover, ERPP makes a breakthrough in achieving knowledge positive accumulation, in contrast to the negative forgetting of existing models. To the best of our knowledge, ERPP is the first model to realize knowledge accumulation. Our code is available at https://anonymous.4open.science/r/ERPP-6D66.
Multi-scenario Instance Embedding Learning for Deep Recommender Systems
Authors: Chaohua Yang, Dugang Liu, Xing Tang, Yuwen Fu, Xiuqiang He, Xiangyu Zhao, Zhong Ming
Venue: SIGIR
Year: 2025

Multi-scenario recommendation (MSR) has become a core component of various online platforms, but its increasing model size has also brought attention to its efficiency optimization. An important effort is to find effective and efficient feature embedding layers for MSR, and existing work focuses on scenario-level feature selection, i.e., all instance embeddings in the same scenario get the same filtering results on the feature set, and the filtering results are different for different scenarios. However, this ignores the information redundancy of the dimension set and the individuality of different instances in the same scenario. To address these limitations, we propose a multi-scenario instance embedding learning (MultiEmb) framework that implements exclusive feature-dimension redundant information removal for different instances within a scenario to obtain the optimal individual embeddings. The core of our MultiEmb is to introduce an instance embedding selection network to effectively complete the above challenging tasks, in which a set of feature selection and dimension selection adaptive components are equipped for each scenario, and their combination completes the optimal embedding selection for each instance. Finally, we evaluate MultiEmb through extensive experiments on two public multi-scenario benchmarks and demonstrate its effectiveness, compatibility, transferability, etc.
Nugget-based Annotation Protocol and Tool For Evaluating Long-form Retrieval-Augmented Generation
Authors: Eugene Yang, Dawn J. Lawrie, Hoa Dang, Ian Soboroff, James Mayfield
Venue: SIGIR
Year: 2025

Retrieval-augmented generation (RAG) summarizes retrieved documents into a text passage that fulfills the information need expressed by the user. Such generated responses should faithfully distill the relevant information and provide sufficient attribution back to the source documents. Nugget-based evaluation was proposed for text summarization and has been adapted to evaluate RAG output in recent shared tasks such as 2024 TREC RAG, BioGen, and NeuCLIR tracks. However, annotating such detailed and nuanced information is complex and errorful. Multiple pieces of information need to be labeled, extracted, linked, and cross-referenced. In this work, we present an annotation protocol and tool tailored to collecting information for evaluating RAG systems. Our tool has four steps: nugget creation, nugget revision, document support assessment, and finally, nugget alignment. Each step aims to minimize the annotator's cognitive load, improving the efficiency and reliability.
Comprehensive List Generation for Multi-Generator Reranking
Authors: Hailan Yang, Zhenyu Qi, Shuchang Liu, Xiaoyu Yang, Xiaobei Wang, Xiang Li, Lantao Hu, Han Li, Kun Gai
Venue: SIGIR
Year: 2025

Reranking models solve the final recommendation lists that best fulfill users' demands. While existing solutions focus on finding parametric models that approximate optimal policies, recent approaches find that it is better to generate multiple lists to compete for a ''pass'' ticket from an evaluator, where the evaluator serves as the supervisor who accurately estimates the performance of the candidate lists. In this work, we show that we can achieve a more efficient and effective list proposal with a multi-generator framework and provide empirical evidence on two public datasets and online A/B tests. More importantly, we verify that the effectiveness of a generator is closely related to how much it complements the views of other generators with sufficiently different rerankings, which derives the metric of list comprehensiveness. With this intuition, we design an automatic complementary generator-finding framework that learns a policy that simultaneously aligns the users' preferences and maximizes the list comprehensiveness metric. The experimental results indicate that the proposed framework can further improve the multi-generator reranking performance.
Invariance Matters: Empowering Social Recommendation via Graph Invariant Learning
Authors: Yonghui Yang, Le Wu, Yuxin Liao, Zhuangzhuang He, Pengyang Shao, Richang Hong, Meng Wang
Venue: SIGIR
Year: 2025

Graph-based social recommender systems have demonstrated great potential in alleviating data sparsity by leveraging high-order user influence embedded in social networks. However, most existing methods rely heavily on the observed social graph, which is often noisy and includes spurious or task-irrelevant connections that can mislead user preference learning. Identifying and removing these noisy relations is crucial but challenging due to the lack of ground-truth annotations. In this paper, we approach the social denoising problem from the perspective of graph invariant learning and propose a novel approach, Social Graph Invariant Learning(SGIL). Specifically, SGIL aims to uncover stable user preferences within the input social graph, thereby enhancing the robustness of graph-based social recommendation systems. To achieve this goal, SGIL first simulates multiple noisy social environments through graph generators. It then seeks to learn environment-invariant user preferences by minimizing invariant risk across these environments. To further promote diversity in the generated social environments, we employ an adversarial training strategy to simulate more potential social noisy distributions. Extensive experimental results demonstrate the effectiveness of the proposed SGIL. The code is available at https://github.com/yimutianyang/SIGIR2025-SGIL.
Knowing You Don't Know: Learning When to Continue Search in Multi-round RAG through Self-Practicing
Authors: Diji Yang, Linda Zeng, Jinmeng Rao, Yi Zhang
Venue: SIGIR
Year: 2025

Retrieval Augmented Generation (RAG) has shown strong capability in enhancing language models' knowledge and reducing AI generative hallucinations, driving its widespread use. However, complex tasks requiring multi-round retrieval remain challenging, and early attempts tend to be overly optimistic without a good sense of self-skepticism. Current multi-round RAG systems may continue searching even when enough information has already been retrieved, or they may provide incorrect answers without having sufficient information or knowledge. Existing solutions either require large amounts of expensive human-labeled process supervision data or lead to subpar performance. This paper aims to address these limitations by introducing a new framework, SIM-RAG, to explicitly enhance RAG systems' self-awareness and multi-round retrieval capabilities. To train SIM-RAG, we first let a RAG system self-practice multi-round retrieval, augmenting existing question-answer pairs with intermediate inner monologue reasoning steps to generate synthetic training data. For each pair, the system may explore multiple retrieval paths, which are labeled as successful if they reach the correct answer and unsuccessful otherwise. Using this data, we train a lightweight information sufficiency Critic. At inference time, the Critic evaluates whether the RAG system has retrieved sufficient information at each round, guiding retrieval decisions and improving system-level self-awareness through in-context reinforcement learning. Experiments across multiple prominent RAG benchmarks show that SIM-RAG is an effective multi-round RAG solution. Furthermore, this framework is system-efficient, adding a lightweight component to RAG without requiring modifications to existing LLMs or search engines, and data-efficient, eliminating the need for costly human-annotated mid-step retrieval process supervision data.
Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
Authors: Zheng Yao, Shuai Wang, Guido Zuccon
Venue: SIGIR
Year: 2025

Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.
Neural Lexical Search with Learned Sparse Retrieval
Authors: Andrew Yates, Carlos Lassance, Cosimo Rulli, Eugene Yang, Sean MacAvaney, Siddharth A. K. Singh, Thong Nguyen, Yibin Lei
Venue: SIGIR
Year: 2025

Learned Sparse Retrieval (LSR) techniques use neural machinery to represent queries and documents as learned bags of words. In contrast with other neural retrieval techniques, such as generative retrieval and dense retrieval, LSR has been shown to be a remarkably robust, transferable, and efficient family of methods for retrieving high-quality search results. This half-day tutorial aims to provide an extensive overview of LSR, ranging from its fundamentals to the latest emerging techniques. By the end of the tutorial, attendees will be familiar with the important design decisions of an LSR system, know how to apply them to text and other modalities, and understand the latest techniques for retrieving with them efficiently. Website: https://lsr-tutorial.github.io