OpenRaiser/Intern-Atlas huggingface dataset:
LLM-Empowered Creator Simulation for Long-Term Evaluation of Recommender Systems Under Information Asymmetry
Authors: Xiaopeng Ye, Chen Xu, Zhongxiang Sun, Jun Xu, Gang Wang, Zhenhua Dong, Ji-Rong Wen
Venue: SIGIR
Year: 2025

Maintaining the long-term sustainability of recommender systems (RS) is crucial. Traditional RS evaluation methods primarily focus on the user's immediate feedback (e.g., click), however, they often overlook the long-term effect involved by the content creators. In the real world, content creators can strategically create and upload new items to the platform by analyzing users' feedback and preference trends. Although previous studies have attempted to model creator behaviors, they often overlook that such behaviors are under conditions of information asymmetry. This asymmetry arises because creators mainly access the user feedback on the items they produce, while the platform has access to the full spectrum of feedback data. However, existing RS simulators often fail to consider such a condition, making the long-term RS evaluation inaccurate. To bridge this gap, we propose a Large Language Model (LLM)-empowered creator simulation agent named CreAgent. By utilizing the belief mechanism from game theory and the fast-and-slow thinking framework, we can simulate the creator's behaviors well under information asymmetry. Furthermore, to enhance CreAgent's simulation ability, we utilize Proximal Policy Optimization to fine-tune CreAgent. Our credibility validation experiments demonstrate that our simulation environment effectively aligns with the behaviors of real-world platforms and creators, thereby enhancing the reliability of long-term evaluations in RS. Furthermore, leveraging this simulator, we can examine whether RS algorithms, such as fairness- and diversity-aware methods, contribute to improving long-term performance for different stakeholders.
Dense Retrieval for Low Resource languages - the Case of Amharic Language
Authors: Tilahun Yeshambel, Moncef Garouani, Serge Molina, Josiane Mothe
Venue: SIGIR
Year: 2025

This paper presents our investigation into dense retrieval models for Amharic, a low-resource language spoken by more than 120 million people. We constructed training datasets tailored to dense retrieval models and evaluated model performance by comparing dense and sparse retrieval approaches on Amharic information retrieval. The study also highlights the challenges and efforts involved in advancing retrieval systems for low-resource languages.
FashionDPO: Fine-tune Fashion Outfit Generation Model using Direct Preference Optimization
Authors: Mingzhe Yu, Yunshan Ma, Lei Wu, Changshuo Wang, Xue Li, Lei Meng
Venue: SIGIR
Year: 2025

Personalized outfit generation aims to construct a set of compatible and personalized fashion items as an outfit. Recently, generative AI models have received widespread attention, as they can generate fashion items for users to complete an incomplete outfit or create a complete outfit. However, they have limitations in terms of lacking diversity and relying on the supervised learning paradigm. Recognizing this gap, we propose a novel framework FashionDPO, which fine-tunes the fashion outfit generation model using direct preference optimization. This framework aims to provide a general fine-tuning approach to fashion generative models, refining a pre-trained fashion outfit generation model using automatically generated feedback, without the need to design a task-specific reward function. To make sure that the feedback is comprehensive and objective, we design a multi-expert feedback generation module which covers three evaluation perspectives, i.e., quality, compatibility and personalization. Experiments on two established datasets, i.e., iFashion and Polyvore-U, demonstrate the effectiveness of our framework in enhancing the model's ability to align with users' personalized preferences while adhering to fashion compatibility principles. Our code and model checkpoints are available at https://github.com/Yzcreator/FashionDPO.
MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation
Authors: Qinhan Yu, Zhiyou Xiao, Binghui Li, Zhengren Wang, Chong Chen, Wentao Zhang
Venue: SIGIR
Year: 2025

Recent advances in Retrieval-Augmented Generation (RAG) have significantly improved response accuracy and relevance by incorporating external knowledge into Large Language Models (LLMs). However, existing RAG methods primarily focus on generating text-only answers, even in Multimodal Retrieval-Augmented Generation (MRAG) scenarios, where multimodal elements are retrieved to assist in generating text answers. To address this, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which we aim to generate multimodal answers that combine both text and images, fully leveraging the multimodal data within a corpus. Despite growing attention to this challenging task, a notable lack of a comprehensive benchmark persists for effectively evaluating its performance. To bridge this gap, we provide MRAMG-Bench, a meticulously curated, human-annotated benchmark comprising 4,346 documents, 14,190 images, and 4,800 QA pairs, distributed across six distinct datasets and spanning three domains: Web, Academia, and Lifestyle. The datasets incorporate diverse difficulty levels and complex multi-image scenarios, providing a robust foundation for evaluating the MRAMG task. To facilitate rigorous evaluation, MRAMG-Bench incorporates a comprehensive suite of both statistical and LLM-based metrics, enabling a thorough analysis of the performance of generative models in the MRAMG task. Additionally, we propose an efficient and flexible multimodal answer generation framework that can leverage LLMs/MLLMs to generate multimodal responses. Our datasets and complete evaluation results for 11 popular generative models are available at https://github.com/MRAMG-Bench/MRAMG.
Fact-Level Calibration and Correction for Long-Form Generations
Authors: Yige Yuan, Bingbing Xu, Hexiang Tan, Fei Sun, Teng Xiao, Wei Li, Huawei Shen, Xueqi Cheng
Venue: SIGIR
Year: 2025

Large language models (LLMs) have achieved remarkable progress across various domains, yet their tendency to generate hallucinations remains a critical barrier to their practical reliability.Confidence calibration addresses this challenge by aligning a model's confidence with its actual accuracy, improving self-evaluation and trustworthiness.However, traditional confidence calibration, operating at response level, are inadequate for long-form generation, which involve complex outputs composed of multiple atomic facts, each with varying confidence, correctness, and relevance to the query.To overcome this limitation, we propose a fact-level confidence calibration framework that evaluates and adjusts confidence at the granularity of individual facts, incorporating both relevance and correctness.This framework identifies finer-grained calibration discrepancies, reduces overconfidence, and reveals confidence variance.Based on this framework, we introduce CARE (Confidence-Aware Fact Correction), a method that leverages high-confidence facts to iteratively refine and correct low-confidence ones.Experimental results demonstrate that our CARE effectively improves the quality of generated content.Our code is available at this link.
Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commerce
Authors: Jingying Zeng, Zhenwei Dai, Hui Liu, Samarth Varshney, Zhiji Liu, Chen Luo, Zhen Li, Qi He, Xianfeng Tang
Venue: SIGIR
Year: 2025

Prompting LLMs offers an efficient way to guide output generation without explicit model training. In the e-commerce domain, prompt-based applications are widely used in query understanding, recommender systems, and customer support. However, adapting LLMs to different tasks often requires extensive prompt engineering by domain experts, along with frequent updates to align with evolving business needs. Additionally, crafting fully unbiased natural language prompts remains a challenge for humans. To address these challenges, we propose a novel framework, Examples as the Prompt (EaP). Specifically, EaP automatically selects the most representative examples to maximize the few-shot capability of LLMs. It is efficient due to its unsupervised example selection and adaptive to potential data distribution shifts. We validate EaP on four real-world production use cases, demonstrating that it achieves comparable or even superior performance comparing to hand-crafted prompts designed by domain experts. Additionally, we introduce EaPlite, which entirely replaces the natural language components of prompts with labeled examples. EaPlite improves LLM inference speed by up to 70% without compromising performance. The online A/B test shows that using EaP and EaPlite for data labeling can bring significant composite revenue gain by 0.06%.
Interactive Code Information Integrated Programming Knowledge Tracing
Authors: Xueqiang Zeng, Guangcheng Fu, Haofei Chen, Qiyun Peng, Mingwen Wang
Venue: SIGIR
Year: 2025

Programming Knowledge Tracing (PKT) aims to estimate students' programming proficiency by analyzing their historical coding activities. It is important to leverage interactive information for PKT code representations, as code submissions are problem-specific interactive solutions evaluated by Online Judge (OJ) feedback. However, most existing PKT models only rely on pre-trained language models to capture static code information, overlooking feedback scores and problem-specific context. To address this issue, we propose an Interactive Information integrated Code Embedding for Programming Knowledge Tracing (IICE-PKT). This model learns a unified code representation by integrating comprehensive interactive information, including code text, OJ feedback scores, problem statement text, problem-skill correlations, and problem difficulties. IICE-PKT consists of three modules: Problem Representation module generates enhanced problem embeddings based on problem-skill correlations and problem difficulties; Code Representation module combines fine-tuned CodeBERT-based code text embeddings, GPT-generated problem text embeddings, and enhanced problem embeddings with supervised feedback scores to produce a unified code representation; Dual-Sequence Modeling module employs two independent GRUs to separately model the problem sequence and the code sequence. Extensive experiments demonstrate the superiority of IICE-PKT. Ablation studies confirm that integrating interactive information into code representations significantly enhances effectiveness.
Scaling Sparse and Dense Retrieval in Decoder-Only LLMs
Authors: Hansi Zeng, Julian Killingback, Hamed Zamani
Venue: SIGIR
Year: 2025

Scaling large language models (LLMs) has shown great potential for improving retrieval model performance; however, previous studies have mainly focused on dense retrieval trained with contrastive loss (CL), neglecting the scaling behavior of other retrieval paradigms and optimization techniques, such as sparse retrieval and knowledge distillation (KD). In this work, we conduct a systematic comparative study on how different retrieval paradigms (sparse vs. dense) and fine-tuning objectives (CL vs. KD vs. their combination) affect retrieval performance across different model scales. Using MSMARCO passages as the training dataset, decoder-only LLMs (Llama-3 series: 1B, 3B, 8B), and a fixed compute budget, we evaluate various training configurations on both in-domain (MSMARCO, TREC DL) and out-of-domain (BEIR) benchmarks. Our key findings reveal that: (1) Scaling behaviors emerge clearly only with CL, where larger models achieve significant performance gains, whereas KD-trained models show minimal improvement, performing similarly across the 1B, 3B, and 8B scales. (2) Sparse retrieval models consistently outperform dense retrieval across both in-domain (MSMARCO, TREC DL) and out-of-domain (BEIR) benchmarks, and they demonstrate greater robustness to imperfect supervised signals. (3) We successfully scale sparse retrieval models with the combination of CL and KD losses at 8B scale, achieving state-of-the-art (SOTA) results in all evaluation sets.
Preference-Strength-Aware Self-Improving Alignment with Generative Preference Models
Authors: Yuanzhao Zhai, Zhuo Zhang, Cheng Yang, Kele Xu, Yue Yu, Wei Li, Hui Wang, Zenglin Xu, Dawei Feng, Bo Ding, Huaimin Wang
Venue: SIGIR
Year: 2025

Self-improving alignment leveraging large language models (LLMs) to automatically generate synthetic preference data has garnered significant attention as a means of reducing reliance on human labelers. These methods typically employ the LLM-as-a-judge mechanism, where the LLM generates responses and then employs itself to judge which response best aligns with the given prompt for curating the binary self-preferred dataset. However, these methods encounter two major challenges: (1) LLM-as-a-judge often produces error-prone evaluations, resulting in low-quality preference annotation, and (2) their optimization strategies often overlook the strength of preferences within binary pairs, leading to overfitting. This paper proposes a novel method, Preference-Strength-aware Optimization (PSO), to address these issues. Specifically, PSO frames the preference annotation process as a judgment token prediction task using the generative preference model to produce reliable judgments. The predicted judgment token indicates the preferred response and its corresponding probability reflects the disparity between responses, referred to as preference strength. Based on this strength, we introduce a new preference-strength-aware loss to adaptively reweight the impact of different response pairs on optimization, concentrating the model's learning on high-quality response pairs. Our experiments demonstrate that PSO significantly improves performance in preference benchmarks, achieving stronger alignment with human preferences, reducing verbose responses, and mitigating overfitting. Furthermore, PSO exhibits robust generalization and sample efficiency, offering a scalable and promising solution for LLM alignment without relying on human-annotated preferences.
Information Retrieval for Artificial General Intelligence: A New Perspective of Information Retrieval Research
Authors: ChengXiang Zhai
Venue: SIGIR
Year: 2025

Traditionally, the users of an information retrieval (IR) system have been human users. We present a new perspective on IR research in which the users of an IR system are intelligent agents instead of human users. Extending the current work on retrieval-augmented generation (RAG), we identify five novel IR tasks that an intelligent agent must be able to perform in order to achieve Human-Level Artificial Intelligence, or Artificial General Intelligence (AGI), including 1) External Information Retrieval (EIR) to access new information unseen by the agent, 2) Provenance Information Retrieval (PIR) to trace the provenance of information, 3) Curriculum Information Retrieval (CIR) to actively acquire the most useful new data and information for lifelong learning, 4) Rule Information Retrieval (RIR) to perform reasoning and problem solving, and 5) Scenario Information Retrieval (SIR) to leverage past scenarios for problem solving and decision making. We compare these new IR tasks with the traditional IR tasks performed by an IR system that serves human users and systematically examine the challenges involved in the five new IR tasks, providing a roadmap for new IR research within the broader context of AGI development.
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Authors: Yuanzhao Zhai, Huanxi Liu, Zhuo Zhang, Tong Lin, Kele Xu, Cheng Yang, Dawei Feng, Bo Ding, Huaimin Wang
Venue: SIGIR
Year: 2025

Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further actions via self-critique and refinement. Through multiple episodes of SLSC-MCTS, LLM agents can effectively utilize step-level critiques while disregarding ineffective ones based on node values, thereby incorporating the critiques more robustly. Additionally, our method further empowers LLM agents in a self-training manner, collecting training data from the constructed decision tree to iteratively fine-tune the LLM agents. The self-training data gathered via SLSC-MCTS is diverse and high-quality, which further enhances the reasoning, critiquing, and refining abilities of LLM agents. Experimental results demonstrate that SLSC-MCTS significantly improves LLM agents during testing, surpassing state-of-the-art baselines and achieving shorter task completion trajectories across information retrieval benchmarks such as WebShop and HotPotQA. After three iterations of self-training, LLM agents established by Llama-3.1-8B-Instruct show substantial improvement, even surpassing human experts in WebShop.
Towards Better Evaluating Multi-Query Sessions: A Measure Based on the Theory of Planned Behavior
Authors: Wenbo Zhang, Fan Zhang, Jia Chen, Wei Lu
Venue: SIGIR
Year: 2025

To evaluate multi-query sessions, recent studies usually add a second ''session'' dimension to the query-level evaluation framework, deriving corresponding session-version evaluation metrics such as sDCG, sRBP, and sINST. However, these existing metrics do not sufficiently consider the different impacts of users' expectations of gains and costs on their behaviors such as query reformulation, nor the bounded rationality characteristic of users in expectation management. To address these issues and better explain user behavior in multi-query sessions, we design a user model based on the Theory of Planned Behavior (TPB), which links user expectations to user behaviors. Within the TPB framework, we propose sTPB, a new measure that adapts to users' expectation management modes by considering users' expectations of gains and costs. To demonstrate the effectiveness of sTPB in evaluating multi-query sessions, we compare it with existing session metrics on two publicly available user search behavior datasets. The results show that sTPB significantly outperforms other metrics in terms of both fitting user behavior and measuring user satisfaction. Additionally, we explore the differences between optimal parameters under different user characteristics and task types in session search evaluation. We find that different user characteristics and task types lead to various preferences in users' choices between continuing to examine results and reformulating queries. Our study not only validates the effectiveness of sTPB in evaluating multi-query sessions but also highlights the necessity of considering the influence of user characteristics and task types when designing metrics.
UTCS: Effective Unsupervised Temporal Community Search with Pre-training of Temporal Dynamics and Subgraph Knowledge
Authors: Yue Zhang, Yankai Chen, Yingli Zhou, Yucan Guo, Xiaolin Han, Chenhao Ma
Venue: SIGIR
Year: 2025

In many real-world applications, the evolving relationships between entities can be modeled as temporal graphs, where each edge has a timestamp representing the interaction time. As a fundamental problem in graph analysis, community search (CS) in temporal graphs has received growing attention but exhibits two major limitations: (1) Traditional methods typically require predefined subgraph structures, which are not always known in advance. (2) Learning-based methods struggle to capture temporal interaction information. To fill this research gap, in this paper, we propose an effective Unsupervised Temporal Community Search with pre-training of temporal dynamics and subgraph knowledge model (UTCS ). UTCS contains two key stages: offline pre-training and online search. In the first stage, we introduce multiple learning objectives to facilitate the pre-training process in the unsupervised learning setting. In the second stage, we identify a candidate subgraph and compute community scores using the pre-trained node representations and a novel scoring mechanism to determine the final community members. Experiments on five real-world datasets demonstrate the effectiveness of the proposed method.
PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usage
Authors: Wenyi Zhang, Ju Jia, Xiaojun Jia, Yihao Huang, Xinfeng Li, Cong Wu, Lina Wang
Venue: SIGIR
Year: 2025

The multimodal datasets can be leveraged to pre-train large-scale vision-language models by providing cross-modal semantics. Current endeavors for determining the usage of datasets mainly focus on single-modal dataset ownership verification through intrusive methods and non-intrusive techniques, while cross-modal approaches remain under-explored. Intrusive methods can adapt to multimodal datasets but degrade model accuracy, while non-intrusive methods rely on label-driven decision boundaries that fail to guarantee stable behaviors for verification. To address these issues, we propose a novel prompt-adapted transferable fingerprinting scheme from a training-free perspective, called PATFinger, which incorporates the global optimal perturbation (GOP) and the adaptive prompts to capture dataset-specific distribution characteristics. Our scheme utilizes inherent dataset attributes as fingerprints instead of compelling the model to learn triggers. The GOP is derived from the sample distribution to maximize embedding drifts between different modalities. Subsequently, our PATFinger re-aligns the adaptive prompt with GOP samples to capture the cross-modal interactions on the carefully crafted surrogate model. This allows the dataset owner to check the usage of datasets by observing specific prediction behaviors linked to the PATFinger during retrieval queries. Extensive experiments demonstrate the effectiveness of our scheme against unauthorized multimodal dataset usage on various cross-modal retrieval architectures by 30% over state-of-the-art baselines.
Hybrid Advertising in the Sponsored Search
Authors: Zhen Zhang, Weian Li, Yuhan Wang, Qi Qi, Kun Huang
Venue: SIGIR
Year: 2025

Online advertisements are a primary revenue source for e-commerce platforms. Traditional advertising models are store-centric, selecting winning stores through auction mechanisms. Recently, a new approach known as joint advertising has emerged, which presents sponsored bundles combining one store and one brand in ad slots. Unlike traditional models, joint advertising allows platforms to collect payments from both brands and stores. However, each of these two advertising models appeals to distinct user groups, leading to low click-through rates when users encounter an undesirable advertising model. To address this limitation and enhance generality, we propose a novel advertising model called ''Hybrid Advertising''. In this model, each ad slot can be allocated to either an independent store or a bundle. To find the optimal auction mechanisms in hybrid advertising, while ensuring nearly dominant strategy incentive compatibility and individual rationality, we introduce the Hybrid Regret Network (HRegNet), a neural network architecture designed for this purpose. Extensive experiments on both synthetic and real-world data demonstrate that the mechanisms generated by HRegNet significantly improve platform revenue compared to established baseline methods.