OpenRaiser/Intern-Atlas huggingface dataset:
A Cost-Effective Framework to Evaluate LLM-Generated Relevance Judgements
Authors: Simone Merlo, Stefano Marchesin, Guglielmo Faggioli, Nicola Ferro
Venue: CIKM
Year: 2025

Large Language Models (LLMs) hugely impacted many research fields, including Information Retrieval (IR), where they are used for many sub-tasks, such as query rewriting and retrieval augmented generation. At the same time, the research community is investigating whether and how to use LLMs to support, or even replace, humans to generate relevance judgments. Indeed, generating relevance judgements automatically - or integrating an LLM in the annotation process - would allow us to improve the number of evaluation collections, also for scenarios where the annotation process is particularly challenging. To validate relevance judgements produced by an LLM they are compared with human-made relevance judgements, measuring the inter-assessor agreement between the human and the LLM. Our work introduces an innovative framework for estimating the quality of LLM-generated relevance judgments, providing statistical guarantees while minimizing human involvement. The proposed framework allows to: i) estimate the quality of LLM-generated relevance judgments with a defined confidence while minimizing human involvement; and ii) estimate the quality of LLM-generated relevance judgments with a fixed budget while providing bounds on the estimate. Our experimental results on three well-known IR collections using multiple LLMs as assessors show it is sufficient to assess 16% of the LLM-generated relevance judgments to estimate the LLM's performance with a 95% confidence.
SmaLLEXT: 1st Workshop on Small and Efficient Large Language Models for Knowledge Extraction
Authors: Felice Antonio Merra, Kristian Skracic, Daniele Malitesta, Jacek Golebiowski, Pasquale Minervini
Venue: CIKM
Year: 2025

The SmaLLEXT workshop (Small and Efficient LLMs for Knowledge Extraction) brings together researchers and practitioners working on compact models that can run under tight memory- and latency budgets while still delivering state-of-the-art accuracy in extracting structured knowledge from unstructured sources. Today's production pipelines-in finance, healthcare, legal, and web-scale analytics-need extraction systems that are fast, verifiable, and economical; current 10-B+-parameter models remain out of reach for many such settings. This workshop focuses on techniques for building smaller, faster, and more adaptable LLMs tailored to knowledge extraction (KE) tasks. Core topics include model compression, knowledge distillation, efficient fine-tuning, optimized retrieval-augmented generation (RAG), and hybrid symbolic-neural approaches. Special emphasis will be placed on practical challenges, such as reducing latency, ensuring factual consistency, and improving robustness in noisy or low-resource settings. By convening a diverse group of experts, the workshop aims to deepen the community's understanding of how lightweight LLMs can be effectively applied to extract, structure, and reason over knowledge at scale. Participants will gain exposure to recent breakthroughs, real-world deployments, and emerging research directions, fostering collaboration around the building of deployable high-impact KE systems in both academic and industrial contexts.
Reverse Chain-of-Thought and Causal Path Verification: A Modular Plugin for Aligning LLMs with Knowledge Graphs
Authors: Dezhuang Miao, Yibin Du, Xiang Li, Xiaoming Zhang, Jiahe Li, Bo Zhang, Bingyu Yan, Lian Zhang, Litian Zhang
Venue: CIKM
Year: 2025

Large language models (LLMs) exhibit strong language understanding capabilities, but encounter challenges when integrating structured knowledge from knowledge graphs (KGs) for complex reasoning tasks such as knowledge graph question answering (KGQA). Existing methods often rely on prompt engineering or fixed templates, which obscure the relational structure and limit generalization. To address these limitations, this paper introduces the Reverse Chain-of-Thought (R-CoT) and Causal Path Verification Plugin, a modular framework that reconstructs retrieved KG triples into reverse chains of sub-questions. Each reasoning step is aligned with a supporting triple, forming interpretable multi-hop paths. In particular, Semantic Causal Scoring (SCS) module is further incorporated to evaluate the causal alignment between each reverse sub-question and the original question through dynamic semantic vector matching. The SCS design avoids frequent interactions with LLMs and effectively filters irrelevant or unsupported reasoning steps. Based on the scoring results, a template-free, model-agnostic R-CoT input format is constructed as a semi-structured sequence. This design preserves the KG structure in natural language form and enables seamless integration with standard LLMs without fine-tuning. Experimental results demonstrate that the R-CoT Plugin consistently improves factual alignment, enhances reasoning stability, and outperforms conventional prompt-based methods in both accuracy and coherence.
Towards Adaptive Personalized Conversational Information Retrieval
Authors: Fengran Mo, Yuchen Hui, Yuxing Tian, Zhaoxuan Tan, Chuan Meng, Zhan Su, Kaiyu Huang, Jian-Yun Nie
Venue: CIKM
Year: 2025

Personalized conversational information retrieval (CIR) systems aim to satisfy users' complex information needs through multi-turn interactions by considering user profiles. However, not all search queries require personalization. The challenge lies in appropriately incorporating personalization elements into search when needed. Most existing studies implicitly incorporate users' personal information and conversational context using large language models without distinguishing the specific requirements for each query turn. Such a ''one-size-fits-all'' personalization strategy might lead to sub-optimal results. In this paper, we propose an adaptive personalization method, in which we first identify the required personalization level for a query and integrate personalized queries with other query reformulations to produce various enhanced queries. Then, we design a personalization-aware ranking fusion approach to assign fusion weights dynamically to different reformulated queries, depending on the required personalization level. The proposed Adaptive Personalized Conversational Information Retrieval framework APCIR is evaluated on two TREC iKAT datasets. The results confirm the effectiveness of adaptive personalization of APCIR by outperforming state-of-the-art methods.
EFU: Enforcing Federated Unlearning via Functional Encryption
Authors: Samaneh Mohammadi, Vasileios Tsouvalas, Iraklis Symeonidis, Ali Balador, Tanir Ozcelebi, Francesco Flammini, Nirvana Meratnia
Venue: CIKM
Year: 2025

Federated unlearning (FU) algorithms allow clients in federated settings to exercise their right to be forgotten by removing the influence of their data from a collaboratively trained model. Existing FU methods maintain data privacy by performing unlearning locally on the client-side and sending targeted updates to the server without exposing forgotten data; yet they often rely on server-side cooperation, revealing the client's intent and identity without enforcement guarantees - compromising autonomy and unlearning privacy. In this work, we propose EFU (Enforced Federated Unlearning), a cryptographically enforced FU framework that enables clients to initiate unlearning while concealing its occurrence from the server. Specifically, EFU leverages functional encryption to bind encrypted updates to specific aggregation functions, ensuring the server can neither perform unauthorized computations nor detect or skip unlearning requests. To further mask behavioral and parameter shifts in the aggregated model, we incorporate auxiliary unlearning losses based on adversarial examples and parameter importance regularization. Extensive experiments show that EFU achieves near-random accuracy on forgotten data while maintaining performance comparable to full retraining across datasets and neural architectures - all while concealing unlearning intent from the server. Furthermore, we demonstrate that EFU is agnostic to the underlying unlearning algorithm, enabling secure, function-hiding, and verifiable unlearning for any client-side FU mechanism that issues targeted updates.
LLM-Enhanced Linear Autoencoders for Recommendation
Authors: Jaewan Moon, Seongmin Park, Jongwuk Lee
Venue: CIKM
Year: 2025

Large language models (LLMs) have been widely adopted to enrich the semantic representation of textual item information in recommender systems. However, existing linear autoencoders (LAEs) that incorporate textual information rely on sparse word co-occurrence patterns, limiting their ability to capture rich textual semantics. To address this, we propose L3AE, the first integration of LLMs into the LAE framework. L3AE effectively integrates the heterogeneous knowledge of textual semantics and user-item interactions through a two-phase optimization strategy. (i) L3AE first constructs a semantic item-to-item correlation matrix from LLM-derived item representations. (ii) It then learns an item-to-item weight matrix from collaborative signals while distilling semantic item correlations as regularization. Notably, each phase of L3AE is optimized through closed-form solutions, ensuring global optimality and computational efficiency. Extensive experiments demonstrate that L3AE consistently outperforms state-of-the-art LLM-enhanced models on three benchmark datasets, achieving gains of 27.6% in Recall@20 and 39.3% in NDCG@20. The source code is available at https://github.com/jaewan7599/L3AE_CIKM2025.
Learning Optimal Personalised Reservation Prices in Impression Ad Auctions with Mixture Density Networks
Authors: Dmitrii Moor, Emma Zetterdahl, Paul van Vliet, Zhenwen Dai, Mounia Lalmas
Venue: CIKM
Year: 2025

Reservation prices have proven effective in boosting revenue in Generalised Second Price (GSP) auctions, particularly in cost-per-click (CPC) settings. However, in domains like music streaming, where ads are consumed passively without user clicks, a cost-per-impression (CPM) model is more appropriate. Additionally, in the music streaming domain, user intent is typically unknown, unlike in sponsored search, making it essential to optimally leverage all available user and contextual information when setting prices. This paper addresses the challenge of optimising reservation prices in GSP auctions with CPM pricing, adopting a personalised approach that accounts for both user- and advertiser-specific factors. Using dataset of 100,000 auctions from a major music streaming service, we determine such optimal prices. To achieve this, we first derive the symmetric Nash equilibrium for GSP auctions in a CPM context. We then introduce a Deep Neural Network-based mixture density model that incorporates this equilibrium into its loss. This model captures advertisers' diverse preferences by learning directly from bidding data. We show how this approach enables the computation of personalised prices for both users and advertisers, boosting auction revenue by an average of +4% across ten markets. Our study further highlights the impact of market competitiveness and advertiser preference heterogeneity on revenue gains, showing that personalised pricing greatly enhances auction performance.
Improving Text Embedding Models with Positive-aware Hard-negative Mining
Authors: Gabriel de Souza Pereira Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, Even Oldridge
Venue: CIKM
Year: 2025

Text embedding models have been popular for information retrieval applications such as semantic search and Question-Answering systems based on Retrieval-Augmented Generation (RAG). Those models are typically Transformer models that are fine-tuned with contrastive learning objectives. One of the challenging aspects of fine-tuning embedding models is selecting high quality hard-negative passages for contrastive learning. In this paper we introduce a family of positive-aware mining methods that use the positive relevance score as an anchor for false negative removal. Our methods are simple, effective, scalable, and lead to faster training and more accurate retrieval models. We provide an ablation study on hard-negative mining methods over their configurations, exploring different teacher and base models. We further demonstrate the efficacy of our proposed mining methods at scale with the NV-Retriever-v1 model, which scored 60.9 on the MTEB Retrieval (BEIR) benchmark and placed 1st upon its publication.
Latent Variable Modeling for Robust Causal Effect Estimation
Authors: Tetsuro Morimura, Tatsushi Oka, Yugo Suzuki, Daisuke Moriwaki
Venue: CIKM
Year: 2025

Latent variable models provide a powerful framework for incorporating and inferring unobserved factors in observational data. In causal inference, they help account for hidden factors influencing treatment or outcome, thereby addressing challenges posed by missing or unmeasured covariates. This paper proposes a new framework that integrates latent variable modeling into the double machine learning (DML) paradigm to enable robust causal effect estimation in the presence of such hidden factors. We consider two scenarios: one where a latent variable affects only the outcome, and another where it may influence both treatment and outcome. To ensure tractability, we incorporate latent variables only in the second stage of DML, separating representation learning from latent inference. We demonstrate the robustness and effectiveness of our method through extensive experiments on both synthetic and real-world datasets.
Get Global Guarantees: On the Probabilistic Nature of Perturbation Robustness
Authors: Wenchuan Mu, Kwan Hui Lim
Venue: CIKM
Year: 2025

Robustness is a critical requirement for deploying machine learning models in safety-sensitive domains, where even imperceptible input perturbations can lead to hazardous outcomes. However, existing robustness assessment techniques prior to deployment often face a trade-off between computational feasibility and measurement precision, limiting their effectiveness in practice. To address these limitations, we provide a systematic comparative study of prevailing robustness definitions and their corresponding evaluation methodologies. Building on this analysis, we propose tower robustness, which is a novel and practical concept setting out from a global perspective. Further, we provide upper and lower bounds of tower robustness, based on hypothesis testing, for quantitative evaluation, enabling more rigorous and efficient pre-deployment assessments. Through empirical investigation, we demonstrate that our approach provides reliable robustness assessments. These findings advance the systematic understanding of robustness and contribute a practical framework for enhancing the safety of machine learning models in safety-critical applications.
Bayesian Privacy Guarantee for User History in Sequential Recommendation Using Randomised Response
Authors: Wenchuan Mu, Kwan Hui Lim
Venue: CIKM
Year: 2025

Sequential recommendation systems play an important role in delivering personalised user experiences, yet they rely heavily on detailed user history, raising serious privacy concerns. In this work, we introduce a novel framework that integrates a randomised response mechanism into sequential recommendation to provide strong privacy guarantees while preserving recommendation effectiveness. By obfuscating user history through controlled probabilistic item substitution based on semantic similarity, our approach ensures that released sequences protect individual behaviour with provable Bayesian posterior privacy. We further propose training strategies tailored for privacy-filtered data, including a frequency-based vocabulary expansion method inspired by subword tokenisation. Experiments on four real-world datasets demonstrate that our approach preserves recommendation quality under strong privacy constraints and outperforms existing baselines even without applying privacy filters.
GraFS: An Integrated GNN-LLM Approach for Inferring Best Functional Substitute Products
Authors: Favour Nerrise, Edward W. Huang, Xiaonan Ji, Karthik Subbian, Danai Koutra
Venue: CIKM
Year: 2025

Identifying and ranking the most functionally substitutable products is a key challenge for improving product selection and recommendations in e-commerce. Functionally substitutable products are items that share core functional attributes, which allow them to serve similar purposes despite variations in product features. Traditional methods struggle with (1) accurately capturing functional similarities between products with unique attributes, and (2) ranking substitutes that best optimize product selection and meet customer needs. We introduce GraFS (Graph-enabled Large Language Model framework for Functional Substitute Selection), which combines Large Language Models (LLMs) to extract textual, semantic similarities from product descriptions and Graph Neural Networks (GNNs) that learn substitution patterns from customer behavior. Specifically, LLMs generate functional similarity embeddings from unstructured, product text attributes, while GNNs aggregate these embeddings with graph-structured customer data to predict substitution scores and rank products within functional groups. This dual approach enables GraFS to identify the top-k most suitable functional substitutes that maximize purchase likelihood while maintaining product diversity. Experiments on four large-scale e-commerce reviews datasets demonstrate that our framework significantly improves upon conventional methods and better captures relationships among functional substitute products by up to %19 NDCG@10 and Pre@10 against baselines.
Multi-modal Adaptive Mixture of Experts for Cold-start Recommendation
Authors: Van-Khang Nguyen, Duc-Hoang Pham, Huy-Son Nguyen, Cam-Van Thi Nguyen, Hoang-Quynh Le, Duc-Trong Le
Venue: CIKM
Year: 2025

Recommendation systems have faced significant challenges in cold-start scenarios, where new items with a limited history of interaction need to be effectively recommended to users. Though multimodal data (e.g., images, text, audio, etc.) offer rich information to address this issue, existing approaches often employ simplistic integration methods such as concatenation, average pooling, or fixed weighting schemes, which fail to capture the complex relationships between modalities. Our study proposes a novel Mixture of Experts framework for multimodal cold-start recommendation (MAMEX), which dynamically leverages latent representation from different modalities. MAMEX utilizes modality-specific expert networks and introduces a learnable gating mechanism that adaptively weights the contribution of each modality based on its content characteristics. This approach enables MAMEX to emphasize the most informative modalities for each item while maintaining robustness when certain modalities are less relevant or missing. Extensive experiments on benchmark datasets show that MAMEX outperforms state-of-the-art models with superior accuracy and adaptability.
A Content-Driven Micro-Video Recommendation Dataset at Scale
Authors: Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, Fajie Yuan
Venue: CIKM
Year: 2025

Micro-form videos have emerged as a popular form of content, leading to extensive research in micro-video recommendation with significant implications for the entertainment, advertising, and e-commerce industries. However, the lack of publicly available large-scale micro-video datasets presents a challenge for developing effective recommender systems. To address this challenge, we introduce a comprehensive and diverse micro-video recommendation dataset, referred to as ''MicroLens.'' This dataset comprises nine million user-item interaction behaviors, one million users, and 91 thousand full-length micro-videos. It includes rich modality information such as titles, cover images, and audio associated with the videos. MicroLens serves as a benchmark for the content-driven micro-video recommendation, allowing researchers to leverage diverse video modality information, particularly the raw video features, to enhance the effectiveness of recommender systems. This goes beyond the traditional reliance on item IDs or off-the-shelf pre-extracted video/visual features, providing new avenues for improving recommendation accuracy and personalization. We have conducted extensive experiments on MicroLens, benchmarking multiple recommender models and video encoders, which have provided valuable insights into the performance of micro-video recommendation. We anticipate that this dataset will not only benefit the recommender system community but also foster advancements in the field of video understanding. Our datasets, code, and additional documents are available at https://github.com/westlake-repl/MicroLens.
PriviRec: Confidential and Decentralized Graph Filtering for Recommender Systems
Authors: Julien Nicolas, César Sabater, Mohamed Maouche, Mark Coates, Sonia Ben Mokhtar
Venue: CIKM
Year: 2025

Recent advances in recommender systems have shown that relying on graph filters, such as the normalized item-item adjacency matrix and the ideal low-pass filter yields competitive performance and scales better than Graph Convolutional Networks-based solutions. However, these solutions require centralizing user data, which raises concerns over data privacy, security, and the monopolization of user data by a few actors. To address those concerns, we propose PriviRec and PriviRec-k, two complementary recommendation frameworks. In PriviRec, we show that it is possible to decompose widely used filters so that they can be computed in a distributed setting using Secure Aggregation and a distributed version of the Randomized Power Method, without revealing individual users contributions. PriviRec-k extends this approach by having users securely aggregate low-rank projections of their contributions, enabling a tunable balance between communication overhead and recommendation accuracy. We demonstrate theoretically as well as experimentally on Gowalla, Yelp2018, and Amazon-Book that our methods achieve performance comparable to centralized state-of-the-art recommender systems and superior to decentralized ones, while preserving confidentiality and low communication and computational overheads.
Addressing Personalized Bias for Unbiased Learning to Rank
Authors: Zechun Niu, Lang Mei, Liu Yang, Ziyuan Zhao, Qiang Yan, Jiaxin Mao, Ji-Rong Wen
Venue: CIKM
Year: 2025

Unbiased learning to rank (ULTR), which aims to learn unbiased ranking models from biased user behavior logs, plays an important role in Web search. Previous research on ULTR has studied a variety of biases in users' clicks, such as position bias, presentation bias, and outlier bias. However, existing work often assumes that the behavior logs are collected from an ''average'' user, neglecting the differences between different users in their search and browsing behaviors. In this paper, we introduce personalized factors into the ULTR framework, which we term the user-aware ULTR problem. Through a formal causal analysis of this problem, we demonstrate that existing user-oblivious methods are biased when different users have different preferences over queries and personalized propensities of examining documents. To address such a personalized bias, we propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives. Specifically, our approach models the distribution of user browsing behaviors for each query and aggregates user-weighted examination probabilities to determine propensities. We theoretically prove that the user-aware estimator is unbiased under some mild assumptions and shows lower variance compared to the straightforward way of calculating a user-dependent propensity for each impression. Finally, we empirically verify the effectiveness of our user-aware estimator by conducting extensive experiments on two semi-synthetic datasets and a real-world dataset.