OpenRaiser/Intern-Atlas huggingface dataset:
GongBu: Easily Fine-tuning LLMs for Domain-specific Adaptation
Authors: Bolin Zhang, Yimin Tian, Shengwei Wang, Zhiying Tu, Dianhui Chu, Zhiqi Shen
Venue: CIKM
Year: 2024

Parameter-Efficient Fine-Tuning (PEFT) adapts large language models (LLMs) to specific domains by updating only a small portion of the parameters. To easily and efficiently adapt LLMs to custom domains, we present a no-code fine-tuning platform, GongBu, supporting 9 PEFT methods and open-source LLMs. GongBu allows LLM fine-tuning through a user-friendly GUI, eliminating the need to write any code. Its features include data selection, accelerated training speed, decoupled deployment, performance monitoring, and error log analysis. The demonstration video is available at https://www.youtube.com/watch?v=QuDR_WNoB9o.
An Enhanced Batch Query Architecture in Real-time Recommendation
Authors: Qiang Zhang, Zhipeng Teng, Disheng Wu, Jiayin Wang
Venue: CIKM
Year: 2024

In industrial recommendation systems on websites and apps, it is essential to recall and predict top-n results relevant to user interests from a content pool of billions within milliseconds. To cope with continuous data growth and improve real-time recommendation performance, we have designed and implemented a high-performance batch query architecture for real-time recommendation systems. Our contributions include optimizing hash structures with a cacheline-aware probing method to enhance coalesced hashing, as well as the implementation of a hybrid storage key-value service built upon it. Our experiments indicate this approach significantly surpasses conventional hash tables in batch query throughput, achieving up to 90% of the query throughput of random memory access when incorporating parallel optimization. The support for NVMe, integrating two-tier storage for hot and cold data, notably reduces resource consumption. Additionally, the system facilitates dynamic updates, automated sharding of attributes and feature embedding tables, and introduces innovative protocols for consistency in batch queries, thereby enhancing the effectiveness of real-time incremental learning updates. This architecture has been deployed and in use in the bilibili recommendation system for over a year, a video content community with hundreds of millions of users, supporting 10x increase in model computation with minimal resource growth, improving outcomes while preserving the system's real-time performance.
HGCH: A Hyperbolic Graph Convolution Network Model for Heterogeneous Collaborative Graph Recommendation
Authors: Lu Zhang, Ning Wu
Venue: CIKM
Year: 2024

User-item interaction data in collaborative filtering and graph modeling tasks often exhibit power-law characteristics, which suggest the suitability of hyperbolic space modeling. Hyperbolic Graph Convolution Neural Networks (HGCNs) are a novel technique that leverages the advantages of GCN and hyperbolic space, and then achieves remarkable results. However, existing HGCN methods have several drawbacks: they fail to fully leverage hyperbolic space properties due to arbitrary embedding initialization and imprecise tangent space aggregation; they overlook auxiliary information that could enrich the collaborative graph; and their training convergence is slow due to margin ranking loss and random negative sampling. To overcome these challenges, we propose Hyperbolic Graph Collaborative for Heterogeneous Recommendation (HGCH), an enhanced HGCN-based model for collaborative filtering that integrates diverse side information into a heterogeneous collaborative graph and improves training convergence speed. HGCH first preserves the long-tailed nature of the graph by initializing node embeddings with power law prior; then it aggregates neighbors in hyperbolic space using the gyromidpoint method for accurate computation; finally, it fuses multiple embeddings from different hyperbolic spaces by the gate fusion with prior. Moreover, HGCH employs a hyperbolic user-specific negative sampling to speed up convergence. We evaluate HGCH on four real datasets, and the results show that HGCH achieves competitive results and outperforms leading baselines, including HGCNs. Extensive ablation studies further confirm its effectiveness.
Momentum Contrastive Bidirectional Encoding with Self-Distillation for Sequential Recommendation
Authors: Dingyi Zhang, Haoyu Wenren, Yue Wang, Yingming Li
Venue: CIKM
Year: 2024

In this paper, we propose a new Momentum Contrastive Bidirectional Encoding network with S elf-D istillation (MoCoBE-SD) to alleviate the data sparsity and noise issues in sequential recommendation by providing rich informative supervisions from both sequence-level and item-level perspectives. In particular, a Momentum Contrastive Bidirectional Encoding (MoCoBE) network is first proposed by constructing momentum updated encoder based on an online bidirectional self-attention encoder, where a momentum contrastive learning task and a masked item prediction task are simultaneously optimized. Building upon MoCoBE, a well-elaborated Self-Distillation (SD) scheme is incorporated to further suppress the noise influence. Specifically, a well-trained sequence encoder by MoCoBE is adopted as the teacher encoder to provide refined supervision for the masked item prediction, which constitutes our MoCoBE-SD framework. Extensive experiments on three public datasets show that MoCoBE-SD outperforms the existing state-of-the-art methods consistently.
InfoMLP: Unlocking the Potential of MLPs for Semi-Supervised Learning with Structured Data
Authors: Hengrui Zhang, Qitian Wu, Chenxiao Yang, Philip S. Yu
Venue: CIKM
Year: 2024

We introduce InfoMLP, an innovative model structured like a Multilayer Perceptron (MLP) for semi-supervised classification of structured data, e.g., graphs. InfoMLP was inspired by our observation that overlapping information between node features and the structure between data points significantly influences the performance gap between feature-only MLPs and advanced graph-based semi-supervised methods, e.g., GNNs. To quantify the overlapping information, we first introduce a tractable metric to quantify the mutual information between node features and graph structure. Based on this, we propose InfoMLP, which seeks to maximize the mutual information between node embeddings derived from the MLP and the structure information. Our info-max objective is split into two sub-objectives: the first is a non-parametric preprocessing step aiming to find the optimal graph-augmented node feature matrix that captures the maximal information about the graph structure, while the second sub-objective is to maximize the mutual information between node embeddings generated from the original node features and those from the graph-augmented node features. Since the message-passing operation is integrated into the preprocessing step, requiring only a single execution per dataset, InfoMLP maintains the same efficiency as a vanilla MLP during both training and testing. We validate the efficacy of our design through experiments on real-world datasets of varying scales supplemented by comprehensive ablation studies. Our results corroborate our analysis and demonstrate the effectiveness of our novel approach.
Effective Utilization of Large-scale Unobserved Data in Recommendation Systems
Authors: Feng Zhang, Yulin Xu, Hongjie Chen, Xu Yuan, Qingwen Liu, Yuning Jiang
Venue: CIKM
Year: 2024

Ranking models play an important role in industrial recommendation systems. However, most ranking models are trained only with the observed items but used to retrieve all items in the entire space, which may suffer from the sample selection bias and the exposure bias. Inspired by the entire space learning framework, we carry out detailed data analyses on large-scale unobserved items and find that they contain quite a few "potentially-positive" samples. In this paper, we propose an "Extract and Transfer" (EAT) framework, utilizing quantities of unobserved items and other domains' data to construct more training data for ranking models. Specifically, we first extract "potentially-positive" samples and negative ones according to their ranking scores from the unobserved data, and then design an Entire Space Transfer Learning (ESTL) model to transfer knowledge between observed and unobserved samples, instead of directly mixing them together to avoid negative transfer. Experiments on production data collected from Taobao validate the proposed method's superiority. Besides, we have deployed EAT on the Taobao recommendation system, obtaining 6.22% IPV (Item Page View) and 3.77% CTR improvement. The code is available at https://github.com/Recommender1/EAT.git1.
P-Rank+: A Scalable Efficient P-Rank Search Algorithm
Authors: Maoyin Zhang, Weiren Yu
Venue: CIKM
Year: 2024

P-Rank (Penetrating-Rank) is a charming measure of structural similarity between objects based on graph topology. It recursively follows the principle that "two objects are considered similar if (a) they are referenced by similar objects and (b) they reference similar objects''. The best-known algorithm for computing P-Rank employs two repeated Singular Value Decompositions (SVDs) coupled with the Woodbury matrix identity. However, this method does not scale well on billion-sized graphs. Worse yet, this algorithm only provides a linear approximation of the P-Rank model and cannot deliver accurate P-Rank values. In this paper, we propose P-Rank+, a fast and efficient algorithm for computing P-Rank similarities, which scales well on large graphs with billions of edges. P-Rank+ leverages dimensionality reduction techniques by performing only one SVD of the graph integrated with Hadamard products in the reduced subspace. Moreover, we provide provable error guarantees for P-Rank+ computation. Experiments on various datasets validate that P-Rank+ is 1--3 orders of magnitude faster than the best-known competitor while achieving excellent scalability on massive graphs.
Advancing Multivariate Time Series Anomaly Detection: A Comprehensive Benchmark with Real-World Data from Alibaba Cloud
Authors: Chaoli Zhang, Yingying Zhang, Lanshu Peng, Qingsong Wen, Yiyuan Yang, Chong-Jiong Fan, Minqi Jiang, Lunting Fan, Liang Sun
Venue: CIKM
Year: 2024

Time series anomaly detection is of significant importance in many real-world applications, including finance, healthcare, network security, industrial equipment, complex computing systems, and space probes. Most of these applications involve multi-sensor systems, thus how to perform multivariate time series anomaly detection (MTSAD) has garnered widespread attention. This broad attention has fueled extensive research endeavors aimed to innovate and develop methods and techniques to improve the efficiency and precision of anomaly detection on multivariate time series data, including both classic machine learning methods and deep learning methods. However, evaluating the performance of these methods remains challenging due to the limited availability of public benchmark datasets for MTSAD, which are often criticized for various reasons. Additionally, there is no consensus on the best metrics for time series anomaly detection, further complicating MTSAD research. In this paper, we advance the benchmarking of time series anomaly detection by addressing datasets, evaluation metrics, and algorithm comparison. To the best of our knowledge, we have generated the largest real-world datasets for MTSAD using the Hologres AIOps system in the Alibaba Cloud platform. We review and compare popular evaluation metrics including recently proposed ones. To evaluate classic machine learning and recent deep learning methods fairly, we have conducted extensive comparisons of these methods on various datasets. We believe that our benchmarks and datasets will promote reproducible results and accelerate the progress of MTSAD research.
M3: A Multi-Image Multi-Modal Entity Alignment Dataset
Authors: Shiqi Zhang, Weixin Zeng, Zhen Tan, Xiang Zhao, Weidong Xiao
Venue: CIKM
Year: 2024

Multi-modal Entity Alignment (MMEA) aims to identify equivalent entities across different multi-modal knowledge graphs (MMKGs), facilitating their integration and enhancing coverage. However, current MMEA datasets have limitations, including low entity coverage, a single image per entity, high inter-image correlation, and images sourced from the same search engine, which do not reflect real-world challenges. The fair comparison and development of alignment solutions may be hindered by these oversimplified scenarios. To address this problem, in this work, we first construct M3, an MMEA benchmark equipped with multiple images from different search engines in real-world scenarios. Additionally, we design a simple and universal multi-image processing module (AMIA), which assigns varying attention weights to images associated with entities to effectively model visual information. Experimental results validate the difficulty of M3, as well as the effectiveness of AMIA. Despite the superior performance of AMIA, there is still room for developing more advanced solutions to address these difficulties. Our dataset is publicly released.
Preference Prototype-Aware Learning for Universal Cross-Domain Recommendation
Authors: Yuxi Zhang, Ji Zhang, Feiyang Xu, Lvying Chen, Bohan Li, Lei Guo, Hongzhi Yin
Venue: CIKM
Year: 2024

Cross-domain recommendation (CDR) aims to suggest items from new domains that align with potential user preferences, based on their historical interactions. Existing methods primarily focus on acquiring item representations by discovering user preferences under specific, yet possibly redundant, item features. However, user preferences may be more strongly associated with interacted items at higher semantic levels, rather than specific item features. Consequently, this item feature-focused recommendation approach can easily become suboptimal or even obsolete when conducting CDR with disturbances of these redundant features. In this paper, we propose a novel Preference Prototype-Aware (PPA) learning method to quantitatively learn user preferences while minimizing disturbances from the source domain. The PPA framework consists of two complementary components: a mix-encoder and a preference prototype-aware decoder, forming an end-to-end unified framework suitable for various real-world scenarios. The mix-encoder employs a mix-network to learn better general representations of interacted items and capture the intrinsic relationships between items across different domains. The preference prototype-aware decoder implements a learnable prototype matching mechanism to quantitatively perceive user preferences, which can accurately capture user preferences at a higher semantic level. This decoder can also avoid disturbances caused by item features from the source domain. The experimental results on public benchmark datasets in different scenarios demonstrate the superiority of the proposed PPA learning method compared to state-of-the-art counterparts. PPA excels not only in providing accurate recommendations but also in offering reliable preference prototypes. Our code is available at https://github.com/zyx-nuaa/PPA-for-CDR.
Language Models-enhanced Semantic Topology Representation Learning For Temporal Knowledge Graph Extrapolation
Authors: Tianli Zhang, Tongya Zheng, Zhenbang Xiao, Zulong Chen, Liangyue Li, Zunlei Feng, Dongxiang Zhang, Mingli Song
Venue: CIKM
Year: 2024

Temporal Knowledge Graph (TKG) extrapolation aims to predict future missing facts based on historical information, which has exhibited both semantics and topology of events. The mainstream methods have advanced the prediction performance by exploring the potential of topology representations of TKGs based on dedicated temporal Graph Neural Networks (GNNs). Until recently, few Language Models (LM) based methods have attempted to model the semantic representations of TKGs, however, lacking specific designs for the topology information. Therefore, we propose a Semantic TOpology REpresentation learning (STORE) framework enhanced by LMs to bridge the gap between the semantics and topology of TKGs. Firstly, we tackle the challenge of long historical facts modeling by a time-aware sampling based on semantic priors to extract concise yet precise facts. Secondly, we handle the challenge of the interaction between topology and semantics by transforming graph representations into virtual tokens that are then integrated with generated prompts and fed into LMs. Finally, multi-head attention is adopted to obtain better semantic topology representations, thereby achieving joint optimization of both temporal GNNs and LMs. Extensive experiments on five datasets show that our STORE outperforms state-of-the-art GNNs- and LM-based methods.
Multi-modal Food Recommendation with Health-aware Knowledge Distillation
Authors: Yixin Zhang, Xin Zhou, Fanglin Zhu, Ning Liu, Wei Guo, Yonghui Xu, Zhiqi Shen, Lizhen Cui
Venue: CIKM
Year: 2024

Food recommendation systems play a pivotal role in shaping dietary salubrity and fostering sustainable lifestyles by recommending recipes and foodstuffs that align with user preferences. Metadata information of a recipe, encompassing multi-modal descriptions, constituent ingredients, and health-related attributes, can furnish a more holistic perspective on the recipe's profile, thereby augmenting recommendation performance. However, existing state-of-the-art methods often overlook the inherent interdependencies between modalities, ingredients, and health factors, leaving the health information pertaining to recipe characteristics underexploited. Notably, our preliminary investigation on two datasets unveiled that the semantic divergence between health-related knowledge and collaborative filtering signals is more pronounced in comparison to other metadata information, thereby potentially impeding the efficacy of food recommendation systems. To address these limitations, we propose HealthRec, a novel multi-modal food recommendation framework with health-aware knowledge distillation. HealthRec employs a global graph representation learning module to capture high-order dependencies across diverse food-related relations, enriching the representations. Subsequently, a co-attention network is leveraged to capture local, recipe-level knowledge transfer between modality-related and ingredient-related embeddings. Additionally, we exploit external supervision signals derived from WHO recommendations, utilizing knowledge distillation during the training phase to transfer local health-aware knowledge into global collaborative embeddings. Extensive experimentation on real-world datasets demonstrates HealthRec's superiority compared to current state-of-the-art recommendation baselines, highlighting its effectiveness in modeling health-aware food recommendations.
Generating Intent-aware Clarifying Questions in Conversational Information Retrieval Systems
Authors: Ziliang Zhao, Zhicheng Dou, Yujia Zhou
Venue: CIKM
Year: 2024

Generating clarifying questions can effectively clarify users' complicated search intent in conversational search systems. However, existing methods based on pre-defined templates are inadequate in understanding explicit user intents, making generated questions monotonous or inaccurate in some cases. In this paper, we define the ''intent'' of a query as a verb representing the potential behavior, action, or task the user may take. We study generating clarifying questions from a new perspective by incorporating the intents explicitly to form ''intent-aware'' questions with high informativeness and accuracy. Since obtaining gold intent-aware questions is expensive, we propose a rule-based method and a continual learning model to generate intent-aware questions as weak supervision signals. The former leverages search results to mine contextual intent-aware words or phrases, and the latter relies on parallel corpora to paraphrase template-based questions by incorporating the intents. The generated weak supervision data are then applied to fine-tune a BART-based model for end-to-end intent-aware question generation. We also explore to prompt a large language model to generate intent-aware questions. Experimental results on a public clarification dataset demonstrate that our proposed methods improve users' search experience compared to existing methods.
NeutronCache: An Efficient Cache-Enhanced Distributed Graph Neural Network Training System
Authors: Chu Zhao, Shengjie Dong, Yuhai Zhao, Yuan Li
Venue: CIKM
Year: 2024

As real-world graph data continues to grow larger and larger, training large graphs in a distributed environment is becoming increasingly prevalent. However, network transmission in a distributed environment can hinder subsequent training steps, resulting in suboptimal training performance. After conducting a comprehensive analysis and experimental demonstration, we have discovered that during the training process, there exist certain data that can be computed once and reused multiple times. In addition, we also found that after a certain iterations of training, the parameter updates during each iteration had minimal effect on the parameters. Based on these findings, we have improved the original implementation and proposed a cache-enhanced distributed graph training system, NeutronCache. It utilizes cached reusable intermediate data and a dynamically adjusted stale embedding reuse strategy, reducing network overhead in distributed systems and accelerating the training process. Through experimental validation, our implementation achieved acceleration ranging from 1.4X to 16.61X on real graph datasets with almost no loss in accuracy.
Zero-shot Knowledge Graph Question Generation via Multi-agent LLMs and Small Models Synthesis
Authors: Runhao Zhao, Jiuyang Tang, Weixin Zeng, Ziyang Chen, Xiang Zhao
Venue: CIKM
Year: 2024

Knowledge Graph Question Generation (KGQG) is the task of generating natural language questions based on the given knowledge graph (KG). Although extensively explored in recent years, prevailing models predominantly depend on labelled data for training deep learning models or employ large parametric frameworks, e.g., Large Language Models (LLMs), which can incur significant deployment costs and pose practical implementation challenges. To address these issues, in this work, we put forward a zero-shot, multi-agent KGQG framework. This framework integrates the capabilities of LLMs with small models to facilitate cost-effective, high-quality question generation. In specific, we develop a professional editorial team architecture accompanied by two workflow optimization tools to reduce unproductive collaboration among LLMs-based agents and enhance the robustness of the system. Extensive experiments demonstrate that our proposed framework derives the new state-of-the-art performance on the zero-shot KGQG tasks, with relative gains of 20.24% and 13.57% on two KGQG datasets, respectively, which rival fully supervised state-of-the-art models.