OpenRaiser/Intern-Atlas huggingface dataset:
Relation-aware Blocking for Scalable Recommendation Systems
Authors: Huizhi Liang, Zehao Liu, Thanet Markchom
Venue: CIKM
Year: 2022

Recommender systems contain rich relation information. The multiple relations in a recommender system form a heterogeneous information network. How to efficiently find similar users and items based on hop-n relations in heterogeneous information networks is one significant challenge to develop scalable recommender systems in the era of big data. Hashing has been popularly used for dimensionality reduction and data size reduction. Current hashing techniques mainly focus on hashing for directly related (i.e. hop-1) features. This paper proposes to develop relation-aware hashing techniques to bridge this gap. The proposed approaches use locality sensitive hashing (LSH) and consider hop-n relations in an information network to construct user or item blocks. They help facilitate efficient neighborhood formation and recommendation making. The experiments conducted on a large-scale real-life dataset show that the proposed approaches are effective.
JavaScript&Me, A Tool to Support Research into Code Transformation and Browser Security
Authors: Susana Lima, Ricardo Morla, João Routar
Venue: CIKM
Year: 2022

Doing research into code variations and their applications to browser security is challenging. One of the most important aspects of this research is to choose a relevant dataset on which machine learning algorithms can be applied to yield useful results. Although JavaScript code is widely available on various sources, such as package managers, code hosting platforms, and websites, collecting a large corpus of JavaScript and curating it is not a simple task. We present a novel open-source tool that helps with this task by allowing the automatic and systematic collection, processing, and transformation of JavaScript code. These three steps are performed by independent modules, and each one can be extended to incorporate new features, such as additional code sources, or transformation tools, adding to the flexibility of our tool and expanding its usability. Additionally, we use our tool to create a corpus of around 270k JavaScript files, including regular, minified, and obfuscated code, on which we perform a brief analysis. The conclusions from this analysis show the importance of properly curating a dataset before using it in research tasks, such as machine learning classifiers, reinforcing the relevance of our tool.
PlatoGL: Effective and Scalable Deep Graph Learning System for Graph-enhanced Real-Time Recommendation
Authors: Dandan Lin, Shijie Sun, Jingtao Ding, Xuehan Ke, Hao Gu, Xing Huang, Chonggang Song, Xuri Zhang, Lingling Yi, Jie Wen, Chuan Chen
Venue: CIKM
Year: 2022

Recently, graph neural network (GNN) approaches have received huge interests in recommendation tasks due to their ability of learning more effective user and item representations. However, existing GNN-based recommendation models cannot support real-time recommendation where the model keeps its freshness by continuously training the streaming data that users produced, leading to negative impact on recommendation performance. To fully support graph-enhanced large-scale recommendation in real-time scenarios, a deep graph learning system is required to dynamically store the streaming data as a graph structure and enable the development of any GNN model incorporated with the capabilities of real-time training and online inference. However, such requirements rule out existing deep graph learning solutions. In this paper, we propose a new deep graph learning system called PlatoGL, where (1) an effective block-based graph storage is designed with non-trivial insertion/deletion mechanism for updating the graph topology in-milliseconds, (2) a non-trivial multi-blocks neighbour sampling method is proposed for efficient graph query, and (3) a cache technique is exploited to improve the storage stability. We have deployed PlatoGL in Wechat, and leveraged its capability in various content recommendation scenarios including live-streaming, article and micro-video. Comprehensive experiments on both deployment performance and benchmark performance~(w.r.t. its key features) demonstrate its effectiveness and scalability. One real-time GNN-based model, developed with PlatoGL, now serves the major online traffic in WeChat live-streaming recommendation scenario.
Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences
Authors: Qianying Lin, Wen-Ji Zhou, Yanshi Wang, Qing Da, Qing-Guo Chen, Bing Wang
Venue: CIKM
Year: 2022

Sequential recommendation predicts users' next behaviors with their historical interactions. Recommending with longer sequences improves recommendation accuracy and increases the degree of personalization. As sequences get longer, existing works have not yet addressed the following two main challenges. Firstly, modeling long-range intra-sequence dependency is difficult with increasing sequence lengths. Secondly, it requires efficient memory and computational speeds. In this paper, we propose a Sparse Attentive Memory (SAM) network for long sequential user behavior modeling. SAM supports efficient training and real-time inference for user behavior sequences with lengths on the scale of thousands. In SAM, we model the target item as the query and the long sequence as the knowledge database, where the former continuously elicits relevant information from the latter. SAM simultaneously models target-sequence dependencies and long-range intra-sequence dependencies with O(L) complexity and O(1) number of sequential updates, which can only be achieved by the self-attention mechanism with O(L2) complexity. Extensive empirical results demonstrate that our proposed solution is effective not only in long user behavior modeling but also on short sequences modeling. Implemented on sequences of length 1000, SAM is successfully deployed on one of the largest international E-commerce platforms. This inference time is within 30ms, with a substantial 7.30% click-through rate improvement for the online A/B test. To the best of our knowledge, it is the first end-to-end long user sequence modeling framework that models intra-sequence and target-sequence dependencies with the aforementioned degree of efficiency and successfully deployed on a large-scale real-time industrial recommender system.
Task Assignment with Federated Preference Learning in Spatial Crowdsourcing
Authors: Jiaxin Liu, Liwei Deng, Hao Miao, Yan Zhao, Kai Zheng
Venue: CIKM
Year: 2022

Spatial Crowdsourcing (SC) is ubiquitous in the online world today. As we have transitioned from crowdsourcing applications (e.g., Wikipedia) to SC applications (e.g., Uber), there is a substantial precedent that SC systems have a responsibility not only to effective task assignment but also to privacy protection. To address these often-conflicting responsibilities, we propose a framework, Task Assignment with Federated Preference Learning, which performs task assignment based on worker preferences while keeping the data decentralized and private in each platform center (e.g., each delivery center of an SC company). The framework includes two phases, i.e., a federated preference learning and a task assignment phase. Specifically, in the first phase, we design a local preference model for each platform center based on historical data. Meanwhile, the horizontal federated learning with a client-server structure is introduced to collaboratively train these local preference models under the orchestration of a central server. The task assignment phase aims to achieve effective and efficient task assignment by considering workers' preferences. Extensive evaluations over real data show the effectiveness and efficiency of the paper's proposals.
Ensemble Learning Methods for Dirty Data
Authors: Ling Liu
Venue: CIKM
Year: 2022

Neural network ensemble is a collaborative learning paradigm that utilizes multiple neural networks to solve a complex learning problem. Constructing predictive models with high generalization performance is an important and yet most challenging goal for robust intelligence systems in the presence of dirty data. Given a target learning task, popular approaches have been dedicated to find the top performing model. However, it is difficult in general to estimate the best model when available data is finite, possibly dirty, and insufficient for the problem. In this keynote, I will give an overview of a diversity-centric ensemble learning framework developed at Georgia Tech, including methodologies and algorithms for measuring, enforcing, and combining multiple neural networks by improving generalization performance of the overall system and maximizing ensemble utility and resilience to dirty data.
Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima
Authors: Hengyu Liu, Qiang Fu, Lun Du, Tiancheng Zhang, Ge Yu, Shi Han, Dongmei Zhang
Venue: CIKM
Year: 2022

Learning rate is one of the most important hyper-parameters that has significant influence for neural network training. Learning rate schedules are widely used in real practice to adjust the learning rate according to pre-defined schedules for the fast convergence and good generalization. However, existing learning rate schedules are all heuristic algorithms and lack theoretical support. Therefore, people usually choose the learning rate schedules through multiple ad-hoc trial, and the obtained learning rate schedules are sub-optimal. To boost the performance of the obtained sub-optimal learning rate schedule, we propose a generic learning rate schedule plugin, called LEArning Rate Perturbation (LEAP), which can be applied to various learning rate schedules to improve the model training by introducing a certain perturbation to the learning rate. We found that, with such simple yet effective strategy, training processing exponentially favors flat minima rather than sharp minima with guaranteed convergence, which leads to better generalization ability. In addition, we conduct extensive experiments which show that training with LEAP can improve the performance of various deep learning models on diverse datasets using various learning rate schedules (including constant learning rate).
Multi-Faceted Hierarchical Multi-Task Learning for Recommender Systems
Authors: Junning Liu, Xinjian Li, Bo An, Zijie Xia, Xu Wang
Venue: CIKM
Year: 2022

There have been many studies on improving the efficiency of shared learning in Multi-Task Learning (MTL). Previous works focused on the "micro" sharing perspective for a small number of tasks, while in Recommender Systems (RS) and many other AI applications, we often need to model a large number of tasks. For example, when using MTL to model various user behaviors in RS, if we differentiate new users and new items from old ones, the number of tasks will increase exponentially with multidimensional relations. This work proposes a Multi-Faceted Hierarchical MTL model (MFH) that exploits the multidimensional task relations in large scale MTLs with a nested hierarchical tree structure. MFH maximizes the shared learning through multi-facets of sharing and improves the performance with heterogeneous task tower design. For the first time, MFH addresses the "macro" perspective of shared learning and defines a "switcher" structure to conceptualize the structures of macro shared learning. We evaluate MFH and SOTA models in a large industry video platform of 10 billion samples and hundreds of millions of monthly active users. Results show that MFH outperforms SOTA MTL models significantly in both offline and online evaluations across all user groups, especially remarkable for new users with an online increase of 9.1% in app time per user and 1.85% in next-day retention rate. MFH currently has been deployed in WeSee, Tencent News, QQ Little World and Tencent Video, several products of Tencent. MFH is especially beneficial to the cold-start problems in RS where new users and new items often suffer from a "local overfitting" phenomenon that we first formalize in this paper.
Efficient Non-sampling Expert Finding
Authors: Hongtao Liu, Zhepeng Lv, Qing Yang, Dongliang Xu, Qiyao Peng
Venue: CIKM
Year: 2022

Expert finding aims at seeking potential users to answer new questions in Community Question Answering (CQA) websites. Most existing methods focus on designing matching frameworks between questions and experts, and rely on negative sampling technology for model training. However, sampling would lose lots of useful information about experts and questions, and make these sampling-based methods suffer the bias and non-robust issues, which may lead to an insufficient matching performance for expert findings. In this paper, we propose a novel Efficient Non-sampling Expert Finding model, named ENEF, which could learn accurate representations of questions and experts from whole training data. In our approach, we adopt a rather basic question encoder and a simple matching framework, then an efficient whole-data optimization method is elaborately designed to learn the model parameters without negative sampling with rather a low space and time complexity. Extensive experimental results on four real-world CQA datasets demonstrate that our model ENEF could achieve better performance and faster training efficiency than existing state-of-the-art expert finding methods.
ExpertBert: Pretraining Expert Finding
Authors: Hongtao Liu, Zhepeng Lv, Qing Yang, Dongliang Xu, Qiyao Peng
Venue: CIKM
Year: 2022

Expert Finding is an important task in Community Question Answering (CQA) platforms, which could help route questions to potential expertise users to answer. The key is to model the question content and experts based on their historical answered questions accurately. Recently Pretrained Language Models (PLMs, e.g., Bert) have shown superior text modeling ability and have been used in expert finding preliminary. However, most PLMs-based models focus on the corpus or document granularity during pretraining, which is inconsistent with the downstream expert modeling and finding task. In this paper, we propose an expert-level pretraining language model named ExpertBert, aiming to model questions, experts as well as question-expert matching effectively in a pretraining manner. In our approach, we aggregate the historical answered questions of an expert as the expert-specific input.Besides, we integrate the target question into the input and design a label-augmented Masked Language Model (MLM) task to further capture the matching pattern between question and experts, which makes the pretraining objectives that more closely resemble the downstream expert finding task. Experimental results and detailed analysis on real-world CQA datasets demonstrate the effectiveness of our ExpertBert.
Memory Augmented Graph Learning Networks for Multivariate Time Series Forecasting
Authors: Xiangyue Liu, Xinqi Lyu, Xiangchi Zhang, Jianliang Gao, Jiamin Chen
Venue: CIKM
Year: 2022

Multivariate time series (MTS) forecasting is a challenging task. In MTS forecasting, We need to consider both intra-series temporal correlations and inter-series spatial correlations simultaneously. However, existing methods capture spatial correlations from the local data of the time series, without taking the global historical information of time series into account. In addition, most methods base on graph neural network mining for the temporal correlations tend to the redundancy of information at adjacent time points in the time-series data, which introduces noise. In this paper, we propose a memory augmented graph learning network (MAGL), which captures the spatial correlations in terms of the global historical features of MTS. Specifically, we use a memory unit to learn from the local data of MTS. The memory unit records the global historical features of the time series, which is used to mine the spatial correlations. We also design a temporal feature distiller to reduce the noise in extracting temporal features. We extensively evaluate our model on four real-world datasets, comparing with several state-of-the-art methods. The experimental results show MAGL outperforms the state-of-the-art baseline methods on several datasets.
I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning
Authors: Yang Liu, Zequn Sun, Guangyao Li, Wei Hu
Venue: CIKM
Year: 2022

Knowledge graph (KG) embedding seeks to learn vector representations for entities and relations. Conventional models reason over graph structures, but they suffer from the issues of graph incompleteness and long-tail entities. Recent studies have used pre-trained language models to learn embeddings based on the textual information of entities and relations, but they cannot take advantage of graph structures. In the paper, we show empirically that these two kinds of features are complementary for KG embedding. To this end, we propose CoLE, a Co-distillation Learning method for KG Embedding that exploits the complementarity of graph structures and text information. Its graph embedding model employs Transformer to reconstruct the representation of an entity from its neighborhood subgraph. Its text embedding model uses a pre-trained language model to generate entity representations from the soft prompts of their names, descriptions and relational neighbors. To let the two models promote each other, we propose co-distillation learning that allows them to distill selective knowledge from each other's prediction logits. In our co-distillation learning, each model serves as both a teacher and a student. Experiments on benchmark datasets demonstrate that the two models outperform their related baselines, and the ensemble method CoLE with co-distillation learning advances the state-of-the-art of KG embedding.
TSUPY: Dynamic Climate Network Analysis Library
Authors: Jinshu Liu, Yunlong Xu, Fatemeh Nargesian, Gourab Ghoshal
Venue: CIKM
Year: 2022

A climate network represents the global climate system as a network where nodes are geographical locations each represented by time-series and edges indicate the interactions of time-series. Network science has been applied to climate data to study the dynamics of a climate network. To enable network dynamics analysis on historical and real-time climate data, the core task is the efficient computation and update of correlation matrices and climate networks. We demonstrate tsupy, a Python library, which extends Jupyter Notebook as instrumentation for performing climate network construction and analysis at interactive speed. This demonstration focuses on how tsupy enables dynamic network analysis on climate data. We also show how tsupy can be applied to neuro-imaging to understand the functional connectivity between brain regions.
Deep Learning for Search and Recommendation
Authors: Wei Liu, Kexin Xie, Linsey Pang, James Bailey, Longbing Cao, Yuxi Zhang
Venue: CIKM
Year: 2022

In the current digital world, web search engines and recommendation systems are continuously evolving, opening up new potential challenges every day which require more sophisticated and efficient data mining and machine learning solutions to satisfy the needs of sellers and consumers as well as marketers. The quality of search and recommendation systems impacts customer retention, time on site, and sales volume. For instance, with often sparse conversion rates, highly personalized contents, heterogeneous digital sources, more rigorous and effective models are required to be developed by research engineers and data scientists. At the same time, deep learning has started to show great impact in many industrial applications which are capable of processing complicated, large-scale and real-time data. Deep learning not only provides more opportunities to increase conversion rates and improve revenue through a positive customer experience, but also provides customers with personalized contents along with their personal shopping journey. Due to this rapid growth of the digital world, there is a need to bring professionals together from both academic research and the industry to solve real-world problems. This workshop fosters the development of a strong research community focused on solving deep learning based large-scale web search, personalized search, recommendation and ranking relevance problems that provide superior digital experience to all users.
Social Graph Transformer Networks for Pedestrian Trajectory Prediction in Complex Social Scenarios
Authors: Yao Liu, Lina Yao, Binghao Li, Xianzhi Wang, Claude Sammut
Venue: CIKM
Year: 2022

Pedestrian trajectory prediction is essential for many modern applications, such as abnormal motion analysis and collision avoidance for improved traffic safety. Previous studies still face challenges in embracing high social interaction, dynamics, and multi-modality for achieving high accuracy with long-time predictions. We propose Social Graph Transformer Networks for multi-modal prediction of pedestrian trajectories, where we combine Graph Convolutional Network and Transformer Network by generating stable resolution pseudo-images from Spatio-temporal graphs through a designed stacking and interception method. Specifically, we adopt adjacency matrices to obtain Spatio-temporal features and Transformer for long-time trajectory predictions. As such, we retrain the advantages of both, i.e., the ability to aggregate information over an arbitrary number of neighbors and to conduct complex time-dependent data processing. Our experimental results show that our model reduces the final displacement error and achieves state-of-the-art in multiple metrics. The module's effectiveness is demonstrated through ablation experiments.
Knowledge Enhanced Multi-Interest Network for the Generation of Recommendation Candidates
Authors: Danyang Liu, Yuji Yang, Mengdi Zhang, Wei Wu, Xing Xie, Guangzhong Sun
Venue: CIKM
Year: 2022

Candidate generation task requires that candidates related to user interests need to be extracted in realtime. Previous works usually transform a user's behavior sequence to a unified embedding, which can not reflect the user's multiple interests. Some recent works like Comirec and Octopus use multi-channel structures to capture users' diverse interests. They cluster users' historical behaviors into several groups, claiming that one group represents one interest. However, these methods have some limitations. First, an item may correspond to multiple interests of users, thereby simply allocating it to just one interest group will make the modeling of users' interests coarse-grained and inaccurate. Second, explaining user interests at the level of items is rather vague and not convincing. In this paper, we propose a Knowledge Enhanced Multi-Interest Network: KEMI, which exploits knowledge graphs to help learn users' diverse interest representations via heterogeneous graph neural networks (HGNNs) and a novel dual memory network. Specifically, we use HGNNs to capture the semantic representation of knowledge entities and a novel dual memory network to learn a user's diverse interests from his behavior sequence. Through memory slots of the user memory network and the item memory network, we can learn multiple interests for each user and each item. Meanwhile, by binding the entities to the channels of memory networks, we enable it to be explained from the perspective of the knowledge graph, which enhances the interpretability and understanding of user interests. We conduct extensive experiments on two industrial and publicly available datasets. Experimental results demonstrate that our model achieves significant improvements over state-of-the-art baseline models.