OpenRaiser/Intern-Atlas huggingface dataset:
Learning to Distill Graph Neural Networks
Authors: Cheng Yang, Yuxin Guo, Yao Xu, Chuan Shi, Jiawei Liu, Chunchen Wang, Xin Li, Ning Guo, Hongzhi Yin
Venue: WSDM
Year: 2023

Graph Neural Networks (GNNs) can effectively capture both the topology and attribute information of a graph, and have been extensively studied in many domains. Recently, there is an emerging trend that equips GNNs with knowledge distillation for better efficiency or effectiveness. However, to the best of our knowledge, existing knowledge distillation methods applied on GNNs all employed predefined distillation processes, which are controlled by several hyper-parameters without any supervision from the performance of distilled models. Such isolation between distillation and evaluation would lead to suboptimal results. In this work, we aim to propose a general knowledge distillation framework that can be applied on any pretrained GNN models to further improve their performance. To address the isolation problem, we propose to parameterize and learn distillation processes suitable for distilling GNNs. Specifically, instead of introducing a unified temperature hyper-parameter as most previous work did, we will learn node-specific distillation temperatures towards better performance of distilled models. We first parameterize each node's temperature by a function of its neighborhood's encodings and predictions, and then design a novel iterative learning process for model distilling and temperature learning. We also introduce a scalable variant of our method to accelerate model training. Experimental results on five benchmark datasets show that our proposed framework can be applied on five popular GNN models and consistently improve their prediction accuracies with 3.12% relative enhancement on average. Besides, the scalable variant enables 8 times faster training speed at the cost of 1% prediction accuracy.
Feature Missing-aware Routing-and-Fusion Network for Customer Lifetime Value Prediction in Advertising
Authors: Xuejiao Yang, Binfeng Jia, Shuangyang Wang, Shijie Zhang
Venue: WSDM
Year: 2023

Nowadays, customer lifetime value (LTV) plays an important role in mobile game advertising, since it can be beneficial to adjust ad bids and ensure that the games are promoted to the most valuable users. Some neural models are utilized for LTV prediction based on the rich user features. However, in the advertising scenario, due to the privacy settings or limited length of log retention, etc, most of existing approaches suffer from the missing feature problem. Moreover, only a small fraction of purchase behaviours can be observed. The label sparsity inevitably limits model expressiveness. To tackle the aforementioned challenges, we propose a feature missing-aware routing-and-fusion network (MarfNet) to reduce the effect of the missing features while training. Specifically, we calculate the missing states of raw features and feature interactions for each sample. Based on the missing states, two missing-aware layers are designed to route samples into different experts, thus each expert can focus on the real features of samples assigned to it. Finally we get the missing-aware representation by the weighted fusion of the experts. To alleviate the label sparsity, we further propose a batch-in dynamic discrimination enhanced (Bidden) loss weight mechanism, which can automatically assign greater loss weights to difficult samples in the training process. Both offline experiments and online A/B tests have validated the superiority of our proposed Bidden-MarfNet.
Ranking-based Group Identification via Factorized Attention on Social Tripartite Graph
Authors: Mingdai Yang, Zhiwei Liu, Liangwei Yang, Xiaolong Liu, Chen Wang, Hao Peng, Philip S. Yu
Venue: WSDM
Year: 2023

Due to the proliferation of social media, a growing number of users search for and join group activities in their daily life. This develops a need for the study on the ranking-based group identification (RGI) task, i.e., recommending groups to users. The major challenge in this task is how to effectively and efficiently leverage the item interaction information from users' and groups' online behaviors, in addition to the information from social interaction between users and groups. Though recent developments of Graph Neural Networks (GNNs) succeed in aggregating both social and user-item interaction simultaneously, they however fail to comprehensively resolve this RGI task. In this paper, we propose a novel GNN-based framework named Contextualized Factorized Attention for Group identification (CFAG). We devise tripartite graph convolution to aggregate information from different types of neighborhoods among users, groups, and items. To cope with the data sparsity issue, we devise a novel propagation augmentation (PA) layer, which is based on our proposed factorized attention mechanism. PA layers efficiently learn the relevance degree of non-neighbor nodes to improve the information propagation to users. Experimental results on three benchmark datasets verify the superiority of CFAG. Additional detailed investigations are conducted to demonstrate the effectiveness of the proposed framework.
"Just To See You Smile": SMILEY, a Voice-Guided GUY GAN
Authors: Qi Yang, Christos Tzelepis, Sergey I. Nikolenko, Ioannis Patras, Aleksandr Farseev
Venue: WSDM
Year: 2023

In this technical demonstration, we present SMILEY, a voice-guided virtual assistant. The system utilizes a deep neural architecture ContraCLIP to manipulate facial attributes using voice instructions, allowing for deeper speaker engagement and smoother customer experience when being used in the "virtual concierge" scenario. We validate the effectiveness of SMILEY and ContraCLIP via a successful real-world case study in Singapore and a large-scale quantitative evaluation.
DGRec: Graph Neural Network for Recommendation with Diversified Embedding Generation
Authors: Liangwei Yang, Shengjie Wang, Yunzhe Tao, Jiankai Sun, Xiaolong Liu, Philip S. Yu, Taiqing Wang
Venue: WSDM
Year: 2023

Graph Neural Network (GNN) based recommender systems have been attracting more and more attention in recent years due to their excellent performance in accuracy. Representing user-item interactions as a bipartite graph, a GNN model generates user and item representations by aggregating embeddings of their neighbors. However, such an aggregation procedure often accumulates information purely based on the graph structure, overlooking the redundancy of the aggregated neighbors and resulting in poor diversity of the recommended list. In this paper, we propose diversifying GNN-based recommender systems by directly improving the embedding generation procedure. Particularly, we utilize the following three modules: submodular neighbor selection to find a subset of diverse neighbors to aggregate for each GNN node, layer attention to assign attention weights for each layer, and loss reweighting to focus on the learning of items belonging to long-tail categories. Blending the three modules into GNN, we present DGRec (Diversified GNN-based Recommender System) for diversified recommendation. Experiments on real-world datasets demonstrate that the proposed method can achieve the best diversity while keeping the accuracy comparable to state-of-the-art GNN-based recommender systems. We open source DGRec at https://github.com/YangLiangwei/DGRec.
Minimum Entropy Principle Guided Graph Neural Networks
Authors: Zhenyu Yang, Ge Zhang, Jia Wu, Jian Yang, Quan Z. Sheng, Hao Peng, Angsheng Li, Shan Xue, Jianlin Su
Venue: WSDM
Year: 2023

Graph neural networks (GNNs) are now the mainstream method for mining graph-structured data and learning low-dimensional node- and graph-level embeddings to serve downstream tasks. However, limited by the bottleneck of interpretability that deep neural networks present, existing GNNs have ignored the issue of estimating the appropriate number of dimensions for the embeddings. Hence, we propose a novel framework called Minimum Graph Entropy principle-guided Dimension Estimation, i.e. MGEDE, that learns the appropriate embedding dimensions for both node and graph representations. In terms of node-level estimation, a minimum entropy function that counts both structure and attribute entropy, appraises the appropriate number of dimensions. In terms of graph-level estimation, each graph is assigned a customized embedding dimension from a candidate set based on the number of dimensions estimated for the node-level embeddings. Comprehensive experiments with node and graph classification tasks and nine benchmark datasets verify the effectiveness and generalizability of MGEDE.
DOCoR: Document-level OpenIE with Coreference Resolution
Authors: Shan Jie Yong, Kuicai Dong, Aixin Sun
Venue: WSDM
Year: 2023

Open Information Extraction (OpenIE) extracts relational fact tuples in the form of from text. Most existing OpenIE solutions operate at sentence level and extract relational tuples solely from a sentence. However, many sentences exist as a part of paragraph or a document, where coreferencing is common. In this demonstration, we present a system which refines the semantic tuples generated by OpenIE with the aid of a coreference resolution tool. Specifically, all coreferential mentions across the entire document are identified and grouped into coreferential clusters. Objects and subjects in the extracted tuples from OpenIE which match any coreferential mentions are then resolved with a suitable representative term. In this way, our system is able to resolve both anaphoric and cataphoric references, to achieve Document-level OpenIE with Coreference Resolution (DOCoR). The demonstration video can be viewed at https://youtu.be/o9ZSWCBvlDs
Variational Reasoning over Incomplete Knowledge Graphs for Conversational Recommendation
Authors: Xiaoyu Zhang, Xin Xin, Dongdong Li, Wenxuan Liu, Pengjie Ren, Zhumin Chen, Jun Ma, Zhaochun Ren
Venue: WSDM
Year: 2023

Conversational recommender systems (CRSs) often utilize external knowledge graphs (KGs) to introduce rich semantic information and recommend relevant items through natural language dialogues. However, original KGs employed in existing CRSs are often incomplete and sparse, which limits the reasoning capability in recommendation. Moreover, only few of existing studies exploit the dialogue context to dynamically refine knowledge from KGs for better recommendation. To address the above issues, we propose the Variational Reasoning over Incomplete KGs Conversational Recommender (VRICR). Our key idea is to incorporate the large dialogue corpus naturally accompanied with CRSs to enhance the incomplete KGs; and perform dynamic knowledge reasoning conditioned on the dialogue context. Specifically, we denote the dialogue-specific subgraphs of KGs as latent variables with categorical priors for adaptive knowledge graphs refactor. We propose a variational Bayesian method to approximate posterior distributions over dialogue-specific subgraphs, which not only leverages the dialogue corpus for restructuring missing entity relations but also dynamically selects knowledge based on the dialogue context. Finally, we infuse the dialogue-specific subgraphs to decode the recommendation and responses. We conduct experiments on two benchmark CRSs datasets. Experimental results confirm the effectiveness of our proposed method.
Towards Autonomous Driving
Authors: Ya-Qin Zhang
Venue: WSDM
Year: 2023

The automotive and transportation industry is going through a tectonic shift in the next decade with the advent of Connectivity, Automation, Sharing, and Electrification (CASE). Autonomous driving presents a historical opportunity to transform the academic, technological, and industrial landscape with advanced sensing and actuation, high definition mapping, new machine learning algorithms, smart planning and control, increasing computing powers, and new infrastructure with 5G, cloud and edge computing. Indeed, we have witnessed unprecedented innovation and activities in the past five years in R&D, investment, joint ventures, road tests and commercial trials, from auto makers, tier-ones, and new forces from the internet and high-tech industries. In this talk, I will speak about this historical opportunity and challenges from technological, industrial and policy perspectives. I will address some of the core controversial and critical issues in the advancement of autonomous driving: open vs closed, Lidar vs cameras, progressive L2-L3-L4 vs new L4, autonomous capabilities vs V2X, Robotaxi vs vertical, China vs global, automakers vs new players, and the evolutional path and end game.
Efficiently Leveraging Multi-level User Intent for Session-based Recommendation via Atten-Mixer Network
Authors: Peiyan Zhang, Jiayan Guo, Chaozhuo Li, Yueqi Xie, Jaeboum Kim, Yan Zhang, Xing Xie, Haohan Wang, Sunghun Kim
Venue: WSDM
Year: 2023

Session-based recommendation (SBR) aims to predict the user's next action based on short and dynamic sessions. Recently, there has been an increasing interest in utilizing various elaborately designed graph neural networks (GNNs) to capture the pair-wise relationships among items, seemingly suggesting the design of more complicated models is the panacea for improving the empirical performance. However, these models achieve relatively marginal improvements with exponential growth in model complexity. In this paper, we dissect the classical GNN-based SBR models and empirically find that some sophisticated GNN propagations are redundant, given the readout module plays a significant role in GNN-based models. Based on this observation, we intuitively propose to remove the GNN propagation part, while the readout module will take on more responsibility in the model reasoning process. To this end, we propose the Multi-Level Attention Mixture Network (Atten-Mixer), which leverages both concept-view and instance-view readouts to achieve multi-level reasoning over item transitions. As simply enumerating all possible high-level concepts is infeasible for large real-world recommender systems, we further incorporate SBR-related inductive biases, i.e., local invariance and inherent priority to prune the search space. Experiments on three benchmarks demonstrate the effectiveness and efficiency of our proposal. We also have already launched the proposed techniques to a large-scale e-commercial online service since April 2021, with significant improvements of top-tier business metrics demonstrated in the online experiments on live traffic.
PiggyBack: Pretrained Visual Question Answering Environment for Backing up Non-deep Learning Professionals
Authors: Zhihao Zhang, Siwen Luo, Junyi Chen, Sijia Lai, Siqu Long, Hyunsuk Chung, Soyeon Caren Han
Venue: WSDM
Year: 2023

We propose a PiggyBack, a Visual Question Answering platform that allows users to apply the state-of-the-art visual-language pretrained models easily. We integrate visual-language models, pretrained by HuggingFace, an open-source API platform of deep learning technologies; however, it cannot be runnable without programming skills or deep learning understanding. Hence, our PiggyBack supports an easy-to-use browser-based user interface with several deep-learning visual language pretrained models for general users and domain experts. The PiggyBack includes the following benefits: Portability due to web-based and thus runs on almost any platform, A comprehensive data creation and processing technique, and ease of use on visual language pretrained models. The demo video can be found at https://youtu.be/iz44RZ1lF4s.
SimSumIoT: A Platform for Simulating the Summarisation from Internet of Things
Authors: Wei Emma Zhang, Adnan Mahmood, Lixin Deng, Minhao Zhu
Venue: WSDM
Year: 2023

Summarising from the Web could be formed as a problem of multi-document Summarisaiton (MDS) from multiple sources. In contrast to the current MDS problem that involves working on benchmark datasets which provide well clustered set of documents, we envisage to build a pipeline for content Summarisaiton from the Web, but narrow down to the Social Internet of Things (SIoT) paradigm, starting at data collection from the IoT objects, then applying natural language processing techniques for grouping and summarising the data, to distributing summaries back to the IoT objects. In this paper, we present our simulation tool, SimSumIoT, that simulates the process of data sharing, receiving, clustering, and Summarisaiton. A Web-based interface is developed for this purpose allowing users to visualize the process through a set of interactions. The Web interface is accessible via http://simsumlot.tk.
A Semantic Search Framework for Similar Audit Issue Recommendation in Financial Industry
Authors: Chuchu Zhang, Can Song, Samarth Agarwal, Huayu Wu, Xuejie Zhang, John Jianan Lu
Venue: WSDM
Year: 2023

Audit issues summarize the findings during audit reviews and provide valuable insights of risks and control gaps in a financial institute. Despite the wide use of data analytics and NLP in financial services, due to the diverse coverage and lack of annotations, there are very few use cases that analyze audit issue writing and derive insights from it. In this paper, we propose a deep learning based semantic search framework to search, rank and recommend similar past issues based on new findings. We adopt a two-step approach. First, a TF-IDF based search algorithm and a Bi-Encoder are used to shortlist a set of issue candidates based on the input query. Then a Cross-Encoder will re-rank the candidates and provide the final recommendation. We will also demonstrate how the models are deployed and integrated with the existing workbench to benefit auditors in their daily work.
Separating Examination and Trust Bias from Click Predictions for Unbiased Relevance Ranking
Authors: Haiyuan Zhao, Jun Xu, Xiao Zhang, Guohao Cai, Zhenhua Dong, Ji-Rong Wen
Venue: WSDM
Year: 2023

Alleviating the examination and trust bias in ranking systems is an important research line in unbiased learning-to-rank (ULTR). Current methods typically use the propensity to correct the biased user clicks and then learn ranking models based on the corrected clicks. Though successes have been achieved, directly modifying the clicks suffers from the inherent high variance because the propensities are usually involved in the denominators of corrected clicks. The problem gets even worse in the situation of mixed examination and trust bias. To address the issue, this paper proposes a novel ULTR method called Decomposed Ranking Debiasing (DRD). DRD is tailored for learning unbiased relevance models with low variance in the existence of examination and trust bias. Unlike existing methods that directly modify the original user clicks, DRD proposes to decompose each click prediction as the combination of a relevance term outputted by the ranking model and other bias terms. The unbiased relevance model, therefore, can be learned by fitting the overall click predictions to the biased user clicks. A joint learning algorithm is developed to learn the relevance and bias models' parameters alternatively. Theoretical analysis showed that, compared with existing methods, DRD has lower variance while retains unbiasedness. Empirical studies indicated that DRD can effectively reduce the variance and outperform the state-of-the-art ULTR baselines.
Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization
Authors: Canzhe Zhao, Yanjie Ze, Jing Dong, Baoxiang Wang, Shuai Li
Venue: WSDM
Year: 2023

Temporal difference (TD) learning with nonlinear function approximation (nonlinear TD learning for short) permits evaluating policies using neural networks, and is a core component in modern deep reinforcement learning. Though significant advances have been made to improve its effectiveness, little attention has been paid to the data privacy faced when applying it in real applications. To mitigate the privacy concerns in practical applications of nonlinear TD learning, in this paper, we consider preserving its privacy under the notion of differential privacy (DP). This problem is challenging since nonlinear TD learning is usually studied in the formulation of stochastic nonconvex-strongly-concave optimization to obtain finite-sample analysis, which requires simultaneously preserving privacy on both primal and dual sides. To this end, we adopt a single-timescale algorithm, which optimizes both sides using learning rates of the same order, to avoid unnecessary privacy costs. Further, we achieve a good trade-off between the privacy and utility guarantees by perturbing gradients on both sides using Gaussian noises with well-calibrated variances. Consequently, our algorithm achieves rigorous (ε,δ)-DP guarantee with the utility upper bounded by ~O((d/log(1/δ))1/8 (nε)1/4) where n is the trajectory length and d is the ambient dimension of the feature space. Extensive experiments conducted in OpenAI Gym validate the advantages of our algorithm.
Long-Document Cross-Lingual Summarization
Authors: Shaohui Zheng, Zhixu Li, Jiaan Wang, Jianfeng Qu, An Liu, Lei Zhao, Zhigang Chen
Venue: WSDM
Year: 2023

Cross-Lingual Summarization (CLS) aims at generating summaries in one language for the given documents in another language. CLS has attracted wide research attention due to its practical significance in the multi-lingual world. Though great contributions have been made, existing CLS works typically focus on short documents, such as news and guides. Different from these short texts, long documents such as academic articles usually discuss complicated subjects and consist of thousands of words, making them non-trivial to process and summarize. To promote CLS research on long documents, we construct Perseus, the first long-document CLS dataset which collects about 94K Chinese scientific documents paired with English summaries. The average length of documents in Perseus is more than 2000 tokens. As a preliminary study on long-document CLS, we build and evaluate various CLS baselines, including pipeline and end-to-end methods. Experimental results on Perseus show the superiority of the end-to-end baseline, which performs the best among all methods. Furthermore, to provide a deeper understanding, we manually analyze the model outputs and discuss specific challenges faced by current approaches. We hope that our work could benchmark long-document CLS and benefit future studies.
Inductive Graph Transformer for Delivery Time Estimation
Authors: Xin Zhou, Jinglong Wang, Yong Liu, Xingyu Wu, Zhiqi Shen, Cyril Leung
Venue: WSDM
Year: 2023

Providing accurate estimated time of package delivery on users' purchasing pages for e-commerce platforms is of great importance to their purchasing decisions and post-purchase experiences. Although this problem shares some common issues with the conventional estimated time of arrival (ETA), it is more challenging with the following aspects: 1) Inductive inference. Models are required to predict ETA for orders with unseen retailers and addresses; 2) High-order interaction of order semantic information. Apart from the spatio-temporal features, the estimated time also varies greatly with other factors, such as the packaging efficiency of retailers, as well as the high-order interaction of these factors. In this paper, we propose an inductive graph transformer (IGT) that leverages raw feature information and structural graph data to estimate package delivery time. Different from previous graph transformer architectures, IGT adopts a decoupled pipeline and trains transformer as a regression function that can capture the multiplex information from both raw feature and dense embeddings encoded by a graph neural network (GNN). In addition, we further simplify the GNN structure by removing its non-linear activation and the learnable linear transformation matrix. The reduced parameter search space and linear information propagation in the simplified GNN enable the IGT to be applied in large-scale industrial scenarios. Experiments on real-world logistics datasets show that our proposed model can significantly outperform the state-of-the-art methods on estimation of delivery time.