OpenRaiser/Intern-Atlas huggingface dataset:
Dual-interest Factorization-heads Attention for Sequential Recommendation
Authors: Guanyu Lin, Chen Gao, Yu Zheng, Jianxin Chang, Yanan Niu, Yang Song, Zhiheng Li, Depeng Jin, Yong Li
Venue: WWW
Year: 2023

Accurate user interest modeling is vital for recommendation scenarios. One of the effective solutions is the sequential recommendation that relies on click behaviors, but this is not elegant in the video feed recommendation where users are passive in receiving the streaming contents and return skip or no-skip behaviors. Here skip and no-skip behaviors can be treated as negative and positive feedback, respectively. With the mixture of positive and negative feedback, it is challenging to capture the transition pattern of behavioral sequence. To do so, FeedRec has exploited a shared vanilla Transformer, which may be inelegant because head interaction of multi-heads attention does not consider different types of feedback. In this paper, we propose Dual-interest Factorization-heads Attention for Sequential Recommendation (short for DFAR) consisting of feedback-aware encoding layer, dual-interest disentangling layer and prediction layer. In the feedback-aware encoding layer, we first suppose each head of multi-heads attention can capture specific feedback relations. Then we further propose factorization-heads attention which can mask specific head interaction and inject feedback information so as to factorize the relation between different types of feedback. Additionally, we propose a dual-interest disentangling layer to decouple positive and negative interests before performing disentanglement on their representations. Finally, we evolve the positive and negative interests by corresponding towers whose outputs are contrastive by BPR loss. Experiments on two real-world datasets show the superiority of our proposed method against state-of-the-art baselines. Further ablation study and visualization also sustain its effectiveness. We release the source code here: https://github.com/tsinghua-fib-lab/WWW2023-DFAR.
EDITS: An Easy-to-difficult Training Strategy for Cloud Failure Prediction
Authors: Qingwei Lin, Tianci Li, Pu Zhao, Yudong Liu, Minghua Ma, Lingling Zheng, Murali Chintalapati, Bo Liu, Paul Wang, Hongyu Zhang, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang
Venue: WWW
Year: 2023

Cloud failures have been a major threat to the reliability of cloud services. Many failure prediction approaches have been proposed to predict cloud failures before they actually occur, so that proactive actions can be taken to ensure service reliability. In industrial practice, existing failure prediction approaches mainly focus on utilizing state-of-the-art time series models to enhance the performance of failure prediction but neglect the training strategy. However, as curriculum learning points out, models perform better when they are trained with data in an order of easy-to-difficult. In this paper, we propose EDITS, a novel training strategy for cloud failure prediction, which greatly improves the performance of the existing cloud failure prediction models. Our experimental results on industrial and public datasets show that EDITS can obviously enhance the performance of cloud failure prediction model. In addition, EDITS also outperforms other curriculum learning methods. More encouragingly, our proposed EDITS has been successfully applied to Microsoft 365 and Azure online service systems, and has obviously reduced financial losses caused by cloud failures.
Skill Graph Construction From Semantic Understanding
Authors: Shiyong Lin, Yiping Yuan, Carol Jin, Yi Pan
Venue: WWW
Year: 2023

LinkedIn is building a skill graph to power a skill-first talent marketplace. Constructing a skill graph from a flat list is not an trivial task, especially by human curation. In this paper, we leverage the pre-trained large language model BERT to achieve this through semantic understanding on synthetically generated texts as training data. We automatically create positive and negative labels from the seed skill graph. The training data are encoded by pre-trained language models into embeddings and they are consumed by the downstream classification module to classify the relationships between skill pairs.
Learned Temporal Aggregations for Fraud Classification on E-Commerce Platforms
Authors: Xiao Ling, David Yan, Bilal Alsallakh, Ashutosh Pandey, Manan Bakshi, Pamela Bhattacharya
Venue: WWW
Year: 2023

Fraud and other types of adversarial behavior are serious problems on customer-to-customer (C2C) e-commerce platforms, where harmful behaviors by bad actors erode user trust and safety. Many modern e-commerce integrity systems utilize machine learning (ML) to detect fraud and bad actors. We discuss the practical problems faced by integrity systems which utilize data associated with user interactions with the platform. Specifically, we focus on the challenge of representing the user interaction events, and aggregating their features. We compare the performance of two paradigms to handle the feature temporality when training the ML models: hand-engineered temporal aggregation and a learned aggregation using a sequence encoder. We show that a model which learns a time-aggregation using a sequence encoder outperforms models trained on handcrafted aggregations on the fraud classification task with a real-world dataset.
Decoding Prompt Syntax: Analysing its Impact on Knowledge Retrieval in Large Language Models
Authors: Stephan Linzbach, Tim Tressel, Laura Kallmeyer, Stefan Dietze, Hajira Jabeen
Venue: WWW
Year: 2023

Large Language Models (LLMs), with their advanced architectures and training on massive language datasets, contain unexplored knowledge. One method to infer this knowledge is through the use of cloze-style prompts. Typically, these prompts are manually designed because the phrasing of these prompts impacts the knowledge retrieval performance, even if the LLM encodes the desired information. In this paper, we study the impact of prompt syntax on the knowledge retrieval capacity of LLMs. We use a template-based approach to paraphrase simple prompts into prompts with a more complex grammatical structure. We then analyse the LLM performance for these structurally different but semantically equivalent prompts. Our study reveals that simple prompts work better than complex forms of sentences. The performance across the syntactical variations for simple relations (1:1) remains best, with a marginal decrease across different typologies. These results reinforce that simple prompt structures are more effective for knowledge retrieval in LLMs and motivate future research into the impact of prompt syntax on various tasks.
CurvDrop: A Ricci Curvature Based Approach to Prevent Graph Neural Networks from Over-Smoothing and Over-Squashing
Authors: Yang Liu, Chuan Zhou, Shirui Pan, Jia Wu, Zhao Li, Hongyang Chen, Peng Zhang
Venue: WWW
Year: 2023

Graph neural networks (GNNs) are powerful models to handle graph data and can achieve state-of-the-art in many critical tasks including node classification and link prediction. However, existing graph neural networks still face both challenges of over-smoothing and over-squashing based on previous literature. To this end, we propose a new Curvature-based topology-aware Dropout sampling technique named CurvDrop, in which we integrate the Discrete Ricci Curvature into graph neural networks to enable more expressive graph models. Also, this work can improve graph neural networks by quantifying connections in graphs and using structural information such as community structures in graphs. As a result, our method can tackle the both challenges of over-smoothing and over-squashing with theoretical justification. Also, numerous experiments on public datasets show the effectiveness and robustness of our proposed method. The code and data are released in https://github.com/liu-yang-maker/Curvature-based-Dropout.
HAPENS: Hardness-Personalized Negative Sampling for Implicit Collaborative Filtering
Authors: Haoxin Liu, Pu Zhao, Si Qin, Yong Shi, Mirror Xu, Qingwei Lin, Dongmei Zhang
Venue: WWW
Year: 2023

For training implicit collaborative filtering (ICF) models, hard negative sampling (HNS) has become a state-of-the-art solution for obtaining negative signals from massive uninteracted items. However, selecting appropriate hardness levels for personalized recommendations remains a fundamental, yet underexplored, problem. Previous HNS works have primarily adjusted the hardness level by tuning a single hyperparameter. However, applying the same hardness level to each user is unsuitable due to varying user behavioral characteristics, the quantity and quality of user records, and different consistencies of models’ inductive biases. Moreover, increasing the number of hyperparameters is not practical due to the massive number of users. To address this important and challenging problem, we propose a model-agnostic and practical approach called hardness-personalized negative sampling (HAPENS). HAPENS uses a two-stage approach: in stage one, it trains the ICF model with a customized objective function that optimizes its worst performance on each user’s interacted item set. In stage two, it utilizes these worst performances as personalized hardness levels with a well-designed sampling distribution, and trains the final model with the same architecture. We evaluated HAPENS on the collected Bing advertising dataset and one public dataset, and the comprehensive experimental results demonstrate its robustness and superiority. Moreover, HAPENS has delivered significant benefits to the Bing advertising system. To the best of our knowledge, we are the first to study this important and challenging problem.
Knowledge Graph Question Answering with Ambiguous Query
Authors: Lihui Liu, Yuzhong Chen, Mahashweta Das, Hao Yang, Hanghang Tong
Venue: WWW
Year: 2023

Knowledge graph question answering aims to identify answers of the query according to the facts in the knowledge graph. In the vast majority of the existing works, the input queries are considered perfect and can precisely express the user’s query intention. However, in reality, input queries might be ambiguous and elusive which only contain a limited amount of information. Directly answering these ambiguous queries may yield unwanted answers and deteriorate user experience. In this paper, we propose PReFNet which focuses on answering ambiguous queries with pseudo relevance feedback on knowledge graphs. In order to leverage the hidden (pseudo) relevance information existed in the results that are initially returned from a given query, PReFNet treats the top-k returned candidate answers as a set of most relevant answers, and uses variational Bayesian inference to infer user’s query intention. To boost the quality of the inferred queries, a neighborhood embedding based VGAE model is used to prune inferior inferred queries. The inferred high quality queries will be returned to the users to help them search with ease. Moreover, all the high-quality candidate nodes will be re-ranked according to the inferred queries. The experiment results show that our proposed method can recommend high-quality query graphs to users and improve the question answering accuracy.
Task Adaptive Multi-learner Network for Joint CTR and CVR Estimation
Authors: Xiaofan Liu, Qinglin Jia, Chuhan Wu, Jingjie Li, Quanyu Dai, Lin Bo, Rui Zhang, Ruiming Tang
Venue: WWW
Year: 2023

CTR and CVR are critical factors in personalized applications, and many methods jointly estimate them via multi-task learning to alleviate the ultra-sparsity of conversion behaviors. However, it is still difficult to predict CVR accurately and robustly due to the limited and even biased knowledge extracted by the single model tower optimized on insufficient conversion samples. In this paper, we propose a task adaptive multi-learner (TAML) framework for joint CTR and CVR prediction. We design a hierarchical task adaptive knowledge representation module with different experts to capture knowledge in different granularities, which can effectively exploit the commonalities between CTR and CVR estimation tasks meanwhile keeping their unique characteristics. We apply multiple learners to extract data knowledge from various views and fuse their predictions to obtain accurate and robust scores. To facilitate knowledge sharing across learners, we further perform self-distillation that uses the fused scores to teach different learners. Thorough offline and online experiments show the superiority of TAML in different Ad ranking tasks, and we have deployed it in Huawei’s online advertising platform to serve the main traffic.
Provenance of Training without Training Data: Towards Privacy-Preserving DNN Model Ownership Verification
Authors: Yunpeng Liu, Kexin Li, Zhuotao Liu, Bihan Wen, Ke Xu, Weiqiang Wang, Wenbiao Zhao, Qi Li
Venue: WWW
Year: 2023

In the era of deep learning, it is critical to protect the intellectual property of high-performance deep neural network (DNN) models. Existing proposals, however, are subject to adversarial ownership forgery (e.g., methods based on watermarks or fingerprints) or require full access to the original training dataset for ownership verification (e.g., methods requiring the replay of the learning process). In this paper, we propose a novel Provenance of Training (PoT) scheme, the first empirical study towards verifying DNN model ownership without accessing any original dataset while being robust against existing attacks. At its core, PoT relies on a coherent model chain built from the intermediate checkpoints saved during model training to serve as the ownership certificate. Through an in-depth analysis of model training, we propose six key properties that a legitimate model chain shall naturally hold. In contrast, it is difficult for the adversary to forge a model chain that satisfies these properties simultaneously without performing actual training. We systematically analyze PoT’s robustness against various possible attacks, including the adaptive attacks that are designed given the full knowledge of PoT’s design, and further perform extensive empirical experiments to demonstrate our security analysis.
SA-Fusion: Multimodal Fusion Approach for Web-based Human-Computer Interaction in the Wild
Authors: Xingyu Liu, Pengfei Ren, Yuchen Chen, Cong Liu, Jing Wang, Haifeng Sun, Qi Qi, Jingyu Wang
Venue: WWW
Year: 2023

Web-based AR technology has broadened human-computer interaction scenes from traditional mechanical devices and flat screens to the real world, resulting in unconstrained environmental challenges such as complex backgrounds, extreme illumination, depth range differences, and hand-object interaction. The previous hand detection and 3D hand pose estimation methods are usually based on single modality such as RGB or depth data, which are not available in some scenarios in unconstrained environments due to the differences between the two modalities. To address this problem, we propose a multimodal fusion approach, named Scene-Adapt Fusion (SA-Fusion), which can fully utilize the complementarity of RGB and depth modalities in web-based HCI tasks. SA-Fusion can be applied in existing hand detection and 3D hand pose estimation frameworks to boost their performance, and can be further integrated into the prototyping AR system to construct a web-based interactive AR application for unconstrained environments. To evaluate the proposed multimodal fusion method, we conduct two user studies on CUG Hand and DexYCB dataset, to demonstrate its effectiveness in terms of accurately detecting hand and estimating 3D hand pose in unconstrained environments and hand-object interaction.
GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks
Authors: Zemin Liu, Xingtong Yu, Yuan Fang, Xinming Zhang
Venue: WWW
Year: 2023

Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks (GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily relies on a large amount of task-specific supervision. To reduce labeling requirement, the “pre-train, fine-tune” and “pre-train, prompt” paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt.
Joint Internal Multi-Interest Exploration and External Domain Alignment for Cross Domain Sequential Recommendation
Authors: Weiming Liu, Xiaolin Zheng, Chaochao Chen, Jiajie Su, Xinting Liao, Mengling Hu, Yanchao Tan
Venue: WWW
Year: 2023

Sequential Cross-Domain Recommendation (CDR) has been popularly studied to utilize different domain knowledge and users’ historical behaviors for the next-item prediction. In this paper, we focus on the cross-domain sequential recommendation problem. This commonly exist problem is rather challenging from two perspectives, i.e., the implicit user historical rating sequences are difficult in modeling and the users/items on different domains are mostly non-overlapped. Most previous sequential CDR approaches cannot solve the cross-domain sequential recommendation problem well, since (1) they cannot sufficiently depict the users’ actual preferences, (2) they cannot leverage and transfer useful knowledge across domains. To tackle the above issues, we propose joint Internal multi-interest exploration and External domain alignment for cross domain Sequential Recommendation model (IESRec). IESRec includes two main modules, i.e., internal multi-interest exploration module and external domain alignment module. To reflect the users’ diverse characteristics with multi-interests evolution, we first propose internal temporal optimal transport method in the internal multi-interest exploration module. We further propose external alignment optimal transport method in the external domain alignment module to reduce domain discrepancy for the item embeddings. Our empirical studies on Amazon datasets demonstrate that IESRec significantly outperforms the state-of-the-art models.
Multiview Representation Learning from Crowdsourced Triplet Comparisons
Authors: Xiaotian Lu, Jiyi Li, Koh Takeuchi, Hisashi Kashima
Venue: WWW
Year: 2023

Crowdsourcing has been used to collect data at scale in numerous fields. Triplet similarity comparison is a type of crowdsourcing task, in which crowd workers are asked the question “among three given objects, which two are more similar?”, which is relatively easy for humans to answer. However, the comparison can be sometimes based on multiple views, i.e., different independent attributes such as color and shape. Each view may lead to different results for the same three objects. Although an algorithm was proposed in prior work to produce multiview embeddings, it involves at least two problems: (1) the existing algorithm cannot independently predict multiview embeddings for a new sample, and (2) different people may prefer different views. In this study, we propose an end-to-end inductive deep learning framework to solve the multiview representation learning problem. The results show that our proposed method can obtain multiview embeddings of any object, in which each view corresponds to an independent attribute of the object. We collected two datasets from a crowdsourcing platform to experimentally investigate the performance of our proposed approach compared to conventional baseline methods.
Differentiable Optimized Product Quantization and Beyond
Authors: Zepu Lu, Defu Lian, Jin Zhang, Zaixi Zhang, Chao Feng, Hao Wang, Enhong Chen
Venue: WWW
Year: 2023

Vector quantization techniques, such as Product Quantization (PQ), play a vital role in approximate nearest neighbor search (ANNs) and maximum inner product search (MIPS) owing to their remarkable search and storage efficiency. However, the indexes in vector quantization cannot be trained together with the inference models since data indexing is not differentiable. To this end, differentiable vector quantization approaches, such as DiffPQ and DeepPQ, have been recently proposed, but existing methods have two drawbacks. First, they do not impose any constraints on codebooks, such that the resultant codebooks lack diversity, leading to limited retrieval performance. Second, since data indexing resorts to operator, differentiability is usually achieved by either relaxation or Straight-Through Estimation (STE), which leads to biased gradient and slow convergence. To address these problems, we propose a Differentiable Optimized Product Quantization method (DOPQ) and beyond in this paper. Particularly, each data is projected into multiple orthogonal spaces, to generate multiple views of data. Thus, each codebook is learned with one view of data, guaranteeing the diversity of codebooks. Moreover, instead of simple differentiable relaxation, DOPQ optimizes the loss based on direct loss minimization, significantly reducing the gradient bias problem. Finally, DOPQ is evaluated with seven datasets of both recommendation and image search tasks. Extensive experimental results show that DOPQ outperforms state-of-the-art baselines by a large margin.
Catch: Collaborative Feature Set Search for Automated Feature Engineering
Authors: Guoshan Lu, Haobo Wang, Saisai Yang, Jing Yuan, Guozheng Yang, Cheng Zang, Gang Chen, Junbo Zhao
Venue: WWW
Year: 2023

Feature engineering often plays a crucial role in building mining systems for tabular data, which traditionally requires experienced human experts to perform. Thanks to the rapid advances in reinforcement learning, it has offered an automated alternative, i.e. automated feature engineering (AutoFE). In this work, through scrutiny of the prior AutoFE methods, we characterize several research challenges that remained in this regime, concerning system-wide efficiency, efficacy, and practicality toward production. We then propose Catch, a full-fledged new AutoFE framework that comprehensively addresses the aforementioned challenges. The core to Catch composes a hierarchical-policy reinforcement learning scheme that manifests a collaborative feature engineering exploration and exploitation grounded on the granularity of the whole feature set. At a higher level of the hierarchy, a decision-making module controls the post-processing of the attained feature engineering transformation. We extensively experiment with Catch on 26 academic standardized tabular datasets and 9 industrialized real-world datasets. Measured by numerous metrics and analyses, Catch establishes a new state-of-the-art, from perspectives performance, latency as well as its practicality towards production. Source code1 can be found at https://github.com/1171000709/Catch.