OpenRaiser/Intern-Atlas huggingface dataset:
Federated Deep Knowledge Tracing
Authors: Jinze Wu, Zhenya Huang, Qi Liu, Defu Lian, Hao Wang, Enhong Chen, Haiping Ma, Shijin Wang
Venue: WSDM
Year: 2021

Knowledge tracing is a fundamental task in intelligent education for tracking the knowledge states of students on necessary concepts. In recent years, Deep Knowledge Tracing (DKT) utilizes recurrent neural networks to model student learning sequences. This approach has achieved significant success and has been widely used in many educational applications. However, in practical scenarios, it tends to suffer from the following critical problems due to data isolation: 1) Data scarcity. Educational data, which is usually distributed across different silos (e.g., schools), is difficult to gather. 2) Different data quality. Students in different silos have different learning schedules, which results in unbalanced learning records, meaning that it is necessary to evaluate the learning data quality independently for different silos. 3) Data incomparability. It is difficult to compare the knowledge states of students with different learning processes from different silos. Inspired by federated learning, in this paper, we propose a novel Federated Deep Knowledge Tracing (FDKT) framework to collectively train high-quality DKT models for multiple silos. In this framework, each client takes charge of training a distributed DKT model and evaluating data quality by leveraging its own local data, while a center server is responsible for aggregating models and updating the parameters for all the clients. In particular, in the client part, we evaluate data quality incorporating different education measurement theories, and we construct two quality-oriented implementations based on FDKT, i.e., FDKTCTT and FDKTIRT-where the means of data quality evaluation follow Classical Test Theory and Item Response Theory, respectively. Moreover, in the server part, we adopt hierarchical model interpolation to uptake local effects for model personalization. Extensive experiments on real-world datasets demonstrate the effectiveness and superiority of the FDKT framework.
Learning and Updating Node Embedding on Dynamic Heterogeneous Information Network
Authors: Yuanzhen Xie, Zijing Ou, Liang Chen, Yang Liu, Kun Xu, Carl Yang, Zibin Zheng
Venue: WSDM
Year: 2021

Heterogeneous information networks consist of multiple types of edges and nodes, which have a strong ability to represent the rich semantics underpinning network structures. Recently, the dynamics of networks has been studied in many tasks such as social media analysis and recommender systems. However, existing methods mainly focus on the static networks or dynamic homogeneous networks, which are incapable or inefficient in modeling dynamic heterogeneous information networks. In this paper, we propose a method named Dynamic Heterogeneous Information Network Embedding (DyHINE), which can update embeddings when the network evolves. The method contains two key designs: (1) A dynamic time-series embedding module which employs a hierarchical attention mechanism to aggregate neighbor features and temporal random walks to capture dynamic interactions; (2) An online real-time updating module which efficiently updates the computed embeddings via a dynamic operator. Experiments on three real-world datasets demonstrate the effectiveness of our model compared with state-of-the-art methods on the task of temporal link prediction.
Split-Treatment Analysis to Rank Heterogeneous Causal Effects for Prospective Interventions
Authors: Yanbo Xu, Divyat Mahajan, Liz Manrao, Amit Sharma, Emre Kiciman
Venue: WSDM
Year: 2021

For many kinds of interventions, such as a new advertisement, marketing intervention, or feature recommendation, it is important to target a specific subset of people for maximizing its benefits at minimum cost or potential harm. However, a key challenge is that no data is available about the effect of such a prospective intervention since it has not been deployed yet. In this work, we propose a split-treatment analysis that ranks the individuals most likely to be positively affected by a prospective intervention using past observational data. Unlike standard causal inference methods, the split-treatment method does not need any observations of the target treatments themselves. Instead it relies on observations of a proxy treatment that is caused by the target treatment. Under reasonable assumptions, we show that the ranking of heterogeneous causal effect based on the proxy treatment is the same as the ranking based on the target treatment's effect. In the absence of any interventional data for cross-validation, Split-Treatment uses sensitivity analyses for unobserved confounding to eliminate unreliable models. We apply Split-Treatment to simulated data and a large-scale, real-world targeting task and validate our discovered rankings via a randomized experiment for the latter.
Privacy Monitoring Service for Conversations
Authors: Qiongkai Xu, Chenchen Xu, Lizhen Qu
Venue: WSDM
Year: 2021

Leakage of personal information in conversations raises serious privacy concerns. Malicious people or bots could pry into sensitive personal information of vulnerable people, such as juveniles, through conversations with them or their digital personal assistants. To address the problem, we present a privacy-leakage warning system that monitors conversations in social media and intercepts the outgoing text messages from a user or a digital assistant, if they impose potential privacy leakage risks. Such messages are redirected to authorized users for approval, before they are sent out. We demonstrate how our system is deployed and used on a social media conversation platform, e.g., Facebook Messenger.
Pretrained Transformers for Text Ranking: BERT and Beyond
Authors: Andrew Yates, Rodrigo Nogueira, Jimmy Lin
Venue: WSDM
Year: 2021

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. Although the most common formulation of text ranking is search, instances of the task can also be found in many text processing applications. This tutorial provides an overview of text ranking with neural network architectures known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) is the best-known example. These models produce high quality results across many domains, tasks, and settings. This tutorial, which is based on the preprint of a forthcoming book to be published by Morgan and & Claypool under the Synthesis Lectures on Human Language Technologies series, provides an overview of existing work as a single point of entry for practitioners who wish to deploy transformers for text ranking in real-world applications and researchers who wish to pursue work in this area. We cover a wide range of techniques, grouped into two categories: transformer models that perform reranking in multi-stage ranking architectures and learned dense representations that perform ranking directly.
Chebyshev Accelerated Spectral Clustering
Authors: Tianyu Yu, Yonghua Zhao, Rongfeng Huang, Shifang Liu, Xinyin Zhang
Venue: WSDM
Year: 2021

Spectral clustering is widely used in modern data analysis. Spectral clustering methods speed up the computation and keep useful information by reducing dimensionality. Recently, graph signal filtering (GSF) has been introduced to further speed up the dimensionality reduction process by avoiding solving eigenvectors. In this work, we first prove that the non-ideal filter not only affects the accuracy of GSF, but also the calculation speed. Then we further propose a modified Kernel Polynomial Method (KPM) to help GSF set the filter properly and effectively. We combine the main steps of KPM and GSF, and propose a novel spectral clustering method: Chebyshev Accelerated Spectral Clustering (CASC). In CASC, we take advantages of the excellent properties of Chebyshev polynomials: Compared with other spectral clustering methods using GSF, CASC spends negligible time on estimating the eigenvalues, and achieves the same accuracy with less computation. The experiments on artificial and real-world data prove that CASC is accurate and fast.
Causal Transfer Random Forest: Combining Logged Data and Randomized Experiments for Robust Prediction
Authors: Shuxi Zeng, Murat Ali Bayir, Joseph J. Pfeiffer III, Denis Charles, Emre Kiciman
Venue: WSDM
Year: 2021

It is often critical for prediction models to be robust to distributional shifts between training and testing data. From a causal perspective, the challenge is to distinguish the stable causal relationships from the unstable spurious correlations across shifts. We describe a causal transfer random forest (CTRF) that combines existing training data with a small amount of data from a randomized experiment to train a model which is robust to the feature shifts and therefore transfers to a new targeting distribution. Theoretically, we justify the robustness of the approach against feature shifts with the knowledge from causal learning. Empirically, we evaluate the CTRF using both synthetic data experiments and real-world experiments in the Bing Ads platform, including a click prediction task and in the context of an end-to-end counterfactual optimization system. The proposed CTRF produces robust predictions and outperforms most baseline methods compared in the presence of feature shifts.
Multi-Interactive Attention Network for Fine-grained Feature Learning in CTR Prediction
Authors: Kai Zhang, Hao Qian, Qing Cui, Qi Liu, Longfei Li, Jun Zhou, Jianhui Ma, Enhong Chen
Venue: WSDM
Year: 2021

In the Click-Through Rate (CTR) prediction scenario, user's sequential behaviors are well utilized to capture the user interest in the recent literature. However, despite being extensively studied, these sequential methods still suffer from three limitations. First, existing methods mostly utilize attention on the behavior of users, which is not always suitable for CTR prediction, because users often click on new products that are irrelevant to any historical behaviors. Second, in the real scenario, there are numerous users that have operations a long time ago, but turn relatively inactive in recent times. Thus, it is hard to precisely capture user's current preferences through early behaviors. Third, multiple representations of user's historical behaviors in different feature subspaces are largely ignored. To remedy these issues, we propose a Multi-Interactive Attention Network (MIAN) to comprehensively extract the latent relationship among all kinds of fine-grained features (e.g., gender, age and occupation in user-profile). Specifically, MIAN contains a Multi-Interactive Layer (MIL) that integrates three local interaction modules to capture multiple representations of user preference through sequential behaviors and simultaneously utilize the fine-grained user-specific as well as context information. In addition, we design a Global Interaction Module (GIM) to learn the high-order interactions and balance the different impacts of multiple features. Finally, Offline experiment results from three datasets, together with an Online A/B test in a large-scale recommendation system, demonstrate the effectiveness of our proposed approach.
The 1st International Workshop on Machine Reasoning: International Machine Reasoning Conference (MRC 2021)
Authors: Yongfeng Zhang, Min Zhang, Hanxiong Chen, Xu Chen, Xianjie Chen, Chuang Gan, Tong Sun, Xin Luna Dong
Venue: WSDM
Year: 2021

Recent years have witnessed the success of machine learning and especially deep learning in many research areas such as Vision and Language Processing, Information Retrieval and Recommender Systems, Social Networks and Conversational Agents. Though various learning approaches have demonstrated satisfying performance in perceptual tasks such as associative learning and matching by extracting useful similarity patterns from data, the area still sees a large amount of research needed to advance the ability of reasoning towards cognitive intelligence in the coming years. This includes but is not limited to neural logical reasoning, neural-symbolic reasoning, causal reasoning, knowledge reasoning and commonsense reasoning. The workshop focuses on the research of machine reasoning techniques and their application in various intelligent tasks. It will gather researchers as well as practitioners in the field for discussions, idea communications, and research promotions. It will also generate insightful debates about the recent progress in machine intelligence to a broader community, including but not limited to CV, IR, NLP, ML, DM, AI and beyond.
WULAI-QA: Web Understanding and Learning with AI towards Document-based Question Answering against COVID-19
Authors: Yuan Zhang, Xiaoqing Zhang, Yichuan Hu, Guanchun Wang, Rui Yan
Venue: WSDM
Year: 2021

With the outbreak of COVID-19, it is urgent and necessary to design a system that can access to information from COVID-19 related documents. Current methods fail to do so since the knowledge about COVID-19, an emerging disease, keeps changing and growing. In this study, we design a dynamic document-based question answering system, namely Web Understanding and Learning with AI (WULAI-QA). WULAI-QA employs feature engineering and online learning to adapt to the non-stationary environment and maintains good and steady performance. We evaluate WULAI-QA's performance on a public question answering (https://www.datafountain.cn/competitions/424) and rank first. We demonstrate that WULAI-QA can learn from user feedback and is easy to use. We believe that WULAI-QA will definitely help people understand COVID-19 and play an important role to fight against the pandemic.
Towards Scalable Spectral Embedding and Data Visualization via Spectral Coarsening
Authors: Zhiqiang Zhao, Ying Zhang, Zhuo Feng
Venue: WSDM
Year: 2021

This paper proposes a scalable multilevel framework for the spectral embedding of large undirected graphs. The proposed method first computes much smaller yet sparse graphs while preserving the key spectral (structural) properties of the original graph, by exploiting a nearly-linear time spectral graph coarsening approach. Then, the resultant spectrally-coarsened graphs are leveraged for the development of much faster algorithms for multilevel spectral graph embedding (clustering) as well as visualization of large data sets. We conducted extensive experiments using a variety of large graphs and datasets and obtained very promising results. For instance, we are able to coarsen the "coPapersCiteseer" graph with 0.43 million nodes and 16 million edges into a much smaller graph with only 13K (32X fewer) nodes and 17K (950X fewer) edges in about 16 seconds; the spectrally-coarsened graphs allow us to achieve up to 1,100X speedup for multilevel spectral graph embedding (clustering) and up to 60X speedup for t-SNE visualization of large data sets.
GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks
Authors: Tianxiang Zhao, Xiang Zhang, Suhang Wang
Venue: WSDM
Year: 2021

Node classification is an important research topic in graph learning. Graph neural networks (GNNs) have achieved state-of-the-art performance of node classification. However, existing GNNs address the problem where node samples for different classes are balanced; while for many real-world scenarios, some classes may have much fewer instances than others. Directly training a GNN classifier in this case would under-represent samples from those minority classes and result in sub-optimal performance. Therefore, it is very important to develop GNNs for imbalanced node classification. However, the work on this is rather limited. Hence, we seek to extend previous imbalanced learning techniques for i.i.d data to the imbalanced node classification task to facilitate GNN classifiers. In particular, we choose to adopt synthetic minority over-sampling algorithms, as they are found to be the most effective and stable. This task is non-trivial, as previous synthetic minority over-sampling algorithms fail to provide relation information for newly synthesized samples, which is vital for learning on graphs. Moreover, node attributes are high-dimensional. Directly over-sampling in the original input domain could generates out-of-domain samples, which may impair the accuracy of the classifier. We propose a novel framework, \method, in which an embedding space is constructed to encode the similarity among the nodes. New samples are synthesize in this space to assure genuineness. In addition, an edge generator is trained simultaneously to model the relation information, and provide it for those new samples. This framework is general and can be easily extended into different variations. The proposed framework is evaluated using three different datasets, and it outperforms all baselines with a large margin.
Scalable Graph Neural Networks with Deep Graph Library
Authors: Da Zheng, Minjie Wang, Quan Gan, Xiang Song, Zheng Zhang, George Karypis
Venue: WSDM
Year: 2021

Learning from graph and relational data plays a major role in many applications including social network analysis, marketing, e-commerce, information retrieval, knowledge modeling, medical and biological sciences, engineering, and others. Recently, Graph Neural Networks (GNNs) have emerged as a promising new learning framework capable of bringing the power of deep representation learning to graph and relational data. This ever-growing body of research has shown that GNNs achieve state-of-the-art performance for problems such as link prediction, fraud detection, target-ligand binding activity prediction, knowledge-graph completion, and product recommendations. In practice, many of the real-world graphs are very large. It is urgent to have scalable solutions to train GNN on large graphs efficiently. The objective of this tutorial is twofold. First, it will provide an overview of the theory behind GNNs, discuss the types of problems that GNNs are well suited for, and introduce some of the most widely used GNN model architectures and problems/applications that are designed to solve. Second, it will introduce the Deep Graph Library (DGL), a scalable GNN framework that simplifies the development of efficient GNN-based training and inference programs at a large scale. To make things concrete, the tutorial will cover state-of-the-art training methods to scale GNN to large graphs and provide hands-on sessions to show how to use DGL to perform scalable training in different settings (multi-GPU training and distributed training). This hands-on part will start with basic graph applications (e.g., node classification and link prediction) to set up the context and move on to train GNNs on large graphs. It will provide tutorials to demonstrate how to apply the techniques in DGL to train GNNs for real-world applications.
Popularity-Opportunity Bias in Collaborative Filtering
Authors: Ziwei Zhu, Yun He, Xing Zhao, Yin Zhang, Jianling Wang, James Caverlee
Venue: WSDM
Year: 2021

This paper connects equal opportunity to popularity bias in implicit recommenders to introduce the problem of popularity-opportunity bias. That is, conditioned on user preferences that a user likes both items, the more popular item is more likely to be recommended (or ranked higher) to the user than the less popular one. This type of bias is harmful, exerting negative effects on the engagement of both users and item providers. Thus, we conduct a three-part study: (i) By a comprehensive empirical study, we identify the existence of the popularity-opportunity bias in fundamental matrix factorization models on four datasets; (ii) coupled with this empirical study, our theoretical study shows that matrix factorization models inherently produce the bias; and (iii) we demonstrate the potential of alleviating this bias by both in-processing and post-processing algorithms. Extensive experiments on four datasets show the effective debiasing performance of these proposed methods compared with baselines designed for conventional popularity bias.
Interpretable Ranking with Generalized Additive Models
Authors: Honglei Zhuang, Xuanhui Wang, Michael Bendersky, Alexander Grushetsky, Yonghui Wu, Petr Mitrichev, Ethan Sterling, Nathan Bell, Walker Ravina, Hai Qian
Venue: WSDM
Year: 2021

Interpretability of ranking models is a crucial yet relatively under-examined research area. Recent progress on this area largely focuses on generating post-hoc explanations for existing black-box ranking models. Though promising, such post-hoc methods cannot provide sufficiently accurate explanations in general, which makes them infeasible in many high-stakes scenarios, especially the ones with legal or policy constraints. Thus, building an intrinsically interpretable ranking model with transparent, self-explainable structure becomes necessary, but this remains less explored in the learning-to-rank setting. In this paper, we lay the groundwork for intrinsically interpretable learning-to-rank by introducing generalized additive models (GAMs) into ranking tasks. Generalized additive models (GAMs) are intrinsically interpretable machine learning models and have been extensively studied on regression and classification tasks. We study how to extend GAMs into ranking models which can handle both item-level and list-level features and propose a novel formulation of ranking GAMs. To instantiate ranking GAMs, we employ neural networks instead of traditional splines or regression trees. We also show that our neural ranking GAMs can be distilled into a set of simple and compact piece-wise linear functions that are much more efficient to evaluate with little accuracy loss. We conduct experiments on three data sets and show that our proposed neural ranking GAMs can outperform other traditional GAM baselines while maintaining similar interpretability.
How Do You Test a Test?: A Multifaceted Examination of Significance Tests
Authors: Nicola Ferro, Mark Sanderson
Venue: WSDM
Year: 2022

We examine three statistical significance tests -- a recently proposed ANOVA model and two baseline tests -- using a suite of measures to determine which is better suited for offline evaluation. We apply our analysis to both the runs of a whole TREC track and also to the runs submitted by six participant groups. The former reveals test behavior in the heterogeneous settings of a large-scale offline evaluation initiative; the latter, almost overlooked in past work (to the best of our knowledge), reveals what happens in the much more restricted case of variants of a single system, i.e. the typical context in which companies and research groups operate. We find the ANOVA test strikingly consistent in large-scale settings, but worryingly inconsistent in some participant experiments. Of greater concern, the participant only experiments show one of our baseline tests (a test widely used in research) can produce a substantial number of inconsistent results. We discuss the implications of this inconsistency for possible publication bias.