OpenRaiser/Intern-Atlas huggingface dataset:
Partial Multi-Label Learning with Meta Disambiguation
Authors: Ming-Kun Xie, Feng Sun, Sheng-Jun Huang
Venue: KDD
Year: 2021

In partial multi-label learning (PML) problems, each instance is partially annotated with a candidate label set, which consists of multiple relevant labels and some noisy labels. To solve PML problems, existing methods typically try to recover the ground-truth information from partial annotations based on extra assumptions on the data structures. While the assumptions hardly hold in real-world applications, the trained model may not generalize well to varied PML tasks. In this paper, we propose a novel approach for partial multi-label learning with meta disambiguation (PML-MD). Instead of relying on extra assumptions, we try to disambiguate between ground-truth and noisy labels in a meta-learning fashion. On one hand, the multi-label classifier is trained by minimizing a confidence-weighted ranking loss, which distinctively utilizes the supervised information according to the label quality; on the other hand, the confidence for each candidate label is adaptively estimated with its performance on a small validation set. To speed up the optimization, these two procedures are performed alternately with an online approximation strategy. Comprehensive experiments on multiple datasets and varied evaluation metrics validate the effectiveness of the proposed method.
Contrastive Multi-View Multiplex Network Embedding with Applications to Robust Network Alignment
Authors: Hao Xiong, Junchi Yan, Li Pan
Venue: KDD
Year: 2021

Despite its success in learning network node representations, network embedding is still relatively new for multiplex networks (MNs) with multiple types of edges. In such networks, the inter-layer anchor links are usually missing, which represent the alignment relations between nodes on different layers and are a crucial prerequisite for many cross-network applications like network alignment. For mining such anchor links between layers for MNs, multiplex network embedding (MNE) has become one of the most promising techniques. In this paper, we consider two problems for MNs: 1) edges can be missing to different extent, and data augmentation may mitigate this issue; 2) the known alignment anchor links between layers can be misleading since the behaviors of nodes on different layers are not always consistent, so the most informative ones should be emphasized compared with those misleading ones. However, most existing works neglect the two problems and simply 1) adopt one structural view for all the layers (e.g. random walk with the same window size) and 2) equally extract information from all the anchor links. We propose an end-to-end contrastive framework called cM2NE for MNE, utilizing multiple structural views for each layer and learning with several plug-in components for different scenarios. Through end-to-end optimization on three levels, the intra-view, inter-view, and inter-layer level, our framework achieves to select the fitted views for different layers and maximize the inter-layer mutual information by emphasizing those most informative anchor links. Extensive experimental results on real-world datasets for node classification and multi-network alignment show that our approach consistently outperforms peer methods.
PAMI: A Computational Module for Joint Estimation and Progression Prediction of Glaucoma
Authors: Linchuan Xu, Ryo Asaoka, Taichi Kiwaki, Hiroshi Murata, Yuri Fujino, Kenji Yamanishi
Venue: KDD
Year: 2021

Glaucoma, which can cause irreversible damage to the sight of human eyes, is conventionally diagnosed by visual field (VF) sensitivity. However, it is labor-intensive and time-consuming to measure VF. Recently, optical coherence tomography (OCT) has been adopted to measure retinal layers thickness (RT) for assisting the diagnosis because glaucoma makes structural changes to RT and it is much less costly to obtain RT. In particular, RT can assist in mainly two manners. One is to estimate a VF from an RT such that clinical doctors only need to obtain an RT of a patient and then convert it to a VF for the diagnosis. The other is to predict future VFs by utilizing both past VFs and RTs, i.e., the prediction of progression of VF over time. The two computational tasks are performed as two data mining tasks because currently there is no knowledge about the exact form of the computations involved. In this paper, we study a novel problem which is the integration of the two data mining tasks. The motivation is that both the two data mining tasks deal with transforming information from the RT domain to the VF domain such that the knowledge discovered in one task can be useful for another. The integration is non-trivial because the two tasks do not share the way of transformation. To address this issue, we design a progression-agnostic and mode-independent (PAMI) module which facilitates cross-task knowledge utilization. We empirically demonstrate that our proposed method outperforms the state-of-the-art method for the estimation by 6.33% in terms of mean of the root mean square error on a real dataset, and outperforms the state-of-the-art method for the progression prediction by 3.49% for the best case.
Removing Disparate Impact on Model Accuracy in Differentially Private Stochastic Gradient Descent
Authors: Depeng Xu, Wei Du, Xintao Wu
Venue: KDD
Year: 2021

In differentially private stochastic gradient descent (DPSGD), gradient clipping and random noise addition disproportionately affect underrepresented and complex classes and subgroups. As a consequence, DPSGD has disparate impact: the accuracy of a model trained using DPSGD tends to decrease more on these classes and subgroups vs. the original, non-private model. If the original model is unfair in the sense that its accuracy is not the same across all subgroups, DPSGD exacerbates this unfairness. In this work, we study the inequality in utility loss due to differential privacy, which compares the changes in prediction accuracy w.r.t. each group between the private model and the non-private model. We analyze the cost of privacy w.r.t. each group and explain how the group sample size along with other factors is related to the privacy impact on group accuracy. Furthermore, we propose a modified DPSGD algorithm, called DPSGD-F, to achieve differential privacy, equal costs of differential privacy, and good utility. DPSGD-F adaptively adjusts the contribution of samples in a group depending on the group clipping bias such that differential privacy has no disparate impact on group accuracy. Our experimental evaluation shows the effectiveness of our removal algorithm on achieving equal costs of differential privacy with satisfactory utility.
Adversarial Robustness in Deep Learning: From Practices to Theories
Authors: Han Xu, Yaxin Li, Xiaorui Liu, Wentao Wang, Jiliang Tang
Venue: KDD
Year: 2021

Deep neural networks (DNNs) have achieved unprecedented accomplishments in various machine learning tasks. However, recent studies demonstrate that DNNs are extremely vulnerable to adversarial examples. They are manually synthesized input samples which look benign but can severely fool the prediction of DNN models. For machine learning practitioners who are applying DNNs, understanding the behavior of adversarial examples will not only help them improve the safety of their models, but also can help them have deeper insights into the working mechanism of the DNNs. In this tutorial, we provide a comprehensive overview on the recent advances of adversarial examples and their countermeasures, from both practical and theoretical perspectives. From the practical aspect, we give a detailed introduction of the popular algorithms to generate adversarial examples under different adversary's goals. We also discuss how the defending strategies are developed to resist these attacks, and how new attacks come out to break these defenses. From the theoretical aspect, we discuss a series of intrinsic behaviors of robust DNNs which are different from traditional DNNs, especially about their optimization and generalization properties. Finally, we introduce DeepRobust, a Pytorch adversarial learning library which aims to build a comprehensive and easy-to-use platform to foster this research field. Via our tutorial, the audience can grip the main ideas of adversarial attacks and defenses and gain a deep insight of DNN's robustness. The tutorial official website is at https://sites.google.com/view/kdd21-tutorial-adv-robust.
PLP 2021: Workshop on Programming Language Processing
Authors: Chang Xu, Siqi Ma, David Lo
Venue: KDD
Year: 2021

The first international Workshop on Programming Language Processing presents interdisciplinary contributions that address programming language procession problems with machine learning and data mining techniques. Recently, there are lots of successful natural language processing methods. But the mining of programming languages could not exactly follow the manner of natural language processing. The difference between natural language and programming language brings in new research challenges and opportunities. The workshop will bring together researchers from machine learning, data mining and software engineering to discuss and debate the path forward for mining the value of programming languages.
2nd International Workshop on Industrial Recommendation Systems (IRS)
Authors: Jianpeng Xu, Lingfei Wu, Xiaolin Pang, Mohit Sharma, Dawei Yin, George Karypis, Justin Basilico, Philip S. Yu
Venue: KDD
Year: 2021

Recommendation systems are used widely across many industries, such as e-commerce, multimedia content platforms and social networks, to provide suggestions that a user will most likely consume or connect; thus, improving the user experience. This motivates people in both industry and research organizations to focus on personalization or recommendation algorithms, which has resulted in a plethora of research papers. While academic research mostly focuses on the performance of recommendation algorithms in terms of ranking quality or accuracy, it often neglects key factors that impact how a recommendation system will perform in a real-world environment. These key factors include but are not limited to: business metric definition and evaluation, recommendation quality control, data and model scalability, model interpretability, model robustness and fairness, and resource limitations, such as computing and memory resources budgets, engineering workforce cost, etc. The gap in constraints and requirements between academic research and industry limits the broad applicability of many of academia's contributions for industrial recommendation systems. This workshop aspires to bridge this gap by bringing together researchers from both academia and industry. Its goal is to serve as a venue through which academic researchers become aware of the additional factors that may affect the adoption of an algorithm into real production systems, and how well it will perform if deployed. Industrial researchers will also benefit from sharing the practical insights, approaches, and frameworks as well.
Numerical Formula Recognition from Tables
Authors: Qingping Yang, Yixuan Cao, Hongwei Li, Ping Luo
Venue: KDD
Year: 2021

Claims over the numerical relationships among some measures are commonly expressed in tabular forms, and widely exist in the published documents on the Web. This paper introduces the problem of numerical formula recognition from tables, namely recognizing all numerical formulas inside a given table. It can well support many interesting downstream applications, such as numerical error correction in tables, formula recommendation in tables. Here, we emphasize that table is a kind of language that adopts a different linguistic paradigm from natural language. It uses visual grammar like visual layout and visual settings (e.g., indentation, font style) to express the grammatical relationships among the table cells. Understanding tables and recognizing formulas require decoding the visual grammar while simultaneously understanding the textual information. Another challenge is that formulas are complicated in terms of diverse math functions and variable-length of arguments. To address these challenges, we convert this task into a uniform framework, extracting relations of table cell pairs in a table. A two-channel neural network model TaFor is proposed to embed both the textual and visual features for a table cell. Our framework achieves the formula-level F1-score = 0.90 on a real-world dataset of 190179 tables while a retrieval-based method achieves F1-score = 0.72. We also perform extensive experiments to demonstrate the effectiveness of each component in our model, and conduct a case study to discuss the limits of the proposed model. With our published data this study also aims to attract the community's interest in deep semantic understanding over tables.
Context-aware Outstanding Fact Mining from Knowledge Graphs
Authors: Yueji Yang, Yuchen Li, Panagiotis Karras, Anthony K. H. Tung
Venue: KDD
Year: 2021

An Outstanding Fact (OF) is an attribute that makes a target entity stand out from its peers. The mining of OFs has important applications, especially in Computational Journalism, such as news promotion, fact-checking, and news story finding. However, existing approaches to OF mining: (i) disregard the context in which the target entity appears, hence may report facts irrelevant to that context; and (ii) require relational data, which are often unavailable or incomplete in many application domains. In this paper, we introduce the novel problem of mining Context-aware Outstanding Facts (COFs) for a target entity under a given context specified by a context entity. We propose FMiner, a context-aware mining framework that leverages knowledge graphs (KGs) for COF mining. FMiner generates COFs in two steps. First, it discovers top-k relevant relationships between the target and the context entity from a KG. We propose novel optimizations and pruning techniques to expedite this operation, as this process is very expensive on large KGs due to its exponential complexity. Second, for each derived relationship, we find the attributes of the target entity that distinguish it from peer entities that have the same relationship with the context entity, yielding the top-l COFs. As such, the mining process is modeled as a top-(k,l) search problem. Context-awareness is ensured by relying on the relevant relationships with the context entity to derive peer entities for COF extraction. Consequently, FMiner can effectively navigate the search to obtain context-aware OFs by incorporating a context entity. We conduct extensive experiments, including a user study, to validate the efficiency and the effectiveness of FMiner.
Improving the Information Disclosure in Mobility-on-Demand Systems
Authors: Yue Yang, Yuan Shi, Dejian Wang, Qisheng Chen, Lei Xu, Hanqian Li, Zhouyu Fu, Xin Li, Hao Zhang
Venue: KDD
Year: 2021

Nowadays, the ubiquity of sharing economy and the booming of ride-sharing services prompt Mobility-on-Demand (MoD) platforms to explore and develop new business modes. Different from forcing full-time drivers to serve the dispatched orders, these modes usually aim to attract part-time drivers to share their vehicles and employ a 'driver-choose-order' pattern by displaying a sequence of orders to drivers as a candidate set. A key issue here is to determine which orders should be displayed to each driver. In this work, we propose a novel framework to tackle this issue, known as the Information Disclosure problem in MoD systems. The problem is solved in two steps combining estimation with optimization: 1) in the estimation step, we investigate the drivers' choice behavior and estimate the probability of choosing an order or ignoring the displayed candidate set. 2) in the optimization step, we transform the problem into determining the optimal edge configuration in a bipartite graph, then we develop a Minimal-Loss Edge Cutting (MLEC) algorithm to solve it. Through extensive experiments on both the simulation and the real-world data from Huolala business, the proposed method remarkably improves users experience and platform efficiency. Based on these promising results, the proposed framework has been successfully deployed in the real-world MoD system in Huolala.
Semi-supervised Bearing Fault Diagnosis with Adversarially-Trained Phase-Consistent Network
Authors: Jaehyuk Yi, Jinkyoo Park
Venue: KDD
Year: 2021

In this study, we propose an adversarially-trained phase-consistent network (APCNet), which is a semi-supervised signal classification approach. The proposed classification model is trained with datasets that contain a small fraction of labeled output so as to design (1) an effective representation of the input time series (vibration signal) to extract important factors for the model to discriminate between different bearing conditions, and (2) a latent representation for the data to reflect the true data distribution precisely. To achieve these goals, APCNet suggests three novelties: the vibration-specific encoder, the phase-consistency regularization, and the adversarially-trained latent distribution alignment of the labeled and unlabeled distributions. We conduct experiments on two public bearing datasets and one public motor operating dataset to evaluate the performance of APCNet. We interpret the model's capabilities with different data label ratios and latent distribution analysis. The results show that APCNet performs well on datasets with small labeled to unlabeled data ratio. Also, we show that APCNet achieves our objectives of capturing important vibration signals features and modeling the true data distribution effectively.
Defending Privacy Against More Knowledgeable Membership Inference Attackers
Authors: Yu Yin, Ke Chen, Lidan Shou, Gang Chen
Venue: KDD
Year: 2021

Membership Inference Attack (MIA) in deep learning is a common form of privacy attack which aims to infer whether a data sample is in a target classifier's training dataset or not. Previous studies of MIA typically tackle either a black-box or a white-box adversary model, assuming an attacker not knowing (or knowing) the structure and parameters of the target classifier while having access to the confidence vector of the query output. With the popularity of privacy protection methods such as differential privacy, it is increasingly easier for an attacker to obtain the defense method adopted by the target classifier, which poses extra challenge to privacy protection. In this paper, we name such attacker a crystal-box adversary. We present definitions for utility and privacy of target classifier, and formulate the design goal of the defense method as an optimization problem. We also conduct theoretical analysis on the respective forms of the optimization for three adversary models, namely black-box, white-box, and crystal-box, and prove that the optimization problem is NP-hard. Thereby we solve a surrogate problem and propose three defense methods, which, if used together, can make trade-off between utility and privacy. A notable advantage of our approach is that it can be used to resist attacks from three adversary models, namely black-box, white-box, and crystal-box, simultaneously. Evaluation results show effectiveness of our proposed approach for defending privacy against MIA and better performance compared to previous defense methods.
Accurate Multivariate Stock Movement Prediction via Data-Axis Transformer with Multi-Level Contexts
Authors: Jaemin Yoo, Yejun Soun, Yong-chan Park, U Kang
Venue: KDD
Year: 2021

How can we efficiently correlate multiple stocks for accurate stock movement prediction? Stock movement prediction has received growing interest in data mining and machine learning communities due to its substantial impact on financial markets. One way to improve the prediction accuracy is to utilize the correlations between multiple stocks, getting a reliable evidence regardless of the random noises of individual prices. However, it has been challenging to acquire accurate correlations between stocks because of their asymmetric and dynamic nature which is also influenced by the global movement of a market. In this work, we propose DTML (Data-axis Transformer with Multi-Level contexts), a novel approach for stock movement prediction that learns the correlations between stocks in an end-to-end way. DTML makes asymmetric and dynamic correlations by a) learning temporal correlations within each stock, b) generating multi-level contexts based on a global market context, and c) utilizing a transformer encoder for learning inter-stock correlations. DTML achieves the state-of-the-art accuracy on six datasets collected from various stock markets from US, China, Japan, and UK, making up to 13.8%p higher profits than the best competitors and the annualized return of 44.4% on investment simulation.
Performance-Adaptive Sampling Strategy Towards Fast and Accurate Graph Neural Networks
Authors: Minji Yoon, Théophile Gervet, Baoxu Shi, Sufeng Niu, Qi He, Jaewon Yang
Venue: KDD
Year: 2021

The main challenge of adapting Graph convolutional networks (GCNs) to large-scale graphs is the scalability issue due to the uncontrollable neighborhood expansion in the aggregation stage. Several sampling algorithms have been proposed to limit the neighborhood expansion. However, these algorithms focus on minimizing the variance in sampling to approximate the original aggregation. This leads to two critical problems: 1) low accuracy because the sampling policy is agnostic to the performance of the target task, and 2) vulnerability to noise or adversarial attacks on the graph. In this paper, we propose a performance-adaptive sampling strategy PASS that samples neighbors informative for a target task. PASS optimizes directly towards task performance, as opposed to variance reduction. PASS trains a sampling policy by propagating gradients of the task performance loss through GCNs and the non-differentiable sampling operation. We dissect the back-propagation process and analyze how PASS learns from the gradients which neighbors are informative and assigned high sampling probabilities. In our extensive experiments, PASS outperforms state-of-the-art sampling methods by up to 10% accuracy on public benchmarks and up to 53% accuracy in the presence of adversarial attacks.
Extremely Compact Non-local Representation Learning
Authors: Ansheng You, Xiangzeng Zhou, Yingya Zhang, Pan Pan, Yinghui Xu
Venue: KDD
Year: 2021

In contrast to regular convolutions with local receptive fields, non-local operations have widely proven an effective method for modeling long-range dependencies. Although lots of prior works have been proposed, prohibitive computation and GPU memory occupation are still the major concerns. Different from that carrying out non-local operations pixel-wise or channel-wise in a computation intensive way, we argue that we can achieve effective non-local operation using a more compact high-order statistic, which can be computed more efficiently and may convey some high-level information. In this paper, we propose an extremely compact non-local learning module (CoNL) with high-order reasoning based on a graph convolution as the core. In our CoNL, a global Hadamard pooling (GHP) as a non-local operation is used to extract a compact second-order feature vector from the input tensor. With the help of a light-weight graph convolution network (GCN), this high-order compact vector is further refined with high-level reasoning. After the GCN refinement, the compact high-order vector intuitively indicates some global semantic characteristics, and is eventually applied to enhance the input tensor through a channel scaling operation. The CoNL module is designed easily pluggable to upgrade existing networks. Extensive experiments on a wide range of tasks demonstrate the effectiveness and efficiency of our work. The proposed CoNL can achieve comparable or superior performance over previous state-of-the-art baselines on video recognition, semantic segmentation, object detection and instance segmentation tasks. For a 96 x 96 x 2048 input, our block consumes 13.6 x less in computational cost than non-local block while 7.6 x smaller in GPU memory occupation.