OpenRaiser/Intern-Atlas huggingface dataset:
Variance Reduction Using In-Experiment Data: Efficient and Targeted Online Measurement for Sparse and Delayed Outcomes
Authors: Alex Deng, Michelle Du, Anna Matlin, Qing Zhang
Venue: KDD
Year: 2023

Improving statistical power is a common challenge for online experimentation platforms so that more hypotheses can be tested and lower effect sizes can be detected. To increase the power without increasing the sample size, it is necessary to consider the variance of experimental outcome metrics. Variance reduction was previously applied to online experimentation based on the idea of using pre-experiment covariate data to account for noise in the final metrics. Since this method relies on correlations between pre-experiment covariates and experiment outcomes, its effectiveness can be limited when testing features for specific product surfaces. We were also motivated by the challenge of attributing sparse, delayed binary outcomes to individual user-product interactions. We present two novel methods for variance reduction that rely exclusively on in-experiment data. The first method is a framework for a model-based leading indicator metric which continually estimates progress toward a delayed binary outcome. The second method is a counterfactual treatment exposure index that quantifies the amount that a user is impacted by the treatment. We applied these methods to past experiments and found that both can achieve variance reduction of 50% or more compared to the delayed outcome metric. The substantial reduction in variance afforded by the two methods presented in this paper has enabled Airbnb's experimentation platform to become more agile and innovative.
Investigating Trojan Attacks on Pre-trained Language Model-powered Database Middleware
Authors: Peiran Dong, Song Guo 0001, Junxiao Wang
Venue: KDD
Year: 2023

The recent success of pre-trained language models (PLMs) such as BERT has resulted in the development of various beneficial database middlewares, including natural language query interfaces and entity matching. This shift has been greatly facilitated by the extensive external knowledge of PLMs. However, as PLMs are often provided by untrusted third parties, their lack of standardization and regulation poses significant security risks that have yet to be fully explored. This paper investigates the security threats posed by malicious PLMs to these emerging database middleware. We specifically propose a novel type of Trojan attack, where a maliciously designed PLM causes unexpected behavior in the database middleware. These Trojan attacks possess the following characteristics: (1) Triggerability: The Trojan-infected database middleware will function normally with normal input, but will likely malfunction when triggered by the attacker. (2) Imperceptibility: There is no need for noticeable modification of the input to trigger the Trojan. (3) Generalizability: The Trojan is capable of targeting a variety of downstream tasks, not just one specific task. We thoroughly evaluate the impact of these Trojan attacks through experiments and analyze potential countermeasures and their limitations. Our findings could aid in the creation of stronger mechanisms for the implementation of PLMs in database middleware.
Fairness in Graph Machine Learning: Recent Advances and Future Prospectives
Authors: Yushun Dong, Oyku Deniz Kose, Yanning Shen, Jundong Li
Venue: KDD
Year: 2023

Graph machine learning algorithms have become popular tools in helping us gain a deeper understanding of the ubiquitous graph data. Despite their effectiveness, most graph machine learning algorithms lack considerations for fairness, which can result in discriminatory outcomes against certain demographic subgroups or individuals. As a result, there is a growing societal concern about mitigating the bias exhibited in these algorithms. To tackle the problem of algorithmic bias in graph machine learning algorithms, this tutorial aims to provide a comprehensive overview of recent research progress in measuring and mitigating the bias in machine learning algorithms on graphs. Specifically, this tutorial first introduces several widely-used fairness notions and the corresponding metrics. Then, we present a well-organized review of the theoretical understanding of bias in graph machine learning algorithms, followed by a summary of existing techniques to debias graph machine learning algorithms. Furthermore, we demonstrate how different real-world applications benefit from these graph machine learning algorithms after debiasing. Finally, we provide insights on current research challenges and open questions to encourage further advances.
From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams
Authors: Iddo Drori, Sarah J. Zhang, Reece Shuttleworth, Sarah Zhang, Keith Tyser, Zad Chin, Pedro Lantigua, Saisamrit Surbehera, Gregory Hunter, Derek Austin, Leonard Tang, Yann Hicke, Sage Simhon, Sathwik Karnik, Darnell Granberry, Madeleine Udell
Venue: KDD
Year: 2023

A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning finals at a human level on finals available online and automatically generate new human-quality final exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve final exams, which differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate a dataset and benchmark of questions from machine learning final exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this final exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. A student survey comparing the quality, appropriateness, and difficulty of machine-generated questions with human-written questions shows that across multiple aspects, machine-generated questions are indistinguishable from human-generated questions and are suitable for final exams. We perform ablation studies comparing zero-shot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and find that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, significantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning large language models such as ChatGPT in class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated, encouraging critical thinking in academic studies.
Time-to-Event Modeling with Hypernetwork based Hawkes Process
Authors: Manisha Dubey, P. K. Srijith, Maunendra Sankar Desarkar
Venue: KDD
Year: 2023

Many real-world applications are associated with collection of events with timestamps, known as time-to-event data. Earthquake occurrences, social networks, and user activity logs can be represented as a sequence of discrete events observed in continuous time. Temporal point process serves as an essential tool for modeling such time-to-event data in continuous time space. Despite having massive amounts of event sequence data from various domains like social media, healthcare etc., real world application of temporal point process faces two major challenges: 1) it is not generalizable to predict events from unseen event sequences in dynamic environment 2) they are not capable of thriving in continually evolving environment with minimal supervision while retaining previously learnt knowledge. To tackle these issues, we propose HyperHawkes, a hypernetwork based temporal point process framework which is capable of modeling time of event occurrence for unseen sequences and consequently, zero-shot learning for time-to-event modeling. We also develop a hypernetwork based continually learning temporal point process for continuous modeling of time-to-event sequences with minimal forgetting. HyperHawkes augments the temporal point process with zero-shot modeling and continual learning capabilities. We demonstrate the application of the proposed framework through our experiments on real-world datasets. Our results show the efficacy of the proposed approach in terms of predicting future events under zero-shot regime for unseen event sequences. We also show that the proposed model is able to learn the time-to-event sequences continually while retaining information from previous event sequences, mitigating catastrophic forgetting in neural temporal point process.
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
Authors: Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, Jayant Kalagnanam
Venue: KDD
Year: 2023

Transformers have gained popularity in time series forecasting for their ability to capture long-sequence interactions. However, their memory and compute-intensive requirements pose a critical bottleneck for long-term forecasting, despite numerous advancements in compute-aware self-attention modules. To address this, we propose TSMixer, a lightweight neural architecture exclusively composed of multi-layer perceptron (MLP) modules. TSMixer is designed for multivariate forecasting and representation learning on patched time series, providing an efficient alternative to Transformers. Our model draws inspiration from the success of MLP-Mixer models in computer vision. We demonstrate the challenges involved in adapting Vision MLP-Mixer for time series and introduce empirically validated components to enhance accuracy. This includes a novel design paradigm of attaching online reconciliation heads to the MLP-Mixer backbone, for explicitly modeling the time-series properties such as hierarchy and channel-correlations. We also propose a Hybrid channel modeling approach to effectively handle noisy channel interactions and generalization across diverse datasets, a common challenge in existing patch channel-mixing methods. Additionally, a simple gated attention mechanism is introduced in the backbone to prioritize important features. By incorporating these lightweight components, we significantly enhance the learning capability of simple MLP structures, outperforming complex Transformer models with minimal computing usage. Moreover, TSMixer's modular design enables compatibility with both supervised and masked self-supervised learning methods, making it a promising building block for time-series Foundation Models. TSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X).
A Lightweight, Efficient and Explainable-by-Design Convolutional Neural Network for Internet Traffic Classification
Authors: Kevin Fauvel, Fuxing Chen, Dario Rossi 0001
Venue: KDD
Year: 2023

Traffic classification, i.e., the identification of the type of applications flowing in a network, is a strategic task for numerous activities (e.g., intrusion detection, routing). This task faces some critical challenges that current deep learning approaches do not address. The design of current approaches do not take into consideration the fact that networking hardware (e.g., routers) often runs with limited computational resources. Further, they do not meet the need for faithful explainability highlighted by regulatory bodies. Finally, these traffic classifiers are evaluated on small datasets which fail to reflect the diversity of applications in real-world settings.
ILRoute: A Graph-based Imitation Learning Method to Unveil Riders' Routing Strategies in Food Delivery Service
Authors: Tao Feng, Huan Yan, Huandong Wang, Wenzhen Huang, Yuyang Han, Hongsen Liao, Jinghua Hao, Yong Li 0008
Venue: KDD
Year: 2023

Pick-up and delivery (PD) services such as online food ordering are playing an increasingly important role in serving people's daily demands. Accurate PD route prediction (PDRP) is important for service providers to efficiently schedule riders to improve service quality. It is crucial to model the decision-making process behind the route choice of riders for PDRP. Recent years have witnessed the success of utilizing imitation learning (IL) to model user decision-making process. Therefore, we propose to deploy an IL framework to solve the PDRP problem. However, there still exist three main challenges: (1) the rider's route decision is affected by multi-source and heterogeneous features and the complex relationships among these features make it hard to explore how they influence the rider's route decision-making; (2) the large route decision-making space make it easy to explore and predict unreasonable routes; (3) the rider's personalized preference is important in modeling the route decision-making process but cannot be fully explored. To tackle the above challenges, we propose ILRoute, a Graph-based imitation learning method for PDRP. ILRoute utilizes a multi-graph neural network (multi-GNN) to extract the multi-source and heterogeneous features and model their complex relationships. To address the large route decision-making space, ILRoute introduces a mobility regularity-aware constraint as prior route choice knowledge to reduce the exploration route decision-making space. To model the personalized preferences of the rider, ILRoute utilizes a personalized constraint mechanism to enhance the personalization of the rider's route decision-making process. Offline experiments conducted on three real-world datasets and online comparisons demonstrate the superiority of our proposed model.
Dependence and Model Selection in LLP: The Problem of Variants
Authors: Gabriel Franco, Mark Crovella, Giovanni Comarela
Venue: KDD
Year: 2023

The problem of Learning from Label Proportions (LLP) has received considerable research attention and has numerous practical applications. In LLP, a hypothesis assigning labels to items is learned using knowledge of only the proportion of labels found in predefined groups, called bags. While a number of algorithmic approaches to learning in this context have been proposed, very little work has addressed the model selection problem for LLP. Nonetheless, it is not obvious how to extend straightforward model selection approaches to LLP, in part because of the lack of item labels. More fundamentally, we argue that a careful approach to model selection for LLP requires consideration of the dependence structure that exists between bags, items, and labels. In this paper we formalize this structure and show how it affects model selection. We show how this leads to improved methods of model selection that we demonstrate outperform the state of the art over a wide range of datasets and LLP algorithms.
Multiplex Heterogeneous Graph Neural Network with Behavior Pattern Modeling
Authors: Chaofan Fu, Guanjie Zheng, Chao Huang 0001, Yanwei Yu, Junyu Dong
Venue: KDD
Year: 2023

Heterogeneous graph neural networks have gained great popularity in tackling various network analysis tasks on heterogeneous network data. However, most existing works mainly focus on general heterogeneous networks, and assume that there is only one type of edge between two nodes, while ignoring the multiplex characteristics between multi-typed nodes in multiplex heterogeneous networks and the different importance of multiplex structures among nodes for node embedding. In addition, the over-smoothing issue of graph neural networks limits existing models to only capturing local structure signals but hardly learning the global relevant information of the network. To tackle these challenges, this work proposes a model called Behavior Pattern based Heterogeneous Graph Neural Network (BPHGNN) for multiplex heterogeneous network embedding. Specifically, BPHGNN can collaboratively learn node representations across different multiplex structures among nodes with adaptive importance learning from local and global perspectives in multiplex heterogeneous networks through depth behavior pattern aggregation and breadth behavior pattern aggregation. Extensive experiments on six real-world networks with various network analytical tasks demonstrate the significant superiority of BPHGNN against state-of-the-art approaches in terms of various evaluation metrics.
AI Explainability 360 Toolkit for Time-Series and Industrial Use Cases
Authors: Giridhar Ganapavarapu, Sumanta Mukherjee, Natalia Martinez Gil, Kanthi K. Sarpatwar, Amaresh Rajasekharan, Amit Dhurandhar, Vijay Arya, Roman Vaculín
Venue: KDD
Year: 2023

With the growing adoption of AI, trust and explainability have become critical which has attracted a lot of research attention over the past decade and has led to the development of many popular AI explainability libraries such as AIX360, Alibi, OmniXAI, etc. Despite that, applying explainability techniques in practice often poses challenges such as lack of consistency between explainers, semantically incorrect explanations, or scalability. Furthermore, one of the key modalities that has been less explored, both from the algorithmic and practice point of view, is time-series. Several application domains involve time-series including Industry 4.0, asset monitoring, supply chain or finance to name a few.
Pre-training Antibody Language Models for Antigen-Specific Computational Antibody Design
Authors: Kaiyuan Gao, Lijun Wu 0003, Jinhua Zhu 0001, Tianbo Peng, Yingce Xia, Liang He 0010, Shufang Xie 0003, Tao Qin 0001, Haiguang Liu, Kun He 0001, Tie-Yan Liu
Venue: KDD
Year: 2023

Antibodies are proteins that effectively protect the human body by binding to pathogens. Recently, deep learning-based computational antibody design has attracted popular attention since it automatically mines the antibody patterns from data that could be complementary to human experiences. However, the computational methods heavily rely on high-quality antibody structure data, which is quite limited. Besides, the complementarity-determining region (CDR), which is the key component of an antibody that determines the specificity and binding affinity, is highly variable and hard to predict. Therefore, the limited availability of high-quality antibody structure data exacerbates the difficulty of CDR generation. Fortunately, there is a large amount of sequence data for antibodies that can help model the CDR and reduce reliance on structure data. By witnessing the success of pre-training models for protein modeling, in this paper, we develop the antibody pre-training language model and incorporate it into the antigen-specific antibody design model in a systemic way. Specifically, we first pre-train a novel antibody language model based on the sequence data, then propose a one-shot way for sequence and structure generation of CDR to mitigate the high cost and error propagation associated with autoregressive methods, and finally leverage the pre-trained antibody model for the antigen-specific antibody generation model with some carefully designed modules. Our experiments demonstrate the superiority of our method over previous baselines in tasks such as sequence and structure generation, CDR-H3 design for antigen binding, and antibody optimization1. The code is available at https://github.com/KyGao/ABGNN.
Distributed Optimization for Big Data Analytics: Beyond Minimization
Authors: Hongchang Gao, Xinwen Zhang
Venue: KDD
Year: 2023

The traditional machine learning model can be formulated as an empirical risk minimization problem, which is typically optimized via stochastic gradient descent (SGD). With the emergence of big data, distributed optimization, e.g., distributed SGD, has been attracting increasing attention to facilitate machine learning models for big data analytics. However, existing distributed optimization mainly focuses on the standard empirical risk minimization problem, failing to deal with the emerging machine learning models that are beyond that category. Thus, of particular interest of this tutorial includes the stochastic minimax optimization, stochastic bilevel optimization, and stochastic compositional optimization, which covers a wide range of emerging machine learning models, e.g., model-agnostic meta-learning models, adversarially robust machine learning models, imbalanced data classification models, etc. Since these models have been widely used in big data analytics, it is necessary to provide a comprehensive introduction about the new distributed optimization algorithms designed for these models. Therefore, the goal of this tutorial is to present the state-of-the-art and recent advances in distributed minimax optimization, distributed bilevel optimization, and distributed compositional optimization. In particular, we will introduce the typical applications in each category and discuss the corresponding distributed optimization algorithms in both centralized and decentralized settings. Through this tutorial, the researchers will be exposed to the fundamental algorithmic design and basic convergence theories, and the practitioners will be able to benefit from this tutorial to apply these algorithms to real-world data mining applications.
KiL 2023 : 3rd International Workshop on Knowledge-infused Learning
Authors: Manas Gaur, Efthymia Tsamoura, Sarath Sreedharan, Sudip Mittal
Venue: KDD
Year: 2023

Recent prolific advances in artificial intelligence through the incorporation of domain knowledge have constituted a new paradigm for AI and data mining communities. For example, the human feedback-based language generation in ChatGPT (a large language model (LLM)), the use of Protein Bank in DeepMind's AlphaFold, and the use of 23 rules of safety in DeepMind's Sparrow have demonstrated the success of teaming human knowledge and AI. In addition, the knowledge retrieval-guided language modeling methods have strengthened the association between knowledge and AI. However, translating research methods and resources into practice presents a new challenge for the machine learning and data/knowledge mining communities. For example, in DARPA's Explainable AI seminar, the need for explainable contextual adaptation is seen as the 3rd phase of AI, facilitating the interplay between data and knowledge for explainability, safety, and, eventually, trust. However, policymakers and practitioners assert serious usability and privacy concerns that constrain adoption, notably in high-consequence domains, such as cybersecurity, healthcare, and other social good domains. In addition, limitations in output quality, measurement, and interactive ability, including both the provision of explanations and the acceptance of user preferences, result in low adoption rates in such domains. This workshop aims to accelerate our pace towards creating innovative methods for integrating knowledge into contemporary AI and data science methods and develop metrics for assessing performance in various applications.