OpenRaiser/Intern-Atlas huggingface dataset:
ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale
Authors: Gopinath Chennupati, Milind Rao, Gurpreet Chadha, Aaron Eakin, Anirudh Raju, Gautam Tiwari, Anit Kumar Sahu, Ariya Rastrow, Jasha Droppo, Andy Oberlin, Buddha Nandanoor, Prahalad Venkataramanan, Zheng Wu, Pankaj Sitpure
Venue: KDD
Year: 2022

Incremental learning is one paradigm to enable model building and updating at scale with streaming data. For end-to-end automatic speech recognition (ASR) tasks, the absence of human annotated labels along with the need for privacy preserving policies for model building makes it a daunting challenge. Motivated by these challenges, in this paper we use a cloud based framework for production systems to demonstrate insights from privacy preserving incremental learning for automatic speech recognition (ILASR). By privacy preserving, we mean, usage of ephemeral data which are not human annotated. This system is a step forward for production level ASR models for incremental/continual learning that offers near real-time test-bed for experimentation in the cloud for end-to-end ASR, while adhering to privacy-preserving policies. We show that the proposed system can improve the production models significantly ($3%$) over a new time period of six months even in the absence of human annotated labels with varying levels of weak supervision and large batch sizes in incremental learning. This improvement is $20%$ over test sets with new words and phrases in the new time period. We demonstrate the effectiveness of model building in a privacy-preserving incremental fashion for ASR while further exploring the utility of having an effective teacher model and use of large batch sizes.
Graph-based Multilingual Language Model: Leveraging Product Relations for Search Relevance
Authors: Nurendra Choudhary, Nikhil Rao 0001, Karthik Subbian, Chandan K. Reddy
Venue: KDD
Year: 2022

The large-scale nature of product catalog and the changing demands of customer queries makes product search a challenging problem. The customer queries are ambiguous and implicit. They may be looking for an exact match of their query, or a functional equivalent (i.e., substitute), or an accessory to go with it (i.e., complement). It is important to distinguish these three categories from merely classifying an item for a customer query as relevant or not. This information can help direct the customer and improve search applications to understand the customer mission. In this paper, we formulate search relevance as a multi-class classification problem and propose a graph-based solution to classify a given query-item pair as exact, substitute, complement, or irrelevant (ESCI). The customer engagement (clicks, add-to-cart, and purchases) between query and items serve as a crucial information for this problem. However, existing approaches rely purely on the textual information (such as BERT) and do not sufficiently focus on the structural relationships. Another challenge in including the structural information is the sparsity of such data in some regions. We propose Structure-Aware multilingual LAnguage Model (SALAM), that utilizes a language model along with a graph neural network, to extract region-specific semantics as well as relational information for the classification of query-product pairs. Our model is first pre-trained on a large region-agnostic dataset and behavioral graph data and then fine-tuned on region-specific versions to address the sparsity. We show in our experiments that SALAM significantly outperforms the current matching frameworks on the ESCI classification task in several regions. We also demonstrate the effectiveness of using a two-phased training setup (i.e., pre-training and fine-tuning) in capturing region-specific information. Also, we provide various challenges and solutions for using the model in an industrial setting and outline its contribution to the e-commerce engine.
Hyperbolic Neural Networks: Theory, Architectures and Applications
Authors: Nurendra Choudhary, Nikhil Rao 0001, Karthik Subbian, Srinivasan H. Sengamedu, Chandan K. Reddy
Venue: KDD
Year: 2022

Recent studies have revealed important properties that are unique to graph datasets such as hierarchies and global structures. This has driven research into hyperbolic space due to their ability to effectively encode the inherent hierarchy present in graph datasets. However, a major bottleneck here is the obscurity of hyperbolic geometry and a better comprehension of its gyrovector operations. In this tutorial, we aim to introduce researchers and practitioners in the data mining community to the hyperbolic equivariants of the Euclidean operations that are necessary to tackle their application to neural networks. We describe the popular hyperbolic variants of GNN architectures and explain their implementation, in contrast to the Euclidean counterparts. Also, we motivate our tutorial through critical analysis of existing applications in the areas of graph mining, knowledge graph reasoning, search, NLP, and computer vision.
TARNet: Task-Aware Reconstruction for Time-Series Transformer
Authors: Ranak Roy Chowdhury, Xiyuan Zhang 0001, Jingbo Shang, Rajesh K. Gupta 0001, Dezhi Hong
Venue: KDD
Year: 2022

Time-series data contains temporal order information that can guide representation learning for predictive end tasks (e.g., classification, regression). Recently, there are some attempts to leverage such order information to first pre-train time-series models by reconstructing time-series values of randomly masked time segments, followed by an end-task fine-tuning on the same dataset, demonstrating improved end-task performance. However, this learning paradigm decouples data reconstruction from the end task. We argue that the representations learnt in this way are not informed by the end task and may, therefore, be sub-optimal for the end-task performance. In fact, the importance of different timestamps can vary significantly in different end tasks. We believe that representations learnt by reconstructing important timestamps would be a better strategy for improving end-task performance. In this work, we propose TARNet, Task-Aware Reconstruction Network, a new model using Transformers to learn task-aware data reconstruction that augments end-task performance. Specifically, we design a data-driven masking strategy that uses self-attention score distribution from end-task training to sample timestamps deemed important by the end task. Then, we mask out data at those timestamps and reconstruct them, thereby making the reconstruction task-aware. This reconstruction task is trained alternately with the end task at every epoch, sharing parameters in a single model, allowing the representation learnt through reconstruction to improve end-task performance. Extensive experiments on tens of classification and regression datasets show that TARNet significantly outperforms state-of-the-art baseline models across all evaluation metrics.
Classifying Multimodal Data Using Transformers
Authors: Watson W. K. Chua, Lu Li, Alvina Goh
Venue: KDD
Year: 2022

The increasing prevalence of multimodal data in our society has led to the increased need for machines to make sense of such data holistically. However, data scientists and machine learning engineers aspiring to work on such data face challenges fusing the knowledge from existing tutorials which often deal with each mode separately. Drawing on our experience in classifying multimodal municipal issue feedback in the Singapore government, we conduct a hands-on tutorial to help flatten the learning curve for practitioners who want to apply machine learning to multimodal data.
Noisy Interactive Graph Search
Authors: Qianhao Cong, Jing Tang 0004, Kai Han 0003, Yuming Huang 0002, Lei Chen 0002, Yeow Meng Chee
Venue: KDD
Year: 2022

The interactive graph search (IGS) problem aims to locate an initially unknown target node leveraging human intelligence. In IGS, we can gradually find the target node by sequentially asking humans some reachability queries like "is the target node reachable from a given node x?". However, human workers may make mistakes when answering these queries. Motivated by this concern, in this paper, we study a noisy version of the IGS problem. Our objective in this problem is to minimize the query complexity while ensuring accuracy. We propose a method to select the query node such that we can push the search process as much as possible and an online method to infer which node is the target after collecting a new answer. By rigorous theoretical analysis, we show that the query complexity of our approach is near-optimal up to a constant factor. The extensive experiments on two real datasets also demonstrate the superiorities of our approach.
4th Workshop on Deep Learning Practice and Theory for High-Dimensional Sparse and Imbalanced Data with KDD 2022
Authors: Roberto Corizzo, Junfeng Ge, Colin Bellinger, Xiaoqiang Zhu, Paula Branco, Kuang-chih Lee, Nathalie Japkowicz, Ruiming Tang, Tao Zhuang, Han Zhu 0001, Biye Jiang, Jiaxin Mao, Weinan Zhang 0001
Venue: KDD
Year: 2022

Recently, we have witnessed that deep learning-based approaches have been widely applied. Particularly, some applications involve data that are high dimensional, sparse or imbalanced, which are different from those applications with dense data processing, such as image classification and speech recognition, where deep learning-based approaches have been extensively studied. One of the main applications is the user-centric platform that consists of great deal of users, items and user generated tabular data which are quite high-dimensional. The characteristics of such data pose unique challenges to the adoption of deep learning in these applications, including modeling, training, and online serving, etc. More and more communities from both academia and industry have initiated the endeavors to solve these challenges. This workshop will provide a venue for both the research and engineering communities to discuss and formulate the challenges, utilize opportunities, and propose new ideas in the practice and theory of deep learning on high-dimensional, sparse and imbalanced data.
2nd Workshop on Online and Adaptive Recommender Systems (OARS)
Authors: Xiquan Cui, Vachik S. Dave, Yi Su 0008, Khalifeh Al Jadda, Srijan Kumar, Julian J. McAuley, Tao Ye 0001, Kamelia Aryafar, Mohammed Korayem
Venue: KDD
Year: 2022

Recommender systems (RecSys) play important roles in helping users navigate, discover, and consume large and highly dynamic information. Today, many RecSys solutions deployed in the real world rely on categorical user-profiles and/or pre-calculated recommendation actions that stay static during a user session. However, recent trends suggest that RecSys need to model user intent in real time and constantly adapt to meet user needs at the moment or change user behavior in situ. There are three primary drivers for this emerging need of online adaptation. First, in order to meet the increasing demand for a better personalized experience, the personalization dimensions and space will grow larger and larger. It would not be feasible to pre-compute recommended actions for all personalization scenarios beyond a certain scale. Second, in many settings the system does not have user prior history to leverage. Estimating user intent in real time is the only feasible way to personalize. As various consumer privacy laws tighten, it is foreseeable that many businesses will reduce their reliance on static user profiles. Therefore, it makes the modeling of user intent in real time an important research topic. Third, a user's intent often changes within a session and between sessions, and user behavior could shift significantly during dramatic events. A RecSys should adapt in real time to meet user needs and be robust against distribution shifts. The online and adaptive recommender systems (OARS) workshop offers a focused discussion of the study and application of OARS, and will bring together an interdisciplinary community of researchers and practitioners from both industry and academia. KDD, as the premier data science conference, is an ideal venue to gather leaders in the field to further research into OARS and promote its adoption. This workshop is complementary to several sessions of the main conference (e.g., recommendation, reinforcement learning, etc.) and brings them together using a practical and focused application.
Discovering Significant Patterns under Sequential False Discovery Control
Authors: Sebastian Dalleiger, Jilles Vreeken
Venue: KDD
Year: 2022

We are interested in discovering those patterns from data with an empirical frequency that is significantly differently than expected. To avoid spurious results, yet achieve high statistical power, we propose to sequentially control for false discoveries during the search. To avoid redundancy, we propose to update our expectations whenever we discover a significant pattern. To efficiently consider the exponentially sized search space, we employ an easy-to-compute upper bound on significance, and propose an effective search strategy for sets of significant patterns. Through an extensive set of experiments on synthetic data, we show that our method, Spass, recovers the ground truth reliably, does so efficiently, and without redundancy. On real-world data we show it works well on both single and multiple classes, on low and high dimensional data, and through case studies that it discovers meaningful results.
Framing Algorithmic Recourse for Anomaly Detection
Authors: Debanjan Datta, Feng Chen 0001, Naren Ramakrishnan
Venue: KDD
Year: 2022

The problem of algorithmic recourse has been explored for supervised machine learning models, to provide more interpretable, transparent and robust outcomes from decision support systems. An unexplored area is that of algorithmic recourse for anomaly detection, specifically for tabular data with only discrete feature values. Here the problem is to present a set of counterfactuals that are deemed normal by the underlying anomaly detection model so that applications can utilize this information for explanation purposes or to recommend countermeasures. We present an approach-Context preserving Algorithmic Recourse for Anomalies in Tabular data(CARAT), that is effective, scalable, and agnostic to the underlying anomaly detection model. CARAT uses a transformer based encoder-decoder model to explain an anomaly by finding features with low likelihood. Subsequently semantically coherent counterfactuals are generated by modifying the highlighted features, using the overall context of features in the anomalous instance(s). Extensive experiments help demonstrate the efficacy of CARAT.
The Good, the Bad, and the Outliers: A Testing Framework for Decision Optimization Model Learning
Authors: Orit Davidovich, Gheorghe-Teodor Bercea, Segev Wasserkrug
Venue: KDD
Year: 2022

Mathematical decision-optimization (DO) models provide decision support in a wide range of scenarios. Often, hard-to-model constraints and objectives are learned from data. Learning, however, can give rise to DO models that fail to capture the real system, leading to poor recommendations. We introduce an open-source framework designed for large-scale testing and solution quality analysis of DO model learning algorithms. Our framework produces multiple optimization problems at random, feeds them to the user's algorithm and collects its predicted optima. By comparing predictions against the ground truth, our framework delivers a comprehensive prediction profile of the algorithm. Thus, it provides a playground for researchers and data scientists to develop, test, and tune their DO model learning algorithms. Our contributions include: (1) an open-source testing framework implementation, (2) a novel way to generate DO ground truth, and (3) a first-of-its-kind, generic, cloud-distributed Ray and Rayvens architecture. We demonstrate the use of our testing framework on two open-source DO model learning algorithms.
Robust Event Forecasting with Spatiotemporal Confounder Learning
Authors: Songgaojun Deng, Huzefa Rangwala, Yue Ning 0001
Venue: KDD
Year: 2022

Data-driven societal event forecasting methods exploit relevant historical information to predict future events. These methods rely on historical labeled data and cannot accurately predict events when data are limited or of poor quality. Studying causal effects between events goes beyond correlation analysis and can contribute to a more robust prediction of events. However, incorporating causality analysis in data-driven event forecasting is challenging due to several factors: (i) Events occur in a complex and dynamic social environment. Many unobserved variables, i.e., hidden confounders, affect both potential causes and outcomes. (ii) Given spatiotemporal non-independent and identically distributed (non-IID) data, modeling hidden confounders for accurate causal effect estimation is not trivial. In this work, we introduce a deep learning framework that integrates causal effect estimation into event forecasting. We first study the problem of Individual Treatment Effect (ITE) estimation from observational event data with spatiotemporal attributes and present a novel causal inference model to estimate ITEs. We then incorporate the learned event-related causal information into event prediction as prior knowledge. Two robust learning modules, including a feature reweighting module and an approximate constraint loss, are introduced to enable prior knowledge injection. We evaluate the proposed causal inference model on real-world event datasets and validate the effectiveness of proposed robust learning modules in event prediction by feeding learned causal information into different deep learning methods. Experimental results demonstrate the strengths of the proposed causal inference model for ITE estimation in societal events and showcase the beneficial properties of robust learning modules in societal event forecasting.
International Workshop on Knowledge Graphs: Open Knowledge Network
Authors: Ying Ding 0001, Amit P. Sheth, Krzysztof W. Janowicz, Sergio Baranzini, Sharat Israni, Ilkay Altintas, Lilit Yeghiazarian, Ellie Young, Sam Klein
Venue: KDD
Year: 2022

Knowledge networks/graphs provide a powerful approach for data discovery, integration, and reuse. The NSF's new Convergence Accelerator program, which focuses on transitioning research to practice and translational research, announced Track A on the Open Knowledge Network (OKN). The program calls for multidisciplinary and multi-sector teams to work together to build a cooperative and shared open knowledge network infrastructure to drive innovation across science, engineering, and humanities. This workshop aims to invite researchers, practitioners, and the general public to brainstorm the ideas related to OKN, collaboratively build KGs for different domains or applications, develop AI algorithms to provide intelligent services based on OKN, and discuss the social and economic implications related to OKN.
Graph-based Representation Learning for Web-scale Recommender Systems
Authors: Ahmed El-Kishky, Michael M. Bronstein, Ying Xiao, Aria Haghighi
Venue: KDD
Year: 2022

Recommender systems are fundamental building blocks of modern consumer web applications that seek to predict user preferences to better serve relevant items. As such, high-quality user and item representations as inputs to recommender systems are crucial for personalized recommendation. To construct these user and item representations, self-supervised graph embedding has emerged as a principled approach to embed relational data such as user social graphs, user membership graphs, user-item engagements, and other heterogeneous graphs. In this tutorial we discuss different families of approaches to self-supervised graph embedding. Within each family, we outline a variety of techniques, their merits and disadvantages, and expound on latest works. Finally, we demonstrate how to effectively utilize the resultant large embedding tables to improve candidate retrieval and ranking in modern industry-scale deep-learning recommender systems.
Efficient Machine Learning on Large-Scale Graphs
Authors: Parker Erickson, Victor E. Lee, Feng Shi, Jiliang Tang
Venue: KDD
Year: 2022

Machine learning on graph data has become a common area of interest across academia and industry. However, due to the size of real-world industry graphs (hundreds of millions of vertices and billions of edges) and the special architecture of graph neural net- works, it is still a challenge for practitioners and researchers to perform machine learning tasks on large-scale graph data. It typi- cally takes a powerful and expensive GPU machine to train a graph neural network on a million-vertex scale graph, let alone doing deep learning on real enterprise graphs. In this tutorial, we will cover how to develop and run performant graph algorithms and graph neural network models with TigerGraph [3], a massively parallel platform for graph analytics, and its Machine Learning Workbench with PyTorch Geometric [4] and DGL [8] support. Using an NFT transaction dataset [6], we will first investigate transactions using graph algorithms by themselves as methods of graph traversing, clustering, classification, and determining similarities between data. Secondly, we will show how to use those graph-derived features such as PageRank and embeddings to empower traditional machine learning models. Finally, we will demonstrate how to train common graph neural networks with TigerGraph and how to implement novel graph neural network models. Participants will use the Tiger- Graph ML Workbench Cloud to perform graph feature engineering and train their machine learning algorithms during the session.
17th International Workshop on Mining and Learning with Graphs (MLG)
Authors: Shobeir Fakhraei, Tim Weninger, Neil Shah, Sami Abu-El-Haija, Saurabh Verma, Tara Safavi
Venue: KDD
Year: 2022

The 17th International Workshop on Mining and Learning with Graphs (MLG) is held in Washington DC, USA on August 15, 2022 and is co-located with the Eighth International Workshop on Deep Learning on Graphs (DLG) as part of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. This workshop is a forum for exchanging ideas and methods for mining and learning with graphs, developing new common understandings of the problems at hand, sharing of data sets where applicable, and leveraging existing knowledge from different disciplines. In doing so, we aim to better understand the overarching principles and the limitations of our current methods, and to inspire research on new algorithms and techniques for mining and learning with graphs. Topics of interest include, but are not limited to, graph mining, statistical relational learning, social network analysis, and network science. The target audience spans researchers and practitioners across academia, government, and industry.