OpenRaiser/Intern-Atlas huggingface dataset:
End-to-End Query Term Weighting
Authors: Karan Samel, Cheng Li 0012, Weize Kong, Tao Chen 0008, Mingyang Zhang 0001, Shaleen Kumar Gupta, Swaraj Khadanga, Wensong Xu, Xingyu Wang, Kashyap Kolipaka, Michael Bendersky, Marc Najork
Venue: KDD
Year: 2023

Bag-of-words based lexical retrieval systems are still the most commonly used methods for real-world search applications. Recently deep learning methods have shown promising results to improve this retrieval performance but are expensive to run in an online fashion, non-trivial to integrate into existing production systems, and might not generalize well in out-of-domain retrieval scenarios. Instead, we build on top of lexical retrievers by proposing a Term Weighting BERT (TW-BERT) model. TW-BERT learns to predict the weight for individual n-gram (e.g., uni-grams and bi-grams) query input terms. These inferred weights and terms can be used directly by a retrieval system to perform a query search. To optimize these term weights, TW-BERT incorporates the scoring function used by the search engine, such as BM25, to score query-document pairs. Given sample query-document pairs we can compute a ranking loss over these matching scores, optimizing the learned query term weights in an end-to-end fashion. Aligning TW-BERT with search engine scorers minimizes the changes needed to integrate it into existing production applications, whereas existing deep learning based search methods would require further infrastructure optimization and hardware requirements. The learned weights can be easily utilized by standard lexical retrievers and by other retrieval techniques such as query expansion. We show that TW-BERT improves retrieval performance over strong term weighting baselines within MSMARCO and in out-of-domain retrieval on TREC datasets.
The 3rd Workshop on Artificial Intelligence-enabled Cybersecurity Analytics
Authors: Sagar Samtani, Shanchieh Yang, Hsinchun Chen
Venue: KDD
Year: 2023

Artificial Intelligence (AI) has gripped modern society as a viable approach to revolutionize operational capabilities across multiple industries. One critical application area that could stand to benefit from the capabilities of AI is cybersecurity. Increasingly, federal funding agencies such as the National Science Foundation are calling for enhanced AI-enabled analytics capabilities to improve cyber threat intelligence, cyber defense generation, and more. To this end, this half-day workshop, not in its third year at ACM KDD, sought to attain significant contributions related to various aspects of AI-enabled cybersecurity analytics. This workshop received a record number of submissions. Submissions were reviewed by a highly-qualified, interdisciplinary group of AI for cybersecurity researchers and practitioners spanning academia and private industry firms.
19th International Workshop on Mining and Learning with Graphs (MLG)
Authors: Neil Shah, Shobeir Fakhraei, Da Zheng 0003, Bahare Fatemi, Leman Akoglu
Venue: KDD
Year: 2023

The 19th International Workshop on Mining and Learning with Graphs (MLG) is held in Long Beach, CA, USA and is co-located with the Tenth International Workshop on Deep Learning on Graphs (DLG) as part of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. This workshop is a forum for exchanging ideas and methods for mining and learning with graphs, developing new common understandings of the problems at hand, sharing of data sets where applicable, and leveraging existing knowledge from different disciplines. In doing so, we aim to better understand the overarching principles and the limitations of our current methods, and to inspire research on new algorithms and techniques for mining and learning with graphs. Topics of interest include, but are not limited to, graph mining, graph machine learning, statistical relational learning, social network analysis, and network science. The target audience spans researchers and practitioners across academia, government, and industry.
Robust NLP for Finance (RobustFin)
Authors: Sameena Shah, Xiaodan Zhu 0001, Gerard de Melo, Armineh Nourbakhsh, Xiaomo Liu, Zhiqiang Ma, Charese Smiley, Zhiyu Chen 0002
Venue: KDD
Year: 2023

Natural language processing (NLP) technologies have been widely applied in business domains such as e-commerce and customer service, but their adoption in the financial sector has been constrained by industry-specific performance standards and regulatory restrictions. This challenge has created new opportunities for core research in related areas. Recent advancements in NLP, such as the advent of large language models, has encouraged adoption in the finance sector. However, compared to other domains, finance has stricter requirements for robustness, explainability, and generalizability. Given this background, we propose to organize the first Robust NLP for Finance (RobustFin) workshop at KDD '23 to encourage the study of and research on robustness and explainability technologies with regard to financial NLP. The goal of the workshop is to extend the applications of NLP in finance, while motivating further research in robust NLP.
Robustness Certification for Structured Prediction with General Inputs via Safe Region Modeling in the Semimetric Output Space
Authors: Huaqing Shao, Lanjun Wang, Junchi Yan
Venue: KDD
Year: 2023

Many real-world machine learning problems involve structured prediction beyond categorical labels. However, most existing robustness certification works are devoted to the classification case. It remains open for robustness certification for more general outputs. In this paper, we propose a novel framework of robustness certification for structured prediction problems, where the output space is modeled as a semimetric space with a distance function that satisfies non-negativity and symmetry but not necessarily the triangle inequality. We further develop our tailored certification methods for binary, numerical, and hybrid inputs in structured prediction. Experiment results show that our method achieves tighter robustness guarantees than the SOTA structured certification baseline for numerical inputs (for which it only supports) with ℓ2 norm perturbation when outputs are measured by intersection over union (IoU) similarity, total variation distance, and perceptual distance. Moreover, we achieve good robustness certification for binary inputs with ℓ0 norm perturbation and hybrid inputs with corresponding perturbation when outputs are measured by Manhattan distance.
QTNet: Theory-based Queue Length Prediction for Urban Traffic
Authors: Ryu Shirakami, Toshiya Kitahara, Koh Takeuchi 0001, Hisashi Kashima
Venue: KDD
Year: 2023

Smart traffic management is the cornerstone of Intelligent Transport Systems (ITS). To achieve smooth travel in urban road networks, ITS provide software-based traffic management based on traffic forecasts. Recently, spatial-temporal graph neural networks (STGNNs) have achieved significant improvements in traffic forecasting by taking into account spatial and temporal dependencies in traffic data. However, in spite of being an indispensable statistic in traffic management in urban areas, the length of congestion queues has not been a prediction target. In addition, existing methods have not considered the use of multimodal traffic data for forecasting. Moreover, given the significant impact of ITS on the real world, black-box predictions with less explainability are unreliable. In this paper, we propose aQueueing-theory-based Neural Network (QTNet), which combines data-driven STGNN methods with queueing-theory-based domain knowledge of traffic engineering in order to achieve accurate and explainable predictions. In our queue length prediction experiments using a real-world dataset collected in urban areas of Tokyo, QTNet outperformed the baseline methods including the state-of-the-art STGNNs by 12.6% in RMSE and 9.9% MAE, and particularly for severe congestion, by 8.1% and 8.4%.
KDD 2023 Workshop on Data Science and AI for Sports
Authors: Huan Song, Panpan Xu, Lin Lee Cheong, Yuanheng Wang
Venue: KDD
Year: 2023

Artificial Intelligence (AI) has enabled breakthroughs on sports analytics in recent years by allowing the extraction of valuable insights and new understandings from vast amounts of sports data. However, due to the highly interdisciplinary nature of the topic, cross-pollination of ideas and methodologies between AI and sports science as well as across different sports remain limited. To address the gap, we propose the first KDD workshop on Data Science and AI for Sports. We aim to bring together researchers and practitioners working on a broad array of sports-related AI/ML use cases, to exchange research ideas, draw connection among various disciplines, and identify challenges and new research questions for future work.
PSLOG: Pretraining with Search Logs for Document Ranking
Authors: Zhan Su, Zhicheng Dou, Yujia Zhou 0002, Ziyuan Zhao, Ji-Rong Wen
Venue: KDD
Year: 2023

Recently, pretrained models have achieved remarkable performance not only in natural language processing but also in information retrieval (IR). Previous studies show that IR-oriented pretraining tasks can achieve better performance than only finetuning pretrained language models in IR datasets. Besides, the massive search log data obtained from mainstream search engines can be used in IR pretraining, for it contains users' implicit judgments of document relevance under a concrete query. However, existing methods mainly use direct query-document click signals to pretrain models. The potential supervision signals from search logs are far from being well explored. In this paper, we propose to comprehensively leverage four query-document relevance relations, including co-interaction and multi-hop relations, to pretrain ranking models in IR. Specifically, we focus on the user's click behavior and construct an Interaction Graph to represent the global relevance relations between queries and documents from all search logs. With the graph, we can consider the co-interaction and multi-hop q-d relationships through their neighbor nodes. Based on the relations extracted from the interaction graph, we propose four strategies to generate contrastive positive and negative q-d pairs and use these data to pretrain ranking models. Experimental results on both industrial and academic datasets demonstrate the effectiveness of our method.
Learning Autoregressive Model in LSM-Tree based Store
Authors: Yunxiang Su, Wenxuan Ma 0003, Shaoxu Song
Venue: KDD
Year: 2023

Database-native machine learning operators are highly desired for efficient I/O and computation costs. While most existing machine learning algorithms assume the time series data fully available and readily ordered by timestamps, it is not the case in practice. Commodity time series databases store the data in pages with possibly overlapping time ranges, known as LSM-Tree based storage. Data points in a page could be incomplete, owing to either missing values or out-of-order arrivals, which may be inserted by the imputed or delayed points in the following pages. Likewise, data points in a page could also be updated by others in another page, for dirty data repairing or re-transmission. A straightforward idea is thus to first merge and order the data points by timestamps, and then apply the existing learning algorithms. It is not only costly in I/O but also prevents pre-computation of model learning. In this paper, we propose to offline learn the AR models locally in each page on incomplete data, and online aggregate the stored models in different pages with the consideration of the aforesaid inserted and updated data points. Remarkably, the proposed method has been deployed and included as a function in an open source time series database, Apache IoTDB. Extensive experiments in the system demonstrate that our proposal LSMAR shows up to one order-of-magnitude improvement in learning time cost. It needs only about 10s of milliseconds for learning over 1 million data points.
Enhance Diffusion to Improve Robust Generalization
Authors: Jianhui Sun, Sanchit Sinha, Aidong Zhang 0001
Venue: KDD
Year: 2023

Deep neural networks are susceptible to human imperceptible adversarial perturbations. One of the strongest defense mechanisms is Adversarial Training (AT). In this paper, we aim to address two predominant problems in AT. First, there is still little consensus on how to set hyperparameters with a performance guarantee for AT research, and customized settings impede a fair comparison between different model designs in AT research. Second, the robustly trained neural networks struggle to generalize well and suffer from tremendous overfitting. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT). We approximate the dynamic of PGD-AT by a continuous-time Stochastic Differential Equation (SDE), and show that the diffusion term of this SDE determines the robust generalization. An immediate implication of this theoretical finding is that robust generalization is positively correlated with the ratio between learning rate and batch size. We further propose a novel approach, Diffusion Enhanced Adversarial Training (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden. We theoretically show that DEAT obtains a tighter generalization bound than PGD-AT. Our empirical investigation is extensive and firmly attests that DEAT universally outperforms PGD-AT by a significant margin.
Hierarchical Reinforcement Learning for Dynamic Autonomous Vehicle Navigation at Intelligent Intersections
Authors: Qian Sun 0005, Le Zhang 0010, Huan Yu 0009, Weijia Zhang 0003, Yu Mei 0002, Hui Xiong 0001
Venue: KDD
Year: 2023

Recent years have witnessed the rapid development of the Cooperative Vehicle Infrastructure System (CVIS), where road infrastructures such as traffic lights (TL) and autonomous vehicles (AVs) can share information among each other and work collaboratively to provide safer and more comfortable transportation experience to human beings. While many efforts have been made to develop efficient and sustainable CVIS solutions, existing approaches on urban intersections heavily rely on domain knowledge and physical assumptions, preventing them from being practically applied. To this end, this paper proposes NavTL, a learning-based framework to jointly control traffic signal plans and autonomous vehicle rerouting in mixed traffic scenarios where human-driven vehicles and AVs co-exist. The objective is to improve travel efficiency and reduce total travel time by minimizing congestion at the intersections while guiding AVs to avoid the temporally congested roads. Specifically, we design a graph-enhanced multi-agent decentralized bi-directional hierarchical reinforcement learning framework by regarding TLs as manager agents and AVs as worker agents. At lower temporal resolution timesteps, each manager sets a goal for the workers within its controlled region. Simultaneously, managers learn to take the signal actions based on the observation from the environment as well as an intention information extracted from its workers. At higher temporal resolution timesteps, each worker makes rerouting decisions along its way to the destination based on its observation from the environment, an intention-enhanced manager state representation, and a goal from its present manager. Finally, extensive experiments on one synthetic and two real-world network-level datasets demonstrate the effectiveness of our proposed framework in terms of improving travel efficiency.
TrustGeo: Uncertainty-Aware Dynamic Graph Learning for Trustworthy IP Geolocation
Authors: Wenxin Tai, Bin Chen, Fan Zhou 0002, Ting Zhong, Goce Trajcevski, Yong Wang 0046, Kai Chen 0005
Venue: KDD
Year: 2023

The rising popularity of online social network services has attracted a lot of research focusing on mining various user patterns. Among them, accurate IP geolocation is essential for a plethora of location-aware applications. However, despite extensive research efforts and significant advances, the "accurate and reliable'' desideratum is yet to be achieved at a higher quality level. This work presents a graph neural network (GNN)-based model, called TrustGeo, for trustworthy street-level IP geolocation. A distinct and important aspect of TrustGeo is the incorporation of sources of uncertainty in the learning process. The results of our extensive experimental evaluations on three real-world datasets demonstrate the superiority of our framework in significantly improving the accuracy and trustworthiness of street-level IP geolocation. Our code and datasets are available at https://github.com/ICDM-UESTC/TrustGeo.
Causal Effect Estimation on Hierarchical Spatial Graph Data
Authors: Koh Takeuchi 0001, Ryo Nishida, Hisashi Kashima, Masaki Onishi
Venue: KDD
Year: 2023

Estimating individual treatment effects from observational data is a fundamental problem in causal inference. To accurately estimate treatment effects in the spatial domain, we need to address certain aspects such as how to use the spatial coordinates of covariates and treatments and how the covariates and the treatments interact spatially. We introduce a new problem of predicting treatment effects on time series outcomes from spatial graph data with a hierarchical structure. To address this problem, we propose a spatial intervention neural network (SINet) that leverages the hierarchical structure of spatial graphs to learn a rich representation of the covariates and the treatments and exploits this representation to predict a time series of treatment outcome. Using a multi-agent simulator, we synthesized a crowd movement guidance dataset and conduct experiments to estimate the conditional average treatment effect, where we considered the initial locations of the crowds as covariates, route guidance as a treatment, and number of agents reaching a goal at each time stamp as the outcome. We employed state-of-the-art spatio-temporal graph neural networks and neural network-based causal inference methods as baselines, and show that our proposed method outperformed baselines both quantitatively and qualitatively.
PERT-GNN: Latency Prediction for Microservice-based Cloud-Native Applications via Graph Neural Networks
Authors: Da Sun Handason Tam, Yang Liu 0263, Huanle Xu, Siyue Xie, Wing Cheong Lau
Venue: KDD
Year: 2023

Cloud-native applications using microservice architectures are rapidly replacing traditional monolithic applications. To meet end-to-end QoS guarantees and enhance user experience, each component microservice must be provisioned with sufficient resources to handle incoming API calls. Accurately predicting the latency of microservices-based applications is critical for optimizing resource allocation, which turns out to be extremely challenging due to the complex dependencies between microservices and the inherent stochasticity. To tackle this problem, various predictors have been designed based on the Microservice Call Graph. However, Microservice Call Graphs do not take into account the API-specific information, cannot capture important temporal dependencies, and cannot scale to large-scale applications.
Counterfactual Video Recommendation for Duration Debiasing
Authors: Shisong Tang, Qing Li 0006, Dingmin Wang, Ci Gao, Wentao Xiao, Dan Zhao 0003, Yong Jiang 0001, Qian Ma, Aoyang Zhang
Venue: KDD
Year: 2023

Duration bias widely exists in video recommendations, where models tend to recommend short videos for the higher ratio of finish playing and thus possibly fail to capture users' true interests. In this paper, we eliminate the duration bias from both data and model. First, based on the extensive data analysis, we observe that play completion rate of videos with the same duration presents a bimodal distribution. Hence, we propose to perform threshold division to construct binary labels as training labels for alleviating the drawback of finish playing labels overly biased towards short videos. Algorithmically, we resort to causal inference, which enables us to inspect causal relationships of video recommendations with a causal graph. We identify that duration has two kinds of effect on prediction: direct and indirect. Duration bias lies in the direct effect, while the indirect effect benefits prediction. To this end, we design a model-agnostic Counterfactual Video Recommendation for Duration Debiasing (CVRDD) framework, which incorporates multi-task learning to estimate different causal effect during training. In the inference phase, we perform counterfactual inference to remove the direct effect of duration for unbiased prediction. We conduct experiments on two industrial datasets, and in addition to achieving highly promising results on traditional top-k recommendation metrics, CVRDD also improves the user watch time.
Online Quality Prediction in Windshield Manufacturing using Data-Efficient Machine Learning
Authors: Hasan Tercan, Tobias Meisen
Venue: KDD
Year: 2023

The digitization of manufacturing processes opens up the possibility of using machine learning methods on process data to predict future product quality. Based on the model predictions, quality improvement actions can be taken at an early stage. However, significant challenges must be overcome to successfully implement the predictions. Production lines are subject to hardware and memory limitations and are characterized by constant changes in quality influencing factors. In this paper, we address these challenges and present an online prediction approach for real-world manufacturing processes. On the one hand, it includes methods for feature extraction and selection from multimodal process and sensor data. On the other hand, a continual learning method based on memory-aware synapses is developed to efficiently train an artificial neural network over process changes. We deploy and evaluate the approach in a windshield production process. Our experimental evaluation shows that the model can accurately predict windshield quality and achieve significant process improvement. By comparing with other learning strategies such as transfer learning, we also show that the continual learning method both prevents catastrophic forgetting of the model and maintains its data efficiency.
From Innovation to Scale (I2S) - Discuss and Learn How to Successfully Build, Commercialize, and Scale AI Innovations in Challenging Market Conditions
Authors: Ankur Teredesai, Michael Zeller, Shenghua Bao, Wee Hyong Tok, Linsey Pang
Venue: KDD
Year: 2023

In recent years, the AI community has witnessed an exciting acceleration in innovation across foundation models, deep learning, new AI applications across numerous verticals, and more. In addition, AI innovations driven by both academic and industry research labs have rapidly been adopted by big tech companies and startups to deliver value-differentiated products and services.