OpenRaiser/Intern-Atlas huggingface dataset:
Multi-Task Self-Supervised Learning for Script Event Prediction
Authors: Bo Zhou, Yubo Chen, Kang Liu, Jun Zhao, Jiexin Xu, Xiaojian Jiang, Jinlong Li
Venue: CIKM
Year: 2021

Most existing approaches to script event prediction rely on manually labeled data heavily, which is often expensive to obtain. To cope with the training data bottleneck, we investigate methods of combining multiple self-supervised tasks, i.e. tasks where models are explicitly trained with automatically generated labels. We propose two self-supervised pre-training tasks:one is End Identification and the other is Contrastive Scoring. Multi-task learning framework is then leveraged to combine these two tasks to jointly train the model. The pre-trained model is then fine-tuned using human-annotated script event prediction training data. Experimental results on the commonly used dataset show that our approach can achieve competitive performance compared to the previous models which are trained with the whole dataset by using just 10% of the training data, and our model trained on the whole dataset outperforms previous models significantly.
PSSL: Self-supervised Learning for Personalized Search with Contrastive Sampling
Authors: Yujia Zhou, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen
Venue: CIKM
Year: 2021

Personalized search plays a crucial role in improving user search experience owing to its ability to build user profiles based on historical behaviors. Previous studies have made great progress in extracting personal signals from the query log and learning user representations. However, neural personalized search is extremely dependent on sufficient data to train the user model. Data sparsity is an inevitable challenge for existing methods to learn high-quality user representations. Moreover, the overemphasis on final ranking quality leads to rough data representations and impairs the generalizability of the model. To tackle these issues, we propose a Personalized Search framework with Self-supervised Learning (PSSL) to enhance data representations. Specifically, we adopt a contrastive sampling method to extract paired self-supervised information from sequences of user behaviors in query logs. Four auxiliary tasks are designed to pre-train the sentence encoder and the sequence encoder used in the ranking model. They are optimized by contrastive loss which aims to close the distance between similar user sequences, queries, and documents. Experimental results on two datasets demonstrate that our proposed model PSSL achieves state-of-the-art performance compared with existing baselines.
Understanding and Resolving Performance Degradation in Deep Graph Convolutional Networks
Authors: Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, Jiashi Feng
Venue: CIKM
Year: 2021

A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation~(PROP) and a TRANsformation operation~(TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.
#StayHome or #Marathon?: Social Media Enhanced Pandemic Surveillance on Spatial-temporal Dynamic Graphs
Authors: Yichao Zhou, Jyun-Yu Jiang, Xiusi Chen, Wei Wang
Venue: CIKM
Year: 2021

COVID-19 has caused lasting damage to almost every domain in public health, society, and economy. To monitor the pandemic trend, existing studies rely on the aggregation of traditional statistical models and epidemic spread theory. In other words, historical statistics of COVID-19, as well as the population mobility data, become the essential knowledge for monitoring the pandemic trend. However, these solutions can barely provide precise prediction and satisfactory explanations on the long-term disease surveillance while the ubiquitous social media resources can be the key enabler for solving this problem. For example, serious discussions may occur on social media before and after some breaking events take place. To take advantage of the social media data, we propose a novel framework, Social Media enhAnced pandemic suRveillance Technique (SMART), which is composed of two modules: (i) information extraction module to construct heterogeneous knowledge graphs based on the extracted events and relationships among them; (ii) time series prediction module to provide both short-term and long-term forecasts of the confirmed cases and fatality at the state-level in the United States and to discover risk factors for COVID-19 interventions. Extensive experiments show that our method largely outperforms the state-of-the-art baselines by 7.3% and 7.4% in confirmed case/fatality prediction, respectively.
SeDyT: A General Framework for Multi-Step Event Forecasting via Sequence Modeling on Dynamic Entity Embeddings
Authors: Hongkuan Zhou, James Orme-Rogers, Rajgopal Kannan, Viktor K. Prasanna
Venue: CIKM
Year: 2021

Temporal Knowledge Graphs store events in the form of subjects, relations, objects, and timestamps which are often represented by dynamic heterogeneous graphs. Event forecasting is a critical and challenging task in Temporal Knowledge Graph reasoning that predicts the subject or object of an event in the future. To obtain temporal embeddings multi-step away in the future, existing methods learn generative models that capture the joint distribution of the observed events. To reduce the high computation costs, these methods rely on unrealistic assumptions of independence and approximations in training and inference. In this work, we propose SeDyT, a discriminative framework that performs sequence modeling on the dynamic entity embeddings to solve the multi-step event forecasting problem. SeDyT consists of two components: a Temporal Graph Neural Network that generates dynamic entity embeddings in the past and a sequence model that predicts the entity embeddings in the future. Compared with the generative models, SeDyT does not rely on any heuristic-based probability model and has low computation complexity in both training and inference. SeDyT is compatible with most Temporal Graph Neural Networks and sequence models. We also design an efficient training method that trains the two components in one gradient descent propagation. We evaluate the performance of SeDyT on five popular datasets. By combining temporal Graph Neural Network models and sequence models, SeDyT achieves an average of 2.4% MRR improvement when not using the validation set and more than 10% MRR improvement when using the validation set.
Tabular Data Concept Type Detection Using Star-Transformers
Authors: Yiwei Zhou, Siffi Singh, Christos Christodoulopoulos
Venue: CIKM
Year: 2021

Tabular data is an invaluable information resource for search, in-formation extraction and question answering about the world. It is critical to understand the semantic concept types for table columns in order to fully exploit the information in tabular data. In this paper, we focus on learning-based approaches for column concept type detection without relying on any metadata or queries to existing knowledge bases. We propose a model that employs both statistical and semantic features of table columns, and use Star-Transformers to gather and scatter information across the whole table to boost the performance on individual columns. We apply distant supervision to construct a tabular dataset with columns annotated with DBpedia classes. Our experiment results show that our model achieves 93.57 accuracy on the dataset, exceeding that of the state-of-the-art baselines.
Subtractive Aggregation for Attributed Network Anomaly Detection
Authors: Shuang Zhou, Qiaoyu Tan, Zhiming Xu, Xiao Huang, Fu-Lai Chung
Venue: CIKM
Year: 2021

Attributed network anomaly detection is essential in various networked systems. It aims to detect nodes that significantly deviate from their corresponding background. In conventional anomaly detection, the background is defined as the vast majority. But in networks, anomalies can be local and look normal when compared with the majority. While several efforts have explored to consider communities as the background, it remains challenging to learn suitable communities for effective anomaly detection. Also, the patterns of anomalies are unknown and it is nontrivial to define criteria of anomalies. To bridge the gap, in this paper, we argue that, by using appropriate models, it is sufficient to simply consider neighbor nodes as the background to detect anomalies. Correspondingly, we propose a novel abnormality-aware graph neural network (AAGNN). It utilizes subtractive aggregation to represent each node as the deviation from its neighbors (the background). Normal nodes with high confidence are employed as labels to learn a tailored hypersphere as the criterion of anomalies. Experiments demonstrate that AAGNN surpasses state-of-the-art methods significantly.
Relation Network and Causal Reasoning for Image Captioning
Authors: Dongming Zhou, Jing Yang
Venue: CIKM
Year: 2021

Image captioning is a cross-modal problem combining computer vision and natural language processing. A typical image captioning model uses a convolutional neural network to extract the features of an image and then uses an Long Short-Term Memory network to transform the representations of the features. However, this method has problems such as not including high-level semantics in the visual network and exposure bias in the language network. To overcome these problems, this paper proposes a novel image captioning model that combines relationship-aware and reinforcement learning. First, we design a relational awareness network as the visual network to mine the latent relationships between objects in an image. Then, a context semantic relational network is proposed to improve the accuracy of image captioning. The context semantic network can generate feature representations for arbitrary pixel positions in an image without association with any specific visual concepts. Subsequently, the high-level context semantics are used as external knowledge to guide the language network in generating sentences. Finally, a policy gradient training algorithm is designed to simplify the state value function in reinforcement learning. We have verified the effectiveness of the model on the MS-COCO and Flickr 30K datasets. The experimental results show that the model proposed in this paper achieves state-of-the-art results.
Self-Supervised Embedding for Subspace Clustering
Authors: Wenjie Zhu, Bo Peng, Chunchun Chen
Venue: CIKM
Year: 2021

Subspace clustering based on data self-expressive model aims to represent each data point as a linear combination of other data points on the dataset. Most existing methods focus on developing the regularization of self-expressive coefficients to solve the transductive unsupervised problem. In this paper, we propose a novel Self-Supervised Embedding for Subspace Clustering (S2ESC), which exploits a low-dimensional feature space where the data points belonging to the same category can be well-connected. Specifically, each data point is encouraged to be close to its transformation which is termed as self-supervised embedding learning. Therefore, the self-expressive coefficients can be learned by representing the data points over the corresponding transformations in the feature space. We introduce the low-rank and sparse regularization to illustrate the performance of self-supervised embedding. Extensive experiments on benchmark datasets demonstrate that our method outperforms the compared traditional subspace clustering methods.
Efficient Learning with Pseudo Labels for Query Cost Estimation
Authors: Shuncheng Liu, Xu Chen, Yan Zhao, Jin Chen, Rui Zhou, Kai Zheng
Venue: CIKM
Year: 2022

Query cost estimation, which is to estimate the query plan cost and query execution cost, is of utmost importance to query optimizers. Query plan cost estimation heavily relies on accurate cardinality estimation, and query execution cost estimation gives good hints on query latency, both of which are challenging in database management systems. Despite decades of research, existing studies either over-simplify the models only using histograms and polynomial calculation that leads to inaccurate estimates, or over-complicate them by using cumbersome neural networks with the requirements for large amounts of training data hence poor computational efficiency. Besides, most of the studies ignore the diversity of query plan structures. In this work, we propose a plan-based query cost estimation framework, called Saturn, which can eStimate cardinality and latency accurately and efficiently, for any query plan structures. Saturn first encodes each query plan tree into a compressed vector by using a traversal-based query plan autoencoder to cope with diverse plan structures. The compressed vectors can be leveraged to distinguish different query types, which is highly useful for downstream tasks. Then a pseudo label generator is designed to acquire all cardinality and latency labels with the execution part of the query plans in the training workload, which can significantly reduce the overhead of collecting the real cardinality and latency labels. Finally, a chain-wise transfer learning module is proposed to estimate the cardinality and latency of the query plan in a pipeline paradigm, which further enhances the efficiency. An extensive empirical study on benchmark data offers evidence that Saturn outperforms the state-of-the-art proposals in terms of accuracy, efficiency, and generalizability for query cost estimation.
HeGA: Heterogeneous Graph Aggregation Network for Trajectory Prediction in High-Density Traffic
Authors: Shuncheng Liu, Xu Chen, Ziniu Wu, Liwei Deng, Han Su, Kai Zheng
Venue: CIKM
Year: 2022

Trajectory prediction enables the fast and accurate response of autonomous driving navigation in complex and dense traffics. In this paper, we present a novel trajectory prediction network called Heterogeneous Graph Aggregation (HeGA) for high-density heterogeneous traffic, where the traffic agents of various categories interact densely with each other. To predict the trajectory of a target agent, HeGA first automatically selects neighbors that interact with it by our proposed adaptive neighbor selector, and then aggregates their interactions based on a novel two-phase aggregation transformer block. At last, the historical residual connection LSTM enhances the historical information awareness and decodes the spatial coordinates as the prediction results. Extensive experiments on real data demonstrate that the proposed network significantly outperforms the existing state-of-the-art competitors by over 27% on average displacement error (ADE) and over 31% on final displacement error (FDE). We also deploy HeGA in a state-of-the-art framework for autonomous driving, demonstrating its superior applicability based on three simulated environments with different densities and complexities.
Approximate and Interactive Processing of Aggregate Queries on Knowledge Graphs: A Demonstration
Authors: Yuxiang Wang, Arijit Khan, Xiaoliang Xu, Shuzhan Ye, Shihuang Pan, Yuhan Zhou
Venue: CIKM
Year: 2022

This paper demonstrates AGQ [26] - our system for approximate and interactive processing of aggregate queries on knowledge graphs (KGs), e.g., "what is the average price of cars produced in Germany?" One can support aggregate queries based on factoid queries, e.g., "find all cars produced in Germany", by applying an aggregate operation on factoid queries' answers. However, this straightforward method is problematic since both the accuracy and efficiency of factoid query processing would impact the performance of aggregate queries. Moreover, returning a one-time, exact result might add computation overhead and hinder users' engagement and interactivity. To this end, we design a system, called AGQ which employs a "sampling-estimation" model to answer aggregate queries over KGs. This is the first work to provide an approximate aggregate result with effective and interactive accuracy guarantees, and without relying on factoid queries. Our demonstration highlights (1) a novel semantic-aware sampling to collect a high-quality random sample through a random walk based on KG embedding, followed by our unbiased (or, consistent) estimators for COUNT, SUM, AVG to compute the approximate aggregate results using the random sample, with a confidence interval-based accuracy guarantee. (2) AGQ supports interactive improvements of accuracy, complex queries with filter, GROUP-BY, MAX/MIN, and different graph shapes, e.g., chain, cycle, star, flower. (3) Its GUI helps users compare simple and complex aggregate queries, intermediate results as the queries progress, confidence intervals, relative errors, and various schemas for different valid answers in a user-friendly and interactive manner. Additionally, our system permits users to input queries in natural languages, keywords, or to select from a set of example graph queries.
Cost-constrained Minimal Steiner Tree Enumeration
Authors: Yuya Sasaki
Venue: CIKM
Year: 2022

The Steiner tree enumeration problem is a well-known problem that asks for enumerating Steiner trees. Although numerous theoretical works proposed algorithms for the problem and analyzed their complexity, there are no practical algorithms and empirical studies. In this paper, we first study the Steiner tree enumeration problem practically. First, we define a practical problem cost-constrained minimal Steiner tree enumeration problem, which enumerates minimal Steiner trees with costs not larger than a given threshold. Second, to address the problem, we propose a binary decision diagram (BDD)-based algorithm. The BDD-based algorithm constructs a BDD that compactly represents the set of minimal Steiner trees and then traverses the BDD for enumeration. We develop a novel frontier-based algorithm to construct such BDDs efficiently. Furthermore, we extend our algorithm to be scalable for large-scale graphs by preprocessing the given graph and controlling the number of generated Steiner trees to reduce memory and computation costs. We validate that our algorithm can efficiently enumerate minimal Steiner trees in real-world graphs.
Graph-based Management and Mining of Blockchain Data
Authors: Arijit Khan, Cuneyt Gurcan Akcora
Venue: CIKM
Year: 2022

The mainstream adoption of blockchains led to the preparation of many decentralized applications and web platforms, including Web 3.0, a peer-to-peer internet with no single authority. The data stored in blockchain can be considered as big data -- massive-volume, dynamic, and heterogeneous. Due to highly connected structure, graph-based modeling is an optimal tool to analyze the data stored in blockchains. Recently, several research works performed graph analysis on the publicly available blockchain data to reveal insights into its business transactions and for critical downstream tasks, e.g., cryptocurrency price prediction, phishing scams and counterfeit token detection. In this tutorial, we discuss relevant literature on blockchain data structures, storage, categories, data extraction and graphs construction, graph mining, topological data analysis, and machine learning methods used, target applications, and the new insights revealed by them, aiming towards providing a clear view of unified graph-data models for UTXO and account-based blockchains. We also emphasize future research directions.
Building Natural Language Processing Applications with EasyNLP
Authors: Chengyu Wang, Minghui Qiu, Jun Huang
Venue: CIKM
Year: 2022

The successful application of Pre-Trained Models (PTMs) has revolutionized the development of Natural Language Processing (NLP) by large-scale self-supervised pre-training. However, it is not easy to obtain high-performing models in domain-specific applications and deploy them online with strict QPS (Query Per Second) requirements for industrial practitioners. To solve these issues, the EasyNLP toolkit is designed for building PTM-based NLP applications with ease, which supports a comprehensive suite of NLP algorithms and is suitable for meeting the inference requirements in industry. It features knowledge-enhanced pre-training that captures rich domain knowledge to better support domain-specific applications. In addition, the knowledge distillation and prompt-based few-shot learning functionalities are provided to improve the performance of large-scale PTMs with little training data available, and to distill models to smaller ones that are suitable for online deployment. EasyNLP provides a unified framework of model training, inference and deployment for real-world applications, using simple high-level APIs or command-line tools. Currently, EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. EasyNLP is also beneficial for academia, as it integrates state-of-the-art methods and models to make it easy for researchers to benchmark and develop their own algorithms. We have released EasyNLP to public at GitHub (https://github.com/alibaba/EasyNLP).
The Third Workshop on Data-driven Intelligent Transportation
Authors: Hua Wei, Guni Sharon, Cathy Wu, Sanjay Chawla, Zhenhui Li
Venue: CIKM
Year: 2022

Traffic is the pulse of the city. Transportation systems can involve humans, vehicles, shipments, information technology, and the physical infrastructure, all interacting in complex ways. Intelligent transportation enables the city to function in a more efficient and effective way. A wide range of city data become increasingly available, such as taxi trips, surveillance camera data, human mobility data from mobile phones or location-based services, events from social media, car accident reports, bike-sharing information, Points-Of-Interest, traffic sensors, public transportation data, and many more. This abundance of data poses a grand challenge to the CIKM research community: How to utilize such data toward city intelligence, across various transportation tasks? The 3rd workshop of "Data-driven Intelligent Transportation" welcomes articles and presentations in the areas of transportation systems, data mining, and artificial intelligence, conveying new advances and developments in theory, modeling, simulation, testing, case studies, as well as large-scale deployment.