Reviewerly/exHarmony huggingface dataset:
On Structural Expressive Power of Graph Transformers

Graph Transformer has recently received wide attention in the research community with its outstanding performance, yet its structural expressive power has not been well analyzed. Inspired by the connections between Weisfeiler-Lehman (WL) graph isomorphism test and graph neural network (GNN), we introduce SEG-WL test (Structural Encoding enhanced G lobal Weisfeiler-Lehman test), a generalized graph isomorphism test algorithm as a powerful theoretical tool for exploring the structural discriminative power of graph Transformers. We theoretically prove that the SEG-WL test is an expressivity upper bound on a wide range of graph Transformers, and the representational power of SEG-WL test can be approximated by a simple Transformer network arbitrarily under certain conditions. With the SEG-WL test, we show how graph Transformers' expressive power is determined by the design of structural encodings, and present conditions that make the expressivity of graph Transformers beyond WL test and GNNs. Moreover, motivated by the popular shortest path distance encoding, we follow the theory-oriented principles and develop a provably stronger structural encoding method, Shortest Path Induced Subgraph (SPIS) encoding. Our theoretical findings provide a novel and practical paradigm for investigating the expressive power of graph Transformers, and extensive synthetic and real-world experiments empirically verify the strengths of our proposed methods.
Uncertainty Quantification in Deep Learning

Deep neural networks (DNNs) have achieved enormous success in a wide range of domains, such as computer vision, natural language processing and scientific areas. However, one key bottleneck of DNNs is that they are ignorant about the uncertainties in their predictions. They can produce wildly wrong predictions without realizing, and can even be confident about their mistakes. Such mistakes can cause misguided decisions-sometimes catastrophic in critical applications, ranging from self-driving cars to cyber security to automatic medical diagnosis. In this tutorial, we present recent advancements in uncertainty quantification for DNNs and their applications across various domains. We first provide an overview of the motivation behind uncertainty quantification, different sources of uncertainty, and evaluation metrics. Then, we delve into several representative uncertainty quantification methods for predictive models, including ensembles, Bayesian neural networks, conformal prediction, and others. We go on to discuss how uncertainty can be utilized for label-efficient learning, continual learning, robust decision-making, and experimental design. Furthermore, we showcase examples of uncertainty-aware DNNs in various domains, such as health, robotics, and scientific machine learning. Finally, we summarize open challenges and future directions in this area.
Investigating Trojan Attacks on Pre-trained Language Model-powered Database Middleware

The recent success of pre-trained language models (PLMs) such as BERT has resulted in the development of various beneficial database middlewares, including natural language query interfaces and entity matching. This shift has been greatly facilitated by the extensive external knowledge of PLMs. However, as PLMs are often provided by untrusted third parties, their lack of standardization and regulation poses significant security risks that have yet to be fully explored. This paper investigates the security threats posed by malicious PLMs to these emerging database middleware. We specifically propose a novel type of Trojan attack, where a maliciously designed PLM causes unexpected behavior in the database middleware. These Trojan attacks possess the following characteristics: (1) Triggerability: The Trojan-infected database middleware will function normally with normal input, but will likely malfunction when triggered by the attacker. (2) Imperceptibility: There is no need for noticeable modification of the input to trigger the Trojan. (3) Generalizability: The Trojan is capable of targeting a variety of downstream tasks, not just one specific task. We thoroughly evaluate the impact of these Trojan attacks through experiments and analyze potential countermeasures and their limitations. Our findings could aid in the creation of stronger mechanisms for the implementation of PLMs in database middleware.
Delving into Global Dialogue Structures: Structure Planning Augmented Response Selection for Multi-turn Conversations

Retrieval-based dialogue systems are a crucial component of natural language processing, employing information retrieval techniques to select responses from a predefined pool of candidates. The advent of pre-trained language models (PLMs) has significantly advanced the field, with a prevailing paradigm that involves post-training PLMs on specific dialogue corpora, followed by fine-tuning for the response selection (RS) task. This post-training process aims to capture dialogue-specific features, as most PLMs are originally trained on plain text. However, prior approaches predominantly rely on self-supervised tasks or session-level graph neural networks during post-training, focusing on capturing underlying patterns of coherent dialogues without explicitly refining the global pattern across the entire dialogue corpus. Consequently, the learned knowledge for organizing coherent dialogues remains isolated, heavily reliant on specific contexts. Additionally, interpreting or visualizing the implicit knowledge acquired through self-supervised tasks proves challenging. In this study, we address these limitations by explicitly refining the knowledge required for response selection and structuring it into a coherent global flow, known as "dialogue structure." This structure captures the inter-dependency of utterances and topic shifts, thereby enhancing the response selection task. To achieve this, we propose a novel structure model comprising a state recognizer and a structure planner. This model effectively captures the flow within the utterance history and plans the trajectory of future utterances. Importantly, the structure model operates orthogonally to the retrieval model, enabling seamless integration with existing retrieval models and facilitating collaborative training. Extensive experiments conducted on three benchmark datasets demonstrate the superior performance of our method over a wide range of competitive baselines, establishing a new state-of-the-art in the field.
Fast Text Generation with Text-Editing Models

Text-editing models have recently become a prominent alternative to seq2seq models for monolingual text-generation tasks such as grammatical error correction, simplification, and style transfer. These tasks share a common trait -- they exhibit a large amount of textual overlap between the source and target texts. Text-editing models take advantage of this observation and learn to generate the output by predicting edit operations applied to the source sequence. In contrast, seq2seq models generate outputs word-by-word from scratch thus making them slow at inference time. Text-editing models provide several benefits over seq2seq models including faster inference speed, higher sample efficiency, and better control and explainability of the outputs. This tutorial provides a comprehensive overview of text-editing models and discusses how they can be used to mitigate hallucination and bias, both pressing challenges in the field of text generation. Finally, we discuss how to optimize latency of large language models via distillation to text-editing models and other means.
The 4th International Workshop on Talent and Management Computing (TMC'2023)

In today's competitive and fast-evolving business environment, it is a critical time for organizations to rethink how to deal with the talent and management related tasks in a quantitative manner. Indeed, thanks to the era of big data, the availability of large-scale talent data provides unparalleled opportunities for business leaders to understand the rules of talent and management, which in turn deliver intelligence for effective decision making and management for their organizations. In the past few years, talent and management computing have increasingly attracted attentions from KDD communities, and a number of research/applied data science efforts have been devoted. To this end, the purpose of this workshop, i.e., the 4th International Workshop on Talent and Management Computing (TMC'2023), is to bring together researchers and practitioners to discuss both the critical problems faced by talent and management related domains, and potential data-driven solutions by leveraging state-of-the-art data mining technologies.
A Data-centric Framework to Endow Graph Neural Networks with Out-Of-Distribution Detection Ability

Out-of-distribution (OOD) detection, which aims to identify OOD samples from in-distribution (ID) ones in test time, has become an essential problem in machine learning. However, existing works are mostly conducted on Euclidean data, and the problem in graph-structured data remains under-explored. Several recent works begin to study graph OOD detection, but they all need to train a graph neural network (GNN) from scratch with high computational cost. In this work, we make the first attempt to endow a well-trained GNN with the OOD detection ability without modifying its parameters. To this end, we design a post-hoc framework with Adaptive Amplifier for Graph OOD Detection, named AAGOD, concentrating on data-centric manipulation. The insight of AAGOD is to superimpose a parameterized amplifier matrix on the adjacency matrix of each original input graph. The amplifier can be seen as prompts and is expected to emphasize the key patterns helpful for graph OOD detection, thereby enlarging the gap between OOD and ID graphs. Then well-trained GNNs can be reused to encode the amplified graphs into vector representations, and pre-defined scoring functions can further convert the representations into detection scores. Specifically, we design a Learnable Amplifier Generator (LAG) to customize amplifiers for different graphs, and propose a Regularized Learning Strategy (RLS) to train parameters with no OOD data required. Experiment results show that AAGOD can be applied on various GNNs to enable the OOD detection ability. Compared with the state-of-the-art baseline in graph OOD detection, on average AAGOD has 6.21% relative enhancement in AUC and a 34 times faster training speed. Code and data are available at https://github.com/BUPT-GAMMA/AAGOD.
PAT: Geometry-Aware Hard-Label Black-Box Adversarial Attacks on Text

Despite a plethora of prior explorations, conducting text adversarial attacks in practical settings is still challenging with the following constraints: black box -- the inner structure of the victim model is unknown; hard label -- the attacker only has access to the top-1 prediction results; and semantic preservation - the perturbation needs to preserve the original semantics. In this paper, we present PAT, a novel adversarial attack method employed under all these constraints. Specifically, PAT explicitly models the adversarial and non-adversarial prototypes and incorporates them to measure semantic changes for replacement selection in the hard-label black-box setting to generate high-quality samples. In each iteration, PAT finds original words that can be replaced back and selects better candidate words for perturbed positions in a geometry-aware manner guided by this estimation, which maximally improves the perturbation construction and minimally impacts the original semantics. Extensive evaluation with benchmark datasets and state-of-the-art models shows that PAT outperforms existing text adversarial attacks in terms of both attack effectiveness and semantic preservation. Moreover, we validate the efficacy of PAT against industry-leading natural language processing platforms in real-world settings.
Geographical Aspects of Cybercrime: A Literature Review

Cybercrime has become a pervasive and complex issue in today’s interconnected world, posing significant threats to individuals, businesses, and governments. This paper aims to provide a comprehensive overview of the diverse aspects related to cybercrime, including its historical context, demographic and geographic dimensions, environmental influences, and preventive strategies. This review provides a holistic overview of the multifaceted dimensions of cybercrime. By understanding its historical context, demographic and geographic aspects, environmental influences, and preventive strategies, policymakers, law enforcement agencies, and researchers can work collaboratively to combat cyber threats effectively. Such a comprehensive approach will help create a safer digital environment and protect individuals, organizations, and societies from the adverse impacts of cybercrime. Moreover, through ongoing research and collaboration, it is possible to develop innovative solutions and adapt to the evolving landscape of cyber threats, ensuring a secure and resilient digital future.
Enhancing Graph Representations Learning with Decorrelated Propagation

In recent years, graph neural networks (GNNs) have been widely used in many domains due to their powerful capability in representation learning on graph-structured data. While a majority of extant studies focus on mitigating the over-smoothing problem, recent works also reveal the limitation of GNN from a new over-correlation perspective which states that the learned representation becomes highly correlated after feature transformation and propagation in GNNs. In this paper, we thoroughly re-examine the issue of over-correlation in deep GNNs, both empirically and theoretically. We demonstrate that the propagation operator in GNNs exacerbates the feature correlation. In addition, we discovered through empirical study that existing decorrelation solutions fall short of maintaining a low feature correlation, potentially encoding redundant information. Thus, to more effectively address the over-correlation problem, we propose a decorrelated propagation scheme (DeProp) as a fundamental component to decorrelate the feature learning in GNN models, which achieves feature decorrelation at the propagation step. Comprehensive experiments on multiple real-world datasets demonstrate that DeProp can be easily integrated into prevalent GNNs, leading to significant performance enhancements. Furthermore, we find that it can be used to solve over-smoothing and over-correlation problems simultaneously and significantly outperform state-of-the-art methods on missing feature settings. The code is available at https://github.com/hualiu829/DeProp.
Multiplex Heterogeneous Graph Neural Network with Behavior Pattern Modeling

Heterogeneous graph neural networks have gained great popularity in tackling various network analysis tasks on heterogeneous network data. However, most existing works mainly focus on general heterogeneous networks, and assume that there is only one type of edge between two nodes, while ignoring the multiplex characteristics between multi-typed nodes in multiplex heterogeneous networks and the different importance of multiplex structures among nodes for node embedding. In addition, the over-smoothing issue of graph neural networks limits existing models to only capturing local structure signals but hardly learning the global relevant information of the network. To tackle these challenges, this work proposes a model called Behavior Pattern based Heterogeneous Graph Neural Network (BPHGNN) for multiplex heterogeneous network embedding. Specifically, BPHGNN can collaboratively learn node representations across different multiplex structures among nodes with adaptive importance learning from local and global perspectives in multiplex heterogeneous networks through depth behavior pattern aggregation and breadth behavior pattern aggregation. Extensive experiments on six real-world networks with various network analytical tasks demonstrate the significant superiority of BPHGNN against state-of-the-art approaches in terms of various evaluation metrics.
Foundations and Applications in Large-scale AI Models: Pre-training, Fine-tuning, and Prompt-based Learning

Deep learning techniques have advanced rapidly in recent years, leading to significant progress in pre-trained and fine-tuned large-scale AI models. For example, in the natural language processing domain, the traditional "pre-train, fine-tune" paradigm is shifting towards the "pre-train, prompt, and predict" paradigm, which has achieved great success on many tasks across different application domains such as ChatGPT/BARD for Conversational AI and P5 for a unified recommendation system. Moreover, there has been a growing interest in models that combine vision and language modalities (vision-language models) which are applied to tasks like Visual Captioning/Generation. Considering the recent technological revolution, it is essential to emphasize these paradigm shifts and highlight the paradigms with the potential to solve different tasks. We thus provide a platform for academic and industrial researchers to showcase their latest work, share research ideas, discuss various challenges, and identify areas where further research is needed in pre-training, fine-tuning, and prompt-learning methods for large-scale AI models. We foster the development of a strong research community focused on solving challenges related to large-scale AI models, providing superior and impactful strategies that can change people's lives in the future.
Dense Representation Learning and Retrieval for Tabular Data Prediction

Data science is concerned with mining data patterns from a database, which is assembled by tabular data. As the routine of machine learning, most of the previous work mining the tabular data's pattern based on a single instance. However, they neglect the similar tabular data instances that could help make the label prediction of the target data instance. Recently, some retrieval-based methods for tabular data label prediction have been proposed, which, however, treat the data as sparse vectors to perform the retrieval, which fails to make use of the semantic information of the tabular data. To address such a problem, in this paper, we propose a novel framework of dense retrieval on tabular data (DERT) to support flexible data representation learning and effective label prediction on tabular data. DERT consists of two major components: (i) the encoder that makes the tabular data as embeddings, which could be trained by flexible neural networks and auxiliary loss functions; (ii) the retrieval and prediction component, which makes use of similar rows in the table to make label prediction of the target row. We test DERT on two tasks based on five real-world datasets and experimental results show that DERT achieves consistent improvements over the state-of-the-art and various baselines.
Online Few-Shot Time Series Classification for Aftershock Detection

Seismic monitoring systems sift through seismograms in real-time, searching for target events, such as underground explosions. In this monitoring system, a burst of aftershocks (minor earthquakes occur after a major earthquake over days or even years) can be a source of confounding signals. Such a burst of aftershock signals can overload the human analysts of the monitoring system. To alleviate this burden at the onset of a sequence of events (e.g., aftershocks), a human analyst can label the first few of these events and start an online classifier to filter out subsequent aftershock events. We propose an online few-shot classification model FewSig for time series data for the above use case. The framework of FewSig consists of a selective model to identify the high-confidence positive events which are used for updating the models and a general classifier to label the remaining events. Our specific technique uses a %two-level decision tree selective model based on sliding DTW distance and a general classifier model based on distance metric learning with Neighborhood Component Analysis (NCA). The algorithm demonstrates surprising robustness when tested on univariate datasets from the UEA/UCR archive. Furthermore, we show two real-world earthquake events where the FewSig reduces the human effort in monitoring applications by filtering out the aftershock events.
A constant round quantum secure protocol for oblivious polynomial evaluation

Oblivious polynomial evaluation (OPE) is a secure two-party cryptographic protocol that enables a receiver to obliviously retrieve polynomial value on its private input for a private polynomial of a sender. This paper deals with the problem of OPE in the quantum domain. Herein, we propose the construction of the first quantum secure OPE, namely QuOPE. Our design is a constant round protocol that only requires quantum communication of one quantum superposition state that achieves all the stipulated security requirements of an OPE. In particular, privacy and security of the sender and receiver’s private data are ensured. In addition, we extend the idea of QuOPE to quantum rational polynomial evaluation (QuORPE). Furthermore, we explore the possibility of utilizing QuOPE as a building block in the automated medical diagnosis platform.
Performance of heterogenous neuro-fuzzy ensembles over medical datasets

Neuro-fuzzy systems combine the abilities of both artificial neural networks and fuzzy systems. They are easily trainable and provide a certain level of interpretability. Their performance has been assessed in different application domains and many attempts have been made to improve it using ensemble learning. However, to the best of our knowledge, no study has investigated the performance of heterogeneous neuro-fuzzy ensembles in a medical context. In this study, we constructed, evaluated, and compared the performance of 26 heterogeneous neuro-fuzzy ensembles on four medical datasets. The five single classifiers used were based on the Takagi-Sugeno-Kang (TSK) and Mamdani fuzzy inference systems. The metrics employed to measure the performance of the ensemble classifiers were the accuracy, precision, and recall. Additionally, the Borda count method and Scott-Knott statistical test were used to rank and cluster the classifiers, respectively. The results show that ensemble classifiers rarely outperform their base classifiers. Moreover, ensembles composed of TSK base classifiers performed best. In addition, we noticed that ensembles comprising four base learners achieved the best performance. Finally, no ensemble classifier managed to score high-performance values across the four datasets.
PSLOG: Pretraining with Search Logs for Document Ranking

Recently, pretrained models have achieved remarkable performance not only in natural language processing but also in information retrieval (IR). Previous studies show that IR-oriented pretraining tasks can achieve better performance than only finetuning pretrained language models in IR datasets. Besides, the massive search log data obtained from mainstream search engines can be used in IR pretraining, for it contains users' implicit judgments of document relevance under a concrete query. However, existing methods mainly use direct query-document click signals to pretrain models. The potential supervision signals from search logs are far from being well explored. In this paper, we propose to comprehensively leverage four query-document relevance relations, including co-interaction and multi-hop relations, to pretrain ranking models in IR. Specifically, we focus on the user's click behavior and construct an Interaction Graph to represent the global relevance relations between queries and documents from all search logs. With the graph, we can consider the co-interaction and multi-hop q-d relationships through their neighbor nodes. Based on the relations extracted from the interaction graph, we propose four strategies to generate contrastive positive and negative q-d pairs and use these data to pretrain ranking models. Experimental results on both industrial and academic datasets demonstrate the effectiveness of our method.