OpenRaiser/Intern-Atlas huggingface dataset:
Online Anomaly Detection Based On Reservoir Sampling and LOF for IoT devices
Venue: arXiv cs.LG 2022
Year: 2022

The growing number of IoT devices and their use to monitor the operation of machines and equipment increases interest in anomaly detection algorithms running on devices. However, the difficulty is the limitations of the available computational and memory resources on the devices. In the case of microcontrollers (MCUs), these are single megabytes of program and several hundred kilobytes of working memory. Consequently, algorithms must be appropriately matched to the capabilities of the devices. In the paper, we analyse the processing pipeline for anomaly detection and implementation of the Local Outliner Factor (LOF) algorithm on a MCU. We also show that it is possible to train such an algorithm directly on the device, which gives great potential to use the solution in real devices.
Learning Time Delay Systems with Neural Ordinary Differential Equations
Venue: arXiv cs.LG 2022
Year: 2022

A novel way of using neural networks to learn the dynamics of time delay systems from sequential data is proposed. A neural network with trainable delays is used to approximate the right hand side of a delay differential equation. We relate the delay differential equation to an ordinary differential equation by discretizing the time history and train the corresponding neural ordinary differential equation (NODE) to learn the dynamics. An example on learning the dynamics of the Mackey-Glass equation using data from chaotic behavior is given. After learning both the nonlinearity and the time delay, we demonstrate that the bifurcation diagram of the neural network matches that of the original system.
Biometric Identification Systems With Both Chosen and Generated Secret Keys by Allowing Correlation
Venue: arXiv cs.IT 2022
Year: 2022

We propose a biometric identification system where the chosen- and generated-secret keys are used simultaneously, and investigate its fundamental limits from information-theoretic perspectives. The system consists of two phases: enrollment and identification phases. In the enrollment phase, for each user, the encoder uses a secret key, which is chosen independently, and the biometric identifier to generate another secret key and a helper {data}. In the identification phase, observing the biometric sequence of the identified user, the decoder estimates index, chosen- and generated-secret keys of the identified user based on the helper {data} stored in the system database. In this study, the capacity region of such a system is characterized. In the problem setting, we allow chosen- and generated-secret keys to be correlated. As a result, by permitting the correlation of the two secret keys, the sum rate of the identification, chosen- and {generated-secret key} rates can achieve a larger value compared to the case where the keys {do not correlate}. Moreover, the minimum amount of the storage rate changes in accordance with both the identification and chosen-secret key rates, but that of the privacy-leakage rate depends only on the identification rate.
Framing Algorithmic Recourse for Anomaly Detection
Venue: arXiv cs.LG 2022
Year: 2022

The problem of algorithmic recourse has been explored for supervised machine learning models, to provide more interpretable, transparent and robust outcomes from decision support systems. An unexplored area is that of algorithmic recourse for anomaly detection, specifically for tabular data with only discrete feature values. Here the problem is to present a set of counterfactuals that are deemed normal by the underlying anomaly detection model so that applications can utilize this information for explanation purposes or to recommend countermeasures. We present an approach -- Context preserving Algorithmic Recourse for Anomalies in Tabular data (CARAT), that is effective, scalable, and agnostic to the underlying anomaly detection model. CARAT uses a transformer based encoder-decoder model to explain an anomaly by finding features with low likelihood. Subsequently semantically coherent counterfactuals are generated by modifying the highlighted features, using the overall context of features in the anomalous instance(s). Extensive experiments help demonstrate the efficacy of CARAT.
Diet Code Is Healthy: Simplifying Programs for Pre-trained Models of Code
Venue: arXiv cs.SE 2022
Year: 2022

Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy which selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCodeBERT provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.
The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning
Venue: arXiv cs.CV 2022
Year: 2022

Vision transformers have recently set off a new wave in the field of medical image analysis due to their remarkable performance on various computer vision tasks. However, recent hybrid-/transformer-based approaches mainly focus on the benefits of transformers in capturing long-range dependency while ignoring the issues of their daunting computational complexity, high training costs, and redundant dependency. In this paper, we propose to employ adaptive pruning to transformers for medical image segmentation and propose a lightweight and effective hybrid network APFormer. To our best knowledge, this is the first work on transformer pruning for medical image analysis tasks. The key features of APFormer mainly are self-supervised self-attention (SSA) to improve the convergence of dependency establishment, Gaussian-prior relative position embedding (GRPE) to foster the learning of position information, and adaptive pruning to eliminate redundant computations and perception information. Specifically, SSA and GRPE consider the well-converged dependency distribution and the Gaussian heatmap distribution separately as the prior knowledge of self-attention and position embedding to ease the training of transformers and lay a solid foundation for the following pruning operation. Then, adaptive transformer pruning, both query-wise and dependency-wise, is performed by adjusting the gate control parameters for both complexity reduction and performance improvement. Extensive experiments on two widely-used datasets demonstrate the prominent segmentation performance of APFormer against the state-of-the-art methods with much fewer parameters and lower GFLOPs. More importantly, we prove, through ablation studies, that adaptive pruning can work as a plug-n-play module for performance improvement on other hybrid-/transformer-based methods. Code is available at https://github.com/xianlin7/APFormer.
Design and Optimization of Aerial-Aided Multi-Access Edge Computing towards 6G
Venue: arXiv cs.NI 2022
Year: 2022

Ubiquity in network coverage is one of the main features of 5G and is expected to be extended to the computing domain in 6G. In order to provide this holistic approach of ubiquity in communication and computation, an integration of satellite, aerial and terrestrial networks is foreseen. In particular, the rising amount of applications such as In-Flight Entertainment and Connectivity Services (IFECS) and SDN-enabled satellites renders network management more challenging. Moreover, due to the stringent Quality of Service (QoS) requirements edge computing gains in importance for these applications. Here, network performance can be boosted by considering components of the aerial network, like aircrafts, as potential Multi-Access Edge Computing (MEC) nodes. Thus, we propose an Aerial-Aided Multi-Access Edge Computing (AA-MEC) architecture that provides a framework for optimal management of computing resources and internet-based services in the sky. Furthermore, we formulate optimization problems to minimize the network latency for the two use cases of providing IFECS to other aircrafts in the sky and providing services for offloading AI/ML-tasks from satellites. Due to the dynamic nature of the satellite and aerial networks, we propose a re-configurable optimization. For the transforming network we continuously identify the optimal MEC node for each application and the optimal path to the destination MEC node. In summary, our results demonstrate that using AA-MEC improves network latency performance by 10.43% compared to the traditional approach of using only terrestrial MEC nodes for latency-critical applications such as online gaming. Furthermore, while comparing our proposed dynamic approach with a static one, we record a benefit of at least 6.7% decrease in flow latency for IFECS and 56.03% decrease for computation offloading.
Deep Active Visual Attention for Real-time Robot Motion Generation: Emergence of Tool-body Assimilation and Adaptive Tool-use
Venue: arXiv cs.RO 2022
Year: 2022

Sufficiently perceiving the environment is a critical factor in robot motion generation. Although the introduction of deep visual processing models have contributed in extending this ability, existing methods lack in the ability to actively modify what to perceive; humans perform internally during visual cognitive processes. This paper addresses the issue by proposing a novel robot motion generation model, inspired by a human cognitive structure. The model incorporates a state-driven active top-down visual attention module, which acquires attentions that can actively change targets based on task states. We term such attentions as role-based attentions, since the acquired attention directed to targets that shared a coherent role throughout the motion. The model was trained on a robot tool-use task, in which the role-based attentions perceived the robot grippers and tool as identical end-effectors, during object picking and object dragging motions respectively. This is analogous to a biological phenomenon called tool-body assimilation, in which one regards a handled tool as an extension of one's body. The results suggested an improvement of flexibility in model's visual perception, which sustained stable attention and motion even if it was provided with untrained tools or exposed to experimenter's distractions.
Current Challenges of Cyber Threat and Vulnerability Identification Using Public Enumerations
Venue: arXiv cs.CR 2022
Year: 2022

Identification of cyber threats is one of the essential tasks for security teams. Currently, cyber threats can be identified using knowledge organized into various formats, enumerations, and knowledge bases. This paper studies the current challenges of identifying vulnerabilities and threats in cyberspace using enumerations and data about assets. Although enumerations are used in practice, we point out several issues that still decrease the quality of vulnerability and threat identification. Since vulnerability identification methods are based on network monitoring and agents, the issues are related to the asset discovery, the precision of vulnerability discovery, and the amount of data. On the other hand, threat identification utilizes graph-based, nature-language, machine-learning, and ontological approaches. The current trend is to propose methods that utilize tactics, techniques, and procedures instead of low-level indicators of compromise to make cyber threat identification more mature. Cooperation between standards from threat, vulnerability, and asset management is also an unresolved issue confirmed by analyzing relationships between public enumerations and knowledge bases. Last, we studied the usability of techniques from the MITRE ATT&CK knowledge base for threat modeling using network monitoring to capture data. Although network traffic is not the most used data source, it allows the modeling of almost all tactics from the MITRE ATT&CK.
Linearizing Partial Search Orders
Venue: arXiv cs.DM 2022
Year: 2022

In recent years, questions about the construction of special orderings of a given graph search were studied by several authors. On the one hand, the so called end-vertex problem introduced by Corneil et al. in 2010 asks for search orderings ending in a special vertex. On the other hand, the problem of finding orderings that induce a given search tree was introduced already in the 1980s by Hagerup and received new attention most recently by Beisegel et al. Here, we introduce a generalization of some of these problems by studying the question whether there is a search ordering that is a linear extension of a given partial order on a graph's vertex set. We show that this problem can be solved in polynomial time on chordal bipartite graphs for LBFS, which also implies the first polynomial-time algorithms for the end-vertex problem and two search tree problems for this combination of graph class and search. Furthermore, we present polynomial-time algorithms for LBFS and MCS on split graphs which generalize known results for the end-vertex and search tree problems.
Generative Anomaly Detection for Time Series Datasets
Venue: arXiv cs.LG 2022
Year: 2022

Traffic congestion anomaly detection is of paramount importance in intelligent traffic systems. The goals of transportation agencies are two-fold: to monitor the general traffic conditions in the area of interest and to locate road segments under abnormal congestion states. Modeling congestion patterns can achieve these goals for citywide roadways, which amounts to learning the distribution of multivariate time series (MTS). However, existing works are either not scalable or unable to capture the spatial-temporal information in MTS simultaneously. To this end, we propose a principled and comprehensive framework consisting of a data-driven generative approach that can perform tractable density estimation for detecting traffic anomalies. Our approach first clusters segments in the feature space and then uses conditional normalizing flow to identify anomalous temporal snapshots at the cluster level in an unsupervised setting. Then, we identify anomalies at the segment level by using a kernel density estimator on the anomalous cluster. Extensive experiments on synthetic datasets show that our approach significantly outperforms several state-of-the-art congestion anomaly detection and diagnosis methods in terms of Recall and F1-Score. We also use the generative model to sample labeled data, which can train classifiers in a supervised setting, alleviating the lack of labeled data for anomaly detection in sparse settings.
Extracting Weighted Finite Automata from Recurrent Neural Networks for Natural Languages
Venue: arXiv cs.CL 2022
Year: 2022

Recurrent Neural Networks (RNNs) have achieved tremendous success in sequential data processing. However, it is quite challenging to interpret and verify RNNs' behaviors directly. To this end, many efforts have been made to extract finite automata from RNNs. Existing approaches such as exact learning are effective in extracting finite-state models to characterize the state dynamics of RNNs for formal languages, but are limited in the scalability to process natural languages. Compositional approaches that are scablable to natural languages fall short in extraction precision. In this paper, we identify the transition sparsity problem that heavily impacts the extraction precision. To address this problem, we propose a transition rule extraction approach, which is scalable to natural language processing models and effective in improving extraction precision. Specifically, we propose an empirical method to complement the missing rules in the transition diagram. In addition, we further adjust the transition matrices to enhance the context-aware ability of the extracted weighted finite automaton (WFA). Finally, we propose two data augmentation tactics to track more dynamic behaviors of the target RNN. Experiments on two popular natural language datasets show that our method can extract WFA from RNN for natural language processing with better precision than existing approaches. Our code is available at https://github.com/weizeming/Extract_WFA_from_RNN_for_NL.
From Kernel Methods to Neural Networks: A Unifying Variational Formulation
Venue: arXiv cs.LG 2022
Year: 2022

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator and on a generic Radon-domain norm. We establish the existence of a minimizer and give the parametric form of the solution(s) under very mild assumptions. When the norm is Hilbertian, the proposed formulation yields a solution that involves radial-basis functions and is compatible with the classical methods of machine learning. By contrast, for the total-variation norm, the solution takes the form of a two-layer neural network with an activation function that is determined by the regularization operator. In particular, we retrieve the popular ReLU networks by letting the operator be the Laplacian. We also characterize the solution for the intermediate regularization norms $\|\cdot\|=\|\cdot\|_{L_p}$ with $p\in(1,2]$. Our framework offers guarantees of universal approximation for a broad family of regularization operators or, equivalently, for a wide variety of shallow neural networks, including the cases (such as ReLU) where the activation function is increasing polynomially. It also explains the favorable role of bias and skip connections in neural architectures.
Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss
Venue: arXiv cs.SD 2022
Year: 2022

In this paper, we tackle the new Language-Based Audio Retrieval task proposed in DCASE 2022. Firstly, we introduce a simple, scalable architecture which ties both the audio and text encoder together. Secondly, we show that using this architecture along with contrastive loss allows the model to significantly beat the performance of the baseline model. Finally, in addition to having an extremely low training memory requirement, we are able to use pretrained models as it is without needing to finetune them. We test our methods and show that using a combination of our methods beats the baseline scores significantly.
Towards a Data-Driven Requirements Engineering Approach: Automatic Analysis of User Reviews
Venue: arXiv cs.CL 2022
Year: 2022

We are concerned by Data Driven Requirements Engineering, and in particular the consideration of user's reviews. These online reviews are a rich source of information for extracting new needs and improvement requests. In this work, we provide an automated analysis using CamemBERT, which is a state-of-the-art language model in French. We created a multi-label classification dataset of 6000 user reviews from three applications in the Health & Fitness field. The results are encouraging and suggest that it's possible to identify automatically the reviews concerning requests for new features. Dataset is available at: https://github.com/Jl-wei/APIA2022-French-user-reviews-classification-dataset.
Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision
Venue: arXiv cs.CL 2022
Year: 2022

Clinical trials are essential for drug development but are extremely expensive and time-consuming to conduct. It is beneficial to study similar historical trials when designing a clinical trial. However, lengthy trial documents and lack of labeled data make trial similarity search difficult. We propose a zero-shot clinical trial retrieval method, Trial2Vec, which learns through self-supervision without annotating similar clinical trials. Specifically, the meta-structure of trial documents (e.g., title, eligibility criteria, target disease) along with clinical knowledge (e.g., UMLS knowledge base https://www.nlm.nih.gov/research/umls/index.html) are leveraged to automatically generate contrastive samples. Besides, Trial2Vec encodes trial documents considering meta-structure thus producing compact embeddings aggregating multi-aspect information from the whole document. We show that our method yields medically interpretable embeddings by visualization and it gets a 15% average improvement over the best baselines on precision/recall for trial retrieval, which is evaluated on our labeled 1600 trial pairs. In addition, we prove the pre-trained embeddings benefit the downstream trial outcome prediction task over 240k trials. Software ias available at https://github.com/RyanWangZf/Trial2Vec.
Modeling Teams Performance Using Deep Representational Learning on Graphs
Venue: arXiv cs.SI 2022
Year: 2022

The large majority of human activities require collaborations within and across formal or informal teams. Our understanding of how the collaborative efforts spent by teams relate to their performance is still a matter of debate. Teamwork results in a highly interconnected ecosystem of potentially overlapping components where tasks are performed in interaction with team members and across other teams. To tackle this problem, we propose a graph neural network model designed to predict a team's performance while identifying the drivers that determine such an outcome. In particular, the model is based on three architectural channels: topological, centrality, and contextual which capture different factors potentially shaping teams' success. We endow the model with two attention mechanisms to boost model performance and allow interpretability. A first mechanism allows pinpointing key members inside the team. A second mechanism allows us to quantify the contributions of the three driver effects in determining the outcome performance. We test model performance on a wide range of domains outperforming most of the classical and neural baselines considered. Moreover, we include synthetic datasets specifically designed to validate how the model disentangles the intended properties on which our model vastly outperforms baselines.