OpenRaiser/Intern-Atlas huggingface dataset:
Relaxed Attention for Transformer Models
Venue: arXiv cs.LG 2022
Year: 2022

The powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the autoregressive transformer decoder complicating the integration of external language models. In this paper, we explore relaxed attention, a simple and easy-to-implement smoothing of the attention weights, yielding a two-fold improvement to the general transformer architecture: First, relaxed attention provides regularization when applied to the self-attention layers in the encoder. Second, we show that it naturally supports the integration of an external language model as it suppresses the implicitly learned internal language model by relaxing the cross attention in the decoder. We demonstrate the benefit of relaxed attention across several tasks with clear improvement in combination with recent benchmark approaches. Specifically, we exceed the former state-of-the-art performance of 26.90% word error rate on the largest public lip-reading LRS3 benchmark with a word error rate of 26.31%, as well as we achieve a top-performing BLEU score of 37.67 on the IWSLT14 (DE$\rightarrow$EN) machine translation task without external language models and virtually no additional model parameters. Code and models will be made publicly available.
On additive MDS codes with linear projections
Venue: arXiv cs.IT 2022
Year: 2022

We support some evidence that a long additive MDS code over a finite field must be equivalent to a linear code. More precisely, let $C$ be an $\mathbb F_q$-linear $(n,q^{hk},n-k+1)_{q^h}$ MDS code over $\mathbb F_{q^h}$. If $k=3$, $h \in \{2,3\}$, $n > \max \{q^{h-1},h q -1\} + 3$, and $C$ has three coordinates from which its projections are equivalent to linear codes, we prove that $C$ itself is equivalent to a linear code. If $k>3$, $n > q+k$, and there are two disjoint subsets of coordinates whose combined size is at most $k-2$ from which the projections of $C$ are equivalent to linear codes, we prove that $C$ is equivalent to a code which is linear over a larger field than $\mathbb F_q$.
Peer-group Behaviour Analytics of Windows Authentications Events Using Hierarchical Bayesian Modelling
Venue: arXiv cs.CR 2022
Year: 2022

Cyber-security analysts face an increasingly large number of alerts received on any given day. This is mainly due to the low precision of many existing methods to detect threats, producing a substantial number of false positives. Usually, several signature-based and statistical anomaly detectors are implemented within a computer network to detect threats. Recent efforts in User and Entity Behaviour Analytics modelling shed a light on how to reduce the burden on Security Operations Centre analysts through a better understanding of peer-group behaviour. Statistically, the challenge consists of accurately grouping users with similar behaviour, and then identifying those who deviate from their peers. This work proposes a new approach for peer-group behaviour modelling of Windows authentication events, using principles from hierarchical Bayesian models. This is a two-stage approach where in the first stage, peer-groups are formed based on a data-driven method, given the user's individual authentication pattern. In the second stage, the counts of users authenticating to different entities are aggregated by an hour and modelled by a Poisson distribution, taking into account seasonality components and hierarchical principles. Finally, we compare grouping users based on their human resources records against the data-driven methods and provide empirical evidence about alert reduction on a real-world authentication data set from a large enterprise network.
Twitter Topic Classification
Venue: arXiv cs.CL 2022
Year: 2022

Social media platforms host discussions about a wide variety of topics that arise everyday. Making sense of all the content and organising it into categories is an arduous task. A common way to deal with this issue is relying on topic modeling, but topics discovered using this technique are difficult to interpret and can differ from corpus to corpus. In this paper, we present a new task based on tweet topic classification and release two associated datasets. Given a wide range of topics covering the most important discussion points in social media, we provide training and testing data from recent time periods that can be used to evaluate tweet classification models. Moreover, we perform a quantitative evaluation and analysis of current general- and domain-specific language models on the task, which provide more insights on the challenges and nature of the task.
EM-Fault It Yourself: Building a Replicable EMFI Setup for Desktop and Server Hardware
Venue: arXiv cs.CR 2022
Year: 2022

EMFI has become a popular fault injection (FI) technique due to its ability to inject faults precisely considering timing and location. Recently, ARM, RISC-V, and even x86 processing units in different packages were shown to be vulnerable to electromagnetic fault injection (EMFI) attacks. However, past publications lack a detailed description of the entire attack setup, hindering researchers and companies from easily replicating the presented attacks on their devices. In this work, we first show how to build an automated EMFI setup with high scanning resolution and good repeatability that is large enough to attack modern desktop and server CPUs. We structurally lay out all details on mechanics, hardware, and software along with this paper. Second, we use our setup to attack a deeply embedded security co-processor in modern AMD systems on a chip (SoCs), the AMD Secure Processor (AMD-SP). Using a previously published code execution exploit, we run two custom payloads on the AMD-SP that utilize the SoC to different degrees. We then visualize these fault locations on SoC photographs allowing us to reason about the SoC's components under attack. Finally, we show that the signature verification process of one of the first executed firmware parts is susceptible to EMFI attacks, undermining the security architecture of the entire SoC. To the best of our knowledge, this is the first reported EMFI attack against an AMD desktop CPU.
Improving Replay-Based Continual Semantic Segmentation with Smart Data Selection
Venue: arXiv cs.CV 2022
Year: 2022

Continual learning for Semantic Segmentation (CSS) is a rapidly emerging field, in which the capabilities of the segmentation model are incrementally improved by learning new classes or new domains. A central challenge in Continual Learning is overcoming the effects of catastrophic forgetting, which refers to the sudden drop in accuracy on previously learned tasks after the model is trained on new classes or domains. In continual classification this challenge is often overcome by replaying a small selection of samples from previous tasks, however replay is rarely considered in CSS. Therefore, we investigate the influences of various replay strategies for semantic segmentation and evaluate them in class- and domain-incremental settings. Our findings suggest that in a class-incremental setting, it is critical to achieve a uniform distribution for the different classes in the buffer to avoid a bias towards newly learned classes. In the domain-incremental setting, it is most effective to select buffer samples by uniformly sampling from the distribution of learned feature representations or by choosing samples with median entropy. Finally, we observe that the effective sampling methods help to decrease the representation shift significantly in early layers, which is a major cause of forgetting in domain-incremental learning.
Frequency Dropout: Feature-Level Regularization via Randomized Filtering
Venue: arXiv cs.CV 2022
Year: 2022

Deep convolutional neural networks have shown remarkable performance on various computer vision tasks, and yet, they are susceptible to picking up spurious correlations from the training signal. So called `shortcuts' can occur during learning, for example, when there are specific frequencies present in the image data that correlate with the output predictions. Both high and low frequencies can be characteristic of the underlying noise distribution caused by the image acquisition rather than in relation to the task-relevant information about the image content. Models that learn features related to this characteristic noise will not generalize well to new data. In this work, we propose a simple yet effective training strategy, Frequency Dropout, to prevent convolutional neural networks from learning frequency-specific imaging features. We employ randomized filtering of feature maps during training which acts as a feature-level regularization. In this study, we consider common image processing filters such as Gaussian smoothing, Laplacian of Gaussian, and Gabor filtering. Our training strategy is model-agnostic and can be used for any computer vision task. We demonstrate the effectiveness of Frequency Dropout on a range of popular architectures and multiple tasks including image classification, domain adaptation, and semantic segmentation using both computer vision and medical imaging datasets. Our results suggest that the proposed approach does not only improve predictive accuracy but also improves robustness against domain shift.
Fine-grained Classification of Solder Joints with α-skew Jensen-Shannon Divergence
Venue: arXiv cs.CV 2022
Year: 2022

Solder joint inspection (SJI) is a critical process in the production of printed circuit boards (PCB). Detection of solder errors during SJI is quite challenging as the solder joints have very small sizes and can take various shapes. In this study, we first show that solders have low feature diversity, and that the SJI can be carried out as a fine-grained image classification task which focuses on hard-to-distinguish object classes. To improve the fine-grained classification accuracy, penalizing confident model predictions by maximizing entropy was found useful in the literature. Inline with this information, we propose using the α-skew Jensen-Shannon divergence (α-JS) for penalizing the confidence in model predictions. We compare the α-JS regularization with both existing entropyregularization based methods and the methods based on attention mechanism, segmentation techniques, transformer models, and specific loss functions for fine-grained image classification tasks. We show that the proposed approach achieves the highest F1-score and competitive accuracy for different models in the finegrained solder joint classification task. Finally, we visualize the activation maps and show that with entropy-regularization, more precise class-discriminative regions are localized, which are also more resilient to noise. Code will be made available here upon acceptance.
Learning the Propagation of Worms in Wireless Sensor Networks
Venue: arXiv cs.LG 2022
Year: 2022

Wireless sensor networks (WSNs) are composed of spatially distributed sensors and are considered vulnerable to attacks by worms and their variants. Due to the distinct strategies of worms propagation, the dynamic behavior varies depending on the different features of the sensors. Modeling the spread of worms can help us understand the worm attack behaviors and analyze the propagation procedure. In this paper, we design a communication model under various worms. We aim to learn our proposed model to analytically derive the dynamics of competitive worms propagation. We develop a new searching space combined with complex neural network models. Furthermore, the experiment results verified our analysis and demonstrated the performance of our proposed learning algorithms.
Audit and Improve Robustness of Private Neural Networks on Encrypted Data
Venue: arXiv cs.LG 2022
Year: 2022

Performing neural network inference on encrypted data without decryption is one popular method to enable privacy-preserving neural networks (PNet) as a service. Compared with regular neural networks deployed for machine-learning-as-a-service, PNet requires additional encoding, e.g., quantized-precision numbers, and polynomial activation. Encrypted input also introduces novel challenges such as adversarial robustness and security. To the best of our knowledge, we are the first to study questions including (i) Whether PNet is more robust against adversarial inputs than regular neural networks? (ii) How to design a robust PNet given the encrypted input without decryption? We propose PNet-Attack to generate black-box adversarial examples that can successfully attack PNet in both target and untarget manners. The attack results show that PNet robustness against adversarial inputs needs to be improved. This is not a trivial task because the PNet model owner does not have access to the plaintext of the input values, which prevents the application of existing detection and defense methods such as input tuning, model normalization, and adversarial training. To tackle this challenge, we propose a new fast and accurate noise insertion method, called RPNet, to design Robust and Private Neural Networks. Our comprehensive experiments show that PNet-Attack reduces at least $2.5\times$ queries than prior works. We theoretically analyze our RPNet methods and demonstrate that RPNet can decrease $\sim 91.88\%$ attack success rate.
Robust, High-Rate Trajectory Tracking on Insect-Scale Soft-Actuated Aerial Robots with Deep-Learned Tube MPC
Venue: arXiv cs.RO 2022
Year: 2022

Accurate and agile trajectory tracking in sub-gram Micro Aerial Vehicles (MAVs) is challenging, as the small scale of the robot induces large model uncertainties, demanding robust feedback controllers, while the fast dynamics and computational constraints prevent the deployment of computationally expensive strategies. In this work, we present an approach for agile and computationally efficient trajectory tracking on the MIT SoftFly, a sub-gram MAV (0.7 grams). Our strategy employs a cascaded control scheme, where an adaptive attitude controller is combined with a neural network policy trained to imitate a trajectory tracking robust tube model predictive controller (RTMPC). The neural network policy is obtained using our recent work, which enables the policy to preserve the robustness of RTMPC, but at a fraction of its computational cost. We experimentally evaluate our approach, achieving position Root Mean Square Errors lower than 1.8 cm even in the more challenging maneuvers, obtaining a 60% reduction in maximum position error compared to our previous work, and demonstrating robustness to large external disturbances
Dataset: Impact Events for Structural Health Monitoring of a Plastic Thin Plate
Venue: arXiv cs.LG 2022
Year: 2022

Nowadays, more and more datasets are published towards research and development of systems and models, enabling direct comparisons, continuous improvement of solutions, and researchers engagement with experimental, real life data. However, especially in the Structural Health Monitoring (SHM) domain, there are plenty of cases where new research projects have a unique combination of structure design and implementation, sensor selection and technological enablers that does not fit with the configuration of relevant individual studies in the literature. Thus, we share the data from our case study to the research community as we did not find any relevant repository available. More specifically, in this paper, we present a novel time-series dataset for impact detection and localization on a plastic thin-plate, towards Structural Health Monitoring applications, using ceramic piezoelectric transducers (PZTs) connected to an Internet of Things (IoT) device. The dataset was collected from an experimental procedure of low-velocity, low-energy impact events that includes at least 3 repetitions for each unique experiment, while the input measurements come from 4 PZT sensors placed at the corners of the plate. For each repetition and sensor, 5000 values are stored with 100 KHz sampling rate. The system is excited with a steel ball, and the height from which it is released varies from 10 cm to 20 cm. The dataset is available in GitHub (https://github.com/Smart-Objects/Impact-Events-Dataset).
Introducing emotions in the reasoning cycle ofnormative aware agents
Venue: arXiv cs.MA 2022
Year: 2022

Human relationships are complex processes that often involve following certain rules that regulate interactions and/or expected outcomes. These rules may be imposed by an authority or established by society. In multi-agent systems, normative systems have extensively addressed aspects such as norm synthesis, norm conflict detection, as well as norm emergence. However, if human behaviour is to be adequately simulated, not only normative aspects but also emotional aspects have to be taken into account. In this paper, we propose a Jason agent architecture that incorporates norms and emotions in its reasoning process to determine which plan (actions) to execute. The proposal is evaluated through a scenario based on a social network, which allows us to analyse the benefits of using emotional normative agents to achieve simulations closer to real human world.
Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems
Venue: arXiv cs.LG 2022
Year: 2022

Deep learning systems have been reported to achieve state-of-the-art performances in many applications, and a key is the existence of well trained classifiers on benchmark datasets. As a main-stream loss function, the cross entropy can easily lead us to find models which demonstrate severe overfitting behavior. In this paper, we show that the existing cross entropy loss minimization problem essentially learns the label conditional entropy (CE) of the underlying data distribution of the dataset. However, the CE learned in this way does not characterize well the information shared by the label and the input. In this paper, we propose a mutual information learning framework where we train deep neural network classifiers via learning the mutual information between the label and the input. Theoretically, we give the population classification error lower bound in terms of the mutual information. In addition, we derive the mutual information lower and upper bounds for a concrete binary classification data model in $\mathbb{R}^n$, and also the error probability lower bound in this scenario. Empirically, we conduct extensive experiments on several benchmark datasets to support our theory. The mutual information learned classifiers (MILCs) achieve far better generalization performances than the conditional entropy learned classifiers (CELCs) with an improvement which can exceed more than 10\% in testing accuracy.
Adaptive Local-Component-aware Graph Convolutional Network for One-shot Skeleton-based Action Recognition
Venue: arXiv cs.CV 2022
Year: 2022

Skeleton-based action recognition receives increasing attention because the skeleton representations reduce the amount of training data by eliminating visual information irrelevant to actions. To further improve the sample efficiency, meta-learning-based one-shot learning solutions were developed for skeleton-based action recognition. These methods find the nearest neighbor according to the similarity between instance-level global average embedding. However, such measurement holds unstable representativity due to inadequate generalized learning on local invariant and noisy features, while intuitively, more fine-grained recognition usually relies on determining key local body movements. To address this limitation, we present the Adaptive Local-Component-aware Graph Convolutional Network, which replaces the comparison metric with a focused sum of similarity measurements on aligned local embedding of action-critical spatial/temporal segments. Comprehensive one-shot experiments on the public benchmark of NTU-RGB+D 120 indicate that our method provides a stronger representation than the global embedding and helps our model reach state-of-the-art.
Recent Approaches for Perceptive Legged Locomotion
Venue: arXiv cs.RO 2022
Year: 2022

As both legged robots and embedded compute have become more capable, researchers have started to focus on field deployment of these robots. Robust autonomy in unstructured environments requires perception of the world around the robot in order to avoid hazards. However, incorporating perception online while maintaining agile motion is more challenging for legged robots than other mobile robots due to the complex planners and controllers required to handle the dynamics of locomotion. This report will compare three recent approaches for perceptive locomotion and discuss the different ways in which vision can be used to enable legged autonomy.
Learning Reconstructability for Drone Aerial Path Planning
Venue: arXiv cs.GR 2022
Year: 2022

We introduce the first learning-based reconstructability predictor to improve view and path planning for large-scale 3D urban scene acquisition using unmanned drones. In contrast to previous heuristic approaches, our method learns a model that explicitly predicts how well a 3D urban scene will be reconstructed from a set of viewpoints. To make such a model trainable and simultaneously applicable to drone path planning, we simulate the proxy-based 3D scene reconstruction during training to set up the prediction. Specifically, the neural network we design is trained to predict the scene reconstructability as a function of the proxy geometry, a set of viewpoints, and optionally a series of scene images acquired in flight. To reconstruct a new urban scene, we first build the 3D scene proxy, then rely on the predicted reconstruction quality and uncertainty measures by our network, based off of the proxy geometry, to guide the drone path planning. We demonstrate that our data-driven reconstructability predictions are more closely correlated to the true reconstruction quality than prior heuristic measures. Further, our learned predictor can be easily integrated into existing path planners to yield improvements. Finally, we devise a new iterative view planning framework, based on the learned reconstructability, and show superior performance of the new planner when reconstructing both synthetic and real scenes.
Reconstructing Robot Operations via Radio-Frequency Side-Channel
Venue: arXiv cs.CR 2022
Year: 2022

Connected teleoperated robotic systems play a key role in ensuring operational workflows are carried out with high levels of accuracy and low margins of error. In recent years, a variety of attacks have been proposed that actively target the robot itself from the cyber domain. However, little attention has been paid to the capabilities of a passive attacker. In this work, we investigate whether an insider adversary can accurately fingerprint robot movements and operational warehousing workflows via the radio frequency side channel in a stealthy manner. Using an SVM for classification, we found that an adversary can fingerprint individual robot movements with at least 96% accuracy, increasing to near perfect accuracy when reconstructing entire warehousing workflows.