OpenRaiser/Intern-Atlas huggingface dataset:
Coordinate descent on the orthogonal group for recurrent neural network training
Venue: arXiv cs.LG 2021
Year: 2021

We propose to use stochastic Riemannian coordinate descent on the orthogonal group for recurrent neural network training. The algorithm rotates successively two columns of the recurrent matrix, an operation that can be efficiently implemented as a multiplication by a Givens matrix. In the case when the coordinate is selected uniformly at random at each iteration, we prove the convergence of the proposed algorithm under standard assumptions on the loss function, stepsize and minibatch noise. In addition, we numerically demonstrate that the Riemannian gradient in recurrent neural network training has an approximately sparse structure. Leveraging this observation, we propose a faster variant of the proposed algorithm that relies on the Gauss-Southwell rule. Experiments on a benchmark recurrent neural network training problem are presented to demonstrate the effectiveness of the proposed algorithm.
Technical Report for HW2VEC -- A Graph Learning Tool for Automating Hardware Security
Venue: arXiv cs.CR 2021
Year: 2021

In this technical report, we present HW2VEC [11], an open-source graph learning tool for hardware security, and its implementation details (Figure 1). HW2VEC provides toolboxes for graph representation extraction in the form of Data Flow Graphs (DFGs) or Abstract Syntax Trees (ASTs) from hardware designs at RTL and GLN levels. Besides, HW2VEC also offers graph learning tools for representing hardware designs in vectors that preserve both structural features and behavioral features. To the best of our knowledge, HW2VEC is the first open-source research tool that supports applying graph learning methods to hardware designs in different abstraction levels for hardware security. We organize the remainder of this technical report as follows: Section 2 introduces the architecture of HW2VEC; Section 3 gives information about the use-case implementations; Section 4 provides the experimental results and demonstrates the performance of HW2VEC for two hardware security applications: HT detection and IP piracy detection; finally, Section 5 will conclude this report.
Zooming Into the Darknet: Characterizing Internet Background Radiation and its Structural Changes
Venue: arXiv cs.CR 2021
Year: 2021

Network telescopes or "Darknets" provide a unique window into Internet-wide malicious activities associated with malware propagation, denial of service attacks, scanning performed for network reconnaissance, and others. Analyses of the resulting data can provide actionable insights to security analysts that can be used to prevent or mitigate cyber-threats. Large Darknets, however, observe millions of nefarious events on a daily basis which makes the transformation of the captured information into meaningful insights challenging. We present a novel framework for characterizing Darknet behavior and its temporal evolution aiming to address this challenge. The proposed framework: (i) Extracts a high dimensional representation of Darknet events composed of features distilled from Darknet data and other external sources; (ii) Learns, in an unsupervised fashion, an information-preserving low-dimensional representation of these events (using deep representation learning) that is amenable to clustering; (iv) Performs clustering of the scanner data in the resulting representation space and provides interpretable insights using optimal decision trees; and (v) Utilizes the clustering outcomes as "signatures" that can be used to detect structural changes in the Darknet activities. We evaluate the proposed system on a large operational Network Telescope and demonstrate its ability to detect real-world, high-impact cybersecurity incidents.
The History of Speech Recognition to the Year 2030
Venue: arXiv cs.CL 2021
Year: 2021

The decade from 2010 to 2020 saw remarkable improvements in automatic speech recognition. Many people now use speech recognition on a daily basis, for example to perform voice search queries, send text messages, and interact with voice assistants like Amazon Alexa and Siri by Apple. Before 2010 most people rarely used speech recognition. Given the remarkable changes in the state of speech recognition over the previous decade, what can we expect over the coming decade? I attempt to forecast the state of speech recognition research and applications by the year 2030. While the changes to general speech recognition accuracy will not be as dramatic as in the previous decade, I suggest we have an exciting decade of progress in speech technology ahead of us.
A Deep Learning Approach to Predict Blood Pressure from PPG Signals
Venue: arXiv cs.LG 2021
Year: 2021

Blood Pressure (BP) is one of the four primary vital signs indicating the status of the body's vital (life-sustaining) functions. BP is difficult to continuously monitor using a sphygmomanometer (i.e. a blood pressure cuff), especially in everyday-setting. However, other health signals which can be easily and continuously acquired, such as photoplethysmography (PPG), show some similarities with the Aortic Pressure waveform. Based on these similarities, in recent years several methods were proposed to predict BP from the PPG signal. Building on these results, we propose an advanced personalized data-driven approach that uses a three-layer deep neural network to estimate BP based on PPG signals. Different from previous work, the proposed model analyzes the PPG signal in time-domain and automatically extracts the most critical features for this specific application, then uses a variation of recurrent neural networks called Long-Short-Term-Memory (LSTM) to map the extracted features to the BP value associated with that time window. Experimental results on two separate standard hospital datasets, yielded absolute errors mean and absolute error standard deviation for systolic and diastolic BP values outperforming prior works.
Manifold-Inspired Single Image Interpolation
Venue: arXiv cs.CV 2021
Year: 2021

Manifold models consider natural-image patches to be on a low-dimensional manifold embedded in a high dimensional state space and each patch and its similar patches to approximately lie on a linear affine subspace. Manifold models are closely related to semi-local similarity, a well-known property of natural images, referring to that for most natural-image patches, several similar patches can be found in its spatial neighborhood. Many approaches to single image interpolation use manifold models to exploit semi-local similarity by two mutually exclusive parts: i) searching each target patch's similar patches and ii) operating on the searched similar patches, the target patch and the measured input pixels to estimate the target patch. Unfortunately, aliasing in the input image makes it challenging for both parts. A very few works explicitly deal with those challenges and only ad-hoc solutions are proposed. To overcome the challenge in the first part, we propose a carefully-designed adaptive technique to remove aliasing in severely aliased regions, which cannot be removed from traditional techniques. This technique enables reliable identification of similar patches even in the presence of strong aliasing. To overcome the challenge in the second part, we propose to use the aliasing-removed image to guide the initialization of the interpolated image and develop a progressive scheme to refine the interpolated image based on manifold models. Experimental results demonstrate that our approach reconstructs edges with both smoothness along contours and sharpness across profiles, and achieves an average Peak Signal-to-Noise Ratio (PSNR) significantly higher than existing model-based approaches.
T$_k$ML-AP: Adversarial Attacks to Top-$k$ Multi-Label Learning
Venue: arXiv cs.CV 2021
Year: 2021

Top-$k$ multi-label learning, which returns the top-$k$ predicted labels from an input, has many practical applications such as image annotation, document analysis, and web search engine. However, the vulnerabilities of such algorithms with regards to dedicated adversarial perturbation attacks have not been extensively studied previously. In this work, we develop methods to create adversarial perturbations that can be used to attack top-$k$ multi-label learning-based image annotation systems (TkML-AP). Our methods explicitly consider the top-$k$ ranking relation and are based on novel loss functions. Experimental evaluations on large-scale benchmark datasets including PASCAL VOC and MS COCO demonstrate the effectiveness of our methods in reducing the performance of state-of-the-art top-$k$ multi-label learning methods, under both untargeted and targeted attacks.
A Plant Root System Algorithm Based on Swarm Intelligence for One-dimensional Biomedical Signal Feature Engineering
Venue: arXiv cs.LG 2021
Year: 2021

To date, very few biomedical signals have transitioned from research applications to clinical applications. This is largely due to the lack of trust in the diagnostic ability of non-stationary signals. To reach the level of clinical diagnostic application, classification using high-quality signal features is necessary. While there has been considerable progress in machine learning in recent years, especially deep learning, progress has been quite limited in the field of feature engineering. This study proposes a feature extraction algorithm based on group intelligence which we call a Plant Root System (PRS) algorithm. Importantly, the correlation between features produced by this PRS algorithm and traditional features is low, and the accuracy of several widely-used classifiers was found to be substantially improved with the addition of PRS features. It is expected that more biomedical signals can be applied to clinical diagnosis using the proposed algorithm.
Self Context and Shape Prior for Sensorless Freehand 3D Ultrasound Reconstruction
Venue: arXiv cs.CV 2021
Year: 2021

3D ultrasound (US) is widely used for its rich diagnostic information. However, it is criticized for its limited field of view. 3D freehand US reconstruction is promising in addressing the problem by providing broad range and freeform scan. The existing deep learning based methods only focus on the basic cases of skill sequences, and the model relies on the training data heavily. The sequences in real clinical practice are a mix of diverse skills and have complex scanning paths. Besides, deep models should adapt themselves to the testing cases with prior knowledge for better robustness, rather than only fit to the training cases. In this paper, we propose a novel approach to sensorless freehand 3D US reconstruction considering the complex skill sequences. Our contribution is three-fold. First, we advance a novel online learning framework by designing a differentiable reconstruction algorithm. It realizes an end-to-end optimization from section sequences to the reconstructed volume. Second, a self-supervised learning method is developed to explore the context information that reconstructed by the testing data itself, promoting the perception of the model. Third, inspired by the effectiveness of shape prior, we also introduce adversarial training to strengthen the learning of anatomical shape prior in the reconstructed volume. By mining the context and structural cues of the testing data, our online learning methods can drive the model to handle complex skill sequences. Experimental results on developmental dysplasia of the hip US and fetal US datasets show that, our proposed method can outperform the start-of-the-art methods regarding the shift errors and path similarities.
Edge Intelligence in Softwarized 6G: Deep Learning-enabled Network Traffic Predictions
Venue: arXiv cs.IT 2021
Year: 2021

The 6G vision is envisaged to enable agile network expansion and rapid deployment of new on-demand microservices (e.g., visibility services for data traffic management, mobile edge computing services) closer to the network's edge IoT devices. However, providing one of the critical features of network visibility services, i.e., data flow prediction in the network, is challenging at the edge devices within a dynamic cloud-native environment as the traffic flow characteristics are random and sporadic. To provide the AI-native services for the 6G vision, we propose a novel edge-native framework to provide an intelligent prognosis technique for data traffic management in this paper. The prognosis model uses long short-term memory (LSTM)-based encoder-decoder deep learning, which we train on real time-series multivariate data records collected from the edge $μ$-boxes of a selected testbed network. Our result accurately predicts the statistical characteristics of data traffic and verifies the trained model against the ground truth observations. Moreover, we validate our novel framework with two performance metrics for each feature of the multivariate data.
A-ULMPM: An Arbitrary Updated Lagrangian Material Point Method for Efficient Simulation of Solids and Fluids
Venue: arXiv cs.GR 2021
Year: 2021

We present an arbitrary updated Lagrangian Material Point Method (A-ULMPM) to alleviate issues, such as the cell-crossing instability and numerical fracture, that plague state of the art Eulerian formulations of MPM, while still allowing for large deformations that arise in fluid simulations. Our proposed framework spans MPM discretizations from total Lagrangian formulations to Eulerian formulations. We design an easy-to-implement physics-based criterion that allows A-ULMPM to update the reference configuration adaptively for measuring physical states including stress, strain, interpolation kernels and their derivatives. For better efficiency and conservation of angular momentum, we further integrate the APIC[Jiang et al. 2015] and MLS-MPM[Hu et al. 2018] formulations in A-ULMPM by augmenting the accuracy of velocity rasterization using both the local velocity and its first-order derivatives. Our theoretical derivations use a nodal discretized Lagrangian, instead of the weak form discretization in MLS-MPM[Hu et al. 2018], and naturally lead to a "modified" MLS-MPM in A-ULMPM, which can recover MLS-MPM using a completely Eulerian formulation. A-ULMPM does not require significant changes to traditional Eulerian formulations of MPM, and is computationally more efficient since it only updates interpolation kernels and their derivatives when large topology changes occur. We present end-to-end 3D simulations of stretching and twisting hyperelastic solids, splashing liquids, and multi-material interactions with large deformations to demonstrate the efficacy of our novel A-ULMPM framework.
Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Text
Venue: arXiv cs.CL 2021
Year: 2021

Chinese sentiment analysis (CSA) has always been one of the challenges in natural language processing due to its complexity and uncertainty. Transformer has succeeded in capturing semantic features, but it uses position encoding to capture sequence features, which has great shortcomings compared with the recurrent model. In this paper, we propose T-E-GRU for Chinese sentiment analysis, which combine transformer encoder and GRU. We conducted experiments on three Chinese comment datasets. In view of the confusion of punctuation marks in Chinese comment texts, we selectively retain some punctuation marks with sentence segmentation ability. The experimental results show that T-E-GRU outperforms classic recurrent model and recurrent model with attention.
A Reinforcement Learning Approach for Scheduling in mmWave Networks
Venue: arXiv cs.IT 2021
Year: 2021

We consider a source that wishes to communicate with a destination at a desired rate, over a mmWave network where links are subject to blockage and nodes to failure (e.g., in a hostile military environment). To achieve resilience to link and node failures, we here explore a state-of-the-art Soft Actor-Critic (SAC) deep reinforcement learning algorithm, that adapts the information flow through the network, without using knowledge of the link capacities or network topology. Numerical evaluations show that our algorithm can achieve the desired rate even in dynamic environments and it is robust against blockage.
Janus: A Systems Engineering Approach to the Design of Industrial Cyber-Physical Systems
Venue: arXiv cs.SE 2021
Year: 2021

The benefits that arise from the adoption of a systems engineering approach to the design of engineered systems are well understood and documented. However , with software systems, different approaches are required given the changeability of requirements and the malleability of software. With the design of industrial cyber-physical systems, one is confronted with the challenge of designing engineered systems that have a significant software component. Furthermore, that software component must be able to seamlessly interact with both the enterprise's business systems and industrial systems. In this paper, we present Janus, which together with the GORITE BDI agent framework, provides a methodology for the design of agent-based industrial cyber-physical systems. Central to the Janus approach is the development of a logical architecture as in traditional systems engineering and then the allocation of the logical requirements to a BDI (Belief Desire Intention) agent architecture which is derived from the physical architecture for the system. Janus has its origins in product manufacturing; in this paper, we apply it to the problem of Fault Location, Isolation and Service Restoration (FLISR) for power substations.
Bucketed PCA Neural Networks with Neurons Mirroring Signals
Venue: arXiv cs.LG 2021
Year: 2021

The bucketed PCA neural network (PCA-NN) with transforms is developed here in an effort to benchmark deep neural networks (DNN's), for problems on supervised classification. Most classical PCA models apply PCA to the entire training data set to establish a reductive representation and then employ non-network tools such as high-order polynomial classifiers. In contrast, the bucketed PCA-NN applies PCA to individual buckets which are constructed in two consecutive phases, as well as retains a genuine architecture of a neural network. This facilitates a fair apple-to-apple comparison to DNN's, esp. to reveal that a major chunk of accuracy achieved by many impressive DNN's could possibly be explained by the bucketed PCA-NN (e.g., 96% out of 98% for the MNIST data set as an example). Compared with most DNN's, the three building blocks of the bucketed PCA-NN are easier to comprehend conceptually - PCA, transforms, and bucketing for error correction. Furthermore, unlike the somewhat quasi-random neurons ubiquitously observed in DNN's, the PCA neurons resemble or mirror the input signals and are more straightforward to decipher as a result.
Simple Framework of ZF Precoding Analysis for Full-Dimensional Massive MIMO Systems
Venue: arXiv cs.IT 2021
Year: 2021

We provide a simple, yet general and accurate framework for analyzing zero-forcing (ZF) precoding performance for a full-dimensional massive multiple-input multiple-output system. Exploiting an order two Neumann series, our framework approximates the expected (average) ZF signal-to-noise ratio and ergodic sum spectral efficiency, while catering for a finite multipath propagation model, as well as correlated and uncorrelated user equipment (UE) positions in both azimuth and elevation domains. The analysis provides clear insights on the influence of propagation and system parameters on ZF performance. We identify how far UEs must lie in azimuth and elevation for the ZF precoder to approach uncorrelated channels. Our framework is useful for predicting the ZF performance degradation arising from channel correlation. For optimal performance, UEs could be separated by around 30$^\circ$ in azimuth and 15$^\circ$ in elevation - conditions which are difficult to meet in reality.
From LSAT: The Progress and Challenges of Complex Reasoning
Venue: arXiv cs.CL 2021
Year: 2021

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.
Information Stealing in Federated Learning Systems Based on Generative Adversarial Networks
Venue: arXiv cs.LG 2021
Year: 2021

An attack on deep learning systems where intelligent machines collaborate to solve problems could cause a node in the network to make a mistake on a critical judgment. At the same time, the security and privacy concerns of AI have galvanized the attention of experts from multiple disciplines. In this research, we successfully mounted adversarial attacks on a federated learning (FL) environment using three different datasets. The attacks leveraged generative adversarial networks (GANs) to affect the learning process and strive to reconstruct the private data of users by learning hidden features from shared local model parameters. The attack was target-oriented drawing data with distinct class distribution from the CIFAR- 10, MNIST, and Fashion-MNIST respectively. Moreover, by measuring the Euclidean distance between the real data and the reconstructed adversarial samples, we evaluated the performance of the adversary in the learning processes in various scenarios. At last, we successfully reconstructed the real data of the victim from the shared global model parameters with all the applied datasets.
ConveRT for FAQ Answering
Venue: arXiv cs.CL 2021
Year: 2021

Knowledgeable FAQ chatbots are a valuable resource to any organization. While powerful and efficient retrieval-based models exist for English, it is rarely the case for other languages for which the same amount of training data is not available. In this paper, we propose a novel pre-training procedure to adapt ConveRT, an English conversational retriever model, to other languages with less training data available. We apply it for the first time to the task of Dutch FAQ answering related to the COVID-19 vaccine. We show it performs better than an open-source alternative in both a low-data regime and a high-data regime.