OpenRaiser/Intern-Atlas huggingface dataset:
FeSHI: Feature Map Based Stealthy Hardware Intrinsic Attack
Venue: arXiv cs.CR 2021
Year: 2021

To reduce the time-to-market and access to state-of-the-art techniques, CNN hardware mapping and deployment on embedded accelerators are often outsourced to untrusted third parties, which is going to be more prevalent in futuristic artificial intelligence of things (AIoT) systems. These AIoT systems anticipate horizontal collaboration among different resource-constrained AIoT node devices, where CNN layers are partitioned and these devices collaboratively compute complex CNN tasks. This horizontal collaboration opens another attack surface to the CNN-based application, like inserting the hardware Trojans (HT) into the embedded accelerators designed for the CNN. Therefore, there is a dire need to explore this attack surface for designing secure embedded hardware accelerators for CNNs. Towards this goal, in this paper, we exploited this attack surface to propose an HT-based attack called FeSHI. Since in horizontal collaboration of RC AIoT devices different sections of CNN architectures are outsourced to different untrusted third parties, the attacker may not know the input image, but it has access to the layer-by-layer output feature maps information for the assigned sections of the CNN architecture. This attack exploits the statistical distribution, i.e., Gaussian distribution, of the layer-by-layer feature maps of the CNN to design two triggers for stealthy HT with a very low probability of triggering. Also, three different novel, stealthy and effective trigger designs are proposed.
Memory-efficient Transformers via Top-$k$ Attention
Venue: arXiv cs.CL 2021
Year: 2021

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length. While these variants are memory and compute efficient, it is not possible to directly use them with popular pre-trained language models trained using vanilla attention, without an expensive corrective pre-training stage. In this work, we propose a simple yet highly accurate approximation for vanilla attention. We process the queries in chunks, and for each query, compute the top-$k$ scores with respect to the keys. Our approach offers several advantages: (a) its memory usage is linear in the input size, similar to linear attention variants, such as Performer and RFA (b) it is a drop-in replacement for vanilla attention that does not require any corrective pre-training, and (c) it can also lead to significant memory savings in the feed-forward layers after casting them into the familiar query-key-value framework. We evaluate the quality of top-$k$ approximation for multi-head attention layers on the Long Range Arena Benchmark, and for feed-forward layers of T5 and UnifiedQA on multiple QA datasets. We show our approach leads to accuracy that is nearly-identical to vanilla attention in multiple setups including training from scratch, fine-tuning, and zero-shot inference.
Multi-User Communication with Extremely Large-Scale MIMO
Venue: arXiv cs.IT 2021
Year: 2021

Extremely large-scale multiple-input multiple-output (XL-MIMO) communication aims to further boost the antenna size significantly than current massive MIMO systems, for which conventional far-field assumption with uniform plane wave (UPW) model may become invalid. This paper studies the modelling and performance analysis for multi-user XL-MIMO communication. With the spherical wavefront phase modelling, and also by taking into account the variations of signal amplitude and projected aperture across array elements, the performance of the three typical beamforming schemes are analyzed, namely the maximal-ratio combining (MRC), zero-forcing (ZF), and minimum mean-square error (MMSE) beamforming. For the special case of two-users, we analytically show that the signal-to-interference-plus-noise ratio (SINR) of all the three beamforming schemes increases as the channels' correlation coefficient decreases. Furthermore, compared to existing UPW model where inter-user interference (IUI) can only be suppressed in angular domain, XL-MIMO enables a new degree-of-freedom (DoF) for IUI suppression by distance separation, even for users along the same direction. Simulation results are provided to validate the modelling and performance analysis of multi-user XL-MIMO communications.
InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping
Venue: arXiv cs.CL 2021
Year: 2021

E-commerce companies have to face abnormal sellers who sell potentially-risky products. Typically, the risk can be identified by jointly considering product content (e.g., title and image) and seller behavior. This work focuses on behavior feature extraction as behavior sequences can provide valuable clues for the risk discovery by reflecting the sellers' operation habits. Traditional feature extraction techniques heavily depend on domain experts and adapt poorly to new tasks. In this paper, we propose a self-supervised method InfoBehavior to automatically extract meaningful representations from ultra-long raw behavior sequences instead of the costly feature selection procedure. InfoBehavior utilizes Bidirectional Transformer as feature encoder due to its excellent capability in modeling long-term dependency. However, it is intractable for commodity GPUs because the time and memory required by Transformer grow quadratically with the increase of sequence length. Thus, we propose a hierarchical grouping strategy to aggregate ultra-long raw behavior sequences to length-processable high-level embedding sequences. Moreover, we introduce two types of pretext tasks. Sequence-related pretext task defines a contrastive-based training objective to correctly select the masked-out coarse-grained/fine-grained behavior sequences against other "distractor" behavior sequences; Domain-related pretext task designs a classification training objective to correctly predict the domain-specific statistical results of anomalous behavior. We show that behavior representations from the pre-trained InfoBehavior can be directly used or integrated with features from other side information to support a wide range of downstream tasks. Experimental results demonstrate that InfoBehavior significantly improves the performance of Product Risk Management and Intellectual Property Protection.
Domain Generalization on Medical Imaging Classification using Episodic Training with Task Augmentation
Venue: arXiv cs.CV 2021
Year: 2021

Medical imaging datasets usually exhibit domain shift due to the variations of scanner vendors, imaging protocols, etc. This raises the concern about the generalization capacity of machine learning models. Domain generalization (DG), which aims to learn a model from multiple source domains such that it can be directly generalized to unseen test domains, seems particularly promising to medical imaging community. To address DG, recent model-agnostic meta-learning (MAML) has been introduced, which transfers the knowledge from previous training tasks to facilitate the learning of novel testing tasks. However, in clinical practice, there are usually only a few annotated source domains available, which decreases the capacity of training task generation and thus increases the risk of overfitting to training tasks in the paradigm. In this paper, we propose a novel DG scheme of episodic training with task augmentation on medical imaging classification. Based on meta-learning, we develop the paradigm of episodic training to construct the knowledge transfer from episodic training-task simulation to the real testing task of DG. Motivated by the limited number of source domains in real-world medical deployment, we consider the unique task-level overfitting and we propose task augmentation to enhance the variety during training task generation to alleviate it. With the established learning framework, we further exploit a novel meta-objective to regularize the deep embedding of training domains. To validate the effectiveness of the proposed method, we perform experiments on histopathological images and abdominal CT images.
Sentiment Analysis of Covid-19 Tweets using Evolutionary Classification-Based LSTM Model
Venue: arXiv cs.CL 2021
Year: 2021

As the Covid-19 outbreaks rapidly all over the world day by day and also affects the lives of million, a number of countries declared complete lock-down to check its intensity. During this lockdown period, social media plat-forms have played an important role to spread information about this pandemic across the world, as people used to express their feelings through the social networks. Considering this catastrophic situation, we developed an experimental approach to analyze the reactions of people on Twitter taking into ac-count the popular words either directly or indirectly based on this pandemic. This paper represents the sentiment analysis on collected large number of tweets on Coronavirus or Covid-19. At first, we analyze the trend of public sentiment on the topics related to Covid-19 epidemic using an evolutionary classification followed by the n-gram analysis. Then we calculated the sentiment ratings on collected tweet based on their class. Finally, we trained the long-short term network using two types of rated tweets to predict sentiment on Covid-19 data and obtained an overall accuracy of 84.46%.
Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition
Venue: arXiv cs.CL 2021
Year: 2021

How to effectively incorporate cross-utterance information cues into a neural language model (LM) has emerged as one of the intriguing issues for automatic speech recognition (ASR). Existing research efforts on improving contextualization of an LM typically regard previous utterances as a sequence of additional input and may fail to capture complex global structural dependencies among these utterances. In view of this, we in this paper seek to represent the historical context information of an utterance as graph-structured data so as to distill cross-utterances, global word interaction relationships. To this end, we apply a graph convolutional network (GCN) on the resulting graph to obtain the corresponding GCN embeddings of historical words. GCN has recently found its versatile applications on social-network analysis, text summarization, and among others due mainly to its ability of effectively capturing rich relational information among elements. However, GCN remains largely underexplored in the context of ASR, especially for dealing with conversational speech. In addition, we frame ASR N-best reranking as a prediction problem, leveraging bidirectional encoder representations from transformers (BERT) as the vehicle to not only seize the local intrinsic word regularity patterns inherent in a candidate hypothesis but also incorporate the cross-utterance, historical word interaction cues distilled by GCN for promoting performance. Extensive experiments conducted on the AMI benchmark dataset seem to confirm the pragmatic utility of our methods, in relation to some current top-of-the-line methods.
Federated Learning Over Wireless Channels: Dynamic Resource Allocation and Task Scheduling
Venue: arXiv cs.DC 2021
Year: 2021

With the development of federated learning (FL), mobile devices (MDs) are able to train their local models with private data and sends them to a central server for aggregation, thereby preventing sensitive raw data leakage. In this paper, we aim to improve the training performance of FL systems in the context of wireless channels and stochastic energy arrivals of MDs. To this purpose, we dynamically optimize MDs' transmission power and training task scheduling. We first model this dynamic programming problem as a constrained Markov decision process (CMDP). Due to high dimensions rooted from our CMDP problem, we propose online stochastic learning methods to simplify the CMDP and design online algorithms to obtain an efficient policy for all MDs. Since there are long-term constraints in our CMDP, we utilize Lagrange multipliers approach to tackle this issue. Furthermore, we prove the convergence of the proposed online stochastic learning algorithm. Numerical results indicate that the proposed algorithms can achieve better performance than the benchmark algorithms.
How Crucial Is It for 6G Networks to Be Autonomous?
Venue: arXiv cs.NI 2021
Year: 2021

The sixth generation (6G), unlike any of the previous generations, is envisioned by 2030 to connect everything. Moreover, in addition to the new use cases, 6G is expected to support, it will need to provide a superior performance over 5G. The global connectivity, large network dimensions, users heterogeneity, extremely low-power consumption, high throughput, ultrahigh reliability, efficient network operation and maintenance, and low-latency requirements to be met by future networks inevitably necessitate the autonomy of 6G. Intelligence, facilitated mainly by the advancement of artificial intelligence (AI) techniques, is a key to achieve autonomy. In this paper, we provide a bird's-eye view of 6G, its vision, progress, and objectives. Furthermore, we present some technologies that would be mainly enabling intelligent globally connected world. In addition to discussing the role of AI for future wireless communications, we, unlike any other review papers, provide our original results which give early evidence for the viability of achieving 6G networks autonomy through leveraging AI advances. Furthermore, we, very importantly, identify 6G implementation challenges and key innovative techniques that promise to solve them. This article serves as a starting point for learners to acquire more knowledge about 6G and also for researchers to promote more development to the field.
AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data
Venue: arXiv cs.LG 2021
Year: 2021

Scoring systems are highly interpretable and widely used to evaluate time-to-event outcomes in healthcare research. However, existing time-to-event scores are predominantly created ad-hoc using a few manually selected variables based on clinician's knowledge, suggesting an unmet need for a robust and efficient generic score-generating method. AutoScore was previously developed as an interpretable machine learning score generator, integrated both machine learning and point-based scores in the strong discriminability and accessibility. We have further extended it to time-to-event data and developed AutoScore-Survival, for automatically generating time-to-event scores with right-censored survival data. Random survival forest provides an efficient solution for selecting variables, and Cox regression was used for score weighting. We illustrated our method in a real-life study of 90-day mortality of patients in intensive care units and compared its performance with survival models (i.e., Cox) and the random survival forest. The AutoScore-Survival-derived scoring model was more parsimonious than survival models built using traditional variable selection methods (e.g., penalized likelihood approach and stepwise variable selection), and its performance was comparable to survival models using the same set of variables. Although AutoScore-Survival achieved a comparable integrated area under the curve of 0.782 (95% CI: 0.767-0.794), the integer-valued time-to-event scores generated are favorable in clinical applications because they are easier to compute and interpret. Our proposed AutoScore-Survival provides an automated, robust and easy-to-use machine learning-based clinical score generator to studies of time-to-event outcomes. It provides a systematic guideline to facilitate the future development of time-to-event scores for clinical applications.
BoolNet: Minimizing The Energy Consumption of Binary Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Recent works on Binary Neural Networks (BNNs) have made promising progress in narrowing the accuracy gap of BNNs to their 32-bit counterparts. However, the accuracy gains are often based on specialized model designs using additional 32-bit components. Furthermore, almost all previous BNNs use 32-bit for feature maps and the shortcuts enclosing the corresponding binary convolution blocks, which helps to effectively maintain the accuracy, but is not friendly to hardware accelerators with limited memory, energy, and computing resources. Thus, we raise the following question: How can accuracy and energy consumption be balanced in a BNN network design? We extensively study this fundamental problem in this work and propose a novel BNN architecture without most commonly used 32-bit components: \textit{BoolNet}. Experimental results on ImageNet demonstrate that BoolNet can achieve 4.6x energy reduction coupled with 1.2\% higher accuracy than the commonly used BNN architecture Bi-RealNet. Code and trained models are available at: https://github.com/hpi-xnor/BoolNet.
Experimental Analysis of Trajectory Control Using Computer Vision and Artificial Intelligence for Autonomous Vehicles
Venue: arXiv cs.RO 2021
Year: 2021

Perception of the lane boundaries is crucial for the tasks related to autonomous trajectory control. In this paper, several methodologies for lane detection are discussed with an experimental illustration: Hough transformation, Blob analysis, and Bird's eye view. Following the abstraction of lane marks from the boundary, the next approach is applying a control law based on the perception to control steering and speed control. In the following, a comparative analysis is made between an open-loop response, PID control, and a neural network control law through graphical statistics. To get the perception of the surrounding a wireless streaming camera connected to Raspberry Pi is used. After pre-processing the signal received by the camera the output is sent back to the Raspberry Pi that processes the input and communicates the control to the motors through Arduino via serial communication.
Underwater Soft Robotic Hand with Multi-Source Coupling Bio-Inspired Soft Palm and Six Fingers Driven by Water Hydraulic
Venue: arXiv cs.RO 2021
Year: 2021

A new fluid-driven soft robot hand in this study uses the idea of the bionics and has the anthropomorphic form, which is oriented to the flexible grasp function. The soft robot hand is composed of a new kind of multi-freedom soft finger and soft palm, which realizes the characteristic grasping function of forehand and backhand. Combined with the fine fluid control system, the soft hand can realize flexible grasping under high pressure, so as to realize flexible grasping operation for different types of target objects in the underwater environment. The soft robot hand was controlled based on water hydraulic platform, Finally, the soft robot hand and the fine fluid control system were connected to form the underwater soft robot hand experiment platform.
Finite-Length Bounds on Hypothesis Testing Subject to Vanishing Type I Error Restrictions
Venue: arXiv cs.IT 2021
Year: 2021

A central problem in Binary Hypothesis Testing (BHT) is to determine the optimal tradeoff between the Type I error (referred to as false alarm) and Type II (referred to as miss) error. In this context, the exponential rate of convergence of the optimal miss error probability -- as the sample size tends to infinity -- given some (positive) restrictions on the false alarm probabilities is a fundamental question to address in theory. Considering the more realistic context of a BHT with a finite number of observations, this paper presents a new non-asymptotic result for the scenario with monotonic (sub-exponential decreasing) restriction on the Type I error probability, which extends the result presented by Strassen in 2009. Building on the use of concentration inequalities, we offer new upper and lower bounds to the optimal Type II error probability for the case of finite observations. Finally, the derived bounds are evaluated and interpreted numerically (as a function of the number samples) for some vanishing Type I error restrictions.
An Extended Multi-Model Regression Approach for Compressive Strength Prediction and Optimization of a Concrete Mixture
Venue: arXiv cs.LG 2021
Year: 2021

Due to the significant delay and cost associated with experimental tests, a model based evaluation of concrete compressive strength is of high value, both for the purpose of strength prediction as well as the mixture optimization. In this regard, several recent studies have employed state-of-the-art regression models in order to achieve a good prediction model, employing available experimental data sets. Nevertheless, while each of the employed models can better adapt to a specific nature of the input data, the accuracy of each individual model is limited due to the sensitivity to the choice of hyperparameters and the learning strategy. In the present work, we take a further step towards improving the accuracy of the prediction model via the weighted combination of multiple regression methods. Moreover, a (GA)-based multi-objective mixture optimization is proposed, building on the obtained multi-regression model. In particular, we present a data aided framework where the regression methods based on artificial neural network, random forest regression, and polynomial regression are jointly implemented to predict the compressive strength of concrete. The outcome of the individual regression models are then combined via a linear weighting strategy and optimized over the training data set as a quadratic convex optimization problem. It is worth mentioning that due to the convexity of the formulated problem, the globally optimum weighting strategy is obtained via standard numerical solvers. Employing the proposed GA-based optimization, a Pareto front of the cost-CS trade-of has been obtained employing the available data set. Moreover, the resulting accuracy of the proposed multi-model prediction method is shown to outperform the available single-model regression methods in the literature by a valuable margin, via numerical simulations.
VeriFly: On-the-fly Assertion Checking via Incrementality
Venue: arXiv cs.PL 2021
Year: 2021

Assertion checking is an invaluable programmer's tool for finding many classes of errors or verifying their absence in dynamic languages such as Prolog. For Prolog programmers this means being able to have relevant properties such as modes, types, determinacy, non-failure, sharing, constraints, cost, etc., checked and errors flagged without having to actually run the program. Such global static analysis tools are arguably most useful the earlier they are used in the software development cycle, and fast response times are essential for interactive use. Triggering a full and precise semantic analysis of a software project every time a change is made can be prohibitively expensive. In our static analysis and verification framework this challenge is addressed through a combination of modular and incremental (context- and path-sensitive) analysis that is responsive to program edits, at different levels of granularity. We describe how the combination of this framework within an integrated development environment (IDE) takes advantage of such incrementality to achieve a high level of reactivity when reflecting analysis and verification results back as colorings and tooltips directly on the program text -- the tool's VeriFly mode. The concrete implementation that we describe is Emacs-based and reuses in part off-the-shelf "on-the-fly" syntax checking facilities (flycheck). We believe that similar extensions are also reproducible with low effort in other mature development environments. Our initial experience with the tool shows quite promising results, with low latency times that provide early, continuous, and precise assertion checking and other semantic feedback to programmers during the development process. The tool supports Prolog natively, as well as other languages by semantic transformation into Horn clauses. This paper is under consideration for acceptance in TPLP.