OpenRaiser/Intern-Atlas huggingface dataset:
Deep Learning Aided Routing for Space-Air-Ground Integrated Networks Relying on Real Satellite, Flight, and Shipping Data
Venue: arXiv cs.NI 2021
Year: 2021

Current maritime communications mainly rely on satellites having meager transmission resources, hence suffering from poorer performance than modern terrestrial wireless networks. With the growth of transcontinental air traffic, the promising concept of aeronautical ad hoc networking relying on commercial passenger airplanes is potentially capable of enhancing satellite-based maritime communications via air-to-ground and multi-hop air-to-air links. In this article, we conceive space-air-ground integrated networks (SAGINs) for supporting ubiquitous maritime communications, where the low-earth-orbit satellite constellations, passenger airplanes, terrestrial base stations, ships, respectively, serve as the space-, air-, ground- and sea-layer. To meet heterogeneous service requirements, and accommodate the time-varying and self-organizing nature of SAGINs, we propose a deep learning (DL) aided multi-objective routing algorithm, which exploits the quasi-predictable network topology and operates in a distributed manner. Our simulation results based on real satellite, flight, and shipping data in the North Atlantic region show that the integrated network enhances the coverage quality by reducing the end-to-end (E2E) delay and by boosting the E2E throughput as well as improving the path-lifetime. The results demonstrate that our DL-aided multi-objective routing algorithm is capable of achieving near Pareto-optimal performance.
A New Remote Monitor and Control System Based on Sigfox IoT Network
Venue: arXiv cs.NI 2021
Year: 2021

We describe a new, low-cost system designed to provide multi-sensor remote condition monitoring of modern scientific laboratories, as well as to allow users to perform actions from remote locations in case of detection of specified events. The system is battery operated and does not require the presence of a Local Area Network (LAN) or WiFi (which are typically not available in case of, e.g. power losses), as it exploits the growing infrastructure of Internet of Things (IoT) Low Power Wide Area Networks (LPWAN). In particular our system exploits the new SigFox ultra-narrow-bandwidth (UNB) infrastructure, and provides for a bidirectional link between the instrumentation and the remote user even in case of power line outages, which are among the most critical situations that a scientific laboratory can withstand. The system can detect the occurrence of predefined events in very short times, and either autonomously react with a series of predefined actions, also allowing a remote user to timely perform additional actions on the system through an user-friendly smartphone application or via a browser interface. The system also embeds a novel power-loss detection architecture, which detects power line failures in less than 2 ms. We provide a full characterization of the prototype, including reaction times, connection latencies, sensors sensitivity, and power consumption.
Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures
Venue: arXiv cs.CL 2021
Year: 2021

Recent years have seen a growing adoption of Transformer models such as BERT in Natural Language Processing and even in Computer Vision. However, due to their size, there has been limited adoption of such models within resource-constrained computing environments. This paper proposes novel pruning algorithm to compress transformer models by eliminating redundant Attention Heads. We apply the A* search algorithm to obtain a pruned model with strict accuracy guarantees. Our results indicate that the method could eliminate as much as 40% of the attention heads in the BERT transformer model with no loss in accuracy.
Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
Venue: arXiv cs.DC 2021
Year: 2021

Today's auto-tuners (e.g., AutoTVM, Ansor) generate efficient tensor programs by navigating a large search space to identify effective implementations, but they do so with opaque hardware details. Thus, their performance could fall behind that of hardware-native libraries (e.g., cuBLAS, cuDNN), which are hand-optimized by device vendors to extract high performance. On the other hand, these vendor libraries have a fixed set of supported functions and lack the customization and automation support afforded by auto-tuners. Bolt is based on the recent trend that vendor libraries are increasingly modularized and reconfigurable via declarative control (e.g., CUTLASS). It enables a novel approach that bridges this gap and achieves the best of both worlds, via hardware-native templated search. Bolt provides new opportunities to rethink end-to-end tensor optimizations at the graph, operator, and model levels. Bolt demonstrates this concept by prototyping on a popular auto-tuner in TVM and a class of widely-used platforms (i.e., NVIDIA GPUs) -- both in large deployment in our production environment. Bolt improves the inference speed of common convolutional neural networks by 2.5x on average over the state of the art, and it auto-tunes these models within 20 minutes.
Learning Continuous Face Representation with Explicit Functions
Venue: arXiv cs.CV 2021
Year: 2021

How to represent a face pattern? While it is presented in a continuous way in our visual system, computers often store and process the face image in a discrete manner with 2D arrays of pixels. In this study, we attempt to learn a continuous representation for face images with explicit functions. First, we propose an explicit model (EmFace) for human face representation in the form of a finite sum of mathematical terms, where each term is an analytic function element. Further, to estimate the unknown parameters of EmFace, a novel neural network, EmNet, is designed with an encoder-decoder structure and trained using the backpropagation algorithm, where the encoder is defined by a deep convolutional neural network and the decoder is an explicit mathematical expression of EmFace. Experimental results show that EmFace has a higher representation performance on faces with various expressions, postures, and other factors, compared to that of other methods. Furthermore, EmFace achieves reasonable performance on several face image processing tasks, including face image restoration, denoising, and transformation.
Wasserstein Distance Maximizing Intrinsic Control
Venue: arXiv cs.LG 2021
Year: 2021

This paper deals with the problem of learning a skill-conditioned policy that acts meaningfully in the absence of a reward signal. Mutual information based objectives have shown some success in learning skills that reach a diverse set of states in this setting. These objectives include a KL-divergence term, which is maximized by visiting distinct states even if those states are not far apart in the MDP. This paper presents an approach that rewards the agent for learning skills that maximize the Wasserstein distance of their state visitation from the start state of the skill. It shows that such an objective leads to a policy that covers more distance in the MDP than diversity based objectives, and validates the results on a variety of Atari environments.
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
Venue: arXiv cs.LG 2021
Year: 2021

Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of modeling long sequences. However, it is still challenging to balance the trade-off between model quality and efficiency to perform a one-size-fits-all approximation for different tasks. To better understand this trade-off, we observe that sparse and low-rank approximations excel in different regimes, determined by the softmax temperature in attention, and sparse + low-rank can outperform each individually. Inspired by the classical robust-PCA algorithm for sparse and low-rank decomposition, we propose Scatterbrain, a novel way to unify sparse (via locality sensitive hashing) and low-rank (via kernel feature map) attention for accurate and efficient approximation. The estimation is unbiased with provably low error. We empirically show that Scatterbrain can achieve 2.1x lower error than baselines when serving as a drop-in replacement in BigGAN image generation and pre-trained T2T-ViT. On a pre-trained T2T Vision transformer, even without fine-tuning, Scatterbrain can reduce 98% of attention memory at the cost of only 1% drop in accuracy. We demonstrate Scatterbrain for end-to-end training with up to 4 points better perplexity and 5 points better average accuracy than sparse or low-rank efficient transformers on language modeling and long-range-arena tasks.
XDEEP-MSI: Explainable Bias-Rejecting Microsatellite Instability Deep Learning System In Colorectal Cancer
Venue: arXiv cs.CV 2021
Year: 2021

We present a system for the prediction of microsatellite instability (MSI) from H&E images of colorectal cancer using deep learning (DL) techniques customized for tissue microarrays (TMAs). The system incorporates an end-to-end image preprocessing module that produces tiles at multiple magnifications in the regions of interest as guided by a tissue classifier module, and a multiple-bias rejecting module. The training and validation TMA samples were obtained from the EPICOLON project and further enriched with samples from a single institution. A systematic study of biases at tile level identified three protected (bias) variables associated with the learned representations of a baseline model: the project of origin of samples, the patient spot and the TMA glass where each spot was placed. A multiple bias rejecting technique based on adversarial training is implemented at the DL architecture so to directly avoid learning the batch effects of those variables. The learned features from the bias-ablated model have maximum discriminative power with respect to the task and minimal statistical mean dependence with the biases. The impact of different magnifications, types of tissues and the model performance at tile vs patient level is analyzed. The AUC at tile level, and including all three selected tissues (tumor epithelium, mucine and lymphocytic regions) and 4 magnifications, was 0.87 +/- 0.03 and increased to 0.9 +/- 0.03 at patient level. To the best of our knowledge, this is the first work that incorporates a multiple bias ablation technique at the DL architecture in digital pathology, and the first using TMAs for the MSI prediction task.
Iterative and greedy algorithms for the sparsity in levels model in compressed sensing
Venue: arXiv cs.IT 2021
Year: 2021

Motivated by the question of optimal functional approximation via compressed sensing, we propose generalizations of the Iterative Hard Thresholding and the Compressive Sampling Matching Pursuit algorithms able to promote sparse in levels signals. We show, by means of numerical experiments, that the proposed algorithms are successfully able to outperform their unstructured variants when the signal exhibits the sparsity structure of interest. Moreover, in the context of piecewise smooth function approximation, we numerically demonstrate that the structure promoting decoders outperform their unstructured variants and the basis pursuit program when the encoder is structure agnostic.
Fuzzm: Finding Memory Bugs through Binary-Only Instrumentation and Fuzzing of WebAssembly
Venue: arXiv cs.CR 2021
Year: 2021

WebAssembly binaries are often compiled from memory-unsafe languages, such as C and C++. Because of WebAssembly's linear memory and missing protection features, e.g., stack canaries, source-level memory vulnerabilities are exploitable in compiled WebAssembly binaries, sometimes even more easily than in native code. This paper addresses the problem of detecting such vulnerabilities through the first binary-only fuzzer for WebAssembly. Our approach, called Fuzzm, combines canary instrumentation to detect overflows and underflows on the stack and the heap, an efficient coverage instrumentation, a WebAssembly VM, and the input generation algorithm of the popular AFL fuzzer. Besides as an oracle for fuzzing, our canaries also serve as a stand-alone binary hardening technique to prevent the exploitation of vulnerable binaries in production. We evaluate Fuzzm with 28 real-world WebAssembly binaries, some compiled from source and some found in the wild without source code. The fuzzer explores thousands of execution paths, triggers dozens of crashes, and performs hundreds of program executions per second. When used for binary hardening, the approach prevents previously published exploits against vulnerable WebAssembly binaries while imposing low runtime overhead.
FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding
Venue: arXiv cs.LG 2021
Year: 2021

Block Floating Point (BFP) can efficiently support quantization for Deep Neural Network (DNN) training by providing a wide dynamic range via a shared exponent across a group of values. In this paper, we propose a Fast First, Accurate Second Training (FAST) system for DNNs, where the weights, activations, and gradients are represented in BFP. FAST supports matrix multiplication with variable precision BFP input operands, enabling incremental increases in DNN precision throughout training. By increasing the BFP precision across both training iterations and DNN layers, FAST can greatly shorten the training time while reducing overall hardware resource usage. Our FAST Multipler-Accumulator (fMAC) supports dot product computations under multiple BFP precisions. We validate our FAST system on multiple DNNs with different datasets, demonstrating a 2-6$\times$ speedup in training on a single-chip platform over prior work based on \textbf{mixed-precision or block} floating point number systems while achieving similar performance in validation accuracy.
A hierarchical behavior prediction framework at signalized intersections
Venue: arXiv cs.RO 2021
Year: 2021

Road user behavior prediction is one of the most critical components in trajectory planning for autonomous driving, especially in urban scenarios involving traffic signals. In this paper, a hierarchical framework is proposed to predict vehicle behaviors at a signalized intersection, using the traffic signal information of the intersection. The framework is composed of two phases: a discrete intention prediction phase and a continuous trajectory prediction phase. In the discrete intention prediction phase, a Bayesian network is adopted to predict the vehicle's high-level intention, after that, maximum entropy inverse reinforcement learning is utilized to learn the human driving model offline; during the online trajectory prediction phase, a driver characteristic is designed and updated to capture the different driving preferences between human drivers. We applied the proposed framework to one of the most challenging scenarios in autonomous driving: the yellow light running scenario. Numerical experiment results are presented in the later part of the paper which show the viability of the method. The accuracy of the Bayesian network for discrete intention prediction is 91.1%, and the prediction results are getting more and more accurate as the yellow time elapses. The average Euclidean distance error in continuous trajectory prediction is only 0.85 m in the yellow light running scenario.
Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing
Venue: arXiv cs.CL 2021
Year: 2021

Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.
An $O(k \log{n})$ algorithm for prefix based ranked autocomplete
Venue: arXiv cs.DS 2021
Year: 2021

Many search engines such as Google, Bing & Yahoo! show search suggestions when users enter search phrases on their interfaces. These suggestions are meant to assist the user in finding what she wants quickly and also suggesting common searches that may result in finding information that is more relevant. It also serves the purpose of helping the user if she is not sure of what to search for, but has a vague idea of what it is that she wants. We present an algorithm that takes time proportional to $O(k \log{n})$, and $O(n)$ extra space for providing the user with the top $k$ ranked suggestions out of a corpus of $n$ possible suggestions based on the prefix of the query that she has entered so far.
A hybrid chaos map with two control parameters to secure image encryption algorithms
Venue: arXiv cs.CR 2021
Year: 2021

In this paper, we introduce a hybrid chaos map for image encryption method with high sensitivity. This new map is sensitive to small changes in the starting point and also in control parameters which result in having more computational complexity. Also, it has uniform distribution that provides resisting of the new system against attacks in security applications. Various tests and plots are demonstrated to show more chaotic behavior of the proposed system. Finally, to show the ability of the generated chaotic map in the existences image cryptography approaches, we further report some results in this area.
Gabor filter incorporated CNN for compression
Venue: arXiv cs.CV 2021
Year: 2021

Convolutional neural networks (CNNs) are remarkably successful in many computer vision tasks. However, the high cost of inference is problematic for embedded and real-time systems, so there are many studies on compressing the networks. On the other hand, recent advances in self-attention models showed that convolution filters are preferable to self-attention in the earlier layers, which indicates that stronger inductive biases are better in the earlier layers. As shown in convolutional filters, strong biases can train specific filters and construct unnecessarily filters to zero. This is analogous to classical image processing tasks, where choosing the suitable filters makes a compact dictionary to represent features. We follow this idea and incorporate Gabor filters in the earlier layers of CNNs for compression. The parameters of Gabor filters are learned through backpropagation, so the features are restricted to Gabor filters. We show that the first layer of VGG-16 for CIFAR-10 has 192 kernels/features, but learning Gabor filters requires an average of 29.4 kernels. Also, using Gabor filters, an average of 83% and 94% of kernels in the first and the second layer, respectively, can be removed on the altered ResNet-20, where the first five layers are exchanged with two layers of larger kernels for CIFAR-10.
Amendable Generation for Dialogue State Tracking
Venue: arXiv cs.CL 2021
Year: 2021

In task-oriented dialogue systems, recent dialogue state tracking methods tend to perform one-pass generation of the dialogue state based on the previous dialogue state. The mistakes of these models made at the current turn are prone to be carried over to the next turn, causing error propagation. In this paper, we propose a novel Amendable Generation for Dialogue State Tracking (AG-DST), which contains a two-pass generation process: (1) generating a primitive dialogue state based on the dialogue of the current turn and the previous dialogue state, and (2) amending the primitive dialogue state from the first pass. With the additional amending generation pass, our model is tasked to learn more robust dialogue state tracking by amending the errors that still exist in the primitive dialogue state, which plays the role of reviser in the double-checking process and alleviates unnecessary error propagation. Experimental results show that AG-DST significantly outperforms previous works in two active DST datasets (MultiWOZ 2.2 and WOZ 2.0), achieving new state-of-the-art performances.
SDP: Scalable Real-time Dynamic Graph Partitioner
Venue: arXiv cs.DC 2021
Year: 2021

Time-evolving large graph has received attention due to their participation in real-world applications such as social networks and PageRank calculation. It is necessary to partition a large-scale dynamic graph in a streaming manner to overcome the memory bottleneck while partitioning the computational load. Reducing network communication and balancing the load between the partitions are the criteria to achieve effective run-time performance in graph partitioning. Moreover, an optimal resource allocation is needed to utilise the resources while catering the graph streams into the partitions. A number of existing partitioning algorithms (ADP, LogGP and LEOPARD) have been proposed to address the above problem. However, these partitioning methods are incapable of scaling the resources and handling the stream of data in real-time. In this study, we propose a dynamic graph partitioning method called Scalable Dynamic Graph Partitioner (SDP) using the streaming partitioning technique. The SDP contributes a novel vertex assigning method, communication-aware balancing method, and a scaling technique to produce an efficient dynamic graph partitioner. Experiment results show that the proposed method achieves up to 90% reduction of communication cost and 60%-70% balancing the load dynamically, compared with previous algorithms. Moreover, the proposed algorithm significantly reduces the execution time during partitioning.
A Protocol for Emotions
Venue: arXiv cs.HC 2021
Year: 2021

We tend to consider emotions a manifestation of our innermost nature of human beings. Emotions characterize our lives in many ways and they chaperon every rational activity we carry out. Despite their pervasiveness, there are still many things we ignore about emotions. Among them, our understanding of how living beings transfer emotions is limited. In particular, there are highly sophisticated interactions between human beings that we would like to comprehend. For instance, think of a movie director who knows in advance the strong emotional impact that a certain scene will have on the spectators. Although many artists rely on some emotional devices, their talent and vision are still the key factors. In this work we analyze high-level protocols for transferring emotions between two intelligent agents. To the best of our knowledge, this is the first attempt to use communication protocols for modeling the exchange of human emotions. By means of a number of examples, we show that our protocols adequately model the engagement of the two parties. Beyond the theoretical interest, our proposal can provide a stepping stone for several applications that we also discuss in this paper.