OpenRaiser/Intern-Atlas huggingface dataset:
Automatic Graph Partitioning for Very Large-scale Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size of neural network models continues to grow. Since such models do not fit into the memory of accelerator devices, they need to be partitioned by model parallelism techniques. Moreover, to accelerate training for huge training data, we need a combination of model and data parallelisms, i.e., hybrid parallelism. Given a model description for PyTorch without any specification for model parallelism, RaNNC automatically partitions the model into a set of subcomponents so that (1) each subcomponent fits a device memory and (2) a high training throughput for pipeline parallelism is achieved by balancing the computation times of the subcomponents. In our experiments, we compared RaNNC with two popular frameworks, Megatron-LM (hybrid parallelism) and GPipe (originally proposed for model parallelism, but a version allowing hybrid parallelism also exists), for training models with increasingly greater numbers of parameters. In the pre-training of enlarged BERT models, RaNNC successfully trained models five times larger than those Megatron-LM could, and RaNNC's training throughputs were comparable to Megatron-LM's when pre-training the same models. RaNNC also achieved better training throughputs than GPipe on both the enlarged BERT model pre-training (GPipe with hybrid parallelism) and the enlarged ResNet models (GPipe with model parallelism) in all of the settings we tried. These results are remarkable, since RaNNC automatically partitions models without any modification to their descriptions; Megatron-LM and GPipe require users to manually rewrite the models' descriptions.
Fast and Accurate Normal Estimation for Point Cloud via Patch Stitching
Venue: arXiv cs.CV 2021
Year: 2021

This paper presents an effective normal estimation method adopting multi-patch stitching for an unstructured point cloud. The majority of learning-based approaches encode a local patch around each point of a whole model and estimate the normals in a point-by-point manner. In contrast, we suggest a more efficient pipeline, in which we introduce a patch-level normal estimation architecture to process a series of overlapping patches. Additionally, a multi-normal selection method based on weights, dubbed as multi-patch stitching, integrates the normals from the overlapping patches. To reduce the adverse effects of sharp corners or noise in a patch, we introduce an adaptive local feature aggregation layer to focus on an anisotropic neighborhood. We then utilize a multi-branch planar experts module to break the mutual influence between underlying piecewise surfaces in a patch. At the stitching stage, we use the learned weights of multi-branch planar experts and distance weights between points to select the best normal from the overlapping parts. Furthermore, we put forward constructing a sparse matrix representation to reduce large-scale retrieval overheads for the loop iterations dramatically. Extensive experiments demonstrate that our method achieves SOTA results with the advantage of lower computational costs and higher robustness to noise over most of the existing approaches.
Environmental sound analysis with mixup based multitask learning and cross-task fusion
Venue: arXiv cs.SD 2021
Year: 2021

Environmental sound analysis is currently getting more and more attentions. In the domain, acoustic scene classification and acoustic event classification are two closely related tasks. In this letter, a two-stage method is proposed for the above tasks. In the first stage, a mixup based MTL solution is proposed to classify both tasks in one single convolutional neural network. Artificial multi-label samples are used in the training of the MTL model, which are mixed up using existing single-task datasets. The multi-task model obtained can effectively recognize both the acoustic scenes and events. Compared with other methods such as re-annotation or synthesis, the mixup based MTL is low-cost, flexible and effective. In the second stage, the MTL model is modified into a single-task model which is fine-tuned using the original dataset corresponding to the specific task. By controlling the frozen layers carefully, the task-specific high level features are fused and the performance of the single classification task is further improved. The proposed method has confirmed the complementary characteristics of acoustic scene and acoustic event classifications. Finally, enhanced by ensemble learning, a satisfactory accuracy of 84.5 percent on TUT acoustic scene 2017 dataset and an accuracy of 77.5 percent on ESC-50 dataset are achieved respectively.
Symbolic Music Generation with Diffusion Models
Venue: arXiv cs.SD 2021
Year: 2021

Score-based generative models and diffusion probabilistic models have been successful at generating high-quality samples in continuous domains such as images and audio. However, due to their Langevin-inspired sampling mechanisms, their application to discrete and sequential data has been limited. In this work, we present a technique for training diffusion models on sequential data by parameterizing the discrete domain in the continuous latent space of a pre-trained variational autoencoder. Our method is non-autoregressive and learns to generate sequences of latent embeddings through the reverse process and offers parallel generation with a constant number of iterative refinement steps. We apply this technique to modeling symbolic music and show strong unconditional generation and post-hoc conditional infilling results compared to autoregressive language models operating over the same continuous embeddings.
Large Scale Autonomous Driving Scenarios Clustering with Self-supervised Feature Extraction
Venue: arXiv cs.CV 2021
Year: 2021

The clustering of autonomous driving scenario data can substantially benefit the autonomous driving validation and simulation systems by improving the simulation tests' completeness and fidelity. This article proposes a comprehensive data clustering framework for a large set of vehicle driving data. Existing algorithms utilize handcrafted features whose quality relies on the judgments of human experts. Additionally, the related feature compression methods are not scalable for a large data-set. Our approach thoroughly considers the traffic elements, including both in-traffic agent objects and map information. Meanwhile, we proposed a self-supervised deep learning approach for spatial and temporal feature extraction to avoid biased data representation. With the newly designed driving data clustering evaluation metrics based on data-augmentation, the accuracy assessment does not require a human-labeled data-set, which is subject to human bias. Via such unprejudiced evaluation metrics, we have shown our approach surpasses the existing methods that rely on handcrafted feature extractions.
Remote Virtual Showdown: A Collaborative Virtual Reality Game for People with Visual Impairments
Venue: arXiv cs.HC 2021
Year: 2021

Many researchers have developed VR systems for people with visual impairments by using various audio feedback techniques. However, there has been much less study of collaborative VR systems in which people with visual impairments and people with able-body can participate together. Therefore, we developed a VR showdown game which is similar to a real Showdown game in which two players can play together in the same virtual environment. We incorporate auditory distance perception using the HRTF (Head Related Transform Function) based on a spatial position in VR. We developed two modes in the showdown game. One is the PVA (Player vs. Agent) mode in which people with visual impairments can play alone and the PVP (Player vs. Player) mode in which people with visual impairments can play with another player in the network environment. We conducted our user studies by comparing the performances of people with visual impairments and people with able-body. The user study results show that people with visual impairments won 67.6% of the games when competing against people with able-body. This paper reports an example of a collaborative VR system for people with visual impairments and also design guideline for developing VR systems for people with visual impairments.
Approximation algorithm for finding short synchronizing words in weighted automata
Venue: arXiv cs.FL 2021
Year: 2021

In this paper we are dealing with the issue of finding possibly short synchronizing words in automata with weight assigned to each letter in the alphabet $Σ$. First we discuss some complexity problems, and then we present new approximation algorithm in four variations.
Using Low-rank Representation of Abundance Maps and Nonnegative Tensor Factorization for Hyperspectral Nonlinear Unmixing
Venue: arXiv cs.CV 2021
Year: 2021

Tensor-based methods have been widely studied to attack inverse problems in hyperspectral imaging since a hyperspectral image (HSI) cube can be naturally represented as a third-order tensor, which can perfectly retain the spatial information in the image. In this article, we extend the linear tensor method to the nonlinear tensor method and propose a nonlinear low-rank tensor unmixing algorithm to solve the generalized bilinear model (GBM). Specifically, the linear and nonlinear parts of the GBM can both be expressed as tensors. Furthermore, the low-rank structures of abundance maps and nonlinear interaction abundance maps are exploited by minimizing their nuclear norm, thus taking full advantage of the high spatial correlation in HSIs. Synthetic and real-data experiments show that the low rank of abundance maps and nonlinear interaction abundance maps exploited in our method can improve the performance of the nonlinear unmixing. A MATLAB demo of this work will be available at https://github.com/LinaZhuang for the sake of reproducibility.
Convolutional Neural Networks for Sleep Stage Scoring on a Two-Channel EEG Signal
Venue: arXiv cs.LG 2021
Year: 2021

Sleeping problems have become one of the major diseases all over the world. To tackle this issue, the basic tool used by specialists is the Polysomnogram, which is a collection of different signals recorded during sleep. After its recording, the specialists have to score the different signals according to one of the standard guidelines. This process is carried out manually, which can be highly time-consuming and very prone to annotation errors. Therefore, over the years, many approaches have been explored in an attempt to support the specialists in this task. In this paper, an approach based on convolutional neural networks is presented, where an in-depth comparison is performed in order to determine the convenience of using more than one signal simultaneously as input. Additionally, the models were also used as parts of an ensemble model to check whether any useful information can be extracted from signal processing a single signal at a time which the dual-signal model cannot identify. Tests have been performed by using a well-known dataset called expanded sleep-EDF, which is the most commonly used dataset as the benchmark for this problem. The tests were carried out with a leave-one-out cross-validation over the patients, which ensures that there is no possible contamination between training and testing. The resulting proposal is a network smaller than previously published ones, but which overcomes the results of any previous models on the same dataset. The best result shows an accuracy of 92.67\% and a Cohen's Kappa value over 0.84 compared to human experts.
Head2HeadFS: Video-based Head Reenactment with Few-shot Learning
Venue: arXiv cs.CV 2021
Year: 2021

Over the past years, a substantial amount of work has been done on the problem of facial reenactment, with the solutions coming mainly from the graphics community. Head reenactment is an even more challenging task, which aims at transferring not only the facial expression, but also the entire head pose from a source person to a target. Current approaches either train person-specific systems, or use facial landmarks to model human heads, a representation that might transfer unwanted identity attributes from the source to the target. We propose head2headFS, a novel easily adaptable pipeline for head reenactment. We condition synthesis of the target person on dense 3D face shape information from the source, which enables high quality expression and pose transfer. Our video-based rendering network is fine-tuned under a few-shot learning strategy, using only a few samples. This allows for fast adaptation of a generic generator trained on a multiple-person dataset, into a person-specific one.
Linear time DBSCAN for sorted 1D data and laser range scan segmentation
Venue: arXiv cs.RO 2021
Year: 2021

This paper introduces new algorithm for line extraction from laser range data including methodology for efficient computation. The task is cast to series of one dimensional problems in various spaces. A fast and simple specialization of DBSCAN algorithm is proposed to solve one dimensional subproblems. Experiments suggest that the method is suitable for real-time applications, handles noise well and may be useful in practice.
Multi-modal Trajectory Prediction for Autonomous Driving with Semantic Map and Dynamic Graph Attention Network
Venue: arXiv cs.CV 2021
Year: 2021

Predicting future trajectories of surrounding obstacles is a crucial task for autonomous driving cars to achieve a high degree of road safety. There are several challenges in trajectory prediction in real-world traffic scenarios, including obeying traffic rules, dealing with social interactions, handling traffic of multi-class movement, and predicting multi-modal trajectories with probability. Inspired by people's natural habit of navigating traffic with attention to their goals and surroundings, this paper presents a unique dynamic graph attention network to solve all those challenges. The network is designed to model the dynamic social interactions among agents and conform to traffic rules with a semantic map. By extending the anchor-based method to multiple types of agents, the proposed method can predict multi-modal trajectories with probabilities for multi-class movements using a single model. We validate our approach on the proprietary autonomous driving dataset for the logistic delivery scenario and two publicly available datasets. The results show that our method outperforms state-of-the-art techniques and demonstrates the potential for trajectory prediction in real-world traffic.
Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers
Venue: arXiv cs.CL 2021
Year: 2021

Generating knowledge grounded responses in both goal and non-goal oriented dialogue systems is an important research challenge. Knowledge Graphs (KG) can be viewed as an abstraction of the real world, which can potentially facilitate a dialogue system to produce knowledge grounded responses. However, integrating KGs into the dialogue generation process in an end-to-end manner is a non-trivial task. This paper proposes a novel architecture for integrating KGs into the response generation process by training a BERT model that learns to answer using the elements of the KG (entities and relations) in a multi-task, end-to-end setting. The k-hop subgraph of the KG is incorporated into the model during training and inference using Graph Laplacian. Empirical evaluation suggests that the model achieves better knowledge groundedness (measured via Entity F1 score) compared to other state-of-the-art models for both goal and non-goal oriented dialogues.
Exploring Edge TPU for Network Intrusion Detection in IoT
Venue: arXiv cs.NI 2021
Year: 2021

This paper explores Google's Edge TPU for implementing a practical network intrusion detection system (NIDS) at the edge of IoT, based on a deep learning approach. While there are a significant number of related works that explore machine learning based NIDS for the IoT edge, they generally do not consider the issue of the required computational and energy resources. The focus of this paper is the exploration of deep learning-based NIDS at the edge of IoT, and in particular the computational and energy efficiency. In particular, the paper studies Google's Edge TPU as a hardware platform, and considers the following three key metrics: computation (inference) time, energy efficiency and the traffic classification performance. Various scaled model sizes of two major deep neural network architectures are used to investigate these three metrics. The performance of the Edge TPU-based implementation is compared with that of an energy efficient embedded CPU (ARM Cortex A53). Our experimental evaluation shows some unexpected results, such as the fact that the CPU significantly outperforms the Edge TPU for small model sizes.
Deep Regression on Manifolds: A 3D Rotation Case Study
Venue: arXiv cs.CV 2021
Year: 2021

Many machine learning problems involve regressing variables on a non-Euclidean manifold -- e.g. a discrete probability distribution, or the 6D pose of an object. One way to tackle these problems through gradient-based learning is to use a differentiable function that maps arbitrary inputs of a Euclidean space onto the manifold. In this paper, we establish a set of desirable properties for such mapping, and in particular highlight the importance of pre-images connectivity/convexity. We illustrate these properties with a case study regarding 3D rotations. Through theoretical considerations and methodological experiments on a variety of tasks, we review various differentiable mappings on the 3D rotation space, and conjecture about the importance of their local linearity. We show that a mapping based on Procrustes orthonormalization generally performs best among the mappings considered, but that a rotation vector representation might also be suitable when restricted to small angles.
Thermal Neural Networks: Lumped-Parameter Thermal Modeling With State-Space Machine Learning
Venue: arXiv cs.LG 2021
Year: 2021

With electric power systems becoming more compact and increasingly powerful, the relevance of thermal stress especially during overload operation is expected to increase ceaselessly. Whenever critical temperatures cannot be measured economically on a sensor base, a thermal model lends itself to estimate those unknown quantities. Thermal models for electric power systems are usually required to be both, real-time capable and of high estimation accuracy. Moreover, ease of implementation and time to production play an increasingly important role. In this work, the thermal neural network (TNN) is introduced, which unifies both, consolidated knowledge in the form of heat-transfer-based lumped-parameter models, and data-driven nonlinear function approximation with supervised machine learning. A quasi-linear parameter-varying system is identified solely from empirical data, where relationships between scheduling variables and system matrices are inferred statistically and automatically. At the same time, a TNN has physically interpretable states through its state-space representation, is end-to-end trainable -- similar to deep learning models -- with automatic differentiation, and requires no material, geometry, nor expert knowledge for its design. Experiments on an electric motor data set show that a TNN achieves higher temperature estimation accuracies than previous white-/grey- or black-box models with a mean squared error of $3.18~\text{K}^2$ and a worst-case error of $5.84~\text{K}$ at 64 model parameters.
Automated Cleanup of the ImageNet Dataset by Model Consensus, Explainability and Confident Learning
Venue: arXiv cs.CV 2021
Year: 2021

The convolutional neural networks (CNNs) trained on ILSVRC12 ImageNet were the backbone of various applications as a generic classifier, a feature extractor or a base model for transfer learning. This paper describes automated heuristics based on model consensus, explainability and confident learning to correct labeling mistakes and remove ambiguous images from this dataset. After making these changes on the training and validation sets, the ImageNet-Clean improves the model performance by 2-2.4 % for SqueezeNet and EfficientNet-B0 models. The results support the importance of larger image corpora and semi-supervised learning, but the original datasets must be fixed to avoid transmitting their mistakes and biases to the student learner. Further contributions describe the training impacts of widescreen input resolutions in portrait and landscape orientations. The trained models and scripts are published on Github (https://github.com/kecsap/imagenet-clean) to clean up ImageNet and ImageNetV2 datasets for reproducible research.
Scheduling in the Secretary Model
Venue: arXiv cs.DS 2021
Year: 2021

This paper studies Makespan Minimization in the secretary model. Formally, jobs, specified by their processing times, are presented in a uniformly random order. An online algorithm has to assign each job permanently and irrevocably to one of m parallel and identical machines such that the expected time it takes to process them all, the makespan, is minimized. We give two deterministic algorithms. First, a straightforward adaptation of the semi-online strategy LightLoad provides a very simple algorithm retaining its competitive ratio of 1.75. A new and sophisticated algorithm is 1.535-competitive. These competitive ratios are not only obtained in expectation but, in fact, for all but a very tiny fraction of job orders. Classically, online makespan minimization only considers the worst-case order. Here, no competitive ratio below 1.885 for deterministic algorithms and 1.581 using randomization is possible. The best randomized algorithm so far is 1.916-competitive. Our results show that classical worst-case orders are quite rare and pessimistic for many applications. They also demonstrate the power of randomization when compared to much stronger deterministic reordering models. We complement our results by providing first lower bounds. A competitive ratio obtained on nearly all possible job orders must be at least 1.257. This implies a lower bound of 1.043 for both deterministic and randomized algorithms in the general model.
Nonlinear Weighted Directed Acyclic Graph and A Priori Estimates for Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

In an attempt to better understand structural benefits and generalization power of deep neural networks, we firstly present a novel graph theoretical formulation of neural network models, including fully connected, residual network (ResNet) and densely connected networks (DenseNet). Secondly, we extend the error analysis of the population risk for two layer network \cite{ew2019prioriTwo} and ResNet \cite{e2019prioriRes} to DenseNet, and show further that for neural networks satisfying certain mild conditions, similar estimates can be obtained. These estimates are a priori in nature since they depend sorely on the information prior to the training process, in particular, the bounds for the estimation errors are independent of the input dimension.
Representing ELMo embeddings as two-dimensional text online
Venue: arXiv cs.CL 2021
Year: 2021

We describe a new addition to the WebVectors toolkit which is used to serve word embedding models over the Web. The new ELMoViz module adds support for contextualized embedding architectures, in particular for ELMo models. The provided visualizations follow the metaphor of `two-dimensional text' by showing lexical substitutes: words which are most semantically similar in context to the words of the input sentence. The system allows the user to change the ELMo layers from which token embeddings are inferred. It also conveys corpus information about the query words and their lexical substitutes (namely their frequency tiers and parts of speech). The module is well integrated into the rest of the WebVectors toolkit, providing lexical hyperlinks to word representations in static embedding models. Two web services have already implemented the new functionality with pre-trained ELMo models for Russian, Norwegian and English.