OpenRaiser/Intern-Atlas huggingface dataset:
Batch Normalization Sampling
Venue: arXiv cs.LG 2018
Year: 2018

Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating this problem through sampling only a small fraction of data for normalization at each iteration. Specifically, we model it as a statistical sampling problem and identify that by sampling less correlated data, we can largely reduce the requirement of the number of data for statistics estimation in BN, which directly simplifies the reduction operations. Based on this conclusion, we propose two sampling strategies, "Batch Sampling" (randomly select several samples from each batch) and "Feature Sampling" (randomly select a small patch from each feature map of all samples), that take both computational efficiency and sample correlation into consideration. Furthermore, we introduce an extremely simple variant of BN, termed as Virtual Dataset Normalization (VDN), that can normalize the activations well with few synthetical random samples. All the proposed methods are evaluated on various datasets and networks, where an overall training speedup by up to 20% on GPU is practically achieved without the support of any specialized libraries, and the loss on accuracy and convergence rate are negligible. Finally, we extend our work to the "micro-batch normalization" problem and yield comparable performance with existing approaches at the case of tiny batch size.
The Good, The Bad & The Ugly Features: A Meta-analysis on User Review About Food Journaling Apps
Venue: arXiv cs.HC 2018
Year: 2018

Users review about an app is a crucial component for open mobile application market, such as the AppStore and the Google play. Analyzing these reviews can reveal user's sentiment towards a feature in the app. There exist several analytical tools to summarize user reviews and extract meaningful sense out of them. However, these tools are still limited in terms of expressiveness and accurately classifying the reviews into more than a positive and a negative review. There is a need to get more insights from user app reviews and direct it to future app development. In this paper, we present our result of analyzing user reviews of 20 food journaling and health tracking apps. We gathered and analyzed reviews per app and classified them into three distinct categories using the sentiment treebank with recursive neural tensor network. We then analyzed the vocabulary frequency per category using the Gensim implementation of Word2Vec model. The analysis result clustered the reviews into good, bad and ugly feature reviews. Different usage patterns were detected from users review. We identified major reasons why users express a certain sentiment towards an app and learned how users' satisfaction or complaints was related to a specific feature. This research could be a guideline for app developers to follow when developing an app to refrain from adopting techniques that might demotivate (hinder) the application use or adopt those perceived positively by the users.
Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
Venue: arXiv cs.DC 2018
Year: 2018

TensorFlow has been the most widely adopted Machine/Deep Learning framework. However, little exists in the literature that provides a thorough understanding of the capabilities which TensorFlow offers for the distributed training of large ML/DL models that need computation and communication at scale. Most commonly used distributed training approaches for TF can be categorized as follows: 1) Google Remote Procedure Call (gRPC), 2) gRPC+X: X=(InfiniBand Verbs, Message Passing Interface, and GPUDirect RDMA), and 3) No-gRPC: Baidu Allreduce with MPI, Horovod with MPI, and Horovod with NVIDIA NCCL. In this paper, we provide an in-depth performance characterization and analysis of these distributed training approaches on various GPU clusters including the Piz Daint system (6 on Top500). We perform experiments to gain novel insights along the following vectors: 1) Application-level scalability of DNN training, 2) Effect of Batch Size on scaling efficiency, 3) Impact of the MPI library used for no-gRPC approaches, and 4) Type and size of DNN architectures. Based on these experiments, we present two key insights: 1) Overall, No-gRPC designs achieve better performance compared to gRPC-based approaches for most configurations, and 2) The performance of No-gRPC is heavily influenced by the gradient aggregation using Allreduce. Finally, we propose a truly CUDA-Aware MPI Allreduce design that exploits CUDA kernels and pointer caching to perform large reductions efficiently. Our proposed designs offer 5-17X better performance than NCCL2 for small and medium messages, and reduces latency by 29% for large messages. The proposed optimizations help Horovod-MPI to achieve approximately 90% scaling efficiency for ResNet-50 training on 64 GPUs. Further, Horovod-MPI achieves 1.8X and 3.2X higher throughput than the native gRPC method for ResNet-50 and MobileNet, respectively, on the Piz Daint cluster.
Anti-lock Brake System for Integrated Electric Parking Brake Actuator Based on Sliding-mode Control
Venue: arXiv cs.SY 2018
Year: 2018

Integrated electric parking brake (iEPB) is popularizing on passenger cars due to its easier operation and automatic functions. As a parking brake, EPB have to act as the secondary brake system in case of hydraulic brake failure. To guarantee the stability and safety of a car during iEPB braking, the rear slip ratio has to be controlled accurately within the optimized value to get the shortest brake distance without undesired loss of control. In this paper, a sliding-mode controller (SMC) is investigated to achieve rear-wheel anti-lock brake control, which is robust against uncertainties and disturbance of the parameters. And a sliding-mode observer (SMO) is present to estimate the load torque of d.c. motor and calculate the brake torque. The tyre/road friction coefficient estimator is designed to obtain the optimal rear slip ratio timely. The simulation model of iEPB system is initially constructed in AMESim and the vehicle model is built in MATLAB/Simulink, and the complete system is co-simulated by these two software simultaneously with different road conditions. Simulation results show that the proposed observer and estimator are feasible. This study may provide a useful method to realize rear slip ratio control so that the safety and stability of vehicle could be improved significantly in specified condition
Lossless (and Lossy) Compression of Random Forests
Venue: arXiv cs.LG 2018
Year: 2018

Ensemble methods are among the state-of-the-art predictive modeling approaches. Applied to modern big data, these methods often require a large number of sub-learners, where the complexity of each learner typically grows with the size of the dataset. This phenomenon results in an increasing demand for storage space, which may be very costly. This problem mostly manifests in a subscriber based environment, where a user-specific ensemble needs to be stored on a personal device with strict storage limitations (such as a cellular device). In this work we introduce a novel method for lossless compression of tree-based ensemble methods, focusing on random forests. Our suggested method is based on probabilistic modeling of the ensemble's trees, followed by model clustering via Bregman divergence. This allows us to find a minimal set of models that provides an accurate description of the trees, and at the same time is small enough to store and maintain. Our compression scheme demonstrates high compression rates on a variety of modern datasets. Importantly, our scheme enables predictions from the compressed format and a perfect reconstruction of the original ensemble. In addition, we introduce a theoretically sound lossy compression scheme, which allows us to control the trade-off between the distortion and the coding rate.
Optimal Energy-Delay in Energy Harvesting Wireless Sensor Networks with Interference Channel
Venue: arXiv cs.NI 2018
Year: 2018

In this work, we investigate the capacity allocation problem in the energy harvesting wireless sensor networks (WSNs) with interference channel. For the fixed topologies of data and energy, we formulate the optimization problem when the data flow remains constant on all data links and each sensor node harvests energy only once in a time slot. We focus on the optimal data rates, power allocations and energy transfers between sensor nodes in a time slot. Our goal is to minimize the total delay in the network under two scenarios, i.e., no energy transfer and energy transfer. Furthermore, since the optimization problem is non-convex and difficult to solve directly. By considering the network with relatively high Signal-to-Interference-plus-Noise Ratio (SINR), the non-convex optimization problem can be transformed into a convex optimization problem by convex approximation. We attain the properties of optimal solution by Lagrange duality and solve the convex optimization problem by CVX solver. The experimental results demonstrate that the total delay of the energy harvesting WSNs with interference channel is more than that in the orthogonal channel; and the energy transfer can help to decrease the total delay. Moreover, we also discuss the extension of our work.
Building Footprint Generation Using Improved Generative Adversarial Networks
Venue: arXiv cs.CV 2018
Year: 2018

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. In this work, we have proposed improved generative adversarial networks (GANs) for the automatic generation of building footprints from satellite images. We used a conditional GAN with a cost function derived from the Wasserstein distance and added a gradient penalty term. The achieved results indicated that the proposed method can significantly improve the quality of building footprint generation compared to conditional generative adversarial networks, the U-Net, and other networks. In addition, our method nearly removes all hyperparameters tuning.
Investigating non-classical correlations between decision fused multi-modal documents
Venue: arXiv cs.IR 2018
Year: 2018

Correlation has been widely used to facilitate various information retrieval methods such as query expansion, relevance feedback, document clustering, and multi-modal fusion. Especially, correlation and independence are important issues when fusing different modalities that influence a multi-modal information retrieval process. The basic idea of correlation is that an observable can help predict or enhance another observable. In quantum mechanics, quantum correlation, called entanglement, is a sort of correlation between the observables measured in atomic-size particles when these particles are not necessarily collected in ensembles. In this paper, we examine a multimodal fusion scenario that might be similar to that encountered in physics by firstly measuring two observables (i.e., text-based relevance and image-based relevance) of a multi-modal document without counting on an ensemble of multi-modal documents already labeled in terms of these two variables. Then, we investigate the existence of non-classical correlations between pairs of multi-modal documents. Despite there are some basic differences between entanglement and classical correlation encountered in the macroscopic world, we investigate the existence of this kind of non-classical correlation through the Bell inequality violation. Here, we experimentally test several novel association methods in a small-scale experiment. However, in the current experiment we did not find any violation of the Bell inequality. Finally, we present a series of interesting discussions, which may provide theoretical and empirical insights and inspirations for future development of this direction.
Outlier Detection using Generative Models with Theoretical Performance Guarantees
Venue: arXiv cs.IT 2018
Year: 2018

This paper considers the problem of recovering signals from compressed measurements contaminated with sparse outliers, which has arisen in many applications. In this paper, we propose a generative model neural network approach for reconstructing the ground truth signals under sparse outliers. We propose an iterative alternating direction method of multipliers (ADMM) algorithm for solving the outlier detection problem via $\ell_1$ norm minimization, and a gradient descent algorithm for solving the outlier detection problem via squared $\ell_1$ norm minimization. We establish the recovery guarantees for reconstruction of signals using generative models in the presence of outliers, and give an upper bound on the number of outliers allowed for recovery. Our results are applicable to both the linear generator neural network and the nonlinear generator neural network with an arbitrary number of layers. We conduct extensive experiments using variational auto-encoder and deep convolutional generative adversarial networks, and the experimental results show that the signals can be successfully reconstructed under outliers using our approach. Our approach outperforms the traditional Lasso and $\ell_2$ minimization approach.
A simulated annealing approach to the student-project allocation problem
Venue: arXiv cs.AI 2018
Year: 2018

We describe a solution to the student-project allocation problem using simulated annealing. The problem involves assigning students to projects, where each student has ranked a fixed number of projects in order of preference. Each project is offered by a specific supervisor (or supervisors), and the goal is to find an optimal matching of students to projects taking into account the students' preferences, the constraint that only one student can be assigned to a given project, and the constraint that supervisors have a maximum workload. We show that when applied to a real dataset from a university physics department, simulated annealing allows the rapid determination of high quality solutions to this allocation problem. The quality of the solution is quantified by a satisfaction metric derived from empirical student survey data. Our approach provides high quality allocations in a matter of minutes that are as good as those found previously by the course organizer using a laborious trial-and-error approach. We investigate how the quality of the allocation is affected by the ratio of the number of projects offered to the number of students and the number of projects ranked by each student. We briefly discuss how our approach can be generalized to include other types of constraints and discuss its potential applicability to wider allocation problems.
An Acceleration Scheme to The Local Directional Pattern
Venue: arXiv cs.CV 2018
Year: 2018

This study seeks to improve the running time of the Local Directional Pattern (LDP) during feature extraction using a newly proposed acceleration scheme to LDP. LDP is considered to be computationally expensive. To confirm this, the running time of the LDP to gray level co-occurrence matrix (GLCM) were it was established that the running time for LDP was two orders of magnitude higher than that of the GLCM. In this study, the performance of the newly proposed acceleration scheme was evaluated against LDP and Local Binary patter (LBP) using images from the publicly available extended Cohn-Kanade (CK+) dataset. Based on our findings, the proposed acceleration scheme significantly improves the running time of the LDP by almost 3 times during feature extraction
Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
Venue: arXiv cs.NE 2018
Year: 2018

This paper presents a new technique for training networks for low-precision communication. Targeting minimal communication between nodes not only enables the use of emerging spiking neuromorphic platforms, but may additionally streamline processing conventionally. Low-power and embedded neuromorphic processors potentially offer dramatic performance-per-Watt improvements over traditional von Neumann processors, however programming these brain-inspired platforms generally requires platform-specific expertise which limits their applicability. To date, the majority of artificial neural networks have not operated using discrete spike-like communication. We present a method for training deep spiking neural networks using an iterative modification of the backpropagation optimization algorithm. This method, which we call Whetstone, effectively and reliably configures a network for a spiking hardware target with little, if any, loss in performance. Whetstone networks use single time step binary communication and do not require a rate code or other spike-based coding scheme, thus producing networks comparable in timing and size to conventional ANNs, albeit with binarized communication. We demonstrate Whetstone on a number of image classification networks, describing how the sharpening process interacts with different training optimizers and changes the distribution of activity within the network. We further note that Whetstone is compatible with several non-classification neural network applications, such as autoencoders and semantic segmentation. Whetstone is widely extendable and currently implemented using custom activation functions within the Keras wrapper to the popular TensorFlow machine learning framework.
Study of Joint Automatic Gain Control and MMSE Receiver Design Techniques for Quantized Multiuser Multiple-Antenna Systems
Venue: arXiv cs.IT 2018
Year: 2018

This paper presents the development of a joint optimization of an automatic gain control (AGC) algorithm and a linear \textit{minimum mean square error} (MMSE) receiver for multi-user multiple input multiple output (MU-MIMO) systems with coarsely quantized signals. The optimization of the AGC is based on the minimization of the \textit{mean square error} (MSE) and the proposed receive filter takes into account the presence of the AGC and the effects due to quantization. Moreover, we provide a lower bound on the capacity of the MU-MIMO system by deriving an expression for the achievable rate. The performance of the proposed Low-Resolution Aware MMSE (LRA-MMSE) receiver and AGC algorithm is evaluated by simulations, and compared with the conventional MMSE receive filter and Zero-Forcing (ZF) receiver using quantization resolution of 2, 3, 4 and 5 bits.
Self-Supervised GAN to Counter Forgetting
Venue: arXiv cs.LG 2018
Year: 2018

GANs involve training two networks in an adversarial game, where each network's task depends on its adversary. Recently, several works have framed GAN training as an online or continual learning problem. We focus on the discriminator, which must perform classification under an (adversarially) shifting data distribution. When trained on sequential tasks, neural networks exhibit \emph{forgetting}. For GANs, discriminator forgetting leads to training instability. To counter forgetting, we encourage the discriminator to maintain useful representations by adding a self-supervision. Conditional GANs have a similar effect using labels. However, our self-supervised GAN does not require labels, and closes the performance gap between conditional and unconditional models. We show that, in doing so, the self-supervised discriminator learns better representations than regular GANs.
Reduced-order Aggregate Dynamical Model for Wind Farms
Venue: arXiv cs.SY 2018
Year: 2018

This paper presents an aggregate reduced-order model for a wind farm composed of identical parallel-connected Type-3 wind turbines. The model for individual turbines includes mechanical dynamics (arising from the turbine and doubly fed induction generator) and electrical dynamics (arising from the rotor-side and grid-side converters and associated filters). The proposed aggregate wind-farm model is structure preserving, in the sense that the parameters of the model are derived by scaling corresponding ones from the individual turbines. The aggregate model hence maps to an equivalent--albeit fictitious--wind turbine that captures the dynamics corresponding to the entire wind farm. The reduced-order model has obvious computational advantages, but more importantly, the presented analysis rigorously formalizes parametric scalings for aggregate wind-turbine models that have been applied with limited justification in prior works. Exhaustive numerical simulations validate the accuracy and computational benefits of the proposed reduced-order model.
A New Cryptographic Approach: Iterated Random Encryption (IRE)
Venue: arXiv cs.CR 2018
Year: 2018

A new cryptographic approach -- Iterated Random Encryption (IRE) -- is presented here. Although it is very simple, and easy to implement, it provides a very high level of security. According to this approach, a sequence of operations applied to a message ($M$) yields the encrypted message ($M_E$). In that series of operations, the one with the most important role is operation 6, which involves a random binary sequence (RBS) generated by using the Hybrid Random Number Generator (HRNG) or the Mathematical Random Number Generator (MRNG). A sequence of anti-operations applied to $M_E$ makes it possible to recover $M$.
Accelerated Inference in Markov Random Fields via Smooth Riemannian Optimization
Venue: arXiv cs.CV 2018
Year: 2018

Markov Random Fields (MRFs) are a popular model for several pattern recognition and reconstruction problems in robotics and computer vision. Inference in MRFs is intractable in general and related work resorts to approximation algorithms. Among those techniques, semidefinite programming (SDP) relaxations have been shown to provide accurate estimates while scaling poorly with the problem size and being typically slow for practical applications. Our first contribution is to design a dual ascent method to solve standard SDP relaxations that takes advantage of the geometric structure of the problem to speed up computation. This technique, named Dual Ascent Riemannian Staircase (DARS), is able to solve large problem instances in seconds. Our second contribution is to develop a second and faster approach. The backbone of this second approach is a novel SDP relaxation combined with a fast and scalable solver based on smooth Riemannian optimization. We show that this approach, named Fast Unconstrained SEmidefinite Solver (FUSES), can solve large problems in milliseconds. Contrarily to local MRF solvers, e.g., loopy belief propagation, our approaches do not require an initial guess. Moreover, we leverage recent results from optimization theory to provide per-instance sub-optimality guarantees. We demonstrate the proposed approaches in multi-class image segmentation problems. Extensive experimental evidence shows that (i) FUSES and DARS produce near-optimal solutions, attaining an objective within 0.1% of the optimum, (ii) FUSES and DARS are remarkably faster than general-purpose SDP solvers, and FUSES is more than two orders of magnitude faster than DARS while attaining similar solution quality, (iii) FUSES is faster than local search methods while being a global solver.