OpenRaiser/Intern-Atlas huggingface dataset:
VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning
Venue: arXiv cs.LG 2018
Year: 2018

In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD). Unlike the choices of snapshot and starting points in SVRG and its proximal variant, Prox-SVRG, the two vectors of VR-SGD are set to the average and last iterate of the previous epoch, respectively. The settings allow us to use much larger learning rates, and also make our convergence analysis more challenging. We also design two different update rules for smooth and non-smooth objective functions, respectively, which means that VR-SGD can tackle non-smooth and/or non-strongly convex problems directly without any reduction techniques. Moreover, we analyze the convergence properties of VR-SGD for strongly convex problems, which show that VR-SGD attains linear convergence. Different from its counterparts that have no convergence guarantees for non-strongly convex problems, we also provide the convergence guarantees of VR-SGD for this case, and empirically verify that VR-SGD with varying learning rates achieves similar performance to its momentum accelerated variant that has the optimal convergence rate $\mathcal{O}(1/T^2)$. Finally, we apply VR-SGD to solve various machine learning problems, such as convex and non-convex empirical risk minimization, and leading eigenvalue computation. Experimental results show that VR-SGD converges significantly faster than SVRG and Prox-SVRG, and usually outperforms state-of-the-art accelerated methods, e.g., Katyusha.
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
Venue: arXiv cs.LG 2018
Year: 2018

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.
Golden Angle Modulation: Approaching the AWGN Capacity
Venue: arXiv cs.IT 2018
Year: 2018

In this work, targeting, e.g., future generation cellular, microwave-links, or optical fiber systems, we propose a new geometric shaping design for golden angle modulation (GAM) based on a (double) truncated Gaussian input distribution. The design improves the mutual information (MI), and the peak-to-average power ratio, over the full signal-to-noise ratio (SNR) range relative to two key GAM schemes introduced in [1],[2]. Inspired by the proposed geometric shaping, a simpler, SNR-dependent, design is also suggested. The performance is numerically evaluated with respect to MI and compared with classical modulation schemes. With the proposed design, the SNR can be decreased relative to classical quadrature amplitude modulation, even for relatively modest target spectral efficiencies. As the GAM design can approach the Gaussian channel capacity, the power/energy efficiency is expected to improve.
A Differential Privacy Mechanism Design Under Matrix-Valued Query
Venue: arXiv cs.LG 2018
Year: 2018

Traditionally, differential privacy mechanism design has been tailored for a scalar-valued query function. Although many mechanisms such as the Laplace and Gaussian mechanisms can be extended to a matrix-valued query function by adding i.i.d. noise to each element of the matrix, this method is often sub-optimal as it forfeits an opportunity to exploit the structural characteristics typically associated with matrix analysis. In this work, we consider the design of differential privacy mechanism specifically for a matrix-valued query function. The proposed solution is to utilize a matrix-variate noise, as opposed to the traditional scalar-valued noise. Particularly, we propose a novel differential privacy mechanism called the Matrix-Variate Gaussian (MVG) mechanism, which adds a matrix-valued noise drawn from a matrix-variate Gaussian distribution. We prove that the MVG mechanism preserves $(ε,δ)$-differential privacy, and show that it allows the structural characteristics of the matrix-valued query function to naturally be exploited. Furthermore, due to the multi-dimensional nature of the MVG mechanism and the matrix-valued query, we introduce the concept of directional noise, which can be utilized to mitigate the impact the noise has on the utility of the query. Finally, we demonstrate the performance of the MVG mechanism and the advantages of directional noise using three matrix-valued queries on three privacy-sensitive datasets. We find that the MVG mechanism notably outperforms four previous state-of-the-art approaches, and provides comparable utility to the non-private baseline. Our work thus presents a promising prospect for both future research and implementation of differential privacy for matrix-valued query functions.
On the role of working memory in trading-off skills and situation awareness in Sudoku
Venue: arXiv cs.HC 2018
Year: 2018

Working memory accounts for the ability of humans to perform cognitive processing, by handling both the representation of information (the mental picture forming the situation awareness) and the space required for processing these information (skill processing). The more complex the skills are, the more processing space they require, the less space becomes available for storage of information. This interplay between situation awareness and skills is critical in many applications. Theoretically, it is less understood in cognition and neuroscience. In the meantime, and practically, it is vital when analysing the mental processes involved in safety-critical domains. In this paper, we use the Sudoku game as a vehicle to study this trade-off. This game combines two features that are present during a user interaction with a software in many safety critical domains: scanning for information and processing of information. We use a society of agents for investigating how this trade-off influences player's proficiency.
Transparent Voting Platform Based on Permissioned Blockchain
Venue: arXiv cs.CY 2018
Year: 2018

Since 2004, different research was handling the challenges in the centralized voting systems, e-voting protocols and recently the decentralized voting. So electronic voting puts forward some difficulties regarding the voter anonymity, the secure casting of the votes and to prevent the voting process from frauding. The Decentralized property of the technology called "blockchain" could have the solution for many of the challenges in voting research area and brings a new secure mechanism of safe and transparent voting. In this paper, a broad comparison between ongoing voting systems has studied by analyzing their structure and the drawbacks that should consider in future to improve the whole election process from keeping the privacy of the voter, casting a vote with the possibility to check if it was counted correctly to publishing the results. The result of the paper will give a new approach to extend the target of the election from small scale to large scale despite the fact of Ethereum limitation which can cast on the blockchain just five votes per minute. The primary challenge is to find an answer for this question: "How to balance between voter privacy and transparency without breaking the important rule where the voter can proof for a specific candidate that he voted for him in a bribe situation?".
Predicting Transportation Modes of GPS Trajectories using Feature Engineering and Noise Removal
Venue: arXiv cs.OH 2018
Year: 2018

Understanding transportation mode from GPS (Global Positioning System) traces is an essential topic in the data mobility domain. In this paper, a framework is proposed to predict transportation modes. This framework follows a sequence of five steps: (i) data preparation, where GPS points are grouped in trajectory samples; (ii) point features generation; (iii) trajectory features extraction; (iv) noise removal; (v) normalization. We show that the extraction of the new point features: bearing rate, the rate of rate of change of the bearing rate and the global and local trajectory features, like medians and percentiles enables many classifiers to achieve high accuracy (96.5%) and f1 (96.3%) scores. We also show that the noise removal task affects the performance of all the models tested. Finally, the empirical tests where we compare this work against state-of-art transportation mode prediction strategies show that our framework is competitive and outperforms most of them.
ADMM-based Networked Stochastic Variational Inference
Venue: arXiv cs.LG 2018
Year: 2018

Owing to the recent advances in "Big Data" modeling and prediction tasks, variational Bayesian estimation has gained popularity due to their ability to provide exact solutions to approximate posteriors. One key technique for approximate inference is stochastic variational inference (SVI). SVI poses variational inference as a stochastic optimization problem and solves it iteratively using noisy gradient estimates. It aims to handle massive data for predictive and classification tasks by applying complex Bayesian models that have observed as well as latent variables. This paper aims to decentralize it allowing parallel computation, secure learning and robustness benefits. We use Alternating Direction Method of Multipliers in a top-down setting to develop a distributed SVI algorithm such that independent learners running inference algorithms only require sharing the estimated model parameters instead of their private datasets. Our work extends the distributed SVI-ADMM algorithm that we first propose, to an ADMM-based networked SVI algorithm in which not only are the learners working distributively but they share information according to rules of a graph by which they form a network. This kind of work lies under the umbrella of `deep learning over networks' and we verify our algorithm for a topic-modeling problem for corpus of Wikipedia articles. We illustrate the results on latent Dirichlet allocation (LDA) topic model in large document classification, compare performance with the centralized algorithm, and use numerical experiments to corroborate the analytical results.
Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
Venue: arXiv cs.LG 2018
Year: 2018

Deep Neural Networks (DNNs) provide state-of-the-art solutions in several difficult machine perceptual tasks. However, their performance relies on the availability of a large set of labeled training data, which limits the breadth of their applicability. Hence, there is a need for new {\em semi-supervised learning} methods for DNNs that can leverage both (a small amount of) labeled and unlabeled training data. In this paper, we develop a general loss function enabling DNNs of any topology to be trained in a semi-supervised manner without extra hyper-parameters. As opposed to current semi-supervised techniques based on topology-specific or unstable approaches, ours is both robust and general. We demonstrate that our approach reaches state-of-the-art performance on the SVHN ($9.82\%$ test error, with $500$ labels and wide Resnet) and CIFAR10 (16.38% test error, with 8000 labels and sigmoid convolutional neural network) data sets.
Fractional Programming for Communication Systems--Part II: Uplink Scheduling via Matching
Venue: arXiv cs.IT 2018
Year: 2018

This two-part paper develops novel methodologies for using fractional programming (FP) techniques to design and optimize communication systems. Part I of this paper proposes a new quadratic transform for FP and treats its application for continuous optimization problems. In this Part II of the paper, we study discrete problems, such as those involving user scheduling, which are considerably more difficult to solve. Unlike the continuous problems, discrete or mixed discrete-continuous problems normally cannot be recast as convex problems. In contrast to the common heuristic of relaxing the discrete variables, this work reformulates the original problem in an FP form amenable to distributed combinatorial optimization. The paper illustrates this methodology by tackling the important and challenging problem of uplink coordinated multi-cell user scheduling in wireless cellular systems. Uplink scheduling is more challenging than downlink scheduling, because uplink user scheduling decisions significantly affect the interference pattern in nearby cells. Further, the discrete scheduling variable needs to be optimized jointly with continuous variables such as transmit power levels and beamformers. The main idea of the proposed FP approach is to decouple the interaction among the interfering links, thereby permitting a distributed and joint optimization of the discrete and continuous variables with provable convergence. The paper shows that the well-known weighted minimum mean-square-error (WMMSE) algorithm can also be derived from a particular use of FP; but our proposed FP-based method significantly outperforms WMMSE when discrete user scheduling variables are involved, both in term of run-time efficiency and optimizing results.
Brain Tumor Type Classification via Capsule Networks
Venue: arXiv cs.CV 2018
Year: 2018

Brain tumor is considered as one of the deadliest and most common form of cancer both in children and in adults. Consequently, determining the correct type of brain tumor in early stages is of significant importance to devise a precise treatment plan and predict patient's response to the adopted treatment. In this regard, there has been a recent surge of interest in designing Convolutional Neural Networks (CNNs) for the problem of brain tumor type classification. However, CNNs typically require large amount of training data and can not properly handle input transformations. Capsule networks (referred to as CapsNets) are brand new machine learning architectures proposed very recently to overcome these shortcomings of CNNs, and posed to revolutionize deep learning solutions. Of particular interest to this work is that Capsule networks are robust to rotation and affine transformation, and require far less training data, which is the case for processing medical image datasets including brain Magnetic Resonance Imaging (MRI) images. In this paper, we focus to achieve the following four objectives: (i) Adopt and incorporate CapsNets for the problem of brain tumor classification to design an improved architecture which maximizes the accuracy of the classification problem at hand; (ii) Investigate the over-fitting problem of CapsNets based on a real set of MRI images; (iii) Explore whether or not CapsNets are capable of providing better fit for the whole brain images or just the segmented tumor, and; (iv) Develop a visualization paradigm for the output of the CapsNet to better explain the learned features. Our results show that the proposed approach can successfully overcome CNNs for the brain tumor classification problem.
Behavioral Learning of Aircraft Landing Sequencing Using a Society of Probabilistic Finite State Machines
Venue: arXiv cs.NE 2018
Year: 2018

Air Traffic Control (ATC) is a complex safety critical environment. A tower controller would be making many decisions in real-time to sequence aircraft. While some optimization tools exist to help the controller in some airports, even in these situations, the real sequence of the aircraft adopted by the controller is significantly different from the one proposed by the optimization algorithm. This is due to the very dynamic nature of the environment. The objective of this paper is to test the hypothesis that one can learn from the sequence adopted by the controller some strategies that can act as heuristics in decision support tools for aircraft sequencing. This aim is tested in this paper by attempting to learn sequences generated from a well-known sequencing method that is being used in the real world. The approach relies on a genetic algorithm (GA) to learn these sequences using a society Probabilistic Finite-state Machines (PFSMs). Each PFSM learns a different sub-space; thus, decomposing the learning problem into a group of agents that need to work together to learn the overall problem. Three sequence metrics (Levenshtein, Hamming and Position distances) are compared as the fitness functions in GA. As the results suggest, it is possible to learn the behavior of the algorithm/heuristic that generated the original sequence from very limited information.
Networking the Boids is More Robust Against Adversarial Learning
Venue: arXiv cs.NE 2018
Year: 2018

Swarm behavior using Boids-like models has been studied primarily using close-proximity spatial sensory information (e.g. vision range). In this study, we propose a novel approach in which the classic definition of boids\textquoteright \ neighborhood that relies on sensory perception and Euclidian space locality is replaced with graph-theoretic network-based proximity mimicking communication and social networks. We demonstrate that networking the boids leads to faster swarming and higher quality of the formation. We further investigate the effect of adversarial learning, whereby an observer attempts to reverse engineer the dynamics of the swarm through observing its behavior. The results show that networking the swarm demonstrated a more robust approach against adversarial learning than a local-proximity neighborhood structure.
Investigating Human Priors for Playing Video Games
Venue: arXiv cs.AI 2018
Year: 2018

What makes humans so good at solving seemingly complex video games? Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors on human performance. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play. Videos and the game manipulations are available at https://rach0012.github.io/humanRL_website/
Parametrized Accelerated Methods Free of Condition Number
Venue: arXiv cs.LG 2018
Year: 2018

Analyses of accelerated (momentum-based) gradient descent usually assume bounded condition number to obtain exponential convergence rates. However, in many real problems, e.g., kernel methods or deep neural networks, the condition number, even locally, can be unbounded, unknown or mis-estimated. This poses problems in both implementing and analyzing accelerated algorithms. In this paper, we address this issue by proposing parametrized accelerated methods by considering the condition number as a free parameter. We provide spectral-level analysis for several important accelerated algorithms, obtain explicit expressions and improve worst case convergence rates. Moreover, we show that those algorithm converge exponentially even when the condition number is unknown or mis-estimated.
An EMG Gesture Recognition System with Flexible High-Density Sensors and Brain-Inspired High-Dimensional Classifier
Venue: arXiv cs.HC 2018
Year: 2018

EMG-based gesture recognition shows promise for human-machine interaction. Systems are often afflicted by signal and electrode variability which degrades performance over time. We present an end-to-end system combating this variability using a large-area, high-density sensor array and a robust classification algorithm. EMG electrodes are fabricated on a flexible substrate and interfaced to a custom wireless device for 64-channel signal acquisition and streaming. We use brain-inspired high-dimensional (HD) computing for processing EMG features in one-shot learning. The HD algorithm is tolerant to noise and electrode misplacement and can quickly learn from few gestures without gradient descent or back-propagation. We achieve an average classification accuracy of 96.64% for five gestures, with only 7% degradation when training and testing across different days. Our system maintains this accuracy when trained with only three trials of gestures; it also demonstrates comparable accuracy with the state-of-the-art when trained with one trial.
Multimodal Sensor-Based Semantic 3D Mapping for a Large-Scale Environment
Venue: arXiv cs.RO 2018
Year: 2018

Semantic 3D mapping is one of the most important fields in robotics, and has been used in many applications, such as robot navigation, surveillance, and virtual reality. In general, semantic 3D mapping is mainly composed of 3D reconstruction and semantic segmentation. As these technologies evolve, there has been great progress in semantic 3D mapping in recent years. Furthermore, the number of robotic applications requiring semantic information in 3D mapping to perform high-level tasks has increased, and many studies on semantic 3D mapping have been published. Existing methods use a camera for both 3D reconstruction and semantic segmentation. However, this is not suitable for large-scale environments and has the disadvantage of high computational complexity. To address this problem, we propose a multimodal sensor-based semantic 3D mapping system using a 3D Lidar combined with a camera. In this study, we build a 3D map by estimating odometry based on a global positioning system (GPS) and an inertial measurement unit (IMU), and use the latest 2D convolutional neural network (CNN) for semantic segmentation. To build a semantic 3D map, we integrate the 3D map with semantic information by using coordinate transformation and Bayes' update scheme. In order to improve the semantic 3D map, we propose a 3D refinement process to correct wrongly segmented voxels and remove traces of moving vehicles in the 3D map. Through experiments on challenging sequences, we demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and intersection over union (IoU). Thus, our method can be used for various applications that require semantic information in 3D map.
A Model for Medical Diagnosis Based on Plantar Pressure
Venue: arXiv cs.CV 2018
Year: 2018

The process of determining which disease or condition explains a person's symptoms and signs can be very complicated and may be inaccurate in some cases. The general belief is that diagnosing diseases relies on doctors' keen intuition, rich experience and professional equipment. In this work, we employ ideas from recent advances in plantar pressure research and from the powerful capacity of the convolutional neural network for learning representations. Here, we propose a model using convolutional neural network based on plantar pressure for medical diagnosis. Our model learns a network that maps plantar pressure data to its corresponding medical diagnostic label. We then apply our model to make the medical diagnosis on datasets we collected from cooperative hospital and achieve an accuracy of 98.36%. We demonstrate that the model base on the convolutional neural network is competitive in medical diagnosis.