OpenRaiser/Intern-Atlas huggingface dataset:
When Machine Learning Meets Wireless Cellular Networks: Deployment, Challenges, and Applications
Venue: arXiv cs.IT 2020
Year: 2020

Artificial intelligence (AI) powered wireless networks promise to revolutionize the conventional operation and structure of current networks from network design to infrastructure management, cost reduction, and user performance improvement. Empowering future networks with AI functionalities will enable a shift from reactive/incident driven operations to proactive/data-driven operations. This paper provides an overview on the integration of AI functionalities in 5G and beyond networks. Key factors for successful AI integration such as data, security, and explainable AI are highlighted. We also summarize the various types of network intelligence as well as machine learning based air interface in future networks. Use case examples for the application of AI to the wireless domain are then summarized. We highlight on applications to the physical layer, mobility management, wireless security, and localization.
ConveRT: Efficient and Accurate Conversational Representations from Transformers
Venue: arXiv cs.CL 2020
Year: 2020

General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.
Adaptive versus Standard Descent Methods and Robustness Against Adversarial Examples
Venue: arXiv cs.LG 2020
Year: 2020

Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. In this paper we study how the choice of optimization algorithm influences the robustness of the resulting classifier to adversarial examples. Specifically we show an example of a learning problem for which the solution found by adaptive optimization algorithms exhibits qualitatively worse robustness properties against both $L_{2}$- and $L_{\infty}$-adversaries than the solution found by non-adaptive algorithms. Then we fully characterize the geometry of the loss landscape of $L_{2}$-adversarial training in least-squares linear regression. The geometry of the loss landscape is subtle and has important consequences for optimization algorithms. Finally we provide experimental evidence which suggests that non-adaptive methods consistently produce more robust models than adaptive methods.
Location Attention for Extrapolation to Longer Sequences
Venue: arXiv cs.LG 2020
Year: 2020

Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set. However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple. In this paper, we first review the notion of extrapolation, why it is important and how one could hope to tackle it. We then focus on a specific type of extrapolation which is especially useful for natural language processing: generalization to sequences that are longer than the training ones. We hypothesize that models with a separate content- and location-based attention are more likely to extrapolate than those with common attention mechanisms. We empirically support our claim for recurrent seq2seq models with our proposed attention on variants of the Lookup Table task. This sheds light on some striking failures of neural models for sequences and on possible methods to approaching such issues.
A generic imperative language for polynomial time
Venue: arXiv cs.LO 2020
Year: 2020

The ramification method in Implicit Computational Complexity has been associated with functional programming, but adapting it to generic imperative programming is highly desirable, given the wider algorithmic applicability of imperative programming. We introduce a new approach to ramification which, among other benefits, adapts readily to fully general imperative programming. The novelty is in ramifying finite second-order objects, namely finite structures, rather than ramifying elements of free algebras. In so doing we bridge between Implicit Complexity's type theoretic characterizations of feasibility, and the data-flow approach of Static Analysis.
Asymptotically Optimal Sampling-based Planners
Venue: arXiv cs.RO 2020
Year: 2020

An asymptotically optimal sampling-based planner employs sampling to solve robot motion planning problems and returns paths with a cost that converges to the optimal solution cost, as the number of samples approaches infinity. This comprehensive article covers the theoretical characteristics of asymptotic optimality of motion planning algorithms, and traces its origins, analysis models, practical performance, extensions, and applications.
Compositional Hierarchical Tensor Factorization: Representing Hierarchical Intrinsic and Extrinsic Causal Factors
Venue: arXiv cs.CV 2020
Year: 2020

Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose properties, such as shape, reflectance, and color, constitute a hierarchy of intrinsic causal factors of object appearance. However, object appearance is the compositional consequence of both an object's intrinsic and extrinsic causal factors, where the extrinsic causal factors are related to illumination, and imaging conditions. Therefore, this paper proposes a unified tensor model of wholes and parts, and introduces a compositional hierarchical tensor factorization that disentangles the hierarchical causal structure of object image formation, and subsumes multilinear block tensor decomposition as a special case. The resulting object representation is an interpretable combinatorial choice of wholes' and parts' representations that renders object recognition robust to occlusion and reduces training data requirements. We demonstrate ourapproach in the context of face recognition by training on an extremely reduced dataset of synthetic images, and report encouragingface verification results on two datasets - the Freiburg dataset, andthe Labeled Face in the Wild (LFW) dataset consisting of real world images, thus, substantiating the suitability of our approach for data starved domains.
Systematic Comparison of the Influence of Different Data Preprocessing Methods on the Performance of Gait Classifications Using Machine Learning
Venue: arXiv cs.LG 2020
Year: 2020

Human movements are characterized by highly non-linear and multi-dimensional interactions within the motor system. Recently, an increasing emphasis on machine-learning applications has led to a significant contribution to the field of gait analysis, e.g., in increasing the classification performance. In order to ensure the generalizability of the machine-learning models, different data preprocessing steps are usually carried out to process the measured raw data before the classifications. In the past, various methods have been used for each of these preprocessing steps. However, there are hardly any standard procedures or rather systematic comparisons of these different methods and their impact on the classification performance. Therefore, the aim of this analysis is to compare different combinations of commonly applied data preprocessing steps and test their effects on the classification performance of gait patterns. A publicly available dataset on intra-individual changes of gait patterns was used for this analysis. Forty-two healthy participants performed 6 sessions of 15 gait trials for 1 day. For each trial, two force plates recorded the 3D ground reaction forces (GRFs). The data was preprocessed with the following steps: GRF filtering, time derivative, time normalization, data reduction, weight normalization and data scaling. Subsequently, combinations of all methods from each preprocessing step were analyzed by comparing their prediction performance in a six-session classification using Support Vector Machines, Random Forest Classifiers, Multi-Layer Perceptrons, and Convolutional Neural Networks. In conclusion, the present results provide first domain-specific recommendations for commonly applied data preprocessing methods and might help to build more comparable and more robust classification models based on machine learning that are suitable for a practical application.
Driving Reinforcement Learning with Models
Venue: arXiv cs.AI 2020
Year: 2020

In this paper we propose a new approach to complement reinforcement learning (RL) with model-based control (in particular, Model Predictive Control - MPC). We introduce an algorithm, the MPC augmented RL (MPRL) that combines RL and MPC in a novel way so that they can augment each other's strengths. We demonstrate the effectiveness of the MPRL by letting it play against the Atari game Pong. For this task, the results highlight how MPRL is able to outperform both RL and MPC when these are used individually.
XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
Venue: arXiv cs.LG 2020
Year: 2020

We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training. XPipe is designed to use multiple GPUs to concurrently and continuously train different parts of a DNN model. To improve GPU utilization and achieve high throughput, it splits a mini-batch into a set of micro-batches. It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches. Most importantly, the novel weight prediction strategy adopted by XPipe enables it to effectively address the weight inconsistency and staleness issues incurred by the asynchronous pipeline parallelism. As a result, XPipe incorporates the advantages of both synchronous and asynchronous pipeline model parallelism approaches. Concretely, it can achieve very comparable (even slightly better) model accuracy as its synchronous counterpart while obtaining higher throughput than it. Experimental results show that XPipe outperforms other state-of-the-art synchronous and asynchronous model parallelism approaches.
Model-Augmented Estimation of Conditional Mutual Information for Feature Selection
Venue: arXiv cs.LG 2020
Year: 2020

Markov blanket feature selection, while theoretically optimal, is generally challenging to implement. This is due to the shortcomings of existing approaches to conditional independence (CI) testing, which tend to struggle either with the curse of dimensionality or computational complexity. We propose a novel two-step approach which facilitates Markov blanket feature selection in high dimensions. First, neural networks are used to map features to low-dimensional representations. In the second step, CI testing is performed by applying the $k$-NN conditional mutual information estimator to the learned feature maps. The mappings are designed to ensure that mapped samples both preserve information and share similar information about the target variable if and only if they are close in Euclidean distance. We show that these properties boost the performance of the $k$-NN estimator in the second step. The performance of the proposed method is evaluated on both synthetic and real data.
Throughput Prediction of Asynchronous SGD in TensorFlow
Venue: arXiv cs.DC 2020
Year: 2020

Modern machine learning frameworks can train neural networks using multiple nodes in parallel, each computing parameter updates with stochastic gradient descent (SGD) and sharing them asynchronously through a central parameter server. Due to communication overhead and bottlenecks, the total throughput of SGD updates in a cluster scales sublinearly, saturating as the number of nodes increases. In this paper, we present a solution to predicting training throughput from profiling traces collected from a single-node configuration. Our approach is able to model the interaction of multiple nodes and the scheduling of concurrent transmissions between the parameter server and each node. By accounting for the dependencies between received parts and pending computations, we predict overlaps between computation and communication and generate synthetic execution traces for configurations with multiple nodes. We validate our approach on TensorFlow training jobs for popular image classification neural networks, on AWS and on our in-house cluster, using nodes equipped with GPUs or only with CPUs. We also investigate the effects of data transmission policies used in TensorFlow and the accuracy of our approach when combined with optimizations of the transmission schedule.
CALPA-NET: Channel-pruning-assisted Deep Residual Network for Steganalysis of Digital Images
Venue: arXiv cs.MM 2020
Year: 2020

Over the past few years, detection performance improvements of deep-learning based steganalyzers have been usually achieved through structure expansion. However, excessive expanded structure results in huge computational cost, storage overheads, and consequently difficulty in training and deployment. In this paper we propose CALPA-NET, a ChAnneL-Pruning-Assisted deep residual network architecture search approach to shrink the network structure of existing vast, over-parameterized deep-learning based steganalyzers. We observe that the broad inverted-pyramid structure of existing deep-learning based steganalyzers might contradict the well-established model diversity oriented philosophy, and therefore is not suitable for steganalysis. Then a hybrid criterion combined with two network pruning schemes is introduced to adaptively shrink every involved convolutional layer in a data-driven manner. The resulting network architecture presents a slender bottleneck-like structure. We have conducted extensive experiments on BOSSBase+BOWS2 dataset, more diverse ALASKA dataset and even a large-scale subset extracted from ImageNet CLS-LOC dataset. The experimental results show that the model structure generated by our proposed CALPA-NET can achieve comparative performance with less than two percent of parameters and about one third FLOPs compared to the original steganalytic model. The new model possesses even better adaptivity, transferability, and scalability.
On uniform boundedness of sequential social learning
Venue: arXiv cs.GT 2020
Year: 2020

In the classical herding model, asymptotic learning refers to situations where individuals eventually take the correct action regardless of their private information. Classical results identify classes of information structures for which such learning occurs. Recent papers have argued that typically, even when asymptotic learning occurs, it takes a very long time. In this paper related questions are referred. The paper studies whether there is a natural family of information structures for which the time it takes until individuals learn is uniformly bounded from above. Indeed, we propose a simple bi-parametric criterion that defines the information structure, and on top of that compute the time by which individuals learn (with high probability) for any pair of parameters. Namely, we identify a family of information structures where individuals learn uniformly fast. The underlying technical tool we deploy is a uniform convergence result on a newly introduced class of `weakly active' supermartingales. This result extends an earlier result of Fudenberg and Levine (1992) on active supermartingales.
Mixed-Initiative variable autonomy for remotely operated mobile robots
Venue: arXiv cs.RO 2020
Year: 2020

This paper presents an Expert-guided Mixed-Initiative Control Switcher (EMICS) for remotely operated mobile robots. The EMICS enables switching between different levels of autonomy during task execution initiated by either the human operator and/or the EMICS. The EMICS is evaluated in two disaster response inspired experiments, one with a simulated robot and test arena, and one with a real robot in a realistic environment. Analyses from the two experiments provide evidence that: a) Human-Initiative (HI) systems outperform systems with single modes of operation, such as pure teleoperation, in navigation tasks; b) in the context of the simulated robot experiment, Mixed-Initiative (MI) systems provide improved performance in navigation tasks, improved operator performance in cognitive demanding secondary tasks, and improved operator workload compared to HI. Results also reinforce previous human-robot interaction evidence regarding the importance of the operator's personality traits and their trust in the autonomous system. Lastly, our experiment on a physical robot provides empirical evidence that identify two major challenges for MI control: a) the design of context-aware MI control systems; and b) the conflict for control between the robot's MI control system and the operator. Insights regarding these challenges are discussed and ways to tackle them are proposed.
Deep Multi-task Prediction of Lung Cancer and Cancer-free Progression from Censored Heterogenous Clinical Imaging
Venue: arXiv cs.CV 2020
Year: 2020

Annual low dose computed tomography (CT) lung screening is currently advised for individuals at high risk of lung cancer (e.g., heavy smokers between 55 and 80 years old). The recommended screening practice significantly reduces all-cause mortality, but the vast majority of screening results are negative for cancer. If patients at very low risk could be identified based on individualized, image-based biomarkers, the health care resources could be more efficiently allocated to higher risk patients and reduce overall exposure to ionizing radiation. In this work, we propose a multi-task (diagnosis and prognosis) deep convolutional neural network to improve the diagnostic accuracy over a baseline model while simultaneously estimating a personalized cancer-free progression time (CFPT). A novel Censored Regression Loss (CRL) is proposed to perform weakly supervised regression so that even single negative screening scans can provide small incremental value. Herein, we study 2287 scans from 1433 de-identified patients from the Vanderbilt Lung Screening Program (VLSP) and Molecular Characterization Laboratories (MCL) cohorts. Using five-fold cross-validation, we train a 3D attention-based network under two scenarios: (1) single-task learning with only classification, and (2) multi-task learning with both classification and regression. The single-task learning leads to a higher AUC compared with the Kaggle challenge winner pre-trained model (0.878 v. 0.856), and multi-task learning significantly improves the single-task one (AUC 0.895, p<0.01, McNemar test). In summary, the image-based predicted CFPT can be used in follow-up year lung cancer prediction and data assessment.
On the Strength of Connectivity of Inhomogeneous Random K-out Graphs
Venue: arXiv cs.IT 2020
Year: 2020

Random graphs are an important tool for modelling and analyzing the underlying properties of complex real-world networks. In this paper, we study a class of random graphs known as the inhomogeneous random K-out graphs which were recently introduced to analyze heterogeneous sensor networks secured by the pairwise scheme. In this model, first, each of the $n$ nodes is classified as type-1 (respectively, type-2) with probability $0<μ<1$ (respectively, $1-μ)$ independently from each other. Next, each type-1 (respectively, type-2) node draws 1 arc towards a node (respectively, $K_n$ arcs towards $K_n$ distinct nodes) selected uniformly at random, and then the orientation of the arcs is ignored. From the literature on homogeneous K-out graphs wherein all nodes select $K_n$ neighbors (i.e., $μ=0$), it is known that when $K_n \geq2$, the graph is $K_n$-connected asymptotically almost surely (a.a.s.) as $n$ gets large. In the inhomogeneous case (i.e., $μ>0$), it was recently established that achieving even 1-connectivity a.a.s. requires $K_n=ω(1)$. Here, we provide a comprehensive set of results to complement these existing results. First, we establish a sharp zero-one law for $k$-connectivity, showing that for the network to be $k$-connected a.a.s., we need to set $K_n = \frac{1}{1-μ}(\log n +(k-2)\log\log n + ω(1))$ for all $k=2, 3, \ldots$. Despite such large scaling of $K_n$ being required for $k$-connectivity, we show that the trivial condition of $K_n \geq 2$ for all $n$ is sufficient to ensure that inhomogeneous K-out graph has a connected component of size $n-O(1)$ whp.
A Polynomial Kernel for Funnel Arc Deletion Set
Venue: arXiv cs.DS 2020
Year: 2020

In Directed Feeback Arc Set (DFAS) we search for a set of at most $k$ arcs which intersect every cycle in the input digraph. It is a well-known open problem in parameterized complexity to decide if DFAS admits a kernel of polynomial size. We consider $\mathcal{C}$-Arc Deletion Set ($\mathcal{C}$-ADS), a variant of DFAS where we want to remove at most $k$ arcs from the input digraph in order to turn it into a digraph of a class $\mathcal{C}$. In this work, we choose $\mathcal{C}$ to be the class of funnels. Funnel-Arc Deletion Set is NP-hard even if the input is a DAG, but is fixed-parameter tractable with respect to $k$. So far no polynomial kernels for this problem were known. Our main result is a kernel for Funnel-Arc Deletion Set with $\mathcal{O}(k^6)$ many vertices and $\mathcal{O}(k^7)$ many arcs, computable in $\mathcal{O}(nm)$ time, where $n$ is the number of vertices and $m$ the number of arcs in the input digraph.
Neural Network Processing Neural Networks: An efficient way to learn higher order functions
Venue: arXiv cs.LG 2020
Year: 2020

Functions are rich in meaning and can be interpreted in a variety of ways. Neural networks were proven to be capable of approximating a large class of functions[1]. In this paper, we propose a new class of neural networks called "Neural Network Processing Neural Networks" (NNPNNs), which inputs neural networks and numerical values, instead of just numerical values. Thus enabling neural networks to represent and process rich structures.
Communication Lower Bound in Convolution Accelerators
Venue: arXiv cs.DC 2020
Year: 2020

In current convolutional neural network (CNN) accelerators, communication (i.e., memory access) dominates the energy consumption. This work provides comprehensive analysis and methodologies to minimize the communication for CNN accelerators. For the off-chip communication, we derive the theoretical lower bound for any convolutional layer and propose a dataflow to reach the lower bound. This fundamental problem has never been solved by prior studies. The on-chip communication is minimized based on an elaborate workload and storage mapping scheme. We in addition design a communication-optimal CNN accelerator architecture. Evaluations based on the 65nm technology demonstrate that the proposed architecture nearly reaches the theoretical minimum communication in a three-level memory hierarchy and it is computation dominant. The gap between the energy efficiency of our accelerator and the theoretical best value is only 37-87%.