OpenRaiser/Intern-Atlas huggingface dataset:
Generative Restricted Kernel Machines: A Framework for Multi-view Generation and Disentangled Feature Learning
Venue: arXiv cs.LG 2020
Year: 2020

This paper introduces a novel framework for generative models based on Restricted Kernel Machines (RKMs) with joint multi-view generation and uncorrelated feature learning, called Gen-RKM. To enable joint multi-view generation, this mechanism uses a shared representation of data from various views. Furthermore, the model has a primal and dual formulation to incorporate both kernel-based and (deep convolutional) neural network based models within the same setting. When using neural networks as explicit feature-maps, a novel training procedure is proposed, which jointly learns the features and shared subspace representation. The latent variables are given by the eigen-decomposition of the kernel matrix, where the mutual orthogonality of eigenvectors represents the learned uncorrelated features. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of generated samples on various standard datasets.
Control What You Can: Intrinsically Motivated Task-Planning Agent
Venue: arXiv cs.LG 2020
Year: 2020

We present a novel intrinsically motivated agent that learns how to control the environment in the fastest possible manner by optimizing learning progress. It learns what can be controlled, how to allocate time and attention, and the relations between objects using surprise based motivation. The effectiveness of our method is demonstrated in a synthetic as well as a robotic manipulation environment yielding considerably improved performance and smaller sample complexity. In a nutshell, our work combines several task-level planning agent structures (backtracking search on task graph, probabilistic road-maps, allocation of search efforts) with intrinsic motivation to achieve learning from scratch.
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Venue: arXiv cs.CL 2020
Year: 2020

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
A Survey of Multi-Access Edge Computing in 5G and Beyond: Fundamentals, Technology Integration, and State-of-the-Art
Venue: arXiv cs.NI 2020
Year: 2020

Driven by the emergence of new compute-intensive applications and the vision of the Internet of Things (IoT), it is foreseen that the emerging 5G network will face an unprecedented increase in traffic volume and computation demands. However, end users mostly have limited storage capacities and finite processing capabilities, thus how to run compute-intensive applications on resource-constrained users has recently become a natural concern. Mobile edge computing (MEC), a key technology in the emerging fifth generation (5G) network, can optimize mobile resources by hosting compute-intensive applications, process large data before sending to the cloud, provide the cloud computing capabilities within the radio access network (RAN) in close proximity to mobile users, and offer context-aware services with the help of RAN information. Therefore, MEC enables a wide variety of applications, where the real-time response is strictly required, e.g., driverless vehicles, augmented reality, robotics, and immerse media. Indeed, the paradigm shift from 4G to 5G could become a reality with the advent of new technological concepts. The successful realization of MEC in the 5G network is still in its infancy and demands for constant efforts from both academic and industry communities. In this survey, we first provide a holistic overview of MEC technology and its potential use cases and applications. Then, we outline up-to-date researches on the integration of MEC with the new technologies that will be deployed in 5G and beyond. We also summarize testbeds and experimental evaluations, and open source activities, for edge computing. We further summarize lessons learned from state-of-the-art research works as well as discuss challenges and potential future directions for MEC research.
Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors
Venue: arXiv cs.CV 2020
Year: 2020

Deep neural networks as image priors have been recently introduced for problems such as denoising, super-resolution and inpainting with promising performance gains over hand-crafted image priors such as sparsity and low-rank. Unlike learned generative priors they do not require any training over large datasets. However, few theoretical guarantees exist in the scope of using untrained neural network priors for inverse imaging problems. We explore new applications and theory for untrained neural network priors. Specifically, we consider the problem of solving linear inverse problems, such as compressive sensing, as well as non-linear problems, such as compressive phase retrieval. We model images to lie in the range of an untrained deep generative network with a fixed seed. We further present a projected gradient descent scheme that can be used for both compressive sensing and phase retrieval and provide rigorous theoretical guarantees for its convergence. We also show both theoretically as well as empirically that with deep network priors, one can achieve better compression rates for the same image quality compared to hand crafted priors.
Understanding More about Human and Machine Attention in Deep Neural Networks
Venue: arXiv cs.CV 2020
Year: 2020

Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known as Human attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is called Machine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks, diverse representative backbones, and famous architectures, corresponding real human gaze data, and systematically conducted large-scale quantitative studies, we quantify the consistency between artificial attention and human visual attention and offer novel insights into existing artificial attention mechanisms by giving preliminary answers to several key questions related to human and artificial attention mechanisms. Overall results demonstrate that human attention can benchmark the meaningful `ground-truth' in attention-driven tasks, where the more the artificial attention is close to human attention, the better the performance; for higher-level vision tasks, it is case-by-case. It would be advisable for attention-driven tasks to explicitly force a better alignment between artificial and human attention to boost the performance; such alignment would also improve the network explainability for higher-level computer vision tasks.
Variance Reduction for Evolution Strategies via Structured Control Variates
Venue: arXiv cs.NE 2020
Year: 2020

Evolution Strategies (ES) are a powerful class of blackbox optimization techniques that recently became a competitive alternative to state-of-the-art policy gradient (PG) algorithms for reinforcement learning (RL). We propose a new method for improving accuracy of the ES algorithms, that as opposed to recent approaches utilizing only Monte Carlo structure of the gradient estimator, takes advantage of the underlying MDP structure to reduce the variance. We observe that the gradient estimator of the ES objective can be alternatively computed using reparametrization and PG estimators, which leads to new control variate techniques for gradient estimation in ES optimization. We provide theoretical insights and show through extensive experiments that this RL-specific variance reduction approach outperforms general purpose variance reduction methods.
Learning as the Unsupervised Alignment of Conceptual Systems
Venue: arXiv cs.LG 2020
Year: 2020

Concept induction requires the extraction and naming of concepts from noisy perceptual experience. For supervised approaches, as the number of concepts grows, so does the number of required training examples. Philosophers, psychologists, and computer scientists, have long recognized that children can learn to label objects without being explicitly taught. In a series of computational experiments, we highlight how information in the environment can be used to build and align conceptual systems. Unlike supervised learning, the learning problem becomes easier the more concepts and systems there are to master. The key insight is that each concept has a unique signature within one conceptual system (e.g., images) that is recapitulated in other systems (e.g., text or audio). As predicted, children's early concepts form readily aligned systems.
Unsupervised Ensemble Classification with Sequential and Networked Data
Venue: arXiv cs.LG 2020
Year: 2020

Ensemble learning, the machine learning paradigm where multiple algorithms are combined, has exhibited promising perfomance in a variety of tasks. The present work focuses on unsupervised ensemble classification. The term unsupervised refers to the ensemble combiner who has no knowledge of the ground-truth labels that each classifier has been trained on. While most prior works on unsupervised ensemble classification are designed for independent and identically distributed (i.i.d.) data, the present work introduces an unsupervised scheme for learning from ensembles of classifiers in the presence of data dependencies. Two types of data dependencies are considered: sequential data and networked data whose dependencies are captured by a graph. Moment matching and Expectation Maximization algorithms are developed for the aforementioned cases, and their performance is evaluated on synthetic and real datasets.
Deep learning approach to description and classification of fungi microscopic images
Venue: arXiv cs.CV 2020
Year: 2020

Diagnosis of fungal infections can rely on microscopic examination, however, in many cases, it does not allow unambiguous identification of the species due to their visual similarity. Therefore, it is usually necessary to use additional biochemical tests. That involves additional costs and extends the identification process up to 10 days. Such a delay in the implementation of targeted treatment is grave in consequences as the mortality rate for immunosuppressed patients is high. In this paper, we apply machine learning approach based on deep learning and bag-of-words to classify microscopic images of various fungi species. Our approach makes the last stage of biochemical identification redundant, shortening the identification process by 2-3 days and reducing the cost of the diagnostic examination.
Learning Activation Functions: A new paradigm for understanding Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

The scope of research in the domain of activation functions remains limited and centered around improving the ease of optimization or generalization quality of neural networks (NNs). However, to develop a deeper understanding of deep learning, it becomes important to look at the non linear component of NNs more carefully. In this paper, we aim to provide a generic form of activation function along with appropriate mathematical grounding so as to allow for insights into the working of NNs in future. We propose "Self-Learnable Activation Functions" (SLAF), which are learned during training and are capable of approximating most of the existing activation functions. SLAF is given as a weighted sum of pre-defined basis elements which can serve for a good approximation of the optimal activation function. The coefficients for these basis elements allow a search in the entire space of continuous functions (consisting of all the conventional activations). We propose various training routines which can be used to achieve performance with SLAF equipped neural networks (SLNNs). We prove that SLNNs can approximate any neural network with lipschitz continuous activations, to any arbitrary error highlighting their capacity and possible equivalence with standard NNs. Also, SLNNs can be completely represented as a collections of finite degree polynomial upto the very last layer obviating several hyper parameters like width and depth. Since the optimization of SLNNs is still a challenge, we show that using SLAF along with standard activations (like ReLU) can provide performance improvements with only a small increase in number of parameters.
On List Decoding of Insertion and Deletion Errors
Venue: arXiv cs.IT 2020
Year: 2020

Insdel errors occur in communication systems caused by the loss of positional information of the message. Since the work by Guruswami and Wang, there have been some further investigations on the list decoding of insertion codes, deletion codes and insdel codes. However, unlike classical Hamming metric or even rank-metric, there are still many unsolved problems on list decoding of insdel codes. The contributions of this paper mainly consist of two parts. Firstly, we analyze the list decodability of random insdel codes. We show that list decoding of random insdel codes surpasses the Singleton bound when there are more insertion errors than deletion errors and the alphabet size is sufficiently large. Furthermore, our results reveal the existence of an insdel code that can be list decoded against insdel errors beyond its minimum insdel distance while still having polynomial list size. This provides a more complete picture on the list decodability of insdel codes when both insertion and deletion errors happen. Secondly, we construct a family of explicit insdel codes with efficient list decoding algorithm. As a result, we derive a Zyablov-type bound for insdel errors. Recently, after our results appeared, Guruswami et al. provided a complete solution for another open problem on list decoding of insdel codes. In contrast to the problems we considered, they provided a region containing all possible insertion and deletion errors that are still list decodable by some q-ary insdel codes of non-zero rate. More specifically, for a fixed number of insertion and deletion errors, while our paper focuses on maximizing the rate of a code that is list decodable against that amount of insertion and deletion errors, Guruswami et al. focuses on finding out the existence of a code with asymptotically non-zero rate which is list decodable against this amount of insertion and deletion errors.
Using Human Ratings for Feedback Control: A Supervised Learning Approach with Application to Rehabilitation Robotics
Venue: arXiv cs.RO 2020
Year: 2020

This paper presents a method for tailoring a parametric controller based on human ratings. The method leverages supervised learning concepts in order to train a reward model from data. It is applied to a gait rehabilitation robot with the goal of teaching the robot how to walk patients physiologically. In this context, the reward model judges the physiology of the gait cycle (instead of therapists) using sensor measurements provided by the robot and the automatic feedback controller chooses the input settings of the robot to maximize the reward. The key advantage of the proposed method is that only a few input adaptations are necessary to achieve a physiological gait cycle. Experiments with nondisabled subjects show that the proposed method permits the incorporation of human expertise into a control law and to automatically walk patients physiologically.
Channel-by-Channel Demosaicking Networks with Embedded Spectral Correlation
Venue: arXiv cs.CV 2020
Year: 2020

Demosaicking is standardly the first step in today's Image Signal Processing (ISP) pipeline of digital cameras. It reconstructs image RGB values from the spatially and spectrally sparse Color Filter Array (CFA) samples, which are the original raw data digitized from electrical signals. High quality and low cost demosaicking is not only necessary for photography, but also fundamental for many machine vision tasks. This paper proposes an accurate and fast demosaicking model based on Convolutional Neural Networks (CNN) for the Bayer CFA, which is the most popular color filter arrangement adopted by digital camera manufacturers. Observing that each channel has different estimation complexity, we reconstruct each channel by an individual sub-network. Moreover, instead of directly estimating the red and blue values, our model infers the green-red and green-blue color difference. This strategy allows recovering the most complex channel by a light weight network. Although the total size of our model is significantly smaller than the state of the art demosaicking networks, it achieves substantially higher performance in both demosaicking quality and computational cost, as validated by extensive experiments. Source code will be released along with paper publication.
Most Important Fundamental Rule of Poker Strategy
Venue: arXiv cs.AI 2020
Year: 2020

Poker is a large complex game of imperfect information, which has been singled out as a major AI challenge problem. Recently there has been a series of breakthroughs culminating in agents that have successfully defeated the strongest human players in two-player no-limit Texas hold 'em. The strongest agents are based on algorithms for approximating Nash equilibrium strategies, which are stored in massive binary files and unintelligible to humans. A recent line of research has explored approaches for extrapolating knowledge from strong game-theoretic strategies that can be understood by humans. This would be useful when humans are the ultimate decision maker and allow humans to make better decisions from massive algorithmically-generated strategies. Using techniques from machine learning we have uncovered a new simple, fundamental rule of poker strategy that leads to a significant improvement in performance over the best prior rule and can also easily be applied by human players.
Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation
Venue: arXiv cs.LG 2020
Year: 2020

Recent work in variational inference (VI) uses ideas from Monte Carlo estimation to tighten the lower bounds on the log-likelihood that are used as objectives. However, there is no systematic understanding of how optimizing different objectives relates to approximating the posterior distribution. Developing such a connection is important if the ideas are to be applied to inference-i.e., applications that require an approximate posterior and not just an approximation of the log-likelihood. Given a VI objective defined by a Monte Carlo estimator of the likelihood, we use a "divide and couple" procedure to identify augmented proposal and target distributions. The divergence between these is equal to the gap between the VI objective and the log-likelihood. Thus, after maximizing the VI objective, the augmented variational distribution may be used to approximate the posterior distribution.
A laser-microfabricated electrohydrodynamic thruster for centimeter-scale aerial robots
Venue: arXiv cs.RO 2020
Year: 2020

To date, insect scale robots capable of controlled flight have used flapping wings for generating lift, but this requires a complex and failure-prone mechanism. A simpler alternative is electrohydrodynamic (EHD) thrust, which requires no moving mechanical parts. In EHD, corona discharge generates a flow of ions in an electric field between two electrodes; the high-velocity ions transfer their kinetic energy to neutral air molecules through collisions, accelerating the gas and creating thrust. We introduce a fabrication process for EHD thruster based on 355 nm laser micromachining and our approach allows for greater flexibility in materials selection. Our four-thruster device measures 1.8 x 2.5 cm and is composed of steel emitters and a lightweight carbon fiber mesh. The current and thrust characteristics of each individual thruster of the quad thruster is determined and agrees with Townsend relation. The mass of the quad thruster is 37 mg and the measured thrust is greater than its weight (362.6 uN). The robot is able to lift off at a voltage of 4.6 kV with a thrust to weight ratio of 1.38.
Compound Probabilistic Context-Free Grammars for Grammar Induction
Venue: arXiv cs.CL 2020
Year: 2020

We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context-free grammar. In contrast to traditional formulations which learn a single stochastic grammar, our grammar's rule probabilities are modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Inference in this grammar is performed by collapsed variational inference, in which an amortized variational posterior is placed on the continuous variable, and the latent trees are marginalized out with dynamic programming. Experiments on English and Chinese show the effectiveness of our approach compared to recent state-of-the-art methods when evaluated on unsupervised parsing.
Learning Fair Representations for Kernel Models
Venue: arXiv cs.LG 2020
Year: 2020

Fair representations are a powerful tool for establishing criteria like statistical parity, proxy non-discrimination, and equality of opportunity in learned models. Existing techniques for learning these representations are typically model-agnostic, as they preprocess the original data such that the output satisfies some fairness criterion, and can be used with arbitrary learning methods. In contrast, we demonstrate the promise of learning a model-aware fair representation, focusing on kernel-based models. We leverage the classical Sufficient Dimension Reduction (SDR) framework to construct representations as subspaces of the reproducing kernel Hilbert space (RKHS), whose member functions are guaranteed to satisfy fairness. Our method supports several fairness criteria, continuous and discrete data, and multiple protected attributes. We further show how to calibrate the accuracy tradeoff by characterizing it in terms of the principal angles between subspaces of the RKHS. Finally, we apply our approach to obtain the first Fair Gaussian Process (FGP) prior for fair Bayesian learning, and show that it is competitive with, and in some cases outperforms, state-of-the-art methods on real data.