huashen218/convxai-cia-dataset huggingface dataset:
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
Keywords: Ternary, 4-bits, low precision, knowledge distillation, knowledge transfer, model compression
TL;DR: We show that knowledge transfer techniques can improve the accuracy of low precision networks and set new state-of-the-art accuracy for ternary and 4-bits precision.

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems -- the models (often deep networks or wide networks or both) are compute and memory intensive. Low precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low precision networks can be significantly improved by using knowledge distillation techniques. We call our approach Apprentice and show state-of-the-art accuracies using ternary precision and 4-bit precision for many variants of ResNet architecture on ImageNet dataset. We study three schemes in which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.
Understanding Deep Neural Networks with Rectified Linear Units
Keywords: expressive power, benefits of depth, empirical risk minimization, global optimality, computational hardness, combinatorial optimization
TL;DR: This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.

In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to {\em global optimality} with runtime polynomial in the data size albeit exponential in the input dimension. Further, we improve on the known lower bounds on size (from exponential to super exponential) for approximating a ReLU deep net function by a shallower ReLU net. Our gap theorems hold for smoothly parametrized families of ``hard'' functions, contrary to countable, discrete families known in the literature.  An example consequence of our gap theorems is the following: for every natural number k there exists a function representable by a ReLU DNN with k2 hidden layers and total size k3, such that any ReLU DNN with at most k hidden layers will require at least 12kk+1−1 total nodes. Finally, for the family of \Rn→\R DNNs with ReLU activations, we show a new lowerbound on the number of affine pieces, which is larger than previous constructions in certain regimes of the network architecture and most distinctively our lowerbound is demonstrated by an explicit construction of a \emph{smoothly parameterized} family of functions attaining this scaling. Our construction utilizes the theory of zonotopes from polyhedral theory.
HexaConv
Keywords: hexagonal, group, symmetry, representation learning, rotation, equivariance, invariance
TL;DR: We introduce G-HexaConv, a group equivariant convolutional neural network on hexagonal lattices.

The effectiveness of Convolutional Neural Networks stems in large part from their ability to exploit the translation invariance that is inherent in many learning problems. Recently, it was shown that CNNs can exploit other invariances, such as rotation invariance, by using group convolutions instead of planar convolutions. However, for reasons of performance and ease of implementation, it has been necessary to limit the group convolution to transformations that can be applied to the filters without interpolation. Thus, for images with square pixels, only integer translations, rotations by multiples of 90 degrees, and reflections are admissible.
        
        Whereas the square tiling provides a 4-fold rotational symmetry, a hexagonal tiling of the plane has a 6-fold rotational symmetry. In this paper we show how one can efficiently implement planar convolution and group convolution over hexagonal lattices, by re-using existing highly optimized convolution routines. We find that, due to the reduced anisotropy of hexagonal filters, planar HexaConv provides better accuracy than planar convolution with square filters, given a fixed parameter budget. Furthermore, we find that the increased degree of symmetry of the hexagonal grid increases the effectiveness of group convolutions, by allowing for more parameter sharing. We show that our method significantly outperforms conventional CNNs on the AID aerial scene classification dataset, even outperforming ImageNet pre-trained models.
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
Keywords: Reinforcement learning
TL;DR: We extend recent insights related to softmax consistency to achieve state-of-the-art results in continuous control.

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL, which exploits an observation that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. The introduction of relative entropy regularization allows Trust-PCL to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL significantly improves the solution quality and sample efficiency of TRPO.
Generative networks as inverse problems with Scattering transforms
Keywords: Unsupervised Learning, Inverse Problems, Convolutional Networks, Generative Models, Scattering Transform
TL;DR: We introduce generative networks that do not require to be learned with a discriminator or an encoder; they are obtained by inverting a special embedding operator defined by a wavelet Scattering transform.

Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be optimized with a discriminator or an encoder. The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images. This embedding is computed with a wavelet Scattering transform. Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.
i-RevNet: Deep Invertible Networks

It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. 
        An analysis of i-RevNet’s learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.
Training GANs with Optimism
Keywords: GANs, Optimistic Mirror Decent, Cycling, Last Iterate Convergence, Optimistic Adam
TL;DR: We propose the use of optimistic mirror decent to address cycling problems in the training of GANs. We also introduce the Optimistic Adam algorithm

We address the issue of limit cycling behavior in training Generative Adversarial Networks and propose the use of Optimistic Mirror Decent (OMD) for training Wasserstein GANs. Recent theoretical results have shown that optimistic mirror decent (OMD) can enjoy faster regret rates in the context of zero-sum games. WGANs is exactly a context of solving a zero-sum game with simultaneous no-regret dynamics.  Moreover, we show that optimistic mirror decent addresses the limit cycling problem in training WGANs. We formally show that in the case of bi-linear zero-sum games the last iterate of OMD dynamics converges to an equilibrium, in contrast to GD dynamics which are bound to cycle. We also portray the huge qualitative difference between GD and OMD dynamics with toy examples, even when GD is modified with many adaptations proposed in the recent literature, such as gradient penalty or momentum. We apply OMD WGAN training to a bioinformatics problem of generating DNA sequences. We observe that models trained with OMD achieve consistently smaller KL divergence with respect to the true underlying distribution, than models trained with GD variants. Finally, we introduce a new algorithm, Optimistic Adam, which is an optimistic variant of Adam. We apply it to WGAN training on CIFAR10 and observe improved performance in terms of inception score as compared to Adam.
Don't Decay the Learning Rate, Increase the Batch Size
Keywords: batch size, learning rate, simulated annealing, large batch training, scaling rules, stochastic gradient descent, sgd, imagenet, optimization
TL;DR: Decaying the learning rate and increasing the batch size during training are equivalent.

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate ϵ and scaling the batch size B∝ϵ. Finally, one can increase the momentum coefficient m and scale B∝1/(1−m), although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
Keywords: Common-sense Physical Reasoning, Intuitive Physics, Representation Learning, Model building
TL;DR: We introduce a novel approach to common-sense physical reasoning that learns to discover objects and model their physical interactions from raw visual images in a purely unsupervised fashion

Common-sense physical reasoning is an essential ingredient for any intelligent agent operating in the real-world. For example, it can be used to simulate the environment, or to infer the state of parts of the world that are currently unobserved. In order to match real-world conditions this causal knowledge must be learned without access to supervised data. To address this problem we present a novel method that learns to discover objects and model their physical interactions from raw visual images in a purely unsupervised fashion. It incorporates prior knowledge about the compositional nature of human perception to factor interactions between object-pairs and learn efficiently. On videos of bouncing balls we show the superior modelling capabilities of our method compared to other unsupervised neural approaches that do not incorporate such prior knowledge. We demonstrate its ability to handle occlusion and show that it can extrapolate learned knowledge to scenes with different numbers of objects.
A Simple Neural Attentive Meta-Learner
Keywords: meta-learning, few-shot learning
TL;DR: a simple RNN-based meta-learner that achieves SOTA performance on popular benchmarks

Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on a distribution of similar tasks, in the hopes of generalization to novel but related tasks by learning a high-level strategy that captures the essence of the problem it is asked to solve. However, many recent meta-learning approaches are extensively hand-designed, either using architectures specialized to a particular application, or hard-coding algorithmic components that constrain how the meta-learner solves the task. We propose a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information.  In the most extensive set of meta-learning experiments to date, we evaluate the resulting Simple Neural AttentIve Learner (or SNAIL) on several heavily-benchmarked tasks.  On all tasks, in both supervised and reinforcement learning, SNAIL attains state-of-the-art performance by significant margins.
Critical Percolation as a Framework to Analyze the Training of Deep Networks
Keywords: Deep Convolutional Networks, Loss function landscape, Graph Structured Data, Training Complexity, Theory of deep learning, Percolation theory, Anderson Localization
TL;DR: A toy dataset based on critical percolation in a planar graph provides an analytical window to the training dynamics of deep neural networks

In this paper we approach two relevant deep learning topics: i) tackling of graph structured input data and ii) a better understanding and analysis of deep networks and related learning algorithms. With this in mind we focus on the topological classification of reachability in a particular subset of planar graphs (Mazes). Doing so, we are able to model the topology of data while staying in Euclidean space, thus allowing its processing with standard CNN architectures. We suggest a suitable architecture for this problem and show that it can express a perfect solution to the classification task. The shape of the cost function around this solution is also derived and, remarkably, does not depend on the size of the maze in the large maze limit. Responsible for this behavior are rare events in the dataset which strongly regulate the shape of the cost function near this global minimum. We further identify an obstacle to learning in the form of poorly performing local minima in which the network chooses to ignore some of the inputs. We further support our claims with training experiments and numerical analysis of the cost function on networks with up to 128 layers.
Attacking Binarized Neural Networks
Keywords: adversarial examples, adversarial attacks, binary, binarized neural networks
TL;DR: We conduct adversarial attacks against binarized neural networks and show that we reduce the impact of the strongest attacks, while maintaining comparable accuracy in a black-box setting

Neural networks with low-precision weights and activations offer compelling
        efficiency advantages over their full-precision equivalents. The two most
        frequently discussed benefits of quantization are reduced memory consumption,
        and a faster forward pass when implemented with efficient bitwise
        operations. We propose a third benefit of very low-precision neural networks:
        improved robustness against some adversarial attacks, and in the worst case,
        performance that is on par with full-precision models. We focus on the very
        low-precision case where weights and activations are both quantized to ±1,
        and note that stochastically quantizing weights in just one layer can sharply
        reduce the impact of iterative attacks. We observe that non-scaled binary neural
        networks exhibit a similar effect to the original \emph{defensive distillation}
        procedure that led to \emph{gradient masking}, and a false notion of security.
        We address this by conducting both black-box and white-box experiments with
        binary models that do not artificially mask gradients.
Simulated+Unsupervised Learning With Adaptive Data Generation and Bidirectional Mappings

Collecting a large dataset with high quality annotations is expensive and time-consuming. Recently, Shrivastava et al. (2017) propose Simulated+Unsupervised (S+U) learning: It first learns a mapping from synthetic data to real data, translates a large amount of labeled synthetic data to the ones that resemble real data, and then trains a learning model on the translated data. Bousmalis et al. (2017) propose a similar framework that jointly trains a translation mapping and a learning model. 
        While these algorithms are shown to achieve the state-of-the-art performances on various tasks, it may have a room for improvement, as they do not fully leverage flexibility of data simulation process and consider only the forward (synthetic to real) mapping. While these algorithms are shown to achieve the state-of-the-art performances on various tasks, it may have a room for improvement, as it does not fully leverage flexibility of data simulation process and consider only the forward (synthetic to real) mapping. Inspired by this limitation, we propose a new S+U learning algorithm, which fully leverage the flexibility  of  data  simulators and bidirectional mappings between synthetic data and real data. We show that our approach achieves the improved performance on the gaze estimation task, outperforming (Shrivastava et al., 2017).
Espresso: Efficient Forward Propagation for Binary Deep Neural Networks
Keywords: binary deep neural networks, optimized implementation, bitwise computations
TL;DR: state-of-the-art computational performance implementation of binary neural networks

There are many applications scenarios for which the computational
          performance and memory footprint of the prediction phase of Deep
          Neural Networks (DNNs) need to be optimized. Binary Deep Neural
          Networks (BDNNs) have been shown to be an effective way of achieving
          this objective. In this paper, we show how Convolutional Neural
          Networks (CNNs) can be implemented using binary
          representations. Espresso is a compact, yet powerful
          library written in C/CUDA that features all the functionalities
          required for the forward propagation of CNNs, in a binary file less
          than 400KB, without any external dependencies. Although it is mainly
          designed to take advantage of massive GPU parallelism, Espresso also
          provides an equivalent CPU implementation for CNNs. Espresso
          provides special convolutional and dense layers for BCNNs,
          leveraging bit-packing and bit-wise computations
          for efficient execution. These techniques provide a speed-up of
          matrix-multiplication routines, and at the same time, reduce memory
          usage when storing parameters and activations. We experimentally
          show that Espresso is significantly faster than existing
          implementations of optimized binary neural networks (~ 2
          orders of magnitude). Espresso is released under the Apache 2.0
          license and is available at http://github.com/organization/project.
Expressive power of recurrent neural networks
Keywords: Recurrent Neural Networks, Tensor Train, tensor decompositions, expressive power
TL;DR: We prove the exponential efficiency of recurrent-type neural networks over shallow networks.

Deep neural networks are surprisingly efficient at solving practical tasks,
        but the theory behind this phenomenon is only starting to catch up with
        the practice. Numerous works show that depth is the key to this efficiency.
        A certain class of deep convolutional networks – namely those that correspond
        to the Hierarchical Tucker (HT) tensor decomposition – has been
        proven to have exponentially higher expressive power than shallow networks.
        I.e. a shallow network of exponential width is required to realize
        the same score function as computed by the deep architecture. In this paper,
        we prove the expressive power theorem (an exponential lower bound on
        the width of the equivalent shallow network) for a class of recurrent neural
        networks – ones that correspond to the Tensor Train (TT) decomposition.
        This means that even processing an image patch by patch with an RNN
        can be exponentially more efficient than a (shallow) convolutional network
        with one hidden layer. Using theoretical results on the relation between
        the tensor decompositions we compare expressive powers of the HT- and
        TT-Networks. We also implement the recurrent TT-Networks and provide
        numerical evidence of their expressivity.
Memory Augmented Control Networks
Keywords: planning, memory networks, deep learning, robotics
TL;DR: Memory Augmented Network to plan in partially observable environments.

Planning problems in partially observable environments cannot be solved directly with convolutional networks and require some form of memory. But, even memory networks with sophisticated addressing schemes are unable to learn intelligent reasoning satisfactorily due to the complexity of simultaneously learning to access memory and plan. To mitigate these challenges we propose the Memory Augmented Control Network (MACN). The network splits planning into a hierarchical process. At a lower level, it learns to plan in a locally observed space. At a higher level, it uses a collection of policies computed on locally observed spaces to learn an optimal plan in the global environment it is operating in. The performance of the network is evaluated on path planning tasks in environments in the presence of simple and complex obstacles and in addition, is tested for its ability to generalize to new environments not seen in the training set.
mixup: Beyond Empirical Risk Minimization
Keywords: empirical risk minimization, vicinal risk minimization, generalization, data augmentation, image classification, generative adversarial networks, adversarial examples, random labels
TL;DR: Training on convex combinations between random training examples and their labels improves generalization in deep neural networks

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels.  By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples.  Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures.  We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.