Faster Training by Selecting Samples Using Embeddings

Long training times have increasingly become a burden for researchers by slowing down the pace of innovation , with some models taking days or weeks to train . In this paper , a new , general technique is presented that aims to speed up the training process by using a thinned-down training dataset . By leveraging autoencoders and the unique properties of embedding spaces , we are able to filter training datasets to include only those samples that matter the most . Through evaluation on a standard CIFAR-10 image classification task , this technique is shown to be effective . With this technique , training times can be reduced with a minimal loss in accuracy . Conversely , given a fixed training time budget , the technique was shown to improve accuracy by over 50 % . This technique is a practical tool for achieving better results with large datasets and limited computational budgets . 1 INTRODUCTION . As models become more sophisticated and complex , the time required to train a neural network becomes an increasingly important consideration in the development process . Some networks can take a month to train , even on a multi-GPU system ( Radford et al. , 2017 ) , so any improvement in this training time can result in a faster development cadence . In this paper , we present a technique that reduces training time through dataset optimization . Our technique involves the analysis of a dataset ’ s embedding space to estimate an optimal training subset . Intuitively , one can imagine that early on in the training process , certain samples can more directly lead to convergence . For example , an object classification network can accomplish more learning by being presented with two very different images than two very similar images of the same object class . Thus , by targeting a network with such distinct samples in its infancy , one can reduce the network ’ s loss more effectively . The problem is now reduced to finding these distinct samples within the training dataset . We accomplish this task using a dense representation that is learned a priori in an unsupervised manner using an auto-encoder . By leveraging properties of the embedding space , we are able to find distinct samples by selecting those with distinct embeddings . A model trained with a subset of the full training dataset clearly can not reach the same level of efficacy as one trained with the entire training dataset . Thus , the optimized subset is only used for initial training , after which fine-tuning is performed with the full training dataset . We show that we are able to achieve a reduction in the required number of training steps , corresponding to 91.36 % smaller epochs , on an image classification task , using the CIFAR-10 dataset ( Krizhevsky & Hinton , 2009 ) , with minimal sacrifice in the resultant network ’ s accuracy . In such tests , the optimized dataset was just 8.64 % the size of the full training dataset . Furthermore , given a fixed time budget , our technique can yield even higher accuracies than a full training approach since epochs under the optimized dataset take less time . In the next section , we motivate our work with a discussion of the challenges in training neural networks on large datasets . We then provide an overview of a variety of related work in the literature . Following this review , we describe our dataset optimization technique in detail , along with the domain upon which we have evaluated it . A high level , step-by-step overview of the technique is also provided . In subsequent sections , we present our results and a discussion to evaluate this work ’ s impact on the field . 2 MOTIVATION . Owing to advances in modern semiconductor manufacturing processes , the industry has seen significant increases in the capacities of modern random access memories . However , as the pace of miniaturization slows and larger machine learning models become the norm , the demands placed upon memories have outstripped their increases in size . For instance , in the 2014 ImageNet Large Scale Visual Recognition Challenge , the image detection dataset required 47GB of storage , while the classification / localization dataset consumed 138GB ( Russakovsky et al. , 2015 ) . These numbers are only set to grow as researchers tackle larger datasets with richer , higher resolution data . While these datasets have played a key role in training networks with state-of-the-art accuracy across different tasks , they have also been damaging for training times . The first , less obvious , reason for this is that larger datasets pose significant challenges for existing accelerator architectures ( such as the Google Tensor Processing Unit ( TPU ) ( Jouppi et al. , 2017 ) , the NVIDIA V100 GPU , and various FPGAs ) . Unfortunately , most accelerators only have between 4GB and 16GB of memory available on-board , depending on their price and target applications . Since the full dataset can not fit entirely in accelerator memory , new training examples must be constantly streamed to the accelerators over the PCIe bus , which is approximately two orders of magnitude slower than the on-board memory . Furthermore , if the dataset does not fit in CPU-side memory , transfers may be limited by the computer ’ s underlying storage speed . However , even when CPU-side memory is large enough , training example preprocessing , such as data augmentation and formatting , can overwhelm the CPU if there are not enough cores . From these observations , it can be noted that training optimally on large datasets requires an enterprise-class budget that is out of reach of many researchers . Additionally , large datasets have revealed an even more relevant problem : training inefficiency due to a lack of scalability . Currently , neural network training consists of the repeated presentation of samples to the model . Unfortunately , while these techniques may be viable for smaller networks or datasets , large datasets have shown that they do not scale well . Despite the wider variety of information present in larger datasets , each example must still be presented to the neural network a roughly similar number of times . This leads to significantly diminishing returns where the added examples increase training time significantly while only increasing accuracy slightly , if at all . If we want to be able to continue to incorporate the ever increasing amount of data available into neural networks , this problem can not be ignored . There are a number of existing solutions to both of these challenges , but they fall short . While one could accept that training neural networks on large datasets will be less than ideal , this would constitute giving up and refusing to solve the problem . Instead , we believe that reducing the training set size during only the first part of training will solve these problems with fewer compromises than existing solutions . By reducing the training set size by an order of magnitude , almost all existing datasets can fit entirely in accelerator memory . This requires the data to only be transferred to the accelerator once , eliminating practically all bottlenecks so that the accelerator can proceed at full speed with minimal cost to the rest of the system . Since only fine-tuning must be performed with the full dataset , much of the training time can be accelerated , resulting in significant time savings . 3 RELATED WORK . To date , dataset optimization has not been a major topic in deep learning literature . Most studies focus on improving accuracy , and use as much data and training time as necessary . When training and evaluation time is a concern , researchers have started to investigate the use of network architectures that are inherently quicker to train ( such as in Yamanaka et al . ( 2017 ) ) . A key contribution to training time reduction via dataset optimization was promulgated by Dünner et al . ( 2017 ) . With the goal of allowing datasets to fit entirely in the memory of different compute hardware , a system was developed to intelligently prune datasets through analysis of the duality gap . The authors ’ approach enables better utilization of hardware in heterogeneous systems . Notably , this work is tailored for support vector machines ( SVMs ) ; neural networks and more sophisticated models were out of the work ’ s scope . While dataset optimization is not a well studied area , embedding spaces are . Embeddings and similar latent representations more frequently form the basis for systems ’ functionality . One key area is in generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . GANs learn to generate new data that is similar to a training dataset by jointly training separate generator and discriminator networks . The generator learns to output data , based off of a latent input , that the discriminator deems realistic . These latent input spaces have been found to exhibit unique properties ; namely , two latent vectors that are not distant from each other lead to similar outputs when propagated through the generator . This property is a key insight that enables the presented dataset optimization technique as well . There has also been work in manipulating latent vectors with arithmetic operations . Wu et al . Wu et al . ( 2016 ) applied GANs to 3D voxel data to learn dense representations for 3D models . The authors found very interesting behavior when performing addition and subtraction on embeddings . Sequences of operations on embeddings resulted in new , unseen 3D models being generated that follow human intuition . For instance , one can subtract the latent representation of a chair without arms from the representation of a chair with arms to generate the latent representation of the arms feature . This feature representation could be combined with that for an object without arms to generate a version of that object with arms attached . These results demonstrate that latent representations encode high-level data about the objects they represent and that they are more than just a compressed version of the raw input data . More recent works have aimed to impose desiderata on learned embedding spaces . Hsiao & Grauman ( 2017 ) created an embedding for clothing fashion that encodes information for fashion style , while encoding little else . Wang et al . ( 2016 ) built a system that learns a joint embedding for images and text , allowing for the construction of systems that can handle images and text in an equal manner . 4 APPROACH . To evaluate the technique ’ s efficacy , we have chosen the well-studied task of image classification . Specifically , we are performing classification on the CIFAR-10 dataset ( Krizhevsky & Hinton , 2009 ) . We describe our basic classifier model in detail in Subsection 4.1 . Following this , we discuss the system ’ s autoencoder architecture and the technique we developed . 4.1 CLASSIFIER ARCHITECTURE . Figure 1 illustrates the CIFAR-10 image classifier used in this paper . We follow a basic convolutional neural network ( CNN ) architecture that takes inspiration from AlexNet ( Krizhevsky et al. , 2012 ) . The network architecture has been chosen due to its simplicity ; achieving state-of-the-art accuracy on CIFAR-10 is not the focus of this paper . 5×5 convolutions , 3×3 max-pooling , and batch normalization layers arranged in sequence , following typical CNN designs , gradually reducing dimensionality . Notably , inputs to the network are sized 24×24×3 , rather than 32×32×3 ; this enables data augmentation , described in detail in subsection 4.1.2 Since this is a multi-class , single-label classification problem , output logits from the final fullyconnected layer are passed through a softmax layer , ensuring that all final outputs correspond to predicted posterior probabilities .

Summary: This paper presents the idea of splitting the training process into two phases: fast training on a subset of the original dataset and finetining on the full dataset. To find a good subset of the training dataset it is proposed to train an autoencoder and use its embeddings to choose examples that have large values of the embedding features. The experiments show that on CIFAR-10 dataset this may speed up the convergence.