Sequenced-Replacement Sampling for Deep Learning

1 INTRODUCTION . Stochastic gradient descent ( SGD ) is an approximation scheme where we only keep a mini-batch in order to reduce the computational cost during the training process . Intuitively , it seems that a smaller mini-batch means a larger approximation , and hence the ultimate model learned must be less accurate . Contrary to the intuition , a relatively smaller mini-batch ( above some threshold ) could actually outperform a larger one in many applications . The reason behind this counter-intuitive phenomenon is that a relatively smaller mini-batch induces more stochasticity and hence exploration during the training process . It is true that a larger mini-batch may lead to faster convergence , but a relatively smaller mini-batch size could lead to a better accuracy . See Keskar et al . ( 2016 ) for more details and discussions . The lesson we learn from the discussions above seems to be that more explorations during the training process may promote generalization . Could we introduce another reasonable type of exploration-inducing mechanism to the training process ? In the deep learning literature , neural networks are typically trained over many epochs . In each epoch , each sample in the dataset is used exactly once . As we enter a new epoch , the samples are reshuffled to form new mini-batches . This means that non-replacement sampling is being used in each epoch . Indeed , non-replacement sampling has been shown to lead to faster convergence than replacement sampling under a relatively general setting Recht & Ré ( 2012 ) . Training-over-epochs is also recommended in Bengio ( 2012 ) . On the other hand , for replacement sampling , a randomly chosen mini-batch is continuously being put back to the original dataset before the next draw of mini-batch . It is conceivable that replacement sampling can lead to more accessible configurations of mini-batches and hence more explorationinduction during the training process . If more explorations during the training process really promote generalization , then it seems that replacement sampling may help to drive the stochastic gradient descent to reach a better local minimum . However , there is one drawback with replacement sampling — as the randomly chosen mini-batches are continuously being put back to the original dataset before the next random draw , there is no guarantee that all the samples in the dataset would have been utilized with sufficient frequency even after a large number of training steps . This may significantly delay the convergence . Worse still , if a significant amount of samples are not sufficiently touched , it is hard to believe that one could reach a relatively optimal local minimum . To tackle this drawback , we propose a scheme which we call “ sequenced-replacement sampling ” for training deep neural networks in this paper . The details of sequenced-replacement sampling will be described in the next section . We find that sequenced-replacement sampling can significantly improve the classification accuracy in the CIFAR-100 dataset over the current state-of-the-art results . This confirms our intuition that more accessible configurations of mini-batches may induce more explorations during the training process and hence promote generalization . 2 SEQUENCED-REPLACEMENT SAMPLING . In the usual training-over-epochs approach , the sampling within one epoch is done without replacement . Suppose that we have N samples and the mini-batch size is B . Then the total number of accessible configurations of mini-batches in one epoch is nB−1∑ k=0 ( N − kB B ) , ( 1 ) where nB = [ N B ] is the nearest integer to N/B , which also represents the total number of available mini-batches within one epoch . Suppose that we train for nE epochs before convergence , then the total number of accessible configurations of mini-batches during the entire training process is Nwithout = nE nB−1∑ k=0 ( N − kB B ) . ( 2 ) This is also the total number of accessible configurations of mini-batches for nE nB training iterations . Alternatively , one could also adopt replacement sampling . In this case , the total number of accessible configurations of mini-batches for nE nB training iterations is Nwith = nE nB ( N B ) = nE nB−1∑ k=0 ( N B ) . ( 3 ) One may wonder why we use ( N B ) instead of ( N+B−1 B ) which is the well-known formula for the conventional replacement sampling . The reason goes as follows . If we were to adopt the conventional replacement sampling , we would have carried out replacement right after each single sample is drawn . In other words , the replacement would have been carried out on a sample-to-sample basis during the entire mini-batch ( B samples ) sampling process . However , in our case , we would not carry out replacement until all of the B samples have already been drawn . After the replacement , we start drawing all the next B samples before carrying out the next replacement . Thus we carry out replacement sampling in a batched way , and hence ( N B ) is more relevant . To compare Nwith and Nwithout , we can first consider ( N B ) ( N−kB B ) = N ( N − 1 ) · · · ( N −B + 1 ) ( N − kB ) ( N − kB − 1 ) · · · ( N − ( k + 1 ) B + 1 ) . ( 4 ) Both of the numerator and denominator in the expression above have exactlyB terms . Except for the case of k = 0 , each factor in the sequence of factors in the numerator is larger than the corresponding factor in the sequence of factors in the denominator . So we have ( N B ) > ( N − kB B ) , for small kB , ( 5 ) ( N B ) ( N − kB B ) , for large kB . ( 6 ) For a sufficiently large ( nB − 1 ) , it is obvious that Nwith Nwithout . Thus replacement sampling leads to many more accessible configurations of mini-batches for nE nB training iterations than the training-over-epochs approach . However , as we pointed out earlier , replacement sampling has a risk of not utilizing all the samples in the dataset with sufficient frequency even after a large number of training iterations . This may significantly delay the convergence or even lead to a relatively less optimal local minimum . In this paper , we would like to apply replacement sampling , but we need to tackle the risk of not touching all samples with sufficient frequency . To achieve this goal , we propose “ sequencedreplacement sampling ” , which is illustrated in Figure 1 . The idea goes as follows . Before the training process , we first assign a fixed sequence index to each sample in the dataset . As the training process proceeds , a mini-batch is randomly being drawn in each iteration . Instead of putting back the mini-batch that has just been drawn , we refill the original dataset by successively adding samples according to their sequence index . In this way , the samples in the mini-batch that has just been drawn would not have an equal probability to be drawn in the next iteration . In fact , the samples that have not yet been drawn will receive a higher probability to be drawn in the next iteration . Of course , there is a possibility that exactly the same samples that have just been drawn will be put back to the original dataset through our sequenced-replacement method . However , the chance for such a coincidence is so low that we can ignore it . With sequenced-replacement sampling , the total number of samples in the dataset remains to be N and the number of samples to be drawn in each training iteration is still B . Therefore , the total number of accessible configurations of mini-batches under sequenced-replacement sampling is the same as that under non-sequenced-replacement sampling ( the original replacement sampling ) . However , the landscape of the accessible configurations of mini-batches under sequencedreplacement sampling will not be the same as that under non-sequenced-replacement sampling . When we apply sequenced-replacement sampling , some accessible configurations of mini-batches under non-sequenced-replacement sampling are no longer accessible . These configurations are mostly replaced by new accessible configurations of mini-batches that are not present under nonsequenced-replacement sampling . Therefore , we conclude that with sequenced-replacement sampling , the risk of not touching all samples with sufficient frequency is tackled . As discussed above , sequenced-replacement sampling can lead to many more accessible configurations of mini-batches and hence more exploration-induction during the training process . Our conjecture is that more explorations during the training process promote generalization , and our experiments confirm this . To significantly improve the classification accuracy in the CIFAR-100 dataset over the current stateof-the-art results , we actually need an important trick to facilitate the success of the sequencedreplacement sampling . As we are getting many more accessible configurations of mini-batches and hence more exploration-induction , we need the learning rate λ to maintain the initial large value for a longer period of time . The reason is that mere existence of many more accessible configurations of mini-batches is not sufficient , we may also require a sufficiently large learning step to actually visit various good local minima and find the optimal one . It is the combination of more accessible con- figurations of mini-batches and a sufficiently large learning rate that leads to effective explorations . In fact , this can also be understood mathematically . First of all , with the gradient descent method , each weight ( model parameter ) at time t , w ( t ) i , is updated with the following formula : w ( t+1 ) i = w ( t ) i − λ ∂L ∂wi ∣∣∣∣ wi=w ( t ) i , ( 7 ) where L is the loss function . The accessible configurations of mini-batches affect the factor ∂L ∂wi ∣∣∣ wi=w ( t ) i which dictates the vectorial direction of the gradient descent . More accessible configurations of mini-batches induce more stochasticity through ∂L∂wi ∣∣∣ wi=w ( t ) i . But it is through a sufficiently large learning rate that the weight updates may be given enough momentum to reach various good local minima . 3 EXPERIMENTS 3.1 EXPERIMENT DESIGN CIFAR-10 CIFAR-100 Classes 10 100 Resolution 32×32 32×32 Channels 3 3 Training images per class 5,000 500 Test images per class 1,000 100 Table 1 : Dataset statistics . In order to validate the effectiveness of sequenced-replacement sampling , we apply it to two of the state-of-the-art methods . Then , we experiment them on public datasets and compare them with stateof-the-art methods where the common non-replacement sampling is applied . We report error rate of classification on test data and show plots of training loss and test accuracy for different epochs . We choose public datasets CIFAR-10 and CIFAR-100 Krizhevsky & Hinton ( 2009 ) for experiments . The statistics of the datasets are described in Table 1 . As shown in the table , there are much fewer training images per class in CIFAR-100 than in CIFAR-10 . For pre-processing , we normalize the data with each channel ’ s mean and stardard deviation . Then , we apply horizontal random flipping and random cropping ( from image padded by 4 pixels on each side ) to training images in each batch . We train two of the state-of-the-art architectures , Wide Residual Networks ( Wide ResNet ) Zagoruyko & Komodakis ( 2016 ) and Densely Connected Convolutional Networks ( DenseNet ) Huang et al . ( 2016a ) , with our proposed sequenced-replacement sampling ( WRN-SRS and DN-SRS ) . Wide ResNet . This network has been proposed to expedite the training in deep residual networks He et al . ( 2016a ) , whose training can be very slow due to the depth of the network . Emphasizing the width of the network , instead of the depth , 16-layer-deep Wide Residual Networks were able to outperform thousand-layer-deep residual networks . Not only the architecture is more computationally efficient than residual networks by decreasing the depth of the network , but also it is regarded as one of the state-of-the-art in terms of its effectiveness . DenseNet . This network has been proposed to solve the vanishing gradient and input problems . In the architecture , each layer is connected to every other layer in a feed-forward fashion , in order to ensure maximum information flow between layers . As a result , its effectiveness is regarded as one of the state-of-the-art while it requires relatively little computation . As the two architectures have been shown to be very effective in image classification , we embed our sequenced-replacement sampling approach in them . However , our method is general and not limited to Wide ResNet and DenseNet . In addition , as our method does not change the underlying architectures , the number of the parameters remains the same . Also , our method does not require additional hyper-parameters , so no extra effort on training process is needed . An important clarification is required here . In addition to the original residual unit proposed in He et al . ( 2016a ) , the authors explored a few similar configurations with different orderings of batch normalizations and activation functions He et al . ( 2016b ) . These include “ BN after addition ” , “ ReLU before addition ” , “ ReLU-only pre-activation ” , and “ full pre-activation ” ( see Figure 4 in He et al . ( 2016b ) ) . By “ WRN-SRS ” , we actually refer “ WRN ” to the Wide ResNet with the “ BN after addition ” residual unit , not the original one . The “ BN after addition ” residual unit is unique in the sense that it provides an “ overall ” batch normalization after the addition . As we anticipate , SRS leads to much more fluctuations , and hence significantly more covariate shift . In order for SRS to bring constructive effects and cause minimal covariate shift , we need an “ overall ” batch normalization after the addition . Indeed our experiments have confirmed that SRS correlates much better with the “ BN after addition ” residual unit than the original one . We reserve the terminology “ Wide ResNet ” for the Wide ResNet with the original residual unit . As for DenseNet , it basically uses something like the “ full pre-activation ” residual unit , not the original one either . However , it has three “ overall ” batch normalization units for the entire network to minimize the covariate shift caused by SRS . Thus , for “ DN-SRS ” , we keep the original DenseNet architecture , except that the non-replacement sampling is replaced by SRS . For sequenced-replacement sampling , the “ number of epochs ” is not well defined . In order to have a more intuitive comparison with non-replacement sampling , we define an “ effective epoch ” for sequenced-replacement sampling : effective epoch for SRS = nI none epoch , ( 8 ) where nI is the number of iterations completed using SRS and none epoch is the number of iterations within one epoch using non-replacement sampling . This would be what we mean by “ the epochs for SRS ” from now on . We use the following configuration of the original Wide ResNet . The architecture ’ s depth is set to 28 and the widening factor k is set to 10 unless specified otherwise . The dropout Srivastava et al . ( 2014 ) is also adopted with the keep probability of 0.7 , and weight decay of 0.0005 is applied to all weights . Stochastic gradient descent with momentum of 0.9 is adopted as the optimizer , and the mini-batch size is set to 64.1 The initial learning rate is set to 0.1 and decays by a factor of 10 at the epochs ( 120 , 150 , 175 ) unless specified otherwise . The models are trained for 200 epochs in total . For DenseNet , we set its depth to 190 and its growth rate to 40 . The same configuration of the mini-batch size , weight decay , and the learning rate schedule is applied to DenseNet . Tensorflow is adopted to implement our method , and all experiments are done with a single GPU , Nvidia Titan X .

Summary: This paper constructs a very simple, new scheme for sampling mini-batches. It aims to (i) achieve the noise properties of sampling with replacement while (ii) reduce the probability of not touching a sample in a given epoch. The result is a biased sampling scheme called “sequenced-replacement sampling (SRS)”. Experimental results show that this scheme performs significantly better than a standard baseline on CIFAR-100 with minor improvements on CIFAR-10.