Backdrop: Stochastic Backpropagation

1 INTRODUCTION . Stochastic gradient descent ( SGD ) and its minibatch variants are ubiquitous in virtually all learning tasks ( Ian Goodfellow & Courville ( 2016 ) ; Bottou et al . ( 2018 ) ) . SGD enables deep learning to scale to large datasets , decreases training time and improves generalization . However , there are many problems where there exists a large amount of information but the data is packaged in a small number of information-rich samples . These problems can not take full advantage of the benefits of SGD as the number of training samples is limited . Examples of these include learning from high resolution medical images , satellite imagery and GIS data , cosmological simulations , lattice simulations for quantum systems , and many others . In all of these situations , there are relatively few training samples , but each training sample carries a tremendous amount of information and can be intuitively considered as being comprised of many independent subsamples . Since the benefits of SGD require the existence of a large number of samples , efficiently employing these techniques on the above problems requires careful analysis of the problem in order to restructure the information-rich data samples into smaller independent pieces . This is not always a straight-forward procedure , and may even be impossible without destroying the structure of the data ( Ravanbakhsh et al . ( 2017 ) ; Shanahan et al . ( 2018 ) ) . This motivates us to introduce backdrop , a new technique for stochastic gradient optimization which does not require the user to reformulate the problem or restructure the training data . In this method , the loss function is unmodified and takes the entirety of each sample as input , but the gradient is computed stochastically on a fraction of the paths in the backpropagation pipeline . A closely related class of problems , which can also benefit from backdrop , is optimization objectives defined via non-decomposable loss functions . The rank statistic , a differentiable approximation of the ROC AUC is an example of such loss functions ( Herschtal & Raskutti ( 2004 ) ) . Here again , minibatch SGD optimization is not viable as the loss function can not be well approximated over small batch sizes . Backdrop can also be applied in these problems to significantly improve generalization performance . Main contributions . In this paper , ( a ) we establish a means for stochastic gradient optimization which does not require a modification to the forward pass needed to evaluate the loss function and is particularly useful for problems with non-trivial subsample structure or with non-decomposable loss . ( b ) We explore the technique empirically and demonstrate the significant gains that can be achieved using backdrop in a number of synthetic and real world examples . ( c ) We introduce the “ multiscale GP texture generator ” , a flexible synthetic texture generator with clear hierarchical subsample structure which can be used as a benchmark to measure performance of networks and optimization tools in problems with hierarchical subsample structure . The source code for our implementation of backdrop and the multi-scale GP texture generator is available online . 1.1 RELATED WORK . Non-decomposable loss optimization . The study of optimization problems with complex nondecomposable loss functions has been the subject of active research in the last few years . A number of methods have been suggested including methods for online optimization ( Kar et al . ( 2014 ) ; Kar et al . ( 2015 ) ; Narasimhan et al . ( 2015 ) ) , methods to solve problems with constraints ( Narasimhan ( 2018 ) ) , and indirect plug-in methods ( Narasimhan et al . ( 2014 ) ; Koyejo et al . ( 2014 ) ) . There are also methods that indirectly optimize specific performance measures ( Dembczynski et al . ( 2011 ) ; Ye et al . ( 2012 ) ) . Our contribution is fundamentally different from the previous work on the subject . Firstly , our approach can be applied to any differentiable loss function and does not change based on the exact form of the loss . Secondly , implementation of backdrop is simple , does not require changing the network structure or the optimization technique and can therefore be used in conjunction with any other batch optimization method . Finally , backdrop optimization can be applied to a more general class of problems where the loss is decomposable but the samples have complex hierarchical subsample structure . Study of minibatch sizes . Recently there have been many studies analyzing the effects of different batch sizes on generalization ( Wilson & Martinez ( 2003 ) ; LeCun et al . ( 1998 ) ; Keskar et al . ( 2016 ) ; Masters & Luschi ( 2018 ) ) and on computation efficiency and data parallelization ( Das et al . ( 2016 ) ; Dean et al . ( 2012 ) ; Hoffer et al . ( 2017 ) ) . These studies , however , are primarily concerned with problems such as classification accuracy with cross-entropy or similar decomposable losses . There are no in-depth studies specifically targetting the effects of varying the batch size for non-decomposable loss functions that we are aware of . We address this in passing as part of one of our examples in Sec . 3 . 2 BACKDROP STOCHASTIC GRADIENT DESCENT . Let us consider the following problem . Given dataset S = { x1 , · · · , xN } with iid samples xn , we wish to optimize an empirical loss function L = Lθ ( S ) , where θ are parameters in a given model . We can employ gradient descent ( GD ) , an iterative optimization strategy for finding the minimum of L wherein we take steps proportional to −∇θL . An alternate strategy which can result in better generalization performance is to use stochastic gradient descent ( SGD ) , where instead we take steps according to −∇θLn = −∇θL ( xn ) , i.e . the gradient of the loss computed on a single sample . We can further generalize SGD by evaluating the iteration step on a number of randomly chosen samples called minibatches ( Ian Goodfellow & Courville ( 2016 ) ; Bottou et al . ( 2018 ) ) . We cast GD and minibatch SGD as graphical models in Figs . 1a and 1b , where the solid and dashed lines respectively denote the forward and backward passes of the network and N , B , and N/B denote the number of samples , the number of minibatches and the minibatch size . Now we consider two variations of this problem . First let us assume that the total loss we wish to optimize is not decomposable over the individual samples , i.e . it can not be written as a sum of sample losses , for example it might depend on some global property of the dataset and include cross terms between different samples , i.e . L = f ( ~h ) , where hn is the hidden state computed from sample xn and f is some non-decomposable function . There are many problems where these kinds of losses arise , and we discuss a few examples in Sec . 3 . In extreme cases , the loss requires all N data points and batching the data is no longer meaningful ( i.e . B = 1 ) . In other cases , a large number of samples is required to accurately approximate the loss ( i.e . N/B 1 ) and hence minibatch SGD is also not optimal . In order to deal with this problem and recover stochasticity to the gradient , we propose the following . During optimization , we evaluate the total loss during the forward pass , but for gradient calculation during the backward pass , we only compute δθ with respect to one ( or some ) of the samples . Note that if the loss L can be decomposed to a sum of sample losses L = ∑ Ln , this procedure would be identically equal to SGD . In practice , we implement backdrop by using a Bernoulli random variable to choose which samples to perform the gradient update with respect to ( Fig . 1c ) . We call this method backdrop as it is reminiscent of dropout applied to the backpropagation pipeline . We now consider a different variation of problem . Let us assume that each sample x consists of a number of subsamples x = { σ1 , · · · , σM } where the subsamples σ are no longer required to be iid . For example , an image can be considered as a collection of smaller patches , or a sequence can be considered as a collection of individual sequence elements . σi can also be defined implicitly by the network , for example it could refer to the ( possibly overlapping ) receptor fields of the individual outputs of a convolutional layer . Similar to above , we take the sample loss to be a non-linear function of the subsamples i.e . Ln = f ( ~h ) which again can not be decomposed into a sum over the individual subsamples . We can implement backdrop by defining δθBD in a manner analogous to the previous case . Figure 2 depicts this implementation of backdrop on subsamples , where the B and N/B are as before and the M plate represents the subsamples . Note that while it is possible to think of the first class of problems with non-decomposable losses as a special case of the second class of problems with subsamples , we find that it is conceptually easier to consider these two classes separately . 2.1 IMPLEMENTATION . Since backdrop does not modify the forward pass of the network , it can be easily applied in most architectures without any modification of the structure . The simplest method to implement backdrop is via the insertion of a masking layer Mp defined as follows . For vector ~v = ( v1 , · · · , vN ) we have : Mp ( vn ) ≡ vn , ∇vnMp ( vm ) ≡ δnm N∣∣~b∣∣ 1 bn ( 1− p ) , ( 1 ) where~b is a length N vector of Bernoulli random variables with probability 1− p and |~b|1 counts the number of its nonzero elements . In words , Mp acts as identity during the forward pass but drops a random number of gradients given by probability p during gradient computation . The factor in front of the gradient is normalization such that the ` 1 norm of the gradient is preserved . For the problem where the total loss is a non-decomposable function of the sample latent variables L = f ( ~h ) , we can implement backdrop simply by inserting a masking layer between hn and L ( Fig . 1c ) : LBD = f ( Mp ( ~h ) ) ⇒ δθBD = ∇θf ( Mp ( ~h ) ) = N∣∣~b∣∣ 1 ∑ n bn∇θhn · ∇hnf ( ~h ) , ( 2 ) resulting in only a fraction of the gradients being accumulated for the gradient descent update as desired . We would say backdrop in this case is masking its input along the minibatch direction n , resulting in a smaller effective batch size or EBS , which we define as EBS ≡ N/B × ( 1− p ) . In more complicated cases , for example if we want to implement backdrop on the subsamples as in Fig . 2 , the location of the masking layer along the pipeline of the network may depend on the details of the problem . We will discuss this point with a number of examples in Sec . 3 . However , even when the scale and structure of the subsamples is not known , it is possible to insert masking layers at multiple points along the network and treat the various masking probabilities as hyperparameters . In other words , it is not necessary to know the details of structure of the data in order to implement and take advantage of the generalization benefits provided by backdrop . We note that while this implementation of backdrop is reminiscent to dropout ( Hinton et al . ( 2012 ) ) , there are major differences . Most notably , the philosophy behind the two schemes is fundamentally different . Dropout is a network averaging method that simultaneously trains a multitude of networks and takes an averages of the outcomes to reduce over-fitting . Backdrop , on the other hand , is a tool designed to take advantage of the structure of the data and introduce stochasticity in cases where SGD is not viable . On a technical level , with backdrop , the masking takes place during the backward pass while the forward pass remains undisturbed . Because of this , backdrop can be applied in scenarios where the entire input is required for the task and it can also be simultaneously used with other optimization techniques . For example , it does not suffer the same incompatibility issues that dropout has with batch-normalization ( Li et al . ( 2018 ) ) .

Summary: This paper proposes a stochastic based method, namely Backdrop, for updating the network structures via backpropagation type methods.  Backdrop inserts masking layers along the network; it acts as the identity in the forward pass, but as randomly masks parts of the backward gradient propagation. The paper claims this approach can significantly improves the overall generalization performance.