Excitation Dropout: Encouraging Plasticity in Deep Neural Networks

1 INTRODUCTION . Dropout [ Hinton et al . ( 2012 ) ; Srivastava et al . ( 2014 ) ] is a classical regularization technique that is used in many state-of-the-art deep neural networks , typically applied to fully-connected layers . Standard Dropout selects a fraction of neurons to randomly drop out by zeroing their forward signal . In this work , we propose a scheme for biasing this selection . Our scheme utilizes the contribution of neurons to the prediction made by the network at a certain training iteration stage . Dropout can be interpreted as model averaging technique that avoids overfitting on training data , allowing for better generalization on unseen test data . A recent variant of dropout that targets improved generalization ability is Curriculum Dropout [ Morerio et al . ( 2017 ) ] . It targets adjusting the dropout rate by exponentially increasing the unit suppression rate during training , answering the question How many neurons to drop out over time ? Like Standard Dropout [ Hinton et al . ( 2012 ) ; Srivastava et al . ( 2014 ) ] , Curriculum Dropout selects the neurons to be dropped randomly . In this work , however , we target at determining how the dropped neurons are selected , answering the question Which neurons to drop out ? Our approach is inspired by brain plasticity [ Hebb ( 2005 ) ; Song et al . ( 2000 ) ; Mittal et al . ( 2018 ) ; Miconi et al . ( 2018 ) ] . We deliberately , and temporarily , paralyze/injure neurons to enforce learning alternative paths in a deep network . At training time , neurons that are more relevant to the current prediction are given a higher dropout probability . The relevance of a neuron for making a certain prediction is quantified using Excitation Backprop , a top-down saliency approach proposed by Zhang et al . ( 2016 ) . Excitation Backprop conveniently yields a probability distribution at each layer that reflects neuron saliency , or neuron contribution to the prediction being made . This is utilized in the pipeline of our approach , named Excitation Dropout , which is summarized in Fig . 1 . In particular , we study how this approach improves generalization through utilizing more network ’ s neurons for image classification . We report an increased recognition rate for both CNN models that are fine-tuned and trained from scratch . This improvement is validated on four image/video recognition datasets , and ranges from 1.1 % - 6.3 % over state-of-the-art Curriculum Dropout . Next , we examine the effect of our approach on network utilization . Mittal et al . ( 2018 ) and Ma et al . ( 2017 ) introduce metrics that measure network utilization . We show a consistent increased network utilization using Excitation Dropout on four image/video recognition datasets . For example , averaged over all four benchmarks , we get 76.55 % reduction in conservative filters , filters whose parameters do not change significantly during training , as compared to Standard Dropout . Finally , we study network resilience to neuron dropping at test time . We observe that training with Excitation Dropout leads to models that are a lot more robust when layers are shrunk/compressed by removing units . We demonstrate this when dropping the most relevant neurons , the least relevant neurons , and with a random dropping selection . This can be quite desirable for compressing/distilling [ Hinton et al . ( 2015 ) ] a model , e.g . for deployment on mobile devices . In summary , by encouraging plasticity-like behavior , our contributions are threefold : 1 . Better generalization on test data . 2 . Higher utilization of network neurons . 3 . Resilience to network compression . 2 RELATED WORK . Dropout was first introduced by Hinton et al . ( 2012 ) and Srivastava et al . ( 2014 ) as a way to prevent neural units from co-adapting too much on the training data by randomly omitting subsets of neurons at each iteration of the training phase . Some follow-up works have explored different schemes for determining how much dropout is applied to neurons/weights . Wager et al . ( 2013 ) described the dropout mechanism in terms of an adaptive regularization , establishing a connection to the AdaGrad algorithm . Inspired by information theoretic principles , Achille & Soatto ( 2018 ) propose Information Dropout , a generalization dropout which can be automatically adapted to the data . Kingma et al . ( 2015 ) showed that a relationship between dropout and Bayesian inference can be extended when the dropout rates are directly learned from the data . Kang et al . ( 2017 ) introduces Shakeout which instead of randomly discarding units as dropout does , it randomly enhances or reverses each units contribution to the next layer . Wan et al . ( 2013 ) introduced the DropConnect framework , adding dynamic sparsity on the weights of a deep model . DropConnect generalized Standard Dropout by randomly dropping the weights rather than the neuron activations in the network . Rennie et al . ( 2014 ) proposed a time scheduling for the retaining probability for the neurons in the network . The presented adaptive regularization scheme smoothly decreased in time the number of neurons turned off during training . Recently , Morerio et al . ( 2017 ) proposed Curriculum Dropout to adjust the dropout rate in the opposite direction , exponentially increasing unit suppression rate during training , leading to a better generalization on unseen data . Other works focus on which neurons to drop out . Dropout is usually applied to fully-connected layers of a deep network . Conversely , Wu & Gu ( 2015 ) studied the effect of dropout in convolutional and pooling layers . The selection of neurons to drop depends on the layer where they reside . In contrast , we select neurons within a layer based on their contribution . Wang & Manning ( 2013 ) demonstrate that sampling neurons from a Gaussian approximation gave an order of magnitude speedup and more stability during training . Li et al . ( 2016 ) proposed to use multinomial sampling for dropout , i.e . keeping neurons according to a multinomial distribution with specific probabilities for different neurons . Ba & Frey ( 2013 ) jointly trained a binary belief network with a neural network to regularize its hidden units by selectively setting activations to zero accordingly to their magnitude . While this takes into consideration the magnitude of the forward activations , it does not take into consideration the relationship of these activations to the ground-truth . In contrast , we drop neurons based on how they contribute to a network ’ s decision . We compare our results against Morerio et al . ( 2017 ) . To the best of our knowledge , we are the first to probabilistically select neurons to dropout based on their task-relevance . 3 METHOD . 3.1 BACKGROUND . Saliency maps that quantize the importance of class-specific neurons for an input image are instrumental to our proposed scheme . Popular approaches include Class Activation Maps ( CAM ) [ Zhou et al . ( 2016 ) ] , Gradient-weighted Class Activation Mapping ( Grad-CAM ) [ Selvaraju et al . ( 2017 ) ] , and Excitation Backprop ( EB ) [ Zhang et al . ( 2017 ) ] . A thorough analysis of all saliency methods is out of the scope of this work , and the saliency problem in general is far from solved . We choose to use EB since it produces a valid probability distribution for each network layer . The saliency maps obtained using this approach are evaluated for spatial localization of objects and demonstrate the ability of pointing to the right region of an image [ Zhang et al . ( 2017 ) ] . In a standard CNN , the forward activation of neuron aj is computed by âj = φ ( ∑ i wij âi + bi ) , where âi is the activation coming from the previous layer , φ is a nonlinear activation function , wij and bi are the weight from neuron i to neuron j and the added bias at layer i , respectively . EB devises a backpropagation formulation able to reconstruct the evidence used by a deep model to make decisions . It computes the probability of each neuron recursively using conditional probabilities P ( ai|aj ) in a top-down order starting from a probability distribution over the output units , as follows : P ( ai ) = ∑ aj∈Pi P ( ai|aj ) P ( aj ) ( 1 ) where Pi is the parent node set of ai . EB passes top-down signals through excitatory connections having non-negative activations , excluding from the competition inhibitory ones . EB is designed with an assumption of non-negative activations . Most modern CNNs use ReLU activation functions , which satisfy this assumption . Therefore , negative weights can be assumed to not positively contribute to the final prediction . Assuming Cj the child node set of aj , for each ai ∈ Cj , the conditional winning probability P ( ai|aj ) is defined as P ( ai|aj ) = { Zj âiwij , if wij ≥ 0 , 0 , otherwise ( 2 ) where Zj is a normalization factor such that ∑ ai∈Cj P ( ai|aj ) = 1 . Recursively propagating the top-down signal and preserving the sum of backpropagated probabilities , it is possible to highlight the salient neurons in each layer using Eqn . 1 , i.e . neurons that mostly contribute to a specific task . We will refer to the distribution of P ( ai ) as pEB ( ai ) . 3.2 EXCITATION DROPOUT . In the standard formulation of dropout [ Hinton et al . ( 2012 ) ; Srivastava et al . ( 2014 ) ] , the suppression of a neuron in a given layer is modeled by a Bernoulli random variable 0 < p ≤ 1 where p is defined as the probability of retaining a neuron . Given a specific layer where dropout is applied , during the training phase , each neuron is turned off with a probability 1− p. We argue for a different approach that is guided in the way it selects neurons to be dropped . In a training iteration , certain paths have high excitation contributing to the resulting classification , while other regions of the network have low response . We encourage learning alternative paths ( plasticity ) through the temporary damaging of the currently highly excited path . We re-define the probability of retaining a neuron as a function of its contribution in the currently highly excited path p = 1− ( 1− P ) ∗ ( N − 1 ) ∗ pEB ( ( 1− P ) ∗N − 1 ) ∗ pEB + P ( 3 ) where pEB is the probability backpropagated through the EB formulation ( Eqn . 1 ) in layer l , P is the base probability of retaining a neuron when all neurons are equally contributing to the prediction and N is the number of neurons in a fully-connected layer l or the number of filters in a convolutional layer l. The retaining probability defined in Eqn . 3 drops neurons which contribute the most to the recognition of a specific class , with higher probability . Dropping out highly relevant neurons , we retain less relevant ones and thus encourage them to awaken . We also study how this compares to dropping the least relevant neurons ( Adaptive Dropout by Ba & Frey ( 2013 ) ) in the Appendix . Fig . 2 shows p as a function of pEB . To gain some intuition for Eqn . 3 , we can look more closely at the graph : 1 ) If neuron ai has pEB ( ai ) = 1 : This results in a retaining probability of p = 0 . We do not want to keep a neuron which has a high contribution to the correct label . 2 ) If neuron ai has pEB ( ai ) = 0 : This results in a retaining probability of p = 1 . We want to keep a neuron which has not contributed to the correct classification of an image . 3 ) If neuron ai has pEB ( ai ) = 1/N , i.e . pEB is a uniform probability distribution : This results in a retaining probability p = P . We want to keep a neuron with base probability P since all neurons contribute equally . Eqn . 3 provides a dropout probability for each neuron , which is then used as the parameter of a Bernoulli distribution giving a binary dropout mask . During training , each image in a batch leads to different excitatory connections in the network and therefore has a different pEB distribution , consequently leading to a different dropout mask . Fig . 1 presents the pipeline of Excitation Dropout at training time , and a run-time analysis is presented in the Appendix .

Summary: The authors propose a data-dependent dropout variant that produces dropout candidates based on their predictive saliency / relevance. Results are reported for 4 datasets (Cifar10, Cifar100, Caltech256 and UCF101) and 4 different models (CNN-2, AlexNet, VGG16 and VGG19), and suggest an increase in generalization performance over other dropout approaches (curriculum dropout, standard dropout, no dropout), as well as increase in the network's plasticity as measured by some existing metrics from the literature. The authors conclude that Excitation Dropout results in better network utilization and offers advantages for network compression (in the sense of neuron pruning).