Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout

1 INTRODUCTION . Recent years have seen a rapid development of deep neural network in the computer vision area , especially for visual object recognition tasks . From AlexNet , the winner of ImageNet Large Scale Visual Recognition Challenge ( ILSVRC ) ( Krizhevsky et al. , 2012 ) , to later VGG network ( Simonyan & Zisserman , 2014 ) and GoogLeNet ( Szegedy et al. , 2015 ) , CNNs have shown significant success with its shocking improvement of accuracy . Researchers gradually realized that the depth of the network always plays an important role in the final accuracy of one model due to increased expressiveness ( Håstad , 1987 ; Håstad & Goldmann , 1991 ; Szegedy et al. , 2015 ) . However , simply increasing the depth does not always help due to the induced vanishing gradient problem ( Glorot & Bengio , 2010 ) . ResNet ( He et al. , 2016a ) has been proposed to promote the flow of information across layers without attenuation by introducing identity skip-connections , which sums together the input and the output of several convolutional layers . In 2016 , Densely connected network ( DenseNet ) ( Huang et al. , 2016a ) came out , which replaces the simple summation in ResNet with concatenation after realizing the summation may also impede the information flow . Despite the improved information flow , DenseNet still suffers from the overfitting problem , especially when the network goes deeper . Standard dropout ( Srivastava et al. , 2014 ) has been used to combat such problem , but can not work effectively on DenseNet . The reasons are twofold : 1 ) Feature-reuse will be weakened by standard dropout as it could make features dropped at previous layers no longer be used at later layers . 2 ) Standard dropout method does not interact well with convolutional layers because of the spatial correlation inside feature maps ( Tompson et al. , 2015 ) . Since dense connectivity increases the number of feature maps tremendously — especially at deep layers — the effectiveness of standard dropout would further be reduced . In this paper , we design a specialized dropout method to resolve these problems . In particular , three aspects of dropout design are addressed : 1 ) Where to put dropout layers in the network ? 2 ) What is the best dropout granularity ? 3 ) How to assign appropriate dropout ( or survival ) probabilities for different layers ? Meanwhile , we show that the idea to re-design the dropout method from the three aspects also applies to other CNN models like ResNet . The contributions of the paper can be summarized as follows : • First , we propose a new structure named pre-dropout to solve the possible feature-reuse obstruction when applying standard dropout method on DenseNet . • Second , we are the first to show that channel-wise dropout ( compared to layer-wise and unit-wise ) fit best for CNN through both detailed analysis and experimental results . • Third , we propose three distinct probability schedules , and via experiments we find out the best one for DenseNet . • Last , we provide a good insight for future practitioners , inspiring them regarding what should be considered and which is the best option when applying the dropout method on a CNN model . Experiments in our paper suggest that DenseNets with our proposed specialized dropout method outperforms other comparable DenseNet and state-of-art CNN models in terms of accuracy , and following the same idea dropout methods designed for other CNN models could also achieve consistent improvements over the standard dropout method . 2 RELATED WORK . In this section , first we will review some basic ideas behind DenseNet , which is also the foundation of our method . Then the standard dropout method along with some of its variants will be introduced as a counterpart of our proposed dropout approach . 2.1 DENSENET . DenseNet was first proposed by ( Huang et al. , 2016a ) , which features a special structure—dense blocks . Figure 1a gives an example of the dense block . One dense block consists of several convolutional layers , each of which output k feature maps , where k is referred to as the growth rate in the network . The most important property of DenseNet is that for each convolutional layer inside the dense block , the input is the concatenation of all feature maps from the preceding layers within the same block , which is also known as the dense connectivity . With dense connectivity previous output features could be reused at later layers . X ` = L ` ( [ X0 , X1 , · · · , X ` −1 ] ) ( 1 ) Equation 1 shows such kind of relationship clearly . Here Xi represents the output at layer i . [ , ] denotes concatenation operation . L is a composite function of batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) , rectified linear unit ( ReLU ) ( Glorot et al. , 2011 ) and a 3× 3 convolutional layer . With the help of feature-reuse , DenseNet achieves a better performance and turns out to be more robust ( Huang et al. , 2017 ) . However , dense connectivity will cost a large amount of parameters . To ease the consumption of parameters , a variant structure , named DenseNet-BC , came out . In this structure , a 1× 1 layer is added before each 3× 3 layer to reduce the depth of input to 4k . 2.2 DROPOUT . Standard dropout method ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) discourages co-adaptation between units by randomly dropping out unit values . Stochastic depth ( Huang et al. , 2016b ) extends it to layer level by randomly dropping out whole layers . Other alternative examples include DropConnect ( Wan et al. , 2013 ) , which generalizes dropout by dropping individual connections instead of units ( i.e. , dropping several connections together ) , and Swapout ( Singh et al. , 2016 ) which skips layers randomly at a unit level . These previous works mainly focus on one aspect of dropout method , i.e. , the dropout granularity . We are the first who gives a thorough study of the dropout design from all three aspects : dropout location , dropout granularity and dropout probability . In our evaluation , we will not only give the overall accuracy improvement , but also the breakdown along all these three aspects . Meanwhile , all these methods above will impede the feature-reuse in DenseNet since dropped features in previous layers will never be available to later layers . Figure 1b shows an example of the feature-reuse obstruction when applying standard dropout method on DenseNet . 3 MODEL DESCRIPTION . As previously mentioned , standard dropout method will have some limitations on DenseNet : 1. it could impede feature-reuse ; 2 . The effect of dropout will be weakened by the spatial correlation . To solve these problems and further improve model generalization ability , we propose the following structures . 3.1 PRE-DROPOUT STRUCTURE . Pre-dropout structure aims at solving the possible feature-reuse obstruction when applying standard dropout to DenseNet . As mentioned in Section 2.2 , due to the dense connectivity , standard dropout will make features dropped at previous layers no longer be used at later layers , which will weaken the effect of feature-reuse in DenseNet . To retain the feature-reuse , we come up with a simple yet novel idea named pre-dropout , which instead places dropout layers before the composite functions so that complete outputs from previous convolutional layers can be transferred directly to later layers before the dropout method is applied . Meanwhile one extra benefit from pre-dropout is that we can stimulate much more feature-reuse patterns in the network because of different dropout patterns applied before different layers . Figure 2 illustrates the differences between standard dropout and pre-dropout . In the following , we will explain these two benefits in details . Suppose the input to Figure 2a is Xstandard0 , then it would also be used as the input to the dropout layer B1 . The calculation of B1 could be regarded as element-wisely multiplying Xstandard1 with Θ1 , a tensor of random variables following Bernoulli ( p1 ) . Thus we can get X standard 1 = Θ1 Xstandard1 = Θ1 L1 ( Xstandard0 ) ( 2 ) where represents element-wise multiplication . Then Xstandard0 and X standard 1 will be concatenated together as the input to L2 , i.e. , Xstandard0 ⊕ X standard 1 , here ⊕ denotes concatenation . So on and so forth . Finally , the output of the dense block can be written as Wstandard = Xstandard0 ⊕X standard 1 ⊕ · · · ⊕X standard n ( 3 ) Similarly we can get the mathematical representations from Figure 2b , which shows the data flow in a dense block with pre-dropout method , X pre 1 = Θ1,2 X pre 1 = Θ1,2 L1 ( Θ0,1 X pre 0 ) ( 4 ) Wpre = ( Θ0 , n+1 Xpre0 ) ⊕ ( Θ1 , n+1 X pre 1 ) ⊕ · · · ⊕ ( Θn , n+1 Xpren ) ( 5 ) where Θi , j represents the tensor of dropout layer connecting from the output of layer i to the input of layer j , in which random variables follow Bernoulli ( pi , j ) . Comparing Equations 2 with 4 , we can notice that pre-dropout method will allow better feature-reuse than standard dropout method . For instance , the outputs of standard dropout would become zero and remains the same as inputs to all following layers when Θ1 equals to zero . However pre-dropout solves this problem . In pre-dropout method , Xpre1 would be multiplied by different independent tensors Θ1,2 , Θ1,3 , · · · , Θ1 , n+1 , such that for any specific Xpre1ijk even if Θ1,2 makes X pre 1ijk be zero at L2 , we still have a chance to reuse this feature at later layers . Meanwhile we could also find that pre-dropout method could stimulate much more feature-reuse patterns in the network . For example , in Figure 2a , once Xstandard1 goes through dropout layer B1 , the same output will always be reused . Whereas in pre-dropout method , every time before Xpre1 is utilized as the input to the next layer , it would be multiplied by a different tensor . In Figure 2b , the contributions of Xpre1 are actually two distinct features , since Θ1,2 and Θ1 , n+1 are independent . Note that similar feature-reuse obstruction also exists when applying standard dropout on other CNN models with shortcut connections , such as ResNet ( He et al. , 2016a ) , Wide-ResNet ( Zagoruyko & Komodakis , 2016 ) and RoR ( Zhang et al. , 2017 ) . Thus pre-dropout method could work for those networks as well . 3.2 CHANNEL-WISE DROPOUT . When designing our specialized dropout method , dropout granularity is also an important aspect to be considered . Figure 3 shows three different granularity : unit-wise , channel-wise and layer-wise . Standard dropout is one kind of unit-wise method , which has been proved useful when applied on fully connected layers ( Srivastava et al. , 2014 ) . It helps improve the generalization capability by breaking the strong dependence between neurons from different layers.However when applying the same method on convolutional layers , the effectiveness of standard unit-wise method will be hampered due to the strong spatial correlation between neurons within a feature map — although dropping one neuron can stop other neurons from replying on that particular one , they will still be able to learn from the correlated neurons in the same feature map . To cope with the spatial correlation , we need dropout at better granularity . Layer-wise method drops the entire layers , whcih refer to the outputs from previous layers , inside the input tensor . However , layer-wise dropout is prone to discard too many useful features . Channel-wise method , which will drop a entire feature map at a given probability , strikes a nice trade-off between the two granularity above . Our experiments also confirmed that channel-wise works the best for regularizing DenseNet . Meanwhile since the spatial correlation exists in all types of convolutional layers , our analysis above should also work for other CNN models whenever a dropout method is applied .

Summary: This paper proposes a special dropout procedure for densenet. The main argument is standard dropout strategy may impede the feature-reuse in Densenet, so the authors propose a pre-dropout technique, which implements the dropout before the nonlinear activation function so that it can be feeded to later layers. Also other tricks are discussed, for example, channel-wise dropout, and probability schedule that assigns different probabilities for different layers in a heuristic way.