Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

1 INTRODUCTION . 1.1 MLP AND ITS TRAINING OBJECTIVE . A multilayer perceptron ( MLP ) is one of the simplest and the oldest artificial neural network architectures . Its origin dates back to 1958 ( Rosenblatt , 1958 ) . Its basic building blocks are linear regression and nonlinear activation functions . One can stack them up to create an MLP with L layers ( See Equation 1 ) . ŷ = f ( L ) ( W ( L ) , . . . , f ( 2 ) ( W ( 2 ) f ( 1 ) ( W ( 1 ) x ) ) , . . . ) ( 1 ) where x , ŷ , W ( l ) , and f ( l ) are an input vector , an output vector , the weight matrix , and the nonlinear activation function at the lth layer , respectively . Nonlinearity is necessary since without it , an MLP will collapse into one matrix . The choice of a nonlinear function is a hyperparameter . ReLU ( Nair & Hinton , 2010 ) is one of the most widely used activation functions . In a supervised setup , where we have a pair of a data sample x ( i ) and a label y ( i ) , we aim to reduce the loss ( distance ) between the model output ŷ ( i ) and the label y ( i ) . The loss , LW ( 1 ) , ... , W ( L ) ( ŷ ( i ) , y ( i ) ) , is parameterized by the weights of all L layers . If we have m pairs of data samples and labels , { ( x ( 1 ) , y ( 1 ) ) , . . . , ( x ( m ) , y ( m ) ) } , the objective is to minimize the expectation of the loss ( See Equation 2 ) . Ex , y∼p̂dataLW ( 1 ) , ... , W ( L ) ( x , y ) = 1 m m∑ i=1 LW ( 1 ) , ... , W ( L ) ( x ( i ) , y ( i ) ) ( 2 ) Since m can be a large number , we typically batch the m data samples into multiple smaller batches and perform stochastic gradient descent to speed up the optimization process and to reduce memory consumption . Backpropagation ( Rumelhart et al. , 1986 ) is a useful tool to perform gradient descent . This allows us to easily differentiate the loss function with respect to the weights of all L layers , taking advantage of the chain rule . So far , we talked about the basics of MLPs and how to optimize them in a supervised setup . In the past decade , there have been many works to improve deep neural network architectures and their performances . In the next section , we ’ ll go through some of the works that can be used to generalize MLPs . 2 RELATED WORK . 2.1 DROPOUT . Dropout ( Srivastava et al. , 2014 ) was introduced to prevent neural networks from overfitting . Every neuron in layer l − 1 has probability 1 − p of being present . That is , with the probability of p , a neuron is dropped out and the weights that connect to the neurons in layer l will not play a role . p is a hyperparameter whose optimum value can depend on the data , neural network architecture , etc . Dropout makes the neural network stochastic . During training , this behavior is desired since this effectively trains many neural networks that share most of the weights . Obviously , the weights that are dropped out won ’ t be shared between the sampled neural networks . This stochasticity works as a regularizer which prevents overfitting . The stochasticity isn ’ t always desired at test time . In order to make it deterministic , the weights are scaled down by multiplying by p. That is , every weight that once was dropped out during training will be used at test time , but its magnitude is scaled down to account for the fact that it used to be not present during training . In Bayesian statistics , the trained neural network ( s ) can be seen as a posterior predictive distribution . The authors compare the performance between the Monte-Carlo model averaging and the weight scaling . They empirically show that the difference is minimal . In some situations , the stochasticity of a neural network is desired , especially if we want to estimate the uncertainty of model predictions . Unlike regular neural networks , where the trained weights are deterministic , Bayesian neural networks learn the distributions of weights , which can capture uncertainty ( Tishby et al. , 1989 ) . At test time , the posterior predictive distribution is used for inference ( See Equation 3 ) . p ( y | x , X , Y ) = ∫ p ( y | x , X , Y , w ) p ( w | X , Y ) dw ( 3 ) where w = { W ( i ) } Li=1 . Equation 3 is intractable . When a regular neural network has dropouts , it ’ s possible to approximate the posterior predictive distribution by performing T stochastic forward passes and averaging the results . This is called ” Monte-Carlo dropout ” ( Gal & Ghahramani , 2016 ) . In practice , the difference between the Monte-Carlo model averaging and weight scaling is minimal . However , we can use the variance or entropy of the results of T stochastic forward passes to estimate the model uncertainty . 2.2 BATCH NORMALIZATION . Batch normalization was first introduced to address the internal covariate shift problem ( Ioffe & Szegedy , 2015 ) . Normalization is done simply by subtracting the batch mean from every element in the batch and dividing it by the batch standard deviation . It ’ s empirically shown that batch normalization allows higher learning rates to be used , and thus leading to faster convergence . 2.3 WHITENING INPUTS OF NEURAL NETWORKS . Whitening decorrelates and normalizes inputs . Instead of using computationally expensive existing techniques , such as ICA ( independent component analysis ) ( Hyvärinen & Oja , 2000 ) , it was shown that using a batch normalization layer followed by a dropout layer can also achieve whitening ( Chen et al. , 2019 ) . The authors called this the IC ( Independent-Component ) layer . They argued that their IC layer can reduce the mutual information and correlation between the neurons by a factor of p2 and p , respectively , where p is the dropout probability . The authors suggested that an IC layer should be placed before a weight layer . They empirically showed that modifying ResNet ( He et al. , 2016 ) to include IC layers led to a more stable training process , faster convergence , and better convergence limit . 2.4 SKIP CONNECTIONS . Skip connections were first introduced in convolutional neural networks ( CNNs ) ( He et al. , 2016 ) . Before then , having deeper CNNs with more layers didn ’ t necessarily lead to better performance , but rather they suffered from the degradation problem . ResNets solved this problem by introducing skip connections . They managed to train deep CNNs , even with 152 layers , which wasn ’ t seen before , and showed great results on some computer vision benchmark datasets . 3 METHODOLOGY . 3.1 MOTIVATION . We want to take advantage of the previous works on whitening neural network inputs ( Chen et al. , 2019 ) and skip connections ( He et al. , 2016 ) , and apply them to MLPs . In each of their original papers , the ideas were tested on CNNs , not on MLPs . However , considering that both a fully connected layer and a convolutional layer are a linear function of f that satisfies Equation 4 , and that their main job is feature extraction , we assume that MLPs can also benefit from them . f ( x+ y ) = f ( x ) + f ( y ) and f ( ax ) = af ( x ) ( 4 ) The biggest difference between a convolutional layer and a fully connected layer is that the former works locally , but the latter works globally . Also , even the latest neural network architectures , such as the Transformer ( Vaswani et al. , 2017 ) , don ’ t use IC layers within their fully connected layers . They can potentially benefit from our work . 3.2 NEURAL NETWORK ARCHITECTURE . Our MLP closely follows ( Chen et al. , 2019 ) and ( He et al. , 2016 ) . Our MLP is composed of a series of two basic building blocks . One is called the residual block and the other is called the downsample block . The residual block starts with an IC layer , followed by a fully connected layer and ReLU nonlinearity . This is repeated twice , as it was done in the original ResNet paper ( He et al. , 2016 ) . The skip connection is added before the last ReLU layer ( See Equation 5 ) . y = ReLU ( W ( 2 ) Dropout ( BN ( ReLU ( W ( 1 ) Dropout ( BN ( x ) ) ) ) ) + x ) ( 5 ) where x is an input vector to the residual block , BN is a batch normalization layer , Dropout is a dropout layer , ReLU is a ReLU nonlinearity , W1 is the first fully connected layer , and W2 is the second fully connected layer . This building preserves the dimension of the input vector . That is , x , y ∈ RN and W ( 1 ) , W ( 2 ) ∈ RN×N . This is done intentionally so that the skip connections can be made easily . The downsample block also starts with an IC layer , followed by a fully connected layer and ReLU nonlinearity ( See Equation 6 ) . y = ReLU ( WDropout ( BN ( x ) ) ) ( 6 ) The dimension is halved after the weight matrix W . That is , x ∈ RN , W ∈ RN2 ×N , and y ∈ RN2 . This block reduces the number of features so that the network can condense information which will later be used for classification or regression . After a series of the two building blocks , one fully connected layer is appended at the end for the regression or classification task . We will only carry out classification tasks in our experiments for simplicity . Therefore , the softmax function is used for the last nonlinearity and the cross-entropy loss will be used for the loss function in Equation 2 . Although our MLP consists of the functions and layers that already exist , we haven ’ t yet found any works that look exactly like ours . The most similar works were found at ( Martinez et al. , 2017 ) and ( Siravenha et al. , 2019 ) . ( Martinez et al. , 2017 ) used an MLP , whose order is FC → BN → ReLU → Dropout→ FC → BN → ReLU → Dropout , where FC stands for fully-connected . A skip connection is added after the last Dropout . Since ReLU is added between BN and Dropout , their MLP architecture does not benefit from whitening . ( Siravenha et al. , 2019 ) used an MLP , whose order is FC → BN → Dropout → FC → BN → Dropout → ReLU , where FC stands for fully-connected . A skip connection is added before the last ReLU . Their block does include BN → Dropout→ FC , like our MLP . However , there is no nonlinearity after the FC . Without nonlinearity after a fully-connected layer , it won ’ t benefit from potentially more complex decision boundaries . Since our neural network architecture includes dropouts , we can also take advantage of the MonteCarlo dropout ( Gal & Ghahramani , 2016 ) to model the prediction uncertainty that a model makes on test examples .

Summary: This paper proposes a general architecture for MLPs. The main ingredients are skip connections and whitening. These methods have been proven to be effective in CNNs. The authors show that MLPs do benefit from them as well. The proposed skip connection (SO,SI - skip-out, skip-in) is added before the last ReLu (R) of each block. The whitening (BND) is implemented by a batch normalization (BN) layer followed by a dropout (D) layer before the fully-connected layer (FC). The proposed block is then: SO-BND-FC-R-BND-FC-SI-R-BND-FCS-R. While FC has the same number of input as output, FCS halves its number of output. The experimental section shows the performance of such MLPs, ablating skips and whitenings. Two datasets of faces are used (IMDB-Wiki and Adience) with 3 classification task of age and gender recognition.