Year: 2021

Abstract:
A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g., faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/anonymous.

heading: 1 INTRODUCTION

heading: 1.1 MLP AND ITS TRAINING OBJECTIVE
text: A multilayer perceptron (MLP) is one of the simplest and the oldest artificial neural network architectures. Its origin dates back to 1958 (Rosenblatt, 1958). Its basic building blocks are linear regression and nonlinear activation functions. One can stack them up to create an MLP with L layers (See Equation 1).
ŷ = f (L)(W (L), . . . , f (2)(W (2)f (1)(W (1)x)), . . . ) (1)
where x, ŷ, W (l), and f (l) are an input vector, an output vector, the weight matrix, and the nonlinear activation function at the lth layer, respectively. Nonlinearity is necessary since without it, an MLP will collapse into one matrix. The choice of a nonlinear function is a hyperparameter. ReLU (Nair & Hinton, 2010) is one of the most widely used activation functions.
In a supervised setup, where we have a pair of a data sample x(i) and a label y(i), we aim to reduce the loss (distance) between the model output ŷ(i) and the label y(i). The loss, LW (1),...,W (L)(ŷ
(i),y(i)), is parameterized by the weights of all L layers. If we have m pairs of data samples and labels, {(x(1),y(1)), . . . , (x(m),y(m))}, the objective is to minimize the expectation of the loss (See Equation 2).
Ex,y∼p̂dataLW (1),...,W (L)(x,y) = 1
m m∑ i=1 LW (1),...,W (L)(x (i),y(i)) (2)
Since m can be a large number, we typically batch the m data samples into multiple smaller batches and perform stochastic gradient descent to speed up the optimization process and to reduce memory consumption.
Backpropagation (Rumelhart et al., 1986) is a useful tool to perform gradient descent. This allows us to easily differentiate the loss function with respect to the weights of all L layers, taking advantage of the chain rule.
So far, we talked about the basics of MLPs and how to optimize them in a supervised setup. In the past decade, there have been many works to improve deep neural network architectures and their
performances. In the next section, we’ll go through some of the works that can be used to generalize MLPs.

heading: 2 RELATED WORK

heading: 2.1 DROPOUT
text: Dropout (Srivastava et al., 2014) was introduced to prevent neural networks from overfitting. Every neuron in layer l − 1 has probability 1 − p of being present. That is, with the probability of p, a neuron is dropped out and the weights that connect to the neurons in layer l will not play a role. p is a hyperparameter whose optimum value can depend on the data, neural network architecture, etc.
Dropout makes the neural network stochastic. During training, this behavior is desired since this effectively trains many neural networks that share most of the weights. Obviously, the weights that are dropped out won’t be shared between the sampled neural networks. This stochasticity works as a regularizer which prevents overfitting.
The stochasticity isn’t always desired at test time. In order to make it deterministic, the weights are scaled down by multiplying by p. That is, every weight that once was dropped out during training will be used at test time, but its magnitude is scaled down to account for the fact that it used to be not present during training.
In Bayesian statistics, the trained neural network(s) can be seen as a posterior predictive distribution. The authors compare the performance between the Monte-Carlo model averaging and the weight scaling. They empirically show that the difference is minimal.
In some situations, the stochasticity of a neural network is desired, especially if we want to estimate the uncertainty of model predictions. Unlike regular neural networks, where the trained weights are deterministic, Bayesian neural networks learn the distributions of weights, which can capture uncertainty (Tishby et al., 1989). At test time, the posterior predictive distribution is used for inference (See Equation 3).
p(y | x,X,Y) = ∫ p(y | x,X,Y,w)p(w | X,Y)dw (3)
where w = {W(i)}Li=1. Equation 3 is intractable. When a regular neural network has dropouts, it’s possible to approximate the posterior predictive distribution by performing T stochastic forward passes and averaging the results. This is called ”Monte-Carlo dropout” (Gal & Ghahramani, 2016). In practice, the difference between the Monte-Carlo model averaging and weight scaling is minimal. However, we can use the variance or entropy of the results of T stochastic forward passes to estimate the model uncertainty.

heading: 2.2 BATCH NORMALIZATION
text: Batch normalization was first introduced to address the internal covariate shift problem (Ioffe & Szegedy, 2015). Normalization is done simply by subtracting the batch mean from every element in the batch and dividing it by the batch standard deviation. It’s empirically shown that batch normalization allows higher learning rates to be used, and thus leading to faster convergence.

heading: 2.3 WHITENING INPUTS OF NEURAL NETWORKS
text: Whitening decorrelates and normalizes inputs. Instead of using computationally expensive existing techniques, such as ICA (independent component analysis) (Hyvärinen & Oja, 2000), it was shown that using a batch normalization layer followed by a dropout layer can also achieve whitening (Chen et al., 2019). The authors called this the IC (Independent-Component) layer. They argued that their IC layer can reduce the mutual information and correlation between the neurons by a factor of p2 and p, respectively, where p is the dropout probability.
The authors suggested that an IC layer should be placed before a weight layer. They empirically showed that modifying ResNet (He et al., 2016) to include IC layers led to a more stable training process, faster convergence, and better convergence limit.

heading: 2.4 SKIP CONNECTIONS
text: Skip connections were first introduced in convolutional neural networks (CNNs) (He et al., 2016). Before then, having deeper CNNs with more layers didn’t necessarily lead to better performance, but rather they suffered from the degradation problem. ResNets solved this problem by introducing skip connections. They managed to train deep CNNs, even with 152 layers, which wasn’t seen before, and showed great results on some computer vision benchmark datasets.

heading: 3 METHODOLOGY

heading: 3.1 MOTIVATION
text: We want to take advantage of the previous works on whitening neural network inputs (Chen et al., 2019) and skip connections (He et al., 2016), and apply them to MLPs. In each of their original papers, the ideas were tested on CNNs, not on MLPs. However, considering that both a fully connected layer and a convolutional layer are a linear function of f that satisfies Equation 4, and that their main job is feature extraction, we assume that MLPs can also benefit from them.
f(x+ y) = f(x) + f(y) and f(ax) = af(x) (4)
The biggest difference between a convolutional layer and a fully connected layer is that the former works locally, but the latter works globally.
Also, even the latest neural network architectures, such as the Transformer (Vaswani et al., 2017), don’t use IC layers within their fully connected layers. They can potentially benefit from our work.

heading: 3.2 NEURAL NETWORK ARCHITECTURE
text: Our MLP closely follows (Chen et al., 2019) and (He et al., 2016). Our MLP is composed of a series of two basic building blocks. One is called the residual block and the other is called the downsample block.
The residual block starts with an IC layer, followed by a fully connected layer and ReLU nonlinearity. This is repeated twice, as it was done in the original ResNet paper (He et al., 2016). The skip connection is added before the last ReLU layer (See Equation 5).
y = ReLU(W (2)Dropout(BN(ReLU(W (1)Dropout(BN(x))))) + x) (5)
where x is an input vector to the residual block, BN is a batch normalization layer, Dropout is a dropout layer, ReLU is a ReLU nonlinearity, W1 is the first fully connected layer, and W2 is the second fully connected layer. This building preserves the dimension of the input vector. That is, x,y ∈ RN and W (1),W (2) ∈ RN×N . This is done intentionally so that the skip connections can be made easily.
The downsample block also starts with an IC layer, followed by a fully connected layer and ReLU nonlinearity (See Equation 6).
y = ReLU(WDropout(BN(x))) (6)
The dimension is halved after the weight matrix W . That is, x ∈ RN , W ∈ RN2 ×N , and y ∈ RN2 . This block reduces the number of features so that the network can condense information which will later be used for classification or regression.
After a series of the two building blocks, one fully connected layer is appended at the end for the regression or classification task. We will only carry out classification tasks in our experiments for
simplicity. Therefore, the softmax function is used for the last nonlinearity and the cross-entropy loss will be used for the loss function in Equation 2.
Although our MLP consists of the functions and layers that already exist, we haven’t yet found any works that look exactly like ours. The most similar works were found at (Martinez et al., 2017) and (Siravenha et al., 2019).
(Martinez et al., 2017) used an MLP, whose order is FC → BN → ReLU → Dropout→ FC → BN → ReLU → Dropout, where FC stands for fully-connected. A skip connection is added after the last Dropout. Since ReLU is added between BN and Dropout, their MLP architecture does not benefit from whitening.
(Siravenha et al., 2019) used an MLP, whose order is FC → BN → Dropout → FC → BN → Dropout → ReLU , where FC stands for fully-connected. A skip connection is added before the last ReLU . Their block does include BN → Dropout→ FC, like our MLP. However, there is no nonlinearity after the FC. Without nonlinearity after a fully-connected layer, it won’t benefit from potentially more complex decision boundaries.
Since our neural network architecture includes dropouts, we can also take advantage of the MonteCarlo dropout (Gal & Ghahramani, 2016) to model the prediction uncertainty that a model makes on test examples.

heading: 4 EXPERIMENTS

heading: 4.1 AGE AND GENDER DATASETS
text: To test our method in Section 3.2, we chose the age and gender datasets (Eidinger et al., 2014) (Rothe et al., 2015). The Adience dataset (Eidinger et al., 2014) is a five-fold cross-validation dataset. This dataset comes from Flickr album photos and is the most commonly used dataset for age and gender estimation. You can check the leaderboard at https://paperswithcode.com/. The IMDB-WIKI dataset (Rothe et al., 2015) is bigger than Adience. It comes from celebrity images from IMDB and Wikipedia. We’ll report the peformance metrics (i.e., the cross entropy loss and accuracy) on the Adience dataset with five-fold cross-validation so that we can compare our results with others. The IMDB-WIKI dataset is only used for hyperparameter tuning and pretraining.
The original images from the Adience dataset have the size of 816× 816× 3 pixels. Although CNNs are known to be good at extracting spatial features from such data, the point of our experiment is to test and compare the MLP architectures. Therefore, we transform the images to fixed-size vectors.
We use RetinaFace (Deng et al., 2020) to get the bounding box and the five facial landmarks, with which we can affine-transform the images to arrays of size 112 × 112 × 3 pixels (See Figure 1). Besides reducing the size of the data, this also has a benefit of removing background noise.
Next, we use ArcFace (Deng et al., 2019) to further reduce the size and extract features from the affine-transformed images. ArcFace is a powerful face-embedding extractor which can be used for face recognition. This transformation transforms 112 × 112 × 3 pixel arrays to 512-dimensional
vectors. It was shown in (Swaminathan et al., 2020) that a classifier can be built on such vectors to classify gender. We take this approach further to even classify age as well.
Table 1 shows the number of data samples before and after removal, along with the gender distribution. Table 2 shows the number of data samples removed and the reason they were removed. Figure 2 shows the age distribution of each dataset. The IMDB-WIKI dataset has a more fine-grained age distribution than the Adience dataset.

heading: 4.2 TRAINING AND HYPERPARAMETERS
text: We first pretrain three MLPs with the IMDB-WIKI dataset:
1. 2-class gender classification.
2. 8-class age classification1.
3. 101-class age classification.
There are a number of hyperparameters to be considered to train the models. Some of them are just fixed by following the best practices. For example, we fix the dropout rate to 0.05 as it was shown in (Chen et al., 2019). We use AdamW (Loshchilov & Hutter, 2019) for the optimizer, as it was proven to be better than using vanilla Adam (Kingma & Ba, 2015). And we use ExponentialLR for learning rate scheduling. Now the remaining hyperparameters to be determined are the number
1The 101 age categories are converted to the 8 categories by projecting them to the nearest numbers of the Adience dataset.
of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay. They were determined using automatic hyperparameter tuning with Ray Tune (Liaw et al., 2018). We also use automatic mixed precision to speed up training. 10% of the IMDB-WIKI datset was held-out from the other 90% so that weights are backpropagated from the 90% of the data, but the best hyperparameters are determined that have the lowest cross-entropy loss on the 10%.
As for the 2-class gender classification task, the number of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay were chosen to be 4, 4, 512, 2e− 2, 4e− 3, and 6e− 2, respectively. As for the 8-class and 101-class age classification tasks, the number of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay were chosen to be 2, 2, 128, 2e− 3, 1e− 4, and 4e− 1, respectively. Figure 3 shows the neural network architecture of the 2-class gender classification model.
After the models are trained on the IMDB-WIKI dataset, we fine-tune the models on the Adience dataset. This fine-tuning is running five-fold cross-validation five times to get the least biased results. In this case, the 101-class age classification task should be turned into an 8-class classification task, since the Adience dataset only has 8 age classes. We didn’t specifically change the network architecture, since it should be able to learn such a thing by itself by updating its weights.

heading: 4.3 RESULTS
text: Table 3 shows the loss and accuracy of all of the trained models. The training stopped if validation loss didn’t drop for five consecutive epochs, to reduce training time. All of the models are pretty light-weight. The number of parameters is all between 800, 000 and 900, 000. Training time was very fast. We trained everything on our local machine with an RTX 2060 max-q 6 GB. Automatic hyperparameter tuning, pretraining on the IMDB-WIKI dataset, and five times of five-fold crossvalidation on the Adience dataset, took about 1 hour and 10 minutes, 5 minutes, and 13 minutes, respectively.
Although our models were not specifically designed to have the best age and gender classification performance (e.g., A CNN on raw image data probably would have been a better choice), but to test out different MLP architectures, the performance is very competitive, especially for the gender classification task. However, this table is not so easy to interpret. Good performance on the validation split doesn’t necessarily lead to good performance on the test split. Also, low cross-entropy loss on the test split somehow doesn’t lead to higher accuracy on the test split either.
Therefore, we made the cross-entropy loss vs. epochs plots, in Figure 4. As you can observe, the biggest performance gap between the model with both the IC layer and skip connections and the models with one or both of them removed is displayed in the validation loss with pretrained initialization. The validation loss is consistently lower than the others when the network is initialized with pretrained models. This shows that a network trained on one dataset with IC layers and skip connections helps to generalize to other datasets.
2We didn’t add the IC layer (batch norm and dropout) in the beginning of the neural network. The face embedding vectors themselves were supposed to have their own meanings (e.g., the L2 norm is fixed to 1) and we didn’t want to harm them by whitening.
As mentioned in 2.1, since our models with the IC layers include dropout, we can use the MonteCarlo dropout to estimate the prediction uncertainty that a model makes. We ran 512 forward passes and computed the entropy values. Obviously the more forward passes we make, the better the estimate is, since it’s sampling from a distribution. The authors in (Gal & Ghahramani, 2016) used 100 forward passes, but since our model is very light-weight, running 512 passes on a CPU was not a problem. See Figure 5 for the faces with low and high entropy. Interestingly, four out of the ten highest entropy faces are either baby or kid faces, which has low frequency in the Adience dataset.

heading: 5 CONCLUSION
text: MLPs have a long history in neural networks. Although they are ubiquitously used, a systematic research to generalize them has been missing. By placing an IC (Independent-Component) layer to whiten inputs before every fully-connected layer, followed by ReLU nonlinearity, and adding skip connections, our MLP empirically showed it can have a better convergence limit, when its weights are initialized from a pretrained network. Also, since our MLP architecture includes dropouts, we were able to show that our MLP can estimate the uncertainty of model predictions. In the future work, we want to test our idea on different datasets to see how well it can generalize.

Review summaries:
The paper proposes a network architecture for MLP networks. The authors build upon work by Chen et al. (2019) to achieve whitening by batchnorm followed by dropout (IC Layer). Further they use skip-connections (He et al., 2016) in their architecture. The authors tested their approach in an ablation study for gender and age classification. With dropout included in their architecture, they computed the entropy values, which showed high values for some baby or kid faces, that were infrequent in the dataset.

This paper proposes a general architecture for MLPs. The main ingredients are skip connections and whitening. These methods have been proven to be effective in CNNs. The authors show that MLPs do benefit from them as well. The proposed skip connection (SO,SI - skip-out, skip-in) is added before the last ReLu (R) of each block. The whitening (BND) is implemented by a batch normalization (BN) layer followed by a dropout (D) layer before the fully-connected layer (FC). The proposed block is then: SO-BND-FC-R-BND-FC-SI-R-BND-FCS-R. While FC has the same number of input as output, FCS halves its number of output. The experimental section shows the performance of such MLPs, ablating skips and whitenings. Two datasets of faces are used (IMDB-Wiki and Adience) with 3 classification task of age and gender recognition.