Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections
Authors: Taewoon Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: MLP
batch normalization
dropout
residual connections
Bayesian inference

Abstract: A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g., faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/anonymous.

Under review as a conference paper at ICLR 2022
GENERALIZING MLPs WITH DROPOUTS, NORMALIZATION, AND SKIP CONNECTIONS
BATCH
Anonymous authors Paper under double-blind review
ABSTRACT
A multilayer perceptron (MLP) is typically made of multiple fully connected lay- ers with nonlinear activation functions. There have been several approaches to make them better (e-g., faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connec- tions, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker im- ages at htthtps://ggttee thub. com/ wtt./oom/////oa.. anonymous
INTRODUCTION
1.1 MLP AND ITS TRAINING OBJECTIVE
multilayer perceptron (MLP) is one the simplest and the oldest artificial neural network ar- chitectures. Its origin dates back to 1958 (Rosenblati) 1958). Its basic building blocks are linear regression and nonlinear activation functions. One can stack them up to create an MLP with L layers (See Equation T.
i ((( (W(L), (W) (w))))))))....
where ÿ. ( and (() are an input vector, an output vector, the weight matrix, and the nonlinear activation function at the (th layer, respectively. Nonlinearity is necessary since without it, an MLP will collapse into one matrix. The choice of a nonlinear function is a hyperparameter. ReLU (Nair & Hinton 2010 is one of the most widely used activation functions.
In supervised setup, where we have pair of a data sample a+()) and a label y (+) we aim reduce the loss (distance) between the model output g() and the label ( The loss,
Lwo,..,,wie (z(),, (")), parameterized by the weights of all L layers. If we have pairs of data samples and labels, { (xx y()), (m)))), (m)) }. the objective to minimize the expectation of the loss (See Equation?? 2).
Ex.y-faar Iww)...... (x) v)
Σ Two, was (2)()) ())
Since m can be large number, we typically batch the m data samples into multiple smaller batches and perform stochastic gradient descent to speed up the optimization process and to reduce memory consumption.
Backpropagation (Rumelhart al. 1986 is useful tool perform gradient descent. This allows us to easily differentiate the loss function with respect to the weights of all L layers, taking advantage of the chain rule.
So far, we talked about the basics of MLPs and how to optimize them in a supervised setup. In the past decade, there have been many works improve deep neural network architectures and their

---

Under review as a conference paper at ICLR 2022
performances. In the next section, we Ill go through some of the works that can be used to generalize MLPs.
2
RELATED WORK
2.1 DROPOUT
Dropout (Srivastava et al. 2014) was introduced to prevent neural networks from overfitting. Every neuron in layer / 7- has probability 1- of being present. That is, with the probability of p. neuron is dropped out and the weights that connect to the neurons in layer will not play role. is hyperparameter whose optimum value can depend on the data, neural network architecture, etc.
Dropout makes the neural network stochastic. During training, this behavior is desired since this effectively trains many neural networks that share most of the weights. Obviously, the weights that are dropped out won't be shared between the sampled neural networks. This stochasticity works as a regularizer which prevents overfitting.
The stochasticity isn't always desired at test time. In order to make it deterministic, the weights are scaled down by multiplying by p. That is, every weight that once was dropped out during training will be used at test time, but its magnitude is scaled down to account for the fact that it used to be not present during training.
In Bayesian statistics, the trained neural network(s) can be seen as posterior predictive distribution. The authors compare the performance between the Monte-Carlo model averaging and the weight scaling. They empirically show that the difference is minimal.
In some situations, the stochasticity a neural network is desired, especially if we want to estimate the uncertainty of model predictions. Unlike regular neural networks, where the trained weights deterministic, Bayesian neural networks learn the distributions weights, which can capture uncer- tainty (Tishby et al. 1989). At test time, the posterior predictive distribution is used for inference (See Equation
ply xXx ply x, X, w))(w X, Y)dw
where w { [W Equation intractable. When regular neural network has dropouts, it's possible to approximate the posterior predictive distribution by performing T stochastic forward passes and averaging the results. This called "Monte- Carlo dropout" " (Gal Ghahraman] 2016). In practice, the difference between the Monte-Carie model averaging and weight scaling is minimal. However, we can use the variance or entropy of the results of stochastic forward passes to estimate the model uncertainty.
2.2 BATCH NORMALIZATION
Batch normalization was first introduced to address the internal covariate shift problem (loffe & Szegedy| 2015). Normalization is done simply by subtracting the batch mean from every element in the batch and dividing it by the batch standard deviation. It's empirically shown that batch normal- ization allows higher learning rates to be used, and thus leading to faster convergence.
2.3 WHITENING INPUTS OF NEURAL NETWORKS
Whitening decorrelates and normalizes inputs. Instead of using computationally expensive existing techniques, such as ICA (independent component analysis) (Hyvärinen & Oja 2000), it was shown that using batch normalization layer followed by a dropout layer can also achieve whitening (Chen layer. They argued that their
et al. 2019). The authors called this the IC IC layer can reduce the mutual information and correlation between the neurons by factor of p and p, respectively, where is the dropout probability.
2

---

Under review as a conference paper at ICLR 2022
The authors suggested that an IC layer should be placed before weight layer. They empirically showed that modifying ResNet (He et al. 2016) to include IC layers led to more stable training process, faster convergence, and better convergence limit.
2.4 SKIP CONNECTIONS
Skip connections were first introduced in convolutional neural networks (CNNs) He et al. 2016). Before then, having deeper CNNs with more layers didn't 't necessarily lead to better performance, but rather they suffered from the degradation problem. ResNets solved this problem by introducing skip connections. They managed train deep CNNs, even with 152 layers, which wasn't 't seen before, and showed great results on some computer vision benchmark datascts.
3 METHODOLOGY
3.1 MOTIVATION
We want to take advantage of the previous works whitening neural network inputs (Chen al., 2019) and skip connections (He et al., 2016), and apply them to MLPs. In each of their original papers, the ideas were tested on CNNs, nected layer and convolutional layer are a linear function of f that satisfies Equation 4 and that their main job is feature extraction, we assume that MLPs can also benefit from them.
on MLPs. However, considering that both a fully con-
f(x y) f(x) + f(y) and f(ax) = af
The biggest difference between a convolutional layer and a fully connected layer is that the former works locally, but the latter works globally.
Also, even the latest neural network architectures, such as the Transformer (Vaswani et al. 2017). don't use IC layers within their fully connected layers. They can potentially benefit from work.
3.2 NEURAL NETWORK ARCHITECTURE
Our MLP closely follows (Chen et al. 2019) and (He et al. 2016). Our MIP is composed of a series of two basic building blocks. One called the residual block and the other is called the downsample block.
The residual block starts with an IC layer, followed by fully connected layer and ReLU nonlinear- ity. This is repeated twice, as it was done in the original ResNet paper (He et al. 2016). The skip connection added before the last ReLU layer (See Equation 5p.
y ReLU (2 (B (Bopo + x))
(5)
where æ is an imput vector to the residual block, BN is batch normalization layer, Dropout is a dropout layer, ReLU is a ReLU nonlinearity, W, is the first fully connected layer, and W2 is the second fully connected layer. This building preserves the dimension of the input vector. That is, x, y E N and W11), ((2) E NxN, This is done intentionally so that the skip connections can be made easily.
The downsample block also starts with an IC layer, followed by a fully connected layer and ReLU nonlinearity (See Equation
y ReLU (W (oropppot(BNaa
The dimension is halved after the weight matrix W. That is, R. W E x and $ This block reduces the number of features so that the network can condense information which will later be used for classification regression.
After a series of the two building blocks, one fully connected layer is appended at the end for the regression or classification task. We will only carry out classification tasks in our experiments for
3

---

Under review as a conference paper at ICLR 2022
simplicity. Therefore, the softmax function is used for the last nonlinearity and the eross-entropy loss will used for the loss function Equation 2
Although MLP consists of the functions and layers that already exist, we haven't It yet found any works that look exactly like ours, The most similar works were found at (Martinez et al. 2017) and 2019).
Siravenha et al.
Martinez et al. BN - ReLU + Dropout, where FC stands for conlecconnected. A skip connection is added after the last Dropoul. Since ReLU is added between BN and Dropoul, their MLP architecture does not benefit from whitening.
2017
used an MLP, whose order is FC + BN + ReLU + Dropout + FC -
Siravenha et al. 2019) used an MLP, whose order is FC - BN + Dropout + FC BN - Dropout + LU, where FC stands for fully-connected. A skip connection is added before the last ReLU' Their block does include BN + Dropout + FC, like our MLP. However, there is no nonlinearity after the FC. Without nonlinearity after fully-connected layer, won't benefit from potentially more complex decision boundaries.
Since our neural network architecture includes dropouts, we can also take advantage of the Monte- Carlo dropout (Gal & Ghahramani 2016) to model the prediction uncertainty that model makes on test examples.
4 EXPERIMENTS
4.1 AGE AND GENDER DATASETS
To test our method in Section we chose the age and gender datasets (Eidinger et al. 20144) (Rothe) et al., 2015). The Adience dataset (Eidinger et al. 2014) is five-fold cross-validation dataset. This dataset comes from Flickr album photos and the most commonly used dataset for age and gender estimation. You can check the leaderboard at pllttps:: http://pppssitttthhoooeso
com/ The IMDB-WIKI dataset RRtthe et al. 2015) is bigger than Adience. comes from celebrity images from IMDB and Wikipedia. We'll I1 report the peformance metrics (i.e., the
entropy loss and accuracy) on the Adience dataset with five-fold cross-vass validation so that we can compare our results with others. The IMDB- WIKI dataset is only used for hyperparameter tuming and pretraining.
The original images from the Adience dataset have the size of 816 816 x pixels. Although CNNs known to be good extracting spatial features from such data, the point of our experiment is to test and compare the MLP architectures. Therefore, we transform the images to fixed-size vectors.
We use RetinaFace (Deng et al. 2020) to get the bounding box and the five facial landmarks, with which we can affine-transform the images to arrays of size 112 x 112 x 3 pixels (See Figure Besides reducing the size of the data, this also has benefit of removing background noise.
(a) Original image, 816 816 3 pixels
(b) frfine-tranffommed to 112 x 112 x pixels
Figure 1: A sample image from the Adience dataset. Instead of using the original images, such as Figure we use affine-transformed images, as in Figure lb
Next, ArcFace (Deng et al. 2019 to further reduce the size and extract features from the transfomaadf images. ArcFace is a powerful face-embedding extractor which can used for face recognition. This transformation transforms 112 112 x 3 pixel arrays 512-dimensional

---

Under review as a conference paper at ICLR 2022
vectors. It was shown in (Swaminathan al. 2020) that classifier can be built such vectors to classify gender. We take this approach further to even classify age as well.
Table shows the number of data samples before and after removal, along with the gender distribu-
tion. Table shows the number of data samples removed and the reason they were removed. Figure
2 shows the age distribution of each dataset. The IMDB-WIKI dataset has fine-grained age
distribution than the Adience dataset.
Gender
Dataset
number of images before removal
number of images after removal
femalc
male
Adience
19,370
17,055 (88.04% original)
9,103
7,952
IMDB-WIK]
523,051
398,251 (76.14 14% of original)
163,228
235,023
Table The statistics of the Adience and IMDB- WIKI datasets
Dataset
process image
age found
DO gender found
face detected
had quality (det_seore <0.9)
more than one face
embeddings
total
Adience
0
748
1,170
322
75
2,315 (11,95% of original)
IMDB- -III
33,109
2,471
10,938
24.515
4,283
49,457
27
124,800 (23.86 % of original)
Table 2: The number of images removed
100
(a) The age distribution of the Adience dataset
(b) The age distribution of the IMDB-WIKI dataset
The age distribution of the Adience (Figure 2a and MMDB-WIKE (Figure 2b datasets. Adience has coarse granied 8 age categories (i.e., 0-2, 4-6, 8- 15-20, 25-32, 38-43, 48-53, and 60- (001, while IMBB-WIK has fine-grained 101 age categories (i.c., from 0 to 100).
Figure 2:
4.2 TRAINING AND IYPERPAARMETERS
We first pretrain three MLPs with the IMDB-WIIE dataset:
1. 2-class gender classification.
2. 8-class age classificatior
3. 101-class age classification.
There are a number of hyperparameters to be considered to train the models. Some of them are just fixed by following the best practices. For example, fix the dropout rate to 0.05 it was shown in (Chen et al.] 2019). We use AdamW (Loshchilov & Hutter 2019) for the optimizer, as was ExponentialL for learning rate scheduling. Now the remaining hyperparameters to be determined are the number
proven be better than using vanilla Adam (Kingma & Ba] 2015). And
'The 101 age categories are converted to the 8 categories by projecting them to the nearest numbers of the Adience dataset.
5

---

Under review as a conference paper at ICLR 2022
of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay. They were determined using automatic hyperparameter tuning with Ray Tune (Liaw al. 2018). We also automatic mixed precision to speed up training. 10% of the IMDB- WIKI datset was held-out from the other 90% so that weights backpropagated from the 90% of the data, but the best hyperparameters determined that have the lowest cross-entropy loss on the 10%.
As for the 2-class gender classification task, the number of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay were chosen be 4, 512, 2c 4e and 6e 2, respectively.
As for the 8-class and 101-class age classification tasks, the number of residual blocks, number of downsample blocks, batch size, initial learning rate, weight decay coefficient, and multiplicative factor of learning rate decay were chosen to be 2, 2, 128, 2c 3, lc 4, and 4e 1, respectively.
Figure shows the neural network architecture of the 2-class gender classification model.
NELL
Diigou
Doggedt
HELU
EN
Drupedt
Drogeut
Disputt
Diopeut
Disputt
RULU
Softrar
Figure The neural network architecture of the 2-class gender classification mode
The skip
connections are depicted with the arrows that go over multiple layers.
After the models are trained on the IMDB-WIKI dataset, we fine-tune the models on the Adience dataset. This fine-tuning rumning five-fold cross-validation five times to get the least biased results. In this case, the 10l-class age classification task should be turned into an 8-class classification task, since the Adience dataset only has age classes. We didn't specifically change the network architecture, since it should be able to learn such thing by itself by updating its weights.
4.3 RESULTS
Table 3 shows the loss and accuracy of all of the trained models. The training stopped if validation loss didn't drop for five consecutive epochs, to reduce training time. All of the models are pretty leigh, The number of parameters is all between 800, 000 and 900, 000. Training time was very fast. We trained everything on our local machine with an RTX 2060 max-q 6 GB. Automatic hyperparameter tuning, pretraining the IMDB-WII dataset, and five times of five-fold cross- validation on the Adience dataset, took about hour and 10 minutes, minutes, and 13 minutes, respectively.
Although our models were not specifically designed to have the best age and gender classification performance (c.g., A CNN on raw image data probably would have been better choice), but to test out different MI.P architectures, the performance is very competitive, especially for the gender clas- sification task. However, this table is not so easy interpret. Good performance the validation split doesn't 't necessarily lead to good performance on the test split. Also, low cross-entropy loss on the test split somehow doesn't lead higher accuracy on the test split either.
Therefore, we made the cross-entropy loss vs. epochs plots, in Figure 4 the biggest performance gap between the model with both the IC layer and skip connections and the models with one or both of them removed is displayed in the validation loss with pretrained initialization. The validation loss is consistently lower than the others when the network is initialized with pretrained models. This shows that a network trained on one dataset with IC layers and skip connections helps generalize to other datasets.
As you can observe,
We didn't it add the IC layer (batch norm and dropout) in the beginning of the neural network. The face embedding vectors themselves didn't 't want harm them by whitening.
supposed to have their own meanings (e.g., the 1?2 norm is fixed to 1) and

---

Under review as a conference paper at ICLR 2022
2-300 Clanni
MURON DOOAA
OLT 01100000100
gra ci Eurlin,
MUNON SUU000
000004 10.000
ELTE4A QUUBRRO
BABET GUROO
ROGES HAAAON
RARZZ WURDSO
GOOST MAALIIO
2-dmm
3- prade padiccledioo Ecstios
WOTE ON 1150
DANST ON 50S5
T.lbw grade wea.e.lloooooeee Eoation
7:dks gra doriiiat
00030 00030000077
00805-00000
3-clas pradi: chaiiiraiiss
01100000000
DONY
3-ilas pradi claoifatiins
00005000001
019350000000
ONE AIAOOOS
igo
004422000111
quii
0005-000145
0:130-0000000
00065-00000))
S-clas speciiun claat
011000000128
00185-000900
resiled cllsueg
02345-00000 02305-00071
Digus
1000000
resdidd s-bin.pe classll
03279-933584
(eillkdd selbrrge ciucticatios
resilidd ael clssticatios
rexitid ag clastiic
theccec becherage chveticates
The loss and accuracy of the trained models. The "modified" 8-class age classification models are the ones that were initially trained for the 101-class age classification task, but then repurposed to 8-class classification. Random initialization is done with Kaiming He uniform random initialization (He al. 2015). We five-fold cross validation five times. The reported numbers are the averages of the 25 runs. Early stopping was done to save training time. That's why some trainings lasted longer epochs than the others.
Table 3:
As mentioned in 2.1 since our models with the IC layers include dropout, we can use the Monte- Carlo dropout to estimate the prediction uncertainty that a model makes. We ran 512 forward passes and computed the entropy values. Obviously the more forward passes make, the better the estimate is, since it's sampling from a distribution. The authors (Gal & Ghahramani 2016) used 100 forward passes, but since model is very light-weight, running 512 passes CPU was not a problem. See Figure for the faces with low and high entropy. Interestingly. four out of the ten highest entropy faces either baby or kid faces, which has low frequency in the Adience dataset.
CONCLUSION
MLPPs have long history in neural networks. Although they ubiquitously used, systematic research to generalize them has been missing. By placing an IC [Independent-Componee)) layer to whiten inputs before every fully-connected layer, followed by ReLU nonlinearity, and adding skip connections, our MLP empirically showed it can have a better convergence limit, when its weights are initialized from a pretrained network. Also, since our MLP architecture includes dropouts, we were able to show that our MLP can estimate the uncertainty of model predictions. In the future work, we want to test our idea on different datasets to see how well can generalize.

---

Under review as a conference paper at ICLR 2022
(a) Training loss, random initialization
(b) Validation loss,
(c) Training loss, pretrained initialization
(d) Validation loss, pretrained initialization
random initialization
(e) Training loss, random initialization
(f) Validation loss, random initialization
(g) Training loss, pretrained initialization
(h) Validation loss, pretrained initialization
(i) Training loss, random initialization
(i) Validation loss, random initialization
(k) Training loss, pretrained initialization
(I) Validation loss, pretrained initialization
Figure 4: Cross-entropy loss vs. epoch in training gender and classification, with random and pre- trained initialization. The top, middle, and the bottom rows the 2-class gender, 8-class age, and "modified" 8-class age classification task, respectively. Random initialization is done with Kaiming He uniform random initialization (He al. 2015). We five-fold cross validation five times. The reported numbers are the averages of the 25 runs.
Figure 5: Top row faces are the ten Adience pictures with the lowest entropy. Their entropy values all zero, which means that the model is 100% sure of its predictions. Bottom row faces are the ten Adience ones with the highest entropy. Their entropy values are all close to 0.6931, which is the maximum entropy for binary classification.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
Guangyong Chen, Pengfei Chen, Yujun Shi, Chang- Yu Hsieh, Benben Liao, and Shengyu Zhang. Rethinking the usage of batch normalization and dropout in the training of deep neural networks, 2019.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4685-4694, 2019. doi: 10.1109/CVP22201..0002.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition (CVPR), June 2020.
Eran Eidinger, Roee Enbar, and Tal Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions Information Forensics and Security, 911): 2179, 2014. doi: 10. 2014.2359646.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model un- certainty in deep learning. Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Ma- chine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMIR. URL URL.http:
http: /ttp:////occceesss.eeeee www.:smww.....r.....m / wt/ps///va//aall/ html
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. 2015 IEEE International Conference Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10.1109/CVRR.2016.990
Steven C. Y. Hung. Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. CoRR, abs/19 1bs.099000666,, 2019, URL http::ttaxii......///// org abs/ 1910. 06562)
A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applica- tions. Neural Networks, 13(4):411-430. 2000. ISSN 0893-6080.
doi: https///oi..rr//I URL htttps:// https://www.cr www.://wienceeirec...cc wtwp:/ededeeet.ccccceeeeeee.. www.s/ico/scceeeeee.mooo..eeeeee
1016/5089936008000000002222
atticle/p11/00830000000000
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on In- lernational Conference on Machine Learning Volume 37, ICML' 15, pp. 448-456. JMLR.org, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.66900 2015.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph Gonzalez, and Ion Sto- ica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118, 2018.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Julieta Martinez, Rayat Hossain, J. Romero, and Little. A simple yet effective baseline for 3d human pose estimation. 2017 IEEE International Conference Computer Vision (ICCV), pp- 2659-2668, 2017.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781 9781665589077.

---

Under review as a conference paper at ICLR 2022
F. Rosenblatt. The perceptron: probabilisticc model for information storage and organization in the brain. Psychological review, 65 8:386-408, 1958.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from
single image. In IEEE International Conference on Computer Vision Workshops (ICCVW),
December 2015.
D. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533- 536,, 1986.
Ana CQ Siravenha, Mylena N F Reis, Iraquitan Cordeiro, Renan Arthur Tourinho, Bruno D. Gomes, and Schubert R. Carvalho. Residual mip network for mental fatigue classification in mining workers from brain data. In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pp- 407-412, 2019. doi: 10. 10.1109/BRACIS.. 200.....0..0077...
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1999 1958, 2014. URL http:// wwtp:./rr//ppee///...../
srivastava 1a..mmm11....11111111..mmm html
Avinash Swaminathan, Mridul Chaba, Deepak Kumar Sharma, and Yogesh Chaba. Gender clas- sification using facial embeddings: A novel approach. Procedia Computer Science, 167:2634- 2642, 2020. ISSN 1877-0509. doi: https://bbi.rr//1011100j h1t.0000066..rrro00000.....114 URL https: / //www. www.ecedieniiiiccccooooooo rect. cww/ wtt.:/cccccccceeeee
In-
ternational Conference Computational Intelligence and Data Science.
Naftali Tishby, Esther Levin, and Sara A. Solla. Consistent inference of probabilities in layered networks: Predictions and generalization. In Anon (ed.), I/CNN Int Jr Conf Neural Network, pp. 403-409. Publ by IEEE, December 1989, IJCNN International Joint Conference on Neural Networks: Conference date: 88-06-1989 Through 22-06-1989.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, un- definedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 3 International Conference on Neural Information Processing Systems, NIPS 17, pp. Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 978151086096
Ke Zhang, Na Liu, Xingfang Yuan, Xinyao Guo, Ce Gao, Zhenbing Zhao, and Zhanyu Ma. Fine- grained age estimation the wild with attention Istm networks. IEEE Trans. Cir. and Sys. for Video Technol., 30(9):3140-3155 September 2020. ISSN 1051-8215. doi: 10. 10.1109/TCSVT. 2019.29 09994293 URL httph: http:////oo.o////////// www.10........o.../o.oo 110.//9001100/11111 TCSVT. 2936410
10


Reviews:
[{"review_id": "wxUMPGSyGzx", "reviewer": "Reviewer_7ffG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The primary strength of the paper is the careful experimental study the authors conducted to show the utility of the Independent Component layer and skip connections in building deep feedforward neural networks. However, I believe the paper's contributions are marginal.\n\nThere are a few suggestions in which the authors may try to build from the current version. \n\n1) Can the authors comment on any significant differences observed in the experimental findings from MLPs and CNNs? \n\n2) What are the reasons behind the improvements in performance with skip connections and the IC layer? Instead of just observing the final accuracy and loss numbers, it will be interesting to see if skip connections and IC layer improve performance because of some underlying phenomenon. Examples of such phenomenons can be smoothening the loss landscape or some underlying implicit bias of skip connections. It will be interesting to track down different function properties learned by the neural networks along the trajectory. \n\nAnother question is whether we have faster optimization with the skip connections and the IC layer.\n\n3) The authors can maneuver into studying the properties of the trained network with and without the IC layer and skip connections. E.g., do the trained networks have better adversarial robustness with the IC layer and skip connections.\n\nThe authors can build upon many of these questions to deepen their study on the structure of neural networks.", "summary_of_the_paper": "The authors conduct a careful study on improving the generalization capabilities of multilayer perceptions using dropout, batch normalization, and skip-connections. They show that including an Independent Component layer (using batch normalization and dropout) before every fully connected layer, followed by a ReLU nonlinearity, and adding skip connections, help pretrained MLPs to generalize well on different datasets.", "summary_of_the_review": "The authors have conducted a careful experimental study to build better MLPs. However, in its current form, I believe the paper's contributions are marginal. However, I think the direction of study is pretty exciting and can be extended to study many related exciting directions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bEFPWfrj6K", "reviewer": "Reviewer_qoCZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript itself is clearly written and self-contained. \nThe idea of the authors builds upon successful deep learning techniques. \nBased on these, the authors suggest a new neural network architecture for MLPs. \nTo our point of view the idea of having residual blocks combined with down-projection layers may be of a certain interest, however from an overall point of view the technical novelties of the paper seem limited. \nSince there are no new theoretical analyses, that would justify the proposed network architecture, it would be necessary that the authors provide a convincing experimental evaluation of their approach, which seems to be lacking and which is therefore our major concern with the paper, although it should be mentioned that it seems that the experiments in the publication have been carried out in a technically correct way.\nAt the end of the paper, the authors provide an analysis on MC Dropout with their architecture, which seems interesting, but which would also require more detailed comparisons to the behavior of MC Dropout with other architectures and a more detailed quantitative analysis w.r.t. data that is infrequent in the training sets.\n\n###### Major concern\n\nThe authors mainly only provide an ablation study, but no extensive comparison to other architectures/methods and the results don't look superior to already reported results. \nEspecially, for age classification their results seem worse than the reported result from Zhang et al. (2020).\nThe architecture seems to have better results for gender classification (best reported test accuracy in table seems better than Hung et al. (2019)), however based on the validation result, the corresponding architecture would possibly not have been selected and furthermore this best architecture doesn't use relevant novelties of their architecture idea (no IC).\nFurthermore, it would be interesting, whether the architecture would also work in an end-to-end way well (e.g., at the top of a CNN), and it would be interesting how well the architecture especially works for tabular datasets as e.g., the Tox21 dataset with circular fingerprints (e.g., ECFP features) or e.g., datasets from the UCI machine learning repository.", "summary_of_the_paper": "The paper proposes a network architecture for MLP networks.\nThe authors build upon work by Chen et al. (2019) to achieve whitening by batchnorm followed by dropout (IC Layer).\nFurther they use skip-connections (He et al., 2016) in their architecture.\nThe authors tested their approach in an ablation study for gender and age classification.\nWith dropout included in their architecture, they computed the entropy values, which showed high values\nfor some baby or kid faces, that were infrequent in the dataset.", "summary_of_the_review": "Since the experimental evaluation of the proposed network architecture was not convincing for us, and since there are no new theoretical analyses in the publication, we have doubts on the relevance of this paper for ICLR and therefore recommend rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R5RVq_iVSL", "reviewer": "Reviewer_8xR5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors claim that no-one had proposed the actual block for MLP before. Although several previously published works came close, either the order of layers were slightly different, or they were shown for CNNs. So, potentially, the proposed block architecture could be useful to many practitioners. Unfortunately, the empirical results do not convince one that their particular block is superior. Indeed, only the absence of both skips and whitening is shown to be worse consistently across all tasks. Other combinations offer less clear results, with some winning for some tasks and some other winning for other tasks. Also, the ablation does not include previously published similar blocks (FC-BN-R-D by Martinez et.al. 2017, and FC-BN-D-FC-BN-D-R by Siravenha et.al.2019). Therefore it is not made clear that the particular order of layer in the proposed block is providing any benefits empirically. Furthermore, such claims should be supported across multiple datasets of several types. Here, only 2 similar face datasets are used on 3 similar tasks of age and gender recognitions. \n", "summary_of_the_paper": "This paper proposes a general architecture for MLPs. The main ingredients are skip connections and whitening. These methods have been proven to be effective in CNNs. The authors show that MLPs do benefit from them as well. The proposed skip connection (SO,SI - skip-out, skip-in) is added before the last ReLu (R) of each block. The whitening (BND) is implemented by a batch normalization (BN) layer followed by a dropout (D) layer before the fully-connected layer (FC). The proposed block is then: SO-BND-FC-R-BND-FC-SI-R-BND-FCS-R. While FC has the same number of input as output, FCS halves its number of output. The experimental section shows the performance of such MLPs, ablating skips and whitenings. Two datasets of faces are used (IMDB-Wiki and Adience) with 3 classification task of age and gender recognition. \n", "summary_of_the_review": "A (somewhat)novel, generic MLP block is proposed. Unfortunately it is not shown to outperform other blocks empirically.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1j3afp_xfRh", "reviewer": "Reviewer_8daM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is casually written, the 'whitening' is not a real whitening. This kind of basic analysis is of little interest to the community. The authors may investigate some fast whitening methods that are still not widely used in transformers to make the work a little more useful.\n", "summary_of_the_paper": "This paper does not bring new knowledge to the community. MLPs have been widely generalized with dropout,BN and skip connections.", "summary_of_the_review": "The paper is below the ICLR standard.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously recommend rejecting this submission, and I concur with that recommendation. This submission is not appropriate for a machine learning conference like ICLR. It does not display a thorough understanding of the literature nor does it make a sufficiently valuable contribution. There is no need to "generalize" MLPs, the community knows quite well that we can use dropout, skip connections, and batch norm with them. Even the original dropout paper applies dropout on fully connected ReLU MLPs.

As another example, the submission attributes skip connections to He et al. 2016, but skip connections (also known as "shortcut connections" were in common use in the late 1980s and throughout the 1990s in the Connectionist community, including for non-convolutional simple feedforward neural networks or "MLPs". They were a well known technique throughout neural network history, although the advent of deeper layered neural network architectures perhaps gave them new importance. He et al. certainly popularized them for modern neural network architectures and popularized their residual formulation. The earliest reference I could find easily for skip connections was "Learning to Tell Two Spirals Apart" which was published in 1988 by Kevin J. Lang and Michael. J. Witbrock, but in general such architectural tricks were not viewed as particularly remarkable in the 1990s neural networks literature.
Decision: Reject