Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization
Authors: David Peer
Bart Keulen
Sebastian Stabinger
Justus Piater
Antonio Rodriguez-sanchez
Venue: Accepted by TMLR ()

Abstract: Training deep neural networks is a very demanding task, especially challenging is how to adapt architectures to improve the performance of trained models. We can find that sometimes, shallow networks generalize better than deep networks, and the addition of more layers results in higher training and test errors. The deep residual learning framework addresses this degradation problem by adding skip connections to several neural network layers. It would at first seem counter-intuitive that such skip connections are needed to train deep networks successfully as the expressivity of a network would grow exponentially with depth. In this paper, we first analyze the flow of information through neural networks. We introduce and evaluate the batch-entropy which quantifies the flow of information through each layer of a neural network. We prove empirically and theoretically that a positive batch-entropy is required for gradient descent-based training approaches to optimize a given loss function successfully. Based on those insights, we introduce batch-entropy regularization to enable gradient descent-based training algorithms to optimize the flow of information through each hidden layer individually. With batch-entropy regularization, gradient descent optimizers can transform untrainable networks into trainable networks. We show empirically that we can therefore train a "vanilla" fully connected network and convolutional neural network---no skip connections, batch normalization, dropout, or any other architectural tweak---with 500 layers by simply adding the batch-entropy regularization term to the loss function. The effect of batch-entropy regularization is not only evaluated on vanilla neural networks, but also on residual networks, autoencoders, and also transformer models over a wide range of computer vision as well as natural language processing tasks.

Published in Transactions on Machine Learning Research (07/2022)
Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization
David Peer
davdd.pperddeeeoopinnnn..tt
DeepOpinion
University of Innsbruck, Austria
Bart Keulen
bart@@bartkeulen..oom com
University of Innsbruck,, Austria
Sebastian Stabinger
schastin..adhngeeBddeepppnnnnnno
DeepOpinion
University of Innsbruck, Austria
Justus Piater
University of Innsbruck, Austria
Antonio Rodriguez-Sánchas
University of Innsbruck, Austria
Reviewed on OpenReview: https: /ttp://ppppe openreview. net/ forum? id=1.oh155D2f
Abstract
Training deep neural networks is very demanding task, especially challenging is how to adapt architectures to improve the performance of trained models. We can find that some- times, shallow networks generalize better than deep networks, and the addition of more layers results in higher training and test errors. The deep residual learning framework addresses this degradation problem by adding skip connections to several neural network layers (He) al. 20166. would at first seem counter-intuitive that such skip conncctions are needed to train deep networks successfully tially with depth. In this paper, first analyze the flow of information through neural networks. We introduce and evaluate the batch-entropy which quantifies the flow of infor- mation through each layer of a neural network. We prove empirically and theoretically that a positive batch-entropy is required for gradient descent-based training approaches to optimize a given loss function successfully. Based on those insights, we introduce batch-entnne entropy regu- lorization to enable gradient descent-basod training algorithms to optimize the flow of infor- mation through each hidden layer individually. With batch-entropy regularization, gradient descent optimizers
the expressivily of network would grow exponen-
transform untrainable networks into trainable networks. We show empirically that we therefore train 'vanilla' fully connected network and convolutional neural network-ne no skip connections, batch normalization, dropout, or any other architec- tural tweak-wit 500 layers by simply adding the batch-entropy regularization term to the loss function. The effect of batch-entropy regularization is not only evaluated on vanilla. neural works but also on residual networks, auloencoders, and also transformer models over a wide range of computer vision as well as natural language processing tasks. Our code basc is available at httts:///gttnn www.uu...ommpeeeeeeeee http:///wwraaad//a/ae/eoooaaaaaa

---

Published in Transactions on Machine Learning Research (07/ (07/202) /202)
1 Introduction
The training of deep neural networks on large-scale datasets pushed slate-of-the-arl results in many differ- ent areas such as computer vision, natural language processing, time-series prediction, medicine, or drug discovery (Alassafi et al. 2022 Devlin et al. 2019 He ct al., 2016 Tran ct al. 2021 Xu et al. 2019). One reason for this achievement is that the expressivily of neural nelworks grows exponentially with the depth of the network (Raghu et al. 2017). The development and training of deep neural networks on a given task is challenging and many different methods and improvements have been developed
simplify the training
of deep networks, such more careful weight initialization (Glorot & Bengioo] 2010 He et al. 2015).
better activation functions (Dahl et. al. 2013), regularization methods (Tofle & Szegedy 2015 Srivastava
et al. 2014), and innovative architectural designs (He et al.) 2016 Srivastava et al. 2015
Vaswani et al,
2017 Nevertheless, designing novel architectures with high performance for different downstream tasks is a laborious one and often, cannot simply be achieved through the creation of deeper networks (Tan & Le 2019) as at times, they may perform worse on a given task than shallower architectures or even sometimes not be trainable all. Surprisingly for such cases, the deeper the network, the lower the accuracy (He al., 2016), which is known the degradation problem. This degradation problem can be overcome through the use of residual connections.
In this paper, we analyze the degradation problem from an information theoretical point of view. To approximate the amount of information that flows through each layer, we introduce the batch-entropy and show empirically that trainable networks maintain a flow of information through the network. On the other hand, whenever this flow of information collapses i.c. the batch-entropy becomes zero, the network cannot be optimized with gradient descent. Through dimensionality reduction, we found experimentally that the loss surface becomes highly non-convex which provides hint about the reason behind the difficulty in the training of such networks (Li et al. 2018). We study this information flow collapse not only empirically, but also from a theoretical perspective and prove that classical loss functions such as the cross-entrops loss cannot be minimized with gradient descent if the batch-entropy of a single layer is zero. Based on these insights, we introduce the autch-entropy regularization term, which ensures a positive batch-entropy and hence enables the training of deep 'vanilla' neural networks. More precisely, we extend the loss function in order to optimize not only for the task specific objective, but to also optimize the network for learnability in case it is not trainable. We will demonstrate that deep "vanilla" fully connected and convolutional networks- skip connections, batch normalization, dropout, or any other architectural tweak special activation function with 500 layers can then be successfully trained. Many different normalization techniques exist that regularize the variance during forward propagation. Those techniques are mainly developed to accelerate the training of deep neural networks (Toffe & Szegedy, 2015: Ba et al. 2016 Salimans & Kingma 2016 Klambaucr et al. 2017a). We show empirically in this paper that they can not be used to train very deep vanilla neural networks.
Another related rescarch area studies signal propagation in random networks (Poole et al. 2016) that enabled the training of very deep networks. For example Xiao et al. (2018) developed orthogonal kernels in order to train very deep convolutional networks. While this orthogona initialization scheme is successful in training very deep networks without skip connections, a special initialization scheme for convolutional filters with only a single non-zero entry per convolutional filter is required. In contrast, we provide a solution, inspired by information theory, where trainability is part of the objective such that untrainable networks are transformed into trainable networks during the training process. We not only show that no special initialization scheme for convolutional kernels is required, but also demonstrate the success of the proposed approach on different types of layers such as convolutional fully connected layers.
We hope that our work will enable researchers to experiment with a much wider range of deep networks with novel layer types. To motivate this further, we study the effect of batch-entropy regularization on fully connected networks, convolutional networks, residual networks, autoencoders, as well as transformer models, for both computer vision and natural language processing tasks.
The main contributions of this paper are as follows:
2

---

Published in Transactions on Machine Learning Research (07/2022)
Batch-entropy is applied in order to estimate the amount of information that flows through every layer inside a network.
A batch-entropy regularization term is introduced to enable gradient descent. to optimize the flow of information through neural network to improve trainability.
We demonstrate that deep, vanilla, fully connected, as well as convolutional networks can be trained with batch-entropy regularization.
We evaluate the effect of the proposed approach on fully connected networks, convolutional networks, residual networks, autocncoders, as well as transformer models, for both computer vision and natural language processing tasks.
Related work is given in section 2 In section 3 we introduce the batch-entropy to approximate the amount of information that flows through a network and the batch-entropy regularization which enables gradient descent to optimize the flow of information through each layer. In section | experimentally analyze the positive effects of batch-entropy regularization. We finish our paper with a discussion on our findings and propose future work in section 5
2 Related Work
Oplimization problems in deep neural networks have been the subject of a wide range of studies. These studies have shown that a careful initialization of parameters has a significant effect on the training (LeCun ct al.
alll|20122 (012)) and revealed the problem of vanishing or exploding errors and how keep the error flow constant
(Hochreiter & Schmidhuber 1997p. Initialization methods that initialize weights such that gradients are not vanishing or exploding at the beginning of the training have been the subject of extensive study (Glorot & Bengio 2010 He et al.) 2015 Hinton et al.) 2006 Krähenbühl et al., 2016 Mishkin & Matas, 2016). Unfortunately, deep neural networks are still difficult to optimize, even when those initialization
methodologies are used (Srivastava et al. 2015 t. He et. al. (2016) called this the degradation problem and
showed that special architectures with skip connections (Srivastava et al. 2015 He et al. 2016) could avoid this problem. Later, Peer et al. (2021b) found that certain layers in neural networks exist that may be the source of the degradation in performance of very deep models and proved that skip-connections would bypass those misleading layers. The same authors showed that those layers do not contribute to the classification and therefore, around 60% of the layers of a trained residual network can be pruned with negligible effect on the lest-error (Peer et al. 2021a).
Many different normalization techniques exist that regularize the variance during forward propagation. Batch normalization accolerates the training by reducing shifts of output distributions of hidden layers (Toffe & Szegedy 2015). The ellect of batch normalivation strongly depends on the batch-size. Therefore, Ba et al. (2016) introduced layer normalization which uses the mean and variance from all of the summed inputs in order to reduce the training time. Weight normalization decouples the length of weight vectors from their direction in order to accelerate the training of deep networks (Salimans & Kingma, 2016). Deeper networks can indeed be trained with self normalizing networks Klambauer et al.] (2017a), but they do not work on vanilla feedforward neural networks with ReLU activation functions. They implement special scaled exponential linear unit in order to achieve training of networks with 32 layers as was demonstrated empirically in the original paper.
Although these techniques accelerate the training of deep networks, they do not improve the trainability of very deep vanilla networks will be shown later in this paper. Another research area studies this problem by analyzing signal propagation in neural networks through mean field theory. The theoretical framework by Poolc et al. (2016) revealed maximum depth which signals can propagate through at initialization for fully connected networks. This was extended by Xiao el al./ (2018) to derive a mean field theory for signal propa- gation in random convolutional neural networks. The authors specifically introduced orthogonal kernels for convolutional layers in order train very deep vanilla convolutional networks. Martens et al. (2021) intro- duced deep kernel shading to train deep neural networks without skip connections or normalization layers, but the authors use combination of precise parameter initialization, activation function transformations
3

---

Published in Transactions on Machine Learning Research (07 (07/2022)
well as small architectural tweaks together with the second-order K-FAC optimizer Martens & Grosse 2015) to achieve this.
Other authors have studied neural networks from an information theoretical viewpoint to improve their performance: Blot et al. (2018) proposed a regularizer in order to improve the intermediate representations of single samples x by calculating the entropy of a sample conditioned to the corresponding label y- The authors showed that a model that is invariant to many transformations will produce the same representation for different imputs, which improves the performance of the model. Achille & Soatto (2018) showed that representation can be made invariant against muisances- maisance--random variables that affect the observed data, but are not informative for the task-by limiting its information throughput. This can be achieved by injecting noise (e.g. dropout) or creating a bottleneck (e.g. & pooling layer) through a regularization term which penalizes the information that is contained in the weights. Gilad-Bachren Bachrach et al. (2003) studied the tradcoff between the complexity of data representation and the accuracy of the model. Another information theory based regularization method proposod by Pereyra et al. (2017). which penalizes low entropy output distribut which acts a strong regularizer in supervised learning, improving the overall performance of the model. Upper bounds on the generalization error of learning algorithms based information-theone analysis have also been derived (Xu & Raginsky 2017).
3 Methods
In this section, the flow of information through neural networks is analyzed from theoretical perspective. We first introduce the batch-entropy to measure the amount of information that is propagated through each individual layer. We then prove that a positive batch-entropy is required at each layer in order to successfully optimize a given objective (e.g. the cross-entropy between the output and the ground truth) with gradient descent. Based on those insights we introduce regularization term that improves trainability.
3.1 Notation
x2, 12,, "..k]] JT Scalar values are represented with lower case letters such k vectors written in bold [21 and matrices are represented with bold upper case letters W. Sets of vectors are represented by X {x..)...
We call a training set S E X x y which contains items (e.g. images) X' and their labels 3 We assume that the ground truth y E y is one-hot encoded. e is the dimensionality of a single label. The input to a layer is of dimension m and its output dimension n. Weight matrices W E Raxm and bias terms E Raxi are trainable parameters of a network. The outputs of given layer L {1,..., Lb for a neural network with L layers are vectors ax+ fz) with z W (x) b computed for given input x E X' with nonlinearity function ʃ and a (x) = The output of an individual neuron i al the output for layer / would be (x).
The network is trained with mini-batches that satisfy B C X. Without loss of generality it is assumed that batches B are uniformly distributed w.r.t. the class labels. The surrogate loss, optimized with gradient descent, calculated using the cross-entropy loss as L(x) Lce (g()) y) for input x with corresponding label y, where g(x) = softmax(0) with 0- w= + b². The gradient is then
aW 0g 00 aw. (g(x) xx) y) a axx)
OC OL 0g a0
The gradient for the whole mini-bater batch can be calculated as
OC(B)
ac(x))
OWL.
B
OWL.
For simplicity we put our focus on classification problems although the theory can extended e.g. regression problems.
4

---

Published in Transactions on Machine Learning Research (07 (07/2022)
H=1.0
H=0.2
H=0.0
(a) Neuron with large
(b) Neuron with small output range.
(c) Neuron with constant output values.
output range.
Figure 1: Histogram of 1024 output values of three different neurons from a fully connected network trained on MNIST. The x-axis shows the output-value and the y-axis how often the neuron fired with this value. On the top of each plot, the estimated batch-entropy eq. (3) is shown. More neurons are shown in appendix A
3.2 Quantifying the Flow of Information through each Layer
In order to analyze the flow of information through neural networks during forward propagation, we quan- tify the amount of information that is propagated through each individual layer the average amount of information that is propagated through each individual newron within layer. Output values of neurons are continuous and therefore, the amount of information passed through neuron can be calculated using the differential entropy (Shannon, 1948h:
1 - (x(x) log (x) de.
(1)
Unfortunately, the underlying data-generating distribution of individual neurons is unknown. Nevertheless, we have access to samples of the distribution by measuring output values of neurons for different inputs of our training dataset X. Based on those samples, the entropy of each neuron can be approximated with e.g. nearest neighbor distances (Beirlant et al. 1997). We make use of the batch-entropy to regularize neural networks with gradient descent, which requires differentiable batch-entropy. Nearest neighbor methods provide good approximations of the real entropy but are not differentiable. Hence, a batch-entropy using nearest neighbor methods is not suitable a regularizer. Therefore, we search for an approzimation of the entropy that is differentiable and be used in combination with a gradient descent based optimizer.
To develop differentiable function that approximates the entropy, we analyzed the output values of different, randomll selected neurons from a fully connected network. Histograms for three randomly selected neurons sampled from a neural network trained on MNIST are shown in fig. [ More neurons for detailed analysis are shown in appendix A It can be seen that output values of almost all neurons seem be Gaussian distributed. Using the empirically motivated assumption that neurons are Gaussian distributed, the following differentiable approximation of the enty can be derived:
H log(2 (mT ox + e),
(
where a% the standard deviation of a single neuron k computed for the values of mini-batch mathematical derivation of the differential entropy for the normal distribution is given by Michalowicz et al.) (2013). A bias term of = 1 introduced in all our experiments to not only ensure numerical stability, with we also ensure that H E oo).
Figure 1 shows how entropy values (H) correlate with the output distribution: For example, the neuron shown in fig. [b has a smaller batch-entropy than the neuron shown in fig, Ia because the latter neuron fires values a much larger range and hence propagate more information the next layer. In the extreme case shown in fig. Ic. where a neuron outputs the same constant value for each input, the entropy is zero, which is reasonable each input leads to the same output.
5

---

Published in Transactions on Machine Learning Research (07 22222
Lm15, .BE botama., Accuracymé6.755 75%
L-30, LBE botamb Accuracym1..59%
24-
80
0.35
60
0:30
1.
6.00
Steps
Steps
(a) FNN with 15 layers.
(b) FNN with 30 layers.
Figure 2: Flow of information through different layers two fully connected networks (FNN) with 15 layers (a) and 30 layers (b).The axis shows the layer of the network which is evaluated and the axis is the corresponding batch-entropy. The color scheme indicates the step after which the batch-entropy was evaluated. Early training stages are blue, later training stages are shown in red. At the beginning of training (bluc), the batch-entropy decreases we go up in the network hicrarchy (from left to right).
The entropy of a complete layer with n neurons can be calculated by averaging the entropy of all neurons as follows:
H' Σ log(2 2e of + c),
(3)
where og is the standard deviation of neuron is of layer computed for the values of the mini-batch B. We call eq. (3) the balch-enlropy of layer l.
By computing the batch-entropy (eq. ), we can measure the flow of information through a network, so that two different networks can casily be compared in terms of how information propagates through them. An example of such a comparison is shown in fig. 2 for network with 15 layers that is trainable, and a network with 30 layers that is untrainable using gradient descent. Both networks proposed by He et al. (2015) and trained on MNIST for 2 epochs. Figure 2a shows that the shallow network was successfully trained on MNIST. At the beginning of the training (blue dots), the flow of information is high in the early layers and decreases as we go deeper in the network. After analyzing the batch-entropy at the output layer, we found that it is slightly larger than zero, which would indicate that at least some useful information propagated from the first to the last layer. During training, the information flow increases also for later layers (fig. [a] color change from blue to red). On the other hand, fig, 2b shows the case of deeper network where information is propagated to the output layer. After testing many different hyperparameter setlings, found that such networks cannot be trained al all and the information flow never increases during training. Next, we analyze from theoretical viewpoint why neural networks are not trainable whenever 3 3 E {1,..., L} such that H' 0
initialized with the method
First of all, whenever H 0, then for all subsequent layers k 1 every input of a mini-batch is the same such that Vk > = 0 which implies that H= given in appendix B Therefore,
An information theoretic proof of this statement is sufficient to prove that a network is untrainable whenever I|_ 0. We need to show next that the gradient points into direction that is independent of the desired labels y whenever H4 = 0. A gradient that is independent of its ground truth cannot update weights to improve the performance which will therefore conclude our statement.
Without loss of generality we can assume that labels are one-hot encoded vectors y that are uniformly distributed w.r.t different classes in our mini-batch B. It follows that.
OC(B) OWL
1
[8|
Oc(x))
1 1 g
a²
(4)
B
OWL
C
1=1
6

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
255
25
2.0
2.0
1.5
10
1.0
0.5
0.5
0.0
0.0
1.00
1.00
50
-1.96.2 58 2808.23 =B,
-1 28. 58 28.2000
0.00
00225
25
0,75
0.75
11.001..00
1000 1.00
(a) H -
(b) H >
Figure 3: Comparison of the cross-entropy loss (CCE) surface for networks with different information through- put trained MNIST. X and Y represent two random orthogonal directions in the weight space 2018
Li et al.
where is vector with components 1 of dimension for a dataset consisting of classes. A detailed derivation of the gradient is added to appendix C
It can be seen that the gradient is independent of labels y whenever 44= 0. Therefore, the error signal is invalid w.r. the real objective, and weights are updated in misleading directions. Whenever the output of neuron is gi > weights are updated in order to decrease the output in the next iteration. On the other hand, whenever gi 1 weights are updated in order to increase the output value. Therefore, during the training, weights are adjusted until each output neuron fires with constant value of L. Whenever g = ! 1 8C(B)
the gradient becomes 7WL = O. It then be concluded that the network fires with constant values and the performance of the model accuracy can no longer increase, the gradient is zero.
We analyzed the output values of neurons for networks with =1_ = 0 and have indeed found that those fire with a constant value of 1 after a few training steps. Additionally, we evaluated whether the gradient becomes zero whenever HL by evaluating the loss surface for both cases, Hb > and H 0. The loss surface of the cross-entropy in the weight space is shown in fig. 3 We generated two random orthogonal vectors to plot 3-dimensional loss surface using the methodology proposed by Li et al. (2018), more samples are shown in appendix E Networks easier to optimize whenever the loss surface seemss close to being convex in this highly compressed space (Li et al. 2018). We can confirm this finding as the surface for 0 (fig, 3a) is flat, highly non-convex and that the network was not trainable at all. Additionally, the gradient is zero in this region which would explain why gradient descent cannot find a solution. On the other hand, whenever H² (fig. 3bb we found that the network can be trained successfully and that the test accuracy incrcased during training. The loss surface of the highly compressed space as shown in fig. 3b looks convex and according to Li et. al. (2018), the network is therefore easier to optimize. This theoretical and empirical study leads to the following hypothesis:
Hypothesis 1. A network is not trainable with gradient descent, i (1...., L} such that H²= for network of depth L.
Next we provide a solution to escape from regions where H= in the form of a regularization term that will help gradient descent move into regions where H > This solution will allow at-first untrainable networks to be trained by bending the loss surface such that the gradient points into a direction where the cross-entropy loss can be optimized successfully.
7

---

Published in Transactions on Machine Learning Research (07 (07/2022)
3.3 Batch-Entropy Regularization
We introduce regularization lerm Laλ that allows gradient descent to directly oplimize the batch-entropy within each layer. Thanks to this analysis of batch-entropy, weights of layers can be adjusted such that information flows through the whole network converting invalid crror signals of the cross-entropy into valid error signals.
The batch-entropy is differentiable and can be used directly as part of the loss term. Unfortunately, we found empirically that minimizing the cross-entropy and maximizing the batch-entropy leads to networks with low performance. This is reasonable because H' [0, oo). Because of this, the entropy can always be maximized while the cross-entropy loss could be neglected, leading to sub-optimal networks with low accuracy. This problem can be solved by defining the precise level of information that should be passed through each individual layer. We do so through a parameter a' per layer, which defines the desired level of information that should pass through each layer. Therefore, during the training, weights of each layer can be adjustod until H' =a.
We showed in fig. that the information flow is different for each layer. Unfortunately, were not able to postulate theoretical framework that would describe for each layer I which exact a' value would lead to high generalization sapabilities for a specific model. Figure 2a provides us with hint that the information flow has different compression and expansion phases. Instead of having to specify a precise value for a', we include it as part of the training process. Note that we pre-initialize each a' with the value a at the beginning of the training. We found a through a hyperparameter grid search for each architecture individually. Additionally, at is constrained
a E (Amin, (0) with amin 0, ensuring a flow of information through the network. For Omin we found that it is sufficient to set it small positive value (e-g- 0.2). The layerwise batch-entropy loss for single layer / can therefore be calculated with
C = (H' mnax(ammnn la')))²
(5)
The batch-entropy loss (CBE) for the whole network can be calculated as follows:
B CE T,
T.
CBF
(6)
1=0
The number of additional learnable parameters is L (one a' for each layer) which is negligible for networks with millions of parameters. In order not to exceed the cross-entropy loss (CCE) and be able focus on the LBE whenever the cross-cntropy cannot be optimized, we included 3 hyperparameter which find through hyperparameter search for each experiment individually. Additionally, we scale the LBE with the Leg which ensures that optimizing the information flow is secondary and mainly active whenever the Lce is large Le. in a local minimum:
L = LcE LCE LBE-
(7)
The effect of scaling the Lb with Lee can be analyzed by computing the gradient:
ac
OCCE OWT
a(CEE # LBE) 0LCE OW
δLgE ÔLCE
+ LCE OW!
+ LBE OW!
(8)
OWT
OW
Whenever the cross-entropy is small, the gradient of the LAF becomes negligible and the main objective LCE- is optimized. On the other hand in a local minimum where the LoE is large, the gradient of the CB is additionally increased. We now continue our analysis for the case where the cross-entropy is in a local
minimum:
OC OW = 0+ LCE Ow
3 9c LcE T Ow
+0
(9)
8

---

Published in Transactions on Machine Learning Research (07/202) 2022)
(a) Without LBS: Los Loss after (b) Without LBE: Lee Loss after (c) Without LBE: Lee Loss after 0 steps.
200 steps.
1500 steps.
(d) With LBE: Lce Loss after 0 (e) With LBE: Lc Loss after 200 (f) With LBs: Lcs Loss after steps.
steps.
1500 steps.
Figure 4: Loss surface of fully connected network at different training steps (0, 200 and 1500) when H² at the beginning of the training without LB (a, b, c) and including LB regularization (d, e, f). Compare the errors surfaces in the top row (without LBE) with the ones in the bottom row (with LBE)
A detailed derivation is given in appendix
C
Whenever the cross-entropy cannot be optimized because H' = O, gradient descent can still optimize weights of each layor to ensure that H' 0. We additionally verify this statement empirically in section 4 A graphical illustration of why optimization without the LBE fails and why it works with batch-entropy regularization is shown appendix D Without batch-entropy regularization, the V. = is propagated backwards through the whole network. With batch-entropy regularization, the CB gradient directly applied to the weights of each layer ensuring that H' > 0 as the training proceeds.
We empirically evaluated the loss surface during training shown in fig. 4 and compared the Lce loss after different training steps when LB is used, to further support hypothesis ! It can be seen that the LB loss pushes weights into a direction where the cross-entropy loss can be successfully optimized with gradient descent, while networks that are trained without LB are unable to escape from local minima. We report more loss surfaces after different stages of training in appendix where we provide the cross-cntropy loss surface, the LnE loss surface, and the accuracy surface.
In fig, 2b we have shown that networks with 4 0 are not trainable. We executed this experiment again, but added the batch-entropy regularization term to the loss with = 0.5 and / = 0.2. The results are shown in fig. 5 It can indeed be seen that the nelwork with 30 layers that was not trainable at first (fig. 2b became trainable (fig. (fig.b)), so that the accuracy increased from 11.35% (fig. 2b to 42.95% in only 100 steps of training. In this same figure, we can see that the network that was trainable before (fig, 2ab is not negatively inftuenced by the batch-entropy regularization term. The accuracy is even a bit larger indicating that the LBE have positive effect the training of networks that are trainable without LBE too. Therefore, hypothesise that
Hypothesis 2. Regularizing the information flow through neural networks can improve the performance of the trained model.
We study hypothesis 1 and hypothesis 2 empirically in section on different architectures and tasks.
9

---

Published in Transactions on Machine Learning Research (07 (0222
L-15, LBE beta Accuracy-47.413
L-30, LBE beta- Accuracy-42.855
044
5500
0.3
60
40
0.-
40
0.2
20
01
10
Steps
20
Layar
Layer
(a) FNN with 15 layers.
(b) FNN with 30 layers.
Figure Flow of information through different layers of an FNN, trained on MNIST for 2 epochs with batch-entropy regularization for 3 = 0.2, all af are initialized with 0.5. The z axis shows the layer of the network be evaluated and the y axis is its the batch-entropy. The color scheme indicates the step after which the batch-entropy was evaluated. Early training stages blue, later training stages are shown in red.
4 Experimental Evaluation
We analyze hypothesis ! and hypothesis 2 empirically computer vision well natural language processing tasks for a wide range of architectures, namely fully connected networks, residual networks, transformer models and autoencoders.
4.1 Setup
All experiments implemented in Torch Paszke et al. 2017) and executed Nvidia GPUs. Wandb used for experiment tracking (Biewald, 2020). Our source code is publicly available GitHut including sweep files for each experiment which describe precise hyperparameter ranges that we used in order to reproduce all experiments. We split the training set into training (80%) and validation (20%) and use the validation set to find a good hyperparameter setup for each method. We want to explicitly mention that the training set that we used therefore smaller compared against other approaches, which often use all training samples. Precise hyperparameter values for each experiment are given in the appendix. The performance results, comparing models that are trained with / without LBE, is provided on the test-set to ensure that we do not overfit the validation set through hyperparameter search. Following previous work deep learning (Devlin et al. 2019), we report the average over 5 random restarts using different scods. In total we used different datascts for evaluation, including computer vision as well as natural language processing tasks: MNIST (LeCun & Cortes 2010).. PashionMNIST (Xiao et al. 2017), CIFAR10 and CIFAR100 (Krizhevsky et al. 2009), RTE (Bentivogli et al.) 2009), MRPC (Dolan Brockett 2005) and CoLA (Warstadt al.] (2018). To also evaluate the proposed approach on a more challenging task, while still using only moderate computational resources, we trained model a subset of ImageNet. (100 classes)
but used only 500 images per class to increase the complexity of the task (Deng et. al. 2009) We call this dataset ImageNetioo in this paper.
4.2 Training Deep Vanilla Neural Networks
To further evaluate hypothesis] 1 we trained "vanilla" networks with 500 layers. The accuracy, Lcg and L loss during training are shown fig. for fully connected as well as convolutional neural network. We found that such deep vanilla networks regularization against other normalization techniques reported in appendix F
trainable if LB was used. We also compared batch-entropy
entropy
10

---

Published in Transactions on Machine Learning Research (07/ 2202)
Rarth Samege Lexe
1.30
130
03
(a) FNN Accuracy.
(b) FNN Lce loss..
(c) FNN LE loss.
Ascensy
Cress Entrapy
Batch Entropy
0.54
0.00
(d) CNN Accuracy.
(e) CNN LCE loss.
(f) CNN LBE loss.
Figure 6: Shows the training curve of a 'vanilla' fully connected network (FNN) and 'vanilla' convolutional neural network (CNN). Both networks are 500 layers deep and both are trained on MNIST. The accuracy as well as the two loss terms (LCF, LBR) are shown.
With batch-entropy regularization, this 'vanilla' fully connected network reached a test accuracy of 95.2% which shows that we can indeed train very deep vanilla fully connected networks by optimizing the infor- mation flow using LAF regularization. The training curve is shown in fig. Ga 6c We also tested whether batch-entropy regularization can be used to train deep vanilla convolutional neural networks. More precisely, we trained a network with 500 convolutional layers with ReLU activation functions as shown in fig. 6d 6i We did not use pooling layers, skip connections or any other normalization technique. We reached a test accuracy of 99.2% on MNIST with this setup. Specifie hyperparameters can be found in the source code that we provide. This experiment shows that batch-entropy regularization can not only be used to make untrain- able fully connected networks trainable, but can also be applicd to other architectures such as convolutional neural networks. Another interesting finding is that for fully connected as well as convolutiona ional networks the batch-entropy loss also increased during training, which indicates that the regularization method must be applied during the whole training process to keep the network trainable.
We are not only interested to evaluate the effect of batch-entropy regularization for very deep vanilla networks but also want to analyze the effect on other architectures with less layers. We therefore evaluate wide range of different architectures and depths on computer vision well as natural language processing tasks in the next experiments.
4.3 Fully Connected Neural Networks
In this section, we empirically evaluate the effect of layerwise batch-entropy regularization on fully connected networks of different depth. We executed a quantitative experiment and trained fully connected networks with depths from 10 to 50 and evaluated the test accuracy for each network when trained with and without LBE. All networks contain 1000 neurons per layer that are trained for 100 epochs, with a batch size of 512, using the Adam optimizer (Kigmma & Ba 2015), on MNIST (LeCun & Cortes 2010). Weights are initializod with the method as proposed by He al. (015)). To find a good setup for the baseline, we optimized for the learning rate ir E [le- 4, 5e 4, 1e 3] utilizing grid-search, and took the best run w.r.t the validation set. Specific hyperparancter values which were determined using grid search are provided in appendix H.1 ttalle 6
A quantitative analysis of different fully connected networks is shown in table I The mean accuracy of runs on our test set together with the standard crror is shown. First of all, can be seen that deep networks (depth 30- 50) can only be trained if the Lb loss used, where the batch-entropy of each layer is regularized in order to enable training of deep neural networks with gradient descent (hypothesis 1. For
11

---

Published in Transactions on Machine Learning Research (07/2022)
Depth
LB
Test Accuracy
Std. Error
10.0
No
98.94
0.01
Yes
98.95
0.01
20.0
No
97.86
0.60
Yes
98.48
0.13
30.0
No
11.35
0.00
Yes
92.84
1.86
40.0
No
11.35
0.00
Yes 93.51
0.67
50.0
No
11.35
0.00
Yes
92.93
1.22
Table 1: Comparison of training fully connected networks with or without batch-entropy regularization. All fully connected networks are trained for 100 epochs with a batch size of 512. Weights are initialized with the method as proposed by He et al. 2015). Larger numbers are written in bold.
depth of 10, the test-accuracy almost the same when training with and without CB.. For a depth of 20, batch-entropy regularization shows a very positive effect on the final performance of the model, even though the network is still trainable without the LB.. In this case, the variance between different runs is also heavily reduced because of the use of LAF (0.13 compared to 0.60), which indicates that batch-entropy regularization not only helps to convert untrainable networks into trainable networks, but also has positive effect on the training in general. Another interesting obsorvation we can extract from appendix [H.1 is that a found through grid search- is 0.1 for almost all depths. Only for the deepest network with 50 layers a 0.3. This indicates that (1) a values are quite consistent for given architecture and datasets, which could help to reduce the hyperparameter search space the future runs and (2) deeper networks require a larger flow of information through each layer reach high performance since the deepest network required
larger value.
4.4 Autoencoders
Autoencoders are a class of unsupervised learning algorithms traditionally used for dimensionality reduction or feature learning. Autoencoders neural networks with bottlencck in their architecture, usually in the middle hidden layer. The nelwork predicts the input itself, hence the bottleneck forces the network to learn compressed representation z E RN,x1 x1 of the data, such that for imput x, the encoder returns z fxx). The decoder roconstructs the input from such representation, & = g(z). Assuming cach x E x is Bernoulli dist distribu, the autoencoder is trained by minimizing the binary cross-entropy between the ground-truth x and the reconstructed input &
We trained autoencoders with depths d E {1, 10, 25). where a depth of d means that both the encoder and the decoder have d hidden layers each. The dimension of the bottleneck was set to N, 10 and all hidden layers contain 256 neurons. All networks were trained for 50 epochs with a batch size of 128. The Adam (Kingma & Ba 2015) optimizer with a waterfall schedule was used. An initial learning rate of le -e-3 3 was used and every time the validation error stopped decreasing- estimated over a window of 5 epochs the learning rate was scaled down by a factor of 0.5. Experiments were run for the MNIST (LeCun Cortes 2010) and FashionMNIST (Xiao et al. 2017) datasots. All images were flattened to one-dimensional array of size 784. For evaluation we compare the similarity between the target x and reconstruetion & using the Meon Structural Similarity Inder (MSSIM) (Wang et al. 2004].
socreasing-esttmmteed
The average MSSIM and std. error of 5 runs on the test set are given in Table 2 The first observation is Ihat for a depth of one the performance between the autoencoders with and without batch-entropy reg- ularization is the same, adding batch-entropy regularization has no negative (nor positive) effect on the performance of shallow autoencoders (hypothesis 2). For larger depths-d E (5, 10, 25) a55=aautencoderrs with batch-entropy regularization outperform their counterparts without batch-entropy regularization. The per- formance, though, is not high for shallow autoencoder. For depth of d = 5 the MSSTM is still close
12

---

Published in Transactions on Machine Learning Research (07/2022)
Dataset
Depth
LBF
Test MSSTM
Std. Error
FNShionMNIST
0.61
0.0005
yes
0.61
0.0007
5
n0
0.53
0.0138
yes
0.57
0.0009
10 n1
0.13
0.0001
yes
0.41
0.0084
25
n10
0.13
0.0002
yes
0.41
0.0059
MNIST
n1
0.75
0.0013
yes
0.75
0.0017
no
0.12
0.0005
yes
0.65
0.0071
10
0.12
0.0003
yes
0.42
0.0038
25 no
0.12
0.0004
yes
0.39
0.0168
Table 2: Comparison of training autoencoders with or without batch-entropy regularization. The networks are trained for 50 epochs with a batch size of 128 using the Adam optimizer. A depth of d means that both the encoder and the decoder have d hidden layers each. Larger numbers are written in bold.
to the best MSSIM, especially on the FashionMNIST dataset. For depth larger than 5 we see that the network is only trainable if the La regulaizer is used (hypothesis D Deep autoencoders with depths 5, 10, 25 and 50 and without batch-entropy regularization, all collapse at similar local optima, MSSIM = 0.12 for MNIST and MSSIM 0.13 for FNSH The network outputs the mean value all images in the training set S 1/|$] Σ 1 X; independently of the input, which is a widely known problem of deep autoencoders [Murphy 2012 On the other hand, autoencoders trained with the LB regularizer do not suffer from this problem as shown in table
|s]
4.5 Residual Networks
In the previous experiments we have seen empirically that batch-entropy regularization can indeed convert untrainable fully connected networks, convolutional networks as well as autoencoder networks into trainable networks. In the next experiments we additionally evaluate whether batch-entropy regulariaztion has a positive or negative effect on regularizing networks with residual connections, which are known to overcome the degradation problem. More precisely, He et al. (2016) introduced residual blocks containing multiple convolutional layers that are bypassed with skip connections. The performance is further improved with natch-normmllizttio layers and dropout just before the output layer. In this experiment, we evaluate the effect of optimizing the information flow through networks with residual connections.
We used a learning rate of 0.1 at the beginming of the training and divide it by 10 whenever it plateaus, together with a weight decay of 0.0001 and a momentum of 0.9. The network is trained on four different datascts which are MNIST (LeCun & Cortes 20100, MashionMNNIST (Xiao et al. 2017). CIFAR10 and al. 2009) using SGD with a mini-batch size of 256 (He et. al., 2016). Weights are initialized with the method proposed by He et al. (2015). To compute the LBE, we regularized the information flow of the intermediate layers within a residual block as we found empirically that regularization of the complete residual block has no effect.
CIFAR100 (Krizhevsky
The results on the test set are shown in table 3 Results indicate that generalization of residual networks benefits from batch-entropy regularization as the network trained with LB outperforms the bascline out of 12 cases (hypothesis 2 The standard error decreases as well for almost all evaluations indicating more stable results when executed with different seeds. Another observation is that the median of increased
13

---

Published in Transactions on Machine Learning Research (07/2022)
Dataset
Depth
LB
Test. Accuracy
Std. Error
MNIST
26
No
99.68
0.02
Yes
99.67
0.01
74
No
99.67
0.02
Yes
99.68
0.01
152
No
99.59
0.02
Yes
99.66
0.03
FashionMNIST
26
No
94.33
0.06
Yes
94.25
0.06
74
No
93.89
0.09
Yes 94.18
0.09
152
No 93.33
0.13
Yes
93.55
0.17
CIFAR10
26
No
89.40
0.03
Yes
89.41
0.08
No
89.70
0.20
Yes
89.89
0.22
152
No
88.16
0.31
Yes
88.54
0.19
CIFAR100
26
No
62.85
0.10
Yes
63.12
0.05
74
No
63.95
0.30
Yes
64.63
0.15
152
No
62.47
1.40
Yes
64.68
0.49
ImageNet100
26
No
51.08
0.22
Yes
51.47
0.25
74
No
51.84
0.21
Yes 52.28
0.20
152
No
50.33
1.92
Yes
52.45
0.28
Table 3: Comparison of training different residual networks with or without batch-entropy regularization. All residual networks are trained for 100 epochs with batch size of 256. Weights are initialized with the method as proposed by He et al. (2015). Larger nunbers written in bold.
the complexity of the dataset increased (appendix H.2 For MNIST, FashionMNIST, CIFAR10, and CIFAR100 the median of for all different depths is 0.5, 1.0, 1.0, and 1.5 respectively. It seems reasonable that more complex data (e.g. more classes, more color channels, etc.) would require more information throughput and therefore, a higher batch-entropy.
4.6 Transformer Models
Transformer models are class of models which implement self-attention layers followed by fully connected layers in order to find correlations between different parts of input tokens (c.g. words in sentence) (Devlin et al.] 2019). Those networks are pre-trained in a self-supervised fashion on a huge lext corpus such as wikipedia. After the pre-training they are fine-tuned on a given downstream task. We will now evaluate whether batch-ontropy regularization also beneficial for this type of models, which are already pro-trained. Different from previous experiments on computer vision tasks, transformers are mostly used for natural language processing. More precisely, trained BERT Base with 12 layers and BERT Large model with 24
14

---

Published in Transactions on Machine Learning Research (07/2022)
Dataset
Model
LB
Test Perf.
St.. Error
RTE
BERT Base
67.42
0.89
67.34
1.54
BERT Large
no
69.76
1.20
yes
70.78
0.87
MRPC
BERT Base
n10
88.58
0.70
yes
89.13
0.88
BERT Large
no 89.06
0.37
yes
89.28
0.55
CoLA
BERT Base
55.63
0.97
56.70
1.04
BERT Large
nO 59.39
0.71
yes 60.13
0.63
Table 4: Comparison of training with or without batch-entropy regularization for BERT Bose with 12 layers and BERT Large with layers. All networks are pre-trained on large-scale lext corpus (Devlin et al. 2019) and fine-tuned with batch size of 32. Larger numbers are written in bold.
layers the RTE (Bentivogli et al. 2009 MRPC 2018) dataset.
DDolan & Brockett 2005 and CoLA (Warstadt et al.
Following Devlin et al. (2019), we fine-tuned networks with a batch-size of 32 and a learning rate that we found through a grid-search E [le 3e - 5, 5e 5]. Similar to the findings about residual networks, batch-entropy regularization improves the performance of trained models for transformer models on natural language processing tasks (hypothesis [ Only for BERT Base trained on RTE, the model was slightly worse when including LBF. For the rest of cases, there was an improvement when including LBε regularization. Of special interest is the MPRC, where the BERT Base with LB had a higher score (89.13), than the BERT Large without LB (89.06). These latter results show that those pre-trained models are not fully exploited yet and the cownrsstremmmpeffommancc can be further improved without the need of additional pre-training.
We finalise our analysis by exposing some interesting facts regarding the differences between residual networks and transformer models. For residual networks, we found that the performance improved most when the a regularizes the information flow-is large (* 1.0). On the other hand, for transformer models, small value (+x 0.3) improved their performance the most, which would indicate that the compression and reduction of information are beneficial. Our insights this observation are follows: Transformer models implement pooling layer just before the classifier which pools all output tokens (c.g. 128, 256, 512) except for the first token. Compressing much information as possible into the first token therefore would help for classification. A complete and precise evaluation of this hypothesis is out of scope for this paper and is left for future work.
5 Conclusion and Discussion
In this paper, we analyzed the flow of information through deep neural networks. We introduced the batch-entropy to approximate the amount of information that is propagated through each individual layer. A positive batch-entropy necessary at each layer of the network in order to optimize a given objective function with gradient descent correctly as shown in section 3 Through dimensionality reduction and the analysis of the loss surface, found that whenever the information flow through network is zero, the gradient is also zero almost everywhere.
The batch-entropy regularization term presented in this work optimizes the flow of information through the network by regularizing the batch-entropy each layer individually. We showed through extensive ex- perimental evaluation that networks not trainable first, could be trained thanks the inclusion of the proposed regularizer. Blot al. (2018) also proposed regularizer in order improve the intermediate representations of single samples x but they calculate the entropy of a sample conditioned to the correspond-
15

---

Published in Transactions on Machine Learning Research (07/2022)
ing label y. The authors showed that model that is invariant to many transformations will produce the same representation ion for different inputs, which improves the performance of the model. Instead of penal- izing layer-transfommtions, we add a regularization term to ensure positive flow of information through each layer. Our empirical ovaluation also showed that this regularization term has a positive effect on the performance of a trained model for a wide range of lasks and architectures. Specifically, such positive effect was consistent across different networks, such as fully connected, residual, autoencoder, and transformer networks applied to computer vision and natural language processing tasks.
The results of our experimental evaluation are very promising and we think it can to improve future deep learning research, even so, we would like to finish our analysis with some open questions. First, even though the a value which regularizes the information flow can be consistently set for a network architecture trained on a given task, the value currently determined through a grid search. One solution worth of further analysis to obtain a good initial value for a from an analysis of the entropy of training data, this is line of work on which we are currently working. Second, the batch-entropy is currently computed through different samples of mini-batch and can therefore only be derived whenever the batch size is larger than one. If a batch size of one is used, similar methods gradient accumulation can be better suited, which is worth investigating in future work. Finally, in the case of transformer models, we found that compression of information improves scores, which is a strong indicator that the current pooling layer is not optimal w.r.t a given downstream task. Further research could include not just regularizing networks with LE,, but also a novel pooling layer to improve scores. To end with a final positive outcome, we found a positive effect of batch-entropy regularization on generalization and in our future work we would like to additionally investigate, whether this same method has also positive effect the adversarial robustness of trained model.
References
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, (9(1)11947-199) 1980, 2018
Madini O Alassafi, Mutasem Jarrah, and Reem Alotaibi. Time series predicting of covid-19 based on deep learning. Neurocomputing, 468:335-344, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statisticnl Sciences, 6(1):17-39, 1997.
Luisa. Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL http:::/wwwwwwwdeee com/ Soft- ware available from wandb.com.
Michael Blot, Thomas Robert, Nicolas Thome, and Matthieu Cord. Shade: Information-based regularization for deep learning. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 813-817. IEEE, 2018.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. Second edition, 2006.
G. E. Dahl, T. N. Sainath, and G. E. Hinton. Improving deep neural networks for lvesr using rectified linear units and dropout. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 9609-8613, 2013.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
16

---

Published in Transactions on Machine Learning Research (07/2022)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In NAACI-HLT (1), pp. 4171-4186. Association for Computational Linguistics, 2019.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Ran Bachraa, Amir Navot, and Naftali Tishby, An information theoretic tradeoff between complexity and accuracy. In Learning Theory and Kernel Machines, pp. 595-609. Springer, 2003.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep foedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256,
Chia Laguna Resort, Sardinia, Italy, 13 15 May 2010. PMLR. URL http://pproceedngss.mi prera/ v9/ 19/glooott0a.hce html
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, Dec 2015. doi: 10.1109/1CCV.2010 2015.123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
G. E Hinton, S. Osindero, and Y. W. Tch. fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
Sepp Hochreiter and Jürgen Schnidhuber. Long short-lerm memory. Neural compulation, 9(8):1735 1780, 1997.
Sergey Tofle and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neurol information processing systems, 30, 2017a.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017b.
Alex Krizhevsky, Vinod Nair, and GeolTrey Hinton. Cifar-10, 2009. URL http:://ww.ccr wttp///owwwc...ootoooooo toronto...... edu/
Krrizccfar..o chml
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster), 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. http://yammllecu.comm /xddb/ /exd//emist//////////////m////// /mnist/ 2010. URL hattp://yann.eeeee www.eecwwooo/eeeeeeoooooe.e.eeee wwt.c/cm//eei/eee exdd/mnis///////////////////mm/m
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Newral Information Processing Systems, 31, 2018.
James Martens and Roger Grosse. Optimizing neural networks with tronecker-factordd approximate curva- ture. In International conference on machine learning, pp. 2408-20 2417. PMLR, 2015.
James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszez, Valentin Dalibard, Jascha Sohl- Dickstein, and Samuel S Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv preprint ar arXiv:21 10.01761 2021.
17

---

Published in Transactions on Machine Learning Research (07/2022)
Joseph Victor Michalowicz, Jonathan M Nichols, and Frank Bucholtz. Handbook of differential entropy. Cre Press, 2013.
Dmytro Mishkin and Jiri Matas. All you need is good init. In Yoshua Bengio and Yann LeCun (eds.); ICLR (Poster), 2016.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
David Peer, Sebastian Stabinger, and Antonio Rodriguez-Sanchez. Auto-tuning of deep neural networks by conflicting layer removal, 2021a.
David Peer, Sebastian Stabinger, and Antonio Rodrrguez=SSnchee. Couflicting bundles: Adapting architec- tures towards the improved training of deep neural networks. In Proceedings of the IEFE/ CVF Winter Conference on Applications of Computer Vision, pp. 256 265, 2021b.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR (Workshop), 2017.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential ex- pressivity in deep neural networks through transient chaos. Advonces in newral information processing systems, 29, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference machine learning, pp. 2847-2854. PMLR, 2017.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in newral information processing systems, 29, 2016.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379 423, 1948.
Ravid Shwartz-Wh Ziv and Naftali Tishby. Opening the Black Box Deep Neural Networks via Information. arXiv: 00810 [es}, April 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent noural networks from overfitting. The journal of machine learning research, 15(1): 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. Deep Learning Work- shop, International Conference on Machine Learning, 2015.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36nd International Conference on Machine Learning, 2019.
Naftali Tishby and Noga Zaslavsky. Deep Learning and the Information Bottleneck Principle.
arXiv:1503 150300000 [es], March 2015.
Khoa A Tran, Olga Kondrashova, Andrew Bradley, Elizabeth D Williams, John V Pearson, and Nicola Waddell. Deep learning in cancer diagnosis, prognosis and treatment selection. Genome Medicine, 13(1): 1 17, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30,
2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE transactions image processing, 13(4):600-6) 212,, 2004.
18

---

Published in Transactions on Machine Learning Research (07/2022)
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint Xiv: 1805. 124 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking 2017.
machine learning algorithms. arXiv preprint
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Diekstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pp. 5393-5402. PMLR, 2018.
Aolin Xu and Maxim Raginsky. Information-theoredt analysis of generalization capability of learning algo- rithms. In NTPS, 2017.
Yinqiu Xu, Xuanyi Li, Hequan Yao, and Kejiang Lin. Neural networks in drug discovery: current insights from medicinal chemists, 2019.
A Distribution of Individual Neurons
The distribution of output values of 25 randomly selected neurons for 128 different output values is shown in fig. 7
The neural network was trained the MNIST dataset.
H=0.5
H=0.0
H=0.6
H=0.2
H=0.4
H=0.4
H=0.6
H=0.8
H=0.5
H=0.4
H=0.4
H=0.7
H=0.0
H=0.8
H=0.4
H=0.0
H=0.0
H=0.4
H=0.7
H=100
H=0.0
H=0.9
H=0.2
H=1.0
H=0.5
Figure 7: Histogram of 1024 output values of several neurons from fully connected network trained on MNIST. The x-axis shows the output-value for each neuron and the y-axis depicts the number of times that output-value was observed. On the top each plot, the estimated batch-entropy eq. shown.
B Information Theoretic Perspective on the Entropy of Subsequent Layers
We denote random variable with a capital letter, c.g., X. single event denoted with lower case letter x E X (note that can be multi-dimensional). The probability density function of a continuous random variable is given be p(x) and the mutual information between random variables X and Y is given by T(X;Y).
19

---

Published in Transactions on Machine Learning Research (07/2022)
A neural network can be interpreted as Markov Chain Tishby & Zaslavsky 2015 Shwartz-Ziv & Tishby 2017), where each hidden layer is a (stochastic) map of the previous layer. The training data is represented by random variables X and Y, for the objects and labels respectively, and have joint probability distribution p(z,y) y) with ((; > The output a hidden layer represented by random variable Ti and the output of the network represented by random variable Y. A neural network with L hidden layers forms the following Markov Chain:
Y x - T2
T -
(10)
where Y X indicates that the faclorization of p(x,y) y) is not known. The Dala Processing Inequality (DPI) (Cover& & Thomas 2006) gives the following inequalities:
(Y; I(YT)) 2 I(Y ... 2((:T)) 2 1(Y;)).
(11)
Which implics that our estimate Y can at most contain as much information about Y as X. It also implics that each hidden layer Ti can at most contain as much information about Y as the previous hidden layer Ti-1-
In the worst case all, information about the input gets lost and we will have I(X;T) 0 for some hidden layer I. Given that the only information Ti can contain about Y is through X, we also have ;(Y;T)) 0. The DPI tells us that T(X; Ti) T(X; T1+1) ... T(X; 8), hence T(X; TT) T(X;T++) The output y contains no information about the input X (and Y).
T(X; Ý) 0.
A standard neural network is deterministic, which is as special case of the problem described above. The mutual information between input X and layer I can be written as:
I(x:T)) = H H IX),
(12)
where H(Ti) is the (differential) entropy of layer and H(T X) the conditional (differential) entropy of layer given input X, ie., the entropy that cannot be explained by input X. The conditional entropy represents external information, e.g., noise, that is being added during the processing steps from input X to layer 1. For deterministic neural networks it is safe to assume no external information is added, i.c., H X) O. Given T(X;T:) 0, this implies H(T)) H(T: X) = 0. In other words, if no information besides the input is added to the network and a layer contains no information about the input, it must be zero entropy.
C Derivation of the Gradient if the Batch-Entropy is Zero
Without loss of generality we can assume that labels are onc-hot encoded vectors y that are uniformly distributed w.r.t different classes in our mini-batch B. we assume that the batch-entropy is zero, follows that
20

---

Published in Transactions on Machine Learning Research (07/202) 2022)
0C(B)
(xbx
Gradient of
OWL
[
OWL
(g(x)) yo) at (xx)) )
Gradient defined in section 3.1
B
(g Vo) (xx)))
H =0 g(xo) g(x1) -x)((x)))) g
B
[8]
(g
y6)
a²
H = = ax(xo) a'(xx) = =..aa(x BB) =
1=1
1
g
Yb
a²
T
Rewrite
B
b=1
g
a
Uniformly distributed batches w.r.t different classes of yb
where 1 is a vector with components of dimension e for dataset consisting of e classes.
We next evaluate the gradient not only the cross-entropy is used to train a model, but also batch-entropy regularization as proposed in cq. (7)
Oc
dL
dCoE
oWi = =0+ Loe
+0
Assumption that
Ow
ow
A
B
LCF OWG
Σ
L
K=U
LCB OWi
L C
3
0C
LoE L OW!
Interpretation of Batch-Entropy Regularization.
The following illustration shows why networks with batch-entropy regularization are trainable whenever the batch-entropy zero.
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
Lez V i8 # 0
Vo
Lex V V 0
Vaa
2cp V V
Vg
Vg
Pes V Ds
V P.ecB
(a) Backpropagation without batch-entropy regularization
(b) Backpropagation with batch-entropy regularization
Figure 8: Comparison of backpropagation without (a) and with (b) LB regluarization whenever H² 0.
E
Loss Surfaces of LCE, LE and Accuracy
In this appendix we show the loss surfaces for the Lb well the Lee loss after different stages of training. It can be seen that weights of the nelwork are never pushed into regions that can be optimized w.r.t. the objective function, if the batch-entropy regularization is not used as shown in fig, g fig. and fig. 11
(a) Training without Las Loe (b) Training without LBs LBe
(c) Training without Lbe Accuracy
Loss
Loss
118 Sear
go
(d) Training with LBE Les Loss (e) Training with CB LBE Loss Training with LEe Accuracy
Figure Different loss surfaces and accuracy for network with 30 layers the beginning of training on MNIST. The LAE Loss points away from the local minimum leading to network that can be optimized with cross entropy.
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
ost
(a) Training without LB Los (b) Training without LB LBS
(c) Training without LBF Accuracy
Loss
Loss
(d) Training with LBE Los Loss (e) Training with LB LBE Loss (f) Training with LBE Accuracy
Figure 10: Different loss surfaces and accuracy for network with 30 layers trained on MNIST. After training for 1k steps the LB is close to the cross entropy loss such that the focus of optimization is on minimizing the real objective function (cross entropy) rather than the information flow (LBε)-
(a) Training without Las Loe (b) Training without LBs LBE
(c) Training without Lbe Accuracy
Loss
Loss
(d) Training with LBE Les Loss (e) Training with CB LBE Loss Training with Cee Accuracy
Figure 11: Different loss surfaces and accuracy for a network with layerstrained on MNIST. After training for 2k steps the LBE is close to the entropy loss such that the focus of optimization is minimizing the real objective function (cross entropy) rather than the information flow (LBE).
23

---

Published in Transactions on Machine Learning Research (07 (07/2022)
F Comparison with Normalization Methods
Normalization methods are used to improve the trainability of deep neural networks. In this section we will show that, four commonly used normalization methods batch batch normalization (Toffe Szegedy 2015). layer normalization (Ba al. 2016), weight normalization (Salimans & Kingma, 2016) and nelf-normalizing
neural networks (Klambauer et al. 2017b cannot convert untrainable networks into trainable networks.
For each normalization method we train four 'vanilla' fully connected network of different depths the MNIST dataset. Each network has 500 layers with 1000 neurons per layer and the ReLU activation function is used-cxcept for the self-normalizing ncural network which uses the SELU activation function. Each network Irained for 500 epochs using the cross entropy loss and the Adam optimizer. We evaluated different learning rates and only show the results for the best performing learning highest accuracy on the test set. The results are shown in Table 5 We were not able to train "vanilla" fully connected neural network of 500 layers with any of the normalization methods. As opposed to LBE regularizer, which does allow us to train such deep network.
Normalization Method
Test Accuracy
Batch Normalization
11.74
Layer Normalization
11.35
Self-Normalizing Neural Networks
11.35
Weight Normalization
11.35
LBE Regularization
95.20
Table 5: Comparison of different normalization methods with the LBE regularizer. Results are obtained using a fully connected neural network of 500 layers and 1000 neurons per layer. Different learning rates and seeds were tested and only the best performing model in terms of test accuracy is shown.
G Hyperparameter a
The selection of the hyperparameter is critical for batch-entropy regularization in order to successfully train the network. In the experimental section 4 we executed a grid search in order to find optimal values for a (values are reported in appendix H.. In this section correlations between different hyperparameter and a for fully connected networks as well residual networks are shown.
Fully Connected Networks We trained fully connected networks with learning rates E [3e 5, 5e 5, - 4), depths [10, 20,30, 30, 40, 50, 500 and evaluated which value for a (with a E [0.5, 1.0, 1.5)) producod the result with highest performance. We report the mean of five runs with different seeds. Results for experiments with different depths well as learning rates are shown in fig. 12
24

---

Published in Transactions on Machine Learning Research (07 (07/202) /202))
1.0
0.104
0.6
LBT 4pha
0.6
0.100
0.4
0.000
0.2
0.546
0.8
6.
10000ptht)))
Learning Rate
1s-4
(a) for different depths.
(b) for different learning rates.
Figure 12: Shows the optimal selection of for fully connected networks trained MNIST
In fig. 12a it can be seen that larger a values are preferable as the depth of the network increases. is reasonable that the flow of information through all the layers must be larger for deeper networks.
On the other hand it can also be seen that for all different learning rates 0.1 works best which indicates that does not depend on this hyperparameters and can be selected independently.
We trained residual networks with depths E [26, 74, 152] on MNIST, FashionMNIST, CIFAR10, and CI- FAR100 and evaluated which value for a (with E [0.5, 1.0, 1.5)) produced the result with the highest performance for five different runs. The results are shown in fig. [2
1.20
14-
3.15
12-
Ab1e
3.10
08
3.55
3.00
06-
60
60
100
120
WWIST
FashionWISST
CIARRD
CENATIC
Imagener
Dopth
Doland
(a) for different depths.
(b) for different datasets.
Figure 13: Shows the optimal selection of for residual networks.
Similar to our findings in fig. 12a for fully connected networks, we can see the same pattern also for residual networks in fig. 13a as the optimal alpha value increases with the depth of the network.
Additionally we report in fig. 12b the optimal value for different datasets. It can be seen that the complexity of the dataset increases, a larger a value is desirable. For example for MNIST, which are 10 different classes containing grayscale images the optimal a value is smaller when compared to the more complex colored images of CIFAR10. can be seen that as the number of classes increases e.g. from CIFAR10 CIFAR100 larger a helps further improve the performance of the network.
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2022)
H Hyperparameter Tuning LBE
In this appendix, report the hyperparameters and 3 that used for batch-entropy regularization.
We found the values through grid search. Additionally, we report the validation and test accuracy for each setup.
H.1
Fully Connected Networks
Depth
Alpha
Beta
Val. Accuracy
Test Accuracy
Std. Error
10.0
0.0
0.0000
97.96
98.94
0.01
0.1
0.0005
97.97
98.95
0.01
20.0
0.0
0.0000
95.83
97.86
0.60
0.1
0.0001
96.83
98.48
0.13
30.0
0.0
0.0000
11.17
11.35
0.00
0.1
0.0010
88.70
92.84
1.86
40.0
0.0
0.0000
11.17
11.35
0.00
0.1
0.0005
88.10
93.51
0.67
50.0
0.0
0.0000
11.17
11.35
0.00
0.3
0.0001
87.45
92.93
1.22
Table Evaluation of fully connected works.
H.2 Residual Networks
26

---

Published in Transactions on Machine Learning Research (07/2022)
Dataset
Depth
Alpha
Beta
Val. Accuracy
Test Accuracy
Std. Error
MNIST
26
0.0
0.000
99.51
99.68
0.02
0.5
0.010
99.51
99.67
0.01
74
0.0
0.000
99.47
99.67
0.02
0.5
0.005
99.48
99.68
0.01
152
0.0
0.000
99.39
99.59
0.02
0.001
99.42
99.66
0.03
MNSST
26
0.0
0.000
93.57
94.33
0.06
0.5
0.005
93.69
94.25
0.06
74
0.0
0.000
93.32
93.89
0.09
1.0
0.010
93.39
94.18
0.09
152
0.0
0.000
92.76
93.33
0.13
1.5
0.001
92.99
93.55
0.17
CIFAR10
26
0.0
0.000
89.49
89.40
0.03
1.0
0.005
89.59
89.41
0.08
0.0
0.000
89.71
89.70
0.20
0.5
0.005
90.04
89.89
0.22
152
0.0
0.000
88.20
88.16
0.31
1.0
0.005
88.66
88.54
0.19
CIFAR100
26
0.0
0.000
62.24
62.85
0.10
1.5
0.005
62.53
63.12
0.05
74
0.0
0.000
63.02
63.95
0.30
1.5
0.001
64.03
64.63
0.15
152
0.0
0.000
61.32
62.47
1.40
1.5
0.005
63.82
64.68
0.49
Imagenet100
26
0.0
0.000
48.65
51.08
0.22
1.5
0.010
48.78
51.47
0.25
74
0.0
0.000
49.03
51.84
0.21
1.0
0.001
49.42
52.28
0.20
152
0.0
0.000
47.97
50.33
1.92
1.5
0.010
49.47
52.45
0.28
Table Evaluation of residual networks.
27

---

Published in Transactions on Machine Learning Research (07 (07/2022)
H.3 Autoencoder
Dataset
Model
Depth
Alpha
Beta
Val. MSSTM
Test MSSTM
Std. Error
INSST
AE
0.0
0.0
0.61
0.61
0.0005
1.5
0.1
0.61
0.61
0.0007
0.0
0.0
0.54
0.53
0.0138
1.5
0.2
0.58
0.57
0.0009
10
0.0
0.0
0.13
0.13
0.0001
1.5
0.2
0.41
0.41
0.0084
25
0.0
0.0
0.13
0.13
0.0002
2.0
0.5
0.41
0.41
0.0059
MNIST
AE
0.0
0.0
0.75
0.75
0.0013
1.5
0.1
0.75
0.75
0.0017
5
0.0
0.0
0.11
0.12
0.0005
2.5
0.5
0.64
0.65
0.0071
10
0.0
0.0
0.11
0.12
0.0003
1.5
0.1
0.41
0.42
0.0038
25
0.0
0.0
0.11
0.12
0.0004
1.5
0.1
0.40
0.39
0.0168
Table 8: Evaluation of autoencoders.
H.4 Transformer
Dataset
Model
Alpha
Beta
Val. Performance
Test Performance
Std. Error
RTE
BERT Base
0.0
0.000
68.42
67.42
0.89
0.5
0.010
69.00
67.34
1.54
BERT Large
0.0
0.000
70.38
69.76
1.20
0.2
0.010
72.17
70.78
0.87
MRPC
BERT Base
0.0
0.000
86.62
88.58
0.70
0.2
0.005
86.80
89.13
0.88
BERT Lorge
0.0
0.000
87.04
89.06
0.37
0.3
0.005
88.34
89.28
0.55
CoLA
BERT Base
0.0
0.000
58.48
55.63
0.97
0.2
0.050
58.81
56.70
1.04
BERT Large
0.0
0.000
63.22
59.39
0.71
0.3
0.005
64.31
60.13
0.63
Table 9: Evaluation of transformers.
28


Reviews:
[{"review_id": "gfaperGohG", "reviewer": "Reviewer_LvEn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nGeneral comments: The approach is novel however strongly related\nto normalization as the proposed term is just log variance. \nThe method is not well-embedded into prior literature, especially\nit misses out on the signal propagation theory. It is unclear\nwhether this approach is relevant because the experiments are \nmostly done on small scale datasets and far from SOTA. \n\nPositive aspects/strengths:\n- Presentation and writing of the paper is clear, well-structure and concise\n- Experiments try to tackle a wider variety of tasks to show the generality of the method\n(however, mostly small datasets and far from current SOTA)\n\n\nMajor comments/weaknesses:\na) Novelty: The approach is novel but closely related to all normalization techniques\nand signal propagation theory. The main quantitity, Eq (2) and Eq(3), \nis just the log variance in the layer. The theory revolves around \nH^l being zero, which indicates that the variance of the neuron activations\nis zero. It is well-established in the community that this leads \nto untrainable networks. This is the reason why normalization techniques\nhave been developed (references [1]-[4]).\nThere is also a huge body of work on signal propagation in random networks\n(references [5]-[8]) which also focus, for example, on keeping the variance\nof the neuron (pre-) activations in a certain range.\nThe authors should embed their work into other related works and clearly\nstate their novel contributions, for example at the end of the introduction section.\n\nb) Relevance: The experiments are performed on small scale datasets such \nas MNIST, CIFAR10, CIFAR100 and in a performance range of methods from \nmany years ago. Because of the substantial closely related methods, such \nas all normalization techniques, that were used to train networks on \nmuch larger datasets (ImageNet, etc) and at higher performance ranges, it \nis unclear whether this method is relevant at all. There are many possible\nways in which variance (or information) can be maintained or transformed\nthrough neural network layers, such that a clear improvement over those \nrelated techniques should be demonstrated. Improvement over un-normalized \ndeep neural networks are hardly relevant since it is established that specialized \ntechniques are necessary to train them. \nThe authors should demonstrate the effectiveness of their approach/method \non larger datasets and with architectures operating at usual performance ranges\nclose to or at state-of-the-art. Methods compared should include the different\nnormalization techniques ([1]-[4]) and methods from signal propagation theory (e.g. [5]-[8].\n  \n\nMinor:\nEq (3): some error as i does not appear in the formula on the right\n\n\n\nReferences:\n[1] Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR.\n[2] Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.\n[3] Salimans, T., & Kingma, D. P. (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29.\n[4] Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). Self-normalizing neural networks. Advances in neural information processing systems, 30.\n[5] Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S., & Pennington, J. (2018, July). Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning (pp. 5393-5402). PMLR.\n[6] Lee, Jaehoon, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. \"Deep neural networks as gaussian processes.\" arXiv preprint arXiv:1711.00165 (2017).\n[7] Martens, J., Ballard, A., Desjardins, G., Swirszcz, G., Dalibard, V., Sohl-Dickstein, J., & Schoenholz, S. S. (2021). Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping. arXiv preprint arXiv:2110.01765.\n[8] Blumenfeld, Y., Gilboa, D., & Soudry, D. (2020, November). Beyond signal propagation: is feature diversity necessary in deep neural network initialization?. In International Conference on Machine Learning (pp. 960-969). PMLR.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an approach to regularize information\nflow through neural network layers. This leads to a differentiable\nloss term that is added throughout training.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Requested changes are formulated \"the authors should [..]\" in my statements above. ", "broader_impact_concerns": "Ok. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0uG1BnZKtd", "reviewer": "Reviewer_U3Z5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* I liked Figure 2, which shows that different layers indeed has different entropies. Further, the change in entropy from one layer to the next seems to be continuous, which is rather interesting though I am not sure if this has any significance.\n* The style of identifying a property that makes networks untrainable, and then fixing it to increase trainability is very interesting to me and I applaud the authors for that.\n\nWeakness\n* As per TMLR guidelines, there is no page limit but the length of the paper should determine how rigorous the review should be. In short, the length should justify the contribution. In this paper, I am strongly of the opinion that it does not. The main text is about 15 pages long, which is rather long for the contributions.\n* One of the main points seem to be that Hypothesis 1, which states that a network is not trainable with gradient descent of there is a layer where the entropy is zero, which essentially means that when the output of the layer is invariant to the input. But that is quite obvious that if the output does not depend on the input, the gradient will be zero as it just a constant function. Then the network will indeed be untrainable. Could the authors please clarify if this is an incorrect interpretations ? \n* Regarding the experimental contribution, first, the choice of $\\alpha$ seems important but is not discussed properly, especially how do you choose $\\alpha$ for different datasets and networks ? How does the performance change with different choices of the hyper-parameter i.e. how brittle is it ?\n* Finally, the experimental results are not convincing at all. In CIFAR10 and MNIST, there is no separation between using the regulariser or not using the regulariser. In fact, shallower networks actually perform better. in CIFAR100, the difference is less than 1% and similar with FasionMNIST. For residual layers, the observation is same as for  fully connected layers.Even for BERT, there is no real difference (within 1%). For auto-encoders there appears to be some difference for deeper networks, which is perhaps the most positive experimental contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a measure for the information flow through each layer of the network. The proposed measure is the layerwise batch entropy, which is zero when the variance of the output of a layer is zero and large when the variance is large. The main observation of the paper is that when this entropy is small, deeper networks are not-trainable. Then the paper proposes a regulariser to control this entropy and show that deeper networks are trainable when this entropy is controlled.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* Critical - The main paper should be shortened to a much smaller length in my opinion.\n* Critical - The theoretical section, right now, does not appear to have a major contribution for this paper, in my opinion. So, I think one way to improve it would be to give insights on how to choose $\\alpha$, though I realise that is a difficult problem.\n* Critical - The experimental results need to be strengthened. I think the authors need to make a clear case for when to use this regulariser is useful. Without it, I will be tending towards rejection.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yZ4furfvtU", "reviewer": "Reviewer_H1f1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Training deep neural networks is a critical problem in deep learning. A vanilla network without shortcuts occupies less memory. It is also easy to implement for fast inference.\n\n-The proposed batch-entropy regularization is reasonable to make the gradients stable. Without any architectural tweak, an over 500 layers neural network can be well trained. \n\n-The proposed method can work well on multiple architectures, such as fully connected network transformers.\n\nWeakness: \n\n-The authors only conduct experiments on small datasets, such as CIFAR10, CIFAR100, FashionMNIST, and MNIST. The classification tasks on these small datasets are usually easy to achieve high performance. Besides, it also has a large difference from practical scenarios. \n\n- A network with 152 layers can be well trained. It is interesting to see whether a convolutional neural network over 1000 layers can be trained.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new strategy to train deep neural networks without shortcuts. It adds a batch-entropy regularization to optimize the information flow. With the regularization strategy, vanilla fully connected networks can be well trained. The authors conduct experiments on multiple datasets to validate the effectiveness.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "-The authors are required to validate the proposed method on larger datasets such as ImageNet.", "broader_impact_concerns": "No ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall this is a borderline submission. I do appreciate however the author's rebuttal and the improvements to the paper which I believe address most of the reviewer's concerns.

The reviewer's original concerns can be summarised in two key points:
1. Related work: Relation to existing work needs to be expanded and clarified substantially. Low theoretical novelty. 
2. Relevance: Experiments are only on small scale datasets, where high performance does not require very deep models. So, the proposed method is not tested on regimes where it would have impact.

The change of the focus towards "trainability of untrainable deep models" does sound more appropriate for the contribution. However, in the end of the day, the reason we want to train deeper models is still because we want to apply them to larger scale problems/datasets. So this change of focus does sound a bit like a way to avoid more criticisms of type 2) above. I hope this method will be tested in more interesting problems/datasets in future work.

While I initially agreed with points 1) and 2) and also that the draft was too long for the contribution; The revised manuscript satisfactory addresses point (1,2) in my view in light of TMRL guidelines. While the theoretical contribution is somewhat straightforward and partially addressed in previous work on layer-normalisation (eg. see refs [1-4, 5, 8] pointed out by Rev. LvEn), the newly added experiments  (Appendix F, Table 5) show that the idea does bring something new to the picture relative to existing normalisation methods. It is a common pattern in DL literature that some mathematically trivial ideas can have a big practical impact simply because they enable training of larger models or new model classes.

Decision: Accept (poster)