Batch Normalization Orthogonalizes Representations in Deep Random Networks
Authors: Hadi Daneshmand
Amir Joudaki
Francis Bach
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Batch normalization
Theory of deep neural networks
Markov chains
Random Neural Networks
Optimization for neural networks

Abstract: This paper underlines an elegant property of batch-normalization (BN): Successive batch normalizations with random linear updates make samples increasingly orthogonal. We establish a non-asymptotic characterization of the interplay between depth, width, and the orthogonality of deep representations. More precisely, we prove, under a mild assumption, the deviation of the representations from orthogonality rapidly decays with depth up to a term inversely proportional to the network width. This result has two main theoretical and practical implications: 1) Theoretically, as the depth grows, the distribution of the outputs contracts to a Wasserstein-2 ball around an isotropic normal distribution. Furthermore, the radius of this Wasserstein ball shrinks with the width of the network. 2) Practically, the orthogonality of the representations directly influences the performance of stochastic gradient descent (SGD). When representations are initially aligned, we observe SGD wastes many iterations to disentangle representations before the classification. Nevertheless, we experimentally show that starting optimization from orthogonal representations is sufficient to accelerate SGD, with no need for BN.

Batch Normalization Orthogonalizes Representations in Deep Random Networks
Hadi Daneshmand
Amir Joudaki
INRIA Paris
ETH Zurich amir joudakiiini ethz.
seyed. www.eemaannnnnniia........
Francis Bach INRIA-ENS-PSL Paris francis. bach@inria.fr
Abstract
This paper underlines subtle property of batch-normalization (BN): Successive batch normalizations with random linear transformations make hidden representa- tions increasingly orthogonal across layers of deep neural network. We establish a non-asymptotic characterization of the interplay between depth, width, and the orthogonality of deep representations. More precisely, under a mild assumption, we prove that the deviation of the representations from orthogonality rapidly de- cays with depth up term inversely proportional to the network width. This result has two main implications: 1) Theoretically, as the depth grows, the distri- bution the representation -after the linear layers- contracts to a Wasserstein-2 ball around an isotropic Gaussian distribution. Furthermore, the radius of this Wasserstein ball shrinks with the width of the network. 2) Practically, the orthog- onality of the representations directly influences the performance of stochastic gradient descent (SGD). When representations are initially aligned, we observe SGD wastes many iterations to orthogonalize representations before the classi- fication. Nevertheless, we experimentally show that starting optimization from orthogonal representations is sufficient to accelerate SGD, with no need for BN.
Introduction
Batch Normalization (BN) (loff and Szegedy 20155) enhances training across wide range of deep
network architectures and experimental setups (He et al., 2016 Huang et al. 2017 Silver et al. 2017). The practical success of BN has inspired research into the underlying mechanism of BN (San-) turkar et al., 2018: Karakida et al. 2019 Arora et al., 2019b Bjorck et al. 2018). BN influences first-order optimization methods by avoiding the rank collapse deep representation (Daneshmand jet al. 2020), direction-length decoupling optimization (Kohler et al. 2018). influencing the con- vergence of the steepest descent (Arora al.
2019b Bjorck al. 2018). and smoothing the op-
timization objective function (Santurkar et al. 2018: Karakida et al. 2019). However, the benefits of BN go beyond its critical role in optimization. For example, Frankle al. (2021) shows that
BN networks with random weights also achieve surprisingly high performance after only minor ad- justments of their weights. This striking result motivates study the representational power of random networks with BN.
We study hidden representations across layers of laboratory random BN with linear activations. Consider a batch of samples passing through consecutive BN and linear layers with Gaussian weights. The representations of these samples perturbed by each random linear transforma- tion, followed by a non-linear BN. At first glance, the deep representations appear unpredictable
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

after many stochastic and non-linear transformations. Yet, we show that these transformations or- thogonalize the representations. To prove this statement, we introduce the notion of "orthogonality gap", defined in Section 3 to quantify the deviation of representations from perfectly orthogonal representation. Then, we prove that the orthogonality gap decays exponentially with the network depth and stabilizes around a term inversely related to the network width. More precisely, we prove
E orthogonality gap = o (1-a) depph (a-
batchsize
av/width
holds for a >> that an absolute constant under a mild assumption. In probability theoretic terms, we prove stochastic stability of the Markov chain of hidden representations (Kushner 1967, Kushner and Yin 2003; Khasminskii 2011). The orthogonality of deep representations allows us to prove that the distribution of the representations after linear layers contracts to W ball around isotropic Gaussian distribution as the network depth grows. Moreover, the radius of the ball is inversely proportional to the network width. Omitting details, we prove the following bound holds:
G O (11 (1-a))(ppt depth (batchsize) + (batchsize)*
Wassersteing
av/width
The above equation shows how depth, width, and batch size, interact with the Gaussian approxima- tion of the representations. Since the established rate is exponential with depth, the distribution of the representations stays in a Wasserstein ball around isotropic Gaussian distribution after a few layers. Thus, BN not only stabilizes the distribution of the representations, which is its main promise (loffe and Szegedy 2015), but also enforces Gaussian isotropic distribution in deep layers.
There is growing interest in bridging the gap between neural networks, as the most successful para- metric methods for learning, and Gaussian processes and kernel methods, as well-understood classi- cal models for learning (Jacot al. 2018 Matthews al. 2018 Lee et al. 2019 Bietti and Mairal, 2019 Huang et al.) 2014). This link is inspired by studying random neural networks in the asymp- totic regime of infinite width. The seminal work by Neal (196)) sparks that a single-layer network resembles a Gaussian process as its width goes to infinity. However, increasing the depth may signif- icantly shift the distribution of the representations away from Gaussian (loffe and Szegedy (015)). This distributional shift breaks the link between Gaussian processes and deep neural networks. To ensure Gaussian representations, Matthews et al. (2018) suggests increasing the width of the net- work proportional to the network depth. Here. we show that BN ensures Gaussian representations even for deep networks with finite width. This result bridges the link between deep neural networks and Gaussian processes in the regime of finite width. Many studies rely on deep Gaussian repre- sentations in an infinite width setting (Yang et al. 2019 Sccoenholz et al.] 2017] Pennington al.] 2018 Klambauer et al. 2017 De Palma et al. 2019). Our non aspttsytt Gaussian approximation can be incorporated into their analysis to extend these results to the regime of finite width.
Since training starts from random networks, representations in these networks directly influence training. Hence. recent theoretical studies has investigated the interplay between initial hidden representations and training (Daneshmand et al., 20200] Bjorck et al.] 2018:: Frankle et al., 2021 Schoenholz al. 2017 Saxe et al.] [014]] Bahri et al.) 2020). In particular, is known that hidden representations in random networks without BN become correlated as the network grows in depth, thereby drastically slowing training (Daneshmand et al. 2020 He et al. 2016 Bjorck et al., 2018 Saxe et al.] 2014). On the contrary, we prove that deep representations in networks with BN are almost orthogonal. We experimentally validate that initial orthogonal representations can save train- ing time that would otherwise be needed to orthogonalize them. By proposing novel initialization scheme, we ensure the orthogonality of hidden representations. Such an initialization effectively avoids the training slowdown with depth for vanilla networks, with no need for BN. This observa- tion further motivates studying the inner workings of BN to replace or improve it for deep learning. Theoretically, we made the following contributions:
1. For MLPs with batch normalization, linear activation, and Gaussian weights, we prove that representations across layers become increasingly orthogonal up to a constant inversely proportional to the network width.
2. Leveraging the orthogonality, we prove that the distribution of the representations contracts to a Wasserstein ball around Gaussian distribution the depth grows. Up to the best of

---

our knowledge, this is the first non-asymptotic Gaussian approximation for deep neural networks with finite width.
Experimentally, we made the following contribution"
3. Inspired by our theoretical understanding, we propose a novel weight initialization for stan- dard neural networks that ensure orthogonal representations without BN. Experimentally. we show that this initialization effectively avoids training slowdown with depth in the ab- sence of BN.
2 Preliminaries
2.1 Notations
Akin to Daneshmand et al. 22000), we focus on a Multi-Layer Perceptron (MLP) with batch normal- ization and linear activation. Theoretical studies of linear networks is growing research area (Saxe et al. [2144] Daneshmand et al. 2020 Bartlett et al.) 2019 Arora et al. 2019ap. When weights are initialized randomly, linear and ninelinear networks share similar properties such as the rank col- lapse issue studied in (Daneshmand et al. 2020). For ease of analysis, we assume activations are linear. Yet, we will argue that our findings extend to non-linear Appendix(Gi
We use to denote batch size, and d to denote the width across all layers, which we further assume larger than the batch size d 2 n. Let Hr Rdx= denote representations for n samples in layer f. with Ho E [dx= corresponding to n input samples in the batch with d features. Successive representations are connected by Gaussian weight matrices We (O, Ia/d)
H = Vd Be,,H BN(M) = diag (MM /
(1)
where diag(.) zeros out off- off-diagonal elements of its input matrix, and the scaling factor 1/vd en- sures that all matrices {H&} have unit Frobenius norm (see Appendix 2). The BN function in Eq. ( differs slightly from the commonly used definition for BN as the mean correction is omitted.
However, Daneshmand al. (2020) shows this difference does not change the network properties qualitatively. Appendix further discusses about the role of the mean correction term in our analy- Readers can find all the notations Appendix Table
2.2 The linear independence of hidden representations
Daneshmand et al. (2020) observe that inputs are linearly independent, then their hidden represen- tations remain linearly independent in all layers as long as d 2?n2).. Under technical assumptions, Daneshmand et al. (2200) establishes a lower-bound on the average of the rank of hidden represen- lations over infinite layers. Based on this study, we assume that the linear independence holds and build our analysis upon that. This avoids restating the technical assumptions of Daneshmand et al. 2020 and also further technical refinements of their theorems. The next assumption presents the formal statement of the linear independence property.
Assumption Ai (a, f). There exists an absolute positive constant a such that the minimum singular value of H& is greater than (or equal to) for all k I,...,t.
The linear independence the representations is a shared property across all layers. However, the representations constantly change when passing through random layers. In this paper, mathe- matically characterize the dynamics of the representations across layers.
Orthogonality of deep representations
3.1 A warm-up observation
To illustrate the difference between BN and vanilla networks, we compare hidden representations of two input samples across the layers of these networks. Figure [ plots the absolute value of
Immlementations available http://gghhhcoooooaaaaaaaaaaaaan

---

cosine similarity of these samples across layers. This plot shows stark contrast between vanilla and BN networks: While representations become increasingly orthogonal across layers of a BN network, they become increasingly aligned in vanilla network. More specifically, we observe that BN able to orthognalized almost aligned representations; while, the vanilla network provides almost align representation of two orthogonal samples in deep layers. While this behaviour has been theoretically studied for vanilla networks (Daneshmand et al. 2020; Bjorek et al., 2018 Saxe et al.) 2014), up to the best of our knowledge, it is not theoretically analyzed for BN networks for networks with finite width regime. In the following section, we formalize and prove this orthogonalizing property for BN networks with finite widths.
Abnolute Cosine Similantty
1.0
BK Vanil
6.8
0.4
62-
10
20
30
40
50
Layer
Figure 1: Orthogonality: BN vs. vanilla networks. The horizontal axis shows the number of layers, and the vertical axis shows the absolute value of cosine similarity between two samples across the layers (d 32). Mean and 95% confidence intervals of 20 independent runs.
3.2 Theoretical analysis
The notion of orthogonality gap plays central role in our analysis. Given the hidden representation H E Rdxx, matrix H constitutes inner products between representations of different samples. Note that H H E RRax is different form the covariance matrix H /n E RRddd, The orthogonal- ity gap of И is defined the deviations of H" T Il from identity matrix, after proper scaling. More precisely, define V : Rdxn 10 + R_ as
V(H) = H
H
H
I
(2)
|
The following theorem establishes bound on the orthogonality of representation in layer f.
Theorem 1. Under Assumption A [c, E), the following holds:
E [[((ee)) < 2- 39)
2
+ 3n
(3)
avd
Assumption A is studied by Daneshmand et al. (2020) who note that A o, oo) holds as long as still prove that there is function of representations that d ((n2). If A1 does not hold, Appendix (
decays with depth up to a constant (see
The above result implies that BN is an approximation algorithm for orthogonalizing the hidden rep- resentations. we replace diag(M) - by (M)-1 in BN formula, in Eq. ( then all the hidden representation will become exactly orthogonal. However, computing the inverse of non-diagonal matrix is computationally expensive, which must repeat for all layers throughout training, and differentiation must propagate back through this inversion. The diagonal approximation in BN significantly reduces the computational complexity the matrix inversion. Since the orthogonality gap decays exponential rate with depth, the approximate orthogonality is met after only few layers. Interestingly, this yields a desirable cost-accuracy trade-off: For a larger width, the orthogo- nality is more accurate, due to term 1/Vd in the orthogonality gap, and also the computational gain more significant.
From a different angle, Theorem I proves the stochastic stability of the Markov chain of hidden rep- resentations. In expectation, stochastic processes may obey an inherent Lyapunov-type of stability

---

(Kushner 1967: Kushner and Yin 2003 Khasminskii 2011). One can analyze the mixing and hit- ting times Markov chains based on the stochastic stability of Markov chains. (Kmmeny and Snell 1976 Eberle| 2009). In our analysis, the orthogonality gap is a Lyapunov function characterizing the stability of the chain of hidden representations. This stability opens the door to more theoretical analysis of this chain, such as studying mixing and hitting times. For, example the stability can be used for mixing analysis of the chain {If1....; He,... J. Let π denote the stationary distribution of this chain. Under particular stochastic stability condition called geometric drift condition,
IE ([()) En~x [x([]]] a [[]][H)]] Es-x [x)
holds for a E (0, 1) and measurable function ş R + R (see Theorem 3.6 of (Hairer 2010 The drift condition holds if there exists a Lyapunov function L Rdxa - [0, 1] and constant K 2 such that the following holds:
E ((HHH+))) S (H(()) K
We prove that the above condition holds in Appendix Eq (12). However, such theoretical analyses may not be of interest to the machine learning community, hence we focus on the implications of these results for understanding the underpinnings of neural networks.
is helpful to compare the orthogonality gap in BN networks to studies on vanilla networks (Bjorck) iet al. 2018, Daneshmand et al.] 2020; Saxe et al. 2014). The function implemented by a vanilla linear network properties of the product of i..dd random matrices are the subject of extensive studies in probability theory (Bougerol et al., 2012). Invoking these results, one can check that the orthogonality gap of the hidden representations in vanilla networks rapidly increases since the rank of representations converges to one matrix as the depth grows (Daneshmand al., 2020). We refer readers to Ap- pendix D for more details about vanilla networks.
linear transformation as the product of random weight matrices. The spectral
al. (202T) proves if activation functions obey self-normalization property, then specific kernel of hidden representations becomes well-condition as the network depth grows. However, it is not clear how impose the self-normalization property. Here, we establish the whitening for an explicitly defined normalization used in practice.
Hamnarkably,Agaaraaa
3.3 Experimental validations
Our experiments presented in Fig. 22 validate the exponential decay rate of V with depth. In this plot, we see that log(V) Vi) linearly decreases for = 1,..., 20, then it wiggles around a small constant. Our experiments in Fig. suggest that the ((1/Vd) dependency on width almost tight. Since V(H₂) rapidly converges to a ball, the average of V(H,) over layers estimates the radius of this ball. This plot shows how the average of V(H,), over 500 layers, changes with the network width, validating the ((I/Vd) dependency implied by Theorem
lavg. orthegonality gap)
logro. 3isertwidenn)
5-32
-..0
Loglarthonnniltt gap)
-0.5
515
-45
-15-
-5.0
-2.0
-55
2.5
-3.0
30
3.0
3.5
4.0
Log/widlh)
4.5
5.0
5.5
Depth
(a) Orthogonality vs. depth
(b) Orthogonality vs. width
Figure 2: Orthogonality gap depth and width. Left: (og(V(HH:) vertically versus horizontally. Right:
log( 500 E V(He)) vertically versus log(d) horizontally. The chain starts from a diagonal Ho with one relatively large diagonal value. This structure imposes large orthogonality gap for Ho. Mcan and 95% confidence interval of 20 independent runs.
Consider an input matrix Ho E Rdxn containing n samples in Idd, Assuming that elements of this matrix are i..d. zero-mean and unit variance random variables, the minimum singular value of Ho
5

---

is greater than Vd- n.. In practical applications, the batchsize used for normalization is relatively smaller than the network width, hence | (1), holds. For the intermediate representations, also observed that (a,E) holds -for an independent from &- as long as d >>>>>> n. Let Oo denote the minimum singular value of Ho- For different values of d and n, we check whether A [0. 1og. 1000) holds. Fig. illustrates that this assumption hold for d = 2((n)), which is also confirmed by Daneshmand et al. 2020).
80
70
30
20
10
0
10
15
20
25
30
35
40
Figure 3: Validations for A
Pixel (n,, marks whether A holds in all 10 independent runs: The black failed in at least once (where dgg is minimum singular value Ha). The highlighting Ai holds for = (2(n)).
color indicates A. 1.1... 1000)
blue curve marks (n 28)
4 Gaussian approximation
4.1 Orthogonality yields Gaussian approximation
The established result on the orthogonality gap allows us to show that the representations after linear layers are approximately Gaussian. If He orthogonal, the random matrix WeHfi is equal in distribution to standard Gaussian matrix due to the invariance of standard Gaussian distribution under linear orthogonal transformations. We can formalize this notion of Gaussianity by bounding the Wasserstein-2 distance between the distribution of We and standard Gaussian distribution. Let W2 R2) denote the Wasserstein-2 distance between probability distributions of random variables R1. and R2 The next lemma formally establishes the link between the orthogonality and the distribution of the representations.
Lemma 2. Given G e [ with i..d. zero-mean 1/d-variance Gaussian elements, the following Gaussian approximation holds:
W2 (W, G/Vm) & 2nE (V(,))
(4)
Combining the above result with Theorem I yields the result presented in the next corollary. Corollary 3. For G Rddxa with i.i.d. zero-mean 1/d-variance Gaussian elements,
W2 (W,H,, G/VR) & 4n (1 2
6n
(5)
3
avd
holds under Assumption A a, e).
In other words, the distribution of the representations contracts to a Wasserstein ball around an isotropic Gaussian distribution the depth grows. The radius of the Wasserstein 2 ball is at most O11/ (/1///idde wwidth). As noted in the last section, A is extensively studied by Daneshmand et al. (2020) where it is shown that A1 a > 0, 00) holds as long as 22(22).
4.2 Deep neural networks as Gaussian processes
3 establishes the first non-asymptotic Gaussian approximation for deep random neural networks. For vanilla networks, the Gaussianity guaranteed only in the asymp- totic regime of infinite width (Garriga-Alonso et al. 2019 Neal 1996 Lee et al. 2019 Neal 1996
Leveraging BN, Corollary

---

Hazan and Jaakkola 2015). Particularly, Matthews et al. (2018) links vanilla networks to Gaus- sian processes when their width is infinite and grows in successive layers, while our Gaussian ap- proximation holds for networks with finite width across layers. Table 1 briefly compares Gaussian approximation for vanilla and BN networks.
Network
Width
Depth
Distribution Outputs
Vanilla MLP
infinite infinite
finite
Converges to Gaussian width Converges to Gaussian as width + oo In a O (width - 1/4) - ball around Gaussian
Vanilla Convnet
finite
BN MLP (Cor.
(in)finite
(in)finite
Table 1: Distribution of representations in random vanilla and BN networks. For the convolutional net- work, the width refers to the numher of channels. Results for Vanilla MI.Ps and Vanilla convolution networks are establish by Matthews et al. (2018), AndgGarriga-aooon el. 2019), respectively. Remarkably, Corollary 3 holds for MI. with lincar activations.
The link between Gaussian processes and infinite-width neural networks has inspired several studies to rely on Gaussian representations in deep random networks (Klambauer et al. 2017; De Palma al. 2019 Pennington et al. 2018, Schoenholz et al. 2017 Yang al., 2019). Assuming the rep- resentations are Gaussian, Klambauer et (2017) designed novel activation functions that improve the optimization performance, [De Palma et al. (2019) studies the sensitivity of random networks, Pennington et al. (2018) highlights the spectral universality in random networks, [Schoenholz et al. (2017) studies information propagating through the network layers, and Yang et al. (20T9) studies gradients propagation through the depth. Indeed, our analysis implies that including BN imposes the Gaussian representations required for these analyses. Although our result is established for linear activations, we conjecture similar result holds for non-linear MLPs (see Appendix G
5 The orthogonality and optimization
In the preceding sections, elaborated on the theoretical properties of BN networks in controlled settings. In this section, we demonstrate the practical applications of our findings. In the first part, focus the relationship between depth and orthogonality. Increasing depth drastically slows the training of neural networks with BN. Furthermore, we observe that as depth grows, the train- ing slowdown highly correlates with the orthogonality gap. This observation suggests that SGD needs to orthogonalize deep representations in order to start classification. This intuition leads us to the following question: If orthogonalization is a prerequisite for training, can we save optimization time by starting from orthogonal representations? To test this experimentally, we devised a weight initialization that guarantees orthogonality of representations. Surprisingly, even in a network with- out BN, our experiments showed that this initialization avoids the training slow down, affirmatively answering the question.
Throughout the experiments, we use vanilla MLP (without BN) with a width of 800 across all hidden layers, ReLU activation, and used Xavier's method for weights intialization (Glorot and Bengio, 2010). We use SGD with stepsize 0.01 and batch size 500 and for training. The learning task is classification with cross entropy loss for CIFARIO dataset (Krizhevsky al., 2009), MIT license). We use Py PyTorch (Paszke et al. 2019 BSD license) and Google Colaboratory platform with a single Tesla-P100 GPU with 16GB memory in all the experiments. The reported orthogonality gap is the average of the orthogonality gap of representation in the last layer.
5.1 Orthogonality correlates with optimization performance
In the first experiment, we show that the orthogonality of representations at the initialization corre- lates with optimization speed. For networks with 15, 30, 45, 60, and 75 widths, we register training loss after 30 epochs and compare with the initial orthogonality gap. loss (blue) and the initial orthogonality gap (red) as a function of depth. We observe that represen- tations are more entangled, i.e., orthogonal, when we increase depth, coinciding with the training slowdown. Intuitively, the slowdown is due to the additional time SGD must spend to orthogonalize the representations before classification. In the second experiment, we validate this intuitive argu- ment by tracking the orthogonality gap during training. Figure 4b plots the orthogonality gap of
the training
7

---

output and training loss for network with 20 layers. We observe that SGD updates are iteratively orthogonalizing representations, marked by the reduction in the orthogonality gap.
Orthogonality gap (red)
1.50
Training loss after 30 spochs
0.20
1.05
12
0.55
2.90
0.30
1.
6.90
6.15
20
30
70
Depth
opects
(a) gap and loss vs. depth
(b) gap and loss during training
Figure 4: Orthogonality and Optimization Left: the orthogonality gap at initialization (red, left axis) and the training loss after epochs (blue, right axis) with depth. Right: the orthogonality gap (red, left axis) and the training loss in each epoch (blue, right axis). Mean and 95% confidence interval of 4 independent runs.
5.2 Learning with initial orthogonal representations
We have seen that the slowdown in SGD for deeper networks correlates with the orthogonality gap before training. Here we show that by preemptively orthogonalizing representations, avoid the slowdown with depth. While in MPL with linear activations, hidden representations remain orthogonal simply by taking orthogonal matrices weights (Pennington al.) 2018, Saxe al.) 2014), the same does not hold for networks with non-linear activations, such as ReLU. To enforce the orthogonality in the absence of BN, we introduce a dependency between weights of successive layers that ensures deep representations remain orthogonal. More specifically, we incorporate the SVD decomposition of the hidden representation of each layer into the initialization of the subsequent layer. To emphasis this dependency between layers and to distinguish it from purely orthogonal weight initialization, we refer this iterative orthogonalization.
We take a large batch of samples n 2 d, as the input batch for initialization. Let us assume that weights are initialized up layer We-1 To initialize Wi, we compute SVD decomposition of the representations H₃ = V. where matrices Up E RRdxd and Ve E RRndd are orthogonal. Given this decomposition, initialize W, by
V VUT We 1
(6)
where V Raxd is orthogonal matrix obtained by slicing Ve Raxdd Notably, the in- verse in the above formula exists when is sufficiently larger than d It is easy check that (Well)) < (H+) holds for the above initialization (see Appendix F similar to BN. By enfore- ing the orthogonality, this initialization significantly alleviates the slow down of training with depth (see 5 with no need for BN. This initialization is not limited to MLPs. In Appendix H we com- pare iterative orthogonalization with BN-replacement method recently proposed by Brock et al. (2021). In Appendix ! we propose similar SVD-baser initialization for convolutional networks that effectively accelerates the training of deep convolutional networks.
Discussion
To recap, proved the recurrence of random linear transformations and BN orthogonalizes sam- plcs. Our experiments underline practical applications of this theoretical finding: starting from orthogonal representations effectively avoids the training slowdown with depth for MLPs. Based on our experimental observations in Appendix lution networks used in practice. In other words, a proper initialization ensuring the orthogonality
| we believe that this result extends to standard convo-
We may inductively assume that He is almost orthogonal by the choice of Wi,... Wi-1. Thus, Ex is invertible.

---

Training loss after 30 epochs
144
orthagonal representations
standard representations
11-
10-
Depth
Figure 5: Iterative orthogonalization. Horizontal axis: depth. Vertical axis: the training loss after 30 epochs for Xavier's initialization (blue), our initialization (red). Mean and 95% confidence interval of independent runs.
of hidden representations may replace BN neural architectures. This future research direction has the potentials boost the training of deep neural networks and change benchmarks in deep learning.
Although our theoretical bounds hold for MLPs with linear activations, our experiments confirm similar orthogonal stability for various neural architectures. Appendix Figure demonstrates this stability for MLPs with ReLU activations and a convolutional networks. This plot compares the evolution of the orthogonality gap, through layers, for BN networks with vanilla networks. For vanilla networks, we observe that the gap increases with depth. On the contrary. the gap decreases by adding BN layers and stabilizes at term that is constant with regard to depth. Based servations, we conjecture that hidden representations of modern BN networks obey similar stability. A more formal statement of the conjecture is presented in Appendix G
ob-
Lubana et al. 2021 experimentally compares properties of different normalization techniques in- cluding layer normalization (LN) (Ba et al.) (016). According to this study, LN does not necessarily orthogonalize the outputs of deep neural networks. Hence, more theoretical studies are required to understand the essence of different normalization techniques in deep learning.
Acknowledgements
We thank Gideon Dresdner, Vincent Fortuin, and Ragnar Groot Koerkamp for their helpful com- ments and discussions. This work was funded in part by the French government under management of Agence Nationale de la Recherche part of the "Investissements d' l'avenir" program, reference ANR-199-31AA-0001PPPAAAII
31A Institute). We also acknowledge support from the European Research Council (grant SEQUOIA 724063).
References
Agarwal, N., Awasthi, P., and Kale, S. (2021). A deep conditioning treatment of neural networks. In Algorithmic Learning Theory. PMLR.
Arora, S., Cohen, N., Golowich, N., and Hu, W. (2019a). A convergence analysis of gradient descent for deep linear neural networks. International Conference on Learning Representations.
Arora, S., Li, Z., and Lyu, K. (2019b). Theoretical analysis of auto rate-tuning by batch normaliza- tion. International Conference on Learning Representations.
Ba, J. L.., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arX.v:1607.06450.
Bahri, Y., Kadmon, J., Pennington, J., Schoenholz, S., Sohl-Dickstein, J., and Ganguli, S. (2020).
Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics.

---

Bartlett, P. L., Helmbold, D. P., and Long, P. M. (2019), Gradient descent with identity initializa- tion efficiently learns positive-definite linear transformations by deep residual networks. Neural computation.
Bietti, A. and Mairal, J. (2019). On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems.
Bjorck, J., Gomes, C.,, Selman, B., and Weinberger, Q. (2018). Understanding batch normaliza- tion. Advances in Neural Information Processing Systems.
Bougerol, P. et al. (2012). Products of random matrices with applications to Schrödinger operators. Springer Science Business Media.
Brock, A., De, S., Smith, S. L., and Simonyan, (2021). High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06177
Daneshmand, H., Kohler, J., Bach, F., Hofmann, T., and Lucchi, A. (2020). Batch normalization provably avoids rank collapse for randomly initialised deep networks. Advances in Neural Infor- mation Processing Systems.
De Palma, G., Kiani, B. T., and Lloyd, S. (2019). Random deep neural networks are biased towards simple functions. Advances in Neural Information Processing Systems.
Eberle, A. (2009). Markov processes. Lecture Notes at University of Bonn.
Frankle, J. Schwab, D. J., and Morcos, S. (2021). Training batchnorm and only batchnorm: On the expressive power of random features in cnns. ICLR.
Garriga-Alonso, A., Rasmussen, C. E.,, and Aitchison, L. (2019). Deep convolutional networks as shallow gaussian processes. International Conference Learning Representations.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics.
Hairer, M. (2010). Convergence of markov processes. Lecture notes.
Hazan, T. and Jaakkola, T. (2015). Steps toward deep kernel methods from infinite neural networks. arXiv preprint arXiv: /508.05133.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected con- volutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Huang, P.-S., Avron, H., Sainath, T. N., Sindhwani, V., and Ramabhadran, B. (2014), Kernel meth- ods match deep neural networks timit. In ICASSP.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML.
Jacot, A., Gabriel, F,, and Hongler, C. (2018). Neural tangent kernel: Convergence and generaliza- tion in neural networks. Advances in Neural Information Processing Systems.
Karakida, R., Akaho, S., and Amari, S.-i. (2019). The normalization method for alleviating patho- logical sharpness in wide neural networks. Advances in Neural Information Processing Sys- tems.
Kemeny, J. G. and Snell, J. L. (1976). Markov chains. Springer- Verlag, New York.
Khasminskii, R. (2011). Stochastic stability of differential equations, volume 66. Springer Science & Business Media.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). Self-normalizing neural net- works. Advances in Neural Information Processing Systems.
10

---

Kohler, J., Daneshmand, H., Lucchi, A., Zhou, M., Neymeyr, K., and Hofmann, T. (2018). Expo- nential convergence rates for batch normalization: The power of length angth-direction decoupling in non-convex optimization. arXiv preprint arXiv: /805. 1805.
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Kushner, H. and Yin, G. G. (2003). Stochastic approximation and recursive algorithmss and appli- cations. Springer Science & Business Media.
Kushner, H. J. (1967). Stochastic stability and control. Technical report.
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J. (2019). Deep neural networks as gaussian processes. International Conference on Learning Representations.
Lubana, E. S., Dick, R. P., and Tanaka, H. (2021). Beyond batchnorm: Towards a general under- standing of normalization in deep learning. NeurIPS.
Matthews, A. G. d. G., Rowland, M., Hron, J., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. arXiv preprint / 1271.
Neal, R. M. (1996). Bayesian learning for neural networks. Springer Science Business Media.
Newman, C. M. (1986). The distribution of Iyapunov exponents: Exact results for random matrices. Communicattonn mathematical physics.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., Vetito, Z., Raison, M., Tejani, A., Chil- amkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems.
Pennington, J., Schoenholz, S.,, and Ganguli, S. (2018). The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics.
Santurkar, S., Tsipras, D., Iyas, A., and Madry, A. (2018). How does batch normalization help opti- mization?(no, it is not about internal covariate shift), Advances Neural Information Processing Systems.
Sawa, T. (1972). Finite-sample properties of the k-class estimators. Econometrica: Journal of the Econometric Society, pages 653-680.
Saxe, A. M., McClelland, J. L.,, and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference Learning Representations.
Schoenholz, S. S., Gilmer, J., Ganguli, S., and Sohl-Dickstein, J. (2017). Deep information propa- gation. International Conference on Learning Representations.
Silver, D., Schrittwieser, J.. Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017). Mastering the game of go without human knowledge. nature.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. (2014). Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research.
Yang, G., Pennington, J., Rao, V., Sohl-Dickstein, J., and Schoenholz, S. (2019). mean field theory of batch normalization. International Conference Learning Representations.


Reviews:
[{"review_id": "y35HI_4PfE", "reviewer": "Reviewer_t2tK", "summary": "This paper studies the effect of batch normalization on representations at random initialization.  They formally show that representations become increasingly orthogonal with layer depth up to a term inversely proportional to layer width. Using this, they prove that the distribution of the output contracts to a Wassertein ball around a Gaussian distribution and demonstrate empirically that orthogonal outputs save training time.  Lastly, they propose an initialization that ensures orthogonality even without batch normalization, which they empirically test.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality/Significance:**\nMy greatest concern with this work is that it is not clear that it is a significant contribution beyond the ideas originally proposed in Daneshmand et al 2020 work \"Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks\".  If we consider the contribution of this work to be in the algorithmic introduction of \"iterative orthogonalization\", then this work has not come close to meeting the bar for experimental benchmarking. This being said I think the ideas proposed in this paper, especially the empirical observations in section 6 (i.e. \"SGD needs to orthogonalize deep representations in order to start classification\") are quite interesting.  However, all the experiments provided are on very small scale networks and a much more thorough experimental section would be required to demonstrate this phenomenon holds generally.\n\n\n**Clarity/Quality:**\nWriting is very clear, notation is nicely introduced, and assumptions are clearly defined.  However, some experimental details are difficult to determine from both the main text and captions.\n\n\n\n**Areas for Improvement:**\n\n- \"While the hidden representations of vanilla networks has been theoretically studied...up to the best of our knowledge, there is no theoretical analysis of BN networks\".  This seems like way to strong of a claim.  For example, see Daneshmand et al 2020 work \"Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks\" which you cite quite a bit or Lubana et al 2021 work \"Beyond BatchNorm: Towards a General\nUnderstanding of Normalization in Deep Learning\".\n- In general there seems to be quite a bit of overlap between this work and Daneshmand et al 2020.  Can you discuss the difference between your quantity of study \"orthogonality gap\" (equation 2 in your work) and their quantity \"(stable) rank\" (equation 4 in their work) and your upper bound (equation 3 in your work) and their lower bound (equation 5 in their work).\n- In 4.3 Experimental validation section you are connecting your bound in equation 3 very loosely to the empirics.  Can you construct a quantitive prediction from your theory that you can validate empirically?  Of course you don't know the value of $\\alpha$ but could you estimate it and then plot your bound on the same plot?  Or even plot your bound for varying levels of $\\alpha$ so we can see the shape?  Making this section more quantitative would improve the empirical evidence. Also its not clear what network architecture or data sets is being used here. In general hard to determine where experimental details are.\n- You write \"Corollary 3 establishes the first non-asymptotic Gaussian approximation for deep random neural networks\" and \"the distribution of the outputs contracts to a Wassertein 2 ball around an isotropic Gaussian distribution as depth grow...the radius...is at most $\\mathcal{O}(1/\\sqrt{\\text{width}})$\".  These seem contradictory?  Only asymptotically (infinite depth/width) does the distribution of outputs according to corollary 3 become Gaussian.  Am I missing something?\n\n\n**Specific Comments:**\n- [line 99-100] \"When weights initialized randomly, linear and non-linear networks share similar properties.\" => \"When weights [are] initialized...share similar properties [such as...]\"\n- [line 105] \"diag(M) zeros out off-diagonal elements\" => I found this initially confusing cause in equation (1) you have $diag(MM^\\intercal)$ not $diag(M)$.  Maybe just write $diag(\\cdot)$ when discussing the operator?\n- [line 129-130] \"We used highly correlated inputs for the BN network and perfectly orthogonal inputs for the Vanilla network to contrast their differences.\" => I don't understand this sentence.  In the caption there is no indication of this experimental detail.  Could you elaborate here or in the caption of Fig. 1?\n- [line 138] \"...is different from the covariance matrix... [used in batch normalization (equation (1))]\" you might want to emphasis this point?\n- [equation (2)] Why write $\\|I_n\\|^2_F$ and not just $n$?  I would simplify this expression and put $H^\\intercal H$ into numerator of first term.\n- [line 161] \"This stability opens the door to more theoretical analysis of this chain, such as studying mixing and hitting times\" => This is still a very general statement. Can you be more specific?  What sort of questions could this stability \"open the door\" to answering (potentially using mixing or hitting times as an analysis tool)?\n- [line 169-170] \"one can readily check that the orthogonality gap of hidden representations in vanilla networks...\" Could you summarize this result mathematically so we can compare to the bound in equation (3)?  \n- [fig. 2] \"vertically versus $d$ horizontally.\" => dont you mean $log(d)$ horizontally?\n- [fig. 4] You write earlier \"to distinguish it from purely orthogonal weight initialization, we refer to this as iterative orthogonalization\", but then in your label you write \"orthogonal\" and not \"iterative orthogonalization\" or \"IO\" or something like this.  Additionally you dont even compare to the original \"orthogonal\" initialization as introduced in Saxe et al.\n- [Fig. 4] The yaxis label for fig4.b is \"training orth\"?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "w-rY4wvmIW", "reviewer": "Reviewer_XJdL", "summary": "This paper considers deep linear networks with batch normalization. The main contributions are:\n1. Under a linear independence condition, it is proved that the orthogonality gap is the sum of a term that decreases exponentially fast with the depth, and a term that is basically number_of_samples / sqrt{width}. \n2. Under similar conditions, it is proved that the distribution of the hidden representation is close to the Gaussian distribution, and a Wasserstein distance bound is proved.\n3. It is shown empirically that a larger orthogonality gap at initialization makes training slower, and that SGD decreases the orthogonality gap during training. This paper further proposes an \"iterative orthogonalization\" algorithm, which can enforce orthogonality and accelerate training when the depth is large.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the contributions of this paper are interesting. Intuitively, orthogonality would help with training, and it is nice to see that batch normalization can increase orthogonality. It is also interesting to see that SGD can increase orthogonality during training (even though there is no proof). The iterative orthogonalization algorithm may also be useful in practice.\n\nI have the following concerns:\n1. The main results require the width to be at least n^2, and if I understand it correctly, the width is also equal to the input dimension. It is also mentioned that the prior work by Daneshmand et al. (2020) requires all inputs to be linearly independent, which also seems to be required in this paper. These assumptions are strong.\n2. As we can see in Figure 4 and 7, iterative orthogonalization always decreases the orthogonality gap, but it actually makes training slower for shallow (depth <= 20) networks.\n3. In Figure 7(b), only the loss curves after 20 epochs are plotted, at which time the losses are still high. Can you also provide the loss curves after maybe 200 epochs?\n\nFinally, the prior work \"A deep conditioning treatment of neural networks\" by Naman Agarwal, Pranjal Awasthi, and Satyen Kale may be related, where they studied the orthogonalization property of nonlinear activations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are discussed above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "smwXHQ-u1T", "reviewer": "Reviewer_rVTH", "summary": "The authors study the proposition that batch normalization (BN) orthogonalizes representations in deep networks. To this end, the consider a model with linear activations and random weights. The authors prove that the \"orthogonality gap\", i.e. how non-orthogonal the activations are, decreases with depth when BN is used. It is also shown how the activations contract to a gaussian isotropic distribution with depth. Experimental verification is provided, and the authors also propose an initialization scheme that can be used to substitute BN.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Comments\n---------------\n\nOverall the paper is relatively well written. BN improving the conditioning of neural networks is folklore at this point, and it is important to provide quantitative non-asymptotic proof of this. The model the authors consider -- linear neural networks with random weights and BN without mean subtraction -- is a reasonable starting point towards a more realistic analysis. The theoretical results are reasonable, verified experimentally and interesting. The main issue with the paper is Assumption 1, which seems very strong. Consider e.g. random Guassian input. The singular values would follow the semi-circle law for random matrices, and \\alpha should go to 0 as the dimensionality increases. Thus, it's not clear that it should hold for generic input. What are the reasons to think it would hold for intermediate layers? This assumption is very easy to test experimentally. Could the authors provide experimental verification that Assumption 1 holds in practice?\n\nThe contributions regarding proposing a new initialization scheme that works without BN is not very interesting in my mind. There is already plenty of work in training without normalization (see, e.g., High-Performance Large-Scale Image Recognition Without Normalization and references therein). The authors should compare against related work to show that their initialization is preferred.\n\n\nMinor Comments\n-----------\n\n\nThe authors state that \"The BN function in Eq. (1) differs slightly from the commonly used definition for BN as the mean correction is omitted.\" This is not a minor difference, but a rather significant one. This should be highlighted more.\n\nFig 2b should have error bars.\n\nIt might be interesting to discuss relationships with layernorm. BN is often used for CNNs and LN is often used for MLPs which the author study here.\n\nThe authors state that they focus on \"Multi-Layer Perceptron (MLP) with linear activation.\" This just means no activation function at all right? I find the term linear activation is confusing.\n\n\"Daneshmand et al. (2020) showes this difference does not change the network properties qualitatively.\" Does Daneshmand show this empirically? If so, it might only hold for the architectures they study and not in general.\n\n\n\nPost Rebuttal Comments\n-----------\n\nThe authors have clarified that the main assumption holds for the non-square case for random matrices, and also verified that it holds empirically. The assumption being strong was my main object, after the clarifications from the authors I now believe that it is reasonable. Hence, I will increase my score. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "HmkgRBW27I", "reviewer": "Reviewer_Agdx", "summary": "This papers focuses on the analysis of batch normalization which has been attracting a lot of attention recently.  The key contribution of the paper is in the finding that batch normalization (under some rather mild and sensible assumptions) makes training samples increasingly orthogonal.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is another contribution in the now rather hot area which attempts to understand better batch normalization, and NN normalization in general.  As such, it addresses a relevant and widely interesting topic.  The key findings are also rather interesting, novel, insightful, and practically important.  The bottom line demonstrated by the authors is rather simple to understand: predicated on a few mild assumptions, batch normalization makes training samples increasingly orthogonal.  This simple to understanding finding is rather insightful though and facilitates not only a better understanding of the normalization for its own sake, but also a series of practically important improvements, e.g. as the authors demonstrate, in the context of stochastic gradient descent application.  Overall, I consider the novelty and general technical quality of this paper to be well within the expected range for NIPS.  The quality of writing is also good - I have no objections to make.\n\nPost Rebuttal:\nHaving considered my colleagues' reviews and the authors' response to these, I find myself no less positive about this paper and am in fact even more confident of my original positive recommendation.  Hence I see no need to alter my review or the recommendation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All fine.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a setting and analysis where batch normalization increasingly orthogonalizes data as it is mapped through a network.  Reviewers are uniformly supportive and I recommend acceptance.  Even so, a few of the reviews (and responses) were quite detailed, and I request the authors adjust the paper to clarify these points in their revisions.
Decision: Accept (Spotlight)