Batch Normalization Explained
Authors: Randall Balestriero
Richard Baraniuk
Venue: Submitted to ICLR 2023 (2023)
Keywords: batch normalization
continuous piecewise linear networks
unsupervised learning

Abstract: A critically important, ubiquitous, and yet poorly understood ingredient in modern deep networks (DNs) is batch normalization (BN), which centers and normalizes the feature maps. To date, only limited progress has been made understanding why BN boosts DN learning and inference performance; work has focused exclusively on showing that BN smooths a DN's loss landscape. In this paper, we study BN theoretically from the perspective of function approximation; we exploit the fact that most of today's state-of-the-art DNs are continuous piecewise affine (CPA) splines that fit a predictor to the training data via affine mappings defined over a partition of the input space (the so-called ``linear regions''). We demonstrate that BN is an unsupervised learning technique that -- independent of the DN's weights or  gradient-based learning -- adapts the geometry of a DN's spline partition to match the data. BN provides a ``smart initialization'' that boosts the performance of DN learning, because it adapts even a DN initialized with random weights to align its spline partition with the data. We also show that the variation of BN statistics between mini-batches introduces a dropout-like random perturbation to the partition boundaries and hence the decision boundary for classification problems. This per mini-batch perturbation reduces overfitting and improves generalization by increasing the margin between the training samples and the decision boundary. 

Under review as a conference paper at ICLR 2023
BATCH NORMALIZATION EXPLAINED
Anonymous authors Paper under double-blind review
ABSTRACT
A critically important, ubiquitous, and yet poorly understood ingredient in modern deep networks (DNs) is batch normalization (BN), which centers and normalizes the feature maps. To date, only limited progress has been made understanding why BN boosts DN learning and inference performance; work has focused ex- clusively on showing that BN smooths DN's loss landscape. In this paper, we study BN theoretically from the perspective of function approximation; we exploit the fact that most of today's state-of-the of-the-art DNs are continuous piecewise affine (CPA) splines that fit a predictor to the training data via affine mappings defined over a partition of the input space (the so-called "linear regions"). We demon- strate that BN is unsupervised learning technique that independent of the DN's weights or grudient-bused learning - adapts the geometry of a DN's spline partition match the data. BN provides "smart initialization" that boosts the performance of DN learning, because it adapts even a DN initialized with random weights to align its spline partition with the data. We also show that the variation of BN statistics between mini-batcher introduces a dropout- random perturba- tion to the partition boundaries and hence the decision boundary for classification problems. This per mini-batch perturbation reduces overfitting and improves gen- eralization by increasing the margin between the training samples and the decision boundary.
INTRODUCTION
Deep learning has made major impacts in a wide range of applications. Mathematically, deep (neural) network (DN) maps an input vector æ to a sequence of L feature maps zz, = 1,... L by successively applying the simple nonlinear transformation (termed DN layer)
1 = a (i + ce), f= O..,.,,
(1)
with zu x. We the weight matrix, Cy the bias vector, and a an activation operator that applies scalar nonlinear activation function a to each element of its vector input. The structure of We, ca controls the type of layer (e.g., cireulant matrix for convolutional layer). For regression tasks, the DN prediction is simply 21,, while for classification tasks, 21, is often processed through softmax operator Goodfellow et al.) (2016). The DN parameters We, Cg are learned from a collection of training data samples X' x = 1, ,,...,,7) n. (augmented with the corresponding ground-truth labels y; in supervised settings) by optimizing an objective function (e.g.. squared error or cross- entropy). Learning is typically performed via some flavor of stochastic gradient descent (SGD) over randomized
of training data samples B C X' Goodfellow et al. (2016).
While host of different DN architectures have been developed over the past several years, mod- em, high-performing DNs nearly universally employ batch normalization (BN) loffe & Szegedy (015)) center and normalize the entries of the feature maps using four additional parameters μe, oe, Ge, Te. Define žt,k as tt entry of feature map 2; of length Dr, Wr,k as the bth row of
the weight matrix We, and μe,k, de,k, Be,k, TE, as the kth entries of the BN parameter vectors ue, oe, 3e, Ye, respectively. Then we can write the BN-equipped layer mapping extending (I)
as
11+14 = (WEE.: 2x) pea
7e.k + Be, = L..,,..,DD
(2)
Te,
The parameters μæ: o, are computed as the element- wise mean and standard deviation of W,z, for each mini-batch during training and for the entire training during testing. The parameters (3, 7e

---

Under review as a conference paper at ICLR 2023
layer 0
layer 1
layer 2
layer 3
without BN
with BN
Figure I: Visualization of the input-space spline partition ("lincar regions") of a four-layer DN with 2D input space, 6 units per layer, Beaky-ReLU activation function, and random weights We. The training data samples are denoted with black dots. In cach plot, blue lines correspond to folded hyperplanes introduced by the units of the corresponding layer, while gray lines correspond to (folded) hyperplanes introduced by previous layers. Top row: Without BN (i.e., using ( the folded hyperplanes are spread throughout the input space, resulting in a spline partition that is agnostic to the data, Bottom row: With BN (i.e., using ). the folded hyperplanes are
drawn towards the data, resulting in an adaptive spline partition that even with random weights minimizes the distance between the partition boundaries and the data and thus increases the density of partition regions around the data.
are Iearned along with We via SGD The empirical fact that BN significantly improves both training speed and generalization performance of a DN wide range of tasks has made ubiquitous, as evidenced by the 40,000 citations of the originating paper Toffe & Szegedy (2015).
Only limited progress has been made to date explaining BN, primarily in the context of optimiza- tion. By studying how backpropagation updates the layer weights. [LeCnn weigh et al. (1998) observed that unnormalized feature maps are constrained to live on a low-dimensional subspace that limits the ca- pacity of gradient- basen learning. By slightly altering the BN formula (2). [Salimans & Kingmal (2166) showed that renormalization via og smooths the optimization landscape and enables faster training. Similarly, Bjorek et al. (2018); SSnnturar et al. (2018): Kohler et al. (2019) confirmed BN's impact on the gradient distribution and optimization landscape through large-scale experiments. Us- ing mean field theory. Yang et al. (2019) characterized the gradient statisties of BN in fully connected feed-forwal networks with random weights to show that it regularizes the gradients and improves the optimization landscape conditioning.
One should not take away from the above analyses that BN's only effect is to smooth the optimization loss surface or stabilize gradients. If this were the case, then BN would be redundant in advanced architectures like residual Li et al. (2017) and mollifying networks Gulcehre et al. (2016) that have been proven to have improved optimization landscapes Li et al. (2018); Riedi et al. (2022) and have been coupled with advanced optimization techniques like Adam Kingma & Ba (2014). Quite to the contrary, BN significantly improves the performance of even these advanced networks and techniques.
In this paper, we study BN theoretically from a different perspective that provides new insights into how it boosts DN optimization and inference performance. Our perspective is function approxima- tion; we exploit the fact that most of today's 'ss sfate-of-the-art DNs are continuous piecewise affine (CPA) splines that fit a predictor to the training data via affine mappings defined over partition of
the input space (the so-called "linear regions"), "); see Balestriero & Baraniuk 2021 2018 Balestriero et al. (019)) and Appendix for more details.
Note that the DN bias e from has been subsumed into μe and Ba.
2

---

Under review as a conference paper at ICLR 2023
The key finding of our study is that BN is unsupervised learning technique that that-indee indeependent of the DN's weights or gradient-based learning - adapts the geometry of a DN's spline partition to match the data. Our three main theoretical contributions are follows:
BN adapts the layer/DN input space spline partition to minimize the total least squares (TLS) distance between the spline partition boundaries and the layer/DN inputs, thereby increasing the number of partition regions around the training data and enabling finer approximation (see Fig-
ure [r
The BN parameter μs translates the boundaries towards the data, while the parameter or folds the boundaries towards the data (see Sections and
BN's adaptation of the spline partition provides "smart initialization" that boosts the perfor- mance of DN learning, because it adapts even a DN initialized with random weights We to align the spline partition to the data (see Section
BN's statistics vary between mini-batches, which introduces dropout-like random jitter pertur- bation to the partition boundaries and hence the decision boundary for classífication problems. This jitter reduces overfitting and improves generalization by increasing the margin between the training samples and the decision boundary (see Section 5p.
The proof' for our results are provided in the Appendices; the codebase will be released upon com- pletion the review process to reproduce all the figures and results the study.
2 SINGLE-LAYER ANALYSIS OF BATCH NORMALIZATION
In this section, we investigate how BN impacts one individual DN layer. Our analysis leverages the identification that DN layers using continuous piecewise linear activation functions a in (I) and 2 are splines that partition their input space into convex polytopal regions. We show that the BN parameter lsp translates the regions such that they concentrate around the training data.
2.1
BATCH NORMALIZATION DETAILS
The BN parameters 3e, 7t. along with the DN weights Wi, are learned directly through the opti- mization of the DN's s objective function (e.g., squared error or cross-entropy) evaluated at the train- ing data samples X' {ar,2 i I,... ... n} and labels (if available). Current practice performs the optimization using some flavor of stochastic gradient descent (SGD) over randomized mini- miniebatches of training data samples C X. Our first new result that we can set 7r = 1 and Br = with no negligible impact DN performance for current architectures, training datasets, and tasks. First, we prove in Appendix thatt we can set Yt- = 1 both in theory and practice.
The BN parameter Yt f 0 does not impact the approximation expressivity of DN, because its value be absorbed into W W41,
Proposition 1.
Second, demonstrate numerically in Appendix D that setting 3r has negligible impact
DN performance. Henceforth, we will assume for our theoretical analysis that 7/- = 1, Br = 0 and will clarify for each experiment whether not enforce these constraints.
Let X₁ denote the collection of feature maps 2, at the input to layer e produced by all inputs æ in the entire training data set X, and similarly let B, denote the collection of feature maps zς at the input to layer l produced by all inputs x in the mini-batch B.
For each mini-batch B during training, the BN parameters μe, o are calculated directly as the mean and standard deviation of the current mini-batch feature maps B₁
Σ Z
Wize
(Wiz--Hi
(3)
B
B
Z1EB,
where the right-hand side square is taken element-wise. After SGD learning is complete, a final fixed "test time" mean Ti, and standard deviation of computed using the above formulae over all of the training data i.e., with B, = Xz. Note that no label information enters into the calculation of or
orr more commonly an exponential moving average of the training mini-batel values.
3

---

Under review as a conference paper at ICLR 2023
2.2 DEEP NETWORK SPLINE PARTITION (ONE LAYER)
We focus on the lionshare of modern DNs that employ continuous piecewise-lineat activation functions in ① and (2) To streamline our notation, but without loss of generality, sume that a consists of exactly two linear pieces that connect at the origin, such as the ubiqui- tous ReLU (a(u) max(0, u)). leaky-ReLU (a(u) max(a, u), a > 0), and absolute value (a(u) = max - ut). The extension to more general continuous piecewise-lincee
activation func- tions is straightforward [Balestriero & Baraniuk (0188 202T): moreover, the extension to an infinite class of smooth activation functions (including the sigmoid gated learning unit, among others) fol- lows from a simple probabilistic argument Balestriero & Baraniuk (2019). Inserting pooling opera- tors Goodfellow et al. (2016) between layers does not impact our results (see Appendix B).
DN layer é equipped with BN and employing such a piecewise-linens activation function a continuous piecewise-affine (CPA) spline operator defined by a partition f2 of the layer's 'ss input space R into a collection of convex polytopal regions and a corresponding collection of affine transformations (one for each region) mapping layer inputs 2 to layer outputs že+1- Here we explain how the partition regions in f2 are formed; then in Section 2.3 we begin our investigation of how these regions are transformed by BN.
Define the pre-activation of layer & by h2 such that the layer output 1 = a(h;); from 2 its Ath entry is given by
he. (We,k: Ze) Pe.
Ot.k
Note from that σge 0 as long as || > and as long as not all inputs are orthogonal to We,k- With typical CPA nonlinearities a, the 14th feature map output 2g-1,k a(he,,) linear in he, for all inputs with same sign. The separation between those two linear regimes is formed by the collection of layer inputs zạ that produce pre-activations with hy 0, hence lie on the D2 dimensional hyperplane
Hep { E R he
03 0} ( E (Wek, ze) p.,)
Note that He. is independent of the value of or.k- The boundary 0n, of the layer's input space parti- tion Q4 is obtained simply by combining all of the He, into the hyperplane arrangement Zaslavsky 1975)
an, = -p 1
For additional results on the DN spline partition, see Montufar et al. (2014): Raghu et al. (2017): Serra et al. (2018); Balestriero al. (2019); the only property of interest here is that, for all inputs lying in the same region w E Qe. the layer mapping is a simple affine transformation 21 Ewen( (A((w)=(11 + ( (=(-1w) (see Appendix B
2.3 BATCH NORMALIZATION PARAMETER l TRANSLATES THE SPLINE PARTITION BOUNDARIES TOWARDS THE TRAINING DATA (PART) 1)
With the above background in place, we now demonstrate that the BN parameter μ impacts the spline partition Qe of the input space of DN layer é by translating its boundaries 22, towards the current mini-batch training data X₁-
We begin with some definitions. The Euclidean distance from a point v in laver l's input space to the layer's s tth hyperplane He, is easily calculated as (e.g., Eq. in Amaldi & Coniglio 2013))
KWEK. v) Mex
d(v He,k)
|
as long as || > O. Then, the average squared distance between He, and a collection of points V in layer E's input space is given by
1 Σ d (v. He,r)
of,
Lu(MER, V)=
(8)
|V]
| 112
WEY
4

---

Under review as a conference paper at ICLR 2023
where we have made explicit the dependency of Cj M through He.g Going step further, the total least squares (TLS) distance [Samuelson 1942] Golub & Van Loan 1980 between a collection of points V in layer f's imput space and layer partition f2y is given by
Dr
C V)= Cu(MEM, V).
1=1
we prove that the BN parameter μ4 as computed in (3) is the unique solution of the strictly convex optimization problem of minimizing the average TLS distance (9) between the training data and layer f's hyperplanes He,k and hence spline partition region boundaries dbg.
Appendix E.
Theorem 1. Consider layer of DN as described in (2) and a mini-batch of layer inputs Be C Xz. Then μe in (3 is the unique minimizer of [(pe, Be), and py is the unique minimizer of C(ue, Xe).
In words, at each layer of DN, BN explicitly adapts the input-space partition S2g by using μe to translate its boundaries He.1, .e...... to minimize the TLS distance to the training data. Figure] [] demonstrates empirically in two dimensions how this translation focuses the layer's 's spline partition
on the data. Moreover, this translation takes very special form. We prove in Appendix E.2 that
BN transforms the spline partition boundary alg into a central hyperplane arrangement Stanley et al. (2004).
is worth noting that the above results do not involve any data label information, and so at least as far μ and concerned BN can be interpreted as unsupervised learning technique.
3 MULTIPLE LAYER ANALYSIS OF BATCH NORMALIZATION
We extend the single-layer analysis of the previous section to the composition of two more DN layers. We begin by showing that the layers' continue to translate the hyperplanes that comprise the spline partition boundary such that they concentrate around the training data. We then show that the layers' ag fold those same hyperplanes with the same goal.
3.1 DEEP NETWORK SPLINE PARTITION (MULTIPLE LAYERS)
Taking advantage of the fact that a composition of multiple CPA splines is itself a CPA spline, now extend the results from Section 2.2 regarding one DN layer to the composition of layers 1,...,f, f, l > that maps the DN input æ to the feature map ze+1 3 The denote partition of this mapping by See where introduce the shorthand
denote the mapping through layers 1,..., l.
Appendix B provides closed-form formulas for the per-region affine mappings.
we are primarily interested in the boundary ohg of the spline partition Qie- Recall that the boundary of the spline partition of a single layer was easily found in (4) as the rearrangement of the hyperplanes formed where the layer's pre-activation equals zero. With multiple layers, the situation is almost the same as in (5)
As in Section 2.2
Dj a2g UUL for (o LEERD
h5,e = 0):
(10)
==1==
further details are provided in Appendix B The salient result of interest to us is that 2hge is con- structed from the hyperplanes H,, back back through the preceding layer(s). This process folds those hyperplanes (toy depiction given in Figure g based the preceding layers' partitions such that the folded Hj.K consist of a collection of facets
Fikk= {x E w hijk
0} {x w (W,,,, M.jk), WEW S
(1I)
which simplifies 10 to 22 = - UP Fy where Fr. A Useq, Fanm-
Our analysis applies any composition of DN layers; focus the first layers only for concreteness.
5

---

Under review as a conference paper at ICLR 2023
N2. varying M2,k F2, varying Ma,k
F2, varying di
Figure 2: Translation and folding effected by the BN parameters pe, dg on a two-layer DN with units per layer, 2D input space, and random weights. (a) Varying 22,8 translates the layer 2. unit k hy- perplane H2, (recall (5)) viewed in 2D slice of own input space. (b) Translation
that same hyperplane but now viewed the DN input space, where it becomes the folded hyperplane F2.1 (recall TTP.. (c) Fixing M2,k and varying folds the next layer(s) hyperplanes, viewed in the DN input space.
(a)
(b)
3.2 BATCH NORMALIZATION PARAMETER μ TRANSLATES THE SPLINE PARTITION BOUNDARIES TOWARDS THE TRAINING DATA (PART 2)
In the one-layer case, we saw in Theorem that BN independently translates each DN layer's hy- perplanes Hi.k towards the training data minimize the TLS distance. In the multilayer case, as we have just seen, those hyperplanes He. become folded hyperplanes Ff, (recall (10).
We now demonstrate that the BN parameter μ4 translates the folded hyperplanes Fe, and thus adapts f2 towards the input-space training data X. To this end, denote the squared distance from a point æ in the DN input space to the folded hyperplane Fi,k by
d(x, Fe,)) = min | x' x'EFik
x' 112
(12)
Theorem 2. Consider layer of layer as described in (2) with fixed weight matrices and BN parameters from layers through { 8- and fixed weights Wt. Let ph and pl yield the hyper- planes He,k and Hig, and their corresponding folded hyperplanes Fe, and Fe, Then we have that
d(ze(x), He,x) < (x)) H), = (( <
In words, translating a hyperplane closer to ze in layer E's input space moves the corresponding folded hyperplane closer the DN input that produced zz, which is of particular interest for inputs æ; from the training data X. We also have the following corollary.
Corollary 1. Consider layer >> 1 of a trained BN-equipped DN as described in Theorem 2 Then =x(x) lies hyperplane He. for some k if and only x lies the corresponding folded hyperplane Fe, in the DN input space; that is, (()z= Hy,k) 0 + d(x,Fex) = O.
Figure (b) illustrates empirically how μ2 translates the folded hyperplanes F2. realized by the second layer of toy DN. The impact of the BN parameters a is also erucial albeit not as crucial as μ.. For completeness, we study how this parameter translates and folds adjacent facets composing the folded hyperplanes Fe, in Figure and Appendix C
3.3 DATCH-NORMALIZATION INCREASES THE DENSITY OF PARTITION REGIONS AROUND THE TRAINING DATA
We now extend the toy DN numerical experiments reported in Figures I to more realistic DN archi- tectures and higher-dimensional settings. We focus on three settings all involving random weights We: (i) zero bias C₂ in T. (ii) random bias C₁ in TT. and (iii) BN in (2)
2D Toy Dataset. We continue visualizing the effect of BN in a 2D input space by reproducing the experiment Figure [ but with more realistic DN with 11 layers of width 1024 and training data consisting of 50 samples from a star shape in 2D (see the leftmost plots in Figure 3). For each of the above three settings, Figure 3 visualizes the contribution to the partition boundary (recall (11) from three specific layers (j = 1, 11). The concentration at each point the input space corresponds to the number of folded hyperplane facets Fj.k. passing through e-ball centered on that point. This concentration calculation can be performed efficiently via the technique of Harman & Lacko 2010 that is analyzed in Voelker et al. (2017). Three conclusions are evident from the figure: (i) BN clearly focuses the spline partition the data; (ii) random bias creates a random partition that is diffused over the entire input space; (iii) zero bias creates (more less) central hyperplane arrangement that does not focus the data.
the 2D input space the concentration of

---

Under review as a conference paper at ICLR 2023
data
F3 Vk
Fr.m vk
F11,A vk
Figure 3: Visualization the 2D input space of the contribution 02; to the spline partition bound- ary Q from layers j 1, 7, 11 of an 1 11-layer DN of width 1024. The training data set X consists of 50 samples from a star-shaped distribution (left plots). We plot the concentration of the folded hy- perplane facets in an e-ball around each 2D input space point for the three initialization settings de- scrihed in the text: zero hias, random bias, and BN. Darker color indicates more partition boundaries crossing through that location. Each plot is normal- ized by the maximum concentration attained, the value of which noted.
BN
random
zero bias
around CIFARIO images
around random images
Figure 4: Average concentration of the spline par- tition boundary Q of Resnet9 in e-balls around CIFAR images and around random images. The vertical axis is the average number of folded hy- perplane facets in Q that pass through epsilon hall centered on each kind of image. As in Fig- ure we see
30000
random zaro
20000
10000
contrast to zero random bias initialization BN aligns the spline partition with the data.
0.005
0,010
0.015
0.020
0.005
0.010
0.015
0.020
radius (c)
radius (c)
CIFAR Images. We now consider high-dimensional input space (CIFAR images) with Resner9 architecture (random weights). Since we cannot visualize the spline partition in the 3072D input space, we present summary of the same concentration analysis carried out in Figure for the par- tition boundary by measuring the number folded hyperplanes passing through an e-ball centered around 100 randomly sampled training images (BN statistics are obtained from the full training set). We report these observation in Figure X (left) and again clearly see that in contrast to zero and random bias initialization BN focuses the spline partition to lie close to the data points. To quan- tify the concentration of regions away from the training data, we repeat the same measurement but for 100 iid random Gaussian images that are scaled to the same mean and variance as the CIFAR images. For this case, we observe in Figure (right) that BN only focused the partition around the CIFAR images and not around the random ones. This is concurrence with the low-dimensional experiment in Figure 3
4 BENEFIT ONE: BATCH NORMALIZATION IS A SMART INITIALIZATION
Up this point, our analysis has revealed that BN effects lask-independent, unsupervised learn- ing that aligns a DN's spline partition with the training data independent of any labels. We now demonstrate that BN's utility extends supervised learning tasks like regression and classification that feature both data and labels. In particular, BN can be viewed as providing a "smart initializa- tion" that expedites SGD-based supervised learning. Our results augment recent empirical work on the importance of initialization on DN performance from the perspective of slope variance con- straints Mishkin Matas (2015) Xie al. (2017), singular value constraints Jia al. (2017). and orthogonality constraints Saxe et al. (2013) Bansal et al. (2018).
First, BN translates and folds the hyperplanes and facets that construct the DN's spline partition Ω so that they are closer to the data. TTiis creates more granular partition regions around the training data, which enables better approximation of complicated regression functions [e Vore & Lorentz] (1993) and richer classification boundaries Balestriero et al. (2019); Chen et al. (2021). For binary classification, for example, this result follows from the fact that the decision boundary created by an L-layer DN is precisely the folded hyperplane FI,1 generated by the final layer. We also prove in Proposition 3 in Appendix E..4 that BN primes SGD learning so that the decision boundary passes through the data in every mini-batch.
We now empirically demonstrate the benefits of BN's smart initialization for learning by a com- paring the same three initialization strategies from Section 3.3 in classification problem using

---

Under review as a conference paper at ICLR 2023
Ir:
0.0001
Ir: 0.0005
Ir: 0.001
Ir: 0.002
Top-1 I test accuracy
50
40
30
diverged training with random Init.
ON intaritation
BK intaitatioon Random inttaittttion
SN Initll zadion Randam Inttallzatiion
BN intial barion Randem initaaiizaion
Rancom intraitattinn
60
40
20
20
learning epochs
Figure 5: Image classification using a Resnet50 on Imagenet. Standard data augmentation was used during SGD training but no BN employed. Instead, the DN was initialized with random weights (blue) with random weights and warmed-up BN statistics O across the entire training set (recall Figure] 3 and (3). Each column corresponds to different learning rate used by SGD and multiple runs are produced for the right cases. BN's "smart initialization" reaches a higher-performin partition that is already adapted the training dataset. This findings provides novel and complementary understanding of BN that was previously studied solely when continuously employed during training mean better condition DN's loss landscape. We also show in Figure 5nin the Appendix a similar observation on CIFAR100.
solution faster because it provides SGD with a spline
ResNet9 with random weights. In each experiment, the weights and biases are initialized accord- ing to one of the three strategies (zero bias, random bias, and BN over the entire training set), and then standard SGD is performed over mini-batches. Importantly, in the BN case, the BN parameters Me, Be computed for the initialization are frozen for all SGD iterations; this will showcase BN's role as a smart initialization. As we see from Figure 5 SGD with BN initialization converges faster and to an higher- perhhrrperformiggg classifier. Since BN is only used as an initializer here, the performance gain can be attributed to the better positioning of the ResNet's initial spline partition Ω
This result showcases the importance of DN initialization and how a good initialization alone plays a crucial role in performance, as has also been empirically studied with slope variance constraints Mishkin & Matas (2015): Xie et al. (2017), singular values constraints Jia al. (2017) or with orthogonal constraints Saxe et al. (2013): Bansal et al. (2018).
BENEFIT Two: BATCH NORMALIZATION INCREASES MARGINS
The BN parameters μ,z ae re-calculated for each mini-batch By and hence can be interpreted as stochastic estimators of the mean and standard deviation of the complete training data set Xx. A direct calculation Von Mises (2014) yields the following result.
Proposition 2. Consider layer l 1 of BN-equipped DN as described in (2) Assume that the layer input 2₁ follows an arbitrary iid distribution with zero mean and diagonal covariance matrix Hiag(mp) Then we have that
(w) mp)
| m|
ww)
p) () 3)
var(Me,
var
(13)
BBel
[Bl
[Bel
|B&l
with oi the fourth-order central moment of (We,k, z1) and w/, the coordinate-wise square.
Consequently, during learning, BN's centering and scaling introduces both additive and multiplica- tive noise to the feature maps whose variance increases as the mini-batch size [B&]] decreases. This noise becomes detrimental for small mini-batches, which has been empirically observed in Toffe (017). We illustrate this result Figure where we depict the DN decision boundary realizations from different mini-batched We also provide in Figure the empirical, analytical, and controlled parameter distributions of BN applied on a Gaussian input with varying mean and variance.
Figure 6 suggests the interpretation that BN noise induces "jitter" in the DN decision boundary. Small amounts of jitter can be beneficial, since forces the DN to learn a representation with an increased margin around the decision boundary. Large amounts of jitter can be detrimental, since the increased margin around the decision boundary might be too large for the classification task. This jitter noise is reminiscent of dropout and other techniques that artificially add noise in the DN
8

---

Under review as a conference paper at ICLR 2023
initialization |
learned |3/|=16
initialization [3g 2256
learned [8g| |256
Figure 6: Realizations of the classifica- tion decision boundary on toy 2D binary classification (red vs green) problem ob- tained solely by sampling different mini- batches and thus observing differrnt real- izations of μez, ar, (recall ( initializa- tion and after training. Each mini-bateh produces a different decision boundary de- pieted in blue. For two different mini-batch sizes [Be] 16, 256, we change the vari- ance as per Proposition 2 Larger batch sizes clearly produce smaller variability in the decision houndary both at initialization and after training; 14, distributions are provided Figure
W
W
distrib. of M
Figure 7: Distributions je, (top row) and de, (bot- tom row) for three units a DN layer with Wr M((1, 0, -1], (1, 0.1])) (E is not relevant in this context). The empirical distributions in black are ob- tained by repeatedly sampling mini-batches of size 64 from a training set of size 1000. The analytical distri- butions in green are from [13. The empirical distribu- tions in red are of the noise-controlled BN, where we increase the variances of the BN parameters match the variances that would result from virtual mini-batch size (32) that is smaller than the actual size (64), hence producing more regularizing jitter perturbations. The different realizations of μ4; and for each mini-batch affect the geometry of the DN partition and decision boundary (see Figure 2 current mini-batch.
0.462
3,030
1.502
1.322
1,000
1.005
distrib. of oz
2,000
0.006
5,10
k: = 2
k = 3
input and/or feature maps to improve generalization performance /Srivastava (003)): Pham et al. (2014); Molchanov al. (2017); Wang al. (2018). We focus on the effect of BN jitter DNs for classification problems here, but jitter will also improve DN performance on regression problems.
To further demonstrate that jitter noise increases the margin between the learned decision boundary and the training set (and hence improves generalization), we conducted an experiment where we fed additional Gaussian additive noise and Chi-square multiplicative noise to the layer pre-activations of a DN to increase the variances of μ4 and σ as desired (as in Figure 77. in addition to the BN- induced noise. For a Resnet9, we observed that increasing these variances about 15% increased the classification accuracies (averaged over 5 runs) from 93.34% to 93.68% (CIFAR10), from 72.22% to 72.74% (CIFAR 100) and from 96.16% to 96. 96.41% 4 (SVHN). Note that this performance boost comes in addition to that obtained from BN's smart initialization (recall Section
CONCLUSIONS
In this paper, our theoretical analysis of BN shed light and explained two novel crucial ways on how BN boosts DNs performances. First, BN provides a "smart initialization" that solve total least squares (TLS) optimization adapt the DN input space spline partition the data improve learn- ing and ultimately function approximation. Second, for classification applications, BN introduces a random jitter perturbation the DN decision boundary that forces the model to learn boundaries with increased margins to the closest training samples. From the results that we derived one can directly see how to further improve batch-normalizztion.. For example, by controlling the strength of the noise standard deviation of the batch-normalizatio parameters to further control the decision boundary margin, or by altering the optimization problem that patch-normalizztion minimizes to enforce specific adaptivity of the DN partition to the dataset hand. We hope that this work will motivate researchers to further extend BN into task-specific methods leveraging a priori knowledge of the properties one desires to enforce into their DN.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
The proofs and further derivations of the theoretical results provided throughout the main text are given in the Appendix. Additional explanatory figures are also provided. The codebase to repro- duce not only the quantitative experiments e.g. the performances without learnable Y and B3 or the performance of BN used as an initialization only, but also the qualitative ones will be released upon completion of the review process in the hope to mcrease the theoretical study around BN and to fur- ther improve this eponymous technique. We did not consider any sensitive dataset and focused on the standard computer vision ones which are CIFAR variants and Imagenet. For the architectures we also used the standard Resnet50 and Resnet9, additional details are also provided in the Appendix.
REFERENCES
Edoardo Amaldi and Stefano Coniglio. A basedd point-reassignment heuristic for the k-hyperplane clustering problem. European Journal of Operational Research, 227(():22-29, 2013. ISSN 0377-2217. doi: http:://b..rgplo...000.....22220
URL httpt: //www. sciencedirect http:///waaaaaaar...////////111/
R. Balestriero and R. G. Baraniuk. A spline theory of deep networks. Proc. Int. Conf. Mach. Learn., volume 80, pp. 374 384,333,,44 Jul. 2018.
Randall Balestriero and Richard Baraniuk. From hard to soft: Understanding deep network nonlin- earities via vector quantization and statistical inference. In International Conference on Learning Representations, 2019. URL https: :/ http///ppp/e....//////////
Randall Balestriero and Richard G. Baraniuk. Mad max: Affine spline insights into deep learning. Proceedings of the IEEE, 10965):704-77,, 2021. doi: 10. 10.1100//RRO..2200000000000
Randall Balestricro, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. arXiv preprint 1505:v:9900084444 2019.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular- izations in training deep networks? In Advances in Neural Information Processing Systems, pp. 4261 4271, 2018
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. CoRR, abs/2102. 2021. URL https:
/
11535
Ronald A DeVore and George G Lorentz. Constructive approximation, volume 303. Springer Science & Business Media, 1993.
Gene H Golub and Charles F Van Loan. An analysis of the total least squares problem. SIAM journal numerical analysis, 17(6):883-89) 1980.
I. Goodfellow, Bengio, and A. Courville, Deep Learning, volume 1. MIT Press, 2016. http:
7/www. www.deeeplearningooo.. org
Caglar Gulcehre, Marcin Moczulski, Francesco Visin, and Yoshua Bengio. Mollifying networks. arXiv preprint arXiv: 049800 2016.
Radoslav Harman and Vladimír Lacko. On decompositional algorithms for uniform sampling from n-spheres and balls.. Journal of Multivariate Analysis, 101(10):297-2304, 2010.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: I 1677 2015.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in normalizdd models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
10

---

Under review as a conference paper at ICLR 2023
Kui Jia, Dacheng Tao, Shenghua Gao, and Xiangmin Xu. Improving training of deep neural net- works via singular value bounding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4344 4342,4352.40 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization. In The 22nd International Conference on Artificial Intel- ligence and Statistics, pp. 806-88 (166815,22 2019.
Y LeCun, L. Bottou, G Orr, and K Muller. Efficient backprop, neural networks: Tricks of the trade. Lecture notes in computer sciences, 1524:5-50, 1998.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. arXiv preprint 171.v:17/. 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land-
scape of neural nets. Advances neural information processing systems, 31, 2018.
Dmytro Mishkin and Jiri Matas. All you need is good init. arXiv preprint arXiv:1511.0642 2015.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. arXiv preprint
2017.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pp- 2014.
Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves recurrett neural networks for handwriting recognition. In 2014 14th international conference frontiers in handwriting recognition, pp. 285-290. IEEE, 2014.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 2847-2854. JMLR. org, 2017.
Rudolf H Riedi, Randall Balestriero, and Richard G Baraniuk. Singular value perturbation and deep network optimization. arXiv preprint arXiv:2203.03000 2022.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp- 901-909, 2016.
Paul A Samuelson. A note alternative regressions. Econometrica: Journal of the Econometric Society, pp. 80-83, 1942.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions deep neural networks. In International Conference on Machine Learning, pp. 4558- 4566. PMLR, 2018.
Nitish Srivastava. Improving neural networks with dropout. University of Toronto, 182(566):7, 2013.
Richard P Stanley et al. An introduction to hyperplane arrangements. Geometric combinatorics, 13: 389-496, 2004.
11

---

Under review as a conference paper at ICLR 2023
Aaron R Voelker, Jan Gosmann, and Terrence C Stewart. Efficiently sampling vectors and coor- dinates from the n-sphere and n-ball. Technical report, Tech. Rep.) Waterloo, ON: Centre for Theoretical Neuroscience. doi: 10.13140 ..... 2017.
Richard Von Mises. Mathematical theory of probability and statistics. Academic Press, 2014.
Zichao Wang, Randall Balestriero, and Richard Baraniuk. A max-affine spline perspective of recur- rent neural networks. In International Conference on Learning Representations, 2018.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176- 6185, 2017.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.088. 2019.
Thomas Zaslavsky. Facing up to arrangements: Face-count formulas for partitions of space by hyperplanes: Face-count formulas for partitions of space by hyperplanes, volume 154. American Mathematical Soc., 1975.
A APPENDIX
We provide the following appendices details on the Max-Affine Spline formulation of DNs (Sec. B and the proofs of the various theoretical results from the main text (Sec.
B DETAILS ON CONTINUOUS PIECEWISE AFFINE DEEP NETWORKS
The goal of this section is to provide additional details into the forming of the per-region affine mappings of CPA DNs.
As mentioned in the main text, any DN that is formed from CPA nonlincarities can be expressed itself as CPA operator. The per-region affine mappings are thus entirely defined by the state of the DN nonlinearities. For an activation function such as ReLU, leaky-ReLU or absolute value, the nonlinearity state is completely determined by the sign of the activation input as determines which of the two linear mapping apply to produce its output. Let denote this code as (r(Zz-1) (ox, 1) Dr given by
W + byls S [1, [iz byls > 0
fa, aa
[ge (2)
(14)
where the pre-activation formula above can be replaced with the one from 2 if BN is employed. For max-pooling type of nonlinearity the state corresponds to the argmax obtained in each pooling regions, for details on how to generalize the below in that case we refer the reader to Balestriero & Baraniuk 2018). Based the above, the layer input-output mapping can be written
(W
(15)
where Qe produces a diagonal matrix from the vector ge, and has 0 for ReLU, for absolute value and a > () for MeekyyRReU;; see Balestriero & Baraniuk (018)) for additional details. The up-to-layer- mapping can thus be easily written as
Ze A ( + Bij((T)
(16)
with the following slope and bias parameters
(17)
B (xxx(=)
Σ (QW:QQ--W--- Q W by
(18)
where for clarity abbreviated 11(z+-1) as Qi.
12

---

Under review as a conference paper at ICLR 2023
From the above formulation, it is clear that whenever the codes qe stay the same for different inputs, the layer input-output mapping remains linear. This defines a region we Q2 from (6. the layer- input-space partition region, as
w/ (ze-1 E R qe(3e-1) q) a E 1).:
(19)
In similar way, the up-to-layer-e imput space partition region can be defined.
The multilayer region is defined as
N(EER" = 9g),0 qi E 1a.
(20)
ile
Definition (Layer/DN partition), The layer é (resp. -up-to-layer-E) input space partition given by
Q = (w/. q E {a, 11 10, Pije = (wipe 9.... qi E fa, 1) Vi) /0.
(21)
(22)
Note that SZI forms the entire DN input space partition. Both unit and layer input space partitioning can be rewritten as Power Diagrams, a generalization of Voronoi Diagrams Balestriero et al. (2019). Composing layers then simply refines successively the previously build input space partitioning via a subdivision process to obtain the -up to layer input space partitioning S2p.
C BATCH NORMALIZATION PARAMETER a FOLDS THE PARTITION REGIONS TOWARDS THE TRAINING DATA
We are now in a position to describe the effect of the BN parameter or, which has no effect on the spline partition of individual DN layer but comes into play for composition of more layers. In contrast to the hyperplane translation effected by μ4g, σr optimizes the dihedral angles between adjacent facets of the folded hyperplanes in subsequent layers order swing them closer to the training data.
Define Q? as the square diagonal matrix whose diagonal entry is determined by the sign of the pre-activation he,5, That is,
a/oes: 1/0 hg 2 0,
h < 0
(23)
with a = 0 for ReLU, a > 0 for leaky- and a- for absolute value (recall the definition of the activation function from Section 2.2] see Balestriero Baraniuk (2019) for additional non- linearities). Note that Q; is constant across each region w in the layer's spline partition Qt since, by definition, none the pre-activations he, change sign within region w. We will use Qr(w) to denote the dependency on the region w; to compute Qr(w) given input from w, computes the pre- pre-activations he and applies (23p.
merely samples a layer
In Appendix ?? prove that the BN parameter a adjusts the dihedral folding angles of adjacent facets of each folded hyperplane created by layer f + in order to align the facets with the training data. Figure 2 c) illustrates empirically how o folds the facets of F2, realized by the second layer of toy DN. Since the relevant expressions quickly (but predictably) grow in length with the number of layers, expose the salient points, but without loss of generality, on the composition of the first two DN layers (layers & = 1, 2). In this case, there are two geometric quantities of interest that combine to create the input space spline partition Ω Q2i layer I's hyperplanes H1,; and layer 2's folded hyperplanes Fz,k-
will focus the next theorem
Theorem 3. Given 2-layer (& 1, 2) BN-equipped DN employing leaky-ReLU or absolute value activation function, consider two adjacent regions w, w' from the spline partition Q whose bound- aries contain the facets F2,k,us F F created by folding across the boundaries' shared hyperplane
13

---

Under review as a conference paper at ICLR 2023
H,; 1,
F F2y
f kny
w
Figure 8: Sketch of the situation in Theorem for a two-dimensional input space.
H1 (see Figure 8y The dihedral angles between these three (partial) hyperplanes are given by:
2, H1,1) = arccos
(24)
|II. Qi(w)
||00,,1||
W.J,A Q2
( W WIw.
0 (F2,k H1,s) arccos
(25)
To Qi(w)) Wi
||01,
W. Qi(w))
(ww))aa)
θ (F2, F2.))w)) arccos
(26)
| (w)
/oo. Q (w") (w')
In 24 26). Qi(w) and Qı(w)) differ by only one diagonal entry at index (i. ii one ma-
trix takes the value Ji, and the other the value 01.2" as per Eq. 23 Since (8) implies that
oZ, & min C(pe,s, BB), we have the following two insights that state for f 1. (These results extend in a straightforward fashion for more than two layers as well as for more complicated piecewise linear activation functions.)
On the one hand, if Hi is well-aligned with the training data BJ, then the TLS error, and hence of, will be small. For the absolute value activation function, formulae 24p and 25 then tell us that both 0 (F2,k. H1,;) xe 0 and 0 (F2,k. H1,5) se O, meaning that F2 and 2 will be folded to closely align with Hi.s (and hence the data). The connection between o², and 24 25 lies in the entries of the Q matrix. Basically, the og are used in the denominator of Q and thus bend more or less the angles. For the ReLU/tcaky-Roa ReLU activation function, either F2,A, F22 will be folded to closely align with Ha,s: the other facet will be unchanged/mildlly folded.
On the other hand, if H1,; is not well-aligned with the training data B,, then the TLS error, and hence of. will be large. This will force Q (w) RV Q (w") and thus 0 (F2 F2,k.,)) ) RJ meaning that a poorly aligned layer-1 1 hyperplane H1,s will not appreciably fold intersecting layer-2 facets.
Figure illustrates empirically how the BN parameter σg, measures the quality of the fit of the (folded) hyperplanes to the data the TLS error sense for a toy DN.
In summary, and extrapolating to the general case, the effect of the BN parameter ar is to fold the layer-(e (f+ 1) hyperplanes (also the + 2 and subsequent hyperplanes) that contribute to the spline partition boundary Ω in order to align them with the layer-& hyperplanes that match the data well. Hence, not only μ4 but also σ, plays a crucial role aligning the DN spline partition with the training data (recall Figure
14

---

Under review as a conference paper at ICLR 2023
H1,k
F4,k
Figure 9: Layer-1 hyperplanes H1, (left) and layer-4 folded hyperplanes Fa,s (right) depicted the imput space of a toy 4-layer DN trained on the data points denoted with black dots. The (folded) hyperplanes are colored based on the correspond- ing value oi we,x Il? which is proportional to the total least squares (TLS) fitting error to the data (bluc: small error, close the data points; green: large error, far from the data points).
D ROLE OF THE BN PARAMETERS B, AND PROOF OF PROPOSITION
Without loss of generality, consider a simple two-layer DN to illustrate. Then, is clear that Y1 simply rescales the rows of W2 and the entries of [1
Σ -=1
[22r14 a (w HIL
Vi.k + 31,K3
12.r
22r =
22.r + 32.r
(27)
2,,
Σ k=l 71, 12 ll a (w0) k. - + BI
12
01,k
72.k
12,r B2,r
(28)
02.r
Σ Di k= (ung,r a (w), k.)) - + 11,,5)
122.
01,6
72.r + B2,r
(29)
222r
and so does not need to be optimized separately. Here, [w2,r]x denotes the k-th entry of the r-th row of W2. We obtain (28). because standard activation and pooling functions (e.g.. (leaky-)ReLU, absolute value, max- maxxpoolingg) are such that a(cu) = ca(u). This leaves Be as the only learnable parameter that needs to be considered.
Now, our BN theoretical analysis relies on setting Br O, which corresponds to the standard initialization of BN. In this setting, we saw that BN "fits" the partition boundaries the data samples exactly. Now, by observing the form of ② it is clear that learning 3r enables to "undo" this fitting needed better solve the task-at-hand. However, we have found that most practical scenarios, fixing B₂ O throughout training actually does not alter final performances. In fact, on Imagenet, the topl accuracy a Resnet18 goes from 67.60% to 65.93% when doing such a change, and on a Resnet34, from 68.7 77% to 67. 1... The drop seems to remain the same even more complicated architecture as for a Resnet50, the topl test accuracy goes down from 77.1 1% to 74.98%, where again we emphasize that the exact same drop could potentially be reduced by tuning the syper-parameters. Obviously those numbers might vary depending optimizers and other hyper-parameters, used here the standard ones for those architectures since our point is merely that all our theoretical results relying on Br = 0, Ye = 1 still applies high performing models.
employed for both situations i.e. the
In addition to the above Imagenet results, we provide in Table I the results for the classification accuracy on various dataset and with two architectures. This set of results simply demonstrates that the learnable parameters of BN (y, (3) have very little impact of performances.
Test accuracy of various models when employing (yes) or not (no) the BN learnable param- eters, as was demonstrated in the main paper, those parameters have very little impact on the final test accuracy (no data- aatta-aggmenttiinnn is used).
Table 1:
imagenette
ci
cifar100
svhn
No
Yes
No
Yes
No
Yes
No
Yes
RESNET
79.2
78.8
83.
86.2
50.
54.8
94.2
95.3
CNN
77.7
77.6
87.5
87.6
54.
55.2
96.
95,9
15

---

Under review as a conference paper at ICLR 2023
We provide below the deseriptions of the DN architectures. For the Residual Networks, the notation Resnet4-10 10 for example defines the width factor and depth of each block. We provide an example below for Resnet2-2.
Residual Network
Conv2D (layer[-1], 32, (5, 5), pad="SAME", b=None)) BatchNorm (1ayyrr(11
leaky_relu (1ayer(11)
MaxPoo12D Layer[--], (2, 2)) Dropout (layer[-1], 0.9)
for width in [64, 128, 256, 512]:
for repeat in range (2):
Conv2D (layer[-1,, width, (3, 3), b=None, pad= "AMB") (BacchN leaky [ecky_relu([ayer[11]]
Conv2D(Layerl-1,, width, (3, 3), b-None, pad= pad=")AME")
batchNorm(layer) if layer[-6).shaps (Layy_re + layer[-6]]
layer[-1] shape:
else:
Jeaky_relu(layer:/
Conv2D (layer [-6, width, (3, 3)
b-None, pad="SAMB:
AvgPool2D (layer[-1), (2, 2)
Conv2D (layer[-1,, 512, (1, 1), b-None)) BatchNorm (layer)
aaakyrreu [-1])
GlobalAvgPoo12D (1ay Dense (layer N_CLASSES)
[-1])
We now describe the CNN model that we employed. Notice that if the considered dataset is ima- genette or other (smaller spatial dimension) dataset there an additional first layer of convolution plus spatial pooling to reduce the spatial dimensions of the feature maps.
Convolutional Network
Conv2D (layer BatchNorm (layer)
32, (5, 5), pad="SAME", b=None)
MaxPoo12D (layer[-1), (2, 2))
if args.dar args.dataset = "imagenette":
Conv2D (layer[-1), 64, (5, 5), pad="SAME", b=None) BatchNorm (layer)
(eaky_rel MaxPco12D (1ayer (2, 2))
for k in range (3)
Conv2D (layer[-1), 96, (5, 5), b=None, pad=*SAME") (acchorm(llayer
(eaky_relu
(1nv 96, (1, 1), b=None) (atchNorm(layee)
(eaky_relb(layer1111
Conv2D (1ayeer-1,, 96, (1, 1), b=None)
BatchNorm (layer)
16

---

Under review as a conference paper at ICLR 2023
leaky_relu (layer (layer [-1])
Dropout [-1], 0.7)
MaxPool2D (1ayeeeaaaaarr [layer[-1], (2, 2))
for k in range (3)
(oayer 192, (5, 5), b-None, pad="SMME
(atchNorm((aaye))
Leaky_relu((ayer(1111]
(onv 192, (1, 1), b=None)
BatchNorm(layer)
leaky_relo=(ayer((I1]]
192, (1, 1), b=None)
(archNorm((ayyr) (eeky_rell
Dropout (layer[-1), 0.7)
MaxPool2D (layer[-1), (2, 2))
ConvD(1ayyrr1-1 192, (3, 3), b-None) BatchNorm(Layer)
leaky_reluu(layrr[-1]]
Conv2D 192, (1, 1), b=None))
BatchNorm((ayer) (eaky_relu
GlobalAvgPoo12D (layer [-1]] Dense (lay N_ C_LLASSES)
E PROOFS
We propose this section the various proofs supporting the diverse theoretical claims from the main part of the paper.
PROOF OF THEOREM
Proof. In order prove the theorem we will demonstrate below that the optimum of the total least square optimization problem is reached at the unique global optimum given by the average of the data, hence corresponding to the batch-normalizatiin mean parameter. Then we demonstrate that at this minimum, the value of the total least square loss is given by the variance parameter of batch- normalization.
The optimization problem is given by
D
D2
Kweek 2t
C(u; Z) = Σ Σ
(z,
He,x)
(30)
|| |wer
A= xEZ
k= 1 =EZ
it is clear that the optimization problem
min C(u, 2), PERD:
(31)
can be decomposed into multiple independent optimization problem for each dimension of the vector 14, since are working with an unconstrained optimization problem with separable sum. We thus focus on a single μtş for now. The optimization problem becomes
[wEe 2z Zz-1) ||wa
/4
min EIER
(32)
#EZ
17

---

Under review as a conference paper at ICLR 2023
taking the first derivative leads to
(WEF zz-
p4k
2 Σ
((WeR 2x-1) par
(33)
#EZ
#EZ
(wee Zt-1)
Σ
+ 2Card(Z)
(34)
|202,&
|
#EZ
(35)
the above first derivative of the total least square (quadratic) loss function is thus linear function of Pg being O the unique point given by
(Wee z1-1) 2Card(Z) 1
GwEZkWEK Card(Z)
(36)
#EZ
confirming that the average of the pre-activation feature maps (per- dire-mimmnsii)) is indeed the optimum of the optimization problem. One can verify easily that indeed minimum by taking the second derivative of the total least square which indeed positive and given by 2Card(2) Twe, The above can be
done for each dimension & in similar manner. Now, by inserting this optimal value back into the total least square loss, we obtain the desired result.
E.2 PROOF OF CENTRAL HYPERPLANE ARRANGEMENT
Corollary 2. BN constrains the imput space partition boundaries 024 of each layer i of a BN- equipped DN to be central hyperplane arrangement; indeed, the average of the layer's training data inputs
Hagk
(37)
as long as || O.
Proof. In order to prove the desired result i.e. that there exists nonempty intersection between all the hyperplanes, first demonstrate that the layer input centroid žę-1 indeed one hyperplane, say k. Then it will be direct to see that this holds regardless of and thus the intersection of all hyperplanes contains least Æt-1 which is enough prove the statement.
For data point (in our case 1)- to belong to the tth (unit) hyperplane He of layer f, we must ensure that this point belong to the of the hyperplane defined as (recall (5)
H { E D (WE,. =1-1) [
(38)
in our case we can simply use the data centroid and ensure that it fulfils the hyperplane equality
(we,,, Ze-1) = - WEEk
ExEZ z Card(Z)
(weL, 2)
Σ zez
Card(Z) [M0]
(39)
where the last equation gives in fact the batch-normalization mean parameter. So now, recalling the equation of He, we see that the point žę-1 makes plane projection [uğ]e which equals the bias of the hyperplane effectively making ž,- part of the Saatch-normalieed hyperplane He Doing the above for each Dr we see that the layer input centroid belongs to all the unit hyperplane that are shifted by the correct batch-normalization parameter, hence we directly obtain the desired result
Zt-1
N Hekk kEDz
(40)
concluding the proof.
18

---

Under review as a conference paper at ICLR 2023
E.3 PROOF OF THEOREM 2
Proof. Define by x" the shortest point in Pt,s from æ defined by
x arg min I| u//2. PEPe,
The path from æ to xx a straight line in the input space which we define by
(()) x0 (1 0) xe, 0 e [0,
(41)
s.t. I(0) æ, our original point, and ((1) is the shortest point on the kinked hyperplane. Now, in the input space of layer this parametric line becomes a continuous piecewise affine parametric line defined as
(0) (fé-1 0...00 (z)
(42)
By definition, if Pe, is brought closer æ, it means that 30 < 1 s.t. l(0) Pe,k Similarly this can be defined in the layer input space as follows.
30 < s.t. z0- E He. = 30 < s.t. 1(0) E Pe.k
this demonstrates that when moving the layer hyperplane s.t. it intersects the kinked path z;-1 at a point =6-1(0") with 8' < then the distance the input space is also reduced. Now, the BN fitting is greedy and tried to minimize the length of the straight line between =0- a.k.a (x) and the hyperplane Hy However, notice that if the length this straight line decreases by brining the hyperplane closer to ( then this also decreases the 0' s.t. 20-1(0') E H in turn reducing the distance between æ and Pt.k in the DN input space, giving the desired (second) result. Conversely, if (0) He, then the point lies in the zero-set of the unit, in turn making belong to the kinked hyperplane Pe,k which corresponds to this exact set.
E.4 PROOF OF PROPOSITION
3
Proposition 3. Consider L-layer DN configured learn binary classifier from the labeled training data X using a teaky-ReLU activation function, arbitrary weights We at all layers, BN at layers? 1,..., L and layer L configured as in ( with et. O. Then, for any training mini-batch from X, there will be at least one data point on either side of the decision boundary.
Proof. When using leaky-ReLU the input to the last layer will have positive and negative values in each dimension for at least in the current minibatch. That means that each dimension will have at least negative value and all the other positive or vice-versa. As the last layer is initialized with zero bias, the decision boundary is defined in the last layer input space as the hyperplanes (or zero-set) of each output unit. Also, being on one side or the other of the decision boundary in the DN input space is equivalent to being one side the other of the linear decision boundary in the last layer input space. Combining those two results we obtain that at initialization, there has to be at least sample one side of the decision boundary and the others on the other side.
19

---

Under review as a conference paper at ICLR 2023
test accuracy
Figure 10: Image classification using Resne19 on CIFAR100. No BN or data augmentation was used during SGD training. Instead, the DN initialized with random weights and zero bias (blue), random bias (orange), or via across the entire data set as in Figure [ (green). Each training was repeated 10 times with learning rate cross-validation, and we plot the average test accuracy (of the best valid set learning rate) vs. learning epoch. BN's "smart initialization" reaches a higher- 90 performing solution faster because it provides SGD with spline partition that is alrcady adapted to the training dataset.
30
20
50
learning epochs
E.5
PROOF OF BN STATISTICS VARIANCE
Proof. Let's consider random variable with E(z) = m and Cov(z) - diag(e"). Then we directly have that
E(ww, z))
(w, m)
V Var( ((w, 2)) EE((w, 2) -E((w, 2)))
= E(((w, u, z) (w0 m))
E((w, 2 -m)))
E
Σ uzz(za md)²
Σ
wallz a ma)ww (ze - m.))
E
Σ wp] (zd mm)² Σ
walza my)was (Z8 mar)
d'pid
waPa
(w". (P)
now given the known variance and mean of the (w, z) random variable, the desired result obtained by using the standard empirical variance estimator.
F DATASET DESCRIPTIONS
MNISTThe MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. It was created by "re-mixing" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments. Furthermore, the black and white images from NIST were normalized to fit into 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels.
The MNIST database contains 60, 000 training images and 10, 000 testing images. Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset. The original creators of the database keep list of some of the methods tested on their original paper, they use support- machine to get error rate of 0.8%.
SVHNSVHN is a real-world image dataset for developing machine learning and object recognition algorithms with minimal requirement data preprocessing and formatting It can be seen similar in flavor to MNIST (e.g.. the images are of small cropped digits), but incorporates an order of magnitude more labeled data (over 600, 000 digit images) and comes from significantly harder, unsolved, real world problem (recognizing digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images.
20

---

Under review as a conference paper at ICLR 2023
CIFARIOThe CIFAR-10 10 dataset (Canadian Institute For Advanced Research) is collection of im- ages that are commonly used to train machine learning and computer vision algorithms. one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 000 32x32 color images 10 different classes. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class.
Computer algorithms for recognizing objects in photos often learn by example. CIFAR-10 a set of images that can be used to teach computer how recognize objects. Since the images in CIFAR- 10 are low-resolution (32x32), this dataset can allow researchers to quickly try different algorithms see what works. Various kinds of convolutional neural networks tend to be the best at recognizing the images in CIFAR-10.
CIFAR100This dataset is just like the CIFAR-10, except has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. The 100 classes in the CIFAR- 100 are grouped into 20 superclasses. Each image comes with "fine" label (the class which belongs) and "coarse" label (the superclass to which it belongs).
21


Reviews:
[{"review_id": "zlYxMlW7Ljq", "reviewer": "Reviewer_991a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the inner working of batch normalization. The paper presents two different claims for enhanced learning with batch normalization: \n- *unsupervised*  Blindness to the labels batch normalizes adapts the \"adapts the geometry of a DN’s spline partition\nto match the data\" Apparently, this claim relates to theoretical results which justify the mean-deduction in batch normalization is the minimizer of a function. \n- *supervised* BN can be viewed as a smart initialization which facilitates classification", "summary_of_the_review": "I recommend authors to develop  *smart initialization* results and make it either theoretical or experimentally more sound and convincing. ", "strength_and_weaknesses": "**Strength** \n\nI think that the strongest message of this paper relates to smart initialization. I wished the authors develop an argument around this result. \n\n**Weakness** \n- *claims* The paper is full of vague claims that are not supported by theoretical or at least experimental pieces of evidence. \n  - It is claimed that BN adapts the geometry of a DN's spline partition. Which theorem or proposition does imply an adaption of spline? I believe that the result of Theorem 1 is a simple fact that the average of points has the minimum square distance to a set of points, which does not relate to adapting partition \n  - It is claimed that adapts the geometry partition to **match the data**. Consider a network with random weights. In each layer, partitions are highly random in each layer. How even it is possible to achieve matched data partitioning? What is exactly the meaning of matching, geometry, and adapting here? \n\n - *soundness* Proposition 1 states that BN parameters can be frozen without losing expressive power. The proof neglects that the variance changes with scaling the weight matrix. Hence, the statement and the proof are not correct. \n- *novelty* Thm 1 is only proving the average achieves the minimum squared distance from a set of points. On the other hand, Proposition 2 does not imply an increase in the margin. There is a large gap between what is claimed and what is actually proven. \n- *notations* In equation 5, is $z_\\ell$ the same as those in equation 3? I believe that equation 5 requires a different notation. \n- *literature review* There is recent research on the role of batch normalization for randomly initialized neural nets. The notion of spline adaption may be connected to one of the well-defined properties studied in this literature.  \n- *experiments* To show BN is equivalent to a smart initialization technique, we need to include BN in Fig 5. Does author use the same batchsize used in BN (often of size 32) to implement the smart initialization? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "xW7Dip2fKV", "reviewer": "Reviewer_maYa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper theoretically studies BN from the perspective of function approximation. The authors view the output function of a neural net as continuous piecewise affine splines. An important simplification in this paper is that it fixes the gamma and beta parameters in BN to 1 and 0 respectively. The key finding is that \"BN is an unsupervised learning technique that adapts the geometry of a DN's spline partition to match the data\". Then this paper points out two benefits of BN: (1) providing a \"smart initialization\" that \"adapts weights to align the spline partition to the data\"; (2) introducing a random jitter perturbation to the partition boundaries by varying BN statistics between mini-batches.", "summary_of_the_review": "It is understandable that the current paper only makes intuitive claims because neural net with BN is a very complex object to study. However, I believe it is necessary to make the claims more formal so that the whole field can make progress. I feel that the current paper needs a major revision before getting accepted.", "strength_and_weaknesses": "#### Strength:\n1. It is a good angle to understand BN from the perspective of function approximation.\n\n#### Weakness:\n1. The biggest weakness of this paper is that it first fixes gamma and beta parameters in BN to 1 and 0, then claims that the benefits of BN come from the fact that the output of each BN is normalized to mean 0 and std 1, hence concluding that BN makes the weights minimize a TLS distance in some sense. This logic does not make sense because this minimization does not occur when gamma and beta parameters are not fixed. I agree with the authors that gamma and beta can be fixed without hurting performance, but then the situation looks like this: with gamma and beta unfixed, performance is good, and the minimization does not occur; with gamma and beta fixed, performance is nearly the same, and the minimization occurs. Comparing these two cases, a reasonable conclusion is that it is fixing gamma and beta that makes the minimization happens, not BN; and this minimization does not have much effect on performance.\n2. Benefit 1 is very ambiguous. In my understanding, the claim that BN is a \"smart\" initialization means that the initialization of a neural net with BN makes the training faster than a neural net without BN. However, adding BN not only changes the initialization, but also changes the whole training dynamics. I don't see how this paper separates the change of dynamics from initialization, and thus it is hard to interpret the meaning of \"BN is a smart initialization\".\n3. I believe Benefit 2 is true in some sense, but the statement of Benefit 2 is not very clear. For providing a good theoretical explanation, the authors should at least provide the precise mathematical definitions of \"margin\" and \"jitter\", instead of just putting Figure 6 there. After that, the authors should either (1) use theoretical tools to bridge from jitter to large margin, and from large margin to generalization; or (2) measure margin, jitter, and generalization in experiments to justify their claim empirically.\n4. The point that the stochasticity in batch statistics improves generalization is not new. See, e.g., Section 9.3 of [a recent survey by Huang](https://link.springer.com/content/pdf/10.1007/978-3-031-14595-7.pdf). I also recommend the authors go over all the related papers to better contextualize the current paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mOFMNsWl_C4", "reviewer": "Reviewer_3myR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to provide insights into Batch Normalisation (BN).\nThis is done by formulating Neural Networks as piece-wise affine splines.\nIt is shown that the mean shift in BN moves the spline boundaries closer to the training data.\nFurthermore, two benefits of BN are highlighted.\nThe first benefit is that the mean shift provides a better initialisation for the bias, compared to standard approaches.\nThe second benefit is the stochasticity due to the use of mini-batch statistics, which leads to larger margins.", "summary_of_the_review": "Although this work presents a few interesting ideas, I do not think that this paper provides enough *new* insights to be valuable for the community.\nThe main takeaway for me would have been that neural networks with LeakyRelus can be interpreted as splines, but this has been established in prior work.\nThe fact that sections&nbsp;4 and&nbsp;5 appear to be separated from the actual analysis in sections&nbsp;2 and&nbsp;3 seems to be a strong indication that the applicability of the presented analysis is limited.", "strength_and_weaknesses": "- [+] The paper is generally well-written and easy to follow.\n- [+] The spline formulation to study BN is a refreshing idea.\n- [+] Most of the theoretical derivations seem to be correct.\n- [+] The experiment in section&nbsp;5 nicely illustrates how batch noise maximises the margin around the decision boundary.\n- [-] I do not see the connection between the spline interpretation and the benefit sections (section&nbsp;4 and&nbsp;5).\n- [-] There seem to be no real *new* insights from the spline interpretation.\n- [-] This paper seems to be more about bias parameters than about BN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Y6LZH9TsOin", "reviewer": "Reviewer_jK7q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the effect of batch normalization on the properties of the associated function. In particular, they consider piecewise affine deep neural networks and show that applying BN brings the induced convex polytopes in the input space closer to the training data, which enhances faster training. Consequently, BN can be seen as a practical way to initialize the model. Also, a second observation is that the batches induce some kind of stochasticity in the model, since the resulting function is different for each batch. This potentially helps the model to generalize better. The arguments are supported by theoretical argumentation and empirical demonstrations.", "summary_of_the_review": "Questions:\n\n1. In Sec 2.3 it is shown that in a feature space (the output of a hidden layer) each row of the next weight matrix introduces a hyperplane. The argument is that BN brings this hyperplane closer to the points in this feature space. I guess the implication is that the associated folded hyperplane in the input space moves closer to the data as well. However, I think that this implication is not explained clearly, since it is only shown that the hyperplane in the feature space moves closer to the points therein, and it is not discussed the relationship to the input space.\n\n2. I am not sure why Theorem 2 is necessary, since the behavior of the folded hyperplane moving closer to the training data is implied by the fact that the hyperplane in the feature space moves closer to the points therein. Is there some detail that I am missing?\n\n3. I think that in Sec 3.1 the explanation of the folded hyperplanes is a bit hard to access. I believe that some figures to illustrate the information and the pipeline could help. I acknowledge that the intuition is clear i.e. a hyperplane in a feature space corresponds to a folded hyperplane in the input space and the collection them for all the layes partitions the input space. Also, there might be some  problems with notation e.g. in Eq. 10 the $D_{j}$ seems weird since for $j=1$ it implies that the weight matrix multiplied with the inputs has only $D_j$ rows. Similarly at the end of this page the $\\cup_{\\omega\\in|\\Omega_j}$ perhaps should be $|\\Omega_\\ell$?\n\n4. As regards Sec 4, if I understood correctly you just make one forward pass (using a batch or all data) for the randomly initialized network, you compute the BN parameters which are kept fixed for the rest of the training?\n\n5. In Sec 5, the layer input $z_\\ell$ in the theorem are the features of a single batch or the collection of many different batches?  Also, the results in Fig. 6 imply that a model is trained with BN, and after that a forward pass for every batch gives a different decision boundary?\n\nIn general, the style of the paper as it describes in an accessible way some interesting properties  of a technique that is widely used. The analysis is quite easy to follow and the arguments are supported both by theoretical results and empirical demonstrations. Perhaps some minor changes can be mode to improve even further the accessibility of the manuscript.\n", "strength_and_weaknesses": "I like the paper because it places a simple question and describes the answer in a rather accessible way. I think that the motivation is clear and the idea easy to grasp. The claims are supported by theoretical results, which I have not checked in details, and empirical demonstrations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject