Year: 2018

Abstract:
Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks. Discussions of why this normalization works so well remain unsettled. We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN. We recast the backpropagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations. This view, which we term gradientleast-squares, is an extensible and arithmetically accurate description of BN. To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.

text: Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks. Discussions of why this normalization works so well remain unsettled. We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN. We recast the backpropagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations. This view, which we term gradientleast-squares, is an extensible and arithmetically accurate description of BN. To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.

heading: 1 INTRODUCTION
text: Training deep neural networks has become central to many machine learning tasks in computer vision, speech, and many other application areas. Ioffe & Szegedy (2015) showed empirically that Batch Normalization (BN) enables deep networks to attain faster convergence and lower loss. Reasons for the effectiveness of BN remain an open question (Lipton & Steinhardt, 2018). Existing work towards explaining this have focused on covariate shift; Santurkar et al. (2018) described how BN makes the loss function smoother. This work examines the details of the back-propagation of BN, and recasts it as a least squares fit. This gradient regression zero-centers and decorrelates partial derivatives from the normalized activations; it passes on a scaled residual during back-propagation. Our view provides novel insight into the effectiveness of BN and several existing alternative normalization approaches in the literature.

heading: 1.1 CONTRIBUTIONS
text: Foremost, we draw an unexpected connection between least squares and the gradient computation of BN. This motivates a novel view that complements earlier investigations into why BN is so effective. Our view is consistent with recent empirical surprises regarding ordering of layers within ResNet residual maps (He et al., 2016b) and within shake-shake regularization branches (Huang & Narayanan, 2018). Finally, to demonstrate the extensibility of our view, we motivate and evaluate two variants of BN from the perspective of gradient-least-squares. In the first variant, a least squares explanation motivates the serial chaining of BN and Layer Normalization (LN) (Ba et al., 2016). In the second variant, regularization of the least-squares leads to a version of BN that performs better on batch size two. In both variants, we provide empirical support on CIFAR-10.
In summary, our work presents a view, which we term gradient-least-squares, through which the back-propagation of BN and related work in a neural network can be recast as least squares regression. This regression decomposes gradients into an explained portion and a residual portion; BN back-propagation will be shown to remove the explained portion. Hopefully, gradient-least-squares will be broadly useful in the future design and understanding of neural network components. Figure 1 reviews normalization with batch statistics, and illustrates our main theorem.
{xi}i=1...N be activations. Let µ =
N∑
i=1 xi N and σ2 =
N∑
i=1 (xi − µ)2 N . Let L be a function dependent on the normalized activations zi defined for each j by zj = (xj − µ)
σ This, along with
partial derivatives, are shown in the left figure. Our work establishes a novel identity on the quantities shown in hexagons. The right figure illustrates our main result in a scatter plot, in which each pair ( zi, ∂L
∂zi
) is shown as a data point in the regression.

heading: 2 NORMALIZATION GRADIENTS ARE LEAST-SQUARES RESIDUALS
text: Consider any particular channel within which {xi} are activations to be normalized in BN moment calculations. Ioffe & Szegedy (2015) defined BN as
BN(x) = (x− µ)
σ · c+ b (1)
where σ, µ are batch moments, but b and c are learned per-channel parameters persistent across batches. In BN, the batch dimension and spatial dimensions are marginalized out in the computation of batch moments. For clarity, we consider a simplified version of BN. We ignore the variables b and c in equation 1 responsible for a downstream channel-wise affine transformation. Ignoring b and c is done without loss of generality, since the main observation in this work will focus on the Gaussian normalization and remains agnostic to downstream computations. We also ignore a numerical stability hyperparameter .
We examine back-propagation of partial derivatives through this normalization, where µ and σ are viewed as functions of x. Notably, µ and σ are functions of each xi, and thus the division by σ is not affine. We write the normalized output as
z = (x− µ)
σ (2)
We review ordinary least squares of a single variable with intercept (Friedman et al., 2001).
Let gj = α+βzj + j where α and β are parameters, z and g are observations. zj and gj are entries in z and g respectively. j are i.i.d. Gaussian residuals. We wish to fit α and β
α̂, β̂ = arg min α,β
Ej(‖g − α− βz‖2) (3)
The least-squares problem in equation 3 is satisfied by β̂ = Cov(z, g)
Var(z) and α̂ = E(g)− β̂E(z)
When z are normalized activations and g are partial derivatives, then Ez = 0 and Var(z) = 1. In this special case, the solution simplifies into
β̂ = Cov(z, g) (4) α̂ = E(g) (5)
Theorem 1 (Normalization gradients are least-squares residuals). Let i ∈ {1 . . . N} be indices over some set of activations {xi}. Then the moment statistics are defined by µ = N∑ i=1 xi N and
σ2 = N∑ i=1 (xi − µ)2 N . Let L be a function dependent on the normalized activations zi defined for each j by zj = (xj − µ)
σ . Then, the gradients of L satisfy, for all j ∈ {1, . . . , N}, the following:
σ ∂L ∂xj = ∂L ∂zj − ∂̂L ∂zj
(6)
where
∂̂L ∂zj = α̂+ β̂zj (7)
α̂, β̂ = arg min α,β N∑ i=1 ( ∂L ∂zi − α− βzi )2 (8)
Proof: Normalization gradients are least-squares residuals. The proof involves a derivation of partial derivatives by repeated applications of the chain rule and rules of total derivative. Because {zi} normalized over i has mean 0 and variance 1, the partial derivatives can be rearranged to satisfy the single variable ordinary least squares framework.
Fix j. We expand ∂L
∂xj as a linear combination of
{ ∂L
∂zi } i=1...N
∂L ∂xj = ∂L ∂zj ∂zj ∂xj + N∑ i6=j ∂L ∂zi ∂zi ∂xj
(9)
We state ∂zi ∂xj directly. Steps are in Appendix A under Lemma 1.
∂zi ∂xj =  −1− zjzi σN if i 6= j
N − 1− z2j σN if i = j
(10)
Through substitution of equations 10 into 9, we get
∂L ∂xj = ∂L ∂zj N − 1− z2j σN + N∑ i 6=j [ ∂L ∂zi · −1− zjzi σN ] (11)
σ ∂L ∂xj = ∂L ∂zj + 1 N N∑ i=1 [ (−1− zizj) ∂L ∂zi ] (12)
σ ∂L ∂xj = ∂L ∂zj −
( 1
N N∑ i=1 ∂L ∂zi ) − zj N N∑ i=1 zi ∂L ∂zi (13)
Noting that {zi} normalized over i has mean 0 and variance 1, we recover β̂ and α̂, in the sense of equations 4 and 5, from equation 13.
1
N N∑ i=1 zi ∂L ∂zi = Covi ( zi, ∂L ∂zi ) = β̂ (14)
1
N N∑ i=1 ∂L ∂zi = Ei [ ∂L ∂zi ] − β̂ · 0 = α̂ (15)
Finally, we rearrange equations 15 and 14 into 13 to conclude, as desired,
σ ∂L ∂xj = ∂L ∂zj − α̂− β̂zj = ∂L ∂zj − ∂̂L ∂zj
(16)
During back-propagation of a single batch, the normalization function takes in partial derivatives ∂L ∂z(·) , and removes that which can be explained by least squares of ∂L ∂z(·) against z(·). As illustrated in Figure 1, during back-propagation, the residual then divides away σ to become ∂L
∂x(·) , the gradient
for the unnormalized activations.

heading: 3 RELATED DEEP LEARNING COMPONENTS VIEWED AS GRADIENT CALCULATIONS
text: BN aims to control its output to have mean near 0 and variance near 1, normalized over the dataset; this is related to the original explanation termed internal covariate shift (Ioffe & Szegedy, 2015). Most existing work that improve or re-purpose BN have focused on describing the distribution of activations.
Definition 1. In the context of normalization layers inside a neural network, activations are split into partitions, within which means and variances are computed. We refer to these partitions as normalization partitions. Definition 2. Within the context of a normalization partition, we refer to the moments calculated on the partitions as partition statistics.
Theorem 1 shows that BN has least squares fitting built into the gradient computation. Gradients of the activations being normalized in each batch moment calculation are fit with a single-variable with-intercept least squares model, and only a rescaled residual is kept during back-propagation. We emphasize that the data on which the regression is trained and applied is a subset of empirical activations within a batch, corresponding to the normalization partitions of BN.
To show extensibility, we recast several popular normalization techniques into the gradient-leastsquares view. We refer to activations arising from a single member of a particular batch as an item. BHWC refers to dimensions corresponding to items, height, width, and channels respectively. In non-image applications or fully connected layers, H and W are 1. BN marginalizes out the items and spatial dimensions, but statistics for each channel are kept separate.
In the subsequent sections, we revisit several normalization methods from the perspective of the gradient. Figure 2 reviews the normalization partitions of these methods, and places our main theorem about gradient-least-squares into context.

heading: 3.1 LAYER NORMALIZATION, INSTANCE NORMALIZATION, GROUP NORMALIZATION
text: Ba et al. (2016) introduced Layer Normalization (LN) in the context of large LSTM models and recurrent networks. Only the (H,W,C) dimensions are marginalized in LN, whereas BN marginalizes out the (B,H,W ) dimensions. In our regression framework, the distinction can be understood
as changing the data point partitions in which least squares are fit during back-propagation. LN marginalizes out the channels, but computes separate statistics for each batch item. To summarize, the regression setup in the back-propagation of LN is performed against other channels, rather than against other batch items.
Huang & Belongie (2017) introduced Instance Normalization (IN) in the context of transferring styles across images. IN is is closely related to contrast normalization, an older technique used in image processing. IN emphasizes end-to-end training with derivatives passing through the moments. Only the (H,W ) dimensions are marginalized in IN, whereas BN marginalizes (B,H,W ) dimensions. In our framework, this can be understood as using fewer data points and a finer binning to fit the least squares during back-propagation, as each batch item now falls into its own normalization partition.
Wu & He (2018) introduced Group Normalization (GN) to improve performance on image-related tasks when memory constrains the batch size. Similar to LN, GN also marginalizes out the (H,W,C) dimensions in the moment computations. The partitions of GN are finer: the channels are grouped into disjoint sub-partitions, and the moments are computed for each sub-partition. When the number of groups is one, GN reduces to LN.
In future normalization methods that involve normalizing with respect to different normalization partitions; such methods can pattern match with BN, LN, IN, or GN; the back-propagation can be formulated as a least-squares fit, in which the partial derivatives at normalized activations ∂L
∂z(·)
are fitted against the normalized z(·), and then the residual of the fit is rescaled to become ∂L
∂x(·) .
Figure 2 summarize the normalization partitions for BN, LN, IN, and GN; the figure visualizes, as an example, a one-to-one correspondence between an activation in BN, and a data point in the gradient regression.
Theorem 1 is agnostic to the precise nature of how activations are partitioned before being normalized; thus, equation 9 applies directly to any method that partitions activations and performs Gaussian normalization on each partition. The partitioning of BN, LN, IN, and GN are performed
in different respective manners, and each partition is individually subject to Gaussian normalization. Thus, the gradients of BN, LN, IN, and GN are residuals of regressions in the sense of Theorem 1.

heading: 3.2 WEIGHT NORMALIZATION
text: Salimans & Kingma (2016) introduced Weight Normalization (WN) in LSTMs, and noted improvements in the condition number of deep networks; WN divides each weight tensor by their respective vector 2-norms. In the view of gradient-least-squares, WN has a single-variable intercept-0 regression interpretation in back-propagation, analogous to BN. A raw weight vector v, is normalized and scaled before being used as coefficient weights w = c
‖v‖ v, where c is a learned downstream linear
scaling parameter.
In this regression setup, the length normalized weights of WN are analogous to the Gaussian normalized activations in BN. We write that z = v
‖v‖ =
w
c , and state directly an analogous relationship
between each ‖v‖ ∂L ∂vj
and the regression on {( zi, ∂L
∂zi )} i=1...N . See Appendix B Lemma 2 for
steps that derive the following identity: for loss L and for each component j, we have
‖v‖ ∂L ∂vj = ∂L ∂zj − β̂zj (17)
where β̂ = arg min
β ‖∇zL− βz‖2 = (∇zL)Tz (18)
The L2 normalization of weights in WN appears distinct from the Gaussian normalization of activations in BN; nevertheless, WN can also be recast as a least squares regression.

heading: 3.3 IDENTITY MAPPINGS IN RESNET, AND SHAKE-SHAKE RESNEXT REGULARIZATION

heading: BN ReLU Conv BN ReLU Conv
text: An update to the popular ResNet architecture showed that the network’s residual mappings can be dramatically improved with a new ordering (He et al., 2016b). The improvement moved BN operations into early positions and surprised the authors; we support the change from the perspective of gradient-least-squares. Figure 3 reviews the precise ordering in the two versions. Huang & Narayanan (2018) provides independent empirical support for the BN-early order, in shake-shake regularization (Gastaldi, 2017) architectures. We believe that the surprise arises from a perspective that views BN only as a way to control the distribution of activations; one would place BN after a sequence of convolution layers. In the gradient-least-squares perspective, the first layer of each residual mapping is also the final calculation for these gradients before they are added back into
the main trunk. The improved residual branch constrains the gradients returning from the residual mappings to be zero-centered and decorrelated with respect to some activations inside the branch. We illustrate this idea in Figure 3.

heading: 4 NORMALIZATION APPROACHES MOTIVATED BY LEAST SQUARES
text: Gradient-least-squares views back-propagation in deep neural networks as a solution to a regression problem. Thus, formulations and ideas from a regression perspective would motivate improvements and alternatives to BN. We pursue and evaluate two of these ideas.

heading: 4.1 BN AND LN AS TWO-STEP GRADIENT REGRESSION
text: BN and LN are similar to each other, but they normalize over different partitioning of the activations; in back-propagation, the regressions occur respectively with respect to different partitions of the activations. Suppose that a BN and a LN layer are chained serially in either order. This results in a two-step regression during back-propagation; in reversed order, the residual from the first regression is further explained by a second regression on a different partitioning. In principle, whether this helps would depend on the empirical characteristics of the gradients encountered during training. The second regression could further decorrelate partial gradients from activations. Empirically, we show improvement in a reference ResNet-34-v2 implementation on CIFAR-10 relative to BN with batch size 128. In all cases, only a single per-channel downstream affine transformation is applied, after both normalization layers, for consistency in the number of parameters. See table 1 for CIFAR10 validation performances. We kept all default hyperparameters from the reference implementation: learning schedules, batch sizes, and optimizer settings.

heading: 4.2 ADDRESSING SMALL BATCHES WITH LEAST-SQUARES REGULARIZATION
text: BN performs less well on small batches (Ioffe, 2017). Gradient-least-squares interprets this as gradient regressions failing on correlated data, an issue typically addressed by regularization. We pursue this idea to recover some performance on small batches by use of regularization. Our regularization uses streaming estimates of past gradients to create virtual data in the regression. This performed better than standard BN on the same batch size, but we did not recover the performance of large batches; this is consistent with the idea that regularization could not in general compensate for having much less data. See Appendix C for CIFAR-10 validation performances.

heading: 5 LIMITATIONS AND RELATED WORK

heading: 5.1 SWITCH NORMALIZATION
text: Luo et al. (2018a) introduced Switch Normalization (SwN), a hybrid strategy for combining moment calculations from LN, BN, and IN. SwN uses learnable scalar logits λk for k ∈ Ω = {BN, IN,LN} with corresponding softmax weighting activations wk = exp(λk)∑ k′ exp(λk′) to rescale the contributions to the batch mean for each normalization scheme. It uses an analogous set of parameters λ′k and activations w′k for variances. We sketch the back-propagation of a simplified version of SN in the perspective of gradient-least-squares. We ignore both the division and downstream affine
z → c · z + b. The normalization calculation inside SwN can be written as:
zbhwc = xbhwc − ∑ k∈Ω wkµbhwc,k√∑
k∈Ω w ′ kσ 2 bhwc,k
(19)
where Ω = {BN,LN, IN}. There is potentially a unique mean and variance used for each activation. Equation 19 bears similarities to the setup in Theorem 1, but we leave unresolved whether there is a gradient-least-squares regression interpretation for SN.

heading: 5.2 DECORRELATED BATCH NORMALIZATION AND SPECTRAL NORMALIZATION
text: Decorrelated Batch Normalization (DBN) (Huang et al., 2018) is a generalization of BN that performs Mahalanobis ZCA whitening to decorrelate the channels, using differentiable operations. On some level, the matrix gradient equation resemble the least squares formulation in Theorem 1.
Spectral Normalization (SpN) (Miyato et al., 2018) is an approximate spectral generalization of WN. For DBN and SpN, the regression interpretations remain unresolved.

heading: 5.3 RELATED WORK
text: BN has been instrumental in the training of deeper networks (Ioffe & Szegedy, 2015). Subsequent work resulted in Batch Renormalization (Ioffe, 2017), and further emphasized the importance of passing gradients through the minibatch moments, instead of a gradient-free exponential running average. In gradient-least-squares, use of running accumulators in the training forward pass would stop the gradients from flowing through them during training, and there would be no least-squares. He et al. (2016b) demonstrate empirically the unexpected advantages of placing BN early in residual mappings of ResNet.
Santurkar et al. (2018) showed that BN makes the loss landscape smoother, and gradients more predictable across stochastic gradient descent steps. Balduzzi et al. (2017) found evidence that spatial correlation of gradients explains why ResNet outperforms earlier designs of deep neural networks. Kohler et al. (2018) proved that BN accelerates convergence on least squares loss, but did not consider back-propagation of BN as a least squares residual. Luo et al. (2018b) has recast BN as a stochastic process, resulting in a novel treatment of regularization.

heading: 6 DISCUSSION, AND FUTURE WORK
text: This work makes explicit how BN back-propagation regresses partial derivatives against the normalized activations and keeps the residual. This view, in conjunction with the empirical success of BN, suggests an interpretation of BN as a gradient regression calculation. BN and its variants decorrelate and zero-center the gradients with respect to the normalized activations. Subjectively, this can be viewed as removing systematic errors from the gradients. Our view also support empirical results in literature preferring early BN placement within neural network branches.
Leveraging gradient-least-squares considerations, we ran two sets of normalization experiments, applicable to large batch and small batch settings. Placing a LN layer either before or after BN can be viewed as two-step regression that better explains the residual. We show empirically on CIFAR10 that BN and LN together are better than either individually. In a second set of experiments, we address BN’s performance degradation with small batch size. We regularize the gradient regression with streaming gradient statistics, which empirically recovers some performance on CIFAR-10 relative to basic BN, on batch size two.
Why do empirical improvements in neural networks with BN keep the gradient-least-squares residuals and drop the explained portion? We propose two open approaches for investigating this in future work. A first approach focuses on how changes to the gradient regression result in different formulations; the two empirical experiments in our work contribute to this. A second approach examines the empirical relationships between gradients of activations evaluated on the same parameter values; we can search for a shared noisy component arising from gradients in the same normalization partition. Suppose that the gradient noise correlates with the activations – this is plausible because
the population of internal activations arise from using shared weights – then normalizations could be viewed as a layer that removes systematic noise during back-propagation.
In conclusion, we have presented a novel view that reorganizes the back-propagation of BN as a least squares residual calculation. This view generates novel descriptions of normalization techniques related to BN, and comments on the ordering of layers inside the residual mappings of ResNet. This view is extensible and will motivate novel designs of neural network components in future work.

heading: A PARTIAL DERIVATIVES OF THE NORMALIZATION FUNCTION
text: Lemma 1. Consider the Gaussian normalization function that maps all {xi} into corresponding
{zi}, for i ∈ 1 . . . N . We define µ = 1
N N∑ j=1 xi and σ2 = 1 N N∑ j=1 (xi − µ)2 with σ > 0. The
normalized activations zi are defined as zi = xi − µ σ . Then, the partial derivatives satisfy
∂zj ∂xi =  −1− zizj σN if j 6= i
N − 1− z2i σN if j = i
(20)
Proof. In deriving ∂zj ∂xi
, we will treat the cases of when j 6= i and when j = i separately. We start by examining intermediate quantities of interest as a matter of convenience for later use. We define helper quantities ui = xi − µ. Note that each uj depends on all of xi via µ. Next, we write out useful identities
1
N N∑ j=1 uj = 0 (21)
∂µ ∂xi = 1 N (22)
∂ui ∂xj =  −1 N if i 6= j
N − 1 N if i = j
(23)
We prepare to differentiate with rule of total derivative:
σ2 = 1
N N∑ j=1 u2j (24)
∂(σ2)
∂xi =
1
N N∑ j=1 ∂(σ2) ∂uj ∂uj ∂xi
(25)
Making use of equations 21, 22, 23 and 25, We simplify ∂σ
∂xi for any i as follows.
∂σ ∂xi = ∂σ ∂(σ2)
∂(σ2)
∂xi (26)
= ∂σ
∂(σ2)
1
N N∑ j=1 ∂(σ2) ∂uj ∂uj ∂xi
(27)
=
[ 1
2N (σ2)
−1 2 ] N∑ j=1 (2uj) ∂uj ∂xi
(28)
= 1
2Nσ  N∑ j 6=i (2uj) ∂uj ∂xi + 2ui ∂ui ∂xi  (29) = 1
Nσ  N∑ j 6=i (uj) −1 N + uiN − 1 N  (30)
= 1
Nσ   N∑ j=1
(uj)︸ ︷︷ ︸ =0
−1 N + ui  (31)
= 1
Nσ ui (32)
∂σ ∂xi = xi − µ σN
(33)
We apply the quotient rule on ∂zj ∂xi when j 6= i, then substitute equation 33
∂zj ∂xi = ∂ ∂xi [ xj − µ σ ] (34)
= −σ ∂µ∂xi − (xj − µ) ∂σ ∂xi
σ2 (35)
= −σ N − (xj − µ) xi−µ σN
σ2 (36)
= −1− (xj − µ)xi−µσ2
Nσ (37)
∂zj ∂xi = −1− zizj Nσ (38)
Similarly, when i = j,
∂zi ∂xi = ∂ ∂xi [ xi − µ σ ] (39)
= σ(1− ∂µ∂xi )− (xi − µ) ∂σ ∂xi
σ2 (40)
= σ − −σN − (xi − µ) xi−µ σN
σ2 (41)
= N − 1− (xj − µ)xi−µσ2
Nσ (42)
∂zi ∂xi = N − 1− z2i Nσ (43)

heading: B WEIGHT NORMALIZATION RECAST AS GRADIENT REGRESSION
text: We show steps to recast the gradient of WN as regression. In WN, A raw weight vector v is normalized and scaled and before being used as coefficient weights. Salimans & Kingma (2016) introduced their transformation as w = c
‖v‖ v, where c is a learned downstream linear scaling parameter. In
our regression setup, we ignore c. We define z = v
‖v‖ =
w
c . We derive the analogous relationship
between each ‖v‖ ∂L ∂vj
and the regression on {( zi, ∂L
∂zi )} i=1...N .
Note that intercept-0 single variable least squares β̂ = arg min β
‖∇zL− βz‖2 has the solution
β̂ = (∇zL)Tz ‖z‖2 = (∇zL)Tz, since ‖z‖ = 1.
Lemma 2. Let v ∈ RN be weights, and let z = v ‖v‖ . Let L be a loss function dependent on z. Then, for each component j, we have
‖v‖ ∂L ∂vj = ∂L ∂zj − β̂zj (44)
where β̂ = arg min β
‖∇zL− βz‖2 = (∇zL)Tz.
Proof. Salimans & Kingma (2016) wrote their gradients as follows:
∇cL = ∇wL · v ‖v‖
(45)
∇vL = c
‖v‖ ∇wL− c∇cL ‖v‖2 v (46)
In our notation where z = v
‖v‖ =
w
c , we have
∇zL = c∇wL (47)
To recover β̂ We substitute equation 45, and then subsequently equation 47 into equation 46 ∇vL = ( c
‖v‖
) ∇wL−
c∇wL · v ‖v‖3 v (48)
∇vL = 1
‖v‖
( c∇wL−
(c∇wL · v)v ‖v‖2
) (49)
‖v‖∇vL = ∇zL− (∇zL · v)v ‖v‖2
(50)
‖v‖∇vL = ∇zL− (∇zL · z)︸ ︷︷ ︸ β̂ z (51)
The result follows: for loss L and for each component j, we have
‖v‖ ∂L ∂vj = ∂L ∂zj − β̂zj (52)

heading: C ADDRESSING SMALL BATCHES WITH LEAST-SQUARES REGULARIZATION
text: Let b be an index for different batches; let X refer to data inputs into the neural network (for example image and class label) within a single step of training, and let X(b) refers to the value of all data
inputs in batch b. In our work, we keep track of am exponential running estimates across batches, α̂∗ ∼ EbEi [ ∂L
∂zi ∣∣∣∣ X=X(b) ] and β̂∗ ∼ EbEi [ zi ∂L ∂zi ∣∣∣∣ X=X(b) ] that marginalize the (B,H,W ) di-
mensions into accumulators of shape C. The b subscript of the outer expectation is slightly abusive notation indicating that α̂∗ and β̂∗ are running averages across recent batches with momentum as a hyperparameter that determines the weighting. We regularize the gradient regression with virtual activations and virtual gradients, defined as follows. We append two virtual batch items, broadcast to an appropriate shape, x+ = µb + σb and x− = µb − σb. Here, µb and σb are batch statistics of the real activations. The concatenated tensor undergoes standard BN, which outputs the usual {zi} for the real activations, but z+ = 1 and z− = −1 for the virtual items. The z+ and z− do not affect the feed forward calculations, but they receive virtual gradients during back-propagation:
∂L
∂z+ = α̂∗ + β̂∗ (53)
∂L
∂z− = α̂∗ − β̂∗ (54)
Virtual data ( z+, ∂L
∂z+
) and ( z−, ∂L
∂z−
) regularizes the gradient-least-squares regression. ∂L
∂z+
and ∂L
∂z− eventually modify the gradients received by the real xi activations. The virtual data can be weighted with hyperparameters. In our experiments, we see improvements, robust to a hyperparameter cross-product search over the weightings and the momentum for α̂∗ and β̂∗. The momentum for α̂∗ and β̂∗ were in {.997, .5} and the virtual item weights were in {2i−1}i∈{0,1,2,3}. The performance of larger batches are not recovered; regularized regression could not be reasonably expected to recover the performance of regressing with more data. See table 2 for final validation performances with a reference Tensorflow ResNet-34-v2 implementation on batch size of two. The baseline evaluation with identity (no normalization) experienced noticeable overfitting in terms of cross entropy but not accuracy. The base learning rate was multiplied by 164 relative to the baseline rate used in runs with batch size 128.

Review summaries:
The primary technical contribution comes from Section 2, where it is demonstrated that the normalized back-propagated gradients obtained from a BN layer can be viewed as the residuals of the gradients obtained without BN regressed via a simple two-parameter model of the activations.  In some sense though this result is to be expected, since centering data (i.e., removing the mean as in BN) can be generically viewed as computing the residuals after a least squares fit of a single constant, and similarly for de-trending with respect to a single independent variable, in this case the activations.  So I'm not sure that Theorem 1 is really that much of an insightful breakthrough, even if it may be nice to work through the precise details in the specific case of a BN layer and the relationship to gradients.

The paper aims at a better understanding of the positive impacts of Batch Normalisation (BN) on network generalisation (mainly) and  convergence of learning. First, the authors propose a novel interpretation of the BN re-parametrisation. They show that an affine transform of the variables with their local variance (scale) and mean (shift) can be interpreted as a decomposition of the gradient of the objective function into a regressor assuming that the gradient is parallel to the variables (up to a shift) and the residual part which is the gradient w.r.t. to the new variables. In the second part of the paper, authors review various normalisation proposals (differing mainly in the subset of variables over which the normalisation statistics is computed) as well as the known empirical findings about the dependence of BN on the batch size. The paper presents an experiment that combines two normalisation variants. A further experiment strives at regularising BN for small batch sizes.