user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Normalization Gradients are Least-squares Residuals

Paper content:
Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks.
Discussions of why this normalization works so well remain unsettled.
We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN.
We recast the back-propagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations.
This view, which we term {\em gradient-least-squares}, is an extensible and arithmetically accurate description of BN.
To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.
Training deep neural networks has become central to many machine learning tasks in computer vision, speech, and many other application areas.
BID10 showed empirically that Batch Normalization (BN) enables deep networks to attain faster convergence and lower loss.
Reasons for the effectiveness of BN remain an open question BID12 .
Existing work towards explaining this have focused on covariate shift; Santurkar et al. (2018) described how BN makes the loss function smoother.
This work examines the details of the back-propagation of BN, and recasts it as a least squares fit.
This gradient regression zero-centers and decorrelates partial derivatives from the normalized activations; it passes on a scaled residual during back-propagation.
Our view provides novel insight into the effectiveness of BN and several existing alternative normalization approaches in the literature.
This work makes explicit how BN back-propagation regresses partial derivatives against the normalized activations and keeps the residual.
This view, in conjunction with the empirical success of BN, suggests an interpretation of BN as a gradient regression calculation.
BN and its variants decorrelate and zero-center the gradients with respect to the normalized activations.
Subjectively, this can be viewed as removing systematic errors from the gradients.
Our view also support empirical results in literature preferring early BN placement within neural network branches.Leveraging gradient-least-squares considerations, we ran two sets of normalization experiments, applicable to large batch and small batch settings.
Placing a LN layer either before or after BN can be viewed as two-step regression that better explains the residual.
We show empirically on CIFAR-10 that BN and LN together are better than either individually.
In a second set of experiments, we address BN's performance degradation with small batch size.
We regularize the gradient regression with streaming gradient statistics, which empirically recovers some performance on CIFAR-10 relative to basic BN, on batch size two.Why do empirical improvements in neural networks with BN keep the gradient-least-squares residuals and drop the explained portion?
We propose two open approaches for investigating this in future work.
A first approach focuses on how changes to the gradient regression result in different formulations; the two empirical experiments in our work contribute to this.
A second approach examines the empirical relationships between gradients of activations evaluated on the same parameter values; we can search for a shared noisy component arising from gradients in the same normalization partition.
Suppose that the gradient noise correlates with the activations -this is plausible because the population of internal activations arise from using shared weights -then normalizations could be viewed as a layer that removes systematic noise during back-propagation.In DISPLAYFORM0 Then, the partial derivatives satisfy DISPLAYFORM1 Proof.
In deriving ∂z j ∂x i , we will treat the cases of when j = i and when j = i separately.
We start by examining intermediate quantities of interest as a matter of convenience for later use.
We define helper quantities u i = x i − µ.
Note that each u j depends on all of x i via µ.
Next, we write out useful identities DISPLAYFORM2 We prepare to differentiate with rule of total derivative: DISPLAYFORM3 Making use of equations 21, 22, 23 and 25, We simplify ∂σ ∂x i for any i as follows.
DISPLAYFORM4 We apply the quotient rule on ∂z j ∂x i when j = i, then substitute equation 33 DISPLAYFORM5 Similarly, when i = j, inputs in batch b.
In our work, we keep track of am exponential running estimates across batches, DISPLAYFORM6 DISPLAYFORM7 DISPLAYFORM8 that marginalize the (B, H, W ) dimensions into accumulators of shape C. The b subscript of the outer expectation is slightly abusive notation indicating thatα * andβ * are running averages across recent batches with momentum as a hyperparameter that determines the weighting.
We regularize the gradient regression with virtual activations and virtual gradients, defined as follows.
We append two virtual batch items, broadcast to an appropriate shape, x + = µ b + σ b and x − = µ b − σ b .
Here, µ b and σ b are batch statistics of the real activations.
The concatenated tensor undergoes standard BN, which outputs the usual {z i } for the real activations, but z + = 1 and z − = −1 for the virtual items.
The z + and z − do not affect the feed forward calculations, but they receive virtual gradients during back-propagation: DISPLAYFORM9 Virtual data z + , ∂L ∂z + and z − , ∂L ∂z − regularizes the gradient-least-squares regression.
∂L ∂z + and ∂L ∂z − eventually modify the gradients received by the real x i activations.
The virtual data can be weighted with hyperparameters.
In our experiments, we see improvements, robust to a hyperparameter cross-product search over the weightings and the momentum forα * andβ * .
The momentum forα * andβ * were in {.997, .5} and the virtual item weights were in {2 i−1 } i∈{0,1,2,3} .
The performance of larger batches are not recovered; regularized regression could not be reasonably expected to recover the performance of regressing with more data.
See table 2 for final validation performances with a reference Tensorflow ResNet-34-v2 implementation on batch size of two.
The baseline evaluation with identity (no normalization) experienced noticeable overfitting in terms of cross entropy but not accuracy.
The base learning rate was multiplied by 1 64 relative to the baseline rate used in runs with batch size 128.

assistant: text: Gaussian normalization performs a least-squares fit during back-propagation, which zero-centers and decorrelates partial derivatives from normalized activations.