On the Importance of Gaussianizing Representations
Authors: Daniel Eftekhari
Vardan Papyan
Venue: ICML 2025 poster (2025)
Keywords: mutual information game
gaussianization
power transform
noise robustness
normalization layer
deep learning
information theory
Primary area: deep_learning->other_representation_learning

Abstract: The normal distribution plays a central role in information theory – it is at the same time the best-case signal and worst-case noise distribution, has the greatest representational capacity of any distribution, and offers an equivalence between uncorrelatedness and independence for joint distributions. Accounting for the mean and variance of activations throughout the layers of deep neural networks has had a significant effect on facilitating their effective training, but seldom has a prescription for precisely what distribution these activations should take, and how this might be achieved, been offered. Motivated by the information-theoretic properties of the normal distribution, we address this question and concurrently present normality normalization: a novel normalization layer which encourages normality in the feature representations of neural networks using the power transform and employs additive Gaussian noise during training. Our experiments comprehensively demonstrate the effectiveness of normality normalization, in regards to its generalization performance on an array of widely used model and dataset combinations, its strong performance across various common factors of variation such as model width, depth, and training minibatch size, its suitability for usage wherever existing normalization layers are conventionally used, and as a means to improving model robustness to random perturbations.

On the Importance of Gaussianizing Representations
Daniel Eftekhari Vardan Papyan 312
Abstract
the maximal amount of information (Shannon, 1948). This can be viewed as desirable property in learning systems such as neural networks, where the activations of successive layers equivocates to successive representations of the data.
The normal distribution plays a central role in information theory it is at the same time the best-case signal and worst-case noise distribution, has the greatest representational capacity of any distribution, and offers an equivalence between uncorrelatedness and independence for joint dis- tributions. Accounting for the mean and vari- ance of activations throughout the layers of deep neural networks has had significant effect on facilitating their effective training, but seldom has prescription for precisely what distribu- tion these activations should take, and how this might be achieved, been offered. Motivated by the information -etemaatio--hooettic properties of the normal distribution, we address this question and concur- rently present normality normalization: novel normalization layer which encourages normality in the feature representations of neural networks using the power transform and employs additive Gaussian noise during training. Our experiments comprehensively demonstrate the effectiveness of normality normalization, in regards to its gen- eralization performance on an array of widely used model and dataset combinations, its strong performance across various common factors of variation such as model width, depth, and training minibatch size, its suitability for usage wherever existing normalization layers are conventionally used, and as a means to improving model robust- ness random perturbations.
Moreover, a signal following the normal distribution is max- imally robust to random perturbations (Cover & Thomas, 2006), and thus presents a desirable property for the rep- resentations of learning systems; especially deep neural networks, which are susceptible to random (Ford et al., 2019) and adversarial (Szegedy et al., 2014) perturba- tions. Concomitantly, the normal distribution is information- theoretically the worst-case perturbative noise distribution (Cover & Thomas, 2006), which suggests models gaining robustness Gaussian noise should be robust to any other form of random perturbations.
We show that encouraging deep learning models to encode their activations using the normal distribution in conjunction with applying additive Gaussian noise during training, helps improve generalization. We do so by means of a novel layer normality normalization - so-named because it applies the power transform, a technique used to gaussianize data (Box & Cox, 1964; Yeo & Johnson, 2000), and because it can be viewed as augmentation of existing normalization techniques such as batch (loffe & Szegedy, 2015), layer (Ba et al., 2016), instance (Ulyanov et al., 2016), and group (Wu & He, 2018) normalization.
Our experiments comprehensively demonstrate the general effectiveness of normality normalization, in terms of its generalization performance, its strong performance across various common factors of variation such model width, depth, and training minibatch size, which furthermore serve to highlight why it is effective, its suitability for usage wher- ever existing normalization layers are conventionally used, and its effect on improving model robustness under random perturbations.
1. Introduction
The normal distribution is unique- information theory shows that among all distributions with the same mean and variance, a signal following this distribution encodes
outline some of the desirable properties normality can imbue in learning models, which serve as motivating factors for the development of normality normal- ization. In Section 3 we provide a brief background on the power transform, before presenting normality normalization in Section 4. In Section we describe our experiments, analyze the results, and explore some of the properties of models trained with normality normalization. In Section
In Section 2
'Department of Computer Science, University of Toronto, Toronto, Canada VVector Institule, Toronto, Canada 'Department of Mathematics, University of Toronto, Toronto, Canada. Corre- spondence to: Daniel Eftckhari <defte@cs.tonm tow.oootoo..oo..
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Importance of Gaussianizing Representations
6 we comment on related work and discuss some possible future directions. Finally in Section we contextualize nor- mality normalization in the broader deep learning literature, and provide a few concluding remarks.
This leads the following minimax formulation of the game
min max 7 (X;; X 2) max min I (X; X 2); (2) 2 X
which implies that any deviation from normality, for X or Z, is suboptimal from that player's 's perspective.
2. Motivation
In this section we present motivating factors for encouraging normality in feature representations in conjunction with using additive random noise during learning. Section 5 substantiates the applicability of the motivation through the experimental results.
2.1.2. RELATION TO LEARNING
How might this framework relate to the learning setting? First, previous works have shown that adding noise to the inputs (Bishop, 1995) or to the intermediate activations (Srivastava et al., 2014) of neural networks can be an effec- tive form of regularization, leading to better generalization. Moreover, the mutual information game shows that, among encoding distributions with first and second moment con- straints, the normal distribution is maximally robust to random perturbations. Taken together these suggest that encoding activations using the normal distribution is the most effective way of using noise as a regularizer, because greater degree of regularizing noise in the activations can be tolerated for the same level of corruption.
2.1. Mutual Information Game & Noise Robustness
2. OVERVIEW OF THE FRAMEWORK
Under first and second moment constraints, the normal dis- tribution is at the same time the best possible signal distri- bution, and the worst possible noise distribution; a result which can be studied in the context of the Gaussian channel (Shannon, 1948), and through the lens of the mutual infor- mation game (Cover& & Thomas, 2006). In this framework, X and Z denote two independent random variables, repre- senting the input signal and noise, and Y = X + Z is the output. The mutual information between X and Y is de- noted by (X;Y): Y); X tries to maximize this term, while Z tries to minimize it. Both X and Z can encode their signal using any probability distribution, so that their respective objectives are optimized for.
Second, the mutual information game suggests gaining ro- bustness to Gaussian noise is optimal because it is the worst- case noise distribution. This suggests adding Gaussian noise specifically - to activations during training should have the strongest regularizing effect. Moreover, gaining robustness to noise has previously been demonstrated imply better generalization (Arora al., 2018).
Finally, there exists a close correspondence between the mutual information between the input and the output of a channel subject to additive Gaussian noise, and the mini- mum mean-squared error (MMSE) in estimating the imput given the output (Guo al., 2005). This suggests that when Gaussian noise is added a given layer's activations, quan- tifying the attenuation of the noise across the subsequent layers of the network, as measured by the mean-squared er- ror (MSE) relative to the unperturbed activations, provides a measurable proxy for the mutual information between the activations of successive layers in the presence of noise.
Information theory answers the question of what distribution X should choose to maximize I (X; Y). It also answers the question of what distribution Z should choose to minimize 7 Y) As shown by the following theorem, remark- ably the answer to both questions is the same the normal distribution.
Theorem 2.1. (Cover & Thomas, 2006) Mutual Informa- tion Game. Let X, Z be independent, continuous ran- dom variables with non-zero support over the entire real line, and satisfying the moment conditions F. [X] Ma. E [x]]] 12 + 0% and E E|Z]= uaz E
[Z²]]= = 22² + o².
Further let X*, z" be normally distributed random vari- ables satisfying the same moment conditions, respectively. Then the following series of inequalities holds
2.2. Maximal Representation Capacity and Maximally Compact Representations
The entropy of a random variable is a measure of the number of bits it can encode (Shannon, 1948). and therefore of its representational capacity (Cover & Thomas, 2006). The normal distribution is the maximum entropy distribution for specified mean and variance. This suggests that unit which encodes features using the normal distribution has maximal representation capacity given fixed variance budget, and
T(X ;X_ + +2")
X(";:X* +Z") I X +Z).
Proof. Without loss of generality let μrr O and n! = 0. The first inequality hinges on the entropy power inequal- ity. The second inequality hinges on the maximum entropy of the normal distribution given first and second moment constraints. See Cover& & Thomas (2006) for details. LI
Note that conventional normalization layers respect precisely these constraints, since the mean and variance used to normal- ize the pre-activations.
2

---

On the Importance of Gaussianizing Representations
therefore encodes information compactly as possible. This may then suggest that it is efficient for unit (and by extension layer) to encode its activations using the normal distribution.
estimation, that the transformed variable is normally distributed possible, by minimizing the following NLL.
1 [he; (log (2T) + 1) + log (o² (2))
2 1 1
2
N
(4)
2.3. Maximally Independent Representations
Σ
log (1 + +h1),
N
Previous work has explored the beneficial effects of decorre- lating features in neural networks (Huang et al., 2018; 2019; Pan et al., 2019). Furthermore, other works have shown that preventing feature co-adaptation is beneficial for training deep neural networks (Hinton et al., 2012).
where á (A)
# Σ ve (bgj and 8² (A)
E (v (hg; İ (A))
4. Normality Normalization
For any set of random variables, for example representing the pre-activation values of various units in neural network layer, uncorrelatedness does not imply independence in gen- eral. But for random variables whose marginals are normally distributed, then as shown by Lemma 1, uncorrelatedness does imply independence when they are furthermore jointly normally distributed. Furthermore, for any given (in general, non-zero) degree of correlation between the random vari- ables, they maximally independent- relative any other possible joint distribution when they are jointly normally distributed.
To gaussianize a unit's pre-activations h. normality normal- ization estimates Ả using the method present in Sub- section 4.1, and then applies the power transform given by Equation 3. It subsequently adds Gaussian noise with scal- ing as described in Subsection 4.2. These steps are done between the normalization and affine transformation steps conventionally performed in other normalization layers.
4.1. Estimate of Å
Differentiating Equation 4 w.r.t. X and setting the resulting expression to 0 does not lead to a closed-form solution for Å. which suggests an iterative method for its estimation; for example gradient descent, or a rrot-finding algorithm (Brent, 1971). However, motivated by the NLL's convexity in À (Yeo & Johnson, 2000), we use a quadratic series expansion for its approximation, which we outline in Appendix D.
We use these results to motivate the following argument: for a given level of correlation, encouraging normality in the feature representations of units would lead to the desirable property of maximal independence between them; in the setting where increased unit-wise normality also lends itself to increased joint normality.
With the quadratic form of the NLL, we can estimate Á with one step of the Newton- Rawton-Raphson method:
3. Background: Power Transform
Before introducing normality normalization, we briefly out- line the power transform (Yeo & Johnson, 2000) which our proposed normalization layer employs. Appendix C pro- vides the complete derivation of the negative likelioel (NLL) objective function presented below.
L' Ch; X A= 1)
-= 1
(5)
L"(h; X 1)'
where the series expansion has been taken around* λo 1. The expressions for L' L'(h; À = 1) and L"(h; X = 1) are
outlined in Appendix D.
Consider a random variable H from which a sample = { is 2 The power transform gaussianizes h by applying the following function for each he:
Appendix E provides empirical evidence substantiating the similarity between the NLL and its second-order series ex- pansion around X 1, and furthermore demonstrates the accuracy of obtaining the estimates I using one step of the Newton-Raphson method.
1 (1+ -1)
h 0 0
sb (h; A)
log (1 + hi)
Subsequent to estimating X, the power transform is applied to each of the pre-activations to obtain x, = (h:s ))
(1 h) 2))=> -1)-
0
log (1 - hi.
h< 0, = 2
(3)
"To simplity the presentation, momentarily defer the cases and = and outine the NLL for h 2 only, as the case for h < 0 follows closely by symmetry.
The parameter X is obtained using maximum likelihood
IIn the context of normalization layers, N represents the num- ber of samples being normalized; for example in batch normal- ization, N = BHW for convolutional layers, where B is the minibatch size, and H, W are respectively the height and width of the activation.
"The previously deferred cases of 0 and X are thus inconsequential, in the context of computing an estimate A, by continuity of the quadratic form of the series expansion for the NL... However, these two cases still need to be considered when applying the transformation function itself.
3

---

On the Importance of Gaussianizing Representations
We next discuss few facets of the method.
4.2. Additive Gaussian Noise with Scaling
Normality normalization applies regularizing additive ran- dom noise to the output of the power transform; step which is also motivated through the informationn princi- ples described in Subsection 2.1, and whose regularizing effect is magnified by having gaussianized pre-activations.
Justification for the Second Order Method The justifi- cation for using the Newton-Raphsee method for computing is as follows:
A first-order gradient-basee based method would require iter- ative refinements its estimates of Å in order to find the minima, which would significantly affect runtime. In contrast, the Newton-Raphson method is guaranteed to find the minima of the quadratic loss in one step.
For each input indexed by E {L..... NN, during training" we have ys = 2; + 2; 5 8, where 2; the i-th input's 's post- power transform value, 2; N (0. 1), & 2 0 is the noise factor, and norm of the post-power transform values, normalized by the sample size N.
Nv ||ar 1|12 represents the zero-centered
A first-order gradient-basee method for computing X would require additional hyperparameter for the step size. Due to the quadratic nature of the loss, the Newton- Raphson method necessarily does not require any such additional hyperparameter.
Importantly, scaling each of the sampled noise values 2; for a given channel's minibatch by the channel-specific scaling factor s, leads to an appropriate degree of additive noise for each of the channel's constituent terms 24-- This is significant because for a given minibatch, each channel's norm will differ from the norms of other channels.
The minibatch statistics jt and 6" are available in closed- form. It is therefore natural to seek a closed-form ex- pression for Å, which is facilitated by using the Newton- Raphson method.
Furthermore, we treat s as a constant, so that its constituent terms are not incorporated during backpropagation. ? This is significant because the purpose of is to scale the additive random noise by the minibatch's 's statistics, and not for it to contribute to learning directly by affecting the gradients of the constituent terms.
Location of Series Expansion The choice of taking the series expansion around Åg = 1 is justified using the follow- ing two complementary factors:
=1 1 corresponds to the identity transformation, and hence having Ag = 1 as the point where the series expansion taken, facilitates its recovery if this is opti- mal.
Note that we employ the f fn-ormm for æ rather than the f--ormm because it lends itself to a more robust measure of dispersion (Pham-Git Gia & Hung, 2001).
It equivocates to assuming the least about the nature of the deviations from normality in the sample statistics, since it avoids biasing the form of the series expansion for the loss towards solutions favoring Ả < 1 or X 1.
Algorithm I provides a summary of normality normaliza- tion.
5. Experimental Results & Analysis
Order of Normalization and Power Transform Steps Applying the power transform after the normalization step beneficial, because having zero mean and unit variance activations simplifies several terms in the computation of X, as shown in Appendix D, and improves numerical stability.
5.1. Experimental Setup
For each model and dataset combination, M = 6 models were trained, each with differing random initializations for the model parameters. Wherever a result is reported nu- merically, it is obtained using the mean performance and one standard error from the mean across the M runs. The best performing models for given dataset and model com- bination are shown in bold. Wherever a result is shown graphically, it is displayed using the mean performance, and its 95% confidence interval when applicable. The training
No Additional Learned Parameters Despite having in- creased normality in the features, this came at no additional cost in terms of the number of learnable parameters relative to existing normalization techniques.
Test Time In the case where normality normalization is used to augment batch normalization, in addition to comput- ing global estimates for μ and o², we additionally compute global estimate for X. These are obtained using the re- spective training set running averages for these terms, anal- ogously with batch normalization. At test time, these global estimates μ, o², X are used, rather than the test minibatch statistics themselves.
'We do not apply additive random noise with scaling at test time.
'For clarity the present discussion assumes the case where normality normalization is used to augment batch normalization. However, the discussion applies equally to other normalization layers, such layer, instance, and group normalization. "Implementationally. this is done by disabling gradient tracking when computing these terms.
4

---

On the Importance of Gaussianizing Representations
Algorithm 1 Normality Normalization
Table 1. Validation accuracy across several datasets for vision transformer (ViT) architecture (see training details for model spec- ification), when using LayerNormalNorm (LNN) vs. LayerNorm
Input: u = (u
Output: v= (
(LN). Data augmentations were employed during training.
Learnable Parameters: 8
Noise Factor: 2 0
DATASET
LN
LNN
SVHN
94.61 + 0.31
95.78 + 0.21
Normalization:
CIFARIO
89.97 1 0.16
91.18 1 0.13
j N Ei= Ui
CIFARTOO
66.40
0.42
70.12 + 0.22
0.22
FOOD101
73.25 1 0.19
79.11 1 0.09
_²_ N/2 E N
(u - 2
IMAGENET TOPI
71.34 + 0.16
75.25
+ 0.07
h; V82+e
9.-R
IMAGENET TOP5
89.40 0.11
92.23 + 0.04 0.00
Power Transform and Scaled Additive Noise:
Table 2. Validation accuracy for several ResNet (RN) architecture and dataset combinations, when using BatchNorrmalNom (BNN) BatchNorm (BN). No data augmentations were employed during training.
== 1 C' ()
J; = v (h:: 2)
with gradient tracking disabled:
DATASET
MODEL
BN
BNN
Σラ zz
CIFARIO
RNI8
88.89
0.07
90.41 + 0.09
N >E -
CIFARTOO
RN18
62.02
0.17
65.82 + 0.11
STL10
RN34
58.82 1 0.52
63.86 1 0.45
sample 2 N 10
TINYIN TOPI
RN34
58.22
0.12
60.57 + 0.14
2 + { 5
TINYIN TOP5
RN34
4.116
83.31 + 0.13
CALTECH101
RN50
72.60 1 0.35
74.71 1 0.51
Affine Transform:
FOODIOI
RN50
61.15 - 0.44
63.51 + 0.33
0/2
65
Validation Accuracy (%)
configurations of the m are outlined in Appendix
60
5.2. Generalization Performance
55
We evaluate layer normality normalization (LayerNormal- Norm) and layer normalization (LayerNorm) on variety of models and datasets, as shown Table 1. A similar eval- uation is done for batch normality normalization (BatchNor- malNorm) and batch normalization (BatchNorm), shown in Table 2.
50
45
GN GNN
BN BNN
Normalization Layer
Figure 1. Normality normalization is effective for various nor- malization layers. Validation accuracy for ResNet34 architectures evaluated on the STL10 dataset. Each bar represents the perfor- mance of the ResNet34 architecture, when using given normal- ization layer across the entire network. INN: InstanceNormalNorm, IN: InstanceNorm, GNN: GroupNormalNorm, GN: GroupNorm, BNN: BatehNormmlNNom, BN: BatchNorm.
Normality Normalization is Performant LayerNormal- Norm generally outperforms LayerNorm across multiple architectures and datasets, with similar trend holding be- tween BatchNormalNorm and BatchNorm.
Effective With and Without Data Augmentations Nor- mality normalization is effective for models trained with (Table 1) and without (Table 2) data augmentations. This of value in application areas such as time series analysis and fine-gra grained medical image analysis, where it is often not clear what data augmentations are appropriate.
5.3. Effectiveness Across Normalization Layers
Figure 1 demonstrates the effectiveness of normality nor- malization across various normalization layer types. Here we further augmented group normalization (GroupNorm) to group normality normalization (GroupNormalNorm) and instance normalization (InstanceNorm) instance normal-
CCode is made available https: //tt:::///thbbt com/ SanielFftekhar /tttaarrr///aaaaaaa whwemaaityy-ormaaa ization.
5

---

On the Importance of Gaussianizing Representations
ity normalization (InstanceNormalNorm))
suggests normality normalization may correct for in- creased tendeney towards non-normality as function of model depth.
Table 3 furthermore contrasts decorrelated batch normaliza- tion (Huang et al., 2018) 8) with its augmented form decor- related batch normality normalization, providing further evidence that normality normalization can be employed wherever normalization layers are conventionally used.
71-
Validation Accuracy
69-
RatchNormalNorm
Table As Table 2, but for models using decorrelated Batch- NormalNorm (DBNN) vs. decorrelated BatchNorm (DBN).
BatchNorm
67-
DATASET
MODEL
DBN
DBNN
65
CIFARIO
RN18
90.66 + 0.05
91.50
0.03
CIFARTO0 RN18
65.11 + 0.06
67.53 0.10
63
STLIO RN34
66.76 + 0.29
69.36 0.14
16
28
40
Depth
Figure 3. Normality normalization effective for networks various depths. Validation accuracy on the STL10 dataset for WideResNet architectures with varying depths when controlling for width factor of 2, when using BatchNormalNorn vs. BatchNorm.
5.4. Effectiveness Across Model Configurations
Network Width Figure 2 shows that BatchNormalNorm outperforms BatchNorm across varying WideResNet archi- tecture model widths. Of particular note is that BatchNor- malNorm shows strong performance even in the regime of relatively small network widths, whereas BatchNorm's ' performance deteriorates. This may indicate that for small- width networks, which do not exhibit the Gaussian process limiting approximation attributed to large- width networks (Neal, 1996; Lee al., 2018; Jacot al., 2018; Lee al., 2019), normality normalization provides correcting effect. This could, for example, be beneficial for hardware-limited deep learning applications.
Training Minibatch Size Figure shows that BatchNor- malNorm maintains high level of performance across mini- batch sizes used during training, which provides further evi- dence for normality normalization' 's general effectiveness across variety of configurations.
Validation Accuracy (%)
89
Validation Accuracy (%)
71
88-
87-
69
BatchNormal
86
BatchNorn
67-
16
32
128 256
512
65-
Minibatch Size
63
ButchNormulNorm
Figure 4. Normality normalization is effective across minibatch sizes used during training. Validation accuracy for ResNet18 architectures evaluated on the CIFAR dataset, with varying mini- batch sizes used during training, when using NotchNNormalNomm vs. BatchNorm.
BatchNorm
61
1
2
10
Width Factor
Figure Normality normalization is effective for small and large width networks. Validation accuracy on the STL-10 dataset for WideResNet architectures with varying width factors when controlling for depth of 28, when using NorehNorral/orm vs. BatchNorm.
5.5. Normality of Representations
Figure 5 shows representative Q-Q plots (Wilk & Gnanade- sikan, 1968), a method for assessing normality, for post- power transform feature values when using BatchNormal- Norm, and post-normalization feature values when using BatchNorm. Figure 6 shows an aggregate measure of nor- mality across model layers, derived from several Q-Q plots corresponding to different channel and minibatch combina- tions. The figures correspond to models which have been
Network Depth Figure 3 shows that Norm outperforms BatchNorm across varying model depths. This suggests normality normalization is beneficial both for small and large-depth models. Furthermore, the increased benefit to performance for Notr in deeper networks
6

---

On the Importance of Gaussianizing Representations
50
for each input indexed by i E (l,..., , N). during training we have yi xi 2; 1-2 where at is the i-th input's post-power transform value, (0, 1] is the retention rate.
Oxcerred takkes
2.0
6.
00
N (0, 1), and
R/s 0,5936
8 0.5999
30
65 24
41
330 3.5
50
15
30
30.
3.5
20
30
The second is additive Gaussian noise, but without scaling by each channel's 's minibatch statistics. This corresponds to the proposed method in the case where s is fixed to the
11.0
1.0
-00
Ordened Volues
mean of a standard half-normal distribution," i.e. 8 = v across all channels; and thus does not depend on the channel statistics.
1.0
Orskrndl
00
00
59
455
4.0 687
0.5500
82- 0.6101
R7
330
30 05
4.
30
3. 50
13
30
3.
35
05
20
Figure 7 shows that additive Gaussian noise with scaling is more effective than Gaussian dropout, giving further evi- dence for the novelty and utility of the proposed method. is also more effective than additive Gaussian noise (without scaling), which suggests the norm of the channel statistics plays an important role when using additive random noise.
Figure Representative Q Q plots of feature values for models trained to convergence with BatchNormalNorm (post-power trans- form, top row) vs, BatchNorm nost-normalization, bottom row), measured for the same validation minibatch (ResNet34/STLLIn Left to right: increasing layer number. The x-axis represents the theoretical quantiles of the normal distribution, and the y-axis the sample's s ordered values. A higher Rº value for the line of best fit signifies greater gaussianity the features. BatchNormallorm induces greater gaussianity in the features throughout the model, in comparison to BatchNorm.
64
Validation Accuracy (%)
62
60
1.0
58
Proposed Method
R² (Normality)
0.8
56-
Proposed Method w/o Scaling
Gaussian Dropout
54
0.6
0.7
0.8
0.9
0.95
0.99
Gaussian Dropout Retention Rate
BatchNormalNomm
Figure 7. Additive Gaussian noise with scaling is effective. Val- idation accuracy for models trained with BateaNoommlNNomm (ResNe(34STTLI0) but with varying forms for the noise com- ponent of the normalization layer.
0.4
BatchNorm
Layer Number
Figure 6. The average R2 values for each model layer, derived from several Q-Q plots (see Figure 5) corresponding to 20 channel and 10 validation minibatch combinations, for models trained to convergence with BatchNormalNorm vs. BatchNorm. The plot demonstrates that normality normalization leads to greater gaussianity throughout the model layers.
One reason why additive Gaussian noise with scaling may work better than Gaussian dropout, is because the latter scales activations multiplicatively. which means the effect of the noise is incorporated the backpropagated errors. In contrast, the proposed method's noise component does not contribute the gradient updates directly, because is additive. This would suggest that models trained with normality normalization obtain higher generalization perfor- mance, because they must become robust to misattribution of gradient values during backpropagation, relative to the corrupted activation values during the forward pass.
trained convergence. The plots demonstrate that normal- ity normalization leads to greater gaussianity throughout the model layers.
5.6. Comparison of Additive Gaussian Noise With Scaling and Gaussian Dropout
5.7. Effect of Degree of Gaussianization
Here we contrast the proposed method of additive Gaussian noise with scaling described in Subsection 4.2, with two other noise-bases based techniques.
Here we consider what effect differing degrees of gaus- sianization have on model performance, as measured by the
Thhis value for precisely mirrors how it is calculated in Algo- rithm I, since recall = + E. las - T/.
The first is Gaussian dropout (Srivastava et al., 2014), where
7

---

On the Importance of Gaussianizing Representations
proximity of the estimate À to the Newton-Raphson solution, which given by Equation 5.
RNIRCIFARIO
RNS6/TnnlmmageNe
Validasion Accursy (14)
01.5
Validacion Accurazy
1430
We control the proximity to the Newton-Raphsee solution
09.5
using a parameter a E [0, 1] in the following equation
39.0
68
(h; = L" ("; (h;>= 1)
88.5
À = 1
INN
UAN w/o INN
INN Normalization Layer
Nommliiatiion layer
RN50callech101
RNS0Food101
where corresponds to the Newton-Raphse solu- tion, and decreasing values of a reduce the strength of the gaussianization.
Valicatioo Accuracy (%)
Valication Accunscy (%k
Figure 8 demonstrates that the method's performance in-
22
6
creases with increasing a, and obtains its best performance for a 1. This provides further evidence that increasing gaussianity improves model performance.
BN
BNN wio mohic BNN
BNN wil-mobi BNN
Normalizaioon
Normalization Layer
Figure 9. Controlling for the effects of the power transform and the additive Gaussian noise with scaling components. Each subplot shows the performance of models trained with BatchNor- malNorm with the use of additive Gaussian noise with scaling (BNN), and without (BNN w/o noise), while using BatchNorm (BN) as baseline. Subplot titles indicate model and dataset combination.
Validation Accuracy (%)
73.5
73.0
noise robustness framework presented in Motivation Sub- section 2. 1, and consequently of the benefit gaussianizing representations.
72.5
0.1
0.2
1.0
(Gaussianization)
Speed benchmarks. Appendix A.2 shows that normality normalization increases runtime; with larger deviation training time than at test time.
Figure 8. Increasing gaussianity improves model performance. Validation accuracy for models trained using BatchNormalNorm without noise (ResNet50(Caltcch00), and with varying strengths for the gaussianization (parameterized by a) when applying the power transform.
Normality normalization uniquely maintains gaussianity throughout training. Appendix A.3 provides graphi- cal illustration of the fact that at initialization, layer pre- activations are close to Gaussian regardless of the normaliza- tion layer employed; thus only models trained with normal- ity normalization maintain gaussianity throughout training.
5.8. Controlling for the Power Transform and Additive Noise Components
Figure demonstrates that both components of normality normalization the power transform, and the additive Gaus- sian noise with scaling each contribute meaningfully the increase in performance for models trained with normality normalization.
Normality normalization induces greater feature inde- pendence. Appendix A.4 demonstrates that normality nor- malization imbues models with greater joint normality and greater independence between channel features, throughout the layers of model. This is of value in context of the benefit feature independence thought to provide, which was explored in Motivation Subsection 2.3.
5.9. Additional Experiments & Analysis
We next describe several additional experiments and analy- ses which serve to further demonstrate the effectiveness of normality normalization, and to substantiate the applicabil- ity of the motivation we presented Section 2.
6. Related Work & Future Directions
Power Transforms Various power transforms have been developed (Box & Cox, 1964; Yeo & Johnson, 2000) and their properties studied (Hernandez & Johnson, 1980), for increasing normality in data. Box & Cox (1964) defined power transform which convex in its parameter, but only defined for positive variables. Yeo Johnson (2000)
Normality normalization induces robustness to noise at test time. Appendix A.1 demonstrates that models trained using normality normalization are more robust to random noise at test time. This substantiates the applicability of the
8

---

On the Importance of Gaussianizing Representations
presented an alternative power transform which was further- more defined for the entire real line, preserved the convexity property with respect to its parameter for positive input val- ues (concavity in the parameter for negative input values), and additionally addressed skewed input distributions.
neural network at initialization approximates Gaussian process. This result has been extended to the multi-layer setting by Lee et al. (2018), and Jacot et al. (2018); Lee et al. (2019) suggest the Gaussian process approximation may remain valid beyond network initialization. However, these analyses still necessitate the infinite width limit assumption.
is worth noting that many power transforms were devel- oped with the aim of improving the validity of statistical tests relying on the assumption of normality in the data. This is in contrast with the present work, which uses information-theorete motivation for gaussianizing.
Subsequent work showed that batch normalization lends itself a non-asymptotic approximation to normality throughout the layers of neural networks at initialization (Daneshmand et al., 2021). Given its gaussianizing ef- feet, layers trained with normality normalization may be amenable - throughout training - to a non-asymptotic ap- proximation to Gaussian processes. This could help further address the disparity in the analysis of neural networks in the infinite width limit, for example as in mean-field theory, with the finite width setting (Joudaki et al., 2023).
Gaussianization Alternative approaches to gaussianiza- tion, such as transformations for gaussianizing heavy-tailed distributions (Goerg, 2015). iterative gaussianization tech- niques (Chen & Gopinath, 2000; Laparra et al., 2011), and copula-based gaussianization (Nelsen, 2006), offer interest- ing directions for future work. Non-parametric techniques for gaussianizing, for example those using quantile func- tions (Gilchrist, 2000), may not be easily amenable to the deep learning setting where models are trained using back- propagation and gradient descent.
7. Conclusion
Among the methodological developments that have spurred the advent of deep learning, their success has often been attributed to their effect on the model's ability to learn and encode representations effectively, whether in the activa- tions in the weights. This can be seen, for example, by considering the importance attributed to initializing model weights suitably, or by the effect different activation func- tions have on learning dynamics.
Usage in Other Normalization Layers Works which have previously assumed normality in the pre-activations to motivate and develop their methodology, for example as seen in normalization propagation (Arpit et al., 2016), may benefit from normality normalization's explicit gaussian- izing effect. It would also be interesting explore what effect gaussianizing model weights might have, for exam- ple by using normality normalization to augment weight normalization (Salimans & Kingma, 2016).
Seldom has a prescription for precisely what distribution deep learning model should use to effectively encode its activations, and exactly how this can be achieved, been investigated. The present work addresses this - ffist by moti- vating the normal distribution as the probability distribution of choice, and subsequently by matcrializing this choice through normality normalization.
Adversarial Robustness It would be interesting to tie the present work with those suggesting robustness to fo-norm constrained adversarial perturbations increases when train- ing with Gaussian noise (Cohen et al., 2019; Salman et al., 2019). Furthermore, it has been suggested that adversarial examples and images corrupted with Gaussian noise may be related (Ford et al., 2019). This may indicate gaining robust- ness to Gaussian noise not only in the inputs, but throughout the model, can lead to greater adversarial robustness.
It is perhaps nowhere clearer what representational benefit normality normalization provides, than when considering that no additional learnabee parameters relative to existing normalization layers were introduced. This highlights and precisely controls for the effect of - the importance of encouraging models to encode their representations effec- tively.
However, gaussianizing activations and training with Gaus- sian noise, may only defense the distributional sense; exact knowledge of the weights (and consequently of the activation values), as is often assumed in the adversarial robustness setting, is not captured by the noise-based robust- ness framework, which is only concerned with distributional assumptions over the activation values. Nevertheless it does suggest that, on average, greater robustness may be attain- able.
We presented normality normalization: novel, principledly motivated, normalization layer. Our experiments and analy- sis comprehensively demonstrated the effectiveness of nor- mality normalization, in regards to its generalization per- formance on an array of widely used model and dataset combinations, its consistently strong performance across various common factors of variation such as model width, depth, and training minibatch size, its suitability for usage wherever existing normalization layers are conventionally used, and through its effect on improving model robustness to random perturbations.
Neural Networks as Gaussian Processes Neal (1996) showed that in the limit of infinite width, a single layer
9

---

On the Importance of Gaussianizing Representations
Acknowledgments
G., Dunson, D., and Dudík, M. (eds.), Proceedings of the Fourteenth International Conference Artificial In- telligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Laud- erdale, FI., USA, 11-13 Apr 201 1. PMLR.
We acknowledge the support provided by Compute On- tario (computeontario.ca ca) and the Digital Research Alliance of Canada (alliancecan.ca), and the support of the Natu- ral Sciences and Engineering Research Council of Canada (NSERC), Discovery Grant RGPIN-2021-02527/
Cohen, J., Rosenfeld, E.,, and Kolter, Z. Certified adversarial robustness via randomized smoothing. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 1310- 1320. PMLR, 09-15 Jun 2019.
Impact Statement
This work is of general interest to the machine learning and broader scientific community. There are many potential ap- plications of the work, for which endeavoring to judiciously highlight one such possible application, in place of another, would be precarious undertaking.
Cover, T. M. and Thomas, J. A. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954.
References
Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learn- ing, 2018.
Daneshmand, H., Joudaki, A., and Bach, F. Batch normal- ization orthogonalizes representations in deep random networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4896-4906. Curran Associates, Inc., 2021.
Arpit, D., Zhou, Y., Kota, B., and Govindaraju, V. Normal- ization propagation: A parametric technique for remov- ing internal covariate shift in deep networks. Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, vol- ume 48 of Proceedings of Machine Learning Research, pp. 1168- 1176, New York, New York, USA, 20-22 Jun 2016. PMLR.
Deng, J., Dong, W., Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 N.1119/CVPR.2000 20000094..8888....
Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, abs/ 1166070064500 2016.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Bishop, C. M. Training with noise is equivalent to tikhonov regularization. Neural Computation, 7(1):108-116, Jan 1995. ISSN 0899-7667. doi: 10.1162//nco.19/ 1995.7.1.108.
Bossard, I.., Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Ebner, B. and Henze, N. Tests for multivariate normality critical review with emphasis on weighted (2 (t-sttiissics. TEST, 29(4):845-89) 2020. ISSN 1 1133-0686, 1863- 8260. doi: 11210077/111779902000044444444
Box, G. E. P. and Cox, D. R. An analysis of transforma- tions. Journal of the Royal Statistical Society. Series B (Methodological), 26(2):211-25 1964. ISSN 00359246.
Ford, N., Gilmer, J., Carlini, N., and Cubuk, F. D. Adver- sarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning, 2019.
Brent, R. P. An algorithm with guaranteed convergence for finding a zero of a function. Comput. J., 14:422-425 1971.
Gilchrist, W. Statistical modelling with functile, functions. 01 2000.
Chen, S. and Gopinath, R. Gaussianization. In Leen, T., Dietterich, T., and Tresp. V. (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
Goerg, G. M. The lambert way to gaussianize heavy-tailed data with the inverse of tukey's h transformation as a spe- cial case. The Scientific World Journal, 2015(1):909231, 2015. doi: https://doi.org/10.11 15500001001110000/999333333
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Gordon,
10

---

On the Importance of Gaussianizing Representations
Guo, D., Shamai, S., and Verdu, S. Mutual information and minimum mean-square error in gaussian channels. IEFE Transactions on Information Theory, 51(4):1261-1282, 2005. doi: 10.1109/TTT.2005.8442
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA. USA, 2015.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: N0.1109/CVPR.201..522
Laparra, V., Camps- Valls, G., and Malo, J. Iterative gaus- sianization: From ica to random rotations. IEEE Trans- actions on Neural Networks, 22(4):537-549, 2011. doi:
Henze, N. and Zirkler. B. A class of invariant consistent tests for multivariate normality. Communications in Statistics Theory and Methods, 19(10))355-3617, 1990. doi:
10.1109/TNN.201 1.2106511.
001188000361092200883000000
Le, Y. and Yang. X. Tiny imagenet visual recognition chal- lenge. In CS 23/N, 2015.
Hernandez, and Johnson, R. A. The large-sample behavior of transformations to normality. Journal of the American Statistical Association, 75(772):855-861, 1980. ISSN 01621459.
Lee, J., Sohl-dickstein, J., Pennington, J., Novak, R., Schoenholz, S., and Bahri, Y. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2018
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Improving neural networks by preventing co-adaptation of feature detectors. ArXiv, abs/1207.0580, 2012.
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl- Dickstein, J., and Pennington. J. Wide neural networks of any depth evolve as linear models under gradient de- scent. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, Buch F., Fox, E., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Huang, L., Yang, D., Lang, B., and Deng. J. Decorrelated batch normalization. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 791-800, 2018.
Huang, L., Zhou, Y., Zhu, F., Liu, L., and Shao, L. Iterative normalization: Beyond standardization towards efficient whitening. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4869 4878, 2019.
Li, F.F., Andreeto, M., Ranzato, M., and Perona, P. Caltech Apr 2022.
Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
loffe, S. and Szegedy, C. Batch normalization: Accelerat- ing deep network training by reducing internal covariate shift. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference Machine Learning, vol- ume 37 of Proceedings of Machine Learning Research, pp.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Lille, France, 07-09 Jul 2015. PMLR.
Neal, R. M. Bayesian Learning for Neural Networks. Springer- Verlag, Berlin, Heidelberg, 1996. ISBN 0387947248.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Nelsen, R. B. An Introduction to Copulas (Springer Series in Statistics). Springer- Verlag,--Verlag, Berlin, Heidelberg, 2006. ISBN 0387286594.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Joudaki, A., Daneshmand, H., and Bach, F. On bridging the gap between mean field and finite width deep ran- dom multilayer perceptron with batch normalization. In Krause, A., Brunskill, E., Cho, K., Engelhardr, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15388- 15400. PMLR, 23 Jul 2023.
Pan, X., Zhan, X., Shi, J., Tang. X., and Luo, P. Switch- able whitening for deep representation learning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.

---

On the Importance of Gaussianizing Representations
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., Detito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.,, Bai, J., and Chintala, S. Pytorch: An imperative style, nigh-performance deep learning library, Advances in Neural Information Processing Systems, 32, 2019.
Processing Systems, volume 30. Curran Associates, Inc.,
2017.
Vinh, N. X., Epps. J., and Bailey, J. Information theoretic measures for clusterings comparison: Variants, proper- ties, normalization and correction for chance. Journal of Machine Learning Research, 1 1195)22887728544 2010.
Wilk, M. B. and Gnanadesikan, R. Probability plotting methods for the analysis of data. Biometrika, 55(1):1-17, 1968. ISSN 00063444, 14643510.
Pham-Gia, T. and Hung, T. The mean and median absolute deviations. Mathematical and Computer Modelling, 34 (7):921-936. 2001. ISSN 0895-7177. doi: https://doi.or// 10.1016650995--177000000 10010-9
Wu, Y. and He, K. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Pickles, A. An Introduction Likelihood Analysis, Con- cepts and techniques in modern geography. Geo Books, 1985. ISBN 0780860941903.
Yeo, I.-K. and Johnson, R. A. A new family of power transformations to improve normality or symmetry. Biometrika, 87(4) 2000. ISSN 00063444.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization accelerate training of deep neural networks. In Lee, D., Sugiyama, M., Luxburg, U.. Guyon, I., and Garnett, R. (eds.), Advances in Neu- ral Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Zagoruyko. S. and Komodakis, N. Wide residual networks. In BMVC, 2016.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In Interna- tional Conference Learning Representations, 2018.
Salman, H., Li, J., Razenshteyn, I., Zhang, P., Zhang, H., Bubeck, S., and Yang. G. Provably robust deep learning via adversarially trained smoothed classifiers. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F,, Fox, E., and Garnett, R. (eds.), Advances in Neural Informa- tion Processing Systems, volume 32. Curran Associates, Inc., 2019.
Zhou, M. and Shao, Y. A powerful test for multivariate normality. Journal of Applied Statistics, 41(2):351-36) 2014. doi: 10.008002264763.221333333 24563571.
PMID:
Shannon, C. E. mathematical theory of communication. The Bell System Technical Journal, 77(3):379-423 1948. doi: 10. 1002/j. 1531-5333000 7305. 100005..998.0011111 1948.tb01338.x.
Srivastava, N., Hinton, G. E.,, Krizhevsky, A., Sutskever, I., and Salakhutdinov, Dropout: simple way to prevent neural networks from overfitting. Mach. Learn. Res., 15:1929-1958. 2014.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. Bengio, and LeCun, Y. (eds.), 2nd International Conference on Learning Representa- tions, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Ulyanov, D., Vedaldi, A., and Lempitsky, S. Instance normalization: The missing ingredient for fast stylization. ArXiv, abwp/1/0..080000 2016.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. At- tention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances Neural Information
12

---

On the Importance of Gaussianizing Representations
A. Additional Experiments & Analysis
A.I. Noise Robustness
We use the following framework to measure a model's robustness to noise (a similar setting is used by Arora et al. (2018)). For given data point, consider a pair of units in a neural network, the first in the k-th layer and the second in the f-th layer. For the unit in the k-th layer, let æ denote the data point's post-normalization value. Let Φx,e (ar) be the same data point's post-normalization value for the unit in the subsequent layer &, where the function ox, encapsulates all the intermediate computations between the two normalization layers k and c.
Let y r + z N x E||, where as in Subsection 4.2 2 ~ N (0, 2 0. and here || - æ|| represents global estimate for the zero-centered norm of the post-normalized values, derived from the training set in its entirety. We then define noise robustness as follows:
Table Normality normalization induces robustness to noise at test time. Evaluation of robustness to noise, using the relative error S for various layers & and E, for various models trained with BatchNormalNomm and BatchNorm. Models were evaluated using noise factor = 0.5. Top: S/CNFA///CIARR1O middle: R/S/Ar18/CIFAR 8/CIFAR 100, bottom: ResNec34/STL10, The layer & at which noise is added is denoted the left side of cach row, and cach column denotes subsequent layer e. For each entry, Ç validation set, and over all channels in the k-tth and E-th layers. This was subsequently averaged across T = Monte Carlo draws for the furthermore the average across each of the M 6 trained models. In each table entry, the top
averaged over the entire
random noise, and the values presented value represents the relative error for BatchNNommlNNrm (BNN), and the bottom value for BatchNorm (BN), with the best value shown in hold; lower is better. The tables provide evidence that models trained using normality normalization generally more robust to random noise at test time.
L5
L,
L13
L17
BNN BN
0.044 + 0.002
0.076
0.002
0.124 1 0.002
0.211 + 0.005
LI
0.160 + 0.009
0.266
0.014
0.390 = 0.016
1.002 = 0.032
0.031
0.001
0.049 1 0.001
0.080 0.002
L5
0.060 = 0.002
0.093 + 0.004
0.201 + 0.005
0.048 + 0.001
0.070 + 0.001
1.9
0.060 + 0.003
0.117 - 0.003
0.142 + 0.006
L13
0.203 = 0.005
L5
L9
L13
L17
BNN
0.051 + 0.001
0.076
0.001
0.100 + 0.001
0.387 0.004
L1
BN
0.177 + 0.007
0.333
0.011
0.419 + 0.021
1.922 + 0.076
0.027
0.002
0.038 + 0.003
0.149 + 0.012
L5
0.059 = 0.009
0.073 + 0.009
0.373 - 0.041
0.044 + 0.001
0.151 + 0.002
L9
0.063 + 0.003
0.249 + 0.009
0.257 + 0.001
L13
0.367 + 0.020
L9
L17
L25
L33
BNN
0.373 + 0.018
0.709 0.034
0.452 + 0.044
0.565 + 0.053
L1
BN
0.615 + 0.046
1.280 + 0.080
1.900 + 0.537
2.621 + 0.257
0.141 + 0.004
0.080 + 0.003
0.099 + 0.007
L9
0.120 = 0.005
0.099 - 0.011
0.307 = 0.013
0.102 + 0.006
0.121 = 0.011
L17
0.104 + 0.006
0.324 + 0.012
0.051 + 0.006
L.25
0. - 0.006
13

---

On the Importance of Gaussianizing Representations
Definition A.1 (Noise Robustness). For given realization of the noise sample 2, let CR.e (2, robuustesss be defined as
Ch.e (r,y) = 1 (x) tx (1)ll
1 (2)]l
Thus <k,e (z,3) y) measures the relative discrepancy between ou,t (x) and on,E e (v) when noise factor å is used, and effectively represents the noise's 's attenuation from layer layer l. Averaging Si.e (x, y) over all data points, and over all units the k-th and E-th layers, leads consolidated estimate of the noise robustness.
Table 4 demonstrates the increased robustness to noise ob- tained when using BatchNormalNomm in comparison to BatchNorm. This substantiates the applicability of the noise robustness framework presented in Motivation Subsection 2.1, and consequently of the benefit of gaussianizing repre- sentations in normality normalization.
RNJSECIFARIO
RNISICIFARIO
Spood (ma)
Spred (1n3)
012
A.2. Speed Benchmarks
BNN
00
BN
BNN
eval
Figure 10 shows the average per-sample running time for models using BatchNoomalNNrmm and BatchNorm. The val- calculated by taking the average minibatch runtime train/vvaluation time, for the entire training/validation set, then normalizing by the number of samples in the minibatch. Values are obtained using an NVIDIA V100 GPU. For the purposes of these benchmarks, synchronization between the CPU and GPU was enforced.
RNBISTLIU
RNBESTTLIU
Sperd (m))
Spoed (mu)
68
BN
BNN
00
BN BNN
train
Figure 10. Runtime comparison between models using Batch- NormalNorm (BNN) and BatchNorm (BN) for two sets model & dataset combinations; top: ResNet18/CIFARI0 bottom: ResNet34/STL.10, The left hand plot shows the rumning time during training. and the right hand plot shows the running time during evaluation. See text for details.
The plots show that normality normalization increases run- time; with larger deviation training time than time. It is worth noting however, that the present work serves foundation, both conceptual and methodological, for future works which may continue to leverage the benefits of gaus- sianizing. We helieve improvements to the runtime normality normalization can be obtained by leveraging approximations to the operations performed in the present form of normality normalization; for example the operations log (1 + h)) and raising to the power.
30.
17.0
1.0
2.3
Crocrea yauus
0.0
0.0
Ordered
:1.5
2.0
2² (Normality)
0.8
# 0.9711
4.0
A² 0.9965
370
8P 0,9350
4.0
223
80
2.
4.
33.
-1.6 30
JSS.
30
300
-1.5 06 35 30
Theoretical Quantiies
Theoretical Quantiies
Theorocical Quantiles
0.6
2.0
Ordardd values
1.5
BatchNormalNoom
5.
0.0
0.0
0.4
BatchNorm
3.0
11.0
17
25
33
-107
22.
33 Theoretical Quantties
Layer Number
Theerstical Quantiees
Theoretical Quanties
Figure 11. As in Figure 5 and Figure 6, but for networks at initialization. The plots provide graphical illustration of the fact that initialization, networks using either NorcNNormlNoorm or BatchNorm have close-to Gaussian pre-activations. However, as the networks traind,, BatchNormalNomm enforces and maintains normality while BatchNorm does not, evidenced by Figure 6.
14

---

On the Importance of Gaussianizing Representations
A.3. Normality at Initialization
Figure 11 shows representative Q-Q plots, together with an aggregate measure of normality across model layers, for post-power transform feature values when using BatchNormalNorm, and post-normalization values when using BatchNorm, for models at initialization. It provides graphical illustration of the fact that initialization, the pre-activations are close to Gaussian regardless of the normalization layer employed; and thus that only the model trained with BatchNormalNorm enforces and maintains normality throughout training, as evidenced by Figure 6. Note that the Q-Q plots presented in Figure 5 and Figure 11 are obtained for the same corresponding minibatch and channel combinations.
A.4. Joint Normality and Independence Between Features
Following the motivation we presented in Subsection 2.3, here we explore the potential effect normality normalization may have on increasing joint normality in the features, and the extent to which it may increase the independence between features.
We use the following experimental setup. For each layer of ResNec34/STLI0 model trained convergence using either BatchNormalNorm or BatchNorm, we compute the correlation, joint normality, and mutual information over 10 pairs of channels, and across 10 validation minibatches.
We evaluate joint normality using the negative of the HZ-statistic (Henze & Zirkler, 1990) (higher values indicate greater joint normality), and evaluate independence using the adjusted mutual information (AMI) metric (Vinh et al., 2010) (lower values indicate a greater degree of independence).
We evaluate joint normality across pairs of channels rather than across all of the channels a layer, because measures of joint normality are sensitive to small deviations in sample statistics for finite sample sizes (Zhou & Shao, 2014; Ebner & Henze, 2020). Wherever measure AMI, we use the square root of the number of sampled features the number of bins (a generally accepted rule of thumb) when discretizing the features, and we use uniform binning, which is appropriate for (close to) normally distributed data.
Figure 12 demonstrates that models trained with BatchNormalNorm have higher joint normality, and have greater indepen- dence, across the model's 's layers. This of value context of the benefit feature independence is thought provide, which was explored in Motivation Subsection 2.3.
Correlation Coeficieet
1.0-
MMI
RuchNNommalNorm BatchNorm
Vtw W WW
Adjusted Munual Information
Mm
Between Channels
0.8-
Joint Normality
-250
03
MI
-500-
0.6
-750
-1000
0.1
-1250
00:
33
17
25
17
25
33
Layer Number
Layer Number
Layer Number
Figure 12. Normality normalization induces greater feature independence. Correlation, joint normality, and adjusted mutual informa- tion between pairs of channels for models trained to convergence using BatchNormalNorm vs. BatchNorm (ResNet34/STLL10), The results are obtained by averaging the corresponding statistics across 10 channel pairs, and across 10 validation minibatches. Here joint normality quantified using the negative of the HZ-statistic.
B. Lemmas
Lemma B.1. Bivariate Normality Minimizes Mutual Information. Let X1 N (21;,, and X2 ~ N (12 u2; oz). Their mutual information (X1; (2) is minimized when the random variables are furthermore jointly normally distributed. f.e. X2) N (x: pr. E), with X2.
[1]
[1
of
12
14
I22
Σ
00
of
and p the correlation coefficient between
1he AMI is variation of mutual information, which adjusts for random chance. It is also bounded between 0 and which makes casier to interpret.
15

---

On the Importance of Gaussianizing Representations
Proof. Consider two possible distributions, S for the joint distribution over (X1, X2), where / denotes the probability density function (PDF) of the bivariate normal distribution, and g can be any joint distribution. Our goal is show that the mutual information between X2, when they are distributed according to g, is lower-bounded by the mutual information between X1,X² when they are distributed according to f.
For clarity of presentation, let the number of variables f and take as arguments be clear from context, so that it is understood when they are used to denote their marginal distributions. Furthermore let lg (X1; X2) represent the mutual information when (X1, X2) are distributed according g. with the notation extending analogously to their joint hy (X1, and marginal hg (X1), hg (X2) entropies under g.
We then have
1g (X1: = hg (X1) hg (X2) - hg (X1,
= by (X1) + t, (X2) h1 (X1, X2)
2 by (X1) + thy (X2) - hy (X1, X2)
I (X1;X9)
(8)
1
log (2TTed+)
+ ž log (2meeÿ)
1
1
log (2Tee) (1-p²)
2
1
log 1 p²
2
where the second equality follows because by assumption the marginals are normally distributed, the inequality follows because the normal distribution maximizes entropy, and in the second-last equality we have used the expressions for the entropies of the univariate and bivariate normal distributions.
Consequently, when the random variables are jointly normally distributed, implies 7 (X1;; X2) = 0; thus uncorrelated- ness implics independence."
C. Derivation of the Power Transform Negative Log-Likelihood
As in Section 3, consider a random variable H from which a sample = } is obtained. Recall that the power transform gaussianizes h by applying Equation 3 for each h,, where the parameter À is obtained using maximum likelihood estimation, so that the transformed variable is normally distributed possible.
Denote the transformed random variable as X N' (1;14,²)) where v (h; A), and p- (A), o² = o² (A); i.e. and 2 are in general functions of X. Obtaining the maximum likelihood estimates for (n,,², A), shown next, requires evaluating the probability density function (PDF) of H, given the transformed variable X is normally distributed.
Evaluating the cumulative distribution function (CDF) of H gives!
Fn (h) = P(H h)
=P P (11 (X) 1/2 -15h) -1s))
= x (1++))-1))
Fx (1 -
l'eee preceding result extends straightforwardly to the general multivariate setting, i.c. with more than two random variables. 1ooo simplify the presentation of the derivation, we omit the cases where À and À 2, and outline the NI.I for 0 only, as the case for < follows closely by symmetry.
16

---

On the Importance of Gaussianizing Representations
Differentiating Equation gives the PDF of H:
fa (h: (2)))))))))))))))))))))) o2) (2) = dh
Fи (h)
xx ((L+ 111 (2)
d
1
1 erf
dh
2
o(A) v2
(10)
=1++
-1
(1 + h)
1 ((())
exp
2 (A)
20 (A)
A
= =1+ fx (
The negative log-likelihood (NLL) of the sample according to this distribution is
C (h; (A), (A),
N
log
fu (he (2)
N
1=1
1
log
(1 ha)²-1
exp
-1
(xt 1 (2)))))
(1I)
N
22TO² (X)
20 (A)
i=1
1
log (2iT)
log (o² (2)) + 2N (A)
1
Σ
N
(x. - 11 (A))²
A-1 Σ
log (1 hs)
2
Optimizing the NLL w.r.t. / (A) and o² (A) gives
1 N
(A) = Σ Et.
N
t=1
(12)
1
8² (A) (A)
Σ
(a:
(=:-A(1 (2)))
N
Finally re-writing the NLL using the expressions for (A) and 6² (A), obtain the profile NLL (Pickles, 1985):
C (2 - (log (2T) + 1) 2 log (6² (A)) x
Σ log (1 + hz).
(13)
D. Series Expansion of the Power Transform Loss
Let L2 (h; (X, A = 1)) denote the second-order series expansion of the power transform's 's negative log-likelihood (NLL) centered at Àg= = 1, i.e.
(A 2
C2 (h: (2, = 1)) =
c"(h; A =1)
(14)
17

---

On the Importance of Gaussianizing Representations
We ha
[h; X 1) (1x
A=1
1
+ 1) + 1 log (o2 (A 1))
log (2tT
2
2
(' 1) 0C (hi 2)
ax
A=1
062 (2)
1 N
(15)
Σ
log (1 + h;),
20 (X = 1)
01
A=1
N
ch; A = 1)= = 8 L (h;
0x2
A=1
-1
1062 (A)
80 (A)
+
2 2 (A 11=1))²
0x
1=1
20 (A 1)
012 A=1
where
063 (A)
2
Σ
(vb (1h:) A) (
(ou (he;
(D0(1:>) A)
0ā (1)
)}
(16)
N
2x
-=1
062 (A)
N
(Eb (hx;:
ap (A)
0x 1==1 { )
Σ
(hs - (A= (a2=1)] 1))
(17)
0x
A=1
N
0x
A=1
with
Oub (h4: A)
(1 + hi) (log (1 +1+ h+))
h4s
0%
A=1
0p (A)
N
Oui (hig 2)
(18)
ax
1=1
N
Ox
A=1
1=1
and
0802 (A)
2
NN.
( - (2)) (
(8ib (h:;
Pi (A)
Σ
012
N
OX2
Ox2
i=1
(19)
Oe (h;;
0n (A)
27
+
02
2X
0200 (2)
2
Σ
(
10=11)) (hs (hi-- i (A==1()) 1))
O (h1; A)
(a(A))
0x2
A=1
N
OX2
x=1
Ox2
A=1
i=1
(20)
Or Or (h;;
ap (A)
)
+
(x=1)
OX
A==1
0x
with
@&:b (h;
anp (hss 2) Ox
= (11 h4) (log (1 + h.)))
OA=
A=1
A=1
(21)
0 (A)
1 Sabi (hi A)
ax2
A=1
ax2
A=1
Furthermore, because the power transform is applied after the normalization step (see main text), áa 1) 0 and 8² (X=1)=1
E. Evaluation of Á Estimates
Figure 13 provides representative examples substantiating the similarity between the negative log-likelihood (NLL) and its second-order series expansion around Å 1. The figure furthermore demonstrates the accuracy of obtaining estimates of using one step of the Newton-Raphson method.
Ioo simplify the presentation, we outline the series expansion for 2 0 only, as h < follows closely by symmetry.
18

---

On the Importance of Gaussianizing Representations
NLL NI1 SF
1.5
3.7
3.6
1.5
1.5
1.4
1.1
0.2
0.6
1.4
1.8
0.2
0.6
1.0
1.8
0.2
0.6
1.0
1.8
1.0
1.8
1.7
121
3.6
1.6
1.5
0.6
3.8
2.1
1.7
3.6
1.7
1.5
1.5
1.3
1.4
1.3
0.2
0.5
3.4
1.8
0.2
0.8
1.0
3.8
0.6
3.0
3.8
Figure 13. Normality normalization estimates for Å for given training minibatch (/sNNet1/CIFARRO)) Left right: increasing layer number. Top to bottom: esti- mates from various channels. Normality normalization's 's quadratic series expansion the loss (NLL SE) closely approximates the original loss (NLL), leading to accurate estimates of Ả (marked by x).
F. Training Details
F.I ResNet and WideResNet Experiments
The training configuration of the model and dataset combinations which use batch normality normalization (BatchNor- malNorm/BNN), batch normalization (BatchNorm/BN), instance normality normalization (InttanceNomallNoo///NN))) instance normalization (InstanceNorm/IN), group normality normalization (GroppNormaNoomGGNNY) group normalization (GroupNorm/GN), decorrelated batch normality normalization (DBNN), and decorrelated batch normalization (DBN), are follows.
We used variety of residual network (ResNet) (He et al., 2016) and wide residual network (WideResNet) (Zagoruyko & Komodakis, 2016) architectures in our experiments. For all of the experiments except those using the TinylmageNet (TinyIN), Caltech101, and Food101 datasets, models were trained from random initialization for 200 epochs, with a factor of 10 reduction learning rate each 60-epoch interval. For the experiments using the TinylmageNet (TinyIN), 1altech101, and Food101 datasets, models were trained from random initialization learning rate epochs 40, 70, 90. A group size of 32 was used in all of the relevant group normalization experiments. For the Caltech100 dataset, each used random 90/10% allocation to obtain the training and validation splits respectively. Each such run used its own unique random seed to generate the splits for that run, which facilitates greater precision in the reporting of our aggregate results across the runs.
100 epochs, with factor of 10 reduction in
In all of our experiments involving the ResNet18, ResNet34, and WideResNet architectures, stochastic gradient descent (SGD) with learning rate 0.1, weight decay 5 10 10-4, momentum 0.9, and minibatch size 128 was used. In the experiments involving the ResNet50 architecture on the Caltech 101 and Food101 datasets, SGD with learning rate 0.0125, weight decay 1 x 10-4 momentum 0.9, and minibatch size 32 was used. noise factor of 0.4, used, preliminary experiments
Theee official 1altech101 dataset does not come with its own training/validation split.
19

---

On the Importance of Gaussianizing Representations
demonstrated increases typically resulted in training instability. We also investigated several hyperparameter configurations, including for the learning rate, learning rate scheduler, weight decay, and minibatch size, across all the models and found the present configurations to generally work best across all of them.
E.2. Vision Transformer Experiments
The training configuration of the model and dataset combinations which use layer normality normalization (LayerNormal- Norm/LNN) and layer normalization (LayerNorm/LNN) are as follows. We used a vision transformer (Vaswani et al., 2017; Dosovitskiy et al., 2021) model consisting of 8 transformer layers, 8 attention heads, hidden dimension size of 768, and multi-layer perceptron (MLP) dimension size of 2304. A patch size of 4 was used throughout, except in the Food101 and ImageNet experiments, where it was set to 16.
For all of the experiments except those using the SVHN dataset, learning rate warm-up strategy was employed, where the learning rate was linearly increased from 0-1 fraction of its value, the full learning rate. After the warm-up phase, eyclic learning rate schedule based on cosine annealing with periodic restarts was employed (Loshchilov & Hutter, 2017), with 50 iterations until the first restart, factor of 2 for increasing the number of epochs between two warm restarts, and a minimal admissible learning rate of 1 x 10 Models were trained using minibatch size of 128, and for 900 epochs on the CIFAR1O, CIFAR 100, Food101 datasets, and for 200 epochs on the ImageNet dataset. For the ImageNet experiments, we applied weighted random sampling to sample training examples based on the training set's corresponding inverse class frequency for the data point; we found this to help across all the model configurations used. For the experiments involving the SVHN dataset, models were trained from random initialization for 200 epochs, with a factor of 10 reduction in learning rate at each 60-epoch interval, and minibatch size of 32.
The AdamW optimizer (Kingma& & Ba, 2015; Loshchilov & Hutter, 2019) with learning rate 1 x 10-3, weight decay 5 x 10-2, (31, = (0.9, 0.999). = 1 x 10-8 was used. A noise factor of & = 1.0 was used, as preliminary experiments demonstrated increases typically resulted in training instability. We also investigated several hyperparameter configurations, including for the learning rate, learning rate scheduler, weight decay, and minibatch size, across all the models and found the present configurations to generally work best across all of them.
The data augmentations used for the models presented in Table 1 are as follows. For the models trained on the SVHN dataset, mild random translations and rotations were used. For the models trained on the CIFAR1O and CIFARION datasets, random cropping, random horizontal flips, mild color jitters, and mixup (Zhang et al., 2018) were used. For the models trained on the Food10l dataset, random cropping with resizing, random horizontal flips, moderate color jitters, and mixup were used. For the models trained on the ImageNet dataset, random cropping with resizing, random horizontal flips, and moderate color jitters were used.
F.3. Datasets and Frameworks
The datasets we used were CIFARIO, CIFARI00 (Krizhevsky, 2009), STL10 (Coates et al., 2011), SVHN (Netzer et al., 2011), 1), Caltech101 101 (Li et al., 2022), TinylmageNer (Le Yang, 2015), Food101 (Bossard al., 2014), and ImageNet (Deng al., 2009). We trained our models using the PyTorch (Paszke al., 2019) machine learning framework.
20


Reviews:
[{"review_id": "ZUYv3imK1M", "reviewer": "Reviewer_Eseb", "summary": "The paper presents a novel approach to improving the feature representations in deep neural networks by encouraging normality in activations. The authors introduce Normality Normalization (NormalNorm), a normalization technique based on the power transform to Gaussianize feature distributions and enhance robustness through additive Gaussian noise during training. The paper argues that the normal distribution is optimal for encoding information in neural networks, improving generalization and robustness. Extensive experiments demonstrate the superiority of NormalNorm over traditional normalization techniques like Batch Normalization (BatchNorm), Layer Normalization (LayerNorm), Group Normalization (GroupNorm), and Instance Normalization (InstanceNorm) across multiple model architectures and datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Eseb,\n\nWe address all of your comments below.\n>\n>\"While Normality Normalization improves robustness to random noise, its effectiveness against adversarial perturbations is not fully examined.\"\n>\nand\n>\n>\"Adversarial Robustness Claims: The paper suggests that Normality Normalization improves robustness to random noise, which could imply improved adversarial robustness. Have you tested Normality Normalization against adversarial perturbations (e.g., FGSM, PGD)? If not, do you expect that the method will improve adversarial robustness, and why?\"\n>\nWe would very kindly like to point out that we invoked adversarial robustness only as it pertains to deep neural networks in general being susceptible to perturbations; we did not explicitly claim robustness to adversarial perturbations in the paper. However, in Section 6 Related Work & Future Directions we provide a line of reasoning which suggests greater adversarial robustness may be attainable, given the connection between robustness to random perturbations and adversarial perturbations. Thus we do expect that on average, greater adversarial robustness should be attainable.\n>\n>Regarding the derivation and justification of Equation 2: the NLL.\n>\nIt is great that you inquire about this. We originally decided to defer the derivation for the NLL as it can be found in the literature, for example an outline is provided in (Yeo & Johnson 2000), and a sketch is provided in (Hernandez 1979) for the (related) Box-Cox power transform (https://drive.google.com/file/d/1__hvD4GgwSA3aj2OK9eVnlZg9JOmpSMs/view). We have now included the derivation in the appendix of the paper, and we give the idea here: Begin by taking a random variable $H$ (which can be arbitrarily distributed) and apply the power transform to it (or in practice, a data sample taken from $H$) to obtain $X$. We want $X$ to be as normally distributed as possible, which means we want to maximize the likelihood of $X$ under the Gaussian. This is given by taking the log of the Gaussian PDF for $X$ and maximizing it (this is equivalent to minimizing the negative log-likelihood (NLL)). That is, if you take the (negative) log of the PDF of a Gaussian random variable $X$, then substitute for $x$ the power transform as a function of $h$ (with correct consideration for change of variables), what you will obtain is precisely the NLL we have in our paper.\n>\n>Regarding the choice of ViT architecture.\n>\nWe chose to use a somewhat smaller-scale ViT architecture to enable our extensive experiments on several datasets, and to enable high precision in the reporting of our experimental results through the multiple random seeds. In fact, we were able to obtain $M=6$ total seeds for the ImageNet experiments post-submission, for each of LNN and LN. The updated results for ImageNet, across these $M=6$ seeds, are:\n|Dataset|LN|LNN|\n|----------|----------|----------|\n|ImageNet Top1|71.54 $\\pm$ 0.16|**75.25 $\\pm$ 0.07**|\n|ImageNet Top5|89.40 $\\pm$ 0.11|**92.23 $\\pm$ 0.04**|\n\nThese enable even greater confidence in our experimental results on ImageNet.\n\nTo further address your inquiry, we ran experiments with the additional use of mixup (Zhang et al. 2017) for several of the model & dataset combinations listed in Table 1 (with the experimental setup otherwise identical to that listed in Appendix E.2).\n|Dataset|LN|LNN|\n|----------|----------|----------|\n|CIFAR10|89.97 $\\pm$ 0.16|**91.18 $\\pm$ 0.13**|\n|CIFAR100|66.40 $\\pm$ 0.42|**70.12 $\\pm$ 0.22**|\n|Food101|73.25 $\\pm$ 0.19|**79.11 $\\pm$ 0.09**|\n\nThese results provide strong evidence that models trained with normality normalization continue to improve with the use of additional techniques for improving generalization performance, and that they continue to outperform models trained with other normalization layers. This also demonstrates that the network size is not an obstacle.\n\nFinally, we found that for both small and large width networks, and for small and large depth networks, normality normalization outperforms competing normalization layers. In Section 4.4 Effectiveness Across Model Configurations, in paragraphs Network Width and Network Depth and through Figures 2 & 3 respectively, we provide experimental evidence demonstrating this. We also found this trend to hold true in experiments with various ViT architectures, and we ultimately used the chosen architecture in the paper to enable extensive experiments and with multiple random seeds, as mentioned.\n\nWe have furthermore added experimental results contrasting decorrelated batch normalization (DBN) with decorrelated batch normality normalization (DBNN); please see the thread with Reviewer 7TPa. These experimental results provide further evidence for the strong performance of normality normalization across various normalization layers.\n\nWe believe we have comprehensively addressed your comments here. We would be highly appreciative if you would consider increasing the score for our submission; thank you.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense for the problem or application", "theoretical_claims": "I did not check the correctness of any proofs for theoretical claims", "experimental_designs_or_analyses": "I check the soundness/validity of any experimental designs or analyses", "relation_to_broader_scientific_literature": "The key contributions of the paper are not related to the broader scientific literature", "essential_references_not_discussed": "I am not familar with this area and unsure whether there are more essential literature should be cited", "other_strengths_and_weaknesses": "**Strengths**:\n1. Theoretical Justification: The authors provide a solid information-theoretic foundation for why Gaussianity in activations is beneficial. They reference the mutual information game framework to argue that normal distributions maximize information transmission and robustness.\n2. Methodological Novelty: The proposed Normality Normalization combines the power transform for Gaussianization with additive Gaussian noise with scaling, a distinct approach compared to conventional normalization methods (BatchNorm, LayerNorm, etc.).\n3. Comprehensive Empirical Validation: The method is tested on multiple architectures (ResNets, Vision Transformers, WideResNets). Evaluations span diverse datasets, including CIFAR-10, CIFAR-100, SVHN, TinyImageNet, and ImageNet. Experiments consider factors such as network width, depth, and batch size, demonstrating that the method generalizes well.\n4: Robustness & Generalization Benefits: The paper shows that Normality Normalization enhances test-time robustness to noise. It improves model generalization, often outperforming conventional normalization techniques. Feature representations exhibit greater independence, an attractive property for reducing redundancy.\n5. Strong Quantitative Support: Statistical analysis using Q-Q plots demonstrates that activations become more Gaussianized. The impact of the power transform and Gaussian noise is separately analyzed to isolate their contributions.\n\n**Weaknesses**:\n1. Computational Overhead: The power transform involves estimating a transformation parameter ($\\lambda$) per feature channel, which increases computational complexity. Although the Newton-Raphson method approximates $\\lambda$ efficiently, the added computations may slow down training, as evidenced in the runtime benchmarks.\n2. Lack of Analysis on Adversarial Robustness: While Normality Normalization improves robustness to random noise, its effectiveness against adversarial perturbations is not fully examined. Given prior studies linking Gaussian robustness to adversarial defense, further testing in this area would be valuable.", "other_comments_or_suggestions": "1. To enhance clarity, I recommend adding an explanation for the derivation of Equation (2). Specifically, it would be helpful to outline the reasoning behind this objective function and why it serves as the appropriate optimization target. Providing an intuitive justification—such as its connection to maximizing the Gaussianity of transformed activations or minimizing divergence from a normal distribution—would strengthen the reader’s understanding of its significance within the broader framework of Normality Normalization.", "questions_for_authors": "1.  **Justification of the Optimization Objective in Equation (2)**: Could you provide a more detailed explanation for why Equation (2) is the appropriate objective function? Specifically: What is the intuition behind minimizing this negative log-likelihood (NLL) in the context of Normality Normalization? Does this directly encourage activations to follow a normal distribution, or is there an implicit assumption about the data distribution?\n2. **Adversarial Robustness Claims**: The paper suggests that Normality Normalization improves robustness to random noise, which could imply improved adversarial robustness.  Have you tested Normality Normalization against adversarial perturbations (e.g., FGSM, PGD)? If not, do you expect that the method will improve adversarial robustness, and why?\n3. **Potential Issue with Baseline Model Choice**: In Table 1, the reported performance of ViT on ImageNet appears significantly lower than that of the standard ViT model, likely due to the smaller model depth and reduced number of attention heads. The improvement of Layer Normality Normalization (LNN) over Layer Normalization (LN) is observed on this smaller ViT model. Why did the authors choose this particular ViT architecture instead of using the standard ViT model configurations commonly used for ImageNet? Have the authors evaluated whether the performance gap between LNN and LN remains consistent for larger ViT models (e.g., ViT-B/16, ViT-L/16)? If the improvement diminishes on larger models, does this suggest that the benefits of Normality Normalization are more pronounced in smaller-scale networks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3MeIx26oMr", "reviewer": "Reviewer_zhYV", "summary": "The paper proposes a normality normalization that enforces Gaussian feature distribution using a power transform and additive Gaussian noise. The motivation for using the normal distribution is to enhance the model's robustness to random perturbations, improving generalization.\n\n## update after rebuttal\n\nThe authors have kindly pointed out the parts of their work which I missed or misunderstood.\nHowever, additional clarifications regarding the information-theoretical content of their paper exposed the lack of strong connection between the method and the information-theoretical framework employed. In the manuscript, it is claimed that\n\n> The normal distribution plays a central role in information theory – it is at the same time the best-case signal and worst-case noise distribution,\n\n> the mutual information game suggests gaining robustness to Gaussian noise is optimal because it is the worst case noise distribution\n\nHowever, Gaussian noise is only the worst-case noise for maximizing $I(X;X+Z)$ if we restrict the second moments of $X$; for other constraints (like restricted support, mean absolute value, etc.), the worst-case noise and best-case distributions **are not Gaussian**.\nThus, robustness to AGN is not a generally desirable outcome: information theory suggests that there may be other cases, in which robustness to, e.g., uniform noise, should yield better results.\n\nTherefore, I insist on additional theoretical analysis (and ablation on noise distribution) being conducted. Otherwise, the work in its current state is not well-supported by the information theory (as constraints imposed and the corresponding \"worst-case noise\" seem arbitrary). For more information (and additional concerns), please refer to my final reply.\n\nAs a result, I decided to keep my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zhYV,\n\nWe address all of your comments below.\n>\n>\"The Q-Q plots and $R^2$ metrics do not serve as a proper multivariate Gaussianization metric. Perhaps, special statistical tests should be employed, e.g., the Henze-Zirkle test\"\n>\nWe very kindly note that we in fact already did precisely this in the paper; using the Henze-Zirkle (HZ) test statistic as well. Please see Section A.7. Joint Normality and Independence Between Features, which is referenced in the main text in Section 4.6 under paragraph Normality normalization induces greater feature independence. This precisely addresses your inquiry.\n>\n>Regarding a comparison to an alternative normalization method.\n>\nPlease see the experimental results comparing decorrelated BatchNorm (DBN) (which is also closely related to iterative normalization) with decorrelated BatchNormalNorm (DBNN) in the thread with Reviewer 7TPa, which precisely addresses your inquiry here.\n>\n>\"Robustness to adversarial as well as random perturbations (\"improving model robustness to random perturbations\" from lines 42-43) — are not convincingly substantiated.\"\n>\nWe very kindly point out that we did in fact substantiate the claims we made through the experiments in Section A.3 Noise Robustness, which we also referenced in the main text of the paper in Section 4.6 under paragraph Normality normalization induces robustness to noise at test time. Furthermore, we had only mentioned adversarial robustness as it pertains to deep neural networks in general being susceptible to perturbations; we did not explicitly claim robustness to adversarial perturbations in the paper. However, in Section 6 we provide a line of reasoning which suggests greater adversarial robustness may be attainable, given the connection between robustness to random perturbations and adversarial perturbations.\n>\n>Regarding \"justify the claim of “wherever existing normalization layers are used”?\".\n>\nWe would very kindly like to point out that we indeed investigated normality normalization across several normalization layers, as evidenced by Subsection 4.3 and Figure 1, where we also compared it with InstanceNorm and GroupNorm.\n>\n>Regarding exploration of non-CV tasks.\n>\nIn terms of application areas, our intent was to be extremely comprehensive in our experiments in a domain of choice. Because we evaluate across several normalization layers, and in general extensively demonstrate the effectiveness of normality normalization, we believe this is a very strong and reliable indicator for the method's success translating to other (non-CV) domains.\n>\n>Regarding the series expansion of the NLL.\n>\nThe power transform we use specifically addresses skewed data distributions - please see (Yeo & Johnson 2000) for a detailed investigation. Furthermore, we have indeed assessed the ability for normality normalization to achieve a high degree of gaussianity on complex datasets, as evidenced by Figures 5 & 6, and our work demonstrates that we can achieve better performance by enforcing a unit's pre-activations to be unimodal-normally distributed.\n>\n>Regarding a discussion on alternative normalization layers.\n>\nWe have now cited several works on more recent normalization layers in our paper, including EvoNorm and iterative normalization, as well as weight normalization, filter response normalization, and normalization propagation.\n>\n>\"There is no clear theoretical evidence that $I\\left(X; X+Z\\right)$ is maximized during the training, which is crucial for the application of Theorem 5.1.\"\n>\nWe would with excitement like to clarify this point of inquiry: Our discussion of the mutual information term $I\\left(X; X+Z\\right)$ uses the following argument as motivation for gaussianizing pre-activations: if the pre-activations are gaussianized, then by Theorem 5.1 $I\\left(X; X+Z\\right)$ is *necessarily* maximized. This is because a Gaussian distributed variable $X$ maximizes $I\\left(X; X+Z\\right)$ (as shown by the Theorem) relative to any other distribution for $X$. Thus $I\\left(X; X+Z\\right)$ is maximized when the gaussianity of $X$ is maximized; even if this occurs implicitly.\n>\n>Regarding comparison to work [8a].\n>\nWe found the paper you referenced very interesting and have now cited it in our work. Interestingly, it differs from our work due to the aforementioned point; we discuss the implicit maximization of $I\\left(X; X+Z\\right)$ and use this idea only as motivation in our work for encoding pre-activations using the Gaussian distribution, whereas to the best of our understanding the work you reference develops a framework for explicitly maximizing this term; making the approaches in the two works interestingly quite distinct and quite complementary. We are eager to explore the possible interplay between our work and this work in follow-up work.\n\nWe believe we have comprehensively addressed your comments here. We would be highly appreciative if you would consider increasing the score for our submission; thank you.", "claims_and_evidence": "- Supported Claims include improved generalization (Tables 1–2 show accuracy gains over traditional BatchNorm/LayerNorm) and Gaussianity in the features (Q-Q plots and $R^2$ metrics validate increased normality).\n- The method’s generality claim (\"wherever existing normalization layers are used\" from lines 42, 411) suggests validation also on non-CV tasks. However, such results are absent from the work.\n- Robustness to adversarial as well as random perturbations (\"improving model robustness to random perturbations\" from lines 42-43) — are not convincingly substantiated. Without adversarial experiments, these broader claims seem unsupported.", "methods_and_evaluation_criteria": "- The proposed method combines a power transform with additive Gaussian noise without introducing extra learnable parameters. The approach is implemented as an augmentation to conventional normalization layers.  \n- While the method is clear and evaluated on several vision architectures and datasets, there is a notable absence of comparisons to more recent or alternative normalization methods (e.g. EvoNorm [3a], Iterative Normalization [Huang et al., 2019]) beyond the classical baselines.\n- The Q-Q plots and $R^2$ metrics do not serve as a proper multivariate Gaussianization metric. Perhaps, special statistical tests should be employed, e.g., the Henze-Zirkle test [3b].\n\n[3a] Liu et al. \"Evolving Normalization-Activation Layers\". arXiv:2004.02967\n\n[3b] Norbert Henze and Bernd Zirkler. \"A class of invariant consistent tests for multivariate normality\". Communications in Statistics-theory and Methods, 19:3595–3617, 1990", "theoretical_claims": "- The paper’s derivation of the quadratic approximation for the negative log-likelihood (NLL) to estimate the power transform parameter λ is central. It relies on a series expansion around $\\lambda_0 = 1$, and assumes activations are locally Gaussian-like near $\\lambda_0$. While empirically valid for the tested cases (Figure 14), it risks failure in deeper layers or complex datasets where activations are multi-modal or heavily skewed. Theoretical guarantees are limited to idealized setup, and the method’s robustness depends on the (unverified) assumption that activations are \"close enough\" to Gaussian. This constitutes a significant weakness if one wishes to claim universality. Therefore, the authors should test on multi-modal datasets and analyze layer-wise approximation quality.\n- There is no clear theoretical evidence that $I(X;X+Z)$ is maximized during the training, which is crucial for the application of Theorem 5.1. Therefore, it is unclear whether noise injection serves any meaningful role in this setup (at least, from the Theorem's 5.1 perspective). Please, compare this to the work [8a] from **Essential References Not Discussed**, where the mutual information is maximized explicitly.", "experimental_designs_or_analyses": "The experiments are extensive within the computer vision domain, proving performance on multiple datasets and architectures. Despite this, the experimental design omits comparisons with several state‐of‐the‐art (post-BatchNorm) normalization methods and does not explore non-CV applications, even though the method is promoted as generally applicable.", "relation_to_broader_scientific_literature": "The work is positioned as an extension of BatchNorm (Ioffe & Szegedy, 2015) and LayerNorm (Lei Ba et al., 2016) by explicitly aiming to make Gaussian activations building on classical power transforms (Box & Cox, 1964; Yeo & Johnson, 2000). It also relates to previous work on decorrelation and whitening (Chen & Gopinath, 2000) and on noise-based regularization. However, the discussion would benefit from a more thorough comparison with recent normalization methods and an explicit discussion of how this approach differs fundamentally from methods that already induce some form of Gaussianity (e.g, from the work [8a], see **Essential References Not Discussed**).", "essential_references_not_discussed": "While the paper cites classical works, it omits several recent advances. For instance, EvoNorm (Liu et al., 2021) is not discussed despite being a contemporary normalization-activation layer method. Additionally, although Iterative Normalization (Huang et al., 2019) is referenced, a more detailed discussion comparing it with the proposed approach would emphasise the contribution.\n\nThe work [8a] also uses noise injection and Theorem 5.1 to achieve Gaussian distribution, and, therefore, is closely related to the method proposed in this manuscript.\n\n[8a] Butakov et al. \"Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax\". Proc. of ICLR 2025.", "other_strengths_and_weaknesses": "The impact of BNN on the runtime compared to BN is not very significant, which is a merit of the method.", "other_comments_or_suggestions": "Perhaps, the \"booktabs\" table style should also be used for Table 3.", "questions_for_authors": "1. Could the authors rigorously prove theoretical bounds on approximation error or provide another theoretical analysis to strengthen their claim that the quadratic approximation is universally valid under the power transform’s Gaussianization? In particular, how does the approximation error change in deeper layers or with non-Gaussian activations? What are potential limitations of your approach in situations where the activations exhibit strong multimodality or heavy skewness?\n1. Could you evaluate normality normalization on non-vision tasks to justify the claim of “wherever existing normalization layers are used”? \n1. Could you compare your method with alternative normalization methods (e.g. EvoNorm) beyond the classical baselines?\n1. How is the maximization of $I(X;Y)$ from Theorem 5.1 enforced?\n1. Have you tried leaving only the noise injection (i.e., not applying the power transform)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hPmj6UomtW", "reviewer": "Reviewer_7TPa", "summary": "Full disclosure: I was a reviewer for a paper for ICLR 2025 which seems to be largely mirroring this paper and I assume that this is a resubmission of that paper (I'm reviewer G4ZL here: https://openreview.net/forum?id=9ut3QBscB0) \n\n\nThis paper introduces normality normalization as a  new type of normalization layer that attempts to impose stronger Gaussianity on the activations in neural networks. They motivate the via information theory by invoking classical facts about Gaussian distribution, namely that Gaussian is the “best-case signal” and “worst-case noise,” so adopting Gaussian representations plus Gaussian noise can maximize information capacity and tolerance to perturbations.  The key technical part is applying Yeo-Johnson power transform on the normalized activations and making them marginally closer to normal (marginal likelihood). They also propose an additive Gaussian noise.  Since this is only a way of correcting statistics, this can be applied to both Layer and Batch Norm to create “BatchNormalNorm,” “LayerNormalNorm,” etc. \n\nIn experiments on CIFAR-10/100, SVHN, STL10, TinyImageNet, Caltech101, Food101, and ImageNet, for ResNets, WideResNets, & Vision Transformers), they show normality normalization outperforms standard normalization (Tables 1 & 2). The authors also claim enhanced robustness to random noise via quantitative attenuation metrics.\n\n## Update after rebuttal \n\nAfter reviewing the rebuttal responses, am happy to increase my score from 3 to 4(Accept). I am happy with the main paper and the explanations provided here, and consider this paper to be a good contribution for ICML.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7TPa,\n\nWe address all of your comments below.\n>\n>Regarding the baseline performance levels, and the code snippet you provided.\n>\nTo address your inquiry regarding the baselines, we ran experiments with the additional use of mixup (Zhang et al. 2017) for several of the model & dataset combinations listed in Table 1 (with the experimental setup otherwise identical to that listed in Appendix E.2) across $M=6$ random seeds. The results are as follows:\n|Dataset|LN|LNN|\n|-|-|-|\n|CIFAR10|89.97 $\\pm$ 0.16|**91.18 $\\pm$ 0.13**|\n|CIFAR100|66.40 $\\pm$ 0.42|**70.12 $\\pm$ 0.22**|\n|Food101|73.25 $\\pm$ 0.19|**79.11 $\\pm$ 0.09**|\n\nThese results precisely address your inquiry regarding the baseline performance levels, and provide further substantiating evidence that models trained with normality normalization continue to improve with the use of additional techniques, and consistently outperform other normalization layers.\n\nTo further supplement this, please also see our rebuttal comment to Reviewer ps2s regarding the performance of ResNet18 x CIFAR10 with data augmentations, showing again the improvement in performance of BNN.\n\nNext we address the code snippet provided and the performance therein. We actually ran the code you provided, replacing BatchNorm2d with BatchNormalNorm2d: we were able to obtain a performance of $90.95$% which - especially when considering the small number of training epochs ($15$) used in the code snippet - is a significant improvement over the figure you quoted (This is also significant because it is very reasonable to expect that the difference in performance would likely grow with further number of training iterations, because generally techniques using stochastic regularization (such as our additive Gaussian noise with scaling, or as seen elsewhere ex: with the use of dropout) tend to improve more with number of training iterations).\n\nThis adds to the evidence that, across a wide array of experimental setups - for example here in the code snippet you provided the optimizer employed (Adam) is different than in our ResNet experiments (SGD), the LR scheduler is different (OneCycleLR vs StepLR), gradient clipping is employed whereas it is not in our setting, and other aspects - that normality normalization consistently outperforms other normalization layers.\n\nAltogether these experiments serve to substantively address your inquiry regarding the baseline performance levels.\n>\n>Regarding a direct side-by-side experiment with an alternative normalization layer.\n>\nWe have addressed this as follows: Since we had already invoked decorrelated BatchNorm (DBN) in the text of our paper, we ran experiments using DBN, and compared this with the implementation we developed for decorrelated BatchNormalNorm (DBNN). The experimental setup is consistent with Appendix E.1, and $M=6$ random seeds are also used throughout:\n|Dataset|Model|DBN|DBNN|\n|-|-|-|-|\n|CIFAR10|RN18|90.66 $\\pm$ 0.05|**91.50 $\\pm$ 0.03**|\n|CIFAR100|RN18|65.11 $\\pm$ 0.06|**67.53 $\\pm$ 0.10**|\n|STL10|RN34|66.76 $\\pm$ 0.29|**69.36 $\\pm$ 0.14**|\n\nThese results demonstrate a consistent improvement of DBNN over DBN.\n>\n>Regarding accounting for the training overhead from computing $\\hat{\\lambda}$.\n>\nWe believe our Subsection A.5 Speed Benchmarks and Figure 11, referenced in the main body of the text in Subsection 4.6 Additional Experiments & Analysis under paragraph Speed benchmarks, does just this. We have evaluated both the training time and test time overheads.\n>\n>Regarding additional noise robustness experiments.\n>\nHere we provide additional noise robustness results for ResNet18 x CIFAR100:\n|$\\phantom{-}$|$\\phantom{-}$|L5|L9|L13|L17|\n|-|-|-|-|-|-|\n|**L1**|BNN|**0.047 $\\pm$ 0.002**|**0.074 $\\pm$ 0.001**|**0.100 $\\pm$ 0.002**|**0.386 $\\pm$ 0.005**|\n||BN|0.166 $\\pm$ 0.005|0.316 $\\pm$ 0.006|0.410 $\\pm$ 0.008|1.881 $\\pm$ 0.026|\n|**L5**|BNN||**0.027 $\\pm$ 0.002**|**0.040 $\\pm$ 0.003**|**0.155 $\\pm$ 0.012**|\n||BN||0.069 $\\pm$ 0.007|0.088 $\\pm$ 0.006|0.438 $\\pm$ 0.030|\n|**L9**|BNN|||**0.043 $\\pm$ 0.000**|**0.149 $\\pm$ 0.002**|\n||BN|||0.061 $\\pm$ 0.001|0.250 $\\pm$ 0.003|\n|**L13**|BNN||||**0.258 $\\pm$ 0.002**|\n||BN||||0.396 $\\pm$ 0.011|\n\nThis provides even further evidence for the findings we presented in Subsection A.3 Noise Robustness.\n>\n>Regarding the suggested references you mentioned.\n>\nWe have now included a discussion on the related normalization layers you listed: weight normalization, filter response normalization, normalization propagation, as well as iterative normalization and EvoNorm.\n\nRegarding the copula-based, inverse normal transform, and Lambert W transformation approaches to gaussianizing: we agree these are very interesting avenues for exploration, and have now included a discussion on them in Section 6 Related Work & Future Directions.\n\nWe believe we have comprehensively addressed your comments here. We would be highly appreciative if you would consider increasing the score for our submission; thank you.", "claims_and_evidence": "While the central claim that normality normalization outperforms the classical normalization seems to be validated by results in Table 1 & 2, there seems to be some gap between reported values for baselines and those reported in other papers. This leaves me the impression that the experimental setup (hyper params and training ) for the baselines may be not good enough to fully substantiate the empirical claims made in the paper. For example, I adapted the code found here: https://www.kaggle.com/code/kmldas/cifar10-resnet-90-accuracy-less-than-5-min, for a small resnet, and even the no data augmented reaches 90.25% accuracy. I understand that this is not the same ResNet they used, but the fact that a small ResNet can achieve >90% accuracy without data augmentation, makes the improvement to 90.4% in Table1 when using BNN, somewhat less significant. I'm eager to hear what authors have to clarify this. \n\nthe sample code I used :\n```python\nimport os\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision.transforms as tt\nfrom torchvision.datasets import CIFAR10\nfrom torch.utils.data import DataLoader\nimport time\n\n# Device setup\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\ndef to_device(x, device): return x.to(device, non_blocking=True) if isinstance(x, torch.Tensor) else [to_device(item, device) for item in x]\n\nclass DeviceDataLoader:\n    def __init__(self, dl, device): self.dl, self.device = dl, device\n    def __iter__(self): return (to_device(batch, self.device) for batch in self.dl)\n    def __len__(self): return len(self.dl)\n\n# Model definition\ndef accuracy(outputs, labels):\n    _, preds = torch.max(outputs, dim=1)\n    return torch.tensor(torch.sum(preds == labels).item() / len(preds))\n\ndef conv_block(in_ch, out_ch, pool=False):\n    layers = [nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1), nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True)]\n    if pool: layers.append(nn.MaxPool2d(2))\n    return nn.Sequential(*layers)\n\nclass ResNet9(nn.Module):\n    def __init__(self, in_channels, num_classes):\n        super().__init__()\n        self.conv1 = conv_block(in_channels, 64)\n        self.conv2 = conv_block(64, 128, pool=True)\n        self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))\n        self.conv3 = conv_block(128, 256, pool=True)\n        self.conv4 = conv_block(256, 512, pool=True)\n        self.res2 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))\n        self.classifier = nn.Sequential(nn.MaxPool2d(4), nn.Flatten(), nn.Linear(512, num_classes))\n        \n    def forward(self, x):\n        x = self.conv1(x)\n        x = self.conv2(x)\n        x = self.res1(x) + x\n        x = self.conv3(x)\n        x = self.conv4(x)\n        x = self.res2(x) + x\n        return self.classifier(x)\n    \n    def training_step(self, batch):\n        images, labels = batch \n        out = self(images)\n        return F.cross_entropy(out, labels)\n    \n    def validation_step(self, batch):\n        images, labels = batch \n        out = self(images)\n        loss = F.cross_entropy(out, labels)\n        acc = accuracy(out, labels)\n        return {'val_loss': loss.detach(), 'val_acc': acc}\n        \n    def validation_epoch_end(self, outputs):\n        batch_losses = [x['val_loss'] for x in outputs]\n        epoch_loss = torch.stack(batch_losses).mean()\n        batch_accs = [x['val_acc'] for x in outputs]\n        epoch_acc = torch.stack(batch_accs).mean()\n        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}\n\n# Training functions\n@torch.no_grad()\ndef evaluate(model, val_loader):\n    model.eval()\n    outputs = [model.validation_step(batch) for batch in val_loader]\n    return model.validation_epoch_end(outputs)\n\ndef train_model(epochs, max_lr, model, train_dl, valid_dl, weight_decay=1e-4, grad_clip=0.1):\n    optimizer = torch.optim.Adam(model.parameters(), max_lr, weight_decay=weight_decay)\n    scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, epochs=epochs, steps_per_epoch=len(train_dl))\n    \n    history = []\n    start_time = time.time()\n    \n    for epoch in range(epochs):\n        # Training\n        model.train()\n        train_losses = []\n        lrs = []\n        \n        for batch in train_dl:\n            loss = model.training_step(batch)\n            train_losses.append(loss)\n            loss.backward()\n            \n            # Gradient clipping\n            if grad_clip: \n                nn.utils.clip_grad_value_(model.parameters(), grad_clip)\n            \n            optimizer.step()\n            optimizer.zero_grad()\n            \n            # Record & update learning rate\n            lrs.append(optimizer.param_groups[0]['lr'])\n            scheduler.step()\n        \n        # Validation\n        result = evaluate(model, valid_dl)\n        result['train_loss'] = torch.stack(train_losses).mean().item()\n        result['lrs'] = lrs\n        \n        # Print progress\n        print(f\"Epoch [{epoch}], lr: {lrs[-1]:.5f}, train_loss: {result['train_loss']:.4f}, val_loss: {result['val_loss']:.4f}, val_acc: {result['val_acc']:.4f}\")\n        history.append(result)\n    \n    train_time = time.time() - start_time\n    print(f\"Training completed in {train_time/60:.2f} minutes\")\n    return history\n\nif __name__ == \"__main__\":\n    stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))\n    train_tfms = tt.Compose([tt.ToTensor(), tt.Normalize(*stats)])\n    valid_tfms = tt.Compose([tt.ToTensor(), tt.Normalize(*stats)])\n    \n    # Dataset\n    os.makedirs(data_dir, exist_ok=True)\n    train_ds = CIFAR10(root= './data', train=True, download=True, transform=train_tfms)\n    valid_ds = CIFAR10(root= './data', train=False, download=True, transform=valid_tfms)\n    \n    # DataLoaders\n    batch_size = 400\n    train_dl = DeviceDataLoader(DataLoader(train_ds, batch_size, shuffle=True, num_workers=3, pin_memory=True), device)\n    valid_dl = DeviceDataLoader(DataLoader(valid_ds, batch_size*2, num_workers=3, pin_memory=True), device)\n    \n    # Model\n    model = ResNet9(3, 10).to(device)\n    \n    # Training\n    history = train_model(\n        epochs=15,\n        max_lr=0.01,\n        model=model,\n        train_dl=train_dl,\n        valid_dl=valid_dl,\n        weight_decay=1e-4,\n        grad_clip=0.1\n    )\n   \n```", "methods_and_evaluation_criteria": "Yes. They evaluate classification performance on mainstream benchmarks (CIFAR, SVHN, TinyImageNet, etc.), measure standard top-1 (and occasionally top-5) accuracy, and compare to widely used baselines (BatchNorm, LayerNorm, etc.). They also run controlled studies on batch size, network width, and depth, making a strong case that normality normalization is robust across conditions. These are all common and accurate choices to backup their claims.", "theoretical_claims": "- The lemma (Appendix B) that in the bivariate normal case, uncorrelatedness implies independence and also that normality minimizes mutual information for given correlation. This is known in standard information-theoretic references and is stated correctly. \n- The second-order series expansion for the negative log-likelihood (Appendices C & D) and the single-step Newton–Raphson for estimating  λ looks coherent to me. authors also provide empirical checks", "experimental_designs_or_analyses": "The experimental setup ticks the basic \n- Cross validation \n- They use 6 runs with independent seeds to obtain mean and standard error \n- They ablate key pieces: the power transform alone, noise alone, partial transform strength.\n- They measure Q–Q plots to show actual Gaussianity, which is a key claim of the proposed layer\nHowever, as mentioned in the Claims And Evidence section, I have some reservations about the optimality of the baseline results and would be eager to hear clarifications from authors", "relation_to_broader_scientific_literature": "- Normalization layers: The paper gives a lot of focus to batch and layer normalization, which is well deserved, they also mention some other key works such as Decorrelated BN or Switchable Whitenning.  They also cite crucial references on maximum-entropy properties of the Gaussian distribution (Cover & Thomas), Box–Cox and Yeo–Johnson transforms, and typical random smoothing (Cohen et al.). \n- Whitening/Orthogonalization: Methods like Decorrelated Batch Normalization (Huang et al.) or Iterative Normalization attempt to whiten features, which is tangentially similar to making them normal. The authors focus on univariate normality; whitened or orthonormal constraints tackle correlations as well.\n- Robustness-Enhancing Approaches: The paper cites Randomized smoothing (e.g., Salman et al. 2019), which can provide certified $\\ell_2$-robustness via Gaussian noise injection at inference. data augmentation (mixup, AugMix) also yield stable, noise-resistant features. Comparing normality normalization to these could further elucidate how it complements or surpasses them.", "essential_references_not_discussed": "- Weight Normalization (Salimans & Kingma)\n- Filter Response Normalization (Singh & Krishnan)\n- Normalization Propagation (Arpit 2016)\n- While not Copula-based or rank-based inverse normal transforms as direct ways to Gaussianize data, or Lambert W transformations for heavy-tailed distributions\n- Copula-Based Gaussianization – In statistics, any multivariate data can be transformed to have normal marginals by using a copula. A Gaussian copul method assumes the data can be mapped into a joint Gaussian via monotonic marginal CDF mappings. \nWhile the paper focuses on certain transforms, it doesn’t mention the general copula framework, which might be interesting", "other_strengths_and_weaknesses": "Strengths:\n\n- Straightforward to implement: only adding a power transform step plus scaled Gaussian noise.\n- Motivated theoretically very well, the idea of having Gaussian activations is quite an interesting approach and this paper takes an important step in that direction \n- Clear discussion of theoretical motivations (information-theoretic and statistical underpinnings).\n\nWeaknesses:\n\n- as mentioned earlier, the paper gives somewhat questionable baselines in Tables 1 & 2. I might be wrong but if\n- The experiments in the robust training seem rather thin\n- It would be interesting to account for the training overhead from computing  𝜆 in  each iteration in comparison to standard BN", "other_comments_or_suggestions": "- A direct side-by-side experiment with at least one of the alternative norms that also claim small-batch stability (e.g., FRN or SwitchableNorm) might strengthen the empirical section.\n- Consider using a baselines that are reported in other papers, namely ResNet or ViTs, to make sure that the baselines are not sub-optimal", "questions_for_authors": "- Have authors thought about connection to other domains, namely loss of plasticity in continual learning? Namely, in preserving diversity of hidden units and features during prolonged training", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WKUv4Ga2HP", "reviewer": "Reviewer_ps2s", "summary": "The authors propose adding a \"Gaussianizing\" step into normalisation layers such as batchnorm, which transforms the features so that they are approximately Gaussian-distributed. Specifically, they use the \"power transform\" originally proposed in the field of hypothesis testing, but propose approximating its objective by a quadratic so that the \"correct\" transform may be determined using a single Newton-Raphson step (which is important since this must be done every time a normalisation is used during a forward pass). The authors present many arguments, largely from an information-theoretic perspective, as to why having Gaussian features is desirable / important. This also motivates the authors' use of additive Gaussian noise as regularisation.\n\nThe authors present a wide range of experiments exploring the effect of their method on generalisation under different ablations, with models / datasets focusing on image classification tasks using ResNets and ViTs. They also verify the Gaussianity of the features when using their method vs standard normalisation layers. They find almost universal benefits to generalisation using their method, though, as detailed in a plot in the appendix, the proposed method does increase runtime by about 50% during training, and about 25% during testing.\n\n### Update after rebuttals\nI have maintained my score of \"accept\" - please see rebuttal comment", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ps2s,\n\nWe address all of your comments below.\n>\n>Regarding the use of data augmentations in the ResNet experiments.\n>\nWe have run an experiment to verify the performance of ResNet18 x CIFAR10 using BatchNormalNorm (BNN), and contrast this with a well-documented baseline - which is in line with your expected level of performance when using BatchNorm (BN) (https://github.com/kuangliu/pytorch-cifar). We use the same data augmentations of transforms.RandomCrop(32, padding=4) and transforms.RandomHorizontalFlip() as listed in the repository. Across $M=6$ runs, we obtained a mean validation set accuracy of $94.93$%$\\pm0.05$, which surpasses the reference performance of $93.02$% listed in the repository; this latter figure being in line with your expected level of performance for BN when data augmentations are employed. This serves to demonstrate that BNN continues to scale and outperform with the use of data augmentations in the ResNet experiments; and is analogous to the findings for LayerNormalNorm (LNN) and LayerNorm (LN).\n>\n>Regarding the justification for not employing data augmentations in Table 2.\n>\nWe next provide our justification for this choice in our experimental design. Our goal was for the set of experiments with data augmentations (Table 1) and without data augmentations (Table 2) to serve as an ablation - showing that the method performs well regardless of specific augmentation techniques used.\n\nCrucially: in many application areas, such as in time series analyses, and in fine-grained medical imaging tasks, it is often not clear what data augmentations are appropriate. Therefore, we believe demonstrating that our method performs strongly relative to other normalization layers - with and without the use of data augmentations - is extremely valuable.\n>\n>Regarding clarifying Figures 5 & 6.\n>\nFigure 5 shows the following: In a given layer of a neural network, we take all the post-normalization features for a given channel and minibatch combination. We then compute a QQ-plot and its associated $R^{2}$ value for the line of best fit. Now consider such a plot for three layers at various depths in the network. Thus Figure 5 serves to demonstrate graphically that normality normalization leads to higher normality in the features, as demonstrated by the higher $R^2$ values for the line of best fit. Crucially then, Figure 6 then substantiates these findings quantitatively: for each layer of the neural network (x-axis), we take $200$ QQ-plots corresponding to $20$ channels and $10$ validation minibatch combinations, we compute the $R^{2}$ values for each of these $200$ QQ-plots, then plot the mean $R^{2}$ value for that layer. Thus the figure demonstrates that throughout the layers of a network, normality normalization leads to much higher normality.\n>\n>\"slightly more explanation of where the NLL for the power transform comes from\"\n>\nPlease see the paragraph about the NLL in our response to Reviewer Eseb, which precisely addresses your comment here.\n>\n>Regarding clarifying that Gaussian noise is an additional regularization technique.\n>\nWe believe this is an excellent suggestion and have modified the paper accordingly.\n>\n>Regarding the inclusion of Figure 9 in the main paper.\n>\nWe have made this change now - the camera-ready version of the paper affords an additional page (9 instead of 8) for the main text, thereby making its inclusion very natural.\n>\n>Regarding summarizing the speed results in the main text and describing what parts of the algorithm cause slowdowns.\n>\nWe have now commented in Section 4.6 Additional Experiments & Analysis, under paragraph Speed Benchmarks, your suggested note on the speed difference.\n\nThe main speed differences occur due to the operations log(1+x) (\"log1p\") and raising to the power. During the work, we investigated making series expansion approximations to these operations, and substantiating their efficacy is a promising direction for future work.\n>\n>\"In Section 5.1.2. I didn't fully understand the point about (line 350) about not wanting to corrupt the distributions when regularising.\"\n>\nHere we are suggesting the following subtle distinction: we want to be able to add as much regularizing noise as possible, but without fundamentally corrupting the underlying signal. The key being that when using Gaussian encodings, the threshold for the amount of noise we can add is higher. You may also find the reference (Guo et al. 2005) (also referenced in our manuscript) to be of interest.\n\nWe have furthermore added experimental results contrasting decorrelated BatchNorm (DBN) with decorrelated BatchNormalNorm (DBNN); please see the thread with Reviewer 7TPa. These experimental results provide further evidence for the strong performance of normality normalization across various normalization layers.\n\nWe believe we have comprehensively addressed your comments here. We would be highly appreciative if you would consider increasing the score for our submission; thank you.", "claims_and_evidence": "The main claim made by the paper is that forcing neural networks to have Gaussian distributed features throughout training can have desirable effects on performance, which is supported by their experiments. I have a couple of minor issues / questions about some of the experiments, which I will detail below.", "methods_and_evaluation_criteria": "Yes. The hypothesis is that Gaussianizing representations could have positive effects on model performance. The proposed method effectively Gaussianises the features (as shown in Figure 5) and results in an improvement to validation accuracy on realistic tasks (Figure 1, Table 1, Table 2, etc.).", "theoretical_claims": "The paper does not make any theoretical claims (e.g. theorems, bounds). Derivations for the formulas used by their method are given in appendix C, which I did not check.", "experimental_designs_or_analyses": "The experiments only consider image classification tasks using ResNets and ViTs, though this includes a variety of architectures and datasets, and are run from multiple random seeds with error bars given. In an ideal world, the authors would have evaluated their method in another domain, such as language modelling with transformers, though given the extensive ablations and other investigations, I think this is an appropriate amount of experiments for a conference paper.\n\nIn the ResNet experiments, Table 2 says that data augmentation was not used. Data augmentation is usually used for ResNet experiments on datasets like CIFAR-10 and CIFAR-100, and greatly improves performance (e.g. we would expect closer to 93 or 94% for a ResNet18 on CIFAR-10 using data augmentation, rather than the 89% achieved by the paper's baseline model). This is especially strange given that Table 1 says that data augmentation **was** used in the ViT experiments. I would have preferred to see data augmentation for the ResNet experiments, as this is standard practice.", "relation_to_broader_scientific_literature": "The paper proposes a modification that can be applied to any neural network which utilises normalisation layers, and therefore has very broad scope. Moreover, the specific algorithm proposed in this paper is less important than the proof of concept that Gaussianing features in normalisation layers is beneficial. Further refinements or alternative algorithms to efficiently Gaussianise features in neural networks could be developed in the future if required.", "essential_references_not_discussed": "I am not aware of any essential references that have not been discussed.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper is well written. In particular, the connections to information theory and motivations are clearly and extensively explained.\n- I have not seen any work that explicitly Gaussianises the features in a neural network before, so as far as I'm aware, this is novel.\n- The suggested method does offer convincing improvements in validation accuracy\n- The authors have included a wide range of relevant ablations, analyses, and justifications, either in the main text or the appendix, which answered many questions I had while reading the paper.\n- The authors identify that, when your features are Gaussian, decorrelating / whitening transformations actually imply independent features. This could have important implications for recent works that utilise whitening / decorrelating transformations in optimisation, e.g. https://kellerjordan.github.io/posts/muon/ or https://arxiv.org/abs/2412.13148.\n- The authors try their method on 4 different types of normalisation (batchnorm, layernorm, instancenorm, and group norm) and show improvements in all cases.\n\n**Weaknesses**\n- The ResNet experiments do not use data augmentation, which is standard practice, and no explanation is given as to why.\n- The specific implementation proposed in this paper increases runtime by  25-50%, which is quite significant.\n- Some of the technical detail weren't very clearly explained. Specifically, slightly more explanation of where the NLL for the power transform comes from would make the paper easier to read. Also, it is not entirely clear what the QQ plots in the main paper (Fig 5 and 6) are showing (e.g. which layers are you plotting for, where does each value on the plot come from).", "other_comments_or_suggestions": "- It would be useful if the authors state at the start of section 3.2 that the additive Gaussian noise is an additional regularisation technique enabled / enhanced by the Gaussianisation of the features, and not actually part of the Gaussianisation process itself (assuming I have understood this correctly). I was confused by this while reading the paper.\n- Related to the previous point, I think it'd be nice to have figure 9 (comparing base model vs. base + gaussianised features vs base + gaussianised features + additive gaussian noise regularisation) in the main text. The Motivation section (section 5) could probably be shortened to accommodate this.\n- Please state in the main text a briefly summarised version of the speed results, e.g. \"the method increases training time by roughly 50%\" etc. It'd also be nice to briefly explain what specific part of the algorithm causes such large slowdowns.\n- The authors may be interested to try their method on some \"speedrun\" benchmarks e.g. https://github.com/KellerJordan/cifar10-airbench or https://github.com/KellerJordan/modded-nanogpt. Obviously the wallclock time will not be impressive given the method's current slowdowns, but loss vs iteration / epoch would be interesting.", "questions_for_authors": "- Why did you not use data augmentation on the ResNets? Can you rerun these experiments with data augmentation and include them in the paper?\n- In Section 5.1.2. I didn't fully understand the point about (line 350) about not wanting to corrupt the distributions when regularising. Is corruption not the point of random noise / regularisation like dropout?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The main contribution of this work is demonstrating that Gaussianizing data representations across the layers of deep neural networks can enhance the performance of deep neural networks. This observation provides valuable insights into the inner workings of normalization layers, which are known to induce Gaussian-like distributions at initialization. Interestingly, explicitly enforcing this Gaussian property further improves the performance of convolutional neural networks (CNNs). While there are concerns regarding the depth of insights and theoretical arguments, the experimental contribution is substantial enough for ICML acceptance.
Decision: Accept (poster)