Kernel Normalized Convolutional Networks
Authors: Reza Nasirigerdeh
Reihaneh Torkzadehmahani
Daniel Rueckert
Georgios Kaissis
Venue: Accepted by TMLR ()

Abstract: Existing convolutional neural network architectures frequently rely upon batch normalization (BatchNorm) to effectively train the model. BatchNorm, however, performs poorly with small batch sizes, and is inapplicable to differential privacy. To address these limitations, we propose the kernel normalization (KernelNorm) and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional networks (KNConvNets) as the main building blocks. We implement KNConvNets corresponding to the state-of-the-art ResNets while forgoing the BatchNorm layers. Through extensive experiments, we illustrate that KNConvNets achieve higher or competitive performance compared to the BatchNorm counterparts in image classification and semantic segmentation. They also significantly outperform their batch-independent competitors including those based on layer and group normalization in non-private and differentially private training. Given that, KernelNorm combines the batch-independence property of layer and group normalization with the performance advantage of BatchNorm.

Published in Transactions on Machine Learning Research (02/2024)
Kernel Normalized Convolutional Networks
Reza Nasirigerdch
wea..wasirigrrdei@haa..om
Technical University of Munich
Helmholts Munich
Reihaneh Torkzadehmahani
reihaneh. www.tttaaaaeeeaaannnntt....c
Technical University of Munich
Daniel Rueckert
daniel..uackerrt@uum.d
Technical University of Munich
Imperial College London
Georgios Kaissis
g.aaissi@tumm.c de
Technical University of Munich
Helmholtz Munich
Reviewed on OpenReview: https: // wptp///eee.rvvee...em/ee net/ forum? id= Uv3X
Abstract
Existing convolutional neural network architectures frequently rely upon batch normaliza- tion (BatchNorm) to effectively train the model. BatchNorm, however, performs poorly with small batch sizes, and is inapplicable to differential privacy. To address these limi- tations, we propose the kernel normalization (KernelNorm) and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional nelworks (KNConnNNets) as the main building blocks. We implement KNConvNets corresponding to the state-of-the-art ResNets while forgoing the BatchNorm layers. Through extensive exper- iments, we illustrate that KNConvNets achieve higher or competitive performance compared to the BatchNorm counterparts in image classífication and semantic segmentation. 'hey also significantly outperform their batch-independeni competitors including those based on layer and group normalization in non- non-private and differentially private training. Given that, KernelNorm combines the batch-inddeenddenee properly of layer and group normalization with the performance advantage of BatchNorm
1 Introduction
Convolutional neural networks (CNNs) LeCun et al. 1989 are standard architectures in computer vision tasks such as image classification Krizhevsky et al. 2012 Sermanct et al. 2014) and semantic segmentation
[Long et al. 2015b). Deep CNNs including ResNets (He et al.) 2016a) achieved outstanding performance classification of challenging datasets such as ImageNet (Deng et al. 2009). One of the main building blocks of these CNNs batch normalization (BatchNorm) (Loffe & Szegedy 2015). The BatchNorm layer
considerably enhances the performance of deep CNNs by smoothening the optimization landscape (Santurkar
et al.] 2018), and addressing the problem of vanishing gradients (Bengio et al. 1994 Glorot & Bengio 2010).
BatchNorm, however, has the disadvantage of breaking the independence among the samples in the batch Brock et al. 2021b). This because BatchNorm carries out normalization along the batch dimension (Figure 11), and as a result, the normalized value associated with a given sample depends on the statistics of the other samples in the batch. Consoquently, the effectiveness of BatchNorm is highly dependent on
'Th code available htttps://gitiiii http://itttbb.ccm/eeeeeeee com/reza www.eeeeeaaasiiiieecoeeoe mas.rrrgrreeeeeee/oomtteeer

---

Published in Transactions on Machine Learning Research (02/2024)
batch size. With large batch sizes, the batch normalized models are trained effectively due to more accurate estimation of the batch statistics. Using small batch sizes, on the other hand, BatchNorm causes reduction in model accuracy (Wu & Hel 2018) because of dramatic fluctuations in the batch statistics. BatchNorm, morcover, is inapplicable to differential privacy (DP) (Dwork & Roth 2014). For the theorctical guarantees of DP to hold for the training of neural networks (Abadi et al. 2016), is required to compute the gradients individually for each sample in a batch, clip the per-sample gradients, and then average and inject random noise to limit the information learnt about any particular sample. Because per-sample (individual) gradients are required, the gradients of a given sample are not allowed to be influenced by other samples in the batch. This is not the case for BatchNorm, where samples are normalized using the statistics computed over the other samples in the batch. Consequently, BatchNorm is inherently incompatible with DP.
To overcome the limitations of BatchNorm, the community has introduced batch-indeeendent normalization layers including layer normalization (LayerNorm) (Ba et al. 2016), instance normalization (InstanceNorm) (Uyannvv et al. 2016), group normalization (GroupNorm) (Wu & He 2018), positional normalization (Po- sitionalNorm) [Li el al. 2019), and local context normalization (TocalContextNoom)) (Ortiz et al. 2020). which perform normalization independently for each sample in the batch. These layers do not suffer from the drawbacks of BatchNorm, and might outperform BatchNorm in particular domains such as generative tasks (e.g. LayerNorm in Transformer models (Vaswani et al. 2017)) ) For image classification and semantic seg- mentation, however, they typically do not achieve performance comparable with BatchNorm's in non-private (without DP) training. In DP, moreover, these batch-independdnt layers might not provide the accuracy gain we expect compared to non-private learning. This motivates us to develop alternative layers, which are batch-independent but more efficient in both non-private and differentaally private learning.
Our main contribution is to propose two novel batch-inddpenddet layers called kernel normalization (KernelNorm) and the kernel normalized convolutional (KNConv) layer to further enhance the per- formance of deep CNNs. The distinguishing characteristic of the proposed layers is that they extensively take into account the spatial correlation among the elements during normalization. KernelNorm is similar to a pooling layer, except that it normalizes the elements specified by the kernel window instead of computing the average/maxine /mwxiium of the elements, and it operates over all input channels instead of a single channel (Figure 3g). KNConv is the combination of KernelNorm with convolutional layer, where it applics KernelNorm to the input, and feeds KernelNorm's output to the convolutional layer (Figure 2 From another perspective, KNConv is the same as the convolutional layer except that KNConv first normalizes the input elements specified by the kernel window, and then computes the convolution between the normalized elements and kernel weights. In both aforementioned naive forms, however, KNConv is computationally inefficient because it leads to extremely large number of normalization units, and therefore, considerable computational over- head to normalize the corresponding elements. To tackle this issuc, we present conputationalll-cfficooi KNConv (Algorithm [}, where the output of the convoluttonn is adjusted using the mean and variance of the normalization units. This way, it is not required to normalize the elements, improving the computation time by orders of magnitude.
As an application of the proposed layers, we introduce kernel normalized convolutional networks (KNConvNets) corresponding to residual networks (He et al. 2016a). referred to as KNResNets, which employ KernelNorm and computationally--ee eefficcin KNConv as the main building blocks while forgoing the BatchNorm layers (Section 3). Our last contribution is to draw performance comparisons among KNRes- Nets and the competitors using several benchmark datasets including CIFAR-100 (Krizhevsky et al. 2009). ImageNet Deng et al. 2009), and Cityscapes (Cordts et al. 2016). According to the experimental results (Section KNResNets deliver significantly higher accuracy than the BatchNorm counterparts in image classification on CIFAR-100 using small batch size. KNResNets, moreover, achieve higher or competitive performance compared to the batch normalized ResNets in classification on ImageNet and semantic segmen- tation on CityScapes. Furthermore, KNResNets considerably outperform GroupNorm and LayerNorm based models for almost all considered case studies in non-private and differentially private learning. Considering that, KernelNorm combines the performance advantage of BatchNorm with the batch-independence benefit of LayerNorm and GroupNorm.
2

---

Published in Transactions on Machine Learning Research (02/2024)
Batch
Channe
Charmell
Ctarmal
Height
Haigh
(a) BatchNNrm
(b) LayerNorm
InstanceNorm
(d) GroupNorm
Clarnel
Chammel
Chariel
Charnel
Height
Height
Height
Height
Widhh
Width
Wilth
Width
(e) PositionalNorm
(f) LocalContexttNorm
(g) KernelNorm
Figure 1: Normalization layers differ from one another in their normalization unit (highlighted in blue and green). The normalization layers in (a)-(I) establish a one-lo-one corvespondence wee the input and normalized elements (i.e. no overlap between the normalization units, and no ignorance of an element). The proposed KernelNorm layer does not impose such one-to-one correspondence: Some elements (dash- hatched area) are common among the normalization units, contributing more than once to the output, while some elements (uncolored ones) are ignored during normalization. Due to this unique property of overlapping normalization units, KernelNorm extensively incorporates the spatial correlation among the elements during normalization (akin to the convolutional layer), which is not the case for the other normalization layers.
2 Normalization Layers
Normalization methods can be categorized into input normalization and weight normalization (Salimans
& Kingma 2016 Bansal et al.] 2018 Wang et al., 2020; Qi ct al., 2020). The former techniques perform
normalization on the input tensor, while the latter ones normalize the model weights. The aforementioned
layers including BatchNorm, and the proposed KernelNorm layer as well as divrisive normalization (Heeger
1992
Bonds 1989), (Ren et al. (017)) and local response normalization (LocalResponseNNrrm, Krizhevsky
ct al.
2012) belong to the category of input normalization. Weight standardization (Huang et al. 2017b Qiao
et al.
2019 and normalizer-free networks (Brock et al. 2021a) fall into the category of weight normalization.
In the following, we provide an overview on the existing normalization layers closely related KernelNorm, i.c. the layers which are based input normalization, and employ standard normalization (zero-mean and unit-variance) to normalize the input tensor. For the sake of simplicity, we focus on 2D images, but the concepts are also applicable to 3D images. For a 2D image, the input of a layer is a 4D tensor of shape (n, c, h, w), where n is batch size, e is the number of input channels, h is height, and w is width of the tensor. Normalization layers differ from one another in their normalization unit, which is a group of input elements that are normalized together with the mean and variance of the unit.
The normalization unit of BatchNorm (Figure [p) is 3D tensor of shape (n, h w), implying that BatchNorm incorporates all elements in the batch, height, and width dimensions during normalization. LayerNorm's normalization unit (Figure [b) is 3D tensor of shape (c. h, w), i.c. LayerNorm considers all elements in the channel, height, and width dimensions for normalization. The normalization unit of InstanceNorm (Figure [b) is 2D tensor of shape (h, w), i.e. all elements of the height and width dimensions are taken into account during normalization.
GooupNorm's normalization unit (Figure [ is 3D tensor of shape (cg. h, w), where cy indicates the channel group size. Thus, GroupNorm incorporates all elements in the height and width dimensions and
3

---

Published in Transactions on Machine Learning Research (02/2024)
subset of elements specified by the group size in the channel dimension during normalization. Positional- Norm's normalization unit (Figure T is 1D tensor of shape i.e. PositionalNorm performs channel-wise normalization. The normalization unit of LocalContextNorrn (Figure Lb is 3D tensor of shape (cy, s), where Cg is the group size, and (r, s) is the window size. Therefore, LoctCConto considers a subset of elements in the height, width, and channel dimensions during normalization.
BatchNorm, LayerNorm, InstanceNorm, and GroupNorm consider all elements in the height and width dimensions for normalization, and thus, they are referred to global normalization layers. PositionalNorm and LocalContextNorm,
the other hand, are called local normalization layers (Ortiz et al., 2020) because they incorporate a subset of elements from the aforementioned dimensions during normalization. In spite of their differences, the aforementioned normalization layers including BatchNorm have at least one thing in common: There is a one-to-one correspondence between the original elements in the input and the normalized elements in the output. That is, there is exactly one normalized element associated with each input element. Therefore, these layers do not modify the shape of the input during normalization.
3 Kernel Normalized Convolutional Networks
The KernelNorm and KNConv layers are the main building blocks of KNConwNets. KernelNorm takes the kernel size (k), kw); stride (sh, aw), padding (Ph, Pw); and dropout probability p as hyper-parameeers.. It pads the imput with zeros if padding is specified. The normalization unit of KernelNorm (Figure [g] is a tensor of shape (c, ky, kw), i.e KerneLNorm incorporates all elements in the channel dimension but a subset of elements specified by the kernel size from the height and width dimensions during normalization. The KernelNorm layer (1) applies random dropout (Srivastava et al. 2014) to the normalization unit to obtain the dropped-out out unit, (2) computes mean and variance of the dropped-out unit, and (3) employs the calculated mean and variance to normalize the original normalization unit:
U'= = Dp(U),
(1)
ΣΣ>
U' U'iee bw)
Mur
Kn kw
(2)
d = c. kin kw
Σ (0' (fo, )wr)
to mm
U pat
=
(3)
+
where p is the dropout probability, D, is the dropout operation, U is the normalization unit, U' is the dropped-out unit, Per and o² the mean and variance of the dropped-out unit, respectivcly, e is small number (e.g. 10-5) for numerical stability, and Ú is the normalized unit.
Partially inspired by BatchNorm, KerneNNorm introduces regularizing effect during training by intentionally normalizing the elements of the original unit U using the statistics computed over the dropped-out unit U'. In BatchNorm, the normalization statistics are computed over the batch but not the whole dataset, where the mean and variance of the batch are randomized approximations of those from the whole dataset. The "stochasticity from the batch statistics" creates a regularizing effect in BatchNorm according to Ba et al. (2016). KernelNorm employs dropout to generate similar stochasticity in the mean and variance of the normalization unit. Notice that the naive option of injocting random noise directly into the mean and variance might generate too much randomness, and hinder model convergence. Using dropout in the aforementioned fashion, KernelNorm can control the regularization effect with more flexibility.
The first normalization unit of KernelNorm is bounded to window specified by diagonal points (1, 1) and (kn, kw)) in the height and width dimensions. The coordinates of the next normalization unit are (1, 1 + sw) and (ks, bw + sw), which are obtained by sliding the window su elements along the width dimension. If there are not enough elements for kernel in the width dimension, the window is slid by sn elements in the height dimension, and the above procedure is repeated. Notice that KerneNNorm works on the padded input of shape (n, e, h + 2 Ph, w 2 pw), where (ph: Pw) is the padding size. The output & of KernelNorm
4

---

Published in Transactions on Machine Learning Research (02/2024)
is the concatenation of the normalized units U from Equation 3 along the height and width dimensions.
KernelNorm's 's output is of shape (n, c, hoat, Waut), and it has total of n. where hout and Woat are computed as follows:
haut
Waut
normalization units,
Rw
hout kn h+ 2- 2-Ph ki
W + 2 Pp
kiw
11, Wout kw
+1 +1
8h
Sw
In simple terms, KernelNorm behaves similarly to the pooling layers with two major differences: (1) Ker- nelNorm normalizes the elements specified by the kernel size instead of computing the maximum aaaiiuum/averagee /average over the elements, and (2) KernelNorm operates over all channels rather than a single channel. KernelNorm is a intee-independent and local normalization layer, but differs from the existing normalization layers two aspects: (I) There is not necessarily a one-to-one correspondence between the original elements in the input and the normalized elements in the output of KernelNorm. Stride values less than kernel size lead to overlapping normalization units, where some input elements contribute more than once in the output (akin to the convolutional layer). If the stride value is greater than kernel size, some input elements are completely ignored during normalization. Therefore, the output shape of KernelNorm can be different from the input shape. (IT) KernelNorm can extensively take into account the spatial correlation among the elements during normalization because the overlapping normalization units.
KNConv is the combination of KernelNorm and the traditional convolutional layer (Figure 2. It takes the number of imput channels chin, number of output channels (filters) chout, kernel size (kn, kw), stride (sh: sw); and padding (ph: Pw)), exactly the same the convolutional layer, well as the dropout probability p as
KNConv first applies KernelNorm with kernel size (ki. kw), stride (s), sw), padding (Ph: Pw); and dropout probability to the input tensor. Next, it applies the convolutional layer with chin channels, chout filters, kernel size (k,, kw); stride (ks, kw); and padding of zero to the output of KerneINorm. That is, both kernel size and stride values of the convolutional layer are identical to kernel size of KernelNorm.
From another perspective, KNConv the same as the convolutional layer except that normalizes the input elements specified by the kernel window before computing the convolution. Assuming that U contains the input elements specified by the kernel window, Ö is the normalized version of U from KernelNorm (Equation 3 Z is the kernel weights of a given filter, is the convolution (or dot product) operation, and 6 is the bias value, KNConv computes the output as follows:
KNConv(U,Z,b) = 0 b
(4)
KNConv (or in fact KernelNorm) leads to extremely high number of normalization units, and consoquently, remarkable computational overhead. Thus, KNConv in its simple format outlined in Equation (or as a combination of the KernelNorm and convolutional layers) is computationally inefficient. Compared to the convolutional layer, the additional computational overhead of KNConv originates from (I) calculating the mean and variance of the units using Equation] 2 and (II) normalizing the elements by the mean and variance using Equation 3
Weights
Ingut tensor
Normalized tensor
Bias
Keme Layer
Output
Convolutional Layer
Figure 2: KNConv as the combination of the KernelNorm and convolutional layers. KNConv first applics KernelNorm with kernel size (3, 3) and stride (2,2) to the input tensor, and then gives KernelNorm's output convolutional layer with kernel size and stride (3, 3). That is, the kernel size and stride of the convolutional layer and the kernel size of NernelNorm are identical.
5

---

Published in Transactions on Machine Learning Research (02/2024)
Computtationally-cffcicis KNConv reformulates Equation f in way that it completely eliminates the overhead of normalizing the elements:
Kip
kw
(des in, iw) Hul) (ice in. iw) +
KNCouv ANCCnv(U,,,,) (U,
Vag. te
11
6/2
Ulio kn))
Z6o kn, Hul
Z(tes (irrdrr
+
(5)
Vou, +e
1==1 1/.
1
= (U * Z Pud
2( tw)))
+
V +
According to Equation and Algorithm KNConv applies the convolutional layer to the original unit, computes the mean and standard deviation of the dropped-out unit well as the sum of the kernel weights, and finally adjusts the convolution output using the computed statistics. This way, it is not required to normalize the elements, improving the computation time of KNConv by orders of magnitude.
terms of implementation, KernelNorm employs the unfolding operation in PyTorch 2023b) to imple- ment the sliding window mechanism in the tn__eenn_tee var function in Algorithm Morcover, it uses the var mean function in PyToorch (2023c) to compute the mean and variance over the unfolded tensor along the channel, width, and height dimensions.
The defining characteristic of KernelNorm and KNConv is that they take into consideration the spatial correlation among the elements during normalization on condition that the kernel size is greater than 1 1x1. Existing architectures (initially designed for global normalization), however, do not satisfy this condition. For instance, all ResNets use 1x1 convolution for downsampling and increasing the number of filters. ResNet- 50/100 1011 101/112, in particular, contains bottleneck blocks with single 3x3 and 1x1 convolutional layers, Consequently, the current architectures unable fully utilize the potential of kernel normalization.
KNConwNets are bespoke architectures for kernel normalization, consisting of computatioonally-fliici KNConv and KernelNorm as the main building blocks. KNConvNets are inceh-inddeendentt (free of Batch- Norm), which primarily employ kernel sizes of 2x2 or 3x3 to benefit from the spatial correlation of elements during normalization. In this study, we propose KNConvNets corresponding to ResNets, called KNResNels, for image classification and semantic segmentation.
Algorithm 1: Conputatiioaall--effiinn KNConv layer
Input: input lensor X, number of input channels ch number of output channels chouts kernel size (kh, kw), stride (sh, sw), padding (Ph, Pw), bias flag, dropout probability p, and epsilon
2-dimensional convolutional layer conv layer Conv2d(in channels=chan, out channels-chaus, kernel size=(Kin, kw), stride=(sh, 8w), padding-(ph, Pw), bias=false)
// convolutional layer output conv out = conv_ [an__aayiinpput]]
// mean and variance from Kernellorm
11 x²
mean var(input=X, kernel size-(k,, kw), stride-(sh, 8w), padding (prs Pw),
dropout_
// KiCoow output
kn conv out (conv_out μ Σ conv laye_ V02 +c
// apply bias
if bias then
kn conv out += conv layer.biar
Output: kn conv out

---

Published in Transactions on Machine Learning Research (02/2024)
KNCon 30
RiLUU
RuUU
ReLU
po0
(a) Basic block
(b) Bottleneck block
(c) Transitional block
Figure 3: KNResNet blocks: Basic blocks are employed in KNResNee-18/34, while KNResNet--00 is based
on bottleneck blocks. Transitional blocks are used in all KNResNets for increasing the number of filters and
downsampling. The architectures of KNResNct-18/34/ are available in Figures 6 in Appendix A
Nets comprise three types of blocks: residual basic block, residual bottleneck block, and transitional block (Figure 3t. Basic blocks contain two KNConv layers with kernel size of 2x2, whereas bottleneck blocks consist three KNConv layers with kernel sizes of 2x2, 3x3, and 2x2, respectively. The stride value both basic and bottleneck blocks is 1x1. The padding values of the first and last KNConv layers, however, are 1x1 and zero so that the width and height. of the output remain identical to the input's (necessary condition for residual blocks with identity shorteut). The middle KNConv layer in bottleneck blocks uses 1x1 padding, Transitional blocks include a KNConv layer with kernel size of 2x2 and stride of 1x1 to increase the number of filters, and max-pooling layer with kernel size and stride of 2x2 to downsample the input.
We propose the KNResNet- 18, KNResNet-334 and KNResNet-50 architectures based on the aforementioned block types (Figure 5 in Appendix A). KNResNet-18/ 18/4 uses basic and transitional blocks, while KNResNet- 50 mainly employs bottleneck and transitional blocks. For semantic segmentation, we utilize KNResNet- 18/34/50 as backbone (Figure 6 in Appendix [ but the kernel size of the KNConv and max-pooling layers in basic and transitional blocks is 3x3 instead of 2x2.
4 Evaluation
We compare the performance of KNResNets to the BatchNorm, GroupNorm, LayerNorm, and LocalCon- textNorm counterparls. For image classification, we do not include LocalContex(Norm in our evaluation because its performance is similar to GroupNorm (Ortiz et al.] 2020). The experimental evaluation is divided into four categorics: (I) batch sizo-dependent performance analysis, (II) image classification on Im- ageNet, (III) semantic segmentation on Cityscapes, and (IV) differentially privale image classification on ImageNet32233.
We adopt the original implementation of RosNet-18/34/50 from PyTorch Paszke et al. 2019 and the PreactResNet-115. 18/34/00 (He et al. 2016b) implementation from Kuang (021) The architectures are based BatchNorm. For GroupNorm/ LocalContextNorm related models, BatchNorm is replaced by Group- Vorn/LLoclConnext.Noo
Regarding LayerNorm based architectures, Group.orm with number of groups of 1 (equivalent to LayerNorm) is substituted for BatchNorm. The mumber of groups of GroupNorm is Wu & He| 2018). The number of groups and window size for LocalContextNorm are and 227, respectively (Ortiz et al. 2020).
For low-resolution datasets (CIFAR-100 and ImageNet32x32), with 3x3 convolutional layer and remove the following max- max-pooling layer. Moreover, insert normal- ization layer followed by an activation function before the last averago-pooling layer in the PreactResNet architectures akin to KNResNets (Figure 5 al Appendix A). The aforementioned modifications considerably enhance the accuracy of the competitors. For semantic segmentation, we employ the fully convolutional net- work architecture (Long al. (2015a) with BatchNorm, GroupNorm, LayerNorm, and LocalContext.oome based ResNet-18 18/4e/111/////00 as backbone. For KNResNets, we use fully convolutional versions of KNResNet- 18/34/50 (Figure 6 at Appendix A).
replace the first 7x7 convolutional layer
7

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Test accuracy versus batch size CIFAR-100.
Model
Normalization
Parameters
B-2
B-32
B-256
ResNet-18-LN
LayerNorm
11,220 M
72.68.10.22
73.17:10.16
71.99:10.15
PreactResNett-8-L.N
LayerNorm
11.220
73.5110.10
73.36.10.15
72.91
ResNet-18-GN
GroupNorm
11.220 M
74.62+0.12
74.46+0.05
74.46+0.08
PreactResNet-18-GG
GroupNorm
11.220 M
74.82++0.24
74.74+0..4
74.62+0036
ResNet-18-BN
BatchNorm
11.220 M
72.1110.22
78.5210.20
77.7210.04
PreactResNet-1--BN
BatchNorm
11.220 M
72.5710.19
78.32:10.09
77.83-1.16
KNResNet-18 (ours)
KernelNorm
11.216 M
79.10-0.10
79.29+0002
78.84+0.10
ResNet-3A-LNN
LayerNorm
21.328 M
73.74+00.6
73.88+0.37
72.48+0..7
PreactResNet-34-11
LayerNorm
21.328 M
74.79-0.133
74.34
73.10=0,42
ResNet-34-GG GN
GroupNorm
21.328 M
75.7610.14
75.72+0.06
75.44+0.27
PreactResNet-31-GN
GroupNorm
21.328 M
75.82.10.05
75.85.1.228
75.76110.25
ResNet-34-BN
BatchNorm
21.328
73.06.10.23
79.21
78.2710.19
PreactRes/Net-344BB
BatchNorm
21.328 M
72.20+0.19
79.09+0.03
78.59+0.24
KNRosNet-33 (ours)
KerneINorm
21.323 M
79.28-0.09
79.53:10.15
79.16:10.21
ResNet-50-LN
LayerNorm
23.705 M
75.8310..25
75.7410..1
74.37-0.558
PreactResNet-GO-LNN
LayerNorm
23.705 M
74.2810.31
74.57-10.32
73.4110.15
ResNet-50-GNN
GroupNorm
23.705 M
77.03+0.62
77.02+0.08
74.79+-0.14
PreactResNet-50-GM
GroupNorm
23.705 M
75.67=1.27
76.08
75.52+00113
ResNet-50-Br
BatchNorm
23.705 M
71.02:00.15
80.39
77. 77.89#0.06
PreactResNett-O-BNN
BatchNorm
23.705
70.83.0.41
80.28.10.15
18.881
KNRosNel-50 (ours)
KerneINorm
23.682 M
80.24
80.1810.110
80.09
4.1 Batch size-dependent performance analysis
Dataset. The CIFAR-100 dataset consists of 50000 train and 10000 test samples of shape 32x32 from 100 classes. We adopt the data preprocessing and augmentation scheme widely used for the dataset (Huang ct al. 2017a: He et al. 2016b a): Horizontally flipping and randomly cropping the samples after padding them. The cropping and padding sizes are 32x32 and 4x4, respectively. Additionally, the feature values are divided by 255 for KNResNets, whereas they are normalized using the mean and standard deviation (SD) of the dataset for the competitors.
Training. The models are trained for 150 epochs using the cosine annealing scheduler (Loshchilov Hutter 2017) with learning rate decay of 0.01. The optimizer is SGD with momentum of 0.9 and weight. decay of 0.0005. For learning rate tuning,
given experiment with initial learning rate of 0.2, divide it by 2, and re-run the experiment. We continue this procedure until finding the best lcarning rate (Table 5 in Appendix Then, we repeat the experiment three times, and report the mean and SD over the runs.
Results. Table [ lists the test accuracy values achieved by the models for different batch sizes. According to the table, (I) KNResNets dramatically outperform the BatchNorm counterparts for batch size of 2, (II) KNResNets deliver highly competitive accuracy values compared BatchNorm- based models with batch sizes of 32 and 256, and (III) KNResNets achieve significantly higher accuracy than the batch-independent competitors (LayerNorm and GroupNorm) for all considered batch sizes.
4.2 Image classification ImageNet
Dataset. The ImageNet dataset contains around 1.28 million training and 50000 validation images. Fol- lowing the data preprocessing and augmentation scheme from TorchVision (2023a), the train images are horizontally flipped and randomly cropped to 224x224. The test images are first resized to 256x256, and then center cropped to x24x224. The feature values are normalized using the mean and SD of ImageNet.
Training. We follow the experimental setting from Wu & He 2018) and use the multi-GPU training script from TorchVision (2023a) to train KNResNets and the competitors. We train all models for 100 epochs with total batch size of 256 (8 GPUs with batch size of 32 per GPU) using learning rate of 0.1, which divided by 10 at epochs 30, 60, and 90. The optimizer is SGD with momentum of 0.9 and weight decay of 0.0001.
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Image classification on ImageNett
Model
Normalization
Parameters
Top-1 accuracy
ResNet-18-LN
LayerNorm
11.690 M
68.34
ResNet-18-GN
GroupNorm
11.690 M
68.93
ResNet-18-BN
NarchNorm
11.690 M
70.28
KNResNet--18 (ours)
KerneLNorm
11.685 M
71.17
ResNet-34-LN
LayerNorm
21.798 M
71.64
ResNet-31-GN
GroupNorm
21.798 M
72.63
ResNet-34-BN
BatchNorm
21.798 M
73.99
KNResNett-34 (ours)
KerneLNorm
21.793 M
74.60
ResNet-50-LN
LayerNorm
25.557 M
73.80
ResNet-50-GN
GroupNorm
25.557 M
75.92
ResNet-50-BB
BatchNorm
25.557 M
76.41
KeResNNet-50 (ours)
KormelNorm
25.556 M
76.54
Results. Table demonstrates the Top-1 accuracy values ImageNet for different architectures. As shown in the table, (I) KNResNet-118 and KNRcsNet-34 outperform the BatchNorm counterparts by around 0.9% and 0.6%, respectively, (II) KNResNet-18/34/55 achieves higher accuracy (by about 0.6%-3.0%) than LayerNorm and GroupNorm based competitors, and (III) KNResNet-50 delivers almost the same accuracy as the batch normalized ResNet-50.
4.3 Semantic segmentation on CityScapes
Dataset. The CityScapes dataset contains 2975 train and 500 validation images from 30 classes, 19 of which are employed for evaluation. Following Sun al. (2019); Ortiz al. (2020). the train samples are randomly cropped from 2048x 1024 to 1024x512, horizontally llppeed, and randomly scaled in the range of [0.5, 2.0). The models tested the validation images, which are of shape 2048x 1024.
Training. Following Sun et al. 2019 Ortiz et al. (020), we train the models with learning rate of 0.01, which is gradually decayed by power of 0.9. The models are trained for 500 epochs using 2 GPUs with batch size of 8 per GPU. The optimizer is SGD with momentum of 0.9 and weight decay of 0.0005. Notice that we use SyncBatchNorm instead of BatchNorm the batch normalized models.
Table 3: Semantic segmentation CityScapes.
Model
Normalization
Parameters
mloU
Pixel accuracy
Mean accuracy
18-LN
LayerNorm
13.547 M
59.10+0.46
92.42+0.17
69.43+00558
ResNet-18-GN
GroupNorm
13.547 M
62.33:0.52
93.23:10.01
71.58:10.55
ResNet-11-LLCN
LocalContextNorm
13.547 M
62.25
92.99
71.59+40.68
ResNet-18-BN
BatchNorm
13.547 M
63.90+10006
93.7740.00
73.15
KNRosNet-118 (ours)
KernelNorm
13.525 M
64.37
93.73.10.01
73.46.10.12
ResNet-344LN
LayerNorm
23.655
60.19+0.32
92.73+0.11
70.1240.33
ResNet-34-GN
GroupNorm
23.655 M
64.2110.58
93.59
74.32-10.49
ResNee-33-LLCN
LocalContextNorm
23.655 M
04.75-0.38
93.3110.00
70.35
ResNet-334-BNN
Batch
23.655 M
66.9440.31
94.27+0.03
76.50+0.41
KNResNNet-34 (ours)
KernelNorm
23.399 M
67.61
94.13100.05
76.5810.19
ResNet-50-LN
LayerNorm
32.955 M
57.88-0.88
92.31-40.21
68.25
ResNet-50-GN
GroupNorm
32.955 M
62.1440.68
93.34+0.04
71.66+0.64
ResNet-50-LCN
Lora
32.955 M
64.03.1..02
93.07.1..14
73.40
ResNet-50-BBN
NatchNorm
32.955 M
65.19+0..5
93.98+00.03
74.65+00.2
KNResNet-50 (ours)
KernelNorm
32.874 M
68.02-10.13
94.22-10.04
77.03-00.05
9

---

Published in Transactions on Machine Learning Research (02/2024)
Results. Table 3 lists the mean of class-wise intersection over union (mloU), pixel accuracy, and mean of class-wise pixel accuracy for different architectures. According to the table, (T) KNResNet-18/ and the BatchNormm-ased counterparts achieve highly competitive mloU, pixel accuracy, and mean accuracy, whereas KNRosNet-500 delivers considcrably higher mIoU and mcan accuracy than batch normalized ResNet-500, (II) KNResNets significantly outperform the batch-inddepeneet competitors (the LayerNorm, GroupNorm, and LocalContextNorm based models) terms of all considered performance metrics. Surprisingly, ResNet-50 based models perform worse than ResNet-34 counterparts for the competitors possibly because of the smaller kernel size they employ in ResNet-50 compared to ResNet-34 (1x1 instead of 3x3).
4.4 Differentially private image classification on ImageNet32x32
Dataset. ImageNet32x332 is the down-sampled version of ImageNet, where all images are resized to 32x32. For preprocessing. the feature values are divided by 255 for KNResNet-18, while they normalized by the mean and SD of ImageNet for the layer and group normalized ResNet-18.
Training. We train KNResNet-18 as well as the GroupNorm and LayerNorm counterparts for 100 epochs using the SGD optimizer with zero-momentum and zero-wiight decay, where the learning rate is decayed by factor of 2 at epochs 70, and 90. Note that BatchNorm is inapplicable to differential privacy. All models use the Mish activation (Misra 2019h. For parameter tuning, we consider learning rate values (2.0, 3.0, 4.0), clipping values of {1.0, 2.0}, and batch sizes of {2048, 4096, 8192]. We observe that learning rate of 4.0, clipping value of 2.0, and batch size of 8192 achieve the best performance for all models. Our differentially
privale training is based DP-SGD (Abadi et al., 2016) from the Opacus library YYouselpour et al. 2021 with 8=8.0 and 8-8x10-7, The privacy accountant is RDP (Mironov 2017
Table 4: Differentially private image classification on ImageNeet32x32.
Model
Normalization Parameters
Top-1 accuracy
ResNet-18-BN
BatchNorm
11.682 M
NA
ResNet-18-LN
LayerNorm
11.682 M
20.81
ResNeet-8-GN
GroupNorm
11.682 M
20.99
KNResNet-18 (ours)
KernelNorm
11.678 M
22.01
Results. Table f lists the Top-1 accuracy values on ImageNet32x32 for different models trained in the aforementioned differentially private learning setting. As can be seen in the table, KVResNet-188 achieves significantly higher accuracy than the layer and group normalized R8sNet-18.
5
Discussion
KNResNets incorporate only indeeedendd layers such as the proposed KernelNorm and KNConv layers into their architectures. Thus, they perform well with very small batch sizes (Table 1 and are applicable to differentially private learning (Table 4 which are not the case for the batch normalized models. Unlike the batch-independent competitors such as LayerNorm, GroupNorm, and LocalContext based NesNets, KNResNets provide higher or very competitive performance compared to the batch normalized counterparts in image classification and semantic segmentation (Tables 3 Moreover, KNResNets converge faster than the batch, layer, and group normalized ResNets in non-private and differentially private image classification as shown Figure These results verify our key claim: the kernel normalized models combine the performance benefit of the batch normalized counterparts with the batch-independence advantage of the layer, group, and local-context normalized competitors.
The key property of kernel normalization is the overlapping normalization units, which allows for kernel normalized models to extensively take advantage of the spatial correlation among the elements during nor- malization. Additionally, enables KernelNorm to combined with the convolutional layer effectively as a single KNConv layer (Equation 5 and Algorithm D. The other normalization layers lack this prop- erty. BatchNorm, LayerNorm, and GroupNorm are global normalization layers, which completely ignore the spatial correlation of the elements. LocalContext.Nomm partially considers the spatial correlation dur-
10

---

Published in Transactions on Machine Learning Research (02/2024)
BatchNorm
GroupNorm
LayerNorm
KemelNorm
4
80
Test accuracy (%)
3
Train loss
60
2-
1
40
20
25
50
75 Epoch
100
125
150
25
50
75 Epoch
100
125
150
(a) CLFAR-100 MAA---10-Resshee -0--0 (B-2)
Test accuracy (%))
70
3.
Train loss
60
2-
40
20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
(b) Netggeeet--
(B=256)
4
80
Test accuracy (%)
Train loss
60
2.
50
40
20
40
60
80
100
20
60
80
100
Epoch
Epoch
(c) ImaggeNet
(B-256)
20
Test accuracy (%)
Train loss
15
10
0
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
(d) ImageNet32x33 Re- (s=8.0, 8 6-8x10-?)
Figure 4: Convergence rate of the models for different case studies:
Kernel normalized models converge faster than the competitors. Notice that BatchNorm is inapplicable to differential privacy; B: batch size.
11

---

Published in Transactions on Machine Learning Research (02/2024)
ing normalization because it has no overlapping normalization units, and must use very large window sizes to achieve practical computational efficiency. Our evaluations illustrate that this charact eristic of kernel normalization lead to significant improvement in convergence rate and accuracy achieved by KNResNets.
Normalizing the feature values of the input images using the mean and SD of the whole dataset is a popular data preprocessing technique, which enhances the performance of the existing CNNs due to feeding the normalized values into the first convolutional layer. This is unnecessary for KNConvNets because all KNConv layers inchuding the first one are self-normalizing (they normalize the imput first, and then, compute the convolution). This makes the data preprocessing simpler during training of KNConvNets.
Compared to the corresponding non-normalized networks, the accuracy gain in KNResNets originates from normalization using KernelNorm and regularization effect of dropout. To investigate the contribution of each factor to the accuracy gain, we train KNResNet-550 on CIFAR-100 with batch size of 32 in three cases: (I) without KerneINorm, (II) with KerneINorm and without dropout, (III) with KernelNorm and dropout. The models achieve accuracy values of 71.48%, 78.32%, and 80.18% in (I), (II), and (III), respectively. Given that, normalization using KernelNorm provides accuracy gain of around 7.0% compared to the non-normalized model. Regularization effect of dropout delivers additional accuracy gain of about 2.0%.
Prior studies show that normalization layers can reduce the sharpness of the loss landscape, improving the generalization of the model (Lyu et al., 2022 Keskar et al. 2016). Given that, we train LayerNorm, GroupNorm, and BatchNorm based ResNet- 18 as well as KNResNet-18 on CIFAR-10 to compare the gen- eralization ability and loss landscape of different normalization methods (experimental details in Appendix Ch. The layer, group, batch, and kernel normalized models achieve test. accuracy of 90.32%, 90.58%, 92.11 1%, 93.27%, respectively. Figure 7 (Appendix C visualizes the loss landscape for different normalization layers. According to the figure, KNResNet-18 provides flatter loss landscape compared to batch normalized ResNet- 18, which in turn, has smoother loss landscape than the group and layer normalized counterparts. These results indeed indicate that KNResNet-18 and BatchNorm-basel ResNet-18 with flatter loss landscapes provide higher generalizability (test accuracy) than LayerNorm/ SogeerNorn//rouNooonnn /rroupNorm based ResNet-18.
There is a prior work known as convolutional normalization (ConvNorm) (Liu et al., 2021), which takes into account the convolutional structure during normalization similar to this study. ConvNorm performs normalization on the kernel weights of the convolutional layer (weight normalization). Our proposed layers, on the hther hand, normalize the input tensor (input normalization). In terms of performance on Image the accuracy of KNResNet-II 18 is higher than the accuracy of the HanNNorm-BatchNNonn based ResNet-18 reported in Liu et al. 2021 (71.17% vs. 70.34%).
We explore the effectiveness of KernelNorm on the ConeNert architecture (Liu et al. 2022 in addition to ResNets. ConvNext is a convolutional architecture, but it is heavily inspired by vision transformers (Dosovitskiy et al., 2020), where it uses linear (fully-connected) layers extensively and employs LayerNorm the normalization layer instead of BatchNorm. To draw the comparison, train the original ConuNeatTing model from PyTorrch and the corresponding kernel normalized version (both with around 28.5m parameters) on ImageNet using the training recipe and code from TTochVision (202bb) (more experimental details in Appendix The original model, which is based on LayerNorm, provides accuracy of 80.87%. The kernel normalized counterpart, on the other hand, achieves accuracy of 81.25%, which is 0.38% higher than the bascline. Given that, KernelNorm-basse models are efficient not only with ResNets, but also with more recent architectures such as ConvNext, which incorporates several architectural elements from vision transformers into convolutional networks.
We also make comparison between KNResNets and the Norm- counterparts from the computa- tional efficiency and memory usage perspectives (Tables 0 and 7 in Appendix D). For the batch normalized models, we employ two different implementations of the Batch layer: The CUDA implementation (Py- Torch 2023a] and the custom implementation (D2L) 2023) using primitives provided by PyTorch. Because the underlying layers of KNResNets (i.e. KernelNorm and KNConv) are implemented using primitives from PyTorch, we directly compare KNResNets with ResNets based on the latter implementation of NorchNorm to have fair comparison. According to Table G KNeesNett-50 (our largest model) is only slower than batch normalized ResNet-50 by factor of 1.66. This slowdown is acceptable given the fact that KernelNorm is a local normalization layer with much more normalization units than BatchNorm as a global normalization
12

---

Published in Transactions on Machine Learning Research (02/2024)
layer (Figure D. The CUDA-based implementation of BatchNorm, morcover, is faster than that based primitives from PyTorch by factor of 1.8. We can expect a similar speedup for KNResNets if the underlying layers are implemented in CUDA. Additionally, the memory usage of KNRRsNets is higher than the Batch- Norm counterparts as expected, which relates to the current implementation of the KNConv layer (more details in Appendix Dj. Notice that the most efficient implementation of KNResNets is not the focus of this study, and is left as a future line of improvement. Our current implementation, however, provides enough efficiency that allows for training KNResNet-18/34/// on large datasets such as ImageNet.
6 Conclusion and Future Work
BatchNorm considerably enhances the model convergence rate and accuracy, but it delivers poor perfor- mance with small batch sizes. Moreover, it is unsuitable for differentially private learning due to its depen- dence on the batch statistics. To address these challenges, we propose two novel batch-independent layers called KernelNorm and KNConv, and employ them as the main building blocks for KNConvNets, and the corresponding residual nelworks referred to as KNResNets. Through exlensive experimentation; we show KNResNets deliver higher or very competitive accuracy compared to BatchNorm counterparts in image clas- sification and semantic segmentation. Furthermore, they consistently outperform the batch-independent counterparts such as LayerNorm, GroupNorm, and LonalContex(Norm in non-private and differentially private learning settings. To our knowledge, our work is the first to combine the batch-independence of LayerNorm/ Coarroorm/Grrupple CoooupNorm/LooalCCoeeeee of convolutional networks.
with the performance advantage of BatchNorm in the context
The performance invostigation of KNResNets for object detection, designing KNConvNets corresponding to other popular architectures such as DenseNets (Huang al. 2017a),, and optimized implementations of KerneNNorm and KNResNets in CUDA are promising directions for future studies.
Acknowledgement
We would like to thank Javad Torkzadeh.Mahrd for assisting with the implementations and helpful discus- sions on the computationalll--ffiiieet version of the kernel normalized convolutional layer. We would also like to thank Sameer Ambekar for his helpful suggestion regarding fairer comparison among the normalization layers from the computational efficiency perspective.
This project was funded by the German Ministry of Education and Research as part of the PrivateAIM Project, by the Bavarian State Ministry for Seience and the Arts, and by the Medical Informatics Initiative. The authors of this work take full responsibility for its content.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. Tn Proceedings of the 2016 ACM SIGSAC conference on compuler and communications security, pp. 308 318, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. 16Xi:16677.0650 2016.
Layer normalization. ar preprint
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations training deep networks? Advances in Neural Information Processing Systems, 31, 2018
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE tronsactions on neural networks, 5(2):157- 166, 1994.
AB Bonds. Role of inhibition in the specification of orientation selectivity of cells in the cat striate cortex. Visual neuroscience, 2(1):41-55 55, 1989.
Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation close the performance gap in unnormalized resnets. arXiv preprint or Xrv: 2021a.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recog- nition without normalization. In International Conference on Machine Learning, pp. 1059 1071. PMT.R, 2021b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiclc. The cityscapes datasct for semantic urban scene understand- ing. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 3213 -223,, 2016.
D2L. Batch normalization.
http://w....aapaaaooooooooaaaaar html 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sei., 9:211-407, 2014.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249- 256. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Springer, 2016b.
European conference on compuler vision, pp.
David J Heeger. Normalization of cell responses in cat striate cortex. Visual neuroscience, 9(2):181-197, 1992.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017a.
Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, and Dacheng Tao. Centered weight normalization in acceler- ating training deep neural networks. In Proceedings of the IEEE International Conference on Compuler Vision, pp. 2803 22112 2017b.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 PMLR, 2015.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang, On latch-batch training for deep learning: Generalization gap and sharp minima. In International Con- ference on Learning Representations, 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Πlya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Liu Kuang, Pytorch models for ciafr-10/100. http:///ggttuuu.ooo/o/o/a/ag/agr
2021.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 (4):541-55) 1989.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Boyi Li, Felix Wu, Kilian Q Weinberger, and Serge Belongie. Positional normalization. Advances in Neural Information. Processing Systems, 32, 2019.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of 2018b.
neural nets.
Sheng Liu, Xiao Li, Yuexiang Zhai, Chong You, Zhihui Zhu, Carlos Bernandez-Granda and Qing Qu. Convo- lutional normalization: Improving deep convolutional network robustness and training. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Sys- tems, 2021.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19976-1986,, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015a.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceediings of the IEEE conference on compuler vision and pallern recognilion, pp. 3431 34400, 2015b.
Tlya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Kaifeng Lyu, Zhiyuan Li, and Sanjecv Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. Advances in Neural Information Processing Systems, 35:34689 34708, 2022.
Πya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp. 263 275. TEEE, 2017.
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:190S.0865 2019.
Anthony Ortiz, Caleb Robinson, Dan Morris, Olac Fuentes, Christopher Kiekintveld, Md Mahmudulla Has- san, and Nebojsa Jojic. Local context normalization: Revisiting local normalization. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, pp. 11276 11285, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- begh-performance deep learning library. Advances in Newral Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Py Batch normalization. http:////woooooooo////////////aa mn. BatchNorm2d. hat.r//aaeeeeerrrooo.aa.accaaana
html 2023a.
PyTorch. Unfold operation pytorch. Unfold.html 2023b.
PyTorch. var mean function in pytorch. http:///yooccoooo/oeessslllaaocc var
mean. html 2023c.
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In International conference on machine learning, pp. 7824 7835. PMLR, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint Xrx: 10520, 2019.
Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, and Richard S. Zemel. Normalizing the normal-
izers: Comparing and extending network normalization schemes. In International Conference on Learning Representations, 2017.
Tim Salimans and Diederik P Kingma. Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In Proceedings of the 30th International Conference on Neural Informa- lion Processing Systems, pp. 901-909, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyss and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018.
Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In 2nd International Conference on Learning Represenlations, ICLR 2014, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhovsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958 2014.
Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang. High-resolution representations for labeling pixels and regions. arXiv preprint 1eXic:1904.055. 2019.
TorchVision. Classification training script in pytorch. http:://ggtthb..caao/rrrr//////r
cefarsaooe//Caassfffaaaittttreee 2023a.
TorchVision. Classification training script in pytorch. http:///sttub..oo//ooorr/////s/f
aetreenceeccassfffccctttoooaarre 2023b.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprinl Xrxi 1XX::06070 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural net- works. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11505-11515, 11515, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on compuler vision (ECCV), pp. 3-19, 2018
Ashkan Yousefpour, Tgor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and llya Mironov. Opa- cus: User-friendly differential privacy library in Py Torch. arXiv preprint ar 200: 12298, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2024)
A
KNResNet Architectures
IoNCam Sxt
KoNCom-
ReLu
RULL
ReLU
Basic block
Bottleneck block
Transitional block
(a) KNResNet blocks (image classification)
Transitional Block
Transiioonal Blook
Transitional Block
Maxc-po0--30
Basic Block
Block
Basic Block
KemelNarm
KN
256
256
512
512
724
MMxx-pool-22 pool.poo...o
724
ReL
KNConv
ReL
512
Aug-pon
Lineal
(b) KNResNet-18 (image classification)
4X
5X
3x
2X
Transitional Block
Transiiional Blook
Transitional Block
KNConn- xx
82
Max-pool-3x3
Basic Block
64
256
Basic Block
256
512
Basic Block
512
512
512
Max.poool22
512
ReLu
512
KNCCnv 2x2
512
KereeNNom
612
ReL
512
Avg-2oal
512
Linear
Bosic
(c) KNResNet-34 (image classification)
4X
SX
4X
2X
Botteneck Block
Transitonnl Block
Bottleneck Block
Transitional Block
Boitleneck Block
Transilional Block
Bottleneck Block
KerneNaarm
MMax-pool-3id
Avg-pool
KNConv-
256
512
810
2048
2048
2048
Lineer
ReL
(d) KNResNet-550 (image classification)
Figure 5: KNRcsNets for image classification: The dropout probability of the KNConv and KernelNorm layers are 0.05 and 0.25, respectively. For low-resolution images (e.g. CIFAR-100 with image shape of x2x)2,, the first KNConv layer is replaced by a KNConv layer with kernel size 3x3, stride 1 1x1, and padding 1x1. and the following max-pooling layer is removed. The kX (k=2/3/44f5 notation above the blocks means k blocks of that type. The numbers above arrows indicate the number of input/one /oupput channels the first/ KNConv layer in the block. For KNResNet--18, the number of the output channels of the first. KNConv layer (or the number of input channels of the second KNConv layer) is 256, 256, 512, and 724 for the first, second, third, and fourth set of basic blocks, respectively. For KNResNet-23 KNResNet-50, the number of output channels of the first and second KNConV layers 512 in the first, second, third, and fourth set of bottleneck blocks, respectively. In KNRosNet-500, the last transitional block and the last set of residual blocks use KNConvlxI instead of KNConv2x2 to keep the number of parameters comparable to the original ResNet-50.
is 256, 320, 640, and 843. For
64, 128, 201, and
17

---

Published in Transactions on Machine Learning Research (02/2024)
KNCum-3e
KNCum-30
KNCmne-z
PalL
PaLU
PulL
ReUU
Basic block
Bottleneck block
Transitional block
(a) KNResNet blocks (semantic segmentation)
2X
Blook
Tramsilional Block
Transitional Block
Basic Block
Basic Block
Basic Block
Basic Block
KerneNNom
Max-pool-30
64
128
128
256
512
512
ReLu
(b) KNResNett-18
(semanticc segmentation)
4X
4X
4X
3x
Transitional Block
Transitional Block
Transitional Block
Max-ppol-333 -x3
Bosic Block
Bosic Block
Basic Block
Bosic Block
KemelNorm
KNConv-T
64
128
128
256
512
512
ReLU
612
(c) KNResNet-34
(semantic segmentation)
4X
4X
3X
Bottleneck Block
Transitional Block
Bottleneck Black
Transitional Blook
Bottleneck Block
Transitional Blook
Bottleneck Block
KNConV-TX
Max-pool- 3x3
KNConv3x3
Keme/Norm
64
128
128
256
256
512
512
512
512
Conv-Ix 1x1
ReLu
ReLU
(d) KNResNet-500 (semantic segmentation)
Figure 6: KNResNets for semantic segmentation: The dropout probability of the KNConv and Kernel- Norm layers are 0.1 and 0.5, respectively. For KNRes.et--18, the number of the output channels of the first KNConv layer (or the number of input channels of the second KNConv layer) is 128, 256, 512, and 625 for the first, second, third, and fourth set of basic blocks. For KNResNet-34,, they are 128, 256, 256, and 512, respectively. For KNRosNet-50, the number of input/output channels of the middle KNConv layer are 128, 256, 458, and 512 for the first, second, third, and fourth set of bottleneck blocks. Unlike their counterparts for image classification, the KNConv and max-pooling layers in basic and transitional blocks employ kernel size of 3x3 instead of 2x2.
18

---

Published in Transactions on Machine Learning Research (02/2024)
B Reproducibility
Table 5: Learning rate values achieving the highest accuracy CIFAR-100.
Model
Normalization
B=2
B-32
B-256
ResNet-18-LN
LayerNorm
0.0015625
0.0125
0.05
PreactResNNet-8..0 18-LN
LayerNorm
0.0015625
0.0125
0.05
ResNet-18-GN
GroupNorrm
0.0015625
0.025
0.1
PreactResNNtt18-GG
GroupNorm
0.0015625
0.025
0.1
RosNet-18-BN
BarchNorm
0.00078125
0.025
0.2
PreactResNNet-18-B
BatchNorm
0.00078125
0.025
0.2
KNResNNt-18
KernelNorm
0.0015625
0.05
0.2
ResNet-34-LNN
LayerNorm
0.0015625
0.0125
0.05
PreaclResNet-34LL
LayerNorm
0.0015625
0.0125
0.05
ResNet-34-GN
GroupNorm
0.0015625
0.025
0.1
NretccResNNtt-3I-GNN
GroupNorm
0.0015625
0.025
ResNet-34-BN
BatchNorm
0.00078125
0.025
0.1
333-44B
BatchNorm
0.000390625
0.025
0.2
KNResNNt-33
KernelNorm
0.0015625
0.05
0.2
ResNet-50-LN
LayerNorm
0.00078125
0.0125
0.05
PreactResNet-50-IN
LayerNorm
0.0015625
0.0125
0.05
ResNet-50-GNN
GroupNorm
0.00078125
0.0125
0.05
PreactRcsNet-50-GN
GroupNorm
0.0015625
0.025
0.1
ResNet-50-BN
BatchNorm Norm
0.000390626
0.0125
0.1
PreactResNet-50BBN
BarchNorm
0.000195313
0.0125
0.2
KNRosNot-50
KernelNorm
0.0015625
0.025
0.2
ConvNext on ImageNet: To train the LayerNorm and KernelNorm based ConvNextTing models on ImageNet, we employ the code and recipe from TorchVision (2023b), where the models are trained with total batch size of 1024 using the AdamW optimizer, learning rate of 0.001, and cosine learning rate scheduler for 600 epochs. Note that of 128 per GPU in the original recipe due to the resource limitation.
4 GPUs with batch size of 256 per GPU rather than GPUs with batch size
19

---

Published in Transactions on Machine Learning Research (02/2024)
C
Loss Landscape
5
100
100
4-
80
3
60
Accuracy
Accuracy
LOSs
Loss
2
1
20
1.00 0.75
0.50
0.25
0.25
0.50
0.75
1.00
1.00
0.75
0.50
0.25 0.00
0.25
0.50
0.75
1.00
(a) LayerNorm
(b) GroupNorm
5
100
100
80
3
60
Accuracy
Accuracy
LOSS
Loss
2
40
1
20
-1.00-0.55 0.00 0.25 0.50
1.00
-1.00 -1.0-00.5-0050-0000 0.00 0.25 0.50
1.00
(c) BatchNorm
(d) KernelNorm
Figure 7: Loss landscape of different normalization layers: Kernel normalized ResNet-18 has flatter loss landscape compared to the batch, group, and layer normalized counterparts CIFAR-10.
ResNet-18 CIFAR-10: To compare the generalization ability and loss landscape of different normaliza- tion layers, we train BatchNorm, GroupNorm, LayerNorm, and KernelNorm based ResNet-18 on CIFAR-10 All models are trained for 70 epochs using batch size of 128 and tuned over learning rate values of (0.05, 0.1}. The weight decay is zero. The optimal learning rate is 005/0.05/0.1/000 for ayyer/grou//batee batchhkernel /ernel nor-
malized ResNet-18. The preprocessing and augmentation scheme and the other training settings are the same as the CIFAR-100 experiments Section We employ the source code from Li et al. 2018 b to visualize the loss landscape Figure
20

---

Published in Transactions on Machine Learning Research (02/2024)
D Running Time and Memory Usage
Table 6: Training and inference time per epoch for ImageNet: The experiments are conducted with 8
NVIDIA A40 GPUs with batch size of 32 per GPU; m: minutes, s: seconds.
Model
Normalization
Implementation
Training time
Inference time
ResNet-50-BN
Batch
CUDA
13m 23s
6s
ResNet-50-BN
BatchNorm
Primitives from PyTorrc Primitives from PyTorch
23m 49s
10s
KNResNet-50 (ours)
KernelNorm
39m 33s
19s
ResNet-34-BN
BatchNorm
CUDA
9m 12s
5s
ResNet-34-BN
BatchNorm
Primitives from PyTorch Primitives from PyTorch
12m 46s
58
KNResNet-34 (ours)
KernelNorm
27m 15s
12s
ResNet-18-PB
BatchNorm
CUDA
5m 28s
4s
ResNet-18-BN
BatchNorm
Primitives from PyTorch
7m 46s
4s
KNRosNet-18 (ours)
KerneINorm
Primitives from PyTorch
13m 58s
7s
Table 7: Memory usage on ImageNet: The experiments are conducted with single NVIDIA RTX A6000 GPU with batch size of 32; GB: Gigabytes.
Model
Normalization
Implementation
Memory usage (GB)
Res.Net-50-BN
BatchNorm
CUDA
5.7
Res.Net-5O-BN
BatchNorm
Primitives from PyTorch Primitives from PyTorch
8.2
KNResNet--50 (ours)
KernelNorm
13.6
ResNNet334-BN
BatchNorm
CUDA
3.6
ResNNet-34BBN
BatchNorm
Primitives from PyTorch Primitives from PyTorch
4.4
KNResNet--34 (ours)
KernelNorm
9.4
ResNet-18-BN
BatchNorm
CUDA
3.2
ResNeet-1-BBN
BatchNorm
Primitives from PyTorch Primitives from PyTorch
3.7
KNResNet-118 (ours)
KernelNorm
7.2
The memory usage of KNResNets is higher than the BatchNorm counterparts. This observation is related to the current implementation the KNConv layer, where the unfolding operation is performed in the kn mean var function (Algorithu 1) to compute the mean and variance of the units. We implemented KNConv in this fashion avoid changing the CUDA implementation of the convolutional layer, which requires huge engincering and implementation effort, and is outside the scope of our expertisc.
In hypothetical implementation of KNConv in CUDA, it would be possible to compute the mean/variance of the units directly inside the convolutional layer, and completely remove the kn mean var function, leading to substantially reducing the memory usage. This is because the units to compute convolution and mean/ mean/variance vaaiiance are the same, and those units are already available the convolutional layer implementation.
Table 8: Inference time memory usage for different stride, width (W) and height (H) values. The experiments are carried out with a single NVIDIA RTX A6000 GPU using batch size of 256 on the test set of CIFAR-100. The model contains four KNConv layers with kernel size of 3x3 and 256 channels; s: seconds, GB: Gigabytes.
W/H-32x32
W/H-64x64
W/H-128x 128
Stride=1x1 1x
2.44s
2.80GB
8.43s
4.94GB
33.45s
13.44GB
Stride=2x2
0.64s
2.24GB
1.07s
2.72GB
2.91s
4.59GB
Stride=3x3
0.58s
2.16GB
0.79s
2.40GB
1.71s
3.28GB
21


Reviews:
[{"review_id": "Yr5RWJ5CeL", "reviewer": "Reviewer_awgw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) This is definitely an interesting operator that can potentially replace layernorm or groupnorm in ConvNets. It aligns more closely with the design of convolutional layers and leverages the spatial correlation inherent in the network.\n\n(+) The writing is clear, straightforward, and easy to understand.\n\nHowever, the main weakness lies in the design of the experiments. From my understanding, this manuscript only presents the performance of the proposed method alongside previous ones. While this effectively showcases the capabilities of the proposed method, the information provided is not particularly insightful. More specifically:\n\n(-) The manuscript lacks discussion on the use of dropout. Unlike other input normalizations, this method employs a dropout layer before computing the means. While this approach is logical, there are no experiments demonstrating its impact on performance.\n\nIn the experiments, it appears that the method performs better with smaller datasets like CIFAR and in the early stages of training (as seen in Figure 4). This outcome is often observed with larger regularization factors (such as dropout, higher weight decay). Hence, it's challenging to determine if dropout plays a more significant role than expected in these experiments.\n\n(-) Practically speaking, the training time is excessively long. It is standard practice to implement the proposed operator efficiently; failing to do so could substantially diminish the paper's contribution.\n\n(-) There is no discussion on the additional memory usage. The Kernel norm layer effectively duplicates overlapping regions of an input tensor, which should increase GPU memory consumption. The authors should provide a comparison of GPU memory usage for different kernel shapes. When the stride is small and the kernel size is large, the additional memory requirement could be even more significant. This aspect is crucial as it also influences how researchers might design the kernel shape when utilizing the proposed method.\n\n(-) Similarly, a discussion on the running time (at test time) is necessary. By the same logic that affects memory usage, the impact on running time should also be evaluated and discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new normalization method for Convolutional Neural Networks (ConvNets) termed 'kernel normalization.' This method calculates the mean and variance across a spatial region determined by the convolutional kernel's width, height, and input channels. The authors present empirical verifications in image classification, semantic segmentation, and differential privacy. They demonstrate that classification performance remains unaffected by a reduced training batch size, while performance in the other two tasks shows improvement.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes:\n\n1. Ablation study on the usage of dropout layer.\n\n2. The extra memory / runtime for the proposed method.\n\n3. Extra memory / inference speed as a function of kernel stride / width / height.\n\nAnd the inline citation format is wrong. The authors might mix the usage of \\citet and \\citep.\n\nMissing related work on weight normalization:\n\n[1] Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?, NIPS 2018.\n\n[2] Orthogonal Convolutional Neural Networks, CVPR 2020.\n\n[3] Deep Isometric Learning for Visual Recognition, ICML 2020.", "broader_impact_concerns": "Not Applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K7fuLaaYHj", "reviewer": "Reviewer_vx11", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is easy to follow, authors first point out the limitation of BatchNorm, then they propose a new normalization method, which can overcome the drawback of BN under the setting that BN fails. The empirical experiment verifies their proposed method, which can tackle the problem of BN when combat with small batch size and under differentially private training. Besides, the computation cost of the suggested method is well estimated, with acceptable running time (and could be further improved as the author discusses).\n\nHere are some of my concerns that relate to the paper:\n1. Author indicates that overlap normalization can \"extensively take into account the spatial correlation among the elements\". In traditional CNN, the convolutional layers (Convs) already have this trait, so what is the advantage of overlap normalization in the proposed method over Convs in this case?\n2. LayerNorm achieves better performance in ViT architecture, therefore comparing the proposed method with LayerNorm under ResNet architecture should not be fair.\n3. Generalization ability: The normalization layer, besides helping the model learn more stable and achieve better performances, also indicates that helps improve the generalization of the learned model [1]. So it will be better if the authors can propose some compared results about the generalization performance of their methods with other normalization techniques (for example, the generalization could be clarified through visualizing the loss landscape of the learned model).\n\n**Reference**\n\n[1] Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction. NeurIPS 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the batch-dependent limitation of Batch normalization (the model with BN layers could crash when learned on small batch size, and is not suitable under differentially private training) by introducing a new local, batch-independent normalization (KernelNorm). Authors further design kernel-normalized convolutional networks (KNConvNets), which are based on KernelNorm and can achieve competitive or higher performance of BN and other normalization techniques (LayerNorm, GroupNorm) under different settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to my concerns", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t1dQGlGUE3", "reviewer": "Reviewer_w1bf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The introduced concept of Kernel Normalized ConvNets is novel to the best of my knowledge \n- The presented results are convincing, where the proposed method outperforms the compared methods\n- The paper is well-written and easy to follow \n\n## Weaknesses\n- The evaluation is somewhat limited. The authors mainly study ResNet architectures. Hence it is not clear how generally applicable the kernel normalization is or if the authors found a technique that boosts only ResNet architectures. I suggest that the authors demonstrate the superiority of kernel normalization on architectures like DenseNet, NasNet, MobileNet, EfficientNet, etc. \n- The authors claim, that the proposed kernel normalization improves DP, however, the DP evaluation is very limited. The authors should elaborate on why the DP performance is relatively low, and how DP was evaluated. I also recommend aligning with a common DP evaluation strategy from the literature instead of simply stating the Top-1 accuracy.  \n- The training times of the proposed method are a possible bottleneck for practitioners.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Kernel Normalized Convolutional Networks (KNConvNets), which aim to overcome the limitations of BatchNorm in convolutional neural networks (CNNs). It proposes two batch-independent layers, KernelNorm and KNConv, which take into account spatial correlations during normalization. These layers are integrated into KNConvNets and corresponding residual networks (KNResNets) to achieve competitive or superior performance compared to BatchNorm in image classification and semantic segmentation, even with small batch sizes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the points in the weakness section. \n\n- Evaluation of normalized kernel on models beyond ResNet\n- More rigorous elaboration on DP, KNResNet benefit over other DP methods, and extended evaluation of DP.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All in all, this is a well written piece of work that shows a novel and interesting way to do normalization in conv-nets. The authors show that it is competitive on a reasonable range of tasks and the disadvantages pointed out by the reviewers do not seem insurmountable given the potential benefit this method could bring (in cases where other normalization techniques like batch norm are not working too well).
Decision: Accept (poster)