Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
Authors: Sheng Liu
Xiao Li
Yuexiang Zhai
Chong You
Zhihui Zhu
Carlos Fernandez-Granda
Qing Qu
Venue: NeurIPS 2021 Poster (2021)
Keywords: network architecture design
convolutional sparse coding
weight normalization

Abstract: Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets). In particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convolutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. In contrast, we introduce a simple and efficient ``Convolutional Normalization'' (ConvNorm) method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to be conveniently incorporated into any ConvNets. Our method is inspired by recent work on preconditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), we show that the ConvNorm improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via numerical experiments on CIFAR and ImageNet. Our implementation is available online at \url{https://github.com/shengliu66/ConvNorm}.

Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
Sheng Liu"
Xiao Li
Yuexiang Zhai UC Berkeley ysz@berkeley.co edu
Chong You Google Research cyou@google. com
New York University shengliudnyu. edu
University of Michigan xlxiao@umi ch. edu
Zhihui Zhu
Carlos Fernandez-Granda
Qing Qu
University of Denver zhihui zhu@du. edu
New York University cfgrandaScims nyu. edu
University of Michigan qingquBumich edu
Abstract
Normalization techniques have become a basic component in modern convolutional neural networks (ConvNetts). particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convo- lutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. contrast, we introduce simple and efficient "Convolutional Normal- ization" (ConvNorm) method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to conveniently incorporated into any ConvNets. Our method is inspired by recent work pre- conditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), show that the ConvNorm improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via numerical experiments on CIFAR and ImageNet. Our implementation is available online at https: github......... com coan/shngllii iu66/ /ConvVorn.
show that ConvNorm
Introduction
In the past decade, Convolutional Neural Networks (ConvNets) have achieved phenomenal success in many machine learning and computer vision applications [1-7]. Normalization is one of the most important components of modern network architectures [8]. Early normalization techniques. such as batch normalization (BrtchNorm) [4], are cornerstones for effective training of models beyond few layers. Since then, the values normalization for optimization and learning is extensively studied, and many normalization techniques, such as layer normalization [9]. instance normalization [10], and group normalization [11] are proposed. Many of such normalization techniques are based on estimating certain statistics of neuron inputs from training data. However, precise estimations of the statisties may not always be possible. For example, BatchNorm becomes ineffective when the batch size is small [12], batch samples statistically dependent [13].
'The first two authors contributed to this work equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Weight normalization [14] is powerful alternative to BatchNorm that improves the conditioning of neural network training without the need to estimate statistics from neuron imputs. Weight normalization operates by either reparameterizing or regularizing the network weights so that all the weights have unit Euclidean norm. Since then, various forms of normalization for network weights are proposed and become critical for many tasks such as training Generative Adversarial Networks (GANs) [15] and obtaining robustness to input perturbations [16, 17J. One of the most popular forms of weight normalization is enforcing orthogonality, which has drawn attention from diverse range of research topics. The idea is that weights in each layer should be orthogonal and energy-preserving Orthogonality is argued to play central role for training ultra-deep models [18-22], optimizing recurrent models [23-26], improving generalization [27], obtaining robustness [28, 29], learning disentangled features [30,31], improving the quality of GANs 23], learning low-dimensional emhedding [34], etc.
Exploiting convolution structures for normal- ization. Our work is motivated by the pivotal role of weight normalization in deep learning. In the context of ConvNets, the network weights multi multi-dimensional (e.g., 4-dimensional for a 2D ConvNet Net) convolutional kernels. A vast major- ity of existing literature [27, 277,28,35339 35-39] imposes orthogonal weight regularization for ConvNets by treating multi-dimennsioall convolutional ker- nels as 2D matrices (e.g.. by flattening certain di- mensions) and imposing orthogonality of the ma- trix. However, this choice ignores the translation- invariance properties of convolutional operators and, as shown in [22]. does not guarantee en- ergy preservation. On the other hand, these meth- ods often involve dealing with matrix inversions that are computationally expensive for deep and highly overparameterized networks.
BatchWorm
+
x1/ek
X
Com/Norm Withaatee hunstoms
1
STT
Figure 1: Comparison between BatchNorm and ConvNorm activations of k I...., C channels. BatchNorm subtracts and multiplies the activations of each channel by computed scalars: mean l and variance o², before a per-channel affine transform parameterized by learned param- eters /3 and y; ConyNorm performs per-channel convolution with precomputed kernel v to nor- malize the spectrum of the weight matrix for the convolution layer, following with a channel-wise convolution with learned kernel r as the affine transform.
In contrast, in this work we introduce a new nor- malization method dedicated to ConvNets, which explicitly exploits translation-invarinne proper- ties of convolutional operators. Therefore, we term our method as Convolutional Normalization (ConvNorm). We normalize each output channel for each layer of ConvNets, similar to recent preconditioning methods for convolutional sparse coding [40]. The ConvNorm can be viewed as a reparameterization approach for the kernels, that actually it normalizes the weight of each channel to be tight frame. While extra mathematical hassles do exist incorporating conslation-invariane properties, and it turns out to be a blessing, rather than curse, in terms of computation. it allows us to carry out the inversion operation in our ConvNorm via fast Fourier transform (FFT) in the frequeney domain, for which the computation complexity can be significantly reduced.
Highlights of our method. In summary, for ConvNets our approach enjoys several clear ad advantages over classical normalization methods [41-43], that we list below:
Easy to implement. In contrast to weight regularization methods that often require hyperparameter tuning and heavy computation [41,43], the ConvNorm has no parameter to tune and is efficient to compute. Moreover, the ConvNorm can serve as a simple plug-and-play module that can be conveniently incorporated into training almost any ConvNets.
Improving network robustness. Although the ConvNorm operates on each output channel separately, we show that it actually improves the overall layer-wise Lipschitzness of the ConyNets. Therefore, as demonstrated by our experiments it has superior robustness performance against noise corruptions and adversarial attacks.
Improving network training. We numerically demonstrate that the ConvNorm accelerates training on standard image datasets such CIFAR [44] and ImageNet [45]. Inspired by the work [40, 46), our high-level intuition is that the ConvNorm improves the optimization landscape that optimization algorithms converge faster the desired solutions.
'Tight frame can be viewed as generalization of orthogonality for overcomplete matrices, which is also energy preserving.
2

---

Related work. Besides our work, a few very recent work also exploits the translation inviriincce for designing the normalization techniques of ConvNets. We summarize and explain the difference with our method below.
The work [22, 43] derived a similar notion of orthogonality for convolutional kernels, and adopted penalty based method to enforce orthogonality for network weights. These penalty methods often require careful tuning of the strength of the penalty on a case-by-case basis. In contrast, our method is parameter-free and thus easier to use. Our method also shows better empirical performance in terms of robustness.
Very recent work by [29] presented a method to enforce strict orthogonality of convolutional weights by using Cayley transform. Like our approach, a sub-step of their method utilizes the idea of performing the computation in the Fourier domain. However, as they normalize the whole unstructured weight matrix, computing expensive matrix inversion is inevitable, so that their running time and memory consumption is prohibitive for large networks. In contrast, our method is "orthogonalizing" the weight of each channel instead of the whole layer, that can exploit the convolutional structure to avoid expensive matrix inversion with a much lower computational burden. In the meanwhile, we show that this channel-wise normalization can still improve layer-wise Lipschitz condition.
Organizations. The rest of our paper is organized follows. In Section 2, we introduce the basic notations and provide brief overview of ConvNets. In Section 3, we introduce the design of the proposed ConvNorm and discuss the key intuitions and advantages. In Section 4, we perform extensive experiments on various applications verifying the effectiveness of the proposed method. Finally, we conclude and point to some interesting future directions in Section 5. To streamline our presentation, some technical details are deferred to the Appendices.
2 Preliminary
Review of deep networks. A deep network is essentially a nonlinear mapping f(:) x + y. which can be modeled by a composition of a series of simple maps: f(x) = ((=) o...o (z)) where every f²(-) (1 < T.) is called one "layer". Each layer is composed of linear transform, followed by a simple nonlinear activation function +p().. More precisely, a basic deep network of layers can be defined recursively by interlcaving linear and nonlinear activation layers as
x e+1 f((2²) = p A (z²
(1)
1, with 2y = x. Here A-(-) denotes the linear transform and will be described T--1 in detail soon. For convenience, let use 0 to denote all network parameters in (A4( tmn The goal of deep learning is to fit the observation y with the output fax, 0) for any sample from distribution D, by learning 0. This can be achieved by optimizing a certain loss function E(-), i.e.
for é= 0,,,,,,....
1 m
min L(0; { {xx (") = oE
(f(x',0), v')
given a (large) training dataset {(x), y²) } For example, for a typical classification task, the class label of sample a: is represented by one-hot vector y E & representing its membership in k classes. The loss can be chosen to be either the cross-entropy or f2-loss [48]. In the following, we use (z. y) to present one training sample.
An overview of ConyNets. The ConvNet [49] is special deep network architecture, where each of its linear layer can be implemented much more efficiently via convolutions in comparison to fully connected networks [50]. Because of its efficiency and popularity in machine learning, for the rest of the paper, we focus on ConvNets. Suppose the input data æ has C' channels, represented as
æ = (1), D2 )C),
(2)
where for ID signal Ty Rmmh denotes the kth channel feature of æ. For the éth layer (0 S L-1) of ConvNets, the linear operator A(()) B R + C xm in (1) is convolution operation with C441 output channels,
'In [29]. the results are reported based on ResNe19, whereas our method can be easily added to larger networks, e.g. ResNet18 and ResNet50.
"The nonlinearity could contain BatchNorm [4], pooling, dropout [47], and stride, etc. 'If' the data is 2D, can assume m Xm2 For simplicity, we present our idea based ID signal.
3

---

x
f+1
E+1
[1]
,
Ce
2
s+1
vp
Σ
d
#2/
(1 ( Ce+1):
where denotes the convolution between two items that we will discuss below in more detail. Thus, for the éth layer with C2 input channels and C++1 output channels, we can organize the convolution kernels {azy} } as
a
aice abca
a
an
A
d cc
C
d'cn
Convolution operators. For the simplicity of presentation and analysis, adopt circular con- volution instead linear convolution." For signal, given kernel a R" and an input signal E Rn (in many cases m > n), circular convolution * between and can be written in simple matrix-vector product form via
y = a = Ca x;
where C. denotes circulant matrix of (zero-padded) a.
C.
so [a]
$1 [a]
8m-1 [a]].
which is the concatenation of all eyclic shifts [a] (0 S m 1-m111 1) of length k of the (zero-padded) vector a. Since C, can be decomposed via the discrete Fourier transform (DFT) matrix F:
C. F diag (ā) F, a Fa,
where a denotes the Fourier transform of a vector a. The computation of a x can be carried out efficiently via fast Fourier transform (FFT) in the frequency domain. We refer the readers to the appendix for more technical details.
3 Convolutional Normalization
In the following, we introduce the proposed ConyNorm, that can fully exploit benign convolution structures of ConvNets. It can be efficiently implemented in the frequency domain, and reduce the layer-wise Lipschitz constant. First of all, we build intuitions of the new design from the simplest setting. From this, show how to expand the idea to practical ConvNets and discuss its advantages for training and robustness.
3.1 A warm-up study
Let us build some intuitions by zooming into one layer of ConvNets with both input and output being single-channel,
Zout As(z) a. * Zin:
(4)
where Zin the input signal, a is single kernel, and Zont denotes the output before the nonlinear activation. The form (4) is closely related to recent work on blind deconvolution [46]. More specifically, the work showed that normalizing the output zout via preconditioning climinates bad local minimizers and dramatically improves the optimization landscapes for learning the kernel The basic idea is to multiply preconditioning matrix which approximates the following form' ?
= (C.C..) -1/2
(5)
As observe
"Although there are slight differences between linear and circulant convolutions on the boundaries, actually linear convolution can be reduced to circular convolution simply via zero paoo-padding..
In the work [46], they cook up a matrix by using output samples P ( Em C (C (C=:...)) 1/2
When the input samples =!n are i.i.d. zero mean, it can be showed that P ~ P for large m. For ConvNets. we just use the learned kernel a for cooking up P.

---

Bout
Poont
Ca Ca
-1/2
Ca
Zing
Q(a)
the ConvNorm is essentially reparametrizing the circulant matrix Ca of the kernel a to an orthogonal
circulant matrix Q(a) = (a Ca. with QQ I. Thus, the ConvNorm is improving the conditioning of the vanilla problem and reducing the Lipschitz constant of the operator (r..) in (4). On the other hand, the benefits of this normalization can also be observed in the frequency domain.
Based on (3), we have P F diag (v) = Cv with v = F-1 (iā) (a))-) Thus, we also have
Q(a) = C. Ca = Cren F" (
g(a) â @ [a
with @ denoting entrywise operation and g = = a 2al00o-1 Thus, we can see that:
Although the reparameterization involves matrix inversion, which typically expensive to compute, for convolution it can actually be much more efficiently implemented in the frequency domain via FFT, reducing the complexity from O(n") to O log n).
The reparametrized kernel g effectively an all-pass filter with flat normalized spectrum ă + la @-1 From an information theory perspective, this implies that it can better preserve (in particular, high-frequency) information of the input feature from the previous layer.
3.2
NorvNorm for multiple channels
So far, we only considered one layer ConvNets with single-channel input and output. However, recall from Section 2, modern deep ConvNets are usually designed with many layers; each typical layer is constructed with a linear transformation with multiple input and output channels, followed by strides, normalization, and nonlinear activation. Extension of the normalization approach in Section 3.1 from one layer to multiple layers is easy, which can be done by applying the same normalization repetitively for all the layers. However, generalizing method from a single channel to multiple channels not obvious, that we discuss below.
In [40], the work introduced preconditioning method for normalizing multiple kernels in convo- lutional sparse coding. In the following, we show that such an idea can be adapted to normalize each output channel, reduce the Lipschitz constant of the weight matrix in each layer, and improve training and network robustness. Let us consider any layer (1 < L) within vanilla ConvNet using 1-stride, and take one channel (e.g., k-th channel) of that layer as example. For simplicity of presentation, hide the layer number f. Given Zk.out can be written
Σ== Okj Zj.ins the k-th output channel
21,in
22,nn
Cas
Casa
Care,
okt
Zin
with C2 and Co being the numbers of input and output channels, respectively. For each channel k 1,-.. Co. we normalize the output by
-1/2
C1 Σ
Pk
Caks Cau;
(AcAA))
(6)
so that
oku
P
(AtA)
11/2 A Zin-
(x(As)
Thus, we can see the ConvNorm is essentially a reparameterization the kernels {axy} for the k-th channel. Similar to Section 3.1, the operation can be rewritten the form of convolutions
(Ax) Ps = C Cva+au
Co
An all-pass pass filter is a signal processing filter that passes all frequencies equally in gain, but can change the phase relationship among various frequencies.
5

---

with Ps Cv, and vb F-1
(Een
G2 aki
0-1/2
it can be efficiently implemented via FFT.
Here, as for multiple kernels the matrix A₁ is overcomplete (i.e., A₁ is a wide rectangular matrix), cannot normalize the channel- -wiee weight matrix A₁ to exact orthogonal. However, it can be normalized to light frame with Qa
I. This further implies that we can normalize the spectral norm ||& x of the weight matrix l: in each channel to unity (see Figure (Left)).
Combining the operation for the channels, the ConvNorm for each layer overall can be summarized as follows:
Vartia
Va-ils Convittt
ConvHor
Condition Number
Spectral Norm
oo
Qı
10.0
Zout
Zin-
Pco
Qco
5.0
2,5
(8)
0.0
Convolutional Layers
Convolutional Layers
that we normalize each output channel by dif-
ferent matrix Ps. The proposed ConvNorm has several ad advantages that we discuss below.
Figure 2: Condition number for each channel (averaged over all channels) (Left), and spec- tral norm for each layer (Right) ResNet18 except for skip connection layers. ConvNorm nor- malizes the channel-win condition number to I and reduces the layer-wise spectral norm. We use the method in [51] to calculate the singular values of the weight matrix.
Proposition 3.1 The spectral norm of Q intro- duced in (8) can be bounded by
Co
IIQII <
1Qu1
that spectral norm of Q is bounded by the spectral norms of all the weights ) Proof We defer the proof to the Appendix A.3.
Co.
Efficient implementations. There are many existing results [29, 39, 41] trying to normalize the whole layerwise weight matrix. For ConvNets, the matrix is neither circulant nor block circulant, computing its inversion is often computationally prohibitive. Here, for each layer, we only normalize the weight matrix of the individual output channel. Thus similar to Section 3.1, the inversion in (6) can be much more efficiently computed via FFT by exploiting the benign convolutional structure.
Improving layer-wise Lipschitzness. As we can see from Proposition 3.1, although ConvNorm only normalized the spectral norm of each channel, can actually reduce the spectral norm of the whole weight matrix, improving the Lipschitzness of each layer, see Figure 2 (Right) for a numerical demonstration on ResNet18. As extensively investigated [28, 29, 42], improving the Lipschitzness of the weights for ConvNets will lead enhanced robustness against data corruptions, for which we will demonstrate on the proposed ConvNorm in Section 4.1.
Easier training and better generalization. For deconvolution and convolutional sparse coding problems, the work [40,46] showed that ConvNorm could dramatically improve the corresponding nonconvex optimization landscapes. On the other hand, from algorithmic unrolling perspective for neural network design [52, 53], the ConvNorm is analogous to the preconditioned or conjugate gradient methods [54] which often substantially boost algorithmic convergence. Therefore, we conjecture that the ConvNorm also leads to better optimization landscapes for training ConvNets, that they can be optimized faster better solution qualities of generalization. We empirically show this in Section 4.2.
3.3 Extra technical details
To achieve the full performance and efficiency potentials of the proposed ConvNorm, we discuss some essential implementation details in the following.
Efficient propapropag For ConvNorm, as the normalization matrix in (6) is constructed from the learned kernels, it complicates the computation of the gradient in pnokppropggation when training the network. Fortunately, we observe that treating the normalization matrices {Px} as constants during back-propagation usually does not affect the training and generalization performances, that the computational complexity in training is not increased. We noticed that such a technique has also been recently considered in [55] for self-supervised learning, which is termed stop- prop-gradient.

---

Learnable affine tranform. For each channel, we include an (optional) affine transform after the Vk aku in (7) as follows:
normalization Ps- ont Cryy koout
b 2% ont F% Ug ok,u
where the extra convolutional kernel ry is learned along with the original model parameters. The idea of including this affine transform is analogous to including learnable rescaling in BatchNorm, which
can be considered as an "undo" operation to make sure the identity transform can be represented [4]. The difference between our affine transform and BatchNorm is that we apply channel-wise convolu- tions instead of simple rescaling (see Figure 1). Note that when F& is an inverse kernel of Us (i.e., Ty 24 = 1), the overall transformation becomes an identity. The effectiveness of affine transform is demonstrated in the ablation study in Appendix C.4.
Dealing with stride and 2D convolution. There are extra technicalities that we briefly discuss below. For more details, we refer the readers to Appendix B.
Extension to 21) convolution. Although we introduced the ConvNorm based on ID convolution for the simplicity of presentations, it should be noted that our approach can be easily extended to the 2D case via 2D FFT.
Dealing with stride, Strided convolutions are universal in modern ConvNet architectures such as the ResNet [5], which can be viewed downsampling after unstrided convolution. To deal with stride for our ConwNorm, we first perform an unstrided convolution, normalizing the activations using ConvNorm and then downsampling the normalized activations. In comparison, the method proposed in [29] is incompatible with strided convolutions.
Experiments & Results
In this section, we run extensive experiments on CIFAR and ImageNet, empirically demonstrating two major advantages of our approach: (i) it improves the robustness against adversarial attacks, data scarcity, and label noise corruptions [56-58], and (ii) it makes deep ConvNets easier to train and perform better on problems such classification and GANs [59]. The rest of this section is organized as follows. First, we introduce baseline methods for comparisons, and describe the setups of network architectures, datasets, and training. In Section 4.1 and Section 4.2, demonstrate the effectiveness of our approach on robustness and training. respectively.
Baseline methods for comparisons. We compare our method with three representative methods.
Spectral normalization (SN). For each layer of ConvNets, the work [15] treats multi-diiennsonal convolutional kernels as 2D matrices (e.g.. by flattening certain dimensions) and normalizes its spectrum (i.e.. singular values). It estimates the matrix's ' maximum singular value via power method and then uses it to normalize all the singular values. As we discussed in Section 1, the method does not exploit convolutional structures of ConvNets.
Orthogonalization by Newton's Iteration (ONT). The work [39] whitens the same reshaped matrices as SN, that the reshaped matrices reparametrized orthogonality. However, the method needs to compute full inversions of covariance matrices, which is approximated by Newton's 's iterations. Again, no convolutional structure utilized.
Orthogonal ConvNets (OCNN). Few methods that exploit convolutional structures are [22, 43], which enforce orthogonality on doubly block circulant matrices of kernels via penalties on the loss. Here, we compare with [43].
Setups of dataset, network and training. For all experiments, if not otherwise mentioned, CIFAR- 10 and CIFAR-100 datasets are processed with standard augmentations, i.e., random cropping and flipping. We use 10% of the training set for validation and treat the validation set as held-out test set. For ImageNet, we perform standard random resizing and flipping. For training, we observe our ConvNorm is not sensitive to the learning rate, and thus we fix the initial learning rate 0.1 for all experiments." 9 For experiments on CIFAR-10, we run 120 epochs and divide the learning rate by 10 the 40th and 80th epochs; for CIFAR- 100, we run 150 epochs and divide the learning rate by 10 at the 60th and 120th epoch; for ImageNet, we run 90 epochs and divide the learning rate by 10 at the 30th and 90th epochs. The optimization is done using SGD with a momentum of 0.9 and a weight decay of 0.0001 for all datasets. For networks we use two backbone networks: VGG16 [60] and 18 [5]. We adopt Xavier uniform initialization [61] which is the default initialization in Py PoToch for all networks.
For experiments with ONI, use learning rate 0.01 since the loss would be trained to NaN if with 0.1.
7

---

<
Test Acc.
SN
BN
ONI
OCNN
ConvNorm
Clean
82.52 1: 0.22
82.13 1 0.67
80.70 0.14
82.90 1 0.31
83.23 0.25
FGSM
$2.34
51.72 +: 0.52
48.33
$2.49 0.21
52.87 0.24
PGD-10
45.68 :: 0.40
45.31 +: 0.29
42.30 :: 0.24
45.74 1: 0.13
46.12
0.26
255
PGD-20
44.47 1: 0.37
44.04 1: 0.24 41.08
0.30
44.53 1 0.10
44.75
Table 1: Comparison of ConvNorm to baseline methods under different gradient based attacks. Models are robustly trained following the procedure in [62] using a ResNet18 backbone. Experiments conducted on CIFAR-10 dataset. Results averaged over 4 random seeds.
4.1 Improved robustness
In this section, we demonstrate our method is more robust to various kinds of adversarial at- tacks, as well random label corruptions and small training datasets.
Method
Average Queries
Attack Success rate (%)
SN
2519.32
60.60
ONI
2817.09
55.90
OCNN
2892.81
54.50
Robustness against ad verrsaaiall attacks. Ex- isting results [29.43] show that controlling the layer-wise Lipschitz constants for deep net- works improves robustmess against adversarial attack. Since method improves the Lipschitz- ness of weights (see Figure 2), we demonstrate its robustness under adversarial attack on the CIFAR-10 dataset. We adopt both white-box (gradient based) attack [57, 58) and black-box attack [56] to test the robustness of our proposed method and other baseline methods. The results are presented in Table and Table 2. For the ease of presentation, all technical details about model training and generation of the adversarial examples are postponed to Appendix C.
ConyNorm
2966.16
53.50
Table 2: Comparison ConvNorm to baseline methods SimBA black box attack. The mean value of average queries (the higher, the better) and attack success rate (the lower, the better) through- out 3 runs are reported. Models are trained using a ResNet1 backbone without BN layers.
In the case of gradient based attacks, follow the training procedure described in [62] to train models with our ConyNorm and other baseline methods. We report the performances the robustly trained models both the clean test dataset and datasets that are perturbed by Fast Gradient Sign Method (FGSM) [57] and Projected Gradient Method (PGD) [58]. As shown in Table 1, our ConyNorm outperforms other methods in terms of robustness under white-box attack while maintaining good performance on clean test accuracy.
For black-box attack, we adopt a popular black-box box adversarial attack method, Simple Black-box Adversarial Attacks (SimBA) [56]. By submitting queries to a model for updated test accuracy, the attack method iteratively finds a perturbation where the confidence score drops the most. We report the average queries and success rate after 3072 iterations in Table 2. As we can see, the CorvNorm resists the most queries, and that the SimBA has the lowest attack success rate for ConvNorm compared with other baseline methods.
Noisy Label
Data Scarcity
Robustness against label noise and data scarcity. It has been widely observed that overparameterized ConvNets tend to overfit when label noise presents the amount of train- ing labels is limited [63-66]. Recent work [67] shows that normalizing the weights enforces cer- tain regularizations, which can improve gener- alization performance against both label noise and data scarcity. Since our method is es- sentially reparametrizing and normalizing the weights, we demonstrate the robustness of our approach under these settings on CIFAR-10 with ResNet18 8 backbone.
0.90
Test Accuracy 2.
00.00
0.75
0.50
0.70
Meas
0.75
0.55
DM
0.55
40%
Label noise rale
Subsel percent
Figure Test accuracy for noisy label (Lefft and insufficient training data (Right). Experi- ments are conducted on CIFAR- dataset using a ResNet18 backbone. Error bars corresponding to standard deviations over runs.
Robusiness against label noise. Following the scheme proposed in [68], we simulate noisy labels by randomly flipping 20% to 80% of the labels in the training set. As shown in Figure (Left), our method outperforms the others most noisy rates by a hefty margin when the noise level is high.
8

---

10 (VGG16)
CIFAR-10 (ResNet18)
ImageNet (ResNet18) 8)
lest Accuracy
Test Accuracy
0.45
0.50
0.6
ConvWrrm BN
0.40
Varilla 0.4
0.35
0.30
Convform
0.2
BN
0:25
BN
4DCC 6000 10030 Iteration
2000
6000
800n
10000
10
20
30
40
50
60
70
80
Iteration
Epach
Figure 4: ConvNorm accelerates convergence. VGGI6 trained on CIFAR-10 (Left), and ResNet18 (Middle) trained on CIFAR-10 10 and ImageNet (Right), with and without ConvNorm or BatchNorm. We do not use data augmentation, weight decay, any other regularization in this experiment to isolate the effects of the normalization techniques. Error bars correspond to min/max over 4 runs.
Robustness against data scarcity. We test our method training the network with varying sizes of the training set, obtained by randomly sampling. The results in Figure 3 (Right) show that our ConvNorm achieves on par performance compared with baseline methods, and its performance stays high even when the size of the training data is tiny (e.g., 4500 examples).
4.2 Easier training on classification and GAN
Finally, we compare training convergence speed for classification and performances GAN. Extra experiments on better generalization performance and ablation study can be found in Appendix C.4.
Improved training on supervised learning. We test our method on image classification tasks with two backbone architectures: VGG16 and ResNet18. We show that ConvNorm accelerates the convergence of training. To isolate the effects of the normalization layers for training, train CIFAR-10 10 and ImageNet without using any augmentation, regularization, and learning rate decay. In Figure 4, we show that adding ConvNorm consistently results faster convergence, stable training (less variance in accuracy), and superior performance. On CIFAR-10 10, there is a wide performance gap after the first few iterations of training: 1000 iterations of training with ConvNorm lead to generalization performance comparable to 8000 iterations of training using BatchNorm. In the case of standard settings where data augmentation, regularization and learning rate decay are added, we notice that using ConyNorm and BatchNorm together also yield better test performances compared to only using BatchNorm (See Appendix C.4 for details). Besides the convergence speed of training. the exact training time for different methods is another important factor for measuring the efficiency of such methods. To this end, we empirically compare the training time for different methods and report the results in Appendix D and Table 8.
Improved performance for GANs. It has been found that improving the Lipschitz condition of the discriminator of GAN stabilizes its training [69]. For instance, WGAN-GP [70] demonstrates that adding a gradient penalty (1 (1-GP) regularization to enforce the I-Lipschitzness of the discriminator
stabilizes GAN training and prevents mode collapse. Subsequent works [71, 72] using variants of the 1-GP regularization also show their improvement in GAN. Later on, [15] further reveals the performance of GAN can be significantly improved if the spectral norm (Lipschitz condition) of the discriminator network is strictly enforced to 1. As shown in Figure 2, the proposed ConvNorm also controls the Lipschitz condition of ConvNets. Therefore, we expect our method also ameliorates the performance of GAN.
Metrie
SN
ONI
OCNN
Vanilla
ConvNorn
To demonstrate the effectiveness of the Con- vNorm on GAN, we compare it with other base- line methods introduced previously. In our ex-
8.12
7.07
7.54
7.13
7.62
14.53
29.49
22.15
29.47
21.37
periments, we adopt the same settings and archi- tecture suggested in [15] without any modifica- tion, and we use the inception score (IS) [73], and FID [74] score for quantitative evaluation. As shown Table 3, our ConvNorm achieves the second-best performance to SN."
Table 3: Comparison of ConvNorm to baseline methods on GAN training. Inception score (IS) (the higher, the better) and FID score (the lower, the better) of ResNet with different normalizations. For each pair of model and method, we generate 50k: images 10 times and compute the mean of IS.
The performance of GANs is highly sensitive to the computational budget and the hyperparameters of the networks [75]. and the hyperparameters of SN is tuned for CIFAR- 10 while we use the same hyperparameters as SN.

---

Discussions & Conclusion
In this work, we introduced a new normalization approach for ConvNets, which explicitly exploits translation- translationiinvarianee invariaavaeeee properties of convolutional operators, leading efficient implementation and boosted performances in training, generalization, and robustness. Our work has opened several interesting directions to be further exploited for normalization design of ConvNets: (i) although provided some high-level intuitions why our CoryNorm works, theoretical justifications are needed; (ii) ConvNorm only promotes channel-wise "orthogonality", it would be interesting utilize similar ideas to efficiently normalize the layerwise weight matrices by exploiting convolutional structures. We leave these questions for future investigations.
Acknowledgement
Part of this work was done when XL. and QQ were at Center for Data Science, NYU. SL, XL., CFG, and QQ were partially supported by NSF grant DMS 2009752. SL was partially supported by NSF NRT-HDR Award 1922658. CY acknowledges support from Sänghua-Berkelee Shenzhen Institute Research Fund. ZZ acknowledges support from NSF grant CCF 2008460. QQ also acknowledges support of Moore-Sloan fellowship, and startup fund at the University of Michigan.
10


Reviews:
[{"review_id": "vyiBjGbo0aF", "reviewer": "Reviewer_3mQk", "summary": "The paper proposes to orthogonalize the convolutional kernel in a channel-wise manner. To do the computation more efficiently, they calculate the matrix inversion via FFT in the frequency domain. They compare with the baselines on the tasks of adversarial attack, image classification and GAN.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I think that this paper is the first work proposed to orthogonalize output channels in the frequency domain via FFT. But this is not the first work that orthogonalizes weight in the frequency domain [1]. \n\nClarity: \n\nPros: The paper is well-written and easy to follow overall. \n\nCons: Some typos: \n1. F* is not defined in Eq. 3, though defined in Appendix. \n2. 'Learnable affine transform' in line 235.\n\nQuality: \n\nPros: The paper provides the comparison with multiple baselines on multiple tasks, and shows empirical advantages.\n\nCons: \n1. I think the paper misses the empirical comparison with [1], at least in the task of adversarial attack. The proposed method shares a lot in common. E.g. focus on circular convolutions and Fourier domain, and [1] also shows the advantage in the task of adversarial attack.\n2. It not clear how well the proposed method controls the spectral norm and condition number of the convolutional layer, and how well the input norm gets preserved, compared with baselines. Can you provide the spectral norm, condition number and ||Conv(x)||_2/||x||_2 (i.e. the ratio of the output norm and the input norm) for each layer, for ConvNorm and other baselines? (just in adversarial attack task is good enough)\n3. I feel the run time is also not clear. Can you provide the run time comparison on ConvNorm and baselines under the different kernel settings (e.g. # channel_in, # channel_out, kernel_size, input_size)? I believe it can provide the community a better view of the advantages and limitations of each method.\n4. The paper doesn't compare with orthogonalization-related baselines. And it seems the baselines have a better performance the ConvNorm. For instance, ONI [2] has 70.43 test acc on ImageNet with ResNet18, and OCNN [3] has 78.1 test acc on CIFAR-100 on ResNet18. \n5. What's more, [2, 3] conducts the experiments with more complex architecture (e.g. ResNet 50, ResNet 101), while ConvNorm only shows the results on ResNet 18. So the scalability is my another concern.\n6. I am confused about affine transform. It seems gamma_k is a freely learned kernel. I wonder if the orthogonality gets changed after passing affine transformation? How's the spectral norm, condition number gonna change?\n\nSignificance: I cannot determine the significance at this moment, since the paper doesn't justify itself completely to me. But I believe I will have a better and more overall understanding if more statistics and comparison can be provided. And I will consider raising my score then. \n\n\n[1] Orthogonalizing Convolutional Layers with the Cayley Transform\n\n[2] Controllable Orthogonalization in Training DNNs\n\n[3] Orthogonal Convolutional Neural Networks\n\n===============\npost rebuttal: I have read the rebuttal; and raise my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper presents the limitations (e.g. theoretical explanation) and leaves them to future work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "PiT8mGVmyQu", "reviewer": "Reviewer_FWBT", "summary": "The paper proposes a novel normalization technique for controlling the channel-wise Lipschitz of the convolutional layers. The normalization technique operates on the Fourier domain which could reduce the burden of doing full SVD and at the same time be more accurate compared to using approximate methods. The paper also empirically demonstrates the effectiveness of such normalization by showing experimental results on robustness under different sources of noises, easier training, and extended usage in GAN.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm quite positive in general. The paper is clearly motivated, well written, and smoothly presented. The method proposed is reasonable and well established. Also, the implementation is straightforward and since there is no need to tune hyperparameters, the method could easily serve as a plug-in module for most deep convolutional networks. The experimental part is comprehensive, the authors tried various sets of tasks to empirically test the performances of the proposed method and other methods. \n\nOne remaining concern is that most experiments in this paper are primarily done on CIFAR-10, it would be more interesting to see more experiments conducted on more complex datasets like Imagenet.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "LobgzSPffn", "reviewer": "Reviewer_HTRX", "summary": "This paper exploits the convolutional structure to maintain orthogonality on the convolutional kernel for guarantee the energy preservation in CNN, rather than the naively used methods using flattened convolutional kernels. It orthogonalizes the channel-wise convolutional kernel via preconditioning its circulant matrix transferred by FFT. The proposed ConvNorm also reduces the layer-wise spectral norm of the weight matrices. Experiments on small network (e.g, VGG 16 and 18-layer Resnet) using CIFAR-10 ImageNet verify the effectiveness of ConvNorm in model robustness and training GANs. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+The idea to orthogonalize the channel-wise convolutional kernel via preconditioning its circulant matrix is novel to me.   \n+The related works are comprehensive discussed, as far as I am concerned.  \n+The presentation is well organized  \n+The effectiveness of ConvNorm in model robustness and training GANs are well validated by the experiments. \n\n\n\n**Cons:**\n\n1.One of my main concern is the computational or memory cost of the proposed methods. This paper spends much of contents to show the computational problems of the previous methods (Lines 58~61, Line 76) and claims the efficiency of the proposed ConvNorm. However, I did not see analysis about the computational complexity or the time cost in experiments. What is the wall-clock time cost of the proposed method, compared to the baselines? I would like to see the time cost& memory cost in the described experiments, especially on ImageNet.   \n2. I am not convinced with the claims“Easier training and better generalization” by the experiments: 1) Clearly, ConvNorm do not work well without BN in such a small network (VGG16-ResNet18); 2) the experiments should be conduct on larger network, e.g, 56-layer or 110-layer ResNet on CIFAR (1GPU is enough to run the experiments) or better using ResNet-50 on ImageNet. I wonder whether ConvNorm can work on 110-layer ResNet without BN. \n\n**Other comments:**\n\nWhat is the y-axis of the Figure 2 (Left), the log-scale? It should be described in here. By the way, I believe it is more important to show the layer-wise condition number of the proposed ConvNorm, rather than the channel-wise condition number, because the motivation of orthogonal matrix W is for preserving the magnitude between the output and input, with y=Wx, or $\\frac{\\partial{L}}{\\partial{x}} = W^T \\frac{\\partial{L}}{\\partial{y}}$.\n\nIt is not clear why should stop the gradient shown in Line 229-234.  I believe this can work with feature activation (e.g. BN) in a network, but I conjecture it will suffer training instability without feature activations in deeper networks (e.g., 56-layer/110-layer ResNet for CIFAR classification). It is better to provide the results in deeper networks without feature activation. \n\n**update**:\nI have read authors’ responses and other reviewers’ comments. I acknowledge the authors’ feedback which addresses my concerns. I hope the code (including the additional experiments for comparing layer-wise condition number) will be released if this paper gets accepted. I keep my score as 7. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "JDgLM8aChgJ", "reviewer": "Reviewer_G1ti", "summary": "The paper introduces a new way to enforce orthogonality in convolutional layers. The main idea is to make the computations in the frequency domain where convolutions (or cross-correlations) become products. Hence, exact orthogonality is achieved, while keeping the computational costs low using FFT.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major revision\n\nThe paper is well written, although some of the computations are not easy to follow. Section 3.1 could be enlarged and better explained. If the authors need more space to do that, the previous section regarding neural networks in general could be deleted. It is only useful because it provides some notation.\n\nThe forward pass of the new layer is not very clear in the paper. It could be useful to explain it, because it might seem that you provide a formula for $Q(a)$ and then multiply the $x$ by that matrix, which would require a large computation and which you do not do. It gets more clear in the Appendix, but it is worth mentioning it in the paper. Besides, experiments of the computation time of the forward pass and a comparison with other methods would be valuable in the paper, since you claim that your method is faster and show that analytically.\n\nMinor revisions:\n\nLine 157: \"The basic idea is to multiply a preconditioning matrix which approximates the following form\": the sentence is not very clear in a first reading, now I understand what you mean, but please rephrase.\nLine 177: \"However, generalizing our method from a single channel to multiple channels is not obvious, that we discuss below.\": again, please rephrase this sentence.\n\nOverall Evaluation:\nApart from simple changes that can be made by the authors, the idea of the authors is clever and the implementation efficient. Besides, numerical experiments prove that their approach is valuable. At the moment, I think the paper is a little above the acceptance threshold and could be further improved with experiments regarding inference time.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors present a simple and efficient ConvNorm method that can fully exploit the convolutional structure in the Fourier domain. The paper is clearly written and well motivated. Extensive experiments empirically demonstrate the effectiveness of such normalization. As all reviewers achieve consensus that the paper is well written and can be accepted, I vote for acceptance. The authors are expected to make a thorough revision by considering the the reviews.  

Decision: Accept (Poster)