Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

1 Introduction . In the past decade , Convolutional Neural Networks ( ConvNets ) have achieved phenomenal success in many machine learning and computer vision applications [ 1–7 ] . Normalization is one of the most important components of modern network architectures [ 8 ] . Early normalization techniques , such as batch normalization ( BatchNorm ) [ 4 ] , are cornerstones for effective training of models beyond a few layers . Since then , the values of normalization for optimization and learning is extensively studied , and many normalization techniques , such as layer normalization [ 9 ] , instance normalization [ 10 ] , and group normalization [ 11 ] are proposed . Many of such normalization techniques are based on estimating certain statistics of neuron inputs from training data . However , precise estimations of the statistics may not always be possible . For example , BatchNorm becomes ineffective when the batch size is small [ 12 ] , or batch samples are statistically dependent [ 13 ] . ∗The first two authors contributed to this work equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Weight normalization [ 14 ] is a powerful alternative to BatchNorm that improves the conditioning of neural network training without the need to estimate statistics from neuron inputs . Weight normalization operates by either reparameterizing or regularizing the network weights so that all the weights have unit Euclidean norm . Since then , various forms of normalization for network weights are proposed and become critical for many tasks such as training Generative Adversarial Networks ( GANs ) [ 15 ] and obtaining robustness to input perturbations [ 16 , 17 ] . One of the most popular forms of weight normalization is enforcing orthogonality , which has drawn attention from a diverse range of research topics . The idea is that weights in each layer should be orthogonal and energy-preserving . Orthogonality is argued to play a central role for training ultra-deep models [ 18–22 ] , optimizing recurrent models [ 23–26 ] , improving generalization [ 27 ] , obtaining robustness [ 28 , 29 ] , learning disentangled features [ 30 , 31 ] , improving the quality of GANs [ 32 , 33 ] , learning low-dimensional embedding [ 34 ] , etc . Exploiting convolution structures for normalization . Our work is motivated by the pivotal role of weight normalization in deep learning . In the context of ConvNets , the network weights are multi-dimensional ( e.g. , 4-dimensional for a 2D ConvNet ) convolutional kernels . A vast majority of existing literature [ 27 , 28 , 35–39 ] imposes orthogonal weight regularization for ConvNets by treating multi-dimensional convolutional kernels as 2D matrices ( e.g. , by flattening certain dimensions ) and imposing orthogonality of the matrix . However , this choice ignores the translationinvariance properties of convolutional operators and , as shown in [ 22 ] , does not guarantee energy preservation . On the other hand , these methods often involve dealing with matrix inversions that are computationally expensive for deep and highly overparameterized networks . In contrast , in this work we introduce a new normalization method dedicated to ConvNets , which explicitly exploits translation-invariance proper- ties of convolutional operators . Therefore , we term our method as Convolutional Normalization ( ConvNorm ) . We normalize each output channel for each layer of ConvNets , similar to recent preconditioning methods for convolutional sparse coding [ 40 ] . The ConvNorm can be viewed as a reparameterization approach for the kernels , that actually it normalizes the weight of each channel to be tight frame.2 While extra mathematical hassles do exist in incorporating translation-invariance properties , and it turns out to be a blessing , rather than a curse , in terms of computation , as it allows us to carry out the inversion operation in our ConvNorm via fast Fourier transform ( FFT ) in the frequency domain , for which the computation complexity can be significantly reduced . Highlights of our method . In summary , for ConvNets our approach enjoys several clear advantages over classical normalization methods [ 41–43 ] , that we list below : • Easy to implement . In contrast to weight regularization methods that often require hyperparameter tuning and heavy computation [ 41 , 43 ] , the ConvNorm has no parameter to tune and is efficient to compute . Moreover , the ConvNorm can serve as a simple plug-and-play module that can be conveniently incorporated into training almost any ConvNets . • Improving network robustness . Although the ConvNorm operates on each output channel separately , we show that it actually improves the overall layer-wise Lipschitzness of the ConvNets . Therefore , as demonstrated by our experiments , it has superior robustness performance against noise corruptions and adversarial attacks . • Improving network training . We numerically demonstrate that the ConvNorm accelerates training on standard image datasets such as CIFAR [ 44 ] and ImageNet [ 45 ] . Inspired by the work [ 40 , 46 ] , our high-level intuition is that the ConvNorm improves the optimization landscape that optimization algorithms converge faster to the desired solutions . 2Tight frame can be viewed as a generalization of orthogonality for overcomplete matrices , which is also energy preserving . Related work . Besides our work , a few very recent work also exploits the translation-invariance for designing the normalization techniques of ConvNets . We summarize and explain the difference with our method below . • The work [ 22 , 43 ] derived a similar notion of orthogonality for convolutional kernels , and adopted a penalty based method to enforce orthogonality for network weights . These penalty methods often require careful tuning of the strength of the penalty on a case-by-case basis . In contrast , our method is parameter-free and thus easier to use . Our method also shows better empirical performance in terms of robustness . • Very recent work by [ 29 ] presented a method to enforce strict orthogonality of convolutional weights by using Cayley transform . Like our approach , a sub-step of their method utilizes the idea of performing the computation in the Fourier domain . However , as they normalize the whole unstructured weight matrix , computing expensive matrix inversion is inevitable , so that their running time and memory consumption is prohibitive for large networks.3 In contrast , our method is “ orthogonalizing ” the weight of each channel instead of the whole layer , so that we can exploit the convolutional structure to avoid expensive matrix inversion with a much lower computational burden . In the meanwhile , we show that this channel-wise normalization can still improve layer-wise Lipschitz condition . Organizations . The rest of our paper is organized as follows . In Section 2 , we introduce the basic notations and provide a brief overview of ConvNets . In Section 3 , we introduce the design of the proposed ConvNorm and discuss the key intuitions and advantages . In Section 4 , we perform extensive experiments on various applications verifying the effectiveness of the proposed method . Finally , we conclude and point to some interesting future directions in Section 5 . To streamline our presentation , some technical details are deferred to the Appendices . 2 Preliminary . Review of deep networks . A deep network is essentially a nonlinear mapping f ( · ) : x 7→ y , which can be modeled by a composition of a series of simple maps : f ( x ) = fL−1 ◦ · · · ◦ f1 ◦ f0 ( x ) , where every f ` ( · ) ( 1 ≤ ` ≤ L ) is called one “ layer ” . Each layer is composed of a linear transform , followed by a simple nonlinear activation function ϕ ( · ) .4 More precisely , a basic deep network of L layers can be defined recursively by interleaving linear and nonlinear activation layers as z ` +1 = f ` ( z ` ) = ϕ ◦ A ` ( z ` ) ( 1 ) for ` = 0 , 1 , . . . , L− 1 , with z0 = x . Here A ` ( · ) denotes the linear transform and will be described in detail soon . For convenience , let us use θ to denote all network parameters in { A ` ( · ) } L−1 ` =0 . The goal of deep learning is to fit the observation y with the output f ( x , θ ) for any sample x from a distribution D , by learning θ . This can be achieved by optimizing a certain loss function ` ( · ) , i.e. , min θ∈Θ L ( θ ; { ( xi , yi ) } m i=1 ) : = 1 m m∑ i=1 ` ( f ( xi , θ ) , yi ) , given a ( large ) training dataset { ( xi , yi ) } m i=1 . For example , for a typical classification task , the class label of a sample x is represented by a one-hot vector y ∈ Rk representing its membership in k classes . The loss can be chosen to be either the cross-entropy or ` 2-loss [ 48 ] . In the following , we use ( x , y ) to present one training sample . An overview of ConvNets . The ConvNet [ 49 ] is a special deep network architecture , where each of its linear layer can be implemented much more efficiently via convolutions in comparison to fully connected networks [ 50 ] . Because of its efficiency and popularity in machine learning , for the rest of the paper , we focus on ConvNets . Suppose the input data x has C channels , represented as x = ( x1 , x2 , · · · , xC ) , ( 2 ) where for 1D signal xk ∈ Rm denotes the kth channel feature of x.5 For the ` th layer ( 0 ≤ ` ≤ L−1 ) of ConvNets , the linear operator A ` ( · ) : RC ` ×m 7→ RC ` +1×m in ( 1 ) is a convolution operation with C ` +1 output channels , 3In [ 29 ] , the results are reported based on ResNet9 , whereas our method can be easily added to larger networks , e.g . ResNet18 and ResNet50 . 4The nonlinearity could contain BatchNorm [ 4 ] , pooling , dropout [ 47 ] , and stride , etc . 5If the data is 2D , we can assume x ∈ Rm1×m2 . For simplicity , we present our idea based on 1D signal . z ` +1 = ( z ` +11 , z ` +1 2 , · · · , z ` +1 C ` +1 ) , z ` +1k = ϕ C∑̀ j=1 a ` kj ∗ z ` j ( 1 ≤ k ≤ C ` +1 ) , where ∗ denotes the convolution between two items that we will discuss below in more detail . Thus , for the ` th layer with C ` input channels and C ` +1 output channels , we can organize the convolution kernels { akj } as A ` = a ` 11 a ` 12 · · · a ` 1C ` a ` 21 a ` 22 · · · a ` 2C ` ... ... . . . ... a ` C ` +11 a ` C ` +12 · · · a ` C ` +1C ` . Convolution operators . For the simplicity of presentation and analysis , we adopt circular convolution instead of linear convolution.6 For 1D signal , given a kernel a ∈ Rn and an input signal x ∈ Rm ( in many cases m n ) , a circular convolution ∗ between a and x can be written in a simple matrix-vector product form via y = a ∗ x = Ca · x , where Ca denotes a circulant matrix of ( zero-padded ) a , Ca : = [ s0 [ a ] s1 [ a ] · · · sm−1 [ a ] ] , which is the concatenation of all cyclic shifts sk [ a ] ( 0 ≤ k ≤ m−1 ) of length k of the ( zero-padded ) vector a . Since Ca can be decomposed via the discrete Fourier transform ( DFT ) matrix F : Ca = F ∗ diag ( â ) F , â = Fa , ( 3 ) where â denotes the Fourier transform of a vector a . The computation of a ∗ x can be carried out efficiently via fast Fourier transform ( FFT ) in the frequency domain . We refer the readers to the appendix for more technical details .

Summary: The paper proposes to orthogonalize the convolutional kernel in a channel-wise manner. To do the computation more efficiently, they calculate the matrix inversion via FFT in the frequency domain. They compare with the baselines on the tasks of adversarial attack, image classification and GAN.