Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

1 Introduction . In the past decade , Convolutional Neural Networks ( ConvNets ) have achieved phenomenal success in many machine learning and computer vision applications [ 1–7 ] . Normalization is one of the most important components of modern network architectures [ 8 ] . Early normalization techniques , such as batch normalization ( BatchNorm ) [ 4 ] , are cornerstones for effective training of models beyond a few layers . Since then , the values of normalization for optimization and learning is extensively studied , and many normalization techniques , such as layer normalization [ 9 ] , instance normalization [ 10 ] , and group normalization [ 11 ] are proposed . Many of such normalization techniques are based on estimating certain statistics of neuron inputs from training data . However , precise estimations of the statistics may not always be possible . For example , BatchNorm becomes ineffective when the batch size is small [ 12 ] , or batch samples are statistically dependent [ 13 ] . ∗The first two authors contributed to this work equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Weight normalization [ 14 ] is a powerful alternative to BatchNorm that improves the conditioning of neural network training without the need to estimate statistics from neuron inputs . Weight normalization operates by either reparameterizing or regularizing the network weights so that all the weights have unit Euclidean norm . Since then , various forms of normalization for network weights are proposed and become critical for many tasks such as training Generative Adversarial Networks ( GANs ) [ 15 ] and obtaining robustness to input perturbations [ 16 , 17 ] . One of the most popular forms of weight normalization is enforcing orthogonality , which has drawn attention from a diverse range of research topics . The idea is that weights in each layer should be orthogonal and energy-preserving . Orthogonality is argued to play a central role for training ultra-deep models [ 18–22 ] , optimizing recurrent models [ 23–26 ] , improving generalization [ 27 ] , obtaining robustness [ 28 , 29 ] , learning disentangled features [ 30 , 31 ] , improving the quality of GANs [ 32 , 33 ] , learning low-dimensional embedding [ 34 ] , etc . Exploiting convolution structures for normalization . Our work is motivated by the pivotal role of weight normalization in deep learning . In the context of ConvNets , the network weights are multi-dimensional ( e.g. , 4-dimensional for a 2D ConvNet ) convolutional kernels . A vast majority of existing literature [ 27 , 28 , 35–39 ] imposes orthogonal weight regularization for ConvNets by treating multi-dimensional convolutional kernels as 2D matrices ( e.g. , by flattening certain dimensions ) and imposing orthogonality of the matrix . However , this choice ignores the translationinvariance properties of convolutional operators and , as shown in [ 22 ] , does not guarantee energy preservation . On the other hand , these methods often involve dealing with matrix inversions that are computationally expensive for deep and highly overparameterized networks . In contrast , in this work we introduce a new normalization method dedicated to ConvNets , which explicitly exploits translation-invariance proper- ties of convolutional operators . Therefore , we term our method as Convolutional Normalization ( ConvNorm ) . We normalize each output channel for each layer of ConvNets , similar to recent preconditioning methods for convolutional sparse coding [ 40 ] . The ConvNorm can be viewed as a reparameterization approach for the kernels , that actually it normalizes the weight of each channel to be tight frame.2 While extra mathematical hassles do exist in incorporating translation-invariance properties , and it turns out to be a blessing , rather than a curse , in terms of computation , as it allows us to carry out the inversion operation in our ConvNorm via fast Fourier transform ( FFT ) in the frequency domain , for which the computation complexity can be significantly reduced . Highlights of our method . In summary , for ConvNets our approach enjoys several clear advantages over classical normalization methods [ 41–43 ] , that we list below : • Easy to implement . In contrast to weight regularization methods that often require hyperparameter tuning and heavy computation [ 41 , 43 ] , the ConvNorm has no parameter to tune and is efficient to compute . Moreover , the ConvNorm can serve as a simple plug-and-play module that can be conveniently incorporated into training almost any ConvNets . • Improving network robustness . Although the ConvNorm operates on each output channel separately , we show that it actually improves the overall layer-wise Lipschitzness of the ConvNets . Therefore , as demonstrated by our experiments , it has superior robustness performance against noise corruptions and adversarial attacks . • Improving network training . We numerically demonstrate that the ConvNorm accelerates training on standard image datasets such as CIFAR [ 44 ] and ImageNet [ 45 ] . Inspired by the work [ 40 , 46 ] , our high-level intuition is that the ConvNorm improves the optimization landscape that optimization algorithms converge faster to the desired solutions . 2Tight frame can be viewed as a generalization of orthogonality for overcomplete matrices , which is also energy preserving . Related work . Besides our work , a few very recent work also exploits the translation-invariance for designing the normalization techniques of ConvNets . We summarize and explain the difference with our method below . • The work [ 22 , 43 ] derived a similar notion of orthogonality for convolutional kernels , and adopted a penalty based method to enforce orthogonality for network weights . These penalty methods often require careful tuning of the strength of the penalty on a case-by-case basis . In contrast , our method is parameter-free and thus easier to use . Our method also shows better empirical performance in terms of robustness . • Very recent work by [ 29 ] presented a method to enforce strict orthogonality of convolutional weights by using Cayley transform . Like our approach , a sub-step of their method utilizes the idea of performing the computation in the Fourier domain . However , as they normalize the whole unstructured weight matrix , computing expensive matrix inversion is inevitable , so that their running time and memory consumption is prohibitive for large networks.3 In contrast , our method is “ orthogonalizing ” the weight of each channel instead of the whole layer , so that we can exploit the convolutional structure to avoid expensive matrix inversion with a much lower computational burden . In the meanwhile , we show that this channel-wise normalization can still improve layer-wise Lipschitz condition . Organizations . The rest of our paper is organized as follows . In Section 2 , we introduce the basic notations and provide a brief overview of ConvNets . In Section 3 , we introduce the design of the proposed ConvNorm and discuss the key intuitions and advantages . In Section 4 , we perform extensive experiments on various applications verifying the effectiveness of the proposed method . Finally , we conclude and point to some interesting future directions in Section 5 . To streamline our presentation , some technical details are deferred to the Appendices . 2 Preliminary . Review of deep networks . A deep network is essentially a nonlinear mapping f ( · ) : x 7→ y , which can be modeled by a composition of a series of simple maps : f ( x ) = fL−1 ◦ · · · ◦ f1 ◦ f0 ( x ) , where every f ` ( · ) ( 1 ≤ ` ≤ L ) is called one “ layer ” . Each layer is composed of a linear transform , followed by a simple nonlinear activation function ϕ ( · ) .4 More precisely , a basic deep network of L layers can be defined recursively by interleaving linear and nonlinear activation layers as z ` +1 = f ` ( z ` ) = ϕ ◦ A ` ( z ` ) ( 1 ) for ` = 0 , 1 , . . . , L− 1 , with z0 = x . Here A ` ( · ) denotes the linear transform and will be described in detail soon . For convenience , let us use θ to denote all network parameters in { A ` ( · ) } L−1 ` =0 . The goal of deep learning is to fit the observation y with the output f ( x , θ ) for any sample x from a distribution D , by learning θ . This can be achieved by optimizing a certain loss function ` ( · ) , i.e. , min θ∈Θ L ( θ ; { ( xi , yi ) } m i=1 ) : = 1 m m∑ i=1 ` ( f ( xi , θ ) , yi ) , given a ( large ) training dataset { ( xi , yi ) } m i=1 . For example , for a typical classification task , the class label of a sample x is represented by a one-hot vector y ∈ Rk representing its membership in k classes . The loss can be chosen to be either the cross-entropy or ` 2-loss [ 48 ] . In the following , we use ( x , y ) to present one training sample . An overview of ConvNets . The ConvNet [ 49 ] is a special deep network architecture , where each of its linear layer can be implemented much more efficiently via convolutions in comparison to fully connected networks [ 50 ] . Because of its efficiency and popularity in machine learning , for the rest of the paper , we focus on ConvNets . Suppose the input data x has C channels , represented as x = ( x1 , x2 , · · · , xC ) , ( 2 ) where for 1D signal xk ∈ Rm denotes the kth channel feature of x.5 For the ` th layer ( 0 ≤ ` ≤ L−1 ) of ConvNets , the linear operator A ` ( · ) : RC ` ×m 7→ RC ` +1×m in ( 1 ) is a convolution operation with C ` +1 output channels , 3In [ 29 ] , the results are reported based on ResNet9 , whereas our method can be easily added to larger networks , e.g . ResNet18 and ResNet50 . 4The nonlinearity could contain BatchNorm [ 4 ] , pooling , dropout [ 47 ] , and stride , etc . 5If the data is 2D , we can assume x ∈ Rm1×m2 . For simplicity , we present our idea based on 1D signal . z ` +1 = ( z ` +11 , z ` +1 2 , · · · , z ` +1 C ` +1 ) , z ` +1k = ϕ C∑̀ j=1 a ` kj ∗ z ` j ( 1 ≤ k ≤ C ` +1 ) , where ∗ denotes the convolution between two items that we will discuss below in more detail . Thus , for the ` th layer with C ` input channels and C ` +1 output channels , we can organize the convolution kernels { akj } as A ` = a ` 11 a ` 12 · · · a ` 1C ` a ` 21 a ` 22 · · · a ` 2C ` ... ... . . . ... a ` C ` +11 a ` C ` +12 · · · a ` C ` +1C ` . Convolution operators . For the simplicity of presentation and analysis , we adopt circular convolution instead of linear convolution.6 For 1D signal , given a kernel a ∈ Rn and an input signal x ∈ Rm ( in many cases m n ) , a circular convolution ∗ between a and x can be written in a simple matrix-vector product form via y = a ∗ x = Ca · x , where Ca denotes a circulant matrix of ( zero-padded ) a , Ca : = [ s0 [ a ] s1 [ a ] · · · sm−1 [ a ] ] , which is the concatenation of all cyclic shifts sk [ a ] ( 0 ≤ k ≤ m−1 ) of length k of the ( zero-padded ) vector a . Since Ca can be decomposed via the discrete Fourier transform ( DFT ) matrix F : Ca = F ∗ diag ( â ) F , â = Fa , ( 3 ) where â denotes the Fourier transform of a vector a . The computation of a ∗ x can be carried out efficiently via fast Fourier transform ( FFT ) in the frequency domain . We refer the readers to the appendix for more technical details .

Summary: The paper introduces a new way to enforce orthogonality in convolutional layers. The main idea is to make the computations in the frequency domain where convolutions (or cross-correlations) become products. Hence, exact orthogonality is achieved, while keeping the computational costs low using FFT.