Network Deconvolution
Authors: Chengxi Ye
Matthew Evanusa
Hua He
Anton Mitrokhin
Tom Goldstein
James A. Yorke
Cornelia Fermuller
Yiannis Aloimonos
Venue:  (2020)
Keywords: convolutional networks
network deconvolution
whitening

Abstract: Convolution is a central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image. However, because of the strong  correlations in real-world image data, convolutional kernels are in effect re-learning redundant data. In this work, we show that this redundancy has made neural network training challenging, and propose network deconvolution, a procedure which optimally removes pixel-wise and channel-wise correlations before the data is fed into each layer. Network deconvolution can be efficiently calculated at a fraction of the computational cost of a convolution layer. We also show that the deconvolution filters in the first layer of the network resemble the center-surround structure found in biological neurons in the visual regions of the brain. Filtering with such kernels results in a sparse representation, a desired property that has been missing in the training of neural networks. Learning from the sparse representation promotes faster convergence and superior results without the use of batch normalization. We apply our network deconvolution operation to 10 modern neural network models by replacing batch normalization within each. Extensive experiments show that the network deconvolution operation is able to deliver performance improvement in all cases on the CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Cityscapes, and ImageNet datasets.

Published as a conference paper at ICLR 2020
NETWORK DECONVOLUTION
Chengxi Ye;
Matthew Evanusa,
Hua He,
Anton Mitrokhin,
Tom Goldstein, James A. Yorke;
Cornelia Fermüller,
Yiannis Aloimonos
Department of Computer Science, University of Maryland, College Park (cxy, mevanusa, huah, amitrokh } @umd edu (tomgêcs, yorkel fer@umiacse yoss,yiann yiannisecs, .ud.edu
ABSTRACT
Convolution is central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image. However, because of the strong correlations in real-world image data, convolutional kernels are in effect re-learning redundant data. In this work, we show that this redundancy has made neural network training challenging, and propose network deconvolution, a procedure which optimally removes pixel-wise and channel-wise correlations before the data is fed into each layer. Network deconvolution can be efficiently calculated at a fraction of the computational cost of convolution layer. We also show that the deconvolution filters the first layer of the network resemble the center-surround structure found in biological neurons in the visual regions of the brain. Filtering with such kernels results in a sparse representation, desired property that has been missing in the training of neural networks. Learning from the sparse representation promotes faster convergence and superior results without the use of batch normalization. We apply our network deconvolution operation to 10 modern neural network models by replacing batch normalization within each. Extensive experiments show that the network deconvolution operation is able to deliver performance improvement in all cases on the CIFAR-10, CIFAR- 100, Cityscapes, and ImageNet datasets.
MNIST,
INTRODUCTION
Images of natural scenes that the human eye or camera captures contain adjacent pixels that are statistically highly correlated (Olshausen & Field 1996, Hyvrinen et al., 2009). which can be compared to the correlations introduced by blurring an image with a Gaussian kernel. We can think of images as being convolved by an unknown filter (Figure D.. The correlation effect complicates object recognition tasks and makes neural network training challenging, as adjacent pixels contain redundant information.
has been discovered that there exists visual correlation removal processes animal brains. Visual neurons called retinal ganglion cells and lateral geniculate nucleus cells have developed "Mexican hat"-like circular center-surround receptive field structures to reduce visual information redundancy, found in Hubel and Wiesel's 's famous cat experiment (Hubel & Wiesel 1961 1962). Furthermore, it has been argued that data compression is an essential and fundamental processing step in natural brains, which inherently involves removing redundant information and only keeping the most salient features (Richert et al. 2016).
In this work, we introduce network deconvolation, method to reduce redundant correlation in images. Mathematically speaking, a correlated signal is generated by a convolution: b b= 1 x = Kw (as illustrated in Fig. F riigtt, where k is the kernel and K the corresponding convolution matrix. The purpose of network deconvolution is to remove the correlation effects via: x = K-16, assuming K is an invertible matrix.
Cooresponning Author
I Institute for Physical Science and Technology

---

Published as a conference paper at ICLR 2020
Decomvolatio
Decomolation
???
Convolution
Figure 1: Performing convolution on this real world image using a correlative filter, such as Gaussian kernel, adds correlations to the resulting image, which makes object recognition process removing this blur is called deconvolution. What if, however, what we saw as the real world image was itself the result of some unknown correlative filter, which has made recognition more difficult? Our proposed network deconvolution operation can decorrelate underlying image features which allows neural networks to perform better.
difficult. The
Image data being fed into a convolutional network (CNN) exhibit two types of correlations. Neigh- boring pixels in single image feature map have high pixel-wise correlation. Similarly, in the case of different channels of a hidden layer of the network, there is strong correlation or "cross-talk" refer to this channel-wise correlation. The goal of this paper is to show
between these channels; that both kinds of correlation or redundancy hamper effective learning. Our network deconvolution attempts remove both correlations in the data every layer of network.
Our contributions are the following:
We introduce network deconvolution, a decorrelation method remove both the pixel-wise and channel-wise correlation at each layer of the network.
Our experiments show that deconvolution can replace batch normalization as a generic procedure in variety of modern neural network architectures with better model training.
We prove that this method is the optimal transform if considering L2 optimization.
Deconvolution has been misnomer in convolution architectures. We demonstrate that network deconvolution is indeed deconvolution operation.
We show that network deconvolution reduces redundancy in the data, leading to sparse representations each layer of the network.
We propose novel implicit deconvolution and subsampling based acceleration technique allowing the deconvolution operation to be done at a cost fractional to the corresponding convolution layer.
We demonstrate the network deconvolution operation improves performance in comparison to batch normalization on the CIFAR-10, CIFAR- 100, MNIST, Fashion-MNISS Cityscapes, and ImageNet datasets, using 10 modern neural network models.
RELATED WORK
2.1 NORMALIZATION AND WHITENING
Since its introduction, batch normalization has been the main normalization technique (Ifffe & Szegedy 2015) facilitate the training of deep networks using stochastic gradient descent (SGD). Many techniques have been introduced to address cases for which batch normalization does not perform well. These include training with a small batch size (Wu He 2018), and on recurrent networks (Salimans & Kingmal 2016). However, to our best knowledge, none of these methods has demonstrated improved performance on the ImageNet dataset.
In the signal processing community, network deconvolution could be referred to whitening deconvolution. There have been multiple complicated attempts to whiten the feature channels and to utilize second-order information. For example, authors of (Martens & Grosse 2015 Desjardins et al, 2013) approximate second-order -ondd- information using the Fisher information matrix. There, the whitening is carried out interwoven into the back propagation training process.
The correlation between feature channels has recently been found to hamper the learning. Simple approximate whitening be achieved by removing the correlation in the channels. Channel-wise
2

---

Published as a conference paper at ICLR 2020
decorrelation was first proposed in the backward fashion et 2017). Equivalently, this procedure can also be done in the forward fashion by a change of coordinates (Ye et al 2018 Huang et al. [2018) 2019). However, none of these methods has captured the nature of the convolution operation. which specifically deals with the pixels. Instead, these techniques are most appropriate for the standard linear transform layers in fully-connected networks.
2.2 DECONVOLUTION OF DNA SEQUENCING SIGNALS
Similar correlation issues also exist in the context of DNA sequencing (Ye et al.) (014) where many DNA sequencers start with analyzing correlated signals. There is cross-talk effect between different sensor channels, and signal responses of one nucleotide base spread into its previous and next nucleotide bases. As result the sequencing signals display both channel correlation and pixel correlation. blind deconvolution technique was developed to estimate and remove kernels to recover the unblurred signal.
MOTIVATIONS
3.1
SUBOPTIMALITY OF EXISTING TRAINING METHODS
Deep neural network training has been challenging research topic for decades. Over the past decade, with the regained popularity of deep neural networks, many techniques have been introduced to improve the training process. However, most of these methods are sub-optimal even the most basic linear regression problems.
Assume we are given a linear regression problem with L2 loss (Eq. D. In typical setting, the output y Xw is given by multiplying the inputs X with unknown weight matrix w, which solving for. In our paper, with a slight abuse of notation, X can be the data matrix or the augmented data matrix (X]
Loss
2 11
0||
2 | - -
(1)
Here is the response data to be regressed. With neural networks, gradient descent iterations (and its variants) are used to solve the above problem. To conduct one iteration gradient descent on Eq. [
have:
1 Xx* Xwsst X
Wnew Wold
Here a the step length or learning rate. Basic numerical experiments tell us these iterations can take long to converge. Normalization techniques, which are popular in training neural networks, are beneficial, but are generally not optimal for this simple problem. If methods are suboptimal for simplest linear problems, less likely for them be optimal for more general problems. Our motivation is to find and apply what is linearly optimal to the more challenging problem of network training.
For the L2 regression problem. an optimal solution can be found by setting the gradient to 0: aLoss X"x
Ow
w = (X'x)-
(3)
Here, we ask fundamental question: When gradient descent converge in single iteration?
Proposition 1. Gradient descent converges to the optimal solution in one iteration if 1/2 X I.
Proof. Substituting X T, into the optimal solution (Eq. we have u = 1 X
On the other hand, substituting the same condition with a in Eq. 2 we have Wnew = 1/2 X
Since gradient descent converges in single step, the above proof gives the optimality condition.
3

---

Published as a conference paper at ICLR 2020
X calculates the covariance matrix of the features. The optimal condition suggests that the features should be standardized and uncorrelated with each other. When this condition does not hold, the gradient direction does not point to the optimal solution. In fact, the more correlated the data, the slower the convergence (Richardson 1911). This problem could be handled equivalently (Section A.8) either by correcting the gradient by multiplying the Hessian matrix N (X*X)=
N
or by a change of coordinates so that in the new space we have N X I. This paper applies the latter method for training convolutional networks.
3.2 NEED OF SUPPORT FOR CONVOLUTIONS
Even though normalization methods were developed for training convolutional networks and have been found successful, these methods are more suitable for non-convolutional operations. Existing methods normalize features by channel or by layer, irrespective of whether the underlying operation convolutional or not. We will show in section 4.1 that if the underlying operation is convolution, this usually implies a strong violation of the optimality condition.
3.3 A NEUROLOGICAL BASIS FOR DECONVOLUTION
Many receptive fields in the primate visual cortex exhibit center- surround type behavior. Some receptive fields, called on-center cells respond maximally when a stimuli is given at the center of the receptive field and lack stimuli is given in a circle surrounding it. Some others, called off-center cells respond maximally in the reversed way when lack of stimuli is at the center and the stimuli is given in a circle surrounding it (Fig. (9) (Hubel & Wiesel 1961 1962). It is well-understood that these center-surround fields form the basis of the simple cells in the primate VI cortex.
If the center-surround structures are beneficial for learning, one might expect such structures to be learned in the network training process. As shown in the proof above, the minima is the same with without such structures, so gradient descent does not get an incentive to develop a faster solution, rendering the need to develop such structures externally. As shown later in Fig. 2 our deconvolution kernels strongly resemble center-surround filters like those in nature.
4
THE DECONVOLUTION OPERATION
4.1 THE MATRIX REPRESENTATION OF A CONVOLUTION LAYER
The standard convolution filtering x kernel, can be formulated into one large matrix multiplication
Xw (Fig. 3). In the 2-dimensional case, w is the flattened 2D kernel. The first column of X corresponds to the flattened image patch of 2 If - k. 1 W - kj.. where k is the side length of the kernel. Neighboring columns correspond to shifted patches of æ: X(:, = vec(x[1: : H k, :: + 1)), .... X[: k-2 vec(=lk H, k. WT). A commonly used function im2col has been designed for this operation. Since the columns of X are constructed by shifting large patches of by one pixel, the columns of X are heavily correlated with each other, which strongly violates the optimality condition. This violation slows down the training algorithm (Richardson, 1911). and cannot be addressed by normalization methods (Ioffe & Szegedy 2015)).
W -
For a regular convolution layer in a network, we generally have multiple input feature channels and multiple kernels in layer. We call im2col in each channel, and horizontally concatenate the resulting data matrices from each individual channel to construct the full data matrix, then vectorize and concatenate all the kernels to get w. Matrix vector multiplication is used to calculate the output y. which is then reshaped into the output shape of the layer. Similar constructions can also be developed for the specific convolution layers such as the grouped convolution, where we carry out such constructions for each group. Other scenarios such when the group number equals the channel number (channel-wise conv) or when & = 1 (1 x conv) can be considered as special cases.
In Fig. 3 (top right) we show an illustrative example the resulting calculated covariance matrix of a sample data matrix X in the first layer of a VGG network (Simonyan & Zisserman, 2014) taken from one of our experiments. The first layer is 3 3 convolution that mixes RGB channels. The total dimension of the weights is 27, the corresponding covariance matrix is 27 x 27. The diagonal blocks correspond to the pixel-wise correlation within x 3 neighhorhoods. The off diagonal blocks
4

---

Published as a conference paper at ICLR 2020
correspond to correlation of pixels across different channels. We have empirically seen that natural images demonstrate stronger pixel-wise correlation than cross-channel correlation, as the diagonal blocks are brighter than the off diagonal blocks.
4.2 THE DECONVOLUTION OPERATION
Once the covariance matrix has been calculated, an inverse correction can be applied. is beneficial to conduct the correction in the forward way for numerical accuracy and because the gradient of the correction can also be included the gradient descent training.
Given a data matrix XNx F described above in section 4.1, where N is the number of samples, and F is the number of features, we calculate the covariance matrix Cov = ( 11)² 1x)²(XX (X 1t).
We then calculate an approximated inverse square root of the covariance matrix D Cov (see section 4.5.3) and multiply this with the centered vectors (X -+) D. In a sense, we remove the correlation effects both pixel-wise and channel-wise If computed perfectly, the transformed data has 00.5 Cov - Cov U.5 = I.
the identity matrix as covariance: D (X - n) (X D Cou-0.5
Algorithm| describes the process to construct X and D = (Cov + e. .T- Here iis introduced improve stability. We then apply the deconvolution operation via matrix multiplication to remove the correlation between neighboring pixels and across different channels. The deconvolved data is then multiplied with w. The full equation becomes y = (X - p1) D w + b, or simply y = X D-D-w if X is the augmented data matrix (Fig. 3
We denote the deconvolution operation in the i-th layer as D,. Hence, the input to next layer 21+1 is:
(4) where o is the (right) matrix multiplication operation, Ij is the input coming from the i-th layer, D; the deconvolution operation that input, W; is the weights in the layer, and ft is the activation function.
I++1 fi W, D; 24;
4.3
JUSTIFICATIONS
4.3.1
ON NAMING THE METHOD DECONVOLUTION
The name network deconvolution has also been used in inverting the convolution effects in biological networks (Feizi et al. 2013). We prove that our operation indeed generalized deconvolution operation.
Proposition 2. Removal of pixel-wise correlation (or patch-based whitening) is a deconvolution operation.
Proof. Let 8 be the delta kernel, be arbitrary signal, and X im2col
x * 8 X 8 = X Cov Cov 8 = X Cov Keov x 8
(5)
The above equations show that the deconvolution operation negates the effects of the convolution using kernel Reov 0.5 6.
4.3.2 THE DECONVOLUTION KERNEL
The deconvolution kernel can be found as 0.5 vec(ō), where vec(:) is the Vectorize function, or equivalently by slicing the middle row/column of -05 and reshaping into the kernel size. We visualize the deconvolution kernel from 1024 random images from the ImageNet dataset. The kernels indeed show center-surround structure, which coincides with the biological observation (Fig. 2). The filter in the green channel is on-center cell while the other two off-center cells (Hubel Wiesel 1961 1962).
4.4 OPTIMALITY
Motivated by the ubiquity of senter-surround and interallinhiitiion mechanisms in biological neural systems, we now ask if removal of redundant information, in manner like our network deconvolution, is an optimal procedure for learning in neural networks.
5

---

Published as a conference paper at ICLR 2020
-1
42
3
Figure Visualizing the 15 x 15 deconvolution kernels from 1024 random images from the ImageNet dataset. The kernels in the R, R.G.B channels consistently show surrond structures.
4.4.1 L2 OPTIMIZATIONS
There classic kernel estimation problem (Cho & Lee 2009 Ye et al. 2014) that requires solving for the kernel given the imput data X and the blurred output data y. Because X violates the optimality condition, it takes tens or hundreds of gradient descent iterations to converge to a close enough solution. We have demonstrated that our deconvolution processing is optimal for the kemnel estimation, in contrast to all other normalization methods.
4.4.2 ON TRAINING NEURAL NETWORKS
Training convolutional neural networks is analogous series of kernel estimation problem, where we have to solve for the kernels in each layer. Network deconvolution has favorable near-optimal property for training neural networks.
For simplicity, we assume the activation function is a sample-variant matrix multiplication throughout the network. The popular ReLU (Nair & Hinton 2010) activation falls into this category. Let W be the linear cotioooooooooooiiiononoooooooooo in a certain layer. A the inputs to the layer, B the operation from the output of the current layer to the output the last deconvolution operation in the network. The computation of such network can be formulated as: A.B.
Proposition 3. Network deconvolution is near-optimal if W is an orthogonal transform and if we connect the output y to the L2 loss.
Rewriting the matrix equation from the previous subsection using the Kronecker product, we get: y AWB B (B" @ A))ee(W)) Xeec(W), where @ is the Kronecker product. According to the discussion in the previous subsection, gradient descent is optimal if X = (B" @ A) satisfies the orthogonality condition. In network trained with deconvolution, the input A is orthogonal for each batch of data. If B is also orthogonal, then according to basic properties of the Kronecker product (Golub & van Loan 2013)(Ch 12.3.1), (B3 & A) is also orthogonal. y is orthogonal since it is the output of the last deconvolution operation in our network. If we assume W is an orthog- onal transform, then transforms orthogonal inputs to orthogonal outputs, and approximately orthogonal.
Slight loss of optimality incurs since we do not enforce W to be orthogonal. But the gain here is that the network is unrestricted and is promised to be as powerful as any standard network. On the other hand, is worth mentioning that many practical loss functions such as the cross entropy loss have similar shapes to the T22 loss.
4.5 ACCELERATIONS
We note that in a direct implementation, the runtime of our training using deconvolution is slower than convolution using the wallclock implicit calculation of the matrices existing libraries. We propose acceleration techniques to reduce the deconvolution cost to only a fraction of the convolution layer (Section A.6). Without further optimization, our training speed similar to training a network using batch normalization on the ImageNet dataset while achieving better accuracy. This is desired property when faced with difficult models (Goodfellow et al. 2014) and with problems where the network part is not the major bottleneck Ye al.) 2018).
metric. This is due to the suboptimal support in the

---

Published as a conference paper at ICLR 2020
(Xx)
Im2col
25
Cay
Figure 3: (Left) Given single channel image, and 3 x 3 kernel, the kernel is first flattened into 9 dimensional vector w. The 9 image patches, corresponding to the image regions each kernel entry sees when overlaying the kernel over the image and then shifting the kernel one pixel each step, are flattened into a tall matrix X. It is important to note that because the patches are shifted by just one pixel, the columns of X are highly correlated. The output y is calculated with matrix multiplication Xw, which is then reshaped back into a 2D image. (Top Right) In a convolution layer the matrix X and Cov is calculated from Algorithm (Bottom Right) The pixel-wise and channel-wise correlation removed by multiplying this X matrix with with Cov 1, before the weight training.
4.5.1 IMPLICIT DECONVOLUTION
Following from the associative rule of matrix multiplication, y - w). which suggests that the deconvolution can be carried out implicitly by changing the model parameters, without explicitly deconvolving the data. Once we finish the training, we freeze to the running average. This change of parameters makes a network with deconvolution perform faster at testing time, which is a favorable property for mobile applications. We provide the practical recipe to include the bias term: y (X μ)) D w + b X (D w) b p- W.
4.5.2 FAST COMPUTATION OF THE COVARIANCE MATRICE
We propose simple 5 = 3 5x subsampling technique that speeds up the computation of the covariance matrix by a factor of 10 20. Since the number of involved pixels is usually large compared with the degree of freedom in convariance matrix (Section A.6), this simple strategy provides significant speedups while maintaining the training quality. Thanks to the regularization and the iterative method discuss below, we found the subsampling method to be robust even when the covariance matrix is large.
4.5.3 FAST INVERSE SQUARE ROOT OF THE COVARIANCE MATRIX
Computing the inverse square root has long and fruitful history in computer graphics and numerical mathematics. Fast computation of the inverse square root of scalar with Newton-Schulz iterations has received wide attention in game engines (Lomont 2003). One would expect the same method to seamlessly generalize the matrix case. However, according to numerous experiments, the standard Newton-Schulz iterations suffer from severe numerical instability and explosion after ~ 20
iterations for simple matrices (Section AA.) (Higham, 1986)(Eq. 7.12), (Higham) 2008). Coupled
Newton-Schul. iterations have been designed (Denman & Beavers 1976) (Eq.6.35), (Higham 2008
and been proved to be numerically stable.
We compute the approximate inverse square root of the covariance matrix at low cost using coupled Newton-Schulz iteration, inspired by the Denman- Beannrs iteration method (Denman Beavers 1976). Given symmetric positive definite covariance matrix Cov, the coupled Newton-Schull iterations start with initial values Yo = Coc, Zo = I. The iteration defined as: Y = Y
Y,Y,), Zk+1 } (3I - Z Zt. and Y₁ + Cou Z₁ Cou 3/ (Higham 2008 (Eq.6.35). Note that this coupled iteration has been used in recent works to calculate the square root of a matrix (Lin & Maji 2017). Instead, take the inverse square root from the outputs, as first shown in Ye et al.
7

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
Figure 4: (a) The input image. (b) The absolute value of the zero-meaned input image. (c) The deconvolved input image (min-max normalized, gray areas stands for 0). (d) Taking the absolute value of the deconvolved image.
2018). In contrast with the vanilla Newton- -chuulz method (Higham 2008 Huang et al.) (01 7.12), we found the coupled Newton-Schulz iterations are stable even if iterated for thousands of times.
It is important to point out a practical implementation detail: when we have Chin input feature channels, and the kernel size k k. the size of the covariance matrix (Chin k k) (Chi, x k x k). The covariance matrix becomes large in deeper layers of the network, and inverting such matrix is cumbersome. We take a grouping approach by evenly dividing the feature channels Chin into smaller blocks (Ye al. 2017 Wu & He 2018 Ye et al., 2018); we us b to denote the block size, and usually set B = 64. The mini- covariance
each block has a manageable size of Newton-Schule iterations are therefore conducted on smaller matrices. We notice that only a few (~~ 5) iterations are necessary to achieve good performance. Solving for the x )) The computation of the covariance matrix has complexity 5x5 O(I W k x Chin x B 5x5).
(B x k k) (B k x k).
inverse square root takes ((kk Chiz
O(H x W k x B B deconvolution is simple matrix multiplication with complextity O(Chout X (B x k x k) Chan
). Implicit
B
The overall complexity O Hxw x CCh., xB (k k B)3 Ch (B k: k)²
SXS
Chin ), which is usually a small fraction of the cost of the convolution operation (Section A.6). A
In comparison, the computational complexity of regular convolution layer has complexity of O(H x W x k x k x Chin X Chout).
Algorithm 1 Computing the Deconvolution Matrix
1: Input: C channels of input features [r1, T2, ... xcl
2: for E {l,..., C} do
3: X; = im2ccll(z))
4: end for
5: X =[X1,... [c] Horrioonyylly Concatenate
6: X = Reshape( X) Diviid columns into groups
7: Cov = 1 X
N
8: D& (Cov + 11
4.6 SPARSE REPRESENTATIONS
Our deconvolution applied at each layer removes the pixel-wise and channel-wise correlation and transforms the original dense representations into sparse representations (in terms of heavy-tailed distributions) without losing information. This is desired property and there is whole field with wide applications developed around sparse representations (Hyvrinen et al., (009) 7.7). (OT: Shausen & Field, 1996] Ye et al.) al.] (013). In Fig. 4 we visualize the deconvolution operation on an input and show how the resulting representations d)) are much sparser than the normalized image ( (b)).. We randomly sample 1024 images from the ImageNet and plot the histograms and log density functions before and after deconvolution (Fig. 10). After deconvolution, the log density distribution becomes heavy-tailed. This holds true also for hidden layer representations (Section[A.5
We show in the supplementary material (Section A.4) that the sparse representation makes classic regularizations more effective.
8

---

Published as a conference paper at ICLR 2020
5 A UNIFIED VIEW
Network deconvolution is a forward correction method that has relations to several successful techniques in training neural networks. When we set i decorrelation, as in (Ye et al.) 2018 Huang et al. 2018). When k: = 1, B = 1, network deconvolution Batch Normalization (lofte & Szegedy 2015). If we apply the decorrelation in a backward way in the gradient direction, network deconvolution is similar to SGD2 (Ye et al. 2017), natural gradient
the method becomes channel-wise
descent (Desjardins al. (201)) and K FAAC (Mariens & Grosse 2015). while being more efficient and having better numerical properties (Section A.8
6 EXPERIMENTS
We now describe experimental results validating that network deconvolution is a powerful and successful tool for sharpening the data. Our experiments show that it outperforms identical networks using batch normalization (loffe & Szegedy 2015), major method for training neural networks. As we will see across all experiments, deconvolution not only improves the final accuracy but also decreases the amount of iterations it takes to learn reasonably good set of weights in small number of epochs.
I Linear Regression with L2 loss and Logistic Regression: As a first experiment, we ran network deconvolution on a simple linear regression task to show its efficacy. We select the Fashion-MNIST dataset. It is noteworthy that with binary targets and the L2 loss, the problem has explicit solution we feed the whole dataset as input. This problem is the classic kernel estimation problem, where need solve for 10 optimal 28 x 28 kernels to convolve with the inputs and minimize the L2 loss for binary targets. During our experiment, we notice that it is important to use a small learning rate of 0.02 0.1 for vanilla SGD training to prevent divergence. However, we notice that with deconvolution we can use the optimal learning rate 1.0 and get high accuracy as well. It takes ~ 5 iterations get low cost under the mini-batch setting (Fig. 5())). This even holds if we change the loss to logistic regression loss (Fig. ()).
(a)
(b)
(c)
500
(d)
5G0
530
Eyata
Figure 5: (a-b): Regression losses on Bushion-MNISTI dataset showing the effectiveness of deconvo- lution versus batch normalization on a non-convolutional type layer. (a) One layer, linear regression model with Ly loss. (b) One layer, linear regression model with logistic regression loss. (c-d): Results of a 3-hidden-layer Multi Layer Perceptron (MLP) network on the MNIST dataset.
Convolutional Networks on CIFAR-10/100: We ran deconvolution on the CIFAR-1O 10 and CIFAR- 100 datasets (Table where we compared again the use of network deconvolution versus the use of batch normalization. Across 10 modern network architectures for both datasets, deconvolution consistently improves convergence on these well nell-known datasets. There is a wide performance gap after the first epochs of training. Deconvolution leads to faster convergence: 20-epoch training using deconvolution leads to results that are comparable to 100-epoch training using batch normalization.
In our setting, we remove all batch normalizations in the networks and replace them with deconvo- lution before each
catattiionnilllycooeettttt layer. For the convolutional layers, we set B 64 before calculating the covariance matrix. For the fully-connected layers, we set B equal to the input channel number, which is usually 512. We set the batch size to 128 and the weight decay to 0.001. All models are trained with SGD and a learning rate of 0.1.
Convolutional Networks ImageNet: We tested three widely acknowledged model architectures (VGG-11, 11, ResNet- 18, DenseNet-1211 121) from the Py Torch model zoo and find significant improvements on both networks over the reference models. Notably, for the VGG-11 network, we notice our method

---

Published as a conference paper at ICLR 2020
CIFAR-10
CIFAR-100
Ner Size
RN
NOV
RN 20
ND 20
BN 1D0
ND 100
BN
NO
RN 201
ND 20
BN 100
ND T00
VGG-16
14.71M
14.12%
74.18%
90,07%
03.25%
93, 93.58%
94.56%
2,015
37.94%
63.228
71.97%
72.75%
75.32%
R8
11.17
56.25%
72.89%
92.64%
94.17%%
87%
95.40%
35.73%
72.67%
76.55%
77.70%
78.6%%
Presc--88
55.15%
72.70%
91.933
94.10%
94. 375
95.44%
15.17%
36.52%
70.79%
76.04%
76.
79.14%
DenseNet-12 121
6.8M
59,5616
76.63%
93.251%
94.89%
94.71%
95.88%6
17.90%
42.91%
74,79%
77.63%
77.99%
90.69%
ResNext-29
4.76M
52.14%
69.22%
93.12%
94.05%
95.15%
95.80%
17.98%
30.93%
74,26%
77.38%
78.60%
90.34%
MobileNet
2.28M
54,29%
65.40%
92.52%
90.51%
15.88%
29.01%
66.31%
72.33%
67.52%
DPN-92
34,18M
34,00%
53.02%
92.87%
93.74%
95.14%
95.82%
8.84%
21.89%
74.87%
76.12%
78.87%
90.38%
PNASNetA
0.13M
21.81%
64.19%
75.85%
81.97%
81.22%
$4.45%
10.49%
36.52%
44.60%
55.65%
54.52%
59.44%
SENec-18
11.26M
57.63%
67.21%
92.37%
94.11%
94.57%
95.38%
16.60%
32.22%
71.10%
75.79%
76.41%
78.63%
EfficiemNerr
291M
35,40%
84.21%
86.78%
86.07%
19,03%
22.40%
57.23%
57.59%
59.09%
Table 1: Comparison on CIFAR-10/100 over 10 modern CNN architectures. Models are trained for 1, 20, 100 epochs using batch normalization (BN) and network deconvolution (ND). Every single model shows improved accuracy using network deconvolution.
VGG-11
ResNet- 18
DenseNet-122
Original
BN
Deconv
BN
Deconv
BN
Deconv
ImageNet top-1
69.02 69.02%
70.38%
71.95% %
69.76%
71.24%
74.65%
75.73%
ImageNet top-5
88.63%
89.81 %
90.49%
89.08% %
90.14%
92.17%
92.75%
Table 2: Comparison of accuracies of deconvolution with the model zoo implementation of VGG-11, ResNet-18, DenseNet- 121 on ImageNet with batch normalization using the reference implementation on PyTorch. For VGG-11, we also include the performance of the original network without batch normalization.
has led to significant improved accuracy. The top-1 accuracy is even higher than 71.55%, reported by the reference VGG-13 model trained with batch normalization. The improvement introduced by network deconvolution is twice large that from batch normalization (+1.66%). This fact also suggests that improving the training method may be more effective than improving the architecture.
We keep most of the default settings when training the models. We set B 64 for all deconvolution operations. The networks are trained for 90 epochs with a batch size of 256, and weight decay of 0.0001. The initial learning rates are 0.01,0.1 and 0.1, respectively for VGG- 1.,RReeetttt 18 and DenseNet- 121 as described in the paper. We used cosine annealing to smoothly decrease the learning rate compare the curves.
Generalization to Other Tasks is worth pointing out that network deconvolution can be applied to other tasks that have convolution layers. Further results semantic segmentation the Cityscapes dataset can be found in (Sec. A.8). Also, the same deconvolution procedure for 1 X 1 convolutions can be used for non-convolutional layers, which makes useful for the broader machine learning community. We constructed a 3-layer fully-connected network that has 128 hidden nodes in each layer and used the sigmoid for the activation function. We compare the result with/without batch normalization, and deconvolution, where we remove the correlation between hidden nodes. Indeed, applying deconvolution to MLP networks outperforms batch normalization, as shown Fig. c,,d).
CONCLUSION
In this paper we presented network deconvolution, a novel decorrelation method tailored for convo- lutions, which is inspired by the biological visual system. Our method was evaluated extensively and shown to improve the optimization efficiency over standard batch normalization. We provided a thorough analysis of its performance and demonstrated consistent performance improvement of the deconvolution operation multiple major benchmarks given 10 modern neural network models. Our proposed deconvolution operation is straightforward in terms of implementation and can serve as good alternative batch normalization.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Sunghyun Cho and Seungyong Lee. Fast motion deblurring. In ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia '09, pp. 145:1-145:8, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-
858-2. doi: 00.1145/1614112.66188) 1618491
Eugene D. Denman and Alex N. Beavers, Jr. The matrix sign function and computations in systems. Appl. Math. Comput., 2(1):63- 94, January 1976. ISSN -006-30033 doi: 10.1016/0096-0000 3003(676)))) 90020-5. UtLhttt: doi org/10 1110116 000666-006060000 3003 90020-5
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and koray kavukcuoglu. Natural neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2071 Curran Associates, Inc., 2015. URL http::/paperr: nips wttp:/ccppppppreee.ee htt.://pppr/555-natttr- - - pdf
Soheil Feizi, Daniel Marbach, Muriel Médard, and Manolis Kellis. Network deconvolution as
general method to distinguish direct dependencies in networks. Nuture biotechnology, 2013.
Gene H. Golub and Charles F. van Loan. Matrix Computations. JHU Press, fourth edition, 2013. ISBN 1421407944 9781421407944. URL http:://wwy cs www.ecooo..ccooo. www..oeee.....c///......... go golubandvan htm.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Nicholas J. Higham. Newton's method for the matrix square root. 1986.
Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2008. ISBN 978-0-898716-46-77
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 791-800, 2018
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standadiiaiinnn towards efficient whitening. CoRR, abs/1104.0344. 2019. URL htt.htttp///raa..orr//////
1904.00 03441
D. H. Hubel and T. N. Wiesel. Integrative action in the cat's lateral geniculate body. The Journal of Physiology, 155(2):385-398, 1961. doi: 10.1 1//physll 1961sp006663.. URL https:t//ppppeese:
wwwoooonieee.. wwwwonnniaannry.yoooeoemaaaaaael wiley. com/ doi cicom//do//b/11000 abs/10
1113/ jt13//pphyio....mm 1961. sp006635
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architec- ture in the cat's visual cortex. The Journal of physiology, 160(1):106-155 1962.
Aapo Hyvrinen, Jarmo Hurri, and Patrick O. Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer Publishing Company, Incorporated, 1st edition, 2009. ISBN 1848824904, 7781848824904.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint
2015.
Tsung- Yu Lin and Subhransu Maji. Improved bilinear pooling with cnns. CoRR, abs/ 770.06772.
2017. URL http: org.
Chris Lomont. Fast inverse square root. Tech-315 nical Report, 32, 2003.
James Martens and Roger B. Grosse. Optimizing neural networks with Kronecker-factored ap- proximate curvature. CoRR, abs/1503.05671, 2015. URL htlltttpp//axiivv wwt.:/.rggaaaab//bbeeeae
05671
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
11

---

Published as a conference paper at ICLR 2020
Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583)660766000 609, 1996. ISSN 1476-4687. doi: 10.1038/38160777 URL URL httpt:://///ooooo org wttp:///...111///111111
L. F. Richardson. The approximate arithmetical solution by finite differences of physical problems involving differential equations, with an application the stresses in a masonry dam. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 210:307-357, 1911. ISSN 02643952. URL BRLLhttp:// http:://www... www. httttt::/www..sttr.oooo org/
stable/90994
Micah Richert, Dimitry Fisher, Filip Piekniewski, Eugene M Izhikevich, and Todd L Hylton. Funda- mental principles of cortical computation: unsupervised learning with prediction, compression and feedback. arXiv preprint
2016.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparamcterization to accelerate training of deep neural networks. CoRR, abs/1602.07866. 2016. URL http://
arxiv. org. abs / 602.0022220 07868
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1et::1109.1555 2014.
Yuxin Wu and Kaiming He. Group normalization. CoRR, abs/1803.08494, 2018. URL http:
7/arxiv. httpxiv.orr/bb///800000 08494
Chengxi Ye, Dacheng Tao, Mingli Song, David W. Jacobs, and Min Wu. Sparse norm filtering, CoRR, abs/1305.3971, 2013. URL
Chengxi Ye, Chiaowen Hsiao, and Héctor Corrada Bravo. BlindCall: Ultra-fast base-calling of high-throughput sequencing data by blind deconvolution. Bioinformatics, 3009):111--1119, 01 2014. ISSN 1367-4803. doi: 10. 1..00//biioooomattts/tull000 URL [ttL/atttp::/dl.orgg////
Chengxi Ye, Yezhou Yang. Cornelia Fermüller, and Yiannis Aloimonos. On the importance of consistency in training deep neural networks. CoRR, abs/1708. 00631, 2017. URL http://
trxiv.org/l
Chengxi Ye, Anton Mitrokhin, Cornelia Fermüller, James A. Yorke, and Yiannis Aloimonos. Unsu- pervised learning of dense optical flow and depth from sparse event data. CoRR, has/1809.086255 2018. Utt.http://a org /aas////////////ae 1809. 08625
A APPENDIX
A.1
SOURCE CODE
Source code can be found at:
The models for CIFAR-IOCIIFAR-O00 are adapted from the following repository:
http://gghhbb.oonkuunniiigyoygoh
A.2 GENERALIZZTION SEMANTIC SEGMENTATION
To demonstrate the applicability of network deconvolution to different tasks, we modify a baseline architecture of DeepLabV3 with ResNet-50 backbone for semantic segmentation. We remove the batch normalization layers in both the backbone network and the head network and pre-apply deconvolutions in all the convolution layers. The full networks are trained from scratch on the Cityscape dataset (with 2, 975 training images) using learning rate of 0.1 for 30 epochs with batch size 8. All settings are the same with the official Py PyTorch recipe except we have raised the learning rate from 0.01 to 0.1 for training from scratch. Here report the mean intersection union (mloU) curves of standard training and deconvolution using a crop size of 480. We achieved significantly improved training results (Fig. 6).
12

---

Published as a conference paper at ICLR 2020
(a)
(b)
Figure 6: The mean IoU, pixel-wise accuracy and training loss the Cityscapes dataset using DeepLabV3 with a ResNet-50 backbone. In our setting, we modified all the convolution layers to remove batch normalizations and insert deconvolutions.
A.3 COUPLED/UNCOUPLEDD NEWTON SCHULZ ITERATIONS
Virila NewronScuur
28
10
70
Iterationn
Figure Comparison of coupled/uncoupled Newton-Schull iterations on 27 covariance matrix constructed from the Lenna Image.
We take the Lenna image and construct the 27 x 27 covariance matrix using pixels from 3 windows in 3 color channels. We apply the vanilla Newton-Schull iteration and compare it with the coupled Newton-Schule iteration. The Frobenius Norm of D D Cov I is plotted in Fig. The rounding errors quickly accumulate with the vanilla Newton Schulz iterations, while the coupled iteration is stable. From the curve we set the iteration number 15 for the first layer of the network to thoroughly remove the correlation in the input data. We freeze the deconvolution matrix D after 200 iterations. For the middle layers of the network we set the iteration number to 5.
A.4 REGULARIZATIONS
If features correlate, weight decay regularization is less effective. If X1, X₂ are strongly correlated features, but differ in scale, and if we look at: Wi + W2X2, the weights are likely to co-adapt during the training, and weight decay is likely to more effective on the larger coefficient. The other, small coefficient is left less penalized. Network deconvolution reduces the co-adaptation of weights, and weight decay becomes less ambiguous and more effective. Here we report the accuracies of the VGG-13 network the CIFAR-100 dataset using weight decays of 0.005 and 0.0005. We notice that a stronger weight decay is detrimental to the performance with standard training using batch normalization. In contrast, the network achieves better performance with deconvolution using a stronger weight decay. Each setting is repeated for times, and the mean accuracy curves with confidence intervals of (+/- 1.0 std) shown in Fig. a).
13

---

Published as a conference paper at ICLR 2020
Cross Entropy Loss VGG-11 on ImageNett
7.5
Validation Accuracy
3GD IN Decom
www.
6
5.5
DOD JVNE DOON DES Coomn NR DE 0553
200
600
800
Iterationn
(a)
(b)
Figure 8: (a) The effects of weight decay on stochastic gradient descent (SGD) and batch normal- ization (BN) versus SGD and deconvolution (Deconv), training on the CIFAR- 100 dataset on the VGG-13 network. Here we notice that increased weight decay leads to worse results for standard training. However, in our with deconvolution, the final accuracy actually improves with increased weight decay (.0005 to .005). Each experiment is repeated 5 times. We show the confidence interval (+/- 1 std). (b) training loss of the VGG-11 network on the ImageNet dataset. Only the first 1000 iterations are shown. Comparison is made among SGD, SGD with batch normalization and deconvolution.
+
Figure 9: An on-center cell and off-center cell found in animal vision system. The on-center cell (left) responds maximally when a stimuli is given at the center and lack of stimuli is given in circle surrounding it. The off-center cell (right) responds in the opposite way.
A.5 SPARSE REPRESENTATION FOR CONVOLUTION LAYERS
Network deconvolution reduces redundancy similar the that in the animal vision system (Fig.
The center-surround antagonism results efficient and sparse representations. In Fig. 10 we plot the distribution and log density of the signals the first layer before and after deconvolution. The distribution after the deconvolution has well-known heavy-tailed shape (Hyvrinen et al. 2009 Ye
al. 2013).
Fig. 11 shows the inputs to the 5-th convolution layer in VGG - 11. This input is the output of a L activation function. The deconvolution operation removes the correlation between channels and nearby pixels, resulting in a sharper and sparser representation.
A.6 PERFORMANCE BREAKDOWN
Network deconvolution is a customized new design and relies on less optimized functions such as Im2col. Even so, the slow down is tunable to be around ~ 10 30% modern networks. We plot the walltime vs accuracy plot of the VGG network on the ImageNet dataset. For this plot we use B 16, S 4 (Fig. L Here also break down the computational cost CPU show deconvolution is a low-cost and promising approuch if properly optimized on GPUs. We take random images at various scales and set the input/output channels common values in modern networks.
14

---

Published as a conference paper at ICLR 2020
Histogram of Signals Before/After Deconv
Log Probability Signals Before/After
Input Signal
Input Signal
After Decomy
After Decort
0.12
0.08
0.04
-8
-9
-10
(a)
(b)
Figure 10: (a) Histograms of input signals (pixel values) before/after deconvolution. (b) Log density of the input signals before/after deconvolution.) The x-axis represents the normalized pixel value. After applying deconvolution, the resulting data is much sparser, with most values being zero
(a)
(b)
Figure 11: (a)Input features to the 5-th convolution layer in VGG-11. (b) Taking the absolute value of the deconvolved features. The features have been min-max normalized for visualization. (Best view on a display.)
The CPU timing on a batch size of 128 can be found Table 3 Here we the Newton-Schut iteration times to be 5.
A.7 ACCELERATED CONVERGENCE
We demonstrate the loss curves using different settings when training the VGG- 11 network on the ImageNet dataset(Fig, b)). We can see network deconvolution leads to significantly faster decay in training loss.
15

---

Published as a conference paper at ICLR 2020
VGG-11 Top-1 Validation Accuracy on ImagaNet
70
Accurscy
30
5cD IN Top Arc Decon: Top ACC
10
Wellime(His))
Figure 12: The accuracy vs walltime curves of VGG-11 networks on the ImageNet dataset using batch normalization and network deconvolution.
H
W
Chan
Chout
groups
stride
Tm2Col
Cov
Tru
Cono
256
256
64
1
3
0,069
0.0079
0.00025
0.50
128
128
64
128
1
3
0.315
0.3214
0.02069
0.67
64
64
128
256
2
3
0.045
0.0445
0.00076
0.55
32
32
256
512
4
3
0.022
0.0391
0.00222
0.48
16
16
512
512
8
3
0.011
0.0444
0.01155
0.23
128
128
64
128
64
3
3
0.376
0,0871
0.00024
0.66
64
64
128
256
128
3
0.042
0.0412
0.00082
0.53
32
32
256
512
256
3
3
0.023
0.0377
0.00208
0.47
16
512
512
512
3
0.011
0.0437
0.01194
0.22
128
128
64
128
32
3
3
0.360
0.0939
0.00021
0.67
64
64
128
256
32
3
0.044
0.0425
0.00076
0.55
32
32
256
512
32
3
3
0,023
0.0374
0.00204
0.46
16
16
512
512
32
3
0,011
0.0421
0,01180
0.22
256
256
64
1
5
0.030
0.0046
0.00029
0.49
128
128
64
128
1
5
0.153
0.1284
0.01901
0.66
256
256
64
3
0.338
0.1111
0.00107
0.69
256
256
64
5
0.180
0.0408
0.00107
0.69
256
256
64
7
0.063
0.0210
0.00104
0.70
256
256
3
64
11
3
0.681
0.4810
0.00479
0.99
256
256
64
5
0.315
0.1909
0.00488
1.00
256
256
3.
64
1
11
7
0.199
0.1022
0.00494
0.99
256
256
64
11
0.069
0.0420
0.00496
1.00
Table 3: Breakdown of deconvolution layer component computation time (in sec., measured on CPU) against various layer parameters. The batch size is set to 128, Hl x W are layer dimensions, Chin number of input channels, Chout - number of output channels, groups the number of channel groups, l is the kernel size and stride is the sampling stride.
A.8 FORWARD BACKWARD EQUIVALENCE
We discuss the relation between the correction in the forward way and in the backward way. We thank Prof. Brian Hunt for providing us this simple proof.
Assuming that in one layer of the network have X W X D W3 Y, W2 D-1 W, here D = Coe U.S
aLoss
aLoss
ay
TLoss
xt.
ow,
OY
ow,
OY
16

---

Published as a conference paper at ICLR 2020
BatchSize
Acc
LR
Iter
2
89.12%
0.001
0.01
2
8
91.26%
0.01
0.01
32
91.18%
0.01
le-5
5
128
91.56%
0.1
le-5
5
512
91.66%
0.5
le-5
5
2048
90.64%
1
le-5
5
Table 4: Performance and settings under different batch sizes.
Assuming D is fixed,
ÖLoss
ØLoss
0Y
ŌLoss 0Y
= (X D)..
(7)
aW2
0Y
aw2
One iteration of gradient descent with respect to W1 is:
ŌLoss
Woold ax. ÔLoss
WTew- Woldd ALass 0Y
(8)
oWald
0Y
LLoss
Loss BY
BY (X D)C
JW2
JW;
One iteration gradient descent with respect to W2 is:
Loss
WREw
Wold
Owold
ŌLoss
0Y
ŌLoss
D-1 WT DD''wpld
D Wald a(X DY
(10)
OY oWold
OY
We then reach the familiar form YYe al.
2017;:
W = Wold aD?. xt. ØY Wold - aCov ((xtt ÔLoss
ŌLoss
(11)
0Y
We have proved the equivalence between forward correction and the Newton's method- like backward correction. Carrying out the forward correction as in paper is beneficial because as the neural network gets deep, gets more ill-posed. Another reason is that because D depends on X, the layer gradients are more accurate if we include the inverse square root into the back propagation training. This is easily achievable with the help automatic differentiation implementations:
dri+1
0Di+1
= D o ft W, +
fs W
(12)
Oz:
dz;
Here v; is input to the current layer and ø441 is the input to the next layer.
A.9 INFLUENCE OF BATCH SIZE
We notice network deconvolution works well under various batch sizes. However, different settings need be adjusted to achieve optimal performance. High learning rates can be used for large batch sizes, small learning rates should be used for small batch sizes. When the batch size is small, to avoid overfitting the noise, the number of Newton-Schule iterations should be reduced and the regularization factor should be raised. More results and settings can be found in Table 4
A. IMPLEMENTATION OF FASTDECONV IN PYTORCH
We present the reference implementation in PyTorch. "FastDeconv" can used to replace instances of "nn. "on.Con in the network architectures. Batch normalizations should also be removed.
17

---

import torch
import torch. nn as nn
import torch, nn, functional as
from torch, un, modules import conv
from torch, nn, modules. utils import pair import math
class FastDeccon (conv._Convvdd
ConvNd)
def init (self, in channels, out channels, Hernel_size stride=1, padding=O, dilation=1, groups-1, bias True,
eps=1e-5, 5 momentum=0. 1, b1ock=64, sampling stride=3, freeze-False
freeze_ iter=100);
self. momentum momentum
self. iter n_iter
self.
self. counter = 0
super DDecnn,, self). init in channels, out chamees, pairreerneel size), pair (stride), pair (paddidiing) pair
dilation),
False, (o)i groups, bias, padding mode=' zeros') )
if block in channels::
block in channels else:
in_ channels block 0:
block math. ged (block, in mt_channels
if groups >
grouped conv
block channels // groups
self. block block self. features = kernel ssze we block
ao if groups = 1:
ao ao self. register buffer( running_eanns torch. zeros (self. num features))
ao self. register_buffere running_deconv's torch. eye num features))
else:
self. rregister_bufferr Sunning_meart torch. zeros (kernel_size * 2 in channels)) self. register buffer momiingdeccov', torch, (sel.. num features). repeat ( in channels // block, I, 1))
self. sampl ing stgiiirrideeeiiiieiieiiiriieeeee = sampl ing_ssrrid * stride self. counter = self. freeze_iteee jiter freeze_iter iter
self. freeze freeze
def forwardiself, x):
X, shape
B block
frozen self. freeze and (self, counter self. freeze iter)
if self. training:
self. counter
self. counter $= (self. freeze_iter iter 10)
if self. training and (not frozen):
# 1. im2col: N cols pixels -) Ntpixles cols

---

self, kornel_siiee [0] 1:
X torch. m. functional. (xf self. kernel_size, self. dilatoon, self. padding,
self. sampl innllinng_eeee stride). transpose (1, 2). contiguous
else:
channel wise
x. permite (0, 2, 3, 1). contignous O. view(-1, C) [::self. sampling stride 2,
if self. groups 1:
(C//BNVtpixels, k*k*B)
x = X. view(-1, self. num features, // B), transpose(1, 2). contiguous 0. view(-1,
self. num features)
else:
X = X. view( 1, X. shape 1))
# 2. subtract mean
X mean X. mean (0)
= X mean. unsqueeze (0)
self. f. running mean. mul_ (1 self. momentum)
suaning_mean. self. running mean. add (X mean, detach() self. momentum)
3. calculate COV, COV (-0"(-0.50 5), then deconv
self, groups 1: Cov t / shape[0] self, eps torch, eye shapel]3, drype=X, drype
device=X, device)
Id torch, eye shape[1l. dtype-X. X. dtype, device=X, device) torch. addmm (self. eps, Id, 1. shape[0], X. O,, x) deconv isqrt newton schulz autograd self.n iter)
ae aa else:
Cov transpose (1, (X shape(1)) self. eps * Id
X X. view(-1,, self. groups, self. num features). transpose 1) Id torch. eye (self. num features, dtype=X. dtype, device=X. device). expand groups, self. features,
self.
num_ meature
Cov torch. aaddhmm(self Id, 1. / X. [1], X. transpose 2), X) deconv
autograd (Cov, self. n iter)
# track stats for evaluation
self. runningg (1 - self. momentum)
self. rwwninggnneeaeeeee deconv. add_ (deconv. detach() self. momentum)
ao else:
mean self. running mean
ao deconv self. soming_decons
4. X deconv conv (deconv conv)
if self, groups
self. weightt. view self. num features, // B).
transpose 2), contiguous (), view(-1, self. num features) deconv
self. bias (w (X mean. unsqueeze(1)), view(self, weight. shape[0], -1). sum
(-ew self. num fen_feetres). transpose 2). contiguous
else:
self. weight. view(C // B, self. features) deconv

---

self. bias (w e (X mean, view(-1, self. num nwm features, 1))), view(self. bias. shape)
w. (iew weight, shape)
x P. conv2d(x, self. stride, self. padding, self, dilation, self. groups)
return
def isqrt_newtoon ______nttoo____ schulz aaannllauttorraaa. numlters, norm= norm', method=' denman beavers' dim A. [0]pe[0]
if norm norn'
normà A. norm
else:
normd A. trace
torch, eye (dim, dtype-A, dtype, device=A. device)
Y = A. (nv(n)))))
= torch, eye (dim, dtype=A, dtype, device=A, device)
if method
beavers's
for
in range (tuulleers
0, (3, 0+I 28Y)
torch. addum (1. 5, I, -0. 5, 2, Y)
Y. um (T)
T. (2)
else:
range (numlters))
1.5
28 8
torch. addum 5, Z. -0. torch. matrix power (Z, 3),
sgrt y* torch, sqrtt (normA)
issrrt / torch. sqrt (normA)
return A isqrt
def isqrt newton schulz autograd (Atch( numlters): batchSize, dim, A. shape
normA = A. wiwewwaatthhhe (hatchSize, -1). norm 1), (ieethate,se,,,, 1, 1)
Y A. (normA)
torch. eye (dim, dtype=A. dtype. device=A. device). unsqueeze (0). expand as (f)
torch. eye (dim, dtype A. dtype, Aevice device). unsqueeze (0). expand as (A)
for in range (numlters):
T = 0.5 * (3. 0 * Z, bmm (Y))
Y Y. bam (T)
Z T. bum (Z)
sgrrt Yetorch. sqrt (normd)
isqrt / torch. sqrt (normA)
return A_isgrt


Reviews:
[{"review_id": "r1lY9K6aYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an operation for removing the pixel-wise and channel-wise correlations of input features. This operation can be considered as a generalization form of the former proposed decorrelated batch normalization. The approach has a well-sounded neurological inspired motivation and a solid explanation of the relationship with deconvolution. In the experimental analysis, the authors demonstrate a good performance of the proposed method compared with batch normalization. Also, the authors provide the CPU time of the proposed method, which is very appreciated. \n\nOverall, this paper has reasonable contributions to the learning algorithm of the deep neural network, so I recommend the AC to accept this paper.\n\nHowever, I have 2 negative points on this paper.\nFirst, the computation cost for the im2col based convolution operation in a large kernel (eg. 7x7) is insanely large, and the authors only show us the results using VGG, which only has small 3x3 kernels.\nSecond, the arguments made on the sparse representations is somehow not convincing to me, it is really difficult to say the sparse representations has mad regularizations more effective with only 2 learning curves", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bklx8agbqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the correlation present in the input data, which may affect learning kernels with redundancy. Therefore, they introduce  a deconvolution mechanism on the input features to remove spatial and channel-wise correlation,  before these features are fed to network layers. They also show the deconvolution effect in the first layer resembles the center-surround effects found in biological neuron and induce sparsity in representation. They draw analogy to batch normalization, and show superiority in terms of  convergence and accuracy. \n\nOverall, the concept of the paper is pretty simple and straightforward - basically it removes the correlation present in the input data, specifically in the case of convolution.The experimental results are promising and show fast convergence over using batch normalization, slightly better accuracy.\n\nHowever, it seems that the deconvolution to avoid correlation is helping for classification tasks, but may not be applicable for other related tasks such as semantic segmentation, where simply using BN may work.\n\nWhat is G in the overall complexity? No. of groups? Please define. \n \nOne of the questions that comes to mind is considering PCA normalization for the same purpose. Essentially, the step of computing the covariance is the same and decorrelating the data by  using the orthonormal basis vectors, pretty much with a similar motivation. Also, PCA is a linear transformation, which points to a way of learning in the network training. So, my question is what will be the problem of doing PCA transformation and then performing convolutions on that transformed decorrelated data? Maybe layerwise PCA transformation may also help in performance and reduce complexity? I think this discussion should be added to the normalization and whitening section of related works.\n\nMinor note: The recommended page limit is 8 pages, the paper has now 10 pages. For me, Figure 1 is not conveying much information, and can be easily replaced with text. Also, I do not see the need of an extra section for the neurological basis for deconvolution; the brief discussion in the intro is enough.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgMW2npKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes \"network deconvolution\", a neural network primitive aimed at whitening the activations of each layer of the network. The method is a generalization of batch normalization that not only whitens per channel, but also removes correlations between channels and across spatial locations. Experiments show that the proposed methods improves training speed and predictive accuracy on a number of image classification models.\n\nThe method is novel and the proposed implementation details constitute a significant technical contribution. The experiments are not exhaustive and leave some open questions, but the first results are highly promising. The paper is clearly written, and embeds the proposed method in the literature.\n\nI'd like to see more discussion and experiments on:\n- What's the dependence on batch size? Does the method work better for larger batches? Or are small batches better for the added regularization effect, like what occurs with batch norm?\n- Are results sensitive to the epsilon in algo 1? How is this chosen in practice and how does it interact with the approximate inversion of the covariance matrix?\n- Under what conditions do you observe a speed-up in convergence? The only training curves shown in the main paper are on Fashion-MNIST which I don't think is very interesting. The appendix does offer a bit more information but I feel this deserves more attention.\n- How does this method interact with regularization methods? For batch normalization much of the benefit seems to come from the noise it adds to network activations; is the same true for your method? Does network deconvolution still improve final accuracy if the baseline uses more extensive data augmentation or other regularization?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a feature normalization method for CNNs by decorrelating channel-wise and spatial correlation simultaneously. Overall all reviewers are positive to the acceptance and I support their opinions. The idea and implementation is relatively straightforward but well-motivated and reasonable. Experiments are well-organized and intensive, providing enough evidence to convince its effectiveness in terms of final accuracy and convergence speed. Also, it’s analogy to biological center-surrounded structure is thought provoking. The novelty of the method seems somewhat incremental considering that there already exists a channel-wise decorrelation method, but I think the findings of the paper are interesting and valuable enough for ICLR community and would like to recommend acceptance.
Minor comments: I recommend authors to mention about zero-component analysis (ZCA) normalization, which has been a standard input normalization method for CIFAR datasets. I guess it is quite similar to the proposed method considering 1x1 convolution. Also, comparison with other recent normalization methods (e.g., Group Norm) would be useful. 

Decision: Accept (Spotlight)