Reparameterization through Spatial Gradient Scaling
Authors: Alexander Detkov
Mohammad Salameh
Muhammad Fetrat
Jialin Zhang
Robin Luwei
SHANGLING JUI
Di Niu
Venue: ICLR 2023 poster (2023)
Keywords: reparameterization
deep learning
convolutional neural networks
neural architectures

Abstract: Reparameterization aims to improve the generalization of deep neural networks by transforming a convolution operation into equivalent multi-branched structures during training. However, there exists a gap in understanding how reparameterization may change and benefit learning processes for neural networks. In this paper, we present a novel spatial gradient scaling method to redistribute learning focus among weights in convolutional neural networks. We prove that spatial gradient scaling achieves the same learning dynamics as a branched reparameterization yet without introducing structural changes into the network. We further propose an analytical approach that dynamically learns scalings for each convolutional layer based on the spatial characteristics of its input feature map gauged by mutual information. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that without searching for reparameterized structures, our proposed scaling method outperforms the state-of-the-art reparameterization methods at a lower computational cost.

Published as a conference paper at ICLR 2023
REPARAMETERIZATION THROUGH SPATIAL GRADIENT SCALING
Alexander Detkov',, Mohaamadd Salameh*1 Muhammad Fetrat Qharabagh Jialin Zhang', Wei Lui", Shangling Jui', Di Niu!
UUiiversity of Alberta, Huuawii Technologies, Huawei Kirin Solutions {detkov, fetratqh, dniu)eualberta. ca [mohammad. salaaaddaallaeee salameh, (zhangjialin0,, robin. luwei @
ing @huawei. com @hi wimiisiiioonn licon.ccooooo com
ABSTRACT
Reparameterizatioo aims to improve the generalization of deep neural networks by transforming convolutional layers into equivalent multi-branched structures dur- ing training. However, there exists gap in understanding how reparameterization may change and benefit the learning process of neural networks. In this paper, we present novel spatial gradient scaling method to redistribute learning focus among weights in convolutional networks. We prove that spatial gradient scaling achieves the same learning dynamics as branched reparameterization yet without introducing structural changes into the network. We further propose an analytical approach that dynamically learns scalings for each convolutional layer based on the spatial characteristics of its input feature map gauged by mutual information. Experiments CIFAR- 10, CIFAR-100, and ImageNet show that without search- ing for reparameterized structures, our proposed scaling method outperforms the state-of-the-art reparameterization strategies lower computational cost. The code is available at wttp://ghhtllooooooeeeeeeeeeeeee
INTRODUCTION
The ever-increasing performance of deep learning is largely attributed to progress made in neural architectural design, with a trend of not only building deeper networks (Krizhevsky et al., 2012 Simonyan & Zisserman, 2014) but also introducing complex blocks through multi-branched struc- tures (Szeedyy et al. 2015][2016] 2017). Recently, efforts have been devoted to Neural Architecture Search, Network Morphism, and Reparametrization, which aim to strike balance between network expressiveness, performance, and computational cost. Neural Architecture Search (NAS) (Elsken) al. 2018 Zoph & Le 2017) searches for network topologies in predefined search space, which often involves multi-branched micro-structures. Examples include the DARTS (Liu et al.) 2019 and NAS-Bench-101 (Ying al. 2019) search spaces that span a large number of cell (block) topolo- gies which are stacked together to form a neural network. In Network Morphism (Wei et al. 2016 2017]. well-trained parent network morphed into child network with the goal of adopting on downstream application with minimum re- re-training. Morphism preserves the parent network's s functions and output while yielding child networks that are deeper and wider.
Structural reparameterization (Ding et al. 2021c) attempts to branch and augment certain operations during training into an equivalent but more complex structure with extra learnable parameters. For example, Asymmetric Convolution Block (ACB) (Ding et al. 2019) augments regular 3x3 convo- lution with both horizontal x 3 and vertical 3 x I convolutions, such that training is performed on the reparameterized network which takes advantage of the changed learning dynamics. During inference, the trained reparameterized network is equivalently transformed back to its base simple structure, preserving the original low inference time, while maintaining the boosted performance of the reparameterized model. However, there exists a gap regarding the understanding of how and
"Work done during an internship at Huawei
1 Equal contribution

---

Published as a conference paper at ICLR 2023
when the different learning dynamics of reparameterized model could help its training. In addi- tion, the search for the optimal reparameterized structure over a discrete space (Huang et al. 2022) inevitably increases the computational cost in deep learning.
In this paper, we propose Spatial Gradient Scaling (SGS), an approach that changes learning dy- namics with reparameterization, yet without introducing structural changes to the neural network. We examine the question-CCar we achieve the same effect as branched reparameterization on con- volutional networks without changing the network structure? Our proposed spatial gradient scaling learns a spatially varied scaling for the gradient of convolutional weights, which we prove to have the same learning dynamics as branched reparameterization, without modifying network structure. We further show that scaling gradients by examining the spatial dependence of neighboring pixels in the input (or intermediate) feature maps can boost neural network learning performance, without searching for the reparameterized form. Our main contributions can be summarized as follows:
We investigate new problem of spatially scaling gradients in the kernels of convolutional neural networks. This method enhances the performance of existing architectures only by redistributing learning rates spatially, i... by adaptively strengthening or weakening the gradients of convolution weights according to their relative position in the kernel.
We mathematically establish a connection between the proposed SGS and parallel convo- lutional reparameterization, and show their equivalence in learning. This enables an un- derstanding of how the existing bulti-braacchd reparameterization structures help improve feature learning. This interpretation also suggests an sechitecture-inependdeee terization method by directly inducing the effect via scaled gradients, bypassing the need for complex structure augmentations, and saving on computational costs.
reparame-
We propose a lightweight method to compute gradient scalings for a given network and dataset based the spatial dependencies in the feature maps. Specifically, we make novel use of the inherent mutual information between neighboring pixels of a feature map within the receptive field of convolution kernel to dynamically determine gradient scalings for each network layer, with only a minimum overhead to the original training routine.
Extensive experiments show that the proposed data-driven spatial gradient scaling approach leads to results that compete or outperform state-of-the-an methods on several image classification models and datasets, yet with almost no extra computational cost and memory consumptionn during training that multi-branchee reparameterizztion structures require.
RELATED WORK
2.1 MULTI-BRANCI STRUCTURES
VGG (Simonyan & Zisserman 2014) is a base model for several computer vision tasks. Due to
its limitations, several new structures have been proposed with multiple branches achieve higher performance. GoogleNet (Szegedy et al. 2015) and Inception (Szegedy et al., 2015; 2016] (017) architectures deploy multi-branch structures enrich the learned feature space. ResNet (He al. 2016) uses simplified two-branch structure that adds the input of layer to its output through residual connections. The improvements in top-1 accuracy of ImageNet classification using these structures demonstrate the importance of multiple receptive fields (e.g.. 1 x 1, 1 x K, K x 1, and K x convolutions). diverse connections of layers and combination of parallel branches. These performance improvements often come at computational cost, complex model topologies are less hardware friendly, and have increased computational requirements. Outside expertly designed networks, advancements in Neural Architecture Search (NAS) allow for the automation of network design. Several search spaces and discovered high-performing networks (Ying et al.] 2019 Dong & Yang 2020; Ding et al.] 2021a) utilize multi-branch structures, which shows their ubiquity in modern convolution architectures. Due to the enormous possibilities of branched model topologies, search is often computationally expensive and requires vast computational resources.
2.2 STRUCTURAL REPARAMETRIZATION
Multi-branch structures enhance the performance of ConvNets. This comes at the cost of higher memory and computational power requirements, which is undesirable for inference-time applica-
2

---

Published as a conference paper at ICLR 2023
tions. Structural reparameterization solves this by training with complex nulti-branch model to improve the learned representations but equivalently transforming back to the original simple base model during inference for decreased computational costs. RepVGG (Ding et al. (021c) introduces a family of VGG-like inference models that are trained with ResNet-inspired reparameterization blocks. These reparameterivation blocks consist of parallel 3 x 3 and x convolutions along with an identity branch. After training, using the linearity of convolutions, parallel branches are equiva- lently transformed into single 3 x 3 convolution. The inference VGG-like model has the advantage of both the enhanced learned representations of the complex reparameterized model, and the fast and efficient inference of the simple base model. Similarly, DBB (Ding al.) (021b) structurally reparameterize models by replacing all K x K convolution with multi-branch topology composed of multi-scale and sequential 1 x 1-K x convolutions and average pooling during training. After training, DBB blocks are equivalently reparameterized back to K x K convolutions for efficient inference. Instead of structurally reparameterizing all convolutions, DyRep (Huang et al. 2022) aims to selectively reparameterize only important convolutions improve the training efficiency of the reparameterized model. Our spatial gradient scaling approach has the benefit of a branched reparameterization without the added training cost of an augmented network structure.
ACNet (Ding et al. 2019), through pruning experiments, showed that convolution weights on the central crisscross positions of the 3 x 3 kernels are more important to the model's representational capacity than corner weights. To further enhance the kernel crisscross's importance, they reparame- terize, during training. 3 convolutions with their Asymmetric Convolution Blocks (ACB). ACB comprises of parallel 3 x 3, 1 x 3, and 3 X 1 convolutions. They found that models trained with ACB reparameterization perform better than the base model. Like ACNet, we also emphasize the importance of kernel central positions. However, instead of using ACB blocks, which add significant training cost, we scale the gradients of convolution weights with spatially varying gradient scal- ing. In fact, by using spatial gradient scaling, we can emulate the presence of multi-branch topology without adding to the structure and computational cost of the training model.
2.3 FEATURE SELECTION WITH MUTUAL INFORMATION
Mutual information has extensive applications in the domain of computer vision and medical imag- ing. Mutual information is used by (Pluim et al. 2003). Viola & Wells III (1997) as a metric for comparing the alignment of a 31D model video images. Russakoff et al. 20044
mutual infor- mation to measure similarity between images. In deep learning, Cheng et al. (2018) used the mutual information between inputs, outputs, and target labels of a neural network to infer its power of dis- tinguishing between classes. In this paper, we use mutual information in a novel way to capture dependencies between neighboring elements within a feature map. We use this spatial information as dynamic scale for adjusting the importance of spatial positions in convolution kernel.
METHOD
In this section, we first introduce spatial gradient scaling for convolutional neural networks. We then establish its connection to reparameterization and mathematically show their equivalence. Fi- nally, we describe our nnutual-ifformmtion--ee approach to dynamically determine spatial gradient scaling during the training process at low computational cost.
3.1 SPATIAL GRADIENT SCALING
Gradient scaling adjusts backpropagation by strengthening or diminishing gradients of learnable parameters based on the significance of their position in the convolutional kernel. Let wr(") be the learnable weights for the 1-th convolutional layer at training iteration 4. The shape of w((*) denoted by (Coute; Ciny kn, ky.). where Cout, and Gin; are sizes of output and input channels, respectively, and (kiz k denotes the kernel size. During each iteration of training, we backpropagate the training loss Lerain calculate the gradients of the learnabl parameters aC/aW(") Following matrix calculus notations, let aL/oW denote a derivative tensor whose element at the position indexed by (m, n, O, p) is given by m.p.p.p Then, gradient descent optimization step for
3

---

Published as a conference paper at ICLR 2023
Average Spettll
Distanoe
(a) Kerne Receptive Field (b) Feature Map
(c) Spatial Dependence
(d) Spatial Gradient Scaling
Figure 1: Overview of the framework for learning spatial gradient scalings. In a), we show the kernel receptive field along with elements and their associated pixel distance to the center. We generate a discrete average spatial dependence vs. pixel distance function (c) from the input feature map. Using (c) and pixel distances in (a), we generate the spatial gradient scaling (d). Note that we simplify the process in the figure by considering pixel distance instead of displacement.
the convolutional layer I with weights W2 can be represented by
w = W/O - ((0) f
Oc
Oc
wr(0)
(1)
ow:
aw!")
where ( is learning rate and f(-) is an optimizer dependent element- wise function of the current and past gradients and weights. Our approach scales the gradient by spatial gradient scaling matrix Gf9, to yield the following gradient descent update rule:
w = - - (G())
Oc
W0O,
G
OC OW!")
Wi(0)
(2)
owle
where G is matrix of shape (Ka, kya) and @ denotes element-wise multiplication along dimen- sions Cout, and Gin.- We additionally constrain (()) to be strictly positive with a mean of to prevent large changes in the overall gradient direction and magnitude. To account for various optimizers, scale the gradient before any optimizer-dependent calculations like momentum and weight decay.
Elements of G()) are learned in three steps. First, we define each element in G by its displacement from the center element. Next, we measure the average spatial relatedness between every two pixels in the input feature map that are particularly displaced apart. We denote this as the average spatial dependence and define mathematically as a function of the feature map and displacement Sec-
tion 3.3 Finally, we assign values to elements in G( based on the average spatial dependence of their displacement. The overall process depicted in Figure [T, We assign higher learning priority to elements with higher average spatial dependence. Feature maps with high spatial correlation over large displacements give rise to more uniform spatial gradient scalings. Feature maps with low spatial correlations yield center concentrated scalings.
3.2 EQUIVALENCE TO REPARAMETERRIZATION
We now establish the relationship between the proposed spatial gradient scaling and parallel convo- lution reparameterization. Specifically, we show that backward propagation for single convolution different from that of its N-branch reparameterization, where the latter is equivalent to updating the original convolution with certain spatial gradient scaling.
Consider kaz x kyp convolutional layer with trainable weights wr(() E Roout X0ing xkx, x input
xx() and output y(()) at timestep in training. Following Ding al. (2019) can reparameterize this single convolutional layer into a general N-branch convolutional structure as depicted in Figure
2 (b) with batchnorms after the reparameterization instead of within. Each branch n contains convolution with a receptive field no larger than (Kiz, kn,) To mathematically represent variable- x&y,
sized kernels in different branches, we let Mr,, E {0, 10,1) x denote binary mask, which is matrix in the shape of the corresponding kernel's receptive field, as illustrated in Figure. (c).
Considering a forward pass, for each original convolutional layer with weights W(()) we can find an equivalent N-branch reparameterization, where each branch n contains convolutional weight
4

---

Published as a conference paper at ICLR 2023
X
X
X
X
dx
3x3
3x3
3x3
3x3
Rop
3x3
3x1
1x1
3x3
dW
0
Y
Y
dY
(a) Base
(b) Structural Rep
(c) Mathematical Rep
(d) Spatial Gradient Scaling
Figure 2: An illustration of the equivalence between SGS and structural reparamcterization. The base 3x3 convolution, (a), is structurally reparameterized into (b), 3-branched reparameterization with diverse receptive fields. (c) shows the equivalent binary masked convolutions. (d) shows the equivalent spatial gradient scaling, which is the sum over the binary masks. The gradient is element- wise multiplied by the spatial gradient scaling before passing to the optimizer.
tensor W and associated binary mask Mins such that the reparameteriization yields the same mapping the original convolution, i.e.,
Ni E(Miin 3 wi:)) x(()). (3)
y w(th x
Σ Mi, C wil)
x Σ
where denotes element-wise multiplication across each dimension Cout, and Cany The second equality in Eq. 3 decomposes the weight tensor Wi, while the third equality achieves branched structural form. The equivalence between a convolutional layer and its N-branch reparameterization with masked representation for each branch is illustrated in Figure (c).
However, in backward passes, Wi updates differently in the reparameterized network than in its original form. That is, one step of gradient descent on the reparameterization in Eq. 3 has the following form to yield the merged weights (/(+++) for the next timestep:
w((+1) w/O (00
Σ #=1
Ni
Min @ f
Oc
(! I
ac
w 1,
(4)
aw
(0)
In
1,m
Note that the learning dynamics for the reparameterized network differ from the original convo- lution in Eq. [ which explains why it may attain higher generalization although having identical expressivity. Despite their different topologies,
show in the following lemma that updates for the original convolution (Eq. [ and any of its N-branch reparameterization (Eq. differ only by a constant spatial gradient scaling Gi-
Lemma 1. Assume Ι(-) is linear function in gradient descent optimization algorithm (e.g. momentum, weight decay). For any reparameterization of a convolutional layer that can be rep- resented as summation of N convolutional branches with weights W and binary receptive field mask MI for =1.,..... I, its gradient descent update Equation 4 is equivalent to
w((++1) = w xx00 (
dc w-(
OC
W
G O
(5)
aw
ow
where G. = D Min is a spatial gradient scaling applied to the original convolution.
The proof in Appendix A follows readily from the equations gradient descent and the linearity of convolutions. Lemma has multiple implications. First, it provides an understanding of how branched reparameterization helps to change the backpropagation dynamics; it redistributes learn- important weights in convolutional kernel. Second. Lemma
ing rates spatially to focus allows us to convert the search for a reparameterizztioo structure into an equivalent numerical gradi- scaling search, which more efficient and lends itself analytical methods (as demonstrate
5

---

Published as a conference paper at ICLR 2023
in Section 3.3
Our scaling interpretation of structural reparameterization allows us more flexible search space unconstrained by the computational complexity of the underlying structure. Third, we find agreement between our formalized gradient scaling approach and observational rules of thumb in the reparameterization literature. For example, the current trend of preferring branches with a diverse range of convolutional receptive fields presented by Ding et al. (2021b) may stem from the fact that without it, the gradient scaling is uniform and loses its spatial emphasis.
3.3
MUTUAL INFORMATION BASED SPATIAL GRADIENT SCALING
Center
Neighbors
-11-1
4
Q
Fr 1..)
R) i)
Q
Rota
Multiply
MI
kP.Q...
fi
k.Q...
Transform
Q 2-3a1
Q22-2
ILP.Q axil
(a) Feature Map (Cis f, w)
(b) Spatial Dependency Matrix
(c) Spatial Gradient Scaling (k2, ky)
(d) Kemel Gradient
(kg.&y) Ryy)
[Cin Conrs Riy)
Figure 3: Overview of SGS calculation from the spatial dependencies in the input feature map. (a) a joint distribution is estimated for every pixel in the feature map and its (i,j) neighbor denoted by random variables P and Qi, respectively (b) Mutual information is calculated between P and Q for all (i, and values are placed in spatial dependence matrix (Si) (c) St is transformed to SGS (G+) by equation 8 (d) G is element-wise multiplied to the kernel gradients.
We describe approach finding spatial gradient scalings from the spatial dependencies in the input feature map. We begin by showing how mutual information can be used to quantify the av- erage spatial dependency between (displedd pixels. We then assign values to elements of our gradient scalings based on their spatial displacement to the center of the kernel and its associated average spatial dependency. T an overview of the method.
We quantify the spatial dependence of two pixels through the use of probabilistic dependence. We define random variables P1 and Q1,i,j as pixel values in the I-th layer feature map and their associ- ated (dis))ddspl neighbor values, respectively. The (i. (i.j)-diss) -ddiplacc neighbor for a pixel is the corresponding pixel (i units away. We express spatial dependence, (i( as the normalized mutual information (MI), ((::) :) of the random variables Pt and Qi,..:
Si(i,)) - [(P;: Q1.i.)),
(6)
Displacement (i, i) spans the entire (Kaz )in.) receptive field of the convolution kemnll as demon- strated in Figure 3 (a). We arrange elements of Si(i,j) into a spatial dependence matrix as illus- trated in Figure (b). Intuitively, spatial dependency within pixels results in statistical dependencies in pixel values, which can be detected via mutual information. I(-) is bounded between 0, when variables are independent, and 1, with complete mutual dependence.
We calculate the normalized MI from the Shannon entropy H(-) of the random variables:
i(Pi. Quas) ((P)) + -(QUi)))-HCC Q1,ss)
H(Pi, (1.5))
where (P,, Q1,.g) ( is the joint entropy of P and Qi We calculate the entropy by estimating the distributions of Pt and Qt.... through discrete binning. High-resolution images often contain redundant / spatially repeated neighbor pixels that may cause mutual information to overestimate the amount useful learnable spatial dependence. To account for this, on ImageNet, we remove occurrences of Pt with Qtik that are near in pixel values when we estimate their distributions. In practice, a single image is not large enough to generate accurate distributions for Pt and QL..jj so we aggregate pixel and their (i.j)-mo neighhors values over multiple batches of training data. Due to the inexpensive nature of the mutual information computation, we can afford to perform this calculation for each convolutional layer and every couple of epochs.

---

Published as a conference paper at ICLR 2023
Rep Method
Cost (GPU hrs.)
FLOPs
Params
Acc
Dataset
(M)
(M)
(%)
Origin
2.3
313
15
94.90+0.07
DBB*
9.4
728
34.7
94.97
CIFAR-10
DBB
7.0
728
34.7
95.00+0.1
DyRep*
6.9
597
26.4
05.2210.13
DyRep
7.2
674
25.5
95.0010.06
SGS (ours)
2.5
313
15
95.20+0.00
Origin
2.3
314
15
73. 73.70
DBB*
9,4
728
34.7
74.04
CIFAR-100
DBB
7.0
728
34.7
74.40...1
DyRep*
6.7
582
27.1
74.37
DyRep
6.4
560
22.8
74.10+0.2
SGS (ours)
2.5
313
15
75.50
Table I: Results for VGG-16 on CIFAR-10 10 and CIFAR-100 trained using the official implementa- tion of DyRep (Huang et al. 2022). Training is done on single NVIDIA Tesla V100 GPU. FLOPs and Parameters are averaged across DyRep runs. Results marked with are taken from the official DyRep paper, while the rest are our runs averaged over independent replicas.
To get the spatial gradient scaling, GG, we transform the spatial dependency matrix St with an element-wise transform parameterized by hyperparameter k
h: x St
Gi (k 1]S;
(8)
converts mutual information values into effective gradient scalings. Finally, we normalize the mean value of Gj. An overview of the SGS framework is given pseudo-code in Appendix A.S and details can be found in the corresponding open-source code |
In short, our scaling defines how significant weight elements for feature extraction based their spatial location within the kernel. In order to determine the scaling, we use mutual information to measure the notion of spatial dependence between pixels distance apart. We give high learning priority to the elements with large spatial dependence on the center kernel element.
EXPERIMENTS AND RESULTS
In this section, we assess the effectiveness of spatial gradient scaling in improving model general- ization ability. Following convention (Huang et al. (2022), Ding al. (2021b)), compare test accuracies for ResNet and VGG models trained under otate-of-the-art reparameterization schemes. We adopt the model training code and strategy from Huang al. (0022).
4.1 CIFAR
We train VGG-16 on CIFAR-(10,100)) for 600 epochs with batch size of 128, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 1 x 10-4 We update our spatial gradient scalings every 30 epochs using 20 random batches from the training set. We add epoch warm-up period at the start of training before generating our gradient scalings. Results are shown in Table Additional results are available in Appendix A.1
Our framework uses a single hyperparameter k, which defines a functional mapping between mutual information and spatial gradient scaling. We search for k on CIFAR100 and use the optimal for experiments on CIFARIO 10 and ImageNet. We perform grid search on CIFAR 100 and VGG-16 over A [2,3, 4, 5, 6, 7) using 20% of the training set for validation. As & acts to interpret values of mutual information into meaningful gradient scalings, we may expect to remain constant across models and datasets (we test this claim in Section 4.3)

---

Published as a conference paper at ICLR 2023
Rep
Cost
Avg. FLOPs
Avg. params
Acc
Model
method
(GPU days)
(G)
(M)
(%)
Origin
4.8
1,81
11.7
71.13=0.04
DBB*
8.1
4.13
26.3
70.99
ResNet-18
DyRep*
6.3
2,42
16.9
71.58
DyRep
9.1
2.92
22.1
71.50
SGS (ours)
5.1
1.81
11.7
71.65
Origin
5.3
3.66
21.8
74.17
DBB"
12.8
8.44
49.9
74.33
ResNet-34
DyRep*
7.7
4.72
33.1
74.68
DyRep
10.6
4.95
38.3
74.40=00.03
SGS (ours)
5.8
3.66
21.8
74.62-0.05
Origin
7.5
4.09
25.6
76.95
DBB
13.7
6.79
40.7
76.71
ResNet-50
DyRep*
8.5
5.05
31.5
77.08
DyRep
11.0
5,84
38.3
77.11
SGS (ours)
7.9
4.09
25.6
77.10=0.01
Table Results on ImageNet dataset. We use the official implementation of DyRep (Huang et al., 2022) 8 NVIDIA Tesla V100 GPUs. FLOOSs and Parameters averaged across DyRep runs. Results marked with are taken from DyRep paper; the rest are our runs averaged over 3 seeds.
Our spatial gradient scaling performs equally or better than state-of-the-art reparameterization meth- ods at fraction of their cost. On CIFAR10, our spatial gradient scaling outperforms DBB and performs as well as DyRep while only requiring a third their training time. On CIFAR 100, we obtain over 1% accuracy improvement from DBB and DyRep while taking less than half their GPU hours. We attribute the success of spatial gradient scaling to its enhanced reparameterization space and strategy. Unlike structural reparameterization methods like DBB and DyRep. by the computational complexity of our blocks, which enables us to explore a much larger space of reparameterizations. Our formalism of spatial gradient scaling as an equivalent to reparameteriza- tion also enables us to perform search in an easily implementable continuous space as opposed to discrete structural one. Our mutual information strategy adaptively reparameterizes each convolu- tion throughout the training process efficiently and effectively (as we show in Section
not limited
4.2 IMAGENET
We train the ResNet models for 120 epochs, with a batch size of 256, cosine annealing scheduler with initial Ir of 0.1, color jitter augmentation, and SGD with momentum of 0.1 and weight decay 1 x 10-4 Scalings update every epochs using two random training batches after a one-epoch model warm-up. Hyperparameter k is taken as the optimal found from the CIFARI00 grid search. Results are presented in Table
As with CIFAR, we significantly reduced training times with spatial gradient scaling method for equal or better accuracy compared to the state-of-the-art reparameterization methods. The ben- efits of reparameterization are attained without complicating the model structure with expensive reparameterization blocks. Large convolution kernels, like the 7 x 7 used in ResNet, are difficult structurally reparameterize. First, these large convolutions are expensive in terms of compute and memory. The addition of a parallel reparameterization branch only increases its computational cost further. Second, as convolution size increases, so does the number possible diverse receptive fields (for example, 7 x 7's receptive fields are: 1 1x .... 3 x 5.). DBB and DyRep only consider a tiny fraction of the possible set (7 1,11777xx 1, 1). Our spatial gradient scaling can consider all possible receptive fields, addition, our method completely avoids the computational pitfalls of structural reparameterization, reparameterization happens efficiently the gradient level.
non-standard ones. through
general binary masks. In
8

---

Published as a conference paper at ICLR 2023
SGS
Cost (GPU hrs.)
Acc
Dataset
Method
(%)
Origin
0.40
69.640.1
Grid Search
19.6
71.0=0.1
CIFAR-100
Autocorrelation
0.42
71.1
SGS (ours)
0.42
71.5
Table Results of VGG-11 on CIFAR-100 using different spatial gradient scaling search methods.
4.3 ABLATION STUDIES
Effectiveness of Mutual Information Approach. We demonstrate our mutual information ap- proach's ' efficacy in finding high-performance spatial gradient scalings. We compare to autocorrela- tion, another commonly used dependency measure, as well as a grid search for gradient scalings.
We train all methods on CIFAR 100 with VGG-11 for 200 epochs with batch size of 512, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 5 x 10-4 For autocorrelation and mutual information, update spatial gradient scalings every 10 epochs using 2 random batches from the training set and a warmup of epoch.
Similar to mutual information, cally, we calculate the correlation of a feature map with itself shifted by (i,j), where (f,j) indexes map autocorrelation values into gradient scalings. We perform grid search on over & E {1, 2, 3, 4, 5, 6) using 20% of the training set for validation. For mutual information, we use the optimal & found in Section We take several considerations for grid search over spatial gradient scaling to make the search tractable. First, we reduce the search space by considering a single 3 gradient scaling shared by all convolu- tional layers and constant for all training epochs. We additionally parameterize the scaling matrix by two variables, a, and B8, which determine the ratio of the center element scaling to the edges and cor- ners respectively (shown in Appendix 7 We search over a. B E {0.8, 1.0, 1.25, 1.7, 5.0, 10, 100).
measure spatial dependencies using autocorrelation. Specifi-
into a spatial dependency matrix (Figure 3.3) We use the same & transform
Our mutual information approach (SGS) outperforms both autocorrelation and grid search. While grid search is robust, it suffers from high computational complexity, which requires designing con- strained search space. Unlike SGS, grid search cannot effectively adapt across convolution depth and time without exponential blowup in the search space. While autocorrelation outperforms grid search, our mutual information method performs better. We attribute this to the fact that auto- correlation only measure linear relationships in the feature map, while mutual information can measure both linear and non-linear dependencies.
k- T-rannffrmmaton Search. In this section, we investigate the behavior of the k hyperparameter across models and datasets. Additionally, we corroborate our decision in Section to learn once on CIFAR- 100 and transfer ImageNet. Following the training strategy defined Section 4 train and evaluate models over a range of & values and plot the results in Appendix Figure g
We observe that & curves across models and datasets peak near h: 5. This may imply that k is a robust default value. We also find consistent performance gains over baseline for a large range of k values. This implies that the conventional uniform update of weights suboptimal, and lower testing error can be attained via spatial gradient scaling.
5 CONCLUSION
In this paper, we present Spatial Gradient Scaling (SGS), an approach that improves the generaliza- tion of neural networks by changing the learning dynamics focus on spatially important weights. We achieve this by scaling the convolutions gradients adaptively from the spatial dependencies of feature maps. We propose a mutual information-base approach to compute the gradient scaling with minimum overhead to the original training routine. We prove that our SGS is equivalent to convolu- tional reparameterizzation under certain conditions. This enables us to take advantage the benefits of reparameterization without introducing complex branching into model structures. Experiments show that our method outperforms the of structural reparameterization approaches on several image classification models and datasets a much lower computational cost.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hao Cheng, Dongze Lian, Shenghua Gao, and Yanlin Geng. Bvaluating capability of deep neural networks for image classification via information plane. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 168-182, 2018.
Mingyu Ding, Xiaochen Lian, Linjie Yang. Peng Wang. Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr- nas: Searching efficient high-resolution neural architectures with lightweight transformers. 202/ IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2981-2991,
2021a.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 191 1-1920, 2019.
Xiaohan Ding, X. Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building volution as an inception-likk unit. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10881-10890, 2021b.
Xiaohan Ding, X. Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repygg: Making vgg-style convnets great again. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13728 13737, 2021c.
Xuanyi Dong and Yi Yang. Nas-bench-200 201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, arXiv preprint arXiv: 180..00377 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tao Huang, Shan You, Bohan Zhang, Yuxuan Du, Fei Wang, Chen Qian, and Chang Xu. Dyrep: Bootstrapping training with dynamic re- ceparameterization. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 588-597, 2022.
Alex Krizhevsky. Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60:84 90, 2012.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019. URL uttptt////arxi
org/abs awaaaas11555 09055
Josien PW Pluim, JB Antoine Maintz, and Max A Viergever. Muuua--ifformation--auad reg.aaaattttobbbbddrrrrrerrtttoo of medical images: a survey. IEEE transactions on medical imaging, 22(8):986-1004 2003.
Daniel B Russakoff, Carlo Tomasi, Torsten Rohlfing, and Calvin R Maurer. Image similarity us- ing mutual information of regions. In European conference on computer vision, pp. 596-607. Springer, 2004.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resner and the impact of residual connections on learning. In Thirty-first AAAI con- ference artificial intelligence, 2017.
10

---

Published as a conference paper at ICLR 2023
Paul Viola and William M Wells III. Alignment by maximization of mutual information. Interna- tional journal of computer vision, 24(2):33-154, 1997.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In ICML, 2016.
Tao Wei, Changhu Wang, and Chang Wen Chen. Modularized morphing of neural networks. ArXiv, 2017.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101. Towards reproducible neural architecture search. In International Conference Machine Learning, pp. 7105-7114, 2019.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. 2017. URL https://arxiv. wwwp:/iwwwgg..ooooo 01578

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EXPERIMENTS ON CIFAR-100
We report in Table 4 results for the CIFAR-100 dataset wide range of CNN architectures. We train for 200 epochs with a batch size of 128, a cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 5 x 10-1 For data augmentation, we use random crop, flip, and cutout. Spatial gradient scalings are updated with mutual information every 5 epochs, using 2 random batches from the training set, and with 5. At the start of training, we add a warm-up period of I epoch before generating our gradient scalings. Adding spatial gradient scaling to optimization yields performance gains for several architectures.
Model
Baseline
SGS (Ours)
VGG-11
73.2+0.1
VGG-13
74.7
76.1
VGG-16
74.9
76.2+0.1
VGG-19
74.640.1
76.110.2
ResNet-18
77.8
78.840.1
ResNet-34
79.0
80.010.2
ResNet-50
79.1
79.640.2
ShuffleNet
72.4+0.1
72.8+0.2
MobileNetV2
73.9
74.210.2
SeResNet-118 18
78.4
79.640.1
SeResNet-34
79.
80.340.2
PreActResNet-18
75.740.1
76.340.1
PreActResNet-34
77.440.1
77.9
PyramidNet (a 48)
79.140.1
79.9+0.2
PyramidNet (a = 84)
81.140.1
81.640.1
PyramidNet (a: 270)
83.310.1
83.610.1
Xception
79.0
80.0
Table 4: Results for models trained on CIFAR-100 with and without spatial gradient scaling. Re- sults are averaged over independent replicas.
A.2 COMPARISONS TO ADAPTIVE GRADIENT OPTIMIZERS
In this section, we compare spatial gradient scaling to popular adaptive gradient optimizers. Unlike adaptive optimizers, which typically optimize based the model weights and gradients of previous timesteps, our gradient scaling uses the spatial properties within the training data to effectively scale convolution gradients. In Table [5] we present CIFAR-100 results for various optimizers with and without spatial gradient scaling. We tune optimizer hyperparameters with grid search using 20% of the training data for validation. We focus our search on learning rate and weight decay, leaving other optimizer settings as PyTorch defaults. Training and SGS settings, outside of searched optimizer hyperparameters, are identical to those described in Appendix A We find that spatial gradient scaling improves the performance of even the highest-performing optimizer, SGD Momentum. Moreover, all tested optimizers, Adagrad being the only exception, benefitted from spatial gradient scaling. Even in Adagrad's 's case, can find performance gain by applying gradient scaling post- optimizer calculations and right before the weight update step (shown as Adagrad*) as opposed to directly to the prooppro gradient and before optimizer calculations.
A.3 SENSITIVITY TO DIFFERENT TRAINING SETTINGS
In this section, we empirically study performance improvement by spatial gradient scaling across training hyperparameters. Results are shown in Table 6 Training and SGS settings, outside of those in the ablation study, identical to Appendix A. We find performance improvements the baseline under all tested hyperparameter configurations. This suggests that SGS can be used effectively to improve ConvNets training without requiring extensive hyperparameter tweaking.
12

---

Published as a conference paper at ICLR 2023
Model
Optimizer
No SGS
SGS (Ours)
VGG-11
SGD
68.3
70.7
SGD + Momentum
71.0+0.1
73.2
Adadelta
69. 69.2
70.8
Adagrad
66.21o.1
66.2
Adagrad*
66.210.1
67.0
Adam
68.8+0.2
70.4+0.2
Adamax
68.4+0.2
69.9+0.1
NAdam
69.2
70.2
RAdam
69.8
71.1
ResNer-18
SGD
75.8
76.3
SGD + Momentum
77.610.1
78.510.1
Adadelta
75.8+0.2
76.4+0.1
Adagrad
72.2+0.2
72.3+0.2
Adagrad
72.210.2
72.9
Adam
75.1
76.2
Adamax
74.9400.1
75.8
NAdam
75.640.1
76.0+0.1
RAdam
76.240.2
76.8
Table 5: Results for models trained on CIFAR- using a range of optimizers, with and without spatial gradient scaling. Results are averaged over 3 independent replicas.
Model
Epochs
Batch Size
Baseline
SGS (Ours)
VGG-11
TO0
64
70.4
71.1
128
70.8
72.5+0.2
512
68. 68.8+0.2
70.8
200
64
71.010.1
72.4
128
70.8
73.2
512
68.640.2
71.640.1
600
71.0
73.0+0.1
128
70.1
73.2+0.1
512
68.2+0.1
71.3+0.1
ResNet-13 18
100
64
77.5
78.110.1
128
77.5
78.0
512
75.240.1
75.6+0.1
200
64
77.8+0.2
78.4+0.1
128
77.8
78.8
512
76.2+0.1
76.7+0.1
600
78.0+0.1
78.7
128
77.9
79.0
512
76.4
77.210.2
Table 6: Performance gain of spatial gradient scaling on CIFAR-100 using a range of training hyperparameters with and without spatial gradient scaling, Results are averaged over 3 independent replicas.
13

---

Published as a conference paper at ICLR 2023
A.4 EXPERIMENTS ON SEMANTIC SEGMENTATION AND HUMAN POSE ESTIMATION
We consider semantic segmentation and human pose estimation evaluate our spatial gradient scal- ing. For semantic segmentation, we adapt our training code and strategy from SemSeg 2 We train and evaluate the PSPNet models on the Cityscapes dataset. PSPNet composed of a ResNet- 18/34/50 backbone and an up-sampling head for pixel-level classification. Using the backbone weights of the ResNet models pre-trained on ImageNet with and without SGS (Section 4.2 we fine-tune with and without SGS on Cityscapes. Results are shown in Table
Model PSPNet-18
Baseline (mloU//Accc/llAAcc)
SGS (clooU/AcccccllAAcc
72.8/80.0/95.3
73.7/81.1/95.4
PSPNet-34
76. 10.1.//8999..7
76.8/84.3/95.9
PSPNet-50
75.4/82.8/95.8
75.6/82.8/95.9
Table 7: Results for PSPNet models trained and fine-tuned with and without spatial gradient scaling on the semantic segmentation task.
For human pose estimation, use the training code and strategy from Pytorch-Pose- F We train and evaluate on MPII dataset with MSRA ResNet. MSRA ResNet has Resnet- 18/3//50 backbone which take pretrained from ImageNet with and without SGS. We then fine-tune with and without SGS on MPII. Results are shown in Table
Model
Baseline (PCKh@0.5)
SGS (PCKh@U.5)
MSRA ResNet-18 18
83.8
84.4
MSRA ResNet-34
85.8
86.2
MSRA ResNet-50
85.3
85.6
Table Results for MSRA ResNet models trained and fine-tuned with and without spatial gradient scaling on the human pose estimation task.
A.5 TRAINING PSEUDO CODE
Algorithm Spatial Gradient Scaling Training
Training dataset X. Set of spatial gradient scaling SGS. Number of epochs E' for training. Number of SGS warmup epochs N Number of batches N,, for SGS calculation. Set of convolution layers L.
Input:
for 0, .... (E'...EE-- 1) do if e divisible by N_ then be randomly sample N, batches from X Forward propagate b and assign the input feature map of convolution layer to the set X for in do
for (i.j) in kernel size of I do P. P.Q + List of pixels of X, and their corresponding (i.j)th neighbours
SGS <- Mutual Information between and (i. neighbour Q
for (x. y) in (X.Y) do Forward and backward propagate (xr, for l in L do
for (i,j) in kernel size of / do
Scale the ith convolution spatial (i,j) gradient element with SGS
Update weights with gradients
huttes///ttuub.ccm/hhhhh//eeee
14

---

Published as a conference paper at ICLR 2023
A.6 LEARNED SPATIAL GRADIENT SCALING
In Figure 4 we plot the spatial gradient scalings for the first, seventh, and last convolutional layers of VGG-16 at the start and the end of training on CIFAR-100. We observe more uniformly dis- tributed gradient scaling for beginning layers giving equal relative importance to all kernel weights. Deeper layers have center-focused gradient scalings with most the importance distributed on the center and edge clements as opposed the corners. Interestingly, observe that after training, spatial gradient scalings for deeper layers become even more center-focused, indicating a decrease in the spatial dependencies of the feature map.
Mocel Depth
1.0
0.8
0.6
1.0
0.8
0.9
0.5
1.0
1.7
1.0
1.0
2.5
1.0
0.8
3.2
0.9
Timesop
1.0
0.6
0.8
1.0
0.8
0.3
0.7
0.3
0.2
0.2
0.2
1.7
1.0
0.9
4.7
4.4
1.6
1.0
0.0
Figure 4: Spatial gradient scaling for the first, seventh, and last convolutional layer of VGG16 on CIFAR-100 100 at the beginning and end of training.
We plot gradient scalings for ResNet18 on ImageNet in Figure 5 Like CIFAR, we see uniform spatial gradient scalings early layers and center-focused scaling deeper layers. Contrary to CIFAR, however, we find that gradient scalings "smoothen" over time and become more uniform. Discrepancies between the behavior of spatial gradient scaling on CIFAR and ImageNet warrant future investigation.
Modill Depth
0.7
0.8
0.9
0.7
0.8
3.0
0.8
1.0
2.3
1.0
Tiresop
D.7
0.7
0.8
0.7
0.9
0.9
0.9
0.9
2.8
2.0
0.9
0.5
Figure 5: Spatial gradient sealing for the first, eight, and last convolutional layers of ResNet18 on ImageNet at the beginning and end of training. The first layer is a x 7 convolution.
A.7 WEIGHT MAGNITUDES
Similar to Ding et al. 2019), we investigate the average kernel magnitude matrix of learned weights with and without our spatial gradient scaling. For convolution weight, the average kernel magni- tude matrix is defined the mean of the absolute value of the weight tensor across the input and output channels (leaving the spatial channels intact). We further normalize the mean of the matrix for meaningful comparisons.
Figure depicts the average kernel magnitude matrix for the first, seventh, and last convolutional layer of VGG-16 trained on the CIFAR- 100 dataset with and without spatial gradient scaling. We additionally show the spatial gradient scaling of the last training epoch the SGS training scheme. Similar to Ding al.] (2019), observe that spatial gradient scaling modifies the normally
15

---

Published as a conference paper at ICLR 2023
Model Depth
1.1
0.9
0.9
1.2
Trained Weights
Mag
1.7
1.0
1.0
1.6
(Base)
1.1
0.9
0.9
1.1
1.0
0.3
0.7
0.3
0.2
Spelial Gradient
1.7
0.9
0.9
1.6
4.4
Scaling
1.0
0.3
0.7
0.3
0.2
0.2
1.0
0.3
0.3
0.2
0.3
0.2
Trained Weights Mag
1.0
2.2
0.9
0.9
1.2
5.0
(wi SGS)
1.0
0.3
0.3
0.2 0.3 0.2
0.0
Figure 6: Comparison of the average kernel magnitude matrix of the first, seventh, and last convolu- tion weight between regularly trained VGG16 CIFAR-100 and the network trained with gradient scaling. We also show the spatial gradient scaling during the last epoch of SGS training.
trained kernel magnitude matrix to focus more the center and edge elements. We also qualitatively observe the similarities between the shape of spatial gradient scaling and the final trained weights.
A.8 PROOF FOR EQUIVALENCE REPARAMETERIIATTIO
In this section we prove by induction lemma I for optimizers that are linear functions of current and past weights and gradients i.c.,
Oc Ow!
w
ac ow(") w:(o)
(1) ac aw(r) + +5(1)
where ((t)) and @C/aw(()) are the weights and respective gradients for the I-th convolutional layer at training iteration and f(-) a linear optimizer parameterized by arbitrary ((() and c(r).
We begin by defining two models: base convolutional layer, and its n-branch reparameterizatioon which at I has an identical mapping:
N Min. @ w-(0)
N Xi = Σ 3 ((Min O w/ * X1)
== = w/ X1 =
Σ
==1
Our goal to find a modified update rule for the single convolution such the mappings are identical throughout training i.c., V(t 2 0) the following holds:
i Σ Min (((9
Assume equation 9 holds Y(t < to) (we know this is for sure the case for to = 0). We wish to find an update rule for the single convolution such that equation 9 holds for t to and thus
v(t < (to + 1)). Together with our assumption identical mapping at t = 0, we can then ensure equation holds v(t 0).
16

---

Published as a conference paper at ICLR 2023
Assuming equation 9 holds v(t I to) we find the update equation for the merged weights W/
as:
Wi
w
N
Min
④
Willo) - A(60)
to
((6)
Oc
<(7)w(()))
))
aw(r)
n=1
T=
Σ
Minn
Wi(ta))
(50)
Mi,n Σ
1o
dc
( In
Ow(r)
wy(0)
((4))
Σ
N
Min 3
to
7(1r)
0C
+ ((1))(())
Equation 9
Ow(7) mm
1m
n=l
-w(l)) (40)
Σ
N
Min
(F)
Oc
( 3 w/?)
aw(r) 1n
ww(h)) (4a)
Min
0c
+ ((() Mi, w/t
aw(F)
-w(to) (100
M
0c
+ ((F)
Σ n=1
N
Min o wo/x))
awlm)
1,
-w((t) - (4a)
to
(7)
Mi,1
0C
+ (())
Equation
ow(r)
We then show that when equation Iholds for t then:
dc
Oc
aw!! 1,
aw(!)
First we demonstrate that the convolution gradient is only a function of the input tensor, X and the output gradient, OC/OY (t) and not of the weight W. We make use of the tensor index and Einstein summation notation:
(t) W(t), * ((()
Ct-1 1 Kn
Kw -
(c,, h, w)(l
Σ
Σ
W Ci Kn, (1) X (ci, h + kn, w + )w))
4=0 ky
k===0
y((1)
w(t)
x(1)
h.
2C
Σ dC
Ow(()
k,d,j 0 (1)))
k.jjj korks
0r(1)
OC
kitj
(v (t)
aw(!)
Oc
k.l.m
(t)
(w(()
0C
òk.c, :1,c, &ma Sn x(1)
aY
ac
(1)
0x:!!)
OL
x(1)
dC
= 9 dy x(())
OL
(1(1))
1.
Corted
17

---

Published as a conference paper at ICLR 2023
Given equation we can now show:
(ff) Σ
((Mn o WiHoo)
(Mr&W.tt))
x((1)
Σ ((tb)
ac
dc
0y
0Y
which implies that:
0C
0c
x(1)
( Oc
dY x(1)
ac
ow!
(t)
dy
(1)
aw!")
Finally, picking up where we left off:
w - =W(( (100)
(r) Σ
Min
Oc
ow
ww(1o) ((0)
(())
M
ac
+ +
aw-(7)
w((1)) -1(0))
10
(T))
(M
OC
+5(0))(())
aw!
w((l)) - (1t)
to
Gr
0C
+ c(0
aw!77
We arrive at lemmal] !
18

---

Published as a conference paper at ICLR 2023
K-11-11
K-1,0
K-1++1
r1/
1/a
1/5
9 4/a
K 0,-1
K 0,0
K 0,+1
1/a
1
1/a
x 1+
141,-1 K+1,u
K+1,41
1/3
1/a
1/3
4/b
Figure 7: and 3 parameterization of the spatial gradient scaling used for the scaling grid search. and & determine the ratio between the center element and the edges and corners respectively. An additional multiplication factor is added to ensure normalized mean.
1.2
CIFAR100 VGG-16
ImageNet ResNet18
ImageNet ResNet34
normalized acc
0.8
0.6
0.4
0.0
Figure 8: Results for K search for VGG-16 on CIFAR100 and ResNets on ImageNet. Accuracies are normalized such that the baseline is 0 and reported results of k: = 5 is I.
19


Reviews:
[{"review_id": "zNaERHRicF", "reviewer": "Reviewer_v3Lx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a gradient scaling approach to replicate structural reparametrization that improves training. The main idea is based on the linearity of convolutions and the observation that the effect of reparametrization can be achieved by simple gradient scaling. The results show the benefits of the approach on multiple vision datasets.", "summary_of_the_review": "Overall, the idea is nice and it is backed by theory and experiments.\n\n## Post rebuttal\nI thank the authors for the response. I would recommend adding a paragraph about the intuition behind using mutual information for gradient scaling. Additionally, the extensive experiments across various settings show the practical benefits of the method. I was already positive and I would like to see the paper accepted.", "strength_and_weaknesses": "## Strengths\n1. The idea is simple and elegant. It replaces structural reparametrizations which require high memory and computational overhead during training with a simple gradient scaling that performs similar to such methods.\n2. The paper is well-written, and the idea is backed by the proofs and experimental results.\n\n## Weaknesses\n1. It is not clear why mutual information is a good measure to obtain gradient scales. Specifically, it is not clear to me why one should weight a filter coefficient higher if it is similar to the center pixel in a convolution filter. Even though the results show the benefits it would be good to motivate it intuitively. Please comment on this.\n2. Even though the idea is simple, there were several implementation details (eg., the last para on page 6) required to make it work in practice. I think reproducibility might be a concern due to this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "omZw4u9d_L", "reviewer": "Reviewer_SeDu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to scale the components of the gradients during back-propagation to achieve the same results as branched reparametrization.", "summary_of_the_review": "The paper proposes a novel and elegant idea. I think it is worth sharing with the community.", "strength_and_weaknesses": "The paper is well-written. It has a coherent story and it is interesting to read. The related work section is properly organized. As I was not aware of some of the papers mentioned there, I found it extremely useful.\n\nI find the idea of the paper elegant. Although it is simple, it demonstrates that a simple modification of the learning process can achieve the same result (or better) than a modification of the architecture.\n\nI didn't notice any significant issues. However, I would appreciate if the authors explain why they use mutual information for gradient rescaling. Why not some other function? And if it is not limited to mutual information, would the results differ if the method relies on some other measure for spatial correlation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HJDgwIgcUN", "reviewer": "Reviewer_xBEc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to rescale gradient components based on spatial mutual information when optimizing convolutional neural networks. They show that the proposed spatial gradient scaling is equivariant to previous work on structural re-parameterization (which uses a multi-branch topology network during training and re-parameterize back to the original convolution for inference). They compare their approach to previous work using structural re-parameterization for classification tasks on the CIFAR and ImageNet datasets and demonstrate that the proposed technique improves performance at a lower computational cost.\n", "summary_of_the_review": "I am leaning toward weak rejection but I think this paper still has merits. \n\nTheir claim that the proposed spatial gradient scaling is equivalent to branched reparameterization in previous work without introducing structural changes is correct and it is also supported by their experiments. \n\nHowever, the proposed technique is too heuristic with weak theoretical justification. In experiments, they only compare to branched reparameterization in previous work  (which has already been shown to be equivalent to their proposed approach). So I am not convinced that this technique can bring robust performance gains that cannot be achieved using other optimization algorithms with adaptive gradients. ", "strength_and_weaknesses": "**Strength:**\n1. This paper is relatively well-written and most technical parts are clear to me. The experimental details are also sufficient.\n2. The connection between spatial gradient scaling and parallel convolution parameterization is a good observation. It shows that previous work on muti-branch reparameterization is unnecessary for achieving better performance.\n\n**Weaknesses:**\n1. The proposed approach is heuristic and lacks theoretical foundations. The use of mutual information for gradient rescaling is intuitive but it is hard to justify from a theoretical perspective. \n2. There are useful tricks in deep learning which do not have a theoretical explanation in the beginning. But in that case, the empirical study needs to be more convincing. It may not be enough to only apply their technique to classification tasks and compared only to previous work on structural re-parameterization. In my opinion, it would be convincing if they could show that their performance improvement can not be possibly achieved by using other optimization methods with adaptive gradient (e.g. AdaGrad), and their empirical gain is robust across tasks such as object recognition, image segmentation, and image generation, and across a wide range of base CNN architectures. Of course, I am not asking the authors to run these experiments during the rebuttal period. These are just suggestions to make a strong paper in the future. That level of empirical study is usually not necessary, but for a more heuristic technique, it might be good to demonstrate the robustness of the performance gain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "39pRB_3qW8", "reviewer": "Reviewer_ahFw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to scale the gradients in convolutions through learning pixel/feature dependencies", "summary_of_the_review": "Paper seems interesting, although there are a few improvements that would benefit the paper", "strength_and_weaknesses": "Strengths:\n- The paper seems novel.\n- Derivations seem correct\n\nWeaknesses:\n- Figures seem to be in low resolution\n- Table 1 evaluation is only with VGG (quite old), not sure why it did not include ResNet or more recent networks.\n- Results on CIFAR 100 are quite low compared to the state-of-the-art\n- I missed a comparison with other reparameterization methods, e.g. Salimans and Kingma (Neurips, 2016) and others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster