Channel-Directed Gradients for Optimization of Convolutional Neural Networks
Authors: Dong Lao
Peihao Zhu
Peter Wonka
Ganesh Sundaramoorthi
Venue:  (2021)
Keywords: stochastic optimization
Riemannian geometry
Riemannian gradient flows
convolutional neural nets

Abstract: We introduce optimization methods for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error. The method requires only simple processing of existing stochastic gradients, can be used in conjunction with any optimizer, and has only a linear overhead (in the number of parameters) compared to computation of the stochastic gradient. The method works by computing the gradient of the loss function with respect to output-channel directed re-weighted L2 or Sobolev metrics, which has the effect of smoothing components of the gradient across a certain direction of the parameter tensor. We show that defining the gradients along the output channel direction leads to a performance boost, while other directions can be detrimental. We present the continuum theory of such gradients, its discretization, and application to deep networks. Experiments on benchmark datasets, several networks, and baseline optimizers show that optimizers can be improved in generalization error by simply computing the stochastic gradient with respect to output-channel directed metrics.

Under review as a conference paper at ICLR 2021
CHANNEL-DIRECTER GRADIENTS FOR OPTIMIZATION OF CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce optimization methods for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error. The method requires only simple processing of existing stochastic gradients, can be used in conjunction with any optimizer, and has only a linear overhead (in the number of parameters) compared to computation of the stochastic gradient. The method works by computing the gradient of the loss function with respect output-channe directed re- weightttdd Hº or Sobolev metrics, which has the effect of smoothing components of the gradient across a certain direction of the parameter tensor. We show that defining the gradients along the output channel direction leads to a performance boost, while other directions can detrimental. We present the continuum theory of such gradients, its discretization, and application to deep networks. Experiments on benchmark datasets, several networks, and baseline optimizers show that optimizers be improved in generalization error by simply computing the stochastic gradient with respect to output-channel directed metrics.
INTRODUCTION
Stochastic gradient descent (SGD) is currently the dominant algorithm for optimizing large-scale convolutional neural networks (CNNs) (LeCun et al. (1998); Simonyan & Zisserman (2014); He al. (2016b)). Although there has been large activity optimization methods seeking improve performance, SGD still dominates in terms of its generalization ability. Despite SGD 's dominance, there still often a gap between training and real-world test accuracy performance, which motivates research in improved optimization methods.
In this paper, we derive new optimization methods that are simple modifications of SGD. The methods implicitly induce correlation in the output direction of parameter tensors in CNNs. This is based on the empirical observation that parameter tensors trained networks typically exhibit correlation over output channel dimension (see Figure D]. We thus explore encoding correlation by constructing smooth gradients in the output direction, which we show improves generalization accuracy. This is done by introducing new Riemmanian metrics the parameter tensors, which changes the underlying geometry of the space of tensors, and reformulating the gradient with respect to those metrics.
Our contributions re- re-weighted version of the standard I.2 metric and another that is a Sobolev metric) over the space of parameter tensors. This encodes channel-directed correlation in the gradient optimization without changing the loss. Second, we compute Riemannian gradients with respect to the metrics showing linear complexity (in the number of parameters) over standard gradient computation, and thus derive new optimization methods for CNN training. Finally, we apply the methodology to training CNNs and show the empirical advantage in generalization accuracy. especially with small batch sizes, over standard optimizers (SGD, Adam) on numerous applications (image classification, semantic segmentation, generative adversarial networks) with simple modification of existing optimizers.
follows. First, formulate output channel-
Riemannian metrics (a
1.1 RELATED WORK
We discuss related work in deep network optimization; for detailed survey, see Bottou et al. (2018). SGD, c.g., Bottou (201)), samples a batch of data to tractably estimate the gradient of the loss function. As the stochastic gradient is noisy version of the gradient, learning rates must follow

---

Under review as a conference paper at ICLR 2021
AlexNet
VGG-16
DenseNet
ResNet-50
Output Channels
Input Channels
Architectures Input Channel Correlation 0.0057
AlexNet
VGG-16
DenseNet
ResNet-50
0.0082
0.0029
0.0047
Output Channel Correlation
0.0267
0.0178
0.0116
0.0077
Visualization of parameter tensor of convolutional layers trained on ImageNet. Frequently within in layers (especially deeper layers), there is correlation of the weights along the output channel direction. The table shows output correlation (invariant to re-scaling) relative to input direction. Our method induces parameter correlations in the output direction.
Figure I:
decay schedule in order to converge. Many methods have been formulated to choose learning
rate over epochs and components of the gradient, including adaptive learning rates (e.g.. Duchi
itt al. (2011); Zeiler (2012); Kingma & Ba 2014)); Bengio (2015); Loshchilov & Hutter (01 7
Luo et al. (2019)). ). For instance, Adam Kingma & Ba (2014) adaptively adjusts the learning rate
so that parameters that have changed infrequently based on historical gradients are updated more quickly than parameters that have changed frequently. Another way to interpret such methods is that they change the underlying metric on the space on which the loss function is defined to an iso-tropically scaled version of the TL. metric given by simple diagonal matrix; we change the metrics an-isotropically. We show that our method can be used in conjunction with such methods by simply using the stochastic gradient computed with our metrics to boost performance.
As the stochastic gradient is computed based on sampling, different runs of the algorithm can result in different local optima. To reduce the variance, several methods have been been formulated, e.g.. Defazio et al. (2014);] Johnson & Zhang (2013). We are not motivated by variance reduction, rather, indueing correlation the parameter tensor to improve generalization. However, as our method smooths the gradient, our experiments show reduced variance with our metrics compared to SGD,
Another method motivated by variance reduction Osher et al. (018)) (see applications Wang et al. (2019): [iang et al.) (2020); |Wang et al.] (2020)), where the stochastic gradient is pre-multiplied with an inverse Laplacian smoothing matrix. For CNNs, the gradient with respect to parameters rasterized in row or column order of network filters before smoothing. Our work is inspired by Osher ct al. (2018), though we are motivated by correlation the parameter tensor. Osher et al. (2018 be interpreted
using the gradient of the loss with respect Sobolev metric. One insight over Osher et al. (2018) is that keeping the structure of the parameter tensor and defining the Sobolev metric with respect to the output-channel direction boosts accuracy, while other directions do not. Secondly. we introduce a re- seeeeitttdd T metric that preferentially treats the output-channel direction, and can be implemented with a line of Pytorch code, has linear (in parameter size) complexity, and performs comparably (in many cases) to our channel-directed Sobolev metric, boosting accuracy of SGD. Third, our Sobolev gradient, a variant of the ordinary one, has linear complexity rather than quasi-lined linear (not requiring FFT as Osher et al. (2018)) Sobolev gradients have been used in computer vision Sundaramoorthi et al. (2007); Charpiit al. (2007) for their coarse- coarse-to-fine evolution Sunddraaoorthi et al. (2008); we adapt that formulation to CNNs.
We formulate Sobolev gradients by considering the space of parameter tensors as a Riemannian manifold, and choosing the Sobolev metric on the tangent space. By choosing metric, gradients intrinsic to the manifold can be computed and gradient flows are guaranteed to decrease loss. Other Riemannian metrics have been used for optimization in neural networks, e.g., Amari (1998) ; Marceau- Caron & Ollivier 20 Hoffman et al. (2013): Gunasekar et al. (220)) and tangentially relate to our work. These works are based on Amari s Amari 19998) information geometry probability measures, and the metric considered is the Fisher information metric. The motivation for these methods is re- paraametrizzaionn invariance of optimization, whereas motivation is imposing correlation in the
2

---

Under review as a conference paper at ICLR 2021
parameter space. Other works Gunasekar et al. (2020) use the Hessian metric (in the convex case), but these metrics are data-deppnndnnt and the gradient is challenging to compute, requiring (a large) inverse matrix computation.
2 CHANNEL-DIRECTED GRADIENTS
We now present the theory to define channel-directed gradients. To do this, we formulate new metrics on the space of tensors, and then derive analytic formulas for channel-directed gradients in terms of the standard L gradient. As we show, channcl-directed gradients effectively smooth the components of the L gradient across the output direction of the parameter tensors of the CNN, which induces correlation in that direction in the gradient and thus also the parameter tensor. Another interpretation we are changing the geometry of the loss landscape (without changing the loss) to a more smooth one by changing the metric of the space which the loss is defined.
2.1 BACKGROUND ON RIEMANNIAN GRADIENTS
We present the definition of gradient on a Riemannian manifold, and show the dependence of the gradient on the chosen metric the manifold (see Carmo ( (1992): Abbaham al. (2012) for more details). A manifold X' is space that is locally linear around each point X E X'; this linear space is the tangent space, denoted xx X' A Riemannian manifold has a smoothly varying positive definite bilinear form ( (called the metric) on the tangent space. This metric allows one to define the notion of lengths of curves on the space, addition to other operations, including gradients.
Definition (Gradient of a Function) Let X be Riemannian manifold, and f. X + 4 be function. The directional derivative of f at E X along a direction k E Txx is defined as (f())k== de f(X + =k)l__o- The gradient of f at X X is the vector, Vf(X) TxX, that satisfies the relation
df(x)
(Vf(X), for all k E TxX.
(1)
Note that "the" gradient will depend on the choice of the metric on the manifold. We note that any such gradient will decrease the the function by moving infinitesimally in the tangent space in the direction of negative the gradient as df(X). k: = - [VV(( < 0 when = - fXX), where -|- is the norm induced from the metric. The gradient flow, defined by the differential equation local minimum. In our application of this theory to CNN
X, = f(Xe). will converge optimization, f will be the loss function, and A' will be the space of parameter tensors.
consequence of this definition that the gradient the direction (up to a scale factor) the tangent space that optimizes the following problem:
df(X) kk ||:||
arg max X'VE)
Thus, the gradient can be regarded the most efficient direction it maximizes the ratio of the change in energy by perturbing in a direction l: over the cost (defined by the metric) of k. Thus, by constructing the metric to have small costs for perturbations (directions) that we prefer for gradients, the gradient flow will move in these preferential directions while minimizng the function, and thus land in favoralle local minima.
2.2 DIRECTED METRICS
In existing deep network gradient-based optimization schemes, the underlying metric on the loss function is assumed to the standard Euclidean T.2 metric. We will consider re-weighted version of the L metric and a Sobolev metric that favor correlation in the output channel direction of the gradient and thus the parameter tensors. To formulate the methodology, we start from a continuum formulation, where treat weight tensors in the continuum, formulate the metrics in the continuum and then in the next sub-section derive the gradients with respect to these metrics. Finally, we discretize gradient flows in the implementation to derive iterative schemes.
Let X O x I H. x W + R denote a parameter tensor a layer of convolutional neural network. Here O [0, O] denotes indices to the output channel dimension of the tensor, I = [0,
3

---

Under review as a conference paper at ICLR 2021
H2 weighsed
Figure 2: Visualization of kernels applied to the H gradient under different metrics for À 1. This illustrates the smoothing effect of the metrics. In com- putation, linear cost formulas are applied to compute the gradients not using the convolution interpretation.
H1 H
2
denote the indices the input channel, and H [0, W [0, W] denote indices to the height and width dimension of the spatial filters of the tensor. The metric is defined on the tangent space to the space of such X. An element of the tangent space will have the same form of the tensor, i.e., k 1Xxxxxxxxx W IR. The L (called H from now on) metric is defined as
(k1, A:2) 40
(a, i, h w) k i, h, w) do di då dw,
J.I.H..W
where k1 l2 are in the tangent space of tensors. We now define a re-weighted version of H that favors tangent vectors that have global smoothness in the direction of the O dimension:
[
k2) HQ
hn h, w) (2 h, w) di dw + 0
(k1 k1, 12
ka),,.
(4)
I.H.W
where A > is a hyper-parameter and is is the average value in the output channel direction, i.e.,
kii, h w) 1
o % kío, i, h, do. do.
The metric in splits the tangent vector into global translations in the output channel direction and its orthogonal complement, i.e., the deformation. The weight À is used to control the weighting between the translation and deformation components, i.e., larger values of À means that deformations more heavily influence the norm of the perturbation. As shown in the next sub-section that means gradients with respect to this metric have higher weighted channel-directed translations than deformations.
Next, we introduce channel-directed version of Sobolev metric, defined as follows:
J.H.. En h, w) (2 h, w) di dh du + XO < 0kk1
0k2
(k1, k2)
(6)
Oo 0o
20
where 2 indicates the partial derivative with respect to the the output channel direction. The partial derivative in the o-direction implies that tensor perturbations that are smooth along the o-direction are close with respect to these metrics, which will imply that the corresponding gradients will exhibit smoothness in this direction, i.e., convolution filters that are nearby the output direction will exhibit correlation. The metric is weighted combination of the 17 metric of the derivative in the output direction, and the H metric of the output-directed translation. Note that the traditional Sobolev metric uses the H metric of the perturbation rather than the translation. Our choice is motivated by computational efficiency of the corresponding gradient, to be discussed below. The scale factors of O in the expressions above output channels. The part of the metric with the partial derivative component implies that tensors that differ in the output channel direction by non-smooth perturbation are far away in distance. Tensors that differ by just a channel-directed translation are close.
To
that the metric is scale invariant with respect different sizes of
2.3 COMPUTING BHANNEL-DIRECTIR GRADIENTS
We now compute gradients with respect the metrics defined in the previous sub-section in terms of the H gradient so that existing SGD code can be re-use with minimal changes. To compute the relation between the channel-directed gradients and the usual H" gradient, we note (T) that the directional derivative can be written as an inner product with the gradient with respect to any metrie:
dL(x) (V HO L(X),
= LV), = L(x) Ho
HQ
With this relation, we may compute the channel-directed gradients terms the H gradient (details are in Appendix Dj. Letting f = V Ho L(X), we have
x 0 V L(X) f+ ( - and f= V=VL((X)
V LmL(X,,
(8)
do
4

---

Under review as a conference paper at ICLR 2021
where the last expression is second order ordinary differential equations (ODE), whose solution discuss next. Notice that the re-weighted H gradient (8) re-weights the channel-directed translation component and the deformation component of the H gradient differently, i.c., À gets larger, the channel-directed translation becomes more prominent.
Our Sobolev gradient effectively computes local averages, and by doing so effectively imposes an ordering of the kernels in CNNs so that nearby kernels (according to the distance the output direction) are similar. As ordering of kernels in CNNs is arbitrary, in the sense that permutations of kemnlss in the output direction along with the input channels result in the same output, we are free impose one ordering, which Sobolev effectively does during the optimization so that filters that are close in the o-dimension are similar.
show, in the output channel direction,
In obtaining the expression for the Sobolev gradient below, we have assumed periodic boundary conditions in the O dimension, which further imposes the ordering of filters such that starting and ending filters in the o-dimension are similar. The periodic assumption gives that the Sobolev gradient can be computed with circular convolution with the H gradient, which is simpler to compute in practice. In fact, the Sobolev gradient is given as V ir L(X)(o, i, h, w)
6//K(1-- x((- a)/0) h, w) do, where K(o) = 1+ 0 + 1/6
for o [0, 1].
(9)
2A
Note that the re-weighted H solution also has an interpretation of convolution with respect to a smoothing kernel. Figure shows plots of the kernels for the parameter À chosen in experiments. For each o, the Sobolev or re-weighted H a local average whose weights die far away from o. Thus, the effect of the metrics is to induce smoothness of the gradient along the output channel direction.
The Sobolev gradient need not use the convolution formula, as one can just integrate the ODE twice, an advantage of mean variant of the Sobolev metric. This saves one from having compute the convolution directly, and hence a reduction in computational cost from quadratic (or quasi-linear with an FFT) to linear in O given the H gradient. The Sobolev gradient can be computed as
1 X S (o (0-0))((l00, (l0,,,,,, w) f(i, h, w)) dō 8y (0, i, h,,)
g(o, i, (,,h,w)) w) g(0, i, h, w) + do
(10)
0g (0, (,i,,,,)) w) do
1 = I' o(f(oO, w) fli, h, w))
(11)
X
g(,,i,,,w L: k(o), w)) ,,) do, o 10
(12)
where g V L(X) and f = L(X). These are just three integrals that can be computed in linear complexity with respect to O. The gradient flows under these metrics are given by
X, VLL(X),
(13)
where / denotes the artificial time variable. X is the time derivative of the parameter tensor, and V denotes the gradient with respect to the desired metric.
2.4 PROPERTIES OF CHANNEL DIRECTED GRADIENT FLOWS
By the convolution formula, the Sobolev gradient is smoothing of the //"h gradient. Noting that the gradient flow (13) integrates (smooth) gradients over time, the final tensor will exhibit correlation in the output direction as it sums smooth (correlated) gradients in the output direction and the initialization, which is typically chosen be decorrelated noise.
Correlation in the Weight Tensor:
Coarse-to-Fine Evolution and Removal of Some Local Minima: Sobolev gradient flows evolve according to coarse-scaa scale perturbations before moving to finer scale perturbations (Sundaramoorthi et al. (2008). This avoids being trapped in local minima due to fine-scale structures. Also, since Sobolev balls can fit any 2 ball but not vice-versa, the loss landscape changes (i.e., topologically in the continuum) and some local minima (in L²) may cease to exist numerically. As wide local minima generalize well Chaudhari et al. (2019), the numerical removal of local minima due to fine structures (c.g., sharp minima) may encourage convergence to wide minima and hence generalize better than SGD. The correlated nature the Sobolev (and re- re-weighted HD) gradient makes difficult lock into sharp local minima.
5

---

Under review as a conference paper at ICLR 2021
def reweighted HO go-grad grad=param grad data .lamdda):
#grad: L2 gradient; lambda>0 weights translation of L2 grad grad += lambda combdattorch mean (aan(gre (grad ,0, True), repeat repeat (grad size return grad
Figure 3: Pytorch code to compute the re-weighted Hº (HS) gradient from the 11th gradient.
APPLICATION TO SGD AND IMPLEMENTATION
To apply re-weighted Hº and Sobolev thannel-directer gradients to optimizing CNNs based on SGD or its variants, we discretize the gradient flow (3) according to forward Euler: We approximate the standard H gradient of the loss, H L(X). using mini-batch, as is standard. We then use this approximation of the Hd gradient to approximate the H gradient, V Vg. L()). by discretizing (10)- 12) using a standard Riemann sum. Note that (TO) be computed for each o, the output channel index the tensor, with the cumulative sum (CUMSUM) operation, which is linear in cost, as 11I and (2). We compute the Sobolev gradient for each convolutional layer parameter tensor independent of others. We use À
for H7 gradient and add it to scaled version (by hyper parameter) of the H gradient (as in Figure to avoid over-smoothing. The re-weighted H7 gradient computed by using (8) from the /1" stochastic gradient. Both our gradients require few additional lines of code; the code for re-weighted He is shown in Figure (see Appendix Figure 12 for f/1 code). Thus, our gradients replace the usual one, and other additions to SGD (e.g.. momentum, Adam) can be used.
EXPERIMENTS
We test methods different baseline optimizers and tasks. Our intent is to show that any method can be improved just by switching to either of our gradients. We fix unless specified otherwise. Table shows the settings for each experiment. Experiments are GPU except for GANs, which are run on Tesla v100 GPU due to memory requirements. Table Experimental settings.
single NVIDIA Titan Xp
Task
Dataset
Baseline
Network
Batch Size
Epochs
Initial I.R
ResNet-56
128,32,8
240
0.1
SGD
VGG-16
128,8,6
240
0.01
CIFAR-10
Image Classification
ADAM
ResNett-56
128,32,8
200
fe-3
LS
ResNet-56
128,32,8
240
0.1
MNIST
SGD
Two-layer Conv
0.01
Semantic Segmentation
PascalVOC
SGD
ResNet50
2
70
7e-3
Image Generation (GAN)
CityScapes
SGD
SPADE
2
100
lc44,c-4
Image Classification: We experiment on CIFAR 10 Krizhevsky et al. (2009). We test our gradients with both SGD and ADAM on ResNet-56 He et al. (2016a) and VGG-16 Simoonnnn & Zisserman (2014) following settings Osher al. (2018). For SGD, set the initial learning rate be 0. and 0.01 on ResNet-56 and VGG-16 respectively with momentum 0.9 and weight decay 5c-4. For ADAM. set the initial learning rate 0.01. We decrease the learning rate by factor 10 every 40 epochs as Osher et al. (2018). We run 25 independent trials on SGD and 10 on ADAM (due to lower variance ADAM), and report the average.
In Figure 4 we show an example of training and test accuracy curves (batch size of 8) for baselines as well as Laplacian Smoothing (LS) Osher et al. (2018). which rasterizes before smoothing. We out-perform all methods. We also apply LS (without rasterization) to smooth the gradient in our output-channel directed fashion, which improves I.S, but we still out-perform it.
In Figure (left), we compare the histograms of test accuracy over multiple runs of ours and SGD. Our method achieves higher average test accuracy with reduced variance. To investigate the effect of different channel directions of smoothing, channel-direction We compare approaches under two settings, which are smoothing gradients in all layers and smoothing gradients in only convolutional layers. Figure 5 (right) shows that output-chanma direction is preferred regardless of smoothing method used. This shows that the output channel smoothing essential. Smoothing only convolutional layers rasterized order (as
apply our method well as LS along different

---

Under review as a conference paper at ICLR 2021
Accunacy CIFAB--00
ADAM
KDAN
17.5
15.0
Ezoch
Figure Evolution of training and test accuracy CIFAR-10: an example with batchsize 8. Our metric improves both training and test accuracy.
Smoothing
Smoothing Lapers
BatchSize 8 SGD
92.4
90.0
85.0
92.0
0
21.8
86.0
SGD
91.5
9144
81.
82
88
83
90
$1.0
Test Test-Acturcyy
1S-ChenOrd
Figure 5: Distribution of results CIFAR-10. Left: Histogram of test accuracy. Ours achieves higher average with significantly reduced variance. Right: Results from different methods. Best accuracy obtained from our proposed direction. Ours: SGD+H1; AS-ChanDir: LS applied in our proposed channel direction; Ours+0: output channel smoothing: Ours+R: parameters rasterized into a 1-D vector perform smoothing; Ours+I: Input channel smoothing.
in LS) performs worse than SGD, further showing importance of the output direction. Note LS makes up the loss in convolutional layer smoothing with smoothing in other layers.
Table summarizes results (over 10-25 trials). Both our gradients improve over 17 A greater advantage is achieved with small batch sizes is small as the stochastic gradient is noisy, and our method imposes regularity. Both gradients perform similarly, but f performs better with ADAM.
Table 2: Test accuracy CIFAR-10. Channel-directed gradients improve H in all cases. Up to 11% of errors can be reduced. Results average 25 trials for SGD and 10 trials for ADAM.
Architecture Batch size
ResNet-56
VGG-16
Architecture
ResNet-56
128
32
8
128
8
Batch size
128
32
8
SGD H1
93.24
91.96
86.54
93.02
92.31
91.88
ADAM
91.20
91.04
89.53
93.39
92.27
87.99
93.26
92.77
92.25
HI1
91.42
91.13
90.02
Eroor reduceds
2.2%
3.9%
10.8%
3.4%
6.0%
4.6%
Brror reduced%
2.5%
1.0%
4.7%
93.57
92.25
88.04
93.19
92.79
92.43
91.20
91.06
89.70
Error reduced%
4.9%
3.6%
11.1%
2.4%
6.2%
6.8%
Error reduced%
0.0%
0.2%
1.6%
Effect of Smoothing Parameter: We examine the effect of the smoothing parameter MNIST Le. Cun & Cortes (2010) and Bushion-MNISTE Xiao al. (2017) by varying it from 20. We conduct training on the test set (10000 samples) and test on the training set (60000 samples) to make gen- eralization more challenging. We use a 2-layer CNN with 50 and 100 5 x 5 filters in each layer, respectively, and train with batch size 100. Figure 6 shows the accuracy at the 100th epoch (average over trials). Note = 0 is SGD. Our methods are not sensitive to and improve SGD for any X.
Semantic Segmentation:
The experiments conducted on VOC Everingham et al. 2015) using the popular UNet segmentation network RRonebberger et al. (2015) with ResNet-50 as the encoder (htpps://ithub.comm// http:///wmm//pytoooosseeeeeeeeee
We use initial learning rate 7e-3 and batch size 2 (to fit on Titan Xp memory), and average results over trials. Figure shows results. Both gradients improve segmentation accuracy by ~8% over SGD on the generalization gap from 0.163 to 0.151 (by 7.4%) and 150 (by 8.0%) for 171 and 17. respectively.
We reduced the
7

---

Under review as a conference paper at ICLR 2021
MNIST
Fashion-MNIST
0.9715
0.87
0.971
0.888
Accuracy
0.9705
0.866
0.97
0.884
0.9895
weighted 1
Re weighted L
Sobolev
Sobolev
0.88
10
15
20
Smoothness: A
Smoothness: A
Figure Results on MNIST and Fashion-MNIST with different choice of smoothness. Our methods improve classification accuracy over SGD (i.e., X = 0) for wide range of smoothness.
0.60
0.425
A
0.55
A
Train
0.400
0.50
L059
Test
mlu
0.500
0.325
0.10
Test
0.75
0.35
025
10
20
20
40
50
60
Epoch
Epoch
Figure 7: Semantic Segmentation Results on PaScalVOC. Sobolev ff and re-weighted H7 (H))
improve segmentation accuracy by 8.5% and 7.8% respectively relative to SGD.
We test on semantic labels to image conversion. We perform the experi- ments the current state-of-the-an art model SPADE Park al. (2019). We test CityScapes Cordts et al. (2016) dataset and results are evaluated by Heusel et al. (017) score (lower better). Learning rates are 1e 4 and 4e 4 for the generator and discriminator, respectively. We compare to SGD with momentum 0.9 and weight decay 5e-4. All models are trained with batch size (to fit on Tesla v100 memory). For each optimizer, summarize the results of 24 different trained models. Table 3 provides results. Our methods achieve better average FID score with less variance.
GAN Image Generation:
Method
FID
Table
Results on the image generation task. Our methods achieve better result with reduced variance due to regularity imposed during training.
SGD
65.77 + 11.94
+
60.17 + 6.15
177
57.99 5.01
Speed: With PyTorch, re-weighted Hº adds negligible overhead. Currently, H71 increases training time CIFAR-10 10 by 50% with batch size 128. 70% of this overhead is due to using tensor transpose and saving/loading, which required due limitations of Pytorch library. This can be eliminated by implementing our own Pytorch function in C++; in this case, // would add 15% overhead.
5 CONCLUSION
Using stochastic gradients that promote correlation (and smoothness) in the output-channel dimension of CNN network tensors is effective in improving accuracy of SGD and its variants. We reformulated the gradient (without changing the loss) by changing the underlying Riemannian geometry on the tensor space using two different metrics. In the continuum, Sobolev changes the topology of the loss landscape (possibly removing fine-scale local minima), and so has better theoretical properties. Both the channel-directed re- weighteed H and H gave accuracy boosts, with H performing better with ADAM. Regularity in other tensor dimensions is not effective in improving accuracy. Both gradients have the same (linear) computational complexity and not much cost over SGD (re-weighted 17 is faster), and the code is simple.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ralph Abraham, Jerrold E Marsden, and Tudor Ratiu. Manifolds, tensor analysis, and applications, volume 75, Springer Science & Business Media, 2012.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276, 1998.
Yoshua Bengio. Rmsprop and equilibrated adaptive learning rates for nonconvex optimization. corr abs/1552..43390 2015.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp- 4p.4211443 -36.. Springer, 2012.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-31) 2018.
Manfredo Perdigao do Carmo. Riemannian geometry. Birkhäuser, 1992.
Guillaume Charpiat, Pierre Maurel, J-P Pons, Renaud Keriven, and Olivier Faugeras. Generalized gradients: Priors on minimization flows. International journal of computer vision, 73(3):32--344, 2007.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Sagar A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems, pp. 1646-1654, 2014.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Ju))2121-2119, 2011.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-13) 136, January 2015.
Suriya Gunasekar, Blake Woodworth, and Nathan Srebro. Mirrorless mirror descent: A more natural discretization of riemannian gradient flow. arXiv preprint arXiv:2004.01025 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016a. doi: 10.1 10900/2222006.900... URL http: doi org 1119//109..... 2016
90
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770-778, 2016b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2017.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):13331144, 2013.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pp. 315-323, 2013.

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann
lecun. haamm/exbbmmmmite
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (1)::22222222222 1998.
Zhicong Liang, Bao Wang, Quanquan Gu, Stanley Osher, and Yuan Yao. Exploring private federated learning with laplacian smoothing. arXiv preprint arXiv:2005.002// 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017.
Liangchen Luo, Yuanhao Xiong, and Yan Liu. Adaptive gradient methods with dynamic bound of learning rate. In International Conference on Learning Representations, 2019. URL https
net popeenreveee.nett/ooumm2222
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv: 2017.
Gaétan Marceau-Caron and Yann Ollivier. Practical riemannian neural networks. arXiv preprint arXiv:1602.08007, 2016.
Stanley Osher, Bao Wang, Penghang Yin, Xiyang Luo, Farzin Barekat, Minh Pham, and Alex Lin. Laplacian smoothing gradient descent. arXiv preprint 1rXiv:1806.000 2018.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- normalization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 231-241. Springer, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Ganesh Sundaramoorthi, Anthony Yezzi, and Andrea C Mennucci. Sobolev active contours. Interna- tional Journal of Computer Vision, 73(3):345-366, 2007.
Ganesh Sundaramoorthi, Anthony Yezzi, and Andrea Mennucci. Coarse-to-fin segmentation and tracking using sobolev active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(5):851-864 2008
Bao Wang, Difan Zou, Quanquan Gu, and Stanley Osher. Laplacian smoothing stochastic gradient markov chain monte carlo. arXiv preprint arXiv:1911.00782. 2019.
Bao Wang, Quanquan Gu, March Boedihardjo, Lingxiao Wang, Farzin Barekat, and Stanley J Osher. Dp-lssgd: A stochastic optimization method to lift the utility in prrvacy-resservigg erm. In Mathematical and Scientific Machine Learning, pp. 328-351. PMLR, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- mmist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint 12Xx:12112.700 2012.
10

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL ANALYSIS OF EVOLUTION OF CHANNEL-DIRECTED OPTIMIZATION
Figure and Figure present the evolution of training and test accuracy of ADAM and SGD with different batch sizes. Using channel-directed gradients (FF2 in this experiment) for SGD or ADAM improves test accuracy for any batch size. More prominent performance gains are for smaller batch sizes. This is due to that the stochastic gradient is typically more noisy when the batch size is small, and our proposed channel-directed metries implicitly encode smoothness.
110
Than Accunnty Tealed CIP4032
Teal Accenny Ruuldd E3PO1-30
ROAM
Figure 8: Training and test accuracy on CIFAR-10 with ADAM.
240
12R112
5GD
10
05.3
02:2
Figure 9: Training and test accuracy on CIFAR-10 with SGD.
B ADDITIONAL EXPERIMENTAL VERIFICATION OF OUTPUT- CHUUT- DIRECTION
To investigate the effect of different channel directions of smoothing, we apply our method as well as LS along different channel-directioe Figure [0 [10shows that our output-channel direction is preferred regardless of different smoothing approaches.
93.7
BatchSize
93.3
93.1
92.9
92.7
Ours A Ours All Ours
LS A LS B
LS
Figure 10: Channel-Directed Smoothing Leads to Better Performance. Best accuracy obtained from our proposed direction. A: Channel Directed; B: Input-hannel Directed; All: parame- ters rasterized into I-D vector to perform smoothing; Ours: re-weighted
11

---

Under review as a conference paper at ICLR 2021
C REGULARITY OF TRAINED CONVOLUTIONAL LAYERS
We show that the final weight tensor at convergence in our methods have correlation in the output channel dimension in Figure II] as should be the case the tensor is composed of component that smooth. To show this, we plot the correlation between filters in the weight tensors as a function of the distance in the output channel dimension. This is done over multiple tensor layers in ResNet-56 and over multiple trials of optimization on CIFAR-10. We also show the correlation of filters in the input channel direction. As can be seen, all optimization methods produce tensors that exhibit correlation (in additional smoothness for Sobolev) in the output channel direction while no (or much less) correlation in the input direction. Notice that our methods increase the amount of regularity compared to SGD as it imposes this in optimization.
Output Channel
Input Channel
0.15
0.125
SGD
SGD
Correlation
0.1
Reweighted
Reweighted
Sobolev
Subelev
0.075
0.05
0.025
[-]]
[+]]
Figure 11: Correlation of Final Tensor. Correlation between weights within different channel directions in CIFAR trained ResNet 56 conv layers (over 10 trials). li ji is distance between weight locations in tensor for correlation computation. Sobolev/re- weighted H show strong correlation in output direction, but not input. SGD shows correlation in output direction.
D DETAILED DERIVATIONS FOR SECTION 2.2
We first derive the re-weighted L gradient under H² metric following the same notations from the paper. Consider f A V HO L(X) the standard L gradient, and we want to solve for g _ L LXX..
By and we have
k), 10
(g, mg (g. k) no + + J. R)) 10
(14)
(15)
Note the fact that (g, k no
0 holds for all k. This is because I Tg(k - oo(-k))do - 9 s(k k) do
and kk= do 0 since k is zero-mean. In this way, k and To become set of orthogonal basis
After decomposing f and & into
f -= Π) k= k),
(16)
by simple algebra we have
j= S- Mag g)
(17)
which leads to the result of (8)
We then derive the Sobolev gradient under H² metric, following similar computations in Sun daramoorthi et al. (2007). Consider V L(X) the Sobolev gradient under H' metric. By 6 and we have
VV (V k) no
(V H: k))
(18)
1
/ Ök
OV mi L(X)
(k, vm: 2o + XO do do
(19)
O
R2
Integrating by parts and considering the periodic boundary conditions, we have 82
(V 1o L()), = (Vm Lm)
2O22 do² vm
(20)
190
12

---

Under review as a conference paper at ICLR 2021
Since l can be any perturbation, by uniqueness, we have
VHOL(X) = LH)((X) - XO² 02
V (()
(21)
do?
which is 88. Similarly, for H metric, we have
02 V do² Vn I(X).
Hoo = V m XO2
(22)
First observe that by computed the output-channel directed average of the both sides of the above equation, we see that V Vii "() V Ho (Xx), i.c., the average values are same. One may integrate
(22 twice to solve for the H' gradient. For simplicity, let be the L gradient and g be the H gradient. Integrating twice yields
[ 0g do
(0, i, h, w) dō 1 2.² J (I(00,,,,,,) - F(i, u)) dō
g(o, i, h1 w) = g(0, i, (.,ww. w) +
(23)
/ do
0g (0, i,, h, w) dō
X 1//0 1 (f(00,,,h,,) f(i, h, w)) dō
1
1
=(0, w)
(24)
do 0g (0, i, h, w)
X f((0- (o - (o-
1
000, h, w)
f(i h, w)) dō.
(25)
Note that here we perform normalization by scaling the channel direction by letting [0. 1].
With boundary conditions g()) =1()), do
0g (0) 80 (1) and f g. we have
0g (0, i, h w) - f
1
off(oO, i, h, w) - f(i, h w)) do. For simplicity, we eliminate i, h. w and O in the following derivations. We have
(26)
do
g(0) g(0) 0g (0) + [ (o 10-000 ())() - f)
(27)
do
9(0) o X S ol((o) do + 1 / f((0- ())
1
(28)
Noting f [0) do = g(0) and f fu f(o) do, we integrate both sides over the entire interval [o, 1].
(()) = - + x f odo / off - f)
do + A 1./ [ (o (((--01010) ()(() - do
(29)
= ff + 2X / of(o) do
xx 1 + 1 1 ..6 I (o (0--)) 00) dōdo +f f [L. 1 (o ō) dō do
(30)
1
[ of(o) do + X 1/' Ja
1 (o (o) dō
(1 41 6X
/ +
(31)
2X
5
1
I of 1 1//² 1 d
/²
(1
(0)
do +
(32)
12X
27
0 0 1/6 )5()) do.
(1+
(33)
2
This gives (12 in the main paper.
E
CODE FOR SOBOLEV GRADIENT
The Pytorch code to compute the Sobolev gradient is provided in Figure 12 In theory, the 'cumsum' operation should be the main part of the code with largest computational cost. However, in order to match with standard Pytorch library, tensor operations including 'permute', 'repeat' and 'unsqueeze' are currently required. These operations contribute to over 70% of computational overhead, and can be avoided by if the computation were done using C++.
13

---

Under review as a conference paper at ICLR 2021
def Sobolev_gee -gad (raad (grad=aram grad data lambda)
# #grad: L2 gradient; lambda>0
L = grad size (0)
s = torch arange (L, ttype float32 ) cuda ()
tmp_mean torch mean (nean(grad grad ,0, True repeat 11,1)
tmp_ diff (grad - tmp_mean). ppr_muen) (1,2 ,,,,,,3, ,0,
gp_0 = torch (tmp_ ((
gp_0 0 gp_0. unsqueeze_ (3). repeat (epeat(1,1,1...
s = s. unsuueeze_ (0), wwwuueezeuueeeeeeeeeeeeeuuueeeee (0)zee__0.n (0)_ unsquezze (0), repeat
(tmp_diff diff. size (0), tmp_dmff......d size ,
tmp_ diff. size 12).1)
# Sobolev gradient computation
tmp2 s*gp_0-(s* torch cumsum (tmp_diff -im==)
torch cumsum ( s* tmp_diff dim (1))/((mmmaaal
grad lambda (tmp2. permute (3,0,1
(mp_mean)
return grad
Figure 12: Pytorch code to compute the Sobolev (H)) gradient from the N gradient. The 'permute', 'repeat' and 'unsqueeze' operations are due to standard library limitations, and can be avoided by further code optimization (e.g., writing the function in C++/Cuda that Pytorch calls).
F FURTHER ANALYSIS OF CORRELATION IN CONVOLUTIONAL LAYERS
Existing analysis on regularity in CNNs (Mao et al. (2017) focus on filter level and kernel level regularities in pruning. To the best of our knowledge, the leamnel-directee regularity proposed by paper has not been investigated nor has the regularity been used in optimization.
We show below that the output-direction correlation is not due randomness in the network due to particular weight transformations that leave the output behavior of the CNN fixed. The reason for the output direction correlation is unknown to us, but direction for future investigation.
Could it be due to random noise? No. Figure shows the histogram of correlation of representa- tive convolutional layer from ImageNet pretrained DenseNet. Note that the input channel correlation is distributed as zero-mean Gaussian, which is likely to be due to random noise. In contrast, the output channel correlation (our proposed channel direction) shows positive correlation. There are also outliers (points near 0.3) corresponding to channels with mean value far away from zero (see vertical lines in Figure Tb. This shows that the neural network prefers regularities the output channel direction.
Outpu: Conclation
Input Channel Comelation
Nomalized Correlation
Figure 13: Histogram of correlation of representative tensor. While the input channel correlation distributed as zero-mean Gaussian, output channel shows positive correlation and sparsity. The neural network prefers regularities in the output channel direction.
Could it be due to scaling? No. In modern CNNs, scaling the affine factor in BatchNorm layer could create such structure in the following convolutional layer, without affecting the output of the neural network. Note that in Figure I] we use correlation that is invariant to re-scalings this is not the case. We also investigate this further. Figure 14 presents the scatter plot of mean of tensors the output direction (larger mean corresponds to stronger correlation) and standard deviation of output channels. scalings are contributing to the regularity, there would be a positive correlation between the mean and standard deviation, scaling amplifies both. The plot does. NOT show positive
14

---

Under review as a conference paper at ICLR 2021
correlation between mean and standard deviation in this channel direction, which means the structure not due to simply scaling up all weights within particular channels (producing the same CNN).
Slanderd Devialion
0.06
0.02
0.01
0.02
0.04
Absolute Value of Mean
Figure 14: Scatter plot of mean and standard deviation of output channels. There is no positive correlation between channel mean and standard deviation, showing that the structure in output channel direction is not due to scaling. Each color corresponds to a layer.
15


Reviews:
[{"review_id": "y3Ogy-_SdV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a modification of the gradients of objective function wrt model parameters by changing the underling metric on the manifold of model parameters.  It proposes two metrics, a reweighed L2 metric, and a specific Sobolev metric, and presents how to compute modified gradients in a compute efficient manner.  Motivated by empirical observation that parameters of trained CNN tensors tend to be correlated along output dimension, the paper focuses on metrics that promote such correlation.\n\nThe approach of considering new metrics is appealing in that it allows keeping original loss function, model parameterization, and optimization methodology; it modifies the gradients in potentially beneficial ways leading to better optimization.  The formulation presented is mathematically rigorous.\n\nThe biggest criticism I have is that the baseline systems presented in the paper appear very far behind the best known results. For instance,\n    * State of the art accuracy on CIFAR-10 is over 99% but the models presented in the paper achieve below 90%. This is a very large gap.\n    * For PascalVOC the mean IoU is in high 80s, not sure why the paper cites numbers near 40. I’ve re-read the paper a few times to find out what am I missing but can’t tell why the numbers are so different.\nWith baselines like that it becomes very hard to assess the real value of the proposed approach.\n\nAnother weak aspect of the paper is the choice of output channel dimension for smoothing.  It is empirically motivated based on the correlations in parameters along output dimension.  At the same time results also show that other dimensions such as input channel dimension lead to accuracy degradation.  However, since the method allows for selecting any subset of variables to smooth over, could there be more principled approaches for identifying optimal subset?  Could the optimal metric be somehow learnt?\n\nFor aforementioned reasons I feel the paper needs more work prior to publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UZ549vCIvqK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an optimization method for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error.  The method computes the gradient of the loss function with respect to output-channel directed re-weighted $H^0$ or Sobolev metrics. \n\nThe paper has the following merits:\n+ The paper shows that defining the gradients along the output channel direction leads to a performance boost, while other directions can be detrimental. I wonder whether the method can be used for other networks beyond CNN?\n+ The continuum theory of gradients, its discretization, and application to deep networks are provided. \n+ The effectiveness of the proposed method is demonstrated by experiments on benchmark datasets, several networks, and baseline optimizers.\n+ The method requires only simple processing of existing stochastic gradients, can be used in conjunction with any optimizer, and has only a linear overhead compared to computation of the stochastic gradient.\n\nThe paper is well structured and well written; while I was not able to confirm every single proof and algorithm, I am confident that the topics are presented in a sound way and would make an interesting topic at the conference.\n\n-------------------------------------------------------\nI have read the response, and the rating is not changed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RbdG5X-I87E", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper propose a stochastic optimization method for CNNs, which can obtain an improvement in terms of generation error. The proposed method is used to compute the gradient with respect to output-channel directed re-weighted matrices or Sobolev metrics, and some anabasis is also provided. Some experimental results show that the proposed method can be improved in generalization error.\n\n1. It is not clear that what's the relationship between input channel, output channel correlations and Sobolev gradients.\n2. What's the key difference between Laplacian smoothing and Sobolev gradients? \n3. The relationship and difference between channel-directed gradients and Sobolev gradients should be analyzed. \n4. The experimental analysis is less convincing. The authors should compare the proposed algorithm with sophisticated methods such as manifold optimization algorithms.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IgvAvss33hn", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a method to compute the gradient of the loss function with respect to output-channel directed re-weighted H0 or Sobolev metrics, which has the effect of smoothing components of the gradient across a certain direction of the parameter tensor. The proposed method was used to train CNNs with gradient based optimisation methods such as SGD and Adam. \n\nIn general, the proposed method and motivation are interesting. Experimental results show that the proposed method improves baseline accuracy for small batch sizes.\n\nHowever, there are some major issues with the proposed method and analyses:\n\n- In the proposed method, Sobolev gradients are formulated by considering the space of parameter tensors as a Riemannian manifold, and choosing the Sobolev metric on the tangent space. That is, the method is proposed as a new optimisation method on Riemannian manifolds of parameters. Therefore, the following issues should be resolved by additional analyses and explanations:\n\n— How do you identify the Riemannian manifold of these tensors?\n— How do you assure that tensors remain on the manifold during training of networks?\n— Please compare the proposed method with other Riemannian optimisation methods applied to train CNNs in the experimental analyses.\n\n- Please provide variance of accuracy for all the experiments, esp. for classification tasks to analyse statistical significance of results, since mean accuracy values of baseline and the proposed methods are very close to each other.\n\n- Parameter correlations for CNNs trained on Imagenet were analysed in Figure 1. However, additional analyses on employment of the proposed method for these CNNs were not given in the experimental analyses. Please apply the proposed method on larger benchmark datasets such as Imagenet, and provide these results as well. These results may be also used to examine the claim proposed in Figure 1.\n\n- The proposed code snippets were useful to see how of the methods are implemented using PyTorch. However, we implemented similar experiments using these code snippets, but could not achieve similar results (indeed, obtained worse accuracy wrt baseline in some cases). A reason of this observation may be difference in implementation of other parts. Therefore, it would be useful and more helpful if you could provide the complete, esp. for experiments on image classification using Cifar 10 and semantic segmentation using Pascal VOC.\n\n\nAfter the rebuttal:\n\nI checked comments of other reviewers and response of authors.\n\nFirst, thank you for the detailed response. Since some of my questions and concerns are partially addressed, I improve the overall rating.\n\nHowever, there are still parts which should be improved:\n\n- Regarding manifolds: There are some statements which should be clarified and appropriately analyzed in the paper. For instance, it is stated that \"The manifold of parameter tensors is a linear space, and so any linear combination of the parameter set with a tangent vector (another tensor) will remain on the manifold. We are changing the metric on the tangent space from the ordinary Euclidean metric to our channel directed (Sobolev, and re-weighted L2) metrics, which effectively changes the lengths of paths on this manifold of parameter tensors, making it a non-trivial Riemannian manifold.\"\n-- This statement claims that the \"structure\" of the manifold of tensors changes by changing the metric on tangent space by changing geodesic or paths on manifolds. However, it is not clear how the geodesic or in general, geometry of the manifold changes by just changing the metric on the tangent space (while, the tangent space can change depending on change of the manifold).  \n-- Then, it is claimed that this leads to a non-trivial Riemannian manifold (a nonlinear space), while it is also claimed that the manifold of parameter tensors is a linear space (why is it a linear space?). \n\n- Regarding experimental results: Thank you for the updated results. However, these results show that the accuracy of the proposed method is pretty close to the baseline. To show superiority of the proposed method, the experiments should be extended using larger benchmark datasets.\n\n-- There are Riemannian optimization method that encodes this channel-directed structure, but there are various Riemannian optimization methods which claim to improve accuracy of models in various tasks such as the following:\n\nS. Kumar Roy, M. Harandi, R. Hartley and R. Nock, Siamese Networks: The Tale of Two Manifolds. (oral), Int. Conference on Computer Vision (ICCV), Seoul, 2019.\n\nLei Huang, Xianglong Liu, Bo Lang, Admas Wei Yu, Bo Li, Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks, AAAI 2018 (Oral)\n\nTherefore, the proposed methods should be compared with these methods in the analyses as well.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1W_YMuIIrko", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper follows a very active line of research on gradient-based optimization methods for convolutional neural networks. In particular, this work proposes a method that can be applied as an extension to popular optimization methods such as SGD and Adam. The main idea is to introduce a modification of the tensor space in order to encourage correlations in the output-channel dimension. In particular, they propose two  channel re-weighting strategies: H^0 and H^1.  The theoretical foundation behind the paper seems to be sound and the overall writing is clear. Results indicate that, indeed, the proposed modifications help to improve the performance of the baselines: SGD and Adam. This is shown using several datasets. Furthermore, the proposed method does not produce a significant overhead with respect to the regular operation of the baselines. \nAs a main conclusions, the main topic of the paper is relevant to ICLR and the proposed method shows a contribution, although minor, to the operation of two of the most popular optimization methods used today. As a disclaimer, this research area is not close to my main topics of expertise.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper begins with an observation in standard trained CNNs that the correlations in the output channels are high. Building upon this the paper proposes a new "optimizer" which modifies the gradients to encourage corelations among output channels. They provide a theoretical foundation for the method, by deriving the gradient through placing a riemannian metric on the manifold of parameter tensors which encourages smoothness along the output channel dimension. Two variants (one based on a Sobolev metric) are proposed and are experiments are provided. The underlying idea and the derivation of the gradients were generally appreciated by the reviewers. However some reviewers maintained their concern regarding the effectiveness of the performed experimentation. The gains demonstrated are relatively small over the baselines and more importantly the baselines are quite far off the state of the art baselines for the particular problems. This is the primary reason for my recommendation as experiments are the only source of understanding whether the method is effective (there is little theory - mostly at an intuitive level to justify the form of the optimizer). Overall, I strongly encourage the authors to explore the idea further and strengthen the paper with stronger baselines (perhaps on larger datasets) and resubmit. 
Decision: Reject