Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper follows a very active line of research on gradient-based optimization methods for convolutional neural networks. In particular, this work proposes a method that can be applied as an extension to popular optimization methods such as SGD and Adam. The main idea is to introduce a modification of the tensor space in order to encourage correlations in the output-channel dimension. In particular, they propose two  channel re-weighting strategies: H^0 and H^1.  The theoretical foundation behind the paper seems to be sound and the overall writing is clear. Results indicate that, indeed, the proposed modifications help to improve the performance of the baselines: SGD and Adam. This is shown using several datasets. Furthermore, the proposed method does not produce a significant overhead with respect to the regular operation of the baselines. 
As a main conclusions, the main topic of the paper is relevant to ICLR and the proposed method shows a contribution, although minor, to the operation of two of the most popular optimization methods used today. As a disclaimer, this research area is not close to my main topics of expertise.
review_title: Channel-Directed Gradients for Optimization of Convolutional Neural Networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The paper proposes a method to compute the gradient of the loss function with respect to output-channel directed re-weighted H0 or Sobolev metrics, which has the effect of smoothing components of the gradient across a certain direction of the parameter tensor. The proposed method was used to train CNNs with gradient based optimisation methods such as SGD and Adam. 

In general, the proposed method and motivation are interesting. Experimental results show that the proposed method improves baseline accuracy for small batch sizes.

However, there are some major issues with the proposed method and analyses:

- In the proposed method, Sobolev gradients are formulated by considering the space of parameter tensors as a Riemannian manifold, and choosing the Sobolev metric on the tangent space. That is, the method is proposed as a new optimisation method on Riemannian manifolds of parameters. Therefore, the following issues should be resolved by additional analyses and explanations:

— How do you identify the Riemannian manifold of these tensors?
— How do you assure that tensors remain on the manifold during training of networks?
— Please compare the proposed method with other Riemannian optimisation methods applied to train CNNs in the experimental analyses.

- Please provide variance of accuracy for all the experiments, esp. for classification tasks to analyse statistical significance of results, since mean accuracy values of baseline and the proposed methods are very close to each other.

- Parameter correlations for CNNs trained on Imagenet were analysed in Figure 1. However, additional analyses on employment of the proposed method for these CNNs were not given in the experimental analyses. Please apply the proposed method on larger benchmark datasets such as Imagenet, and provide these results as well. These results may be also used to examine the claim proposed in Figure 1.

- The proposed code snippets were useful to see how of the methods are implemented using PyTorch. However, we implemented similar experiments using these code snippets, but could not achieve similar results (indeed, obtained worse accuracy wrt baseline in some cases). A reason of this observation may be difference in implementation of other parts. Therefore, it would be useful and more helpful if you could provide the complete, esp. for experiments on image classification using Cifar 10 and semantic segmentation using Pascal VOC.


After the rebuttal:

I checked comments of other reviewers and response of authors.

First, thank you for the detailed response. Since some of my questions and concerns are partially addressed, I improve the overall rating.

However, there are still parts which should be improved:

- Regarding manifolds: There are some statements which should be clarified and appropriately analyzed in the paper. For instance, it is stated that "The manifold of parameter tensors is a linear space, and so any linear combination of the parameter set with a tangent vector (another tensor) will remain on the manifold. We are changing the metric on the tangent space from the ordinary Euclidean metric to our channel directed (Sobolev, and re-weighted L2) metrics, which effectively changes the lengths of paths on this manifold of parameter tensors, making it a non-trivial Riemannian manifold."
-- This statement claims that the "structure" of the manifold of tensors changes by changing the metric on tangent space by changing geodesic or paths on manifolds. However, it is not clear how the geodesic or in general, geometry of the manifold changes by just changing the metric on the tangent space (while, the tangent space can change depending on change of the manifold).  
-- Then, it is claimed that this leads to a non-trivial Riemannian manifold (a nonlinear space), while it is also claimed that the manifold of parameter tensors is a linear space (why is it a linear space?). 

- Regarding experimental results: Thank you for the updated results. However, these results show that the accuracy of the proposed method is pretty close to the baseline. To show superiority of the proposed method, the experiments should be extended using larger benchmark datasets.

-- There are Riemannian optimization method that encodes this channel-directed structure, but there are various Riemannian optimization methods which claim to improve accuracy of models in various tasks such as the following:

S. Kumar Roy, M. Harandi, R. Hartley and R. Nock, Siamese Networks: The Tale of Two Manifolds. (oral), Int. Conference on Computer Vision (ICCV), Seoul, 2019.

Lei Huang, Xianglong Liu, Bo Lang, Admas Wei Yu, Bo Li, Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks, AAAI 2018 (Oral)

Therefore, the proposed methods should be compared with these methods in the analyses as well.

review_title: The proposed method should be further explained, and additional analyses should be provided.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an optimization method for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error.  The method computes the gradient of the loss function with respect to output-channel directed re-weighted $H^0$ or Sobolev metrics. 

The paper has the following merits:
+ The paper shows that defining the gradients along the output channel direction leads to a performance boost, while other directions can be detrimental. I wonder whether the method can be used for other networks beyond CNN?
+ The continuum theory of gradients, its discretization, and application to deep networks are provided. 
+ The effectiveness of the proposed method is demonstrated by experiments on benchmark datasets, several networks, and baseline optimizers.
+ The method requires only simple processing of existing stochastic gradients, can be used in conjunction with any optimizer, and has only a linear overhead compared to computation of the stochastic gradient.

The paper is well structured and well written; while I was not able to confirm every single proof and algorithm, I am confident that the topics are presented in a sound way and would make an interesting topic at the conference.

-------------------------------------------------------
I have read the response, and the rating is not changed.

review_title: An intersting paper on CNN optimization

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a modification of the gradients of objective function wrt model parameters by changing the underling metric on the manifold of model parameters.  It proposes two metrics, a reweighed L2 metric, and a specific Sobolev metric, and presents how to compute modified gradients in a compute efficient manner.  Motivated by empirical observation that parameters of trained CNN tensors tend to be correlated along output dimension, the paper focuses on metrics that promote such correlation.

The approach of considering new metrics is appealing in that it allows keeping original loss function, model parameterization, and optimization methodology; it modifies the gradients in potentially beneficial ways leading to better optimization.  The formulation presented is mathematically rigorous.

The biggest criticism I have is that the baseline systems presented in the paper appear very far behind the best known results. For instance,
    * State of the art accuracy on CIFAR-10 is over 99% but the models presented in the paper achieve below 90%. This is a very large gap.
    * For PascalVOC the mean IoU is in high 80s, not sure why the paper cites numbers near 40. I’ve re-read the paper a few times to find out what am I missing but can’t tell why the numbers are so different.
With baselines like that it becomes very hard to assess the real value of the proposed approach.

Another weak aspect of the paper is the choice of output channel dimension for smoothing.  It is empirically motivated based on the correlations in parameters along output dimension.  At the same time results also show that other dimensions such as input channel dimension lead to accuracy degradation.  However, since the method allows for selecting any subset of variables to smooth over, could there be more principled approaches for identifying optimal subset?  Could the optimal metric be somehow learnt?

For aforementioned reasons I feel the paper needs more work prior to publication.
review_title: weak baselines, needs more work

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper propose a stochastic optimization method for CNNs, which can obtain an improvement in terms of generation error. The proposed method is used to compute the gradient with respect to output-channel directed re-weighted matrices or Sobolev metrics, and some anabasis is also provided. Some experimental results show that the proposed method can be improved in generalization error.

1. It is not clear that what's the relationship between input channel, output channel correlations and Sobolev gradients.
2. What's the key difference between Laplacian smoothing and Sobolev gradients? 
3. The relationship and difference between channel-directed gradients and Sobolev gradients should be analyzed. 
4. The experimental analysis is less convincing. The authors should compare the proposed algorithm with sophisticated methods such as manifold optimization algorithms.
review_title: Prof. 

Meta-review: The paper begins with an observation in standard trained CNNs that the correlations in the output channels are high. Building upon this the paper proposes a new "optimizer" which modifies the gradients to encourage corelations among output channels. They provide a theoretical foundation for the method, by deriving the gradient through placing a riemannian metric on the manifold of parameter tensors which encourages smoothness along the output channel dimension. Two variants (one based on a Sobolev metric) are proposed and are experiments are provided. The underlying idea and the derivation of the gradients were generally appreciated by the reviewers. However some reviewers maintained their concern regarding the effectiveness of the performed experimentation. The gains demonstrated are relatively small over the baselines and more importantly the baselines are quite far off the state of the art baselines for the particular problems. This is the primary reason for my recommendation as experiments are the only source of understanding whether the method is effective (there is little theory - mostly at an intuitive level to justify the form of the optimizer). Overall, I strongly encourage the authors to explore the idea further and strengthen the paper with stronger baselines (perhaps on larger datasets) and resubmit. 
Decision: Reject