Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to scale the components of the gradients during back-propagation to achieve the same results as branched reparametrization.
strength_and_weaknesses: The paper is well-written. It has a coherent story and it is interesting to read. The related work section is properly organized. As I was not aware of some of the papers mentioned there, I found it extremely useful.

I find the idea of the paper elegant. Although it is simple, it demonstrates that a simple modification of the learning process can achieve the same result (or better) than a modification of the architecture.

I didn't notice any significant issues. However, I would appreciate if the authors explain why they use mutual information for gradient rescaling. Why not some other function? And if it is not limited to mutual information, would the results differ if the method relies on some other measure for spatial correlation?
clarity,_quality,_novelty_and_reproducibility: The text is well-written. The source code is not shared. However, it seems possible to reimplement the method myself. 
summary_of_the_review: The paper proposes a novel and elegant idea. I think it is worth sharing with the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to scale the gradients in convolutions through learning pixel/feature dependencies
strength_and_weaknesses: Strengths:
- The paper seems novel.
- Derivations seem correct

Weaknesses:
- Figures seem to be in low resolution
- Table 1 evaluation is only with VGG (quite old), not sure why it did not include ResNet or more recent networks.
- Results on CIFAR 100 are quite low compared to the state-of-the-art
- I missed a comparison with other reparameterization methods, e.g. Salimans and Kingma (Neurips, 2016) and others.
clarity,_quality,_novelty_and_reproducibility: Paper is written clearly with graphs that help understand the methodology. 
Reproducibility: I did not find a link to the code
summary_of_the_review: Paper seems interesting, although there are a few improvements that would benefit the paper
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a gradient scaling approach to replicate structural reparametrization that improves training. The main idea is based on the linearity of convolutions and the observation that the effect of reparametrization can be achieved by simple gradient scaling. The results show the benefits of the approach on multiple vision datasets.
strength_and_weaknesses: ## Strengths
1. The idea is simple and elegant. It replaces structural reparametrizations which require high memory and computational overhead during training with a simple gradient scaling that performs similar to such methods.
2. The paper is well-written, and the idea is backed by the proofs and experimental results.

## Weaknesses
1. It is not clear why mutual information is a good measure to obtain gradient scales. Specifically, it is not clear to me why one should weight a filter coefficient higher if it is similar to the center pixel in a convolution filter. Even though the results show the benefits it would be good to motivate it intuitively. Please comment on this.
2. Even though the idea is simple, there were several implementation details (eg., the last para on page 6) required to make it work in practice. I think reproducibility might be a concern due to this.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, the idea is good and overall the paper is of high quality. Reproducibility might be a concern so I would encourage the authors to release the code.
summary_of_the_review: Overall, the idea is nice and it is backed by theory and experiments.

## Post rebuttal
I thank the authors for the response. I would recommend adding a paragraph about the intuition behind using mutual information for gradient scaling. Additionally, the extensive experiments across various settings show the practical benefits of the method. I was already positive and I would like to see the paper accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to rescale gradient components based on spatial mutual information when optimizing convolutional neural networks. They show that the proposed spatial gradient scaling is equivariant to previous work on structural re-parameterization (which uses a multi-branch topology network during training and re-parameterize back to the original convolution for inference). They compare their approach to previous work using structural re-parameterization for classification tasks on the CIFAR and ImageNet datasets and demonstrate that the proposed technique improves performance at a lower computational cost.

strength_and_weaknesses: **Strength:**
1. This paper is relatively well-written and most technical parts are clear to me. The experimental details are also sufficient.
2. The connection between spatial gradient scaling and parallel convolution parameterization is a good observation. It shows that previous work on muti-branch reparameterization is unnecessary for achieving better performance.

**Weaknesses:**
1. The proposed approach is heuristic and lacks theoretical foundations. The use of mutual information for gradient rescaling is intuitive but it is hard to justify from a theoretical perspective. 
2. There are useful tricks in deep learning which do not have a theoretical explanation in the beginning. But in that case, the empirical study needs to be more convincing. It may not be enough to only apply their technique to classification tasks and compared only to previous work on structural re-parameterization. In my opinion, it would be convincing if they could show that their performance improvement can not be possibly achieved by using other optimization methods with adaptive gradient (e.g. AdaGrad), and their empirical gain is robust across tasks such as object recognition, image segmentation, and image generation, and across a wide range of base CNN architectures. Of course, I am not asking the authors to run these experiments during the rebuttal period. These are just suggestions to make a strong paper in the future. That level of empirical study is usually not necessary, but for a more heuristic technique, it might be good to demonstrate the robustness of the performance gain.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**

The paper is well-paced with good explanations and illustrations. The paper contains sufficient technical details.

**Quality:**

I am convinced that their proposed spatial gradient scaling is equivalent to branched reparameterization in previous work without introducing structural changes. I have no questions regarding this contribution and I think their experiments also support that claim well.

However, I do not understand why structural reparameterization is an important and promising direction. As shown by this paper, previous work on structural reparameterization can just be seen as gradient rescaling. Can this line of research be seen as designing data-dependent optimizers dedicated to convolutional neural networks? If that is the case, can we find other optimization methods with adaptive gradients such that gradient rescaling is no longer essential? I think the baselines should not be previous work on structural reparameterization, because we already know they are equivalent. The paper should try to demonstrate why structural reparameterization/spatial gradient scaling is an attractive technique and the practitioners should be recommended to use it. If this cannot be justified theoretically, then more empirical studies need to be conducted to show that performance improvement is robust and can be used on a variety of tasks/architectures, and no other techniques (e.g. adaptive gradient optimizers) can not achieve similar improvement. 

**Reproducibility:**

The paper provided enough experimental details. The code is currently not provided.
summary_of_the_review: I am leaning toward weak rejection but I think this paper still has merits. 

Their claim that the proposed spatial gradient scaling is equivalent to branched reparameterization in previous work without introducing structural changes is correct and it is also supported by their experiments. 

However, the proposed technique is too heuristic with weak theoretical justification. In experiments, they only compare to branched reparameterization in previous work  (which has already been shown to be equivalent to their proposed approach). So I am not convinced that this technique can bring robust performance gains that cannot be achieved using other optimization algorithms with adaptive gradients.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method to rescale components of the gradient during the back propagation to achieve similar performance as branched reparametrization. The paper is well-written and easy to follow. The proposed approach is novel and leads to marginal improvements. Therefore, I recommend acceptance. 
justification_for_why_not_higher_score: Because the reported empirical gains are not significant and there is no theoretical justification for the proposed method.
justification_for_why_not_lower_score: Because the proposed idea is novel and leads to empirical gains.
Decision: Accept: poster