Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author propose a new regularization method for training the deep neural networks, instead of weight decay.
The proposed method is named Logit Attenuating Weight Normalization (LAWN), which constrains the weight norm in 
training with the projected gradient. The experimental results show LAWN is more effective than weight decay in Adam and 
LAMB optimizers.
main_review: 1. The proposed method is almost the same as AdamP [1]. The difference is there are two gradient projection operation.
    The motivation and main approach is very similar to AdamP. But the author did not compare with AdamP and even not mentioned it in 
    related works.
2. The compared method Adam is not fair. It is well-known that Adam with L2 regularization weight decay leads to bad generalization.
    A proper way to add weight decay is to use the decoupled weight decay, i.e., AdamW. Although the author mentioned AdamW in related   
   work section, it seems not be a comparison method in the experimental part. 
3.  Please give the illustration of how to tune E_{free} for different datasets and different learning processes.
4.  Actually, only using Eq (3) on gradient can not make sure Eq (2) hold. The norm of the weight will still gain. There should be a weight 
    projection step. But I did not find it in the proposed method. Why do not use weight  projection step to keep the norm of weight 
    unchanged? If the norm of the weight changes in each step, the author should not state that  the proposed method optimizes the 
    constrained objective function in Eq (2).






[1] Heo B, Chun S, Oh S J, et al. AdamP: Slowing down the slowdown for momentum optimizers on scale-invariant weights[J]. arXiv preprint arXiv:2006.08217, 2020.


summary_of_the_review: The similar idea has been conducted in the work AdamP. And the compared methods are not convincing in the experiment part.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a network training algorithm to overcome the bad local minima for better network generalization. The proposed method, dubbed logit attenuating weight normalization (LAWN) is to constrain the weight norms to constrain the value of logit / softmax. The experiments justifies that the proposed method can be easily combined with different optimizers and improved the generalization ability of learned network. 
main_review: Overall, I appreciate the simple but effective weight normalization method for network training. My major concerns are on the reason/analysis of the effectiveness of LAWN, the sensitivities to its hyper-parameters, the clarity on the details of the model, and comparisons.  

1.  It is understandable that the weight scale should be bounded to improve the stability of network prediction. However, the reason of why the proposed normalization in Eqn.(2) is effective compared with the other approaches, such as the weight decay, LSR, etc., should be more deeply analyzed.

2.  For Eqn. (3), according to appendix D, it is to implement the constrain of lines 4 and 9 in algorithms 1, for optimizing eqn.(2) under the weight constraints. I am afraid that the gradient descent in eqn.(3) may not exactly find the projected gradient, can it practically guarantee that the weights constraint can be satisfied when optimizing eqn. (2)? 

3. The sensitivity of performance to parameter c^l in Eqn. (2) should be given in details. Though it is heuristically set as |\bar{w}^l|, whether the proposed algorithm is sensitive to this parameter should be analyzed.

4. The proposed algorithm is mainly compared with the baseline algorithms such as LSR, weight decay. There are several related works on improving generalization of training algorithm by escaping the bad local minima. The authors are suggested to more thoroughly compare with more related state-of-the-art training algorithms in literature.
summary_of_the_review: The proposed training algorithm is simple, but showed to be able to improve network generalization ability apparently.  I mostly like the results, but the proposed algorithm should be more thoroughly analyzed and compared as pointed out in the above comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a technique called Logit Attenuating Weight Normalization (LAWN) which is a modification that can be applied to gradient-based optimizers. LAWN combines a gradient projection step with weight normalization to overcome the short-comings of adaptive gradient-based optimizers especially when being applied to large batch sizes. The LAWN method is applied to different deep learning models for image-classification and movie recommendations and increased performance is demonstrated. It is also shown that LAWN improves network calibration.
main_review: STRENGTHS:
- The results are promising and the paper contains a lot of experiments. However, some of them are very specific to recommender systems. (See below.)
- The LAWN modification seems to improve the performance of adaptive optimizers on image-classification and recommendation tasks as well as the calibration of deep neural network classifiers.
- The author(s) is/are very transparent about the limits of their method's applicability.

WEAKNESSES:
- Although the author(s) state in the abstract that they want to address the generalization gap between adaptive-optimizers and SGD in the image classification domain, most experiments are limited to very specific recommender systems datasets. The only image-classification datasets considered in this work are CIFAR-10, CIFAR-100 and Imagenet, which limits my enthusiasm for this paper.
- There are some issues with the notation: using $nc$ to denote the number of classes is problematic. I would suggest to use subscripts (e.g., $n_c$ for number of classes or $n_w$ for the number of weight variables) instead of variable names consisting of two letters. Equation 3 defines $g_p^\ell$, however in Algorithm 1 $g_{pt}^\ell$ (line 4) is used. Perhaps it would be better to define Equation 3 also depending on some time step t to make clear that this is what line 4 of Algorithm 1 is referring to.
- I find it strange that SGD failed in the MovieLens and Pinterest datasets for very large batch-sizes. As a reader i am curious why and how it fails. It would be nice if the author(s) elaborate(s) on that matter during the rebuttal period. What accuracy does SGD achieve in that case?
- Why is it necessary to start training the network in a free (unregularized) mode? What happens when the $c^\ell$ is constant? Does the value for $c^\ell$ really differ for each dataset/model? Which test accuracy does one get when setting $E_\text{free}$ to 0 epochs, i.e., when using the norm of the initial weight vectors for normalization? At least for larger batch sizes (4k or 10k) the trend shown in the Figure 4 of the supplementary material seems to suggest that this could further improve the accuracy.
- Table 2 could be extended to also include similar experiments on the Imagenet dataset.

MINOR REMARKS:
- Multiple wrong citations, e.g., "Hoffer et al (Hoffer et al., 2018)" instead of "Hoffer et al. (2018)" or "Salimans and Kingma (Salimans & Kingma, 2016)" instead of "Salimans and Kingma (2016)"
summary_of_the_review: Overall, see some issues with this submission that could be addressed by the author(s) during the rebuttal. I am willing to improve my score when my concerns are addressed appropriately.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to overcome the loss flattening problem in deep neural networks by explicitly constraining the weight norm during training. The paper argues that as a result of scale-variant loss functions in classification and ranking tasks, the loss can be made small by pushing logits to large magnitude. This loss adaptivity phenomenon is argued to hurt generalization since the escaping condition in SGD training is difficult to satisfy if the logits are large. To alleviate such a problem, the paper proposes to fix weight norms during training (LAWN), which is implemented by projected gradient. The empirical result on several datasets shows the advantage of LAWN, especially when a large batch size is used.
main_review: # Strengths:

The motivation for this work is straightforward and reasonable. In a homogeneous neural network, the prediction in classification tasks is invariant to the scale of logits and weights. However, as the neural network is over-parameterized it's often fitting the training data quite well and the weight scales are increased to reduce the loss. The paper argues that the loss flattening worsens the generalization since the model cannot escape from local minimum easily. The proposed LAWN algorithm is clearly presented and some hyperparameter settings are given in the appendix. 


# Weakness:

My major concern for this work is that it neglects batch normalization (BN) layers which are crucial to the performance of ResNets but are invariant to weight norm scaling. In other words, the loss flattening problem cannot be alleviated by constraining weight norms in networks with BN because scaling weight norms in BN has no impact on the logit scales. If the paper argues that even in networks with BN the LAWN still has its advantage, this argument should be explicitly made in the paper. Though a ResNet50 is used in the ImageNet experiment, I did not find any details on how to deal with BN. 

Another concern is about the implementation of weight norm constraint. If my understanding is correct, Equation (3) is how the constraint is enforced. Appendix D shows the projected gradient will not increase the weight norm assuming the learning rate is infinitesimal. However, a large learning rate is necessary for the generalization of DNNs and is often used in training practice of DNNs. In fact, the projected gradient always increases the norm with a practical learning rate since it is orthogonal to the weight, and the learning rate controls the increase of the weight norm. Compared with weight normalization (Salimans & Kingma 2016), Equation (3) is the gradient of v^l when c^l is 1. The difference is not so clear from my point of view.

Overall, I think the paper's analysis on the loss flattening is interesting and I appreciate the effort of authors. But the current manuscript is not ready to be published at ICLR. I hope the paper can be revised to address my above concerns. 

summary_of_the_review: In spite of its interesting motivation, the paper does not properly handle DNNs with Batch Norm such as ResNets and many variants, and the  way the weight norm constraint is implemented is confusing.
review_title: null

Meta-review: The reviewers unanimously recommend rejecting this submission and I concur with this recommendation. The submission essentially introduces a regularization technique to solve the alleged problem of Adam getting worse out-of-sample error for typical image classification problems, e.g. training ResNets on ImageNet. Reviewers raised a variety of issues with the submission. Some found the experiments unconvincing, some were concerned that the submission duplicated closely related work without engaging with and citing that work, and some were concerned by what they viewed as insufficient analysis and comparisons. To me, the most severe issue with the submission is that the experimental evidence for its claims is not sufficiently convincing and the problem it purports to solve has not been convincingly demonstrated, making the work hard to motivate. The other issues raised by the reviewers are less damaging in my view.

Although this is a meta review and not a full de novo review, I would be remiss to not raise a few of the severe issues I see with the results that makes it hard for them to be convincing.
The Adam results in table 1 are far weaker than they should be, raising questions about the experiments as a whole. For example, https://arxiv.org/abs/2102.06356 reports 76.4% top 1 accuracy for ResNet-50 on ImageNet with Adam without increasing the epsilon parameter to a larger value as Choi et al. 2019 did (who also report good Adam results for ResNet-50 on ImageNet). This should also lead us to question one premise of the paper that there is some problem with adaptive optimizers for image classification.

Ok, but perhaps LAWN helps validation error even if there is no gap between SGD and Adam? Sadly, to demonstrate this subordinate claim, LAWN would have to be compared carefully with state of the art regularization techniques and compared with results that use any optimizer, not just Adam. With modern regularization techniques, it isn't hard to get 77%+ top 1 validation accuracy on ImageNet with ResNet-50. See, for example https://arxiv.org/abs/2010.01412v1 which gets 77.5% in 100 epochs and as high as 79.1 with longer training. Since LAWN is claiming to improve generalization, it must be compared with other regularization techniques. It is a type error to primarily compare it with optimizers so even if there weren't concerns with the performance of the existing baselines, there would need to be additional comparisons.

The claims about fixing issues that arise at large batch sizes are prima facie problematic since there isn't strong evidence of an actual problem at the batch sizes considered in the submission.
Decision: Reject