Logit Attenuating Weight Normalization
Authors: Aman Gupta
Rohan Ramanath
Jun Shi
Anika Ramachandran
Sirou Zhu
Mingzhou Zhou
Sathiya Keerthi
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
gradient methods
stochastic optimization
generalization gap
imagenet
adam
large batch training

Abstract: Over-parameterized deep networks trained using gradient-based optimizers is a popular way of solving classification and ranking problems. Without appropriately tuned regularization, such networks have the tendency to make output scores (logits) and network weights large, causing training loss to become too small and the network to lose its adaptivity (ability to move around and escape regions of poor generalization) in the weight space. Adaptive optimizers like Adam, being aggressive at optimizing the train loss, are particularly affected by this. It is well known that, even with weight decay (WD) and normal hyper-parameter tuning, adaptive optimizers lag behind SGD a lot in terms of generalization performance, mainly in the image classification domain.

An alternative to WD for improving a network's adaptivity is to directly control the magnitude of the weights and hence the logits. We propose a method called Logit Attenuating Weight Normalization (LAWN), that can be stacked onto any gradient-based optimizer.  LAWN initially starts off training in a free (unregularized) mode and, after some initial epochs, it constrains the weight norms of layers, thereby controlling the logits and improving adaptivity. This is a new regularization approach that does not use WD anywhere; instead, the number of initial free epochs becomes the new hyper-parameter. The resulting LAWN variant of adaptive optimizers gives a solid lift to generalization performance, making their performance equal or even exceed SGD's performance on benchmark image classification and recommender datasets. Another important feature is that LAWN also greatly improves the adaptive optimizers when used with large batch sizes.


Under review as a conference paper at ICLR 2022
LOGIT ATTENUATING WEIGHT NORMALIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
ever-parameterized deep networks trained using gradient-based optimizers is a popular way of solving classification and ranking problems. Without appropriately tuned regularization, such networks have the tendency to make output scores (logits) and network weights large, causing training loss to become too small and the network to lose its adaptivity (ability to move around and escape regions of poor generalization) in the weight space. Adaptive optimizers like Adam, being aggressive at optimizing the train loss, are particularly affected by this. It is well known that, even with weight decay (WD) and normal pareee-prrmmter tuning, adaptive optimizers lag behind SGD lot in terms of generalization performance, mainly in the image classification domain.
An alternative to WD for improving a network's adaptivity is to directly control the magnitude of the weights and hence the logits. We propose method called Logit Attenuating Weight Normalization (LAWN), that can be stacked onto any gradient-based optimizer. LAWN initially starts off training in a free (unregularized) mode and, after some initial epochs, it constrains the weight norms of layers, thereby controlling the logits and improving adaptivity. This is new regularization approach that does not use WD anywhere; instead, the number of initial free epochs becomes the new hyper-parameter. The resulting LAWN variant of adaptive optimizers gives a solid lift to generalization performance, making their performance equal or even exceed SGD's 's performance on benchmark image classification and recommender datasets. Another important feature is that LAWN also greatly improves the adaptive optimizers when used with large batch sizes.
INTRODUCTION
The advent of large scale deep models with tens of millions to billions of parameters has resulted in three trends in the community: (1) Sfate-offthe-aar performance via over parameterized networks for problems like image classification, language modeling, machine translation, text classification and recommender systems. (2) Development of optimizers like Stochastic Gradient Descent (SGD) with heavy-ball momentum (Qian 1999), Adam (Kingma & Ba 2017). Adam W (Loshchilov & Hutter 2019), LAMB YYou al. 2020 and their extensions with weight decay/l2 regularization to improve generalization performance. (3) Increased emphasis on theory to understand the optimizer landscape, especially how to tune different hyperparameters well to escape poor minima for both large and small batch sizes. In this work, inspired by all three trends, we propose a new training method, explain why it works, and show vastly improved performance over the weight decay method when applied with adaptive optimizers across a wide range of batch sizes for classification and ranking tasks.
Complex deep networks can easily learn to classify large fraction of examples correctly as training progresses. These networks have two characteristics: (a) they have one or more contiguous homogeneoud layers at the end forming the end homogeneous sub-network, and, (b) they are trained with exponential-type loss functions like logistic loss and cross entropy that asymptotically attain their least value of zero when the network score goes to infinity. Let us collectively refer to this network score logit. After the network has learned to correctly classify large fraction of training examples, the weights of the end homogeneous layers and the logits grow to make the training loss (and hence its gradient) very small. This, seen in optimizers when used with no (or mild) weight decayléy regularization, leads to loss flattening (loss and gradient taking very small values) (see Appendix A). This further leads to loss of adaptivity of the network (Szegedy al. 2015). causing training to stall in regions of sub-optimal generalization (see 2
layee
A layer is homogeneous if the activation function of each unit of the layer satisfies ((ax) = ad(I.). Linear and ReI. U are examples of such activation functions.

---

Under review as a conference paper at ICLR 2022
Figure D shows the generalization performance (Test HR @ 10) of Adam without weight decay (green dotted line) applied to a fully homogeneous network on a classification task. After about 115 epochs, Margin p50 (median margin over the training examples; blue dotted line) becomes large and the training gets stuck in the basin of a sub-optimally generalizing minimum. The minimum is also overfitting due to the tussle between good and noisy examples.
70
65-
Optimizer
Batch Size
256
16k
SGD
76.00 (0.04)
74.48 (0.06)
Test HR@00
Margin p50
55-
Adam
71.16 (0.05)
70.60 (0.02)
50-
Adam-I.
76.18 (0.03)
76.07 (0.08)
LAMB+
74.30 (0.01)
73.43 (0.03)
45
LAMB-L
76.48 (0.05)
75.93 (0.02)
40-
Table 1: ImageNet validation accuracy. Comparison SGD, Adam and LAMB (all with WD) and LAWN variants (*-L) of Adam and LAMB+ N on the ImageNet validation set. LAW enables Adam to work on image classification tasks with very little drop performance at large batch sizes. Current optimizers have a much steeper drop-off in performance as batch size increases. Standard error is mentioned in parentheses.
35
0 50
100 150
200 250
300
Epochs
Figure 1: Adam with (continuous lines) and without (dotted lines) logit attenuation on Moviel ens classification. Adam: after 115 epochs (vertical dotted black line), loss flattening sets in, Test HR @ 10 is sub-optimal, with overfitting. Adam with logits attenuation: The introduction of a factor after the 115th epoch reduces Margin p50 but keeps Test HR @ 10 the same. Though Test HR @ 10 then drops initially, with further constrained weight training Adam with logit attenuation eventually achieves a higher Test HR @ 10 than vanilla Adam.
"LAMB+ is modification of the LAMB algorithm. More details can be found in Appendix 33..3
Adaptive gradient algorithms like ADAM are particularly severely affected by loss flattening. These algorithms are based on tracking the local loss behavior each individual weight level and hence they achieve fast convergence. However, in domains such as image classification, their generalization performance quickly plateaus to a value that much worse than the performance finally achieved by SGD, This gap is usually attributed to the inability of adaptive gradient algorithms to escape the basin of a sharp minimum that has large curvature and possibly poor generalization. While weight decay helps improve adaptive optimizers, it is still not sufficient to close the gap with SGD. The training method proposed in this paper closes the gap.
To motivate the method, let us return the experiment of Figure D Consider that training has reached a point where logits are starting to become large. Attenuation of the logit values can be done using the following two ideas: (a) multiply the logits by factor, 0 a < 1 and use this factor for the rest of the training: (b) constrain the norms of layer weights for the rest of the training. Specifically, in the experiment of Figure [ after epoch 115 we shrink the logits to one-fifth, and then continue training while keeping the weight norms of each layer fixed. This leads to superior generalization with no overfitting (see the continuous lines in the figure). It turns out that idea (a) alone is not sufficient since further training will increase the logits again to reduce the training loss; so, idea (b) is very important. Also, in order control the logit magnitudes, instead of idea (a) (waiting till 115 epochs and then reducing the logits by a factor 0.2) we could simply start using idea (b) without applying an factor, at an earlier point, say, after just epochs.
This lays the base for our method, Logit Attenuating Weight Normalization (LAWN). It can be used with any gradient- based base optimizer, though our aim is mainly to improve adaptive optimizers. LAWN begins with normal, unconstrained training in the initial phase and then constrains the weight norms of the layers for the rest of the training. Weight decay is not used anywhere. The training on the constrained norm surfaces is done by employing projected gradients instead of regular gradients.
The LAWN variants of Adam and LAMB achieve impressive performance across multiple network architectures and tasks. At large batch sizes, most optimizers get caught in sub-optimal regions due to lowered stochasticity which is worsened by increased logits. LAWN's attenuation of the logits
2

---

Under review as a conference paper at ICLR 2022
helps avoid this worsening. Due to this, the LAWN variants of Adam and LAMB work significantly better than their base versions at large batch sizes. Table [] which compares the generalization performance of SGD, Adam and LAMB (all with weight decay) against the LAWN variants of Adam and LAMB on the popular Imagenet dataset, powerfully showcases the two observations on LAWN mentioned above. In f we show that these LAWN variants give much better generalization than weight decay on several architectures for image classification (CIFAR, ImageNet) and recommender systems (MovicLens, Pinterest). We end this section with a discussion of related work.
Related work: The issue of loss flattening and the resulting loss of adaptivity is highlighted in (Szegedy et al.) 2015). Several well-known techniques are used to mitigate this issue. f2 regularization is often used in conjunction with optimizers like SGD and adaptive optimizers Duchi et al. (0TT), among others. Recently, decoupled weight decay (Loshchilov & Hutter 2019 Hanson & Prati 1988) has become popular to reign-in network
(Kingma & Ba 2017| Zeiler 2012:
weights and prevent overcontidence on training samples. Other techniques are: (a) label smoothing regularization (Szegedy et al. 2015), which makes the model less confident about predictions by changing the ground label distribution; and (b) flooding @Ishida et al., 2020) which tries to keep the aggregate training loss to be around specified small value. From theory perspective, weight norm bounding has been shown to be useful for improving generalization Neyshabur et al.. 2015) Bartlett et al., (017). Salimans and Kingma (Salimans& & Kingmal 2016) use weight normalization as a transformation; but, by also keeping the scale component, they end up allowing logits to grow large. Hoffer et al (Hoffer et al. 2018) discuss keeping the norms of the parameters fixed, but the method would require a LAWN-like approach to work effectively.
It has been observed that small batch sizes yield better generalization performance (Keskar et al. 2016) due to the noise of mini-batch gradients and the large learning rate. The noise diminishes with increasing batch sizes. Large batch sizes are useful for speeding up the training process by leveraging parallel GPUs. Poor generalization for large batch sizes is attributed to them stalling around "sharp" minimizers (Keskar et al. 2016). Goyal et (Goyal et al., 2018) scale ImageNet training to batch sizes of 8k without loss in generalization performance, by carefully tuning parameters like learning rate and batch normalization. Other recent efforts to train large- models include (Hoffer et al. 2017: You et al. 2017: 2019 Shallue et al., 2018 You et al. 2020). It is important to note that for majority of the cited works, large batch gains do not necessarily hold across tasks or datasets.
2 THE NEED FOR LAWN
In this section we motivate LAWN by describing the problem settings in which normal training struggles and in which LAWN could improve generalization performance. We define these problem settings in 22. then describe what we mean by loss of adaptivity in $2. We review the strengths
and weaknesses of existing methods for avoiding loss of adaptivity in $2.3 With this context, we
elaborate on the LAWN method in 3
Notations. For the rest of the paper we will use multi-class setting as the rumning example. The following notations will be used: n is the number of weight variables; m is the number of training examples; i is the index used to denote the index of a training example; is the target class of the i-th training example; ie is the index used to denote a class; ne the number of classes; w is the weight vector of the deep net; p& (a(w) is the class k probability assigned by the deep net with weight vector w. For an optimizer, 7) denotes the learning rate and B is the batch size.
2.1
WHEN DOES LAWN WORK?
Let us describe the problem settings in which we expect generalization improvement using LAWN
Problem type. We are mainly interested in classification and ranking problems which form a score for the target. In binary classification this score is the logit score of the target class; in multiclass problem, this score is the difference between the target class score and the maximum score over the remaining classes. For ease of presentation we will simply refer to such scores as logit. We consider a loss function that attains its least value (usually zero) asymptotically as logit goes to infinity. The logistic loss for binary classification and the cross entropy loss based on softmax for multi-class classification are important cases. Problems like regression that use a loss function such as the squared loss (which attains its minimum at finite value) may not have much benefit using LAWN. In a multi-task setting, when the total loss is an additive mix of several individual task losses, LAWN can be useful even if just one of them is suited for LAWN.
3

---

Under review as a conference paper at ICLR 2022
Network Complexity. This refers to the complexity of the network in relation to the number of training examples. Most deep networks used in applications are pveamaarameriized. Roughly, we will take aher-parametrizee to mean that the network is so powerful that training easily locates that classifies most examples correctly, i.e., the target class has the highest score among all classes. Deep nets usually have one or more fully connected homogeneous layers (usually with ReLU units) the end. If the deep net is powerful enough to correctly classify most examples correctly, then by making the weights large it is possible to push the loss of most training examples (and thereby the train loss) to very small values.
Generalization Metrics. Our focus is on improving metries that are based on score ordering as opposed to the actual values of scores. More precisely, we are interested in test set metrics such as classification error, AUC, NDCG etc. as opposed to logistic loss, cross entropy loss, probability calibration error etc. Deep networks are known to be poor with respect to the latter metrics (Guo et al., 2017) but which can be improved in the post-training stage; the adaptation of LAWN to improving such metrics will be taken up in a follow-up work F
2.2 ISSUE OF LOSS OF ADAPTIVITY
Consider the problem setting defined in 1. and the minimization of the normally used unregularized training loss, L(w)- = 1 2im1 log Py, ()). With the network being sufficiently powerful, training causes the losses of a large fraction of the training examples to become very small. During training, this happens due to weights becoming large, the logit becoming large, and Py, + 1 for those examples. Due to these, each of the following become very small: the gradient of most example-wiss wise loss terms; E, the covariance of the noise associated with minibatch gradient; and, H, the Hessian of the training loss. We will refer to this collective happening loss flattening.
It has been established via theoretical and empirical arguments that the powerful generalization ability of deep net comes from its ability to escape regions of attraction of "sharper" minima with sub-optimal generalization performance and go to better solutions. Appendix C gives an idea of the escape mechanism for the SGD method using simplistic analysis given by Wu al (Wu al.), which we use just for guidance and motivation. It is worth recalling from there, the following rough
guiding condition for SGD to escape from a poor solution:
Amax ((I n)H
n (m- B) Σ} > 1, B(m 1)
where Amax (A) denotes the largest eigenvalue of A. If training is at sub-optimal generalization point, loss flattening occurs (which means that H and > become small), then the escape condition is difficult to satisfy and hence it becomes difficult for the network to escape from this solution and then train further to go a better solution. A carefully increased learning rate schedule cause the escape followed by the use of normal learning rates to go a better solution can make this happen, but no such automatic sophisticated learning rate adjustment mechanism has been devised yet. We will refer to this inability of the network to escape out of sub-optimal solution due to loss flattening as loss of adaptivity (also see $7 in (Szegedy et al. 2015) ).
2.3 CURRENT METHODS FOR DEALING WITH LOSS OF ADAPTIVITY
Several methods have been suggested in the literature to handle the issue of loss of adaptivity.
We briefly describe three key ones: label smoothing regularization (LSRR (Szegedy et al., 2015).
flooding (Ishida et al. 2020), and le regularization/weigh decay (Loshchilov & Hutter 2019) LSR modifies L(w) via making the target class less confident by fixing its probability as (1 ELSR) and reassigning CLSR to the remaining classes. This makes the loss attain its minimum at finite logit values. Flooding modifies the loss as Lllodding (w) = IL
is forced to move around the hypersurface defined by L Fllodingg so that the training process
FFlodding = 0. l2 regularization is a traditional method that modifies the loss as Le, = L(w) + 2 [|]l (Decoupled) Weight decay. as used in the recent deep net literature decouples the term Aw from the gradient and instead includes the additive term, -Xw at the weight update step. In the next section ($3) we will return to discuss these methods in relation to LAWN
3 THE LAWN METHOD
As we mentioned in 2 we consider deep nets that have one or more homogeneous layers (usually with ReLU units) at the end (top); let fhsn refer to this final homogeneous sub net. Consider weight
2 However, as we show $4.6 LAWN does much hetter calibration than the weight decay method. Originating from the work of Keskar et al (Keskar et al. 2016). it has heen observed in the literature that minima at which the classification function has sharper hehavior has poorer generalization.
4

---

Under review as a conference paper at ICLR 2022
vector tū and corresponding fhsn weights shan- When we extend fhen along radial direction, ie., Wyen = f fhan. a E R4, while keeping the weights of the network outside fhsn unchanged, the classification error set of examples (c.g.. the training error computed training set the generalization error computed on a test set) remains unaffected whereas classification loss varies a lot with a. In particular, if w classifies set of examples strictly correctly, then, goes from O to oc, the average loss on these examples goes from log(nc) to zero asymptotically. When training without any/sufficient weight decay, weights do get large and training loss does become very small. In the previous section ($2) we saw how, when training loss becomes very small, it causes loss flattening, which leads to loss of adaptivity of the network. Therefore, it makes sense to suitably contain the size of the weights. Theory (Neyshabur et al.) 2015 Bartlett et al.) 2017) suggests that, for improving generalization, it a good idea to bound the weights of the entire network. Given that most layers are usually homogeneou [ (e.g., units with ReLU activations), it is appropriate to use layer-wise weight normalization. The essential spirit of LAWN is along this idea.
Consider constrained training via layer-wise weight normalization: ||= = vl where wf is the weight vector accociated with layer f, and the cf are some constants. It is useful to understand how the loss contours behave as the c' go from small to big values. When the cf are large, we know that loss flattening will happen. When the of are small, the distinction between the loss values well classified examples and poorly classified examples diminishes and so, optimizers will find it harder to traverse the contours and go to the right place of best generalization. To choose the right ef values, in LAWN we take simple and natural approach. We initialize the network with weights having small magnitude using a standard weight initialization method and start a given optimizer in its free (unconstrained, without any weight decay) form. At suitable point in that training process, with weights at some Wū we switch to constrained training defined by setting
||wl|| δ¹ where cf | 4I
(1)
The et are fixed for the rest of the LAWN training. Constrained training corresponds to solving the optimization problem,
min L(w) s.t. ||wo' c AA
using a modified version of any given gradient- based optimizer.
In LAWN, we switch from free training to constrained training after some Ffrree epochs, and tune Ereee as a hyperparameter using a coarse grid. Thus, when compared to regularization using weight decay, LAWN (a) does not use weight decay anywhere, and (b) it uses E free hyperparameter instead.
In the future, we plan to try the following automatic method for choosing Esree- Track the logits of training examples (done simply and efficiently by tracking them on the minibatches used) and switch to constrained training when their median value starts reaching high values indicating the onset of loss flattening.
Hoffer et al (Hoffer et al., 2018) give bounded weight normalization method which also constrains the weight norms and uses simple heuristics for setting the ct. However, it does not use the idea of combining free and constrained training. and it does not set up and demonstrate the use of constrained training as powerful method for use with adaptive optimizers for improving the performance by overcoming loss flattening and loss of adaptivity. In we conduct experiments on image classification and recommender datasets and show that Hoffer al's method is quite inferior to LAWN.
Algorithm Adam-LAWW Constrained Phase
1: for in I...T do
2:
Draw batch S, from the training set
3:
9t ComputeGrad( (1--1, St)
4: Project each of to {d (w") ) de = 0} to get of V
5:
m. 3,m-
(1 31 9pt
6:
Ut 82 + (1 B2)
7:
m, = T- 02 = 11
m.
T-
8:
Compute ri- = Voi+e
me
9:
Project each rf to {d' (w = 0 to get of ve
10:
We W;-1 T/
11: Rescale we to satisfy constraints on || Aa
$
12: end for
'In fully homogeneous nets, layer-wise weight normalization is a way of removing some redundancies Dinh
2017 layer-wise weight norms also connect well with implicit bias properties (Lyu Li 2020
5

---

Under review as a conference paper at ICLR 2022
LAWN Implementation. There two ways of implementing the constrained phase of LAWN. The first method is to define an unconstrained vector of and set w" = c' v /||="|| and simply apply the optimizer to vf. This is the implementation suggested by Salimans and Kingma (Salimans & Kingma 2016); see equation (2) there. (Note, however, that Salimans and Kingma (Salimans & Kingmal 2016) keep the radial component by including the scale parameter, g.) The downside with this method is that it puts load on the computational graph, and automatic differentiation through the normalizing transformation increases the computational cost (Huang et al. 2020
The second method is to have the optimizer directly deal with the constraints. At a given weight vector, w, let g = Lw)( For updating w", the projected gradient defined by
s - "-gg-"""" ((w")²"² )f w AA 1
naturally plays the role of the gradient for decreasing the loss on the manifold defined by I = cf and so it is used instead of the gradient in all optimizer related updates. In Appendix D we use
gradient flow to establish this. We use this method in our implementation of the constrained phase of LAWN. The implementation of the constrained phase of Adam-LAWN is given in Algorithm ||
Let us now describe the LAWN method as a complete algorithm. The free and constrained training of LAWN can be done with any given optimizer. LAWN does a total of Fxotat epochs; Erotat is fixed for a given dataset. After Free epochs of free training, with x denoting the weights reached, it sets
e || 'f and switches to do constrained training (solve 2 for the remaining (Erotal free) epochs. Learning rate schedules are important for deep networks attain good performance and LAWN employs the standard linear warmup and decay schedule (Loshchilov & Hutter [019] Liu
et al. 2020; Loshchilov & Hutter, 2016). This schedule has two hyperparameters. Npeak, the peak learning rate, and Ewarmup. the number of warmup epochs. Erree is the additional hyperparameter; as already mentioned, this hyperparameter replaces the weight decay parameter.
LAWN and large batch sizes. For a fixed epoch budget, larger batch sizes require a smaller number of steps; combined with distributed computation this helps speed up training. However, since stochasticity of updates reduces with large batch size, the mechanism of escape from sub-optimal solutions gets affected. Thus, one usually sees a reduction in generalization performance as batch size is increased (Shallue al. 2018). Loss flattening makes this issue worse by affecting adaptivity. LAWN, by helping overcome this issue. leads to a more graceful degradation of performance as a function of batch size. We will empirically demonstrate this in $ The degradation becomes far less (even zero) when the the total number of steps is allowed to decently increase with batch size.
LAWN and weight adaptivity. Research in the last five years is clearly showing that improving weight adaptivity is the key to escaping inferior weights and reaching weights with superior generalization performance. It was believed that the noise associated with stochastic gradient is the only way to ensure such adaptivity, leading to the promotion of smaller batch sizes and larger learning rates. But other ways of improving adaptivity are being suggested. For example, injection of suitable forms of artificial noise has been shown to improve generalization (Wu et al.] [2020] Wen al. 2020). is even being suggested that adaptivity can be improved without any stochasticity and with just suitable deterministic regularization (Geiping et al.) 202T). More interesting research is expected on this important topic of weight adaptivity. In this line of research and methods, LAWN can be thought of as an orthogonal technique of improving adaptivity by suitably constraining the norms of the weights and avoiding loss flattening. While weight decay also helps in a similar way, we will show in 4hat LAWN is much superior.
4 EXPERIMENTS
Our basclines include SGD. Adam (Kingma & Ba 2017) and LAMB You et al. 2020). We add weight decay to all baseline algorithms, and additionally add momentum to SGD. To evaluate LAWN, we consider LAWN-based variants of the adaptive optimizers (Adam and LAMB). SGD has demonstrated strong performance for computer vision tasks (Ren et al. 2015 Goyal et al. 2018), whereas adaptive methods like Adam perform well on other domains (eg. recommender systems, text classification). To demonstrate the efficacy of LAWN across a wide variety of tasks, we conducted experiments the CIFAR (Krizhevsky et al. 2009) and ImageNet (Deng et al.] 2009 Krizhevsky et al. 2012) datasets for image classification, and the MovieLens (Harper & Konstan 2015) and Pinterest (Geng al. 2015) datasets for item recommendation. All model training code

---

Under review as a conference paper at ICLR 2022
was implemented using the PyTorch library (Paszke et al. 2019) and experiments conducted on machines with NVIDIA V100 GPUs. For each experiment, we report average test metric over 3 runs.
Hy pyperparameters: Regular versions of SGD, Adam, and LAMB use weight decay. The LAWN versions do not use weight decay; instead, they use Érree For a given dataset, the total number of epochs, Etotal fixed. Learning rate schedule (with warmup of learning rate from 0 to 1peak in Ewarnup epochs followed by decay to zero in the remaining epochs) has proved beneficial. The above mentioned hyperparameters were all tuned to get the best generalization performance. For Opeak we used equally spaced values in logarithmic scale suited for each (optimizer, dataset) combination. The value of Etotal- the range of values for weight decay, Eree and Ewermap. and additional details used for individual datasets are given in Appendix B Apart from these, SGD's ' momentum value was fixed at 0.9, unless specified otherwise Appendix B For Adam, LAMB, Adam-LAWN and LAMB-LAWN, we fixed f1 = 0.9 and B2 = 0.999. For Adam, we used e = 10-S and the rest of the adaptive optimizers use 10-6 We did not tune c, ,1 and 82. which could have led to further improvements.
Details about datasets, pre-processing and network architectures for all experiments can be found in Appendix B
4.1
LAWN VS. OTHER METHODS FOR CONTROLLING LOSS OF ADAPTIVITY
MovieLLns-Ihh
CIFAR-TO
CIFAR-1O0
Method
BS
10k
BS
100%
BS = 4k
BS 10k
BS = 4k
BS 10k
LSR
68.66
67.34
93.09
92.73
69.92
69.24
WD
70.12
69.28
92.93
92.63
68.91
68.61
Hoffer
44.54
45.52
92.99
91.66
70.66
69.35
LAWN
70.41
70.77
93.74
93.84
73.13
72.97
Table 2: Comparison of test performance of LAWN with other methods for controlling loss of adaptivity on Movielens- 1M, CIFAR- 10 and CIFAR- 100 datasets. The base optimizer used Adam. Two different batch sizes, BS are tried for each dataset. For LSR, the smoothing parameter was fixed at 0.05. LAWN comprehensively outperforms other methods, including weight decay (WD).
We first compared LAWN to three key methods for controlling loss of adaptivity, discussed in 22. The comparison is done the three datasets, Movielens- 1M, CIFAR-10 10 and CIFAR-100. For each dataset we tried two values of batch size. Table 2 gives the results. LAWN clearly outperforms all the other methods. The second overall best method weight decay and hence it is used as the baseline for all remaining experiments of this section. The weight normalization technique suggested by Hoffer et al (Hoffer et al., 2018) does very badly Movielens- 1M; CIFAR-10 and CIFAR-100, 100, though it gives decent performance, it lags behind LAWN a lot. Clearly, this method requires a modification along the lines of LAWN in order for it do well.
4.2 IMAGE CLASSIHICATON FOR CIFAR-10 AND CIFAR- 100
For both CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), we used the VGG-19 CNN network (Simonyan & Zisserman 2014) with fully connected final layer. Our ImageNet experiments use a BesNeet-based He et al. 2015) architecture. All experiments were run with a 300 epoch budget. As seen Table 3 LAWN variants either match or outperform the base variants across batch sizes. Adam-LAWN is particularly impressive. This is in stark contrast to earlier held beliefs that adaptive optimizers cannot match SGD's generalization performance for image classification tasks (Wilson
al. 2017)
Effect of batch size. LAWN variants cause more graceful degradation of performance with batch size, as compared to base variants. Adam-LAWN causes almost no degradation in generalization performance even at batch size 10k (see Figures (aa 2b)..
Effect of Efree- We observed that switching early to LAWN mode (i.e. fixing Eree to less than 10 epochs) usually works well for generalization. See Appendix B for details. This is consistent with hypothesis that constrained training should kick in before loss flattening sets in.
4.3 RECOMMENDATION SYSTEMS
We conducted experiments on the MovieLens-1000 100k, MovieLens- 1M and Pinterest datasets using the popular neural collaborative filtering (NCF) technique (He et al. 2017). The datasets contain ratings provided to various items by users. The model is a 3-layer MLP, the input being user and item embeddings. We use the hit ratio 10 metric (expressed as %), where we reward the model for
7

---

Under review as a conference paper at ICLR 2022
15s1 Aucuney
Text Aucuney
72
Tox 13R 10
Text Accumey
70
Adam Adtun-LAWN
68
Adam Adden-LAWW
Adom Adauel LAWN
Addun- MlWN-LAWW
Adtart
10"
10
107
30
10P
11P
11P
UPP
10
Ratch sioe
Batch
(a) CIFAR-I0
(b) CIFAR-100
(c) MovicLens-IN 1M
(d) ImageNet
Figure 2: Adam-LAWN Adam (weight decay comprehensively tuned) for variety of datasets. Adam-
LAWN causes little to no drop generalization performance with increasing batch size.
ranking a test item in the top 10 of 100 randomly sampled items that a user has not interacted with in the past. We trained for 300 epochs for the two smallest batch sizes, and 500 epochs for the two biggest batch sizes for each dataset. Details about the datasets, pre- precresssinng model and evaluation can be found in Appendix B A summary of the performance of the aforementioned optimizers on all datasets can be found in Table 4 LAWN-basee optimizers consistently outperform their base variants. SGD failed to generalize well large batch sizes and this requires further investigation.
MovieLens- IM
Pinterest
Method
1k
10k
100k
400k
1k
10k
100k
IM
1k
10k
100k
IM
SGD
66.33
65.58
Fail
Fuil
70.91
69.31
Fuil
Fail
86.62
85,57
Fail
Fail
Adam
66.01
66.03
63.20
63.98
69.87
70.12
69.28
68.99
87.27
85.97
85.81
85.30
Adam-L.
66.81
66.91
66.24
66.14
70.80
70.41
70.77
70.66
86.85
86.61
86.04
86.06
LAMB
65.45
65.34
64.23
62.57
69.91
69.77
69.44
68.95
86.63
85.91
85.80
85.65
LAMB-L
66.56
66.54
66.52
66.14
70.86
70.86
70.68
70.34
86.83
86.25
85,99
86.07
Table 4: Test HR@ on Moviel.ens and Pinterest recommendations. Standard error is the range [0.15, 0.25]: Highlighted values indicate the hetter performing method hetween x and x-I...
details are in Appendix B
Weight decay vs. LAWN. Weight decay was used and tuned for all the base optimizers since arrests the uncontrolled growth of network weights, helping avoid of loss of adaptivity. The LAWN variants do not use weight decay but still outperform the base variants.
CIFAR-10
CIFAR-100
Method
256
4k
10k
256
SGD
93,99
93.48
92.99
73,49
71.68
71.07
Adam
93.48
92.93
92.63
70.84
68.91
68.61
Adam-L
93.91
93.74
93.84
72.99
73.12
72.97
LAMB
93.76
93.27
92.91
71.29
69.39
67.76
LAMB-L
93.67
93.22
92.92
71.25
69.68
69.16
Effect of batch size. LAWN variants of Adam and LAMB scale to very large batch sizes (1 million for MovieLens-INM 1M, 400k for MovieLens- without any appreciable loss in accuracy. SGD could only scale to batch size 10k. Adam-LAWN's strong scalability with batch size is consistent with results obtained from the CIFAR experiments (also see Figure 2(c)..
Table 3: Test accuracy on CIFAR- 10 and CIFAR- 100. Standard error is in the range [0.1, 0.45). Details are Appendix B Highlighted values indicate the better performing method between and x-I..
Effect of Efree- Similar to the results of the CIFAR experiments, fixing free a small value works well for LAWN. Details are in Appendix B
4.4 IMAGE CLASSIFICATION FOR IMAGENET
As compared CIFAR, the ImageNet classification problem (Krizhevsky et al. 2012 more representative of real world classification problems. We used a variant of the popular ResNet50 (He et al. 2015) model as the classifier. We considered small (256) and a large (16k) batch size for this experiment, and fixed training budget to be 90 epochs.
SGD, used in conjunction with momentum and weight decay, has long been the optimizer of choice for image classification. Adam is well known to perform worse than SGD for image classification tasks (Wilson) al. 2017). For our experiment, tuned the learning rate and could only get an accuracy of 71.16%. In comparison, Adam-LAWN achieves an accuracy of more than 76%, marginally surpassing the performance of SGD.
Results for batch size 256. Overall results can be found in Table (see
We found it difficult to reproduce ImageNet results using the LAMB algorithm. We made minor modifications (details in Appendix B) to the original algorithm to make it more stable, and call
8

---

Under review as a conference paper at ICLR 2022
the resultant algorithm LAMB+. LAMB-LAWN (the LAWN version of the unmodified LAMB) comprehensively outperforms LAMB+ for batch size 256 by achieving an accuracy close to 76.5%.
Results for batch size 16k. For the large batch size of 16k, we noticed that LAWN retains strong generalization performance (also see Figure ( Both Adam-LAWN and LAMB-LAWN achieve very high accuracy, with Adam-LAWN retaining its performance such large batch size by crossing the 76% test accuracy mark. This is with only additonally tuning for the LAWN variants Eree and Ewarmup.
Remark. Adam has traditionally performed worse than SGD at tasks like image classification.
Recent work (Choi al., 2020; Nado et al. 202T has shown that Adam's inner hyperparameters (that include $1, 32. <) could be the reason for the inferior generalization. The above cited works use sophisticated hyperparameter tuning algorithms over a relatively large search space (see Appendix D of Choi et al. (2020) and conclude that the optimal parameters vary a lot between datasets. While these are important results to close the gap in our understanding of Adam, they do little to improve the practical usability of Adam since it is prohibitively expensive to run the recommended number of training runs required to find the ideal hyperparameters.
4.5 LAWN WORKS WITH OTHER LOSS FUNCTIONS
To understand the effect of LAWN on loss functions other than cross entropy. we conducted experiments with focal loss (Lin et al. 2017). Focal loss was proposed as way of improving performance when the classification problem highly imbalanced, and has also recently found use for calibration of neural networks Muukhoi et al. 2020). is very different from cross-entropy, but still suffers from the issue of loss flattening. We conducted experiments using focal loss and Adam on both item recommendation and image classification. Results are given in Table 5 Adam-LAWN outperformed regular Adam with weight decay each one of the cases. This demonstrates the strengths of LAWN across a variety of loss functions and reinforces LAWN's efficacy in improving generalization performance.
MovieLens-Il IM
CIFAR-10
CIEAR-100
Method
Fl.
FLa
FLg
CE
F
FL2
FLg
CE
FLaa
FL2
FLg
CE
Adam (BSI)
67.33
69.02
67.17
69.28
93.07
91.16
87.88
92.93
69.19
68.85
67.72
68.91
Adam-L (BS1)
71.24
69.68
69.24
70.77
93.89
92.22
89.14
93.74
73.05
73.29
72.89
73.12
Adam (BS2)
66.42
67.40
66.62
68.99
92.51
89.79
86.82
92.63
67.94
67.
66.11
68.61
Adam-L (BS2)
70.53
69.56
68.66
70.66
93,21
91.20
88,93
93.84
72.51
72.55
70.95
72.97
Table 5: Adam vs. Adam-LAWN when used with focal loss (FL). We tried 3 different values (0.5, and 5) for (focal loss parameter) and also compared the results to cross entropy (CE) loss. BS1 and BS2 refer to batch sizes. For MovieLens-IM 1M, BS1 100k and BS2 = 1M. For the CIFAR datasets, BSI 4k and BS2 10k.
4.6 LAWN IMPROVES CALIBRATION
A well calibrated network is one in which the predicted class probability is close to the observed probability of being correct. Many applications require a well calibrated network. It is well known that werrppaamettrizzd deep nets are prone to over predicting probabilities (Guo et al. 2017). One of the main reasons for this is that weights become large, logits become large, and so the network is pushed to give out extreme probability values. Weight decay helps Adam improve calibration over unregularized Adam. Here we demonstrate that Adam-LAW- significantly improves the calibration even over Adam with weight decay. Estimated Calibration Error (ECE) is a standard metric for measuring calibration error (Guo et al. 2017). For CIFAR-10 10 and CIFAR-100, 100, with all hyperparameters tuned, Table f givees the ECE values for Adam and Adam-LAWN. It is clear that than Adam (with weight decay). This improved calibration is important advantage of LAWN.
Method
CIFAR-10
CIFAR-100
Adam
0.132
0.292
Adam-L
0.048
0.188
Table 6: ECE (lower is better) on image classification.
gives much smaller ECE values
5 CONCLUSION
LAWN as a simple and powerful method of modifying deep net training with a base optimizer to improve weight adaptivity and lead to improved generalization. Switching from free to weight norm constrained training at an appropriate point is a key element of the method. We study the performance of the LAWN technique on a variety of tasks, optimizers and batch sizes, demonstrating its efficacy. Tremendous overall enhancement of Adam and the improvement of all base optimizers at large batch sizes using LAWN important highlights.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Peter L.. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-noormliized margin bounds for neural networks. CoRR, abs/1706.08498, 2017.
Dami Choi, Christopher J. Shallue, Zachary Nado, Jachoon Lee, Chris J. Maddison, and George E, Dahl. On Empirical Comparisons of Optimizers for Deep Learning, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. CoRR, abs/1703.04933, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning reseurch, 12(7), 2011.
Jonas Geiping, Micah Goldblum, Phillip E. Pope, Michael Moeller, and Tom Goldstein. Stochastic training not necessary for generalization, 2021.
Xue Geng, Hanwang Zhang, Jingwen Bian, and Tat-Seng Chua. Learning image and user features for recommendation in social networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4274-4282, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in hour, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. CoRR, Abs/1706.04599 2017.
Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. Advances in neural information processing systems, 1:177-185, 1988.
Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. ACM fransactions on interactive intelligent systems (T7IS), 5(4):1-19, 2015.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385. 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp- 173-182, 2017.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint 17Xiv:1705.08741, 2017.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. arXiv preprint arXiv:1803.01814 2018.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. CoRR, abs/2009, 12836, 2020.
Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, and Masashi Sugiyama, Do we need zero training loss after achieving zero training error? arXiv preprint arXiv:2002.08709, 2020.
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point optimization trajectories of deep neural networks. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
10

---

Under review as a conference paper at ICLR 2022
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batel batch training for deep learning: Generalization gap and sharp minima. arXiv preprint
2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-I0 and CIFAR-100 datasets, URL:
https: /wwp:///www..o cs. toronte. wdww-kriz/ekaeeeeee
html 6(1):1, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25:1097-1105,
2012.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. CoRR, abs//708..220 020022 2017.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance the adaptive learning rate and beyond, 2020.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, arXiv preprint /608. /60800383,,, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks, 2020.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip S. Torr, and Puneet K. Dokania. Calibrating deep neural networks using focal loss. CoRR, abs/2002.09437 2020.
Zachary Nado, Justin M Gilmer, Christopher J Shallue, Rohan Anil, and George E Dahl. A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. arXiv preprint arXiv: 102.0033666 2021.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. CoRR, 0bbs//503.00036, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d' d'Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Ning Qian, On the momentum term in gradient descent learning algorithms. Neural networks, 12(1): 145-151, 1999.
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 150.01479 2015.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07866. 2016.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: 1.066000 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. CoRR, 1b5/1512.00067, 2015.
11

---

Under review as a conference paper at ICLR 2022
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of stochastic gradient descent with structured covariance noise. In AISTATS, pp. 3621-3631, 2020.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:/705.08295) 2017.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In ICML, pp. 10367 10376, 2020.
Lei Wu, Chao Ma, and Weinan E. How SGD selects the global minima in over- parameterized learning: A dynamical stability perspective. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Basa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc.
Yang You, Igor Gitman, and Boris Ginsburg, Large batch training convolutional networks, 2017.
Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large-batch training for Istm and beyond. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16, 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes, 2020.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven C. Hoi, and Weinan Towards theoretically understanding why SGD generalizes better than ADAM in deep learning. CoRR, abs/2010.05627, 2020.
12


Reviews:
[{"review_id": "icJAPGambPf", "reviewer": "Reviewer_dDDW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I appreciate the simple but effective weight normalization method for network training. My major concerns are on the reason/analysis of the effectiveness of LAWN, the sensitivities to its hyper-parameters, the clarity on the details of the model, and comparisons.  \n\n1.  It is understandable that the weight scale should be bounded to improve the stability of network prediction. However, the reason of why the proposed normalization in Eqn.(2) is effective compared with the other approaches, such as the weight decay, LSR, etc., should be more deeply analyzed.\n\n2.  For Eqn. (3), according to appendix D, it is to implement the constrain of lines 4 and 9 in algorithms 1, for optimizing eqn.(2) under the weight constraints. I am afraid that the gradient descent in eqn.(3) may not exactly find the projected gradient, can it practically guarantee that the weights constraint can be satisfied when optimizing eqn. (2)? \n\n3. The sensitivity of performance to parameter c^l in Eqn. (2) should be given in details. Though it is heuristically set as |\\bar{w}^l|, whether the proposed algorithm is sensitive to this parameter should be analyzed.\n\n4. The proposed algorithm is mainly compared with the baseline algorithms such as LSR, weight decay. There are several related works on improving generalization of training algorithm by escaping the bad local minima. The authors are suggested to more thoroughly compare with more related state-of-the-art training algorithms in literature.", "summary_of_the_paper": "This paper proposed a network training algorithm to overcome the bad local minima for better network generalization. The proposed method, dubbed logit attenuating weight normalization (LAWN) is to constrain the weight norms to constrain the value of logit / softmax. The experiments justifies that the proposed method can be easily combined with different optimizers and improved the generalization ability of learned network. ", "summary_of_the_review": "The proposed training algorithm is simple, but showed to be able to improve network generalization ability apparently.  I mostly like the results, but the proposed algorithm should be more thoroughly analyzed and compared as pointed out in the above comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bAQ6MBiQHIh", "reviewer": "Reviewer_qsYk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n\nThe motivation for this work is straightforward and reasonable. In a homogeneous neural network, the prediction in classification tasks is invariant to the scale of logits and weights. However, as the neural network is over-parameterized it's often fitting the training data quite well and the weight scales are increased to reduce the loss. The paper argues that the loss flattening worsens the generalization since the model cannot escape from local minimum easily. The proposed LAWN algorithm is clearly presented and some hyperparameter settings are given in the appendix. \n\n\n# Weakness:\n\nMy major concern for this work is that it neglects batch normalization (BN) layers which are crucial to the performance of ResNets but are invariant to weight norm scaling. In other words, the loss flattening problem cannot be alleviated by constraining weight norms in networks with BN because scaling weight norms in BN has no impact on the logit scales. If the paper argues that even in networks with BN the LAWN still has its advantage, this argument should be explicitly made in the paper. Though a ResNet50 is used in the ImageNet experiment, I did not find any details on how to deal with BN. \n\nAnother concern is about the implementation of weight norm constraint. If my understanding is correct, Equation (3) is how the constraint is enforced. Appendix D shows the projected gradient will not increase the weight norm assuming the learning rate is infinitesimal. However, a large learning rate is necessary for the generalization of DNNs and is often used in training practice of DNNs. In fact, the projected gradient always increases the norm with a practical learning rate since it is orthogonal to the weight, and the learning rate controls the increase of the weight norm. Compared with weight normalization (Salimans & Kingma 2016), Equation (3) is the gradient of v^l when c^l is 1. The difference is not so clear from my point of view.\n\nOverall, I think the paper's analysis on the loss flattening is interesting and I appreciate the effort of authors. But the current manuscript is not ready to be published at ICLR. I hope the paper can be revised to address my above concerns. \n", "summary_of_the_paper": "The paper proposes to overcome the loss flattening problem in deep neural networks by explicitly constraining the weight norm during training. The paper argues that as a result of scale-variant loss functions in classification and ranking tasks, the loss can be made small by pushing logits to large magnitude. This loss adaptivity phenomenon is argued to hurt generalization since the escaping condition in SGD training is difficult to satisfy if the logits are large. To alleviate such a problem, the paper proposes to fix weight norms during training (LAWN), which is implemented by projected gradient. The empirical result on several datasets shows the advantage of LAWN, especially when a large batch size is used.", "summary_of_the_review": "In spite of its interesting motivation, the paper does not properly handle DNNs with Batch Norm such as ResNets and many variants, and the  way the weight norm constraint is implemented is confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JLoSs2xN339", "reviewer": "Reviewer_Xf6G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed method is almost the same as AdamP [1]. The difference is there are two gradient projection operation.\n    The motivation and main approach is very similar to AdamP. But the author did not compare with AdamP and even not mentioned it in \n    related works.\n2. The compared method Adam is not fair. It is well-known that Adam with L2 regularization weight decay leads to bad generalization.\n    A proper way to add weight decay is to use the decoupled weight decay, i.e., AdamW. Although the author mentioned AdamW in related   \n   work section, it seems not be a comparison method in the experimental part. \n3.  Please give the illustration of how to tune E_{free} for different datasets and different learning processes.\n4.  Actually, only using Eq (3) on gradient can not make sure Eq (2) hold. The norm of the weight will still gain. There should be a weight \n    projection step. But I did not find it in the proposed method. Why do not use weight  projection step to keep the norm of weight \n    unchanged? If the norm of the weight changes in each step, the author should not state that  the proposed method optimizes the \n    constrained objective function in Eq (2).\n\n\n\n\n\n\n[1] Heo B, Chun S, Oh S J, et al. AdamP: Slowing down the slowdown for momentum optimizers on scale-invariant weights[J]. arXiv preprint arXiv:2006.08217, 2020.\n\n", "summary_of_the_paper": "The author propose a new regularization method for training the deep neural networks, instead of weight decay.\nThe proposed method is named Logit Attenuating Weight Normalization (LAWN), which constrains the weight norm in \ntraining with the projected gradient. The experimental results show LAWN is more effective than weight decay in Adam and \nLAMB optimizers.", "summary_of_the_review": "The similar idea has been conducted in the work AdamP. And the compared methods are not convincing in the experiment part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1nbtJ16Efy", "reviewer": "Reviewer_7vWF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n- The results are promising and the paper contains a lot of experiments. However, some of them are very specific to recommender systems. (See below.)\n- The LAWN modification seems to improve the performance of adaptive optimizers on image-classification and recommendation tasks as well as the calibration of deep neural network classifiers.\n- The author(s) is/are very transparent about the limits of their method's applicability.\n\nWEAKNESSES:\n- Although the author(s) state in the abstract that they want to address the generalization gap between adaptive-optimizers and SGD in the image classification domain, most experiments are limited to very specific recommender systems datasets. The only image-classification datasets considered in this work are CIFAR-10, CIFAR-100 and Imagenet, which limits my enthusiasm for this paper.\n- There are some issues with the notation: using $nc$ to denote the number of classes is problematic. I would suggest to use subscripts (e.g., $n_c$ for number of classes or $n_w$ for the number of weight variables) instead of variable names consisting of two letters. Equation 3 defines $g_p^\\ell$, however in Algorithm 1 $g_{pt}^\\ell$ (line 4) is used. Perhaps it would be better to define Equation 3 also depending on some time step t to make clear that this is what line 4 of Algorithm 1 is referring to.\n- I find it strange that SGD failed in the MovieLens and Pinterest datasets for very large batch-sizes. As a reader i am curious why and how it fails. It would be nice if the author(s) elaborate(s) on that matter during the rebuttal period. What accuracy does SGD achieve in that case?\n- Why is it necessary to start training the network in a free (unregularized) mode? What happens when the $c^\\ell$ is constant? Does the value for $c^\\ell$ really differ for each dataset/model? Which test accuracy does one get when setting $E_\\text{free}$ to 0 epochs, i.e., when using the norm of the initial weight vectors for normalization? At least for larger batch sizes (4k or 10k) the trend shown in the Figure 4 of the supplementary material seems to suggest that this could further improve the accuracy.\n- Table 2 could be extended to also include similar experiments on the Imagenet dataset.\n\nMINOR REMARKS:\n- Multiple wrong citations, e.g., \"Hoffer et al (Hoffer et al., 2018)\" instead of \"Hoffer et al. (2018)\" or \"Salimans and Kingma (Salimans & Kingma, 2016)\" instead of \"Salimans and Kingma (2016)\"", "summary_of_the_paper": "The paper proposes a technique called Logit Attenuating Weight Normalization (LAWN) which is a modification that can be applied to gradient-based optimizers. LAWN combines a gradient projection step with weight normalization to overcome the short-comings of adaptive gradient-based optimizers especially when being applied to large batch sizes. The LAWN method is applied to different deep learning models for image-classification and movie recommendations and increased performance is demonstrated. It is also shown that LAWN improves network calibration.", "summary_of_the_review": "Overall, see some issues with this submission that could be addressed by the author(s) during the rebuttal. I am willing to improve my score when my concerns are addressed appropriately.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously recommend rejecting this submission and I concur with this recommendation. The submission essentially introduces a regularization technique to solve the alleged problem of Adam getting worse out-of-sample error for typical image classification problems, e.g. training ResNets on ImageNet. Reviewers raised a variety of issues with the submission. Some found the experiments unconvincing, some were concerned that the submission duplicated closely related work without engaging with and citing that work, and some were concerned by what they viewed as insufficient analysis and comparisons. To me, the most severe issue with the submission is that the experimental evidence for its claims is not sufficiently convincing and the problem it purports to solve has not been convincingly demonstrated, making the work hard to motivate. The other issues raised by the reviewers are less damaging in my view.

Although this is a meta review and not a full de novo review, I would be remiss to not raise a few of the severe issues I see with the results that makes it hard for them to be convincing.
The Adam results in table 1 are far weaker than they should be, raising questions about the experiments as a whole. For example, https://arxiv.org/abs/2102.06356 reports 76.4% top 1 accuracy for ResNet-50 on ImageNet with Adam without increasing the epsilon parameter to a larger value as Choi et al. 2019 did (who also report good Adam results for ResNet-50 on ImageNet). This should also lead us to question one premise of the paper that there is some problem with adaptive optimizers for image classification.

Ok, but perhaps LAWN helps validation error even if there is no gap between SGD and Adam? Sadly, to demonstrate this subordinate claim, LAWN would have to be compared carefully with state of the art regularization techniques and compared with results that use any optimizer, not just Adam. With modern regularization techniques, it isn't hard to get 77%+ top 1 validation accuracy on ImageNet with ResNet-50. See, for example https://arxiv.org/abs/2010.01412v1 which gets 77.5% in 100 epochs and as high as 79.1 with longer training. Since LAWN is claiming to improve generalization, it must be compared with other regularization techniques. It is a type error to primarily compare it with optimizers so even if there weren't concerns with the performance of the existing baselines, there would need to be additional comparisons.

The claims about fixing issues that arise at large batch sizes are prima facie problematic since there isn't strong evidence of an actual problem at the batch sizes considered in the submission.
Decision: Reject