On the implicit minimization of alternative loss functions when training deep networks
Authors: Alexandre Lemire Paquin
Brahim Chaib-draa
Philippe Giguère
Venue:  (2020)
Keywords: implicit minimization
optimization bias
margin based loss functions
flat minima

Abstract: Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in the loss function.  Conversely, an interesting approach to gain more insights into the implicit bias could be to study how different loss functions  are being implicitly minimized when training the network. In this work, we concentrate our study on the inductive bias occurring when minimizing the cross-entropy loss with different batch sizes and learning rates.  We investigate how three loss functions are being implicitly minimized during training. These three loss functions are the Hinge loss with different margins, the cross-entropy loss with different temperatures and a newly introduced Gcdf loss with different standard deviations. This  Gcdf loss establishes a connection between a sharpness measure for the 0−1 loss and margin based loss functions. We find that a common behavior is emerging for all the loss functions considered.

Under review as a conference paper at ICLR 2020
ON THE IMPLICIT MINIMIZATION OF ALTERNATIVE LOSS FUNCTIONS WHEN TRAINING DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in the loss function. Con- versely, an interesting approach to gain more insights into the implicit bias could be to study how different loss functions are being implicitly minimized when train- ing the network. In this work, we concentrate our study on the inductive bias occurring when minimizing the cross-entropy loss with different batch sizes and learning rates. We investigate how three loss functions are being implicitly mini- mized during training. These three loss functions are the Hinge loss with different margins. the cross-entropy loss with different temperatures and newly intro- duced Gedf loss with different standard deviations. This Gedf loss establishes a connection between a sharpness measure for the 0 1 loss and margin based loss functions. We find that common behavior is emerging for all the loss functions considered.
INTRODUCTION
In the last few years, deep learning has succeeded in establishing state of the art performances in a wide variety of tasks in fields like computer vision, natural language processing and bioinformatics (LeCun et al., 2015). Understanding when and how these networks generalize better is important keep improving their performance. Many works starting mainly from Neyshabur al. (2015), Zhang et al. (017)) and Keskar et al. (2017) hint to a rich interplay between regularization and the optimization process of learning the weights of the network. The idea is that form of inductive bias can be realized implicitly by the optimization algorithm.
In this paper, we investigate the implicit bias induced from using different learning rates and batch sizes when minimizing the cross-entropy loss with SGD. A common theory is that more noise in the gradient bias the solution toward flatter minima (Keskar al. 2017). We draw connection between particular measure of flatness and margin based loss functions
Our contributions are the following:
1. A new loss function (Gedf loss) that can be interpreted measure of flatness for the 0 loss (for the top layer's weights of the network).
2. A methodology consisting in tracking alternative loss functions during training and compar- ing them for given training loss value to try to uncover implicit biases in the optimization algorithm applied to varying the learning rate and batch size in SGD.
3. Experimental results on CIFARIO and MNIST showing that larger learning rates and smaller batch sizes are better at implicitly minimizing the cross-entropy loss with larger temperature parameter, the hinge loss with larger margin parameter and the Gedf loss with larger standard deviation parameter. At the opposite, smaller learning rates and larger batch sizes are better at implicitly minimizing the cross- cross-entropy loss, the hinge loss and the Gedf loss with smaller values of their respective parameter.
'The concept of margin has been link generalization of deep networks; see for example Bartlett al. 2017, Poggio al. 2019) and Jiang al. (2019

---

Under review as a conference paper at ICLR 2020
We do not propose to modify optimization algorithms to try to improve large batch training but we instead try to offer new insights on how the solutions it produces are different from solutions resulting from small batch training (or larger learning rates). The hope is to eventually succeed at incorporating the inductive bias in the objective being optimized instead of relying on the implicit bias of the optimization algorithm. It is not yet clear what extent this goal can be realized (and by what m and we certainly do not claim to be reaching it. We offer only partial understanding of some of the differences between large batch training (or using small learning rates) and small batch training (or using large learning rates) through the behavior of alternative loss functions during training.
2 RELATED WORK
It was observed by Zhang et al.] (2017) that deep networks can often obtain good results with- out explicit regularization even if they have the capacity to essentially memorize the training set. They hypothesized that SGD is probably acting as an implicit regularizer. Also, the earlier work of Neyshabur et al. (2015) brought forward the idea that optimization might be implicitly biasing the trajectory toward low
models. Since then, many works have investigated the idea of implicit regularization for neural networks (linear or non-linear). For example, (Arora et al. (2019) studied how gradient descent finds low rank solutions for matrix completion with deep linear networks. Soudry et al. (2018) showed that gradient descent converges to the max-margin solution for logistic regression and Lyu & Li (2019) provides and extension to deep non-linear homogeneous networks. In contrast to these works, we study empirically how the optimization algorithm implicitly mini- mizes alternative loss functions during the course of training.
A highly studied source of implicit bias from the optimization algorithm is the ability to reach flatter minima. In Keskar et al.) (2017), the worst loss that can be obtained when slightly perturbing the parameters is considered as measure of sharpness while Neyshabur et al. (2017) considered the expected loss under Gaussian noise in the weights. We consider a measure of sharpness (section 3) similar to Neyshabur et al. (2017) and we apply it to the 0 loss directly instead of the usual surrogate cross-entropy loss.
The batch size and the learning rate are two ways to control the noise in the gradient which might influence the sharpness of the resulting solution (see for example /mith & Le (2018), |mmihh et al. 2018)). In conjunction with increasing the learning rate, different strategies like training for more epochs (Hoffer et al.) 2017). "warm up" (Goyal et al. (017) and using a separate learning rate for each layer based the norm of the weights (You al. (2017) have been proposed to improve the performance of large batch training. Instead of trying to offer a new modification to the optimiza- tion algorithm, we try here to capture the inductive bias into computationally efficient to loss functions in the hope of eventually simplifying the design of optimization algorithms.
GCDF LOSS
This section introduces a loss function based on the idea of flat minima. is defined as a measure of sharpness for the 0- - loss. The main motivation for introducing this loss function is that it simultaneously a measure of sharpness and margin based loss function establishing a clear relationship between these ideas. Furthermore, as opposed to the cross-entropy loss and the Hinge loss, is bounded and non-convex (see section [.1 for visual comparison). It thus offers more diversity to the loss functions investigated in this paper. We start with the binary linear case in [.] and then extend to the multi-class case in 3.2 For deep networks, this loss will be applied on the top layer. It is possible extension to our work to consider loss functions applied on multiple layers maybe in a similar fashion to Elsayed et al. (2018).
see for example Arora et al. 2019 about the difficulties capture the implicit bias of gradient descent with norms.
2

---

Under review as a conference paper at ICLR 2020
3.1 BINARY LINEAR CASE
Let f(w, 2) = w x + b, where w, x E R" and b E R. Consider the 0 1 loss for a binary linear where 1 the indicator function. Note that we will
classifier: L(f(w, x).9) = 1 ((w"x+b)=
write all the loss functions for single examples (z. y) throughout the paper and it will be understood that the training loss is obtained by taking the mean over the training set. We smooth (or "robustify") the 0 1 loss by considering
expectation under Gaussian noise in the weights. This loss function will then be denoted by T. (w, y) when the standard deviation is a. Consider the random variable =1) is a zero mean isotropic Gaussian distribution with covariance matrix o² 1. Since (w + e)) x + is distributed as Gaussian distribution with mean w w²x + b and variance |=21" we get that y((w + e) x + 6) is distributed as a Gaussian distribution with mean b) and the same variance. Therefore,
e (O, I21), where (0, o²1)
yw (wTx: x
Lo (w, v, y) = E.Lis(w + x) - Φ (y(w" + b))
(I)
a |||
where is the Gaussian cumulative distribution function (Gedf) given by
[ exp (x)) 22T -00
t2
dt.
(2)
If we assume that is normalized, the loss La is (decrcasing) function of (f(, (it is margin based loss function in the terminology from Lin (004) for example).
3.2 MULTI-CLASS CASE
Suppose the number of classes is m and now consider the affine mapping fWW 2) = Wa + with RR, b E Rm and W Rm For some fixed IR" and denoting by Wj the j row of W, let s, == w. x + bj be the corresponding score for class j. Finally, let sy(c,) (w, + +e,) be the perturbed score, an isotropic Gaussian random variable with mean and covariance matrix g² For given class y, we get
Jx x
bj
P(sy(Ev) F max maxx)(E)))
EP(ss(es) sy(Ev)}
(3)
jfy
Σ P(ss Sy > (cy - k,) xz}
(4)
jfy
8y
(5)
j
since (ey =T follows zero mean Gaussian distribution with variance 2o ||x|| We define
La (W,, = Σ
(6)
|
3F
This is upper bound the probability that the classifier does not predict under Gaussian noise on W. We will experiment with this Gedf loss function on top of feedforward neural networks (and also with other loss functions) in the following sections. In all the experiments, to enforce |||||| = 1 (this now represents the feature vector for the top layer).
normalization
4 IMPLICIT MINIMIZATION OF DIFFERENT LOSS FUNCTIONS
In this section, we track different loss functions while training deep neural networks with the cross- entropy loss varying the learning rates and batch sizes in SGD with momentum. The results in the main text are obtained while training CIFAR10. Results MNIST given in Appendix A The following loss functions are considered: cross-entropy with different values of temperature, Hinge loss with different margin parameters and the Gedf loss with different standard deviation parameters.
3

---

Under review as a conference paper at ICLR 2020
For the cross-entrope loss, the temperature T divides the scores s, before the softmax function. That is, the probability for class j is then given by
exp(s; (T)
Σn exp(sx/T)*
Remark that the positive homogeneity of the Relu implies that normalizing each layer of the network is equivalent to take T equal to the product of the norm of the layers. The cross-entropy loss after normalization the end of training is investigated in [iao al. (2018). In contrast, here multiple values for T and investigate the behavior during training. Given the probabilities for each class, the cross-entropy loss (on single example) is then the negative log probability for the correct class. For its part, the multi-class Hinge loss with margin parameter y (on single example) given by
consider
Σ max 7 + (5) sy))
jzy
The Gedf loss with standard deviation parameter a has been described and motivated in the previous section.
4.1 VISUAL COMPARISON OF THE LOSS FUNCTIONS
Assume that we have two classes and let = = Sy sj (for j * y). An example is correctly classified if > 0. The Gedf loss is then given by o (122)) ), the Hinge loss by max =} and the cross- entropy loss by log( exp( 7 +)) These functions are plotted in figure They share one interesting characteristic on the side 2 > O: when their parameter (o, T or 7) gets larger, the loss takes more time to get closer to zero when z increases. This kind of "heavier tail" behavior can encourage larger z values for some training points at the expense of other closer to zero training points more easily.
Croawerrougy
T=
(a)
(b)
(c)
Figure 1: In (a). we have the Gedf loss for different values of sigma. In (b), the Cross-entropy loss for different temperatures and in (c), the Hinge loss for different margins.
4.2 TRAINING CURVES OF THE ALTERNATIVE LOSS FUNCTIONS
In figures 2 and we investigate the effect of the size of the learning rate by considering the implicit training curves of the Gedf loss, the cross-entropy loss and the Hinge loss for different values of their respective parameter (o. T or 7). The learning rate is kept constant throughout training (no decaying schedule is used). We consider a small learning rate of 0.001 and a larger learning rate of 0.1. For the three loss functions considered, the larger learning rate clearly better at implicitly minimizing them for larger values of their parameter. A similar conclusion holds when considering different batch sizes as is shown in figures 5 13 (Appendix B and 14 (Appendix B In this case, the smaller batch size (256) is much better at implicitly minimizing the loss functions for larger values of their parameter than the larger batch size (16384). As a technical aside, note that ghost batch normalization Hoffer et al. (2017) is used when training with large batch sizes. The gradients are accumulated on a sequence of smaller weights.
of size 256 before updating the
4

---

Under review as a conference paper at ICLR 2020
25
lnes diffecent tempershurs
23
less differert perateruees sma
0.2
0.2
1.5
2.0
22
30 350
290
200
150
303
30
305
150
200
250
302
coech
e90th
(b)
Figure 2: Tracking the the cross-entropy loss with different temperatures while training with the standard cross-entropy loss (T = 1) on CIFAR1O. A relatively large learning rate of 0.1 is used in (a) while much smaller learning rate of 0.001 is used in (b). Even though both learning rates succeed at minimizing to almost zero the loss function they are trained on, the smaller learning rate does not implicitly minimize as well the cross-entropy loss for larger temperatures.
Hinge trve with margins vpesp sporh long
Hinge lons difleeen mangles veruu egorh unal
marge
marg
marge
magh
maigh
magg
marge
magn
margi
magn
22
30
120ch
#2ach
(a)
(b)
Figure 3: Tracking the Hinge loss with different margin parameters while training with the standard cross-entropy loss CIFARIO. A relatively large learning rate of 0.1 is used in (a) while much smaller learning rate of 0.001 is used in (b). Even though both learning rates succeed at minimizing to almost zero the loss function they are trained on, the smaller learning rate does not implicitly minimize as well the Hinge loss for larger margins.
4.3 ALTERNATIVE LOSS VERSUS ACTUAL TRAIN LOSS
At a given fixed training loss value two training runs have made the same progress toward minimiz- ing their objective function but they might not have made the same progress with respect other measures of performance. The other measures of performance considered here are of course our alternative loss functions. In figure we plot the Gedf loss against the actual train loss for different runs corresponding to different learning rates. We can see that smaller learning rates are actually better at minimizing the Gedf loss with smaller a during training while larger learning rates are better at minimizing the Gedf loss with larger There exists an intermediate value (here o = 1) where all the learning rates considered in our experiments are essentially equivalent implicitly minimizing the Gcdf loss. The train enror behaves similarly to the Gedf loss with small value of a. the binary case, the Gedf loss converges pointwise to the - loss almost everywhere (except at 0) when goes to O. It would therefore make sense actually define the Gedf loss for 0 to be the train error (in the binary case; some modifications are needed in the multi-class case). In this light, it is not surprising that figure 6a and 6d are showing similar behavior. In order make even more clear how different choices of learning rates are not implicitly minimizing the alternative loss functions for different parameters (a, T 7) in the same way, plotted the alternative losses
5

---

Under review as a conference paper at ICLR 2020
Gedf lmx with different
epach
Cedt loes difleeer
smalir sigma 0.5
sigma 0.5
10
33
sigma
0.3
30 350
290
200 250 302
30
205 150
200 250 302
Figure 4: Tracking the Gedf loss with different standard deviation parameters while training with the standard cross-entropy loss on CIFARIO. A relatively large learning rate of 0.1 is used (a) while much smaller learning rate of 0.001 is used in (b). Even though both learning rates succeed at minimizing to almost zero the loss function they are trained on, the smaller learning rate does not implicitly minimize as well the Gcdf loss for larger standard deviations.
Grdf Gedf with different dinaa versus epoc's for small bs
Grdf loss with di different s gamos veruus epoch for large bs
Nigma 0.5
Vigmm 0.5 sigma = sigma
sigmt =
sigma
1.4
2.0
250
302 335
(a)
(b)
Figure 5: Tracking the Gedf loss with different standard deviation parameters while training with the standard cross-entropy loss on CIFARIO. A relatively small batch size of 256 used in (a) while much larger batch size of 16384 used in (b). Even though both batch sizes succeed at minimizing almost zero the loss function they are trained on, the larger batch size does not implicitly minimize as well the Godf loss for larger values of a.
against their respective parameter for some fixed training loss value in figure 7 Similar results are obtained on MNIST (see figure 12 in Appendix [A). See also figure I6 (Appendix B for the results when considering different batch sizes CIFARIO.
5 DISCUSSION AND CONCLUSION
Suppose is distributed according to an isotropic Gaussian distribution with covariance matrix o² and mean Under second order approximation to the cross-entropy loss Le x. y) at w,
get E. Le(w v. s3 Lo(w, v. e² Tr(H), where H is the Hessian of Le 2, y). For
simplicity consider the binary case. Furthermore, since we restricted ourselves to loss functions applied the top layer only, assume that applied only to the weights of the top layer. In our setup, the Hessian H is now restricted to the weights of the final layer. Since the 0 - 1 loss is bounded above by the cross-entropy loss (times factor 1/ 1/l))(2)).
get
E, Lo(w + e, w) 1og(2)
~ Le 2r,
0²
Tog (w, xr S
Tr(H).
(9)
log(2)
2 log(22

---

Under review as a conference paper at ICLR 2020
loss sig mam0..3 versus train loss
4.-
Gcdf signam1 versus
0.001
0.001
4.-
0.01
4.-
0.01
35
1.0
35-
1.0
25-
23-
2.0
2.0
0.5
0.5
0.0
0.0
tram
(a)
(b)
45
Gcdf loss with sigma-B versus train loss
50
Train error versus train less for cross-entropy training
0.001
Ir 0.001
4.0
0.1
0.1
3.5
1.0
50
2.0
155
5 Fhord
20
1.0
2.0
25 80 0.5
1.5
2.0
(c)
(d)
Figure 6: For each value of the training loss (cross-entropy) achieved during training CIFARIO, we plot the Gedf loss at that time on the y axis. In (a), we use a smaller value of a = 0.5, in (b) an intermediate value of and in (c) larger value of a 8. The train error is plotted against the train loss in (d). Four training runs corresponding to four different learning rates are drawn in each case.
CEC
455
25
(a)
(b)
(c)
Figure 7: At given fixed cross-entropy training loss (here approximately 0.6 in cases), the Gedf loss for varying a's in (a), the cross-entropy loss with varying temperatures in (b) and the Hinge loss with varying y's in (c) are plotted. Larger learning rates obtain better values of the alternative losses for larger o ss, temperatures and 7's while smaller learning rates are generally better for smaller values of these parameters.
Therefore, optimization algorithm succeeding finding solution with small cross-entropy loss and small mean curvature the cross-entropy loss must have a small Gedf loss also. This might help to explain why larger learning rates and smaller batch sizes good at implicitly minimizing

---

Under review as a conference paper at ICLR 2020
the Gedf loss. Note however that this argument has some weaknesses. First, the approximation is only local and so might not be good for larger values of a. Second, it cannot explain why smaller learning rates and larger batch sizes are better for smaller values of a. Future work could concentrate on finding a rigorous explanation for these results.
Understanding the inductive biases of different optimization algorithms for training deep networks might allow to make the bias more explicit, that is to incorporate in the loss function. We think that one strategy make progress toward this long term goal might be to study how alternative loss functions are being implicitly minimized by a given optimization algorithm. This paper considered the learning rate and batch size parameters when training with SGD. clear avenue for future research is to extend the investigation to adaptive first-order methods (which can sometimes exhibit worse generalization performance than SGD (Wilson et al. 2017 and second-order methods.
REFERENCES
Sanjeev Arora, Naday Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. CoRR, 135//1905.1365, 2019. httpaapppp/aaa....r////aa//////a
13655
Peter I.. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaliized margin bounds for neural networks. In Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6240-6249, 2017. URL httt:::/papessssttpe www.aprrs.iipsee nips. wtc.pps.ccppppaae paper/
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pp. 1550-860 2018 networks for cww.oo---ssffaaaiii.o..
paper/736a 1113336---a------- marga-aa-eaainaeeeaaa
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet in 1 hour. CoRR, abs/1 706.00067,,7,, 2017. URL. httpp://arx.w..rg//bbb//// 02677
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 1731-1741, 2017.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019, URL https:// openreview. net/ forum htt:://rrmmiittoonnnne.t
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-bateh training for deep learning: Generalization gap and sharp minima. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https:/ /ttps:///peeeevieeeo / forttffoommm id=HloyR1ygg
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nature, 521((553))436 444, 2015. doi: 10. 10.8////e 14539. http:/cccp.////oo.oor///00111110
Qianli Liao, Brando Miranda, Andrzej Banburski, Jack Hidary, and Tomaso A. Poggio. A surprising linear relationship predicts test performance in deep networks. CoRR, abs/1807.09659, 2018.
09659
Yi Lin. A note basgin-based loss functions in classification. Statistics and Probabiliry Let- ters, 68( 1):73 - 82, 2004. ISSN 0167-7152. URL http: /tttt::/wwwir /www. scwencedirect. com/
8

---

Under review as a conference paper at ICLR 2020
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. CoRR, 1es/1900605890, 2019. http:t/////a/ii.ooo org/ hww.gg/bb/990../ abs/1906.05890
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of The 28th Conference on Learning Theory, COLT Paris, France, July 3-6, 2015, pp. 1376-1401, 2015. URL http:ttp::// htttp://proceedinggs ml wtt.igssss..pppreer.. prwsss/v4r/
Neyshabur html
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring gen- eralization in deep learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,
Long Beach, CA, USA. pp. 5947-5956, 2017. URL http://papers nips cc/paper/ in deep-learning learning
7176-
Tomaso A. Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks: Approximation, optimization and generalization. CoRR, N00/1908.093755 2019, URL http:
// wtw.ai..rg/aaa///aaaaoaaaaaaaoaa
09375
Samuel L. Smith and Quoe V. Le. A bayesian perspective on generalization and stochastic gradient descent. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings, 2018. URL https: /openzeview wttpe/oomm.dddiiiiea.e...
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL httpps::/ppeeeeeveen..../ net Eorumorriiio. forummid--B1yy1xxxx
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, and Nathan Srebro. The implicit bias of gradient descent on separable data. In 6th International Conference Learning Representations, ICLR 2018, Vancouver BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL wtt.tppss//ppeeeeeeeeee net
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4148-4158, 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. org. abs / 03888
CoRR, abs/1 708.03888, 2017. URL
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL bttp:tts::// https:://ppereeieeee wtwp://wwweetttttoeeeeedeeeeeeee
A RESULTS ON MNIST
This section contains the results when training 6-layer fully connected network with batch nor- malization on MNIST. No data augmentation is used. The optimization algorithm is SGD with momentum (0.9) and without weight decay. The learning rate is constant during all training.
B MORE RESULTS ON CIFARIO
This section contains additional results when training a convolutional network with batch normal- ization on CIFARIO. The network consists two convolutional layers with max pooling followed by 3 fully connected layers. No data augmentation is used. The optimization algorithm is SGD with momentum (0.9) and without weight decay. The leamnngg rate constant during all training.

---

Under review as a conference paper at ICLR 2020
Gool lasa citter sigmas versus epech for
Coal loss different versus epech smallr
Sigma 0.5
sigma 0.2
4.0
sigma
sigma
koue
2.0
360 250
252
(b)
Figure 8: Tracking the Gedf loss with different standard deviation parameters while training with the standard cross-entropy loss on MNIST. A relatively large learning rate of 0.1 is used in (a) while much smaller learning rate of 01.01 is used in (b).
Godf loss with differ zigmas versus opech for large
Ocdf loss with different signus versus opoch for large bs
sigma 0.1
sigma 0.1
sigma -1
sigma -1
sigma
sigme =
sigmt -
sigma
sigma
sigma
2.
Heaking
3.0
2.3
120th
(a)
(b)
Figure 9: Tracking the Gedf loss with different standard deviation parameters while training with the standard cross-entropy loss on MNIST. A relatively small batch size of 256 is used in (a) while a much larger batch size of 16384 is used in (b).
epoch
mangies
margi"
80
mangi"
mergh
12
300
129
250
50
300
150
250
e5eth
690th
(a)
(b)
Figure 10: Tracking the Hinge loss with different margin parameters while training with the standard cross-entropy loss on MNIST. A relatively large learning rate of 0.1 is used (a) while a much smaller learning rate of 0.001 is used in (b).
10

---

Under review as a conference paper at ICLR 2020
2g
entrapy lons
drfierent lemperatures versus epoch lange
Cap entropy icas difterent per res versus upach
1.6
1.6
17
T
coech
138ch
(a)
(b)
Figure 11: Tracking the the cross-entropy loss with different temperatures while training with the standard cross cross-entropy loss (T = 1) on MNIST. A relatively large learning rate of 0.1 is used in (a) while a much smaller learning rate of 0.001 used (b).
COCL
(a)
(b)
(c)
Figure 12: At a given fixed cross-entropy training loss (here approximately 0.08 in all cases), the Gedf loss for varying a's in (a), the cross-entropy loss with varying temperatures in (b) and the Hinge loss with varying 7's in (c) are plotted. Larger learning rates obtain better values of the alternative losses for larger o's, temperatures and >'s while smaller learning rates generally better for smaller values of these parameters. The training dataset is MNIST.
Hinge loss with ferrrnt margin versus eaoch for small by
Hinge lass wit difforent margits varsus opoch for ge
maigi maggi
magg
magh 8
maigi
magn
maigi"
magn
margi =5
magn =
10
120ch
(a)
(b)
Figure 13: Tracking the Hinge loss with different margin parameters while training with the standard cross-entropy loss CIFARIO. A relatively batch size of 256 is used in (a) while much larger batch size of 16384 is used in (b). Even though both batch sizes succeed at minimizing almost zero the loss function they trained on, the larger batch size does not implicitly minimize well the Hinge loss for larger margins.
11

---

Under review as a conference paper at ICLR 2020
enbrapy with
Cigy eroropy
n-4
0.8
0.8
bocp
15
Ueinnng boocs
2.3
3.0
2.2
0.5
0.0
30 320 230
200
250
302
30
202
130
200
250
302
s28th
120ch
(b)
Figure 14: Tracking the cross-entropy loss with different temperature parameters while training with the standard cross-entropy loss (I' 1) on CIFARIO. 10. A relatively batch size of 256 used in (a) while much larger batch size of 16384 is used in (b). Even though both batch sizes succeed at minimizing to almost zero the loss function they are trained on, the larger batch size does not implicitly minimize as well the cross-entropy loss for larger temperatures.
2014
2014
2014
(a)
(b)
(c)
Figure 15: For each value of the training loss (cross-entropy) achieved during training on CIFARIO, we plot the Gedf loss at that time on the y axis. In (a), we use a smaller value of a = 0.5, in (b) an intermediate value of three different batch sizes drawn in each case. Larger batch sizes slightly better at minimizing the Gedf loss with small a during training while smaller batch sizes are better at minimizing the Gedf loss with larger o. There exists an intermediate value (here 1) where all the batch sizes considered our experiments are essentially equivalent at implicitly minimizing the Gcdf loss.
and (c) a larger value
8. Three training runs corresponding to
Gcel 20 333
"Iigo foo PETTE Perign
(b)
(c)
Figure 16: At given fixed cross-entropy training loss (here approximately 0.6 in all cases), the Gedf loss for varying o's in (a), the cross-entropy loss with varying temperatures (b) and the Hinge loss with varying 7's in (c) are plotted. The small batch size obtain better values of the alternative losses for larger a' 's temperatures and y's while the large batch size is better for smaller values these parameters. The training dataset is CIFARIO.
12


Reviews:
[{"review_id": "ryx18TZ6dr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper want to show that minimizing cross-entropy loss will simultaneously minimize Hinge loss with different margins, cross-entropy loss with different temperatures and a newly introduced Gcdf loss with different standard deviations. The main contribution is a new gcdf loss based on Gaussian-perturbed parameters. However, this loss can only be used with linear models. For deep models, the authors suggest that only measure this loss on the top layer of model.\n\nThe motivation is week. Seems most of these loss functions only depend on s_i - s_j, the difference between logits. And the optimization with cross-entropy loss wants to maximize this difference between logits corresponding to true labels and false labels, which is obviously minimize difference loss functions. So I do not feel surprise that optimizing the neural network with cross-entropy loss will minimize other kinds of losses.\n\nThe format is poor. The figures occupy most of the places and thus let me feel the contents of this paper is somewhat weak.\n\nDetailed Comments:\n1. In Sec 3.1, it will be better to mention that y belongs to +1 and -1. Also, in the last sentence in Sec 3.1, the meaning of x is normalized is ambiguous. I guess the authors want to say if x is unit norm?\n2. Will adding the regularization of the feature map norm violate the performance?\n3. What do the authors want to say in Sec 4.3? Does the relation of learning rate and convergence rate of gcdf loss indicate some non-trivial results?\n4. I cannot understand the meaning of Figure 7. Obviously different learning rate may lead to different training process and thus the different solution and different s_i - s_j. But I think this is not related to the different losses. With simple calculation I think we can find all of these losses have some relation with s_i - s_j and thus we can directly say different learning rate lead to different s_i - s_j and there is no need to relate s_i - s_j to these losses.\n\nOverall I find the claims of this paper is somewhat weak. Gcdf loss seems related to some kinds of adversarial robustness that can be individual interest, but the current paper is still far from the standard of publication. Some more interesting and valuable directions can be the theoretically analysis of the equivalence between the objective optimization, e.g. minimizing cross-entropy loss is equivalent to minimizing gcdf loss for example, as well as the empirical comparison between the different losses, like selecting some losses as objective can boost the performance on some aspects.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "ryeeKOZatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adds to a large body of research on implicit regularization:\nin the context of deep networks, that, when SGD with a certain step\nsize or batch size is used, from among parameter vectors that fit\nthe data equally well, some are much more likely to be chosen than\nothers.  This paper attacks this question through the lens of\nloss functions: when SGD is applied to the usual softmax loss\nwith a given learning rate, how does the choice of learning\nrate effect how rapidly other loss functions are reduced?  They\npay special attention to a loss function that they call Gcdf,\nwhich is motivated by \"wide minima\" considerations.  In particular,\nfor the Gcdf to be small, not only must training examples be\nclassified correctly, but randomly perturbing the weights in\nthe last layer should not change this correct prediction.\n\nI am convinced that Figures 6 and 12 of this paper show\nthat optimization of the softmax with a larger step size\nimplicitly optimizes a loss function that rewards robustness\nto a greater extent than when a small step size is used.\nI find this interesting.\n\nThe authors do a nice job of summarizing a lot of related work.\n\nThe Gcdf loss is similar to the ramp loss used in [1]\n(see Section 3.1) and elsewhere, including to analyze\ngeneralization in deep learning.  It is also like the\npotential function optimized by RobustBoost\n(see (4) of [2]) -- the RobustBoost loss function does\nnot scale by the norm of x, but since it is an ensemble\nmethod, the role of x is played by the predictions of\nmembers of the ensemble, which have a fixed scale.\n\nI assume that, when they evaluate the Gcdf loss for a deep\nnetwork, they normalize by the norm of the last hidden\nlayer.  If this is true, it only captures \"wide minima\"\nin the sense of being robust with respect to perturbations\nof the output layer.  (They seem to acknowledge this point\nin their paper.)\n\nThe results in the paper are not described in enough\ndetail to be reproduced.  For example, I don't see\nwhere they specify the architecture of the network\nthat they used in Section 4.\n\nThe experiments are limited and narrow in scope.\n\nIn Figures 2 and 3 I don't see that they have adequately controlled\nfor the effect of the learning rate on how fast the explicitly\nminimized loss is reduced.  Part of the effect observed is simply\nthat, when a small learning rate is used, after a given number of\nepochs, the weights are just not changed much, so that no loss is\nreduced much.  In Figure 2, I find it strange that they did not plot\nthe values for T=1.\n\nFigure 6 is the most interesting to me.  It seems to show that\ntraining the same loss function with a larger learning rate\neffectively optimizes the gcdf loss that rewards sacrificing\ntraining error to achieve stronger robustness.  \n\nSmall point: the first time I read Appendix A, I thought that the\nresults were not there.  It was only later that I saw the figures with\nthe MNIST results.  It would be helpful if the authors wrote\n\"The results are in Figures 8-12\".\n\nWhile, as I wrote above, I did find Figure 6 interesting, I feel that\nthe increment of this research over the large body of previous work\non this topic is not enough to justify publication in ICLR.\n\n\n\n\n[1] Bartlett, Peter L., Dylan J. Foster, and Matus\nJ. Telgarsky. \"Spectrally-normalized margin bounds for neural\nnetworks.\" Advances in Neural Information Processing Systems. 2017.\n\n[2] https://arxiv.org/pdf/0905.2138.pdf\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1lQ53WCKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes a step towards understanding of the implicit bias of optimization algorithms in deep learning. The authors consider alternative loss functions for deep networks: (1) the temperature-scaled cross-entropy loss with different values of the temperature; (2) the hinge-loss with different values of the margin parameter; (3) the Gcdf loss with different values of the variance parameter. The paper introduces the Gcdf loss which is derived as a modification of the 0-1 loss under the noise in the parameters of the linear output layer. The authors propose to use the alternative losses as measures of margin and sharpness associated with a solution found by an optimization algorithm. The experiments show how SGD in different learning scenarios (low/high learning rate and small/large batch) performs implicit minimization of the alternative loss functions with different parameters. Specifically, using larger learning rates/smaller batch sizes is shown to implicitly minimize the losses corresponding to higher values of the temperature/margin/variance. The results provide insights about margins and sharpness of solutions found by different modes of SGD.\n\nThe direction explored in the paper is important for the understanding of the connections between optimization, properties of the loss landscapes (such as sharpness), and generalization. The results reported in the paper are interesting. However, currently I am not convinced that the contributions are sufficient for publication at ICLR as the scope of the performed analysis is limited. In my view, the study is not comprehensive enough and the paper would benefit from incorporating additional results. \n\nDetailed comments:\n\n\n1) My main concern is that currently there is very little explanation provided for the observed experimental findings. The paper would strongly benefit from additional results focused on identification and verification of the mechanisms behind the observed behavior of the optimizer. \n\n2) Many connections mentioned in the paper are left unexplored. It would help to investigate the mentioned connections between the implicit minimization of the considered losses and sharpness, curvature, and generalization. A similar design of the experiment can be used in which the alternative loss values can be tracked alongside with the validation loss (or multiple losses) as well as the measures of sharpness and the characteristics of the Hessian.\n\n3) Another direction for improvement is the extension of the set of analyzed settings (as it was mentioned in the discussion section). This includes performing the analysis for a broader set of architectures (potentially with different normalization schemes), optimizers, and choices of the hyperparameters (momentum, weight decay). These experiments would help to better understand the observed phenomenon and analyze the effect of different settings.\n \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting setting in which the effect of different optimization parameters on the loss function is analyzed.  The analysis is based on considering cross-entropy loss with different softmax parameters, or hinge loss with different margin parameters.  The observations are interesting but ultimately the reviewers felt that the experimental results were not sufficient to warrant publication at ICLR.  The reviews unanimously recommended rejection, and no rebuttal was provided.
Decision: Reject