Demystifying Loss Functions for Classification
Authors: Simon Kornblith
Honglak Lee
Ting Chen
Mohammad Norouzi
Venue:  (2021)
Keywords: softmax
classification
representational similarity
transfer learning
label smoothing
dropout

Abstract: It is common to use the softmax cross-entropy loss to train neural networks on classification datasets where a single class label is assigned to each example. However, it has been shown that modifying softmax cross-entropy with label smoothing or regularizers such as dropout can lead to higher performance. In this paper, we compare a variety of loss functions and output layer regularization strategies that improve performance on image classification tasks. We find differences in the outputs of networks trained with these different objectives, in terms of accuracy, calibration, out-of-distribution robustness, and predictions. However, differences in hidden representations of networks trained with different objectives are restricted to the last few layers; representational similarity reveals no differences among network layers that are not close to the output. We show that all objectives that improve over vanilla softmax loss produce greater class separation in the penultimate layer of the network, which potentially accounts for improved performance on the original task, but results in features that transfer worse to other tasks.

Under review as a conference paper at ICLR 2021
DEMYSTIFYING Loss FUNCTIONS FOR CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
It is common to use the softmax cross- coos--y loss to train neural networks on classification datasets where single class label is assigned to each example. How- ever, it has been shown that modifying softmax cross-entropy with label smoothing or regularizers such as dropout can lead to higher performance. this paper, we compare variety of loss functions and output layer regularization strategies that improve performance on image classification tasks. We find differences the outputs of networks trained with these different objectives, in terms of accuracy, calibration, distiiibbitittoonn robustness, and predictions. However, differences hidden representations of networks trained with different objectives are restricted to the last few layers; representational similarity reveals no differences among network layers that are not close the output. We show that all objectives that im- prove over vanilla softmax loss produce greater class separation in the penultimate layer of the network, which potentially accounts for improved performance on the original task, but results in features that transfer worse to other tasks.
INTRODUCTION
Softmax cnosspyn (Bridle 1990a b) is the canonical loss function for multi-class classification in deep learning. However, the popularity of softmax cross-entropy appears to be driven by the aesthetic appeal of its probabilistic interpretation, rather than by practical superiority. Early studies reported no empirical advantage of softmax cross-entropy over equared-error loss (Richard & Lippmann] 1991] Weigend 1993; Dietterich & Bakiri 1994), and more recent work has found other objectives that yield better performance
certain tasks (e.g. SSzegedy et al.) 2016; Liu et al. 2016] Beyer et al. 2020). These studies show that is possible to achieve meaning ful improvements in accuracy simply by changing the loss function. Nonetheless, there has been little comparison among these alternative objectives, and even less investigation of why some objectives work better than others.
In this paper, perform comprehensive empirical study of the properties of 9 common and less-common loss functions and regularizers for deep learning, on standard image classification benchmarks. Most existing work in this area has proposed attempted to demonstrate its superiority over a limited set of alternatives on benchmark tasks. This approach creates strong incentives demonstrate the superiority of the proposed loss and little incentive to understand its limitations. Our goal is instead to understand when one might want to use one loss function or regularizer over another and, more broadly, to understand the extent which neural network performance and representations can be manipulated through the choice of objective alone. Our key contributions are as follows:
loss function regularizer and
We rigorously benchmark 9 training objectives standard image classification tasks, measuring accuracy, calibration, and out-of-distribution robustness. Many objectives improve over vanilla softmax cross-entropy loss, but no single objective performs best on all benchmarks.
We demonstrate that different loss functions and regularizers produce different patterns of pre- dictions. but combining them does not appear to improve accuracy. However, regularization that affects the input, such as AutoAugment (Cubuk et af., 2019) and Mixup (Zhang et al. 2017). can provide further gains. Our best models achieve state-of -tte-e-rtttt-tttt accuracy (79.1 %794.5% top- top-1/topp 1/top-5) on ImageNet for unmodified ResNet-50 architectures trained from scratch.
Using centered kernel alignment (CKA), we measure the similarity of the hidden representations of networks trained with different objectives. We show that the choice of objective affects representations in network layers close to the output, but earlier layers are highly similar regardless of what loss function is used.

---

Under review as a conference paper at ICLR 2021
We show that all objectives that improve accuracy softmax cross-entropy also lead greater separation between representations of different classes in the penultimate layer. This improvement in class separation may be related to the boost in accuracy these objectives provide. However, representations with greater class separation are also more heavily specialized for the original task, and linear classifiers operating these features perform substantially worse transfer tasks.
2 LOSS FUNCTIONS AND OUTPUT LAYER REGULARIZERS
We investigate 9 loss functions and output layer regularizers.Le. & E RK denote the network's s output ("logit") vector, and let t {0, 1 denote a one-hot vector of targets, where ||t|| 1. Let RRM denote the vector of penultimate layer activations, which gives rise to the output vector as Wx + b, where W R the matrix of final layer weights, and b is a vector of biases.
All investigated loss functions include term that encourages l to have high dot product with t. To avoid solutions that make this dot product large simply by increasing the scale of l, these loss functions must also include one or more contractive terms and/or normalize t. Many "regularizers" correspond to additional contractive terms added to the loss, so we do not draw a firm distinction between losses and regularizers. We describe each loss in detail below. Hyperparameters provided in Appendix A.
Softmax cross-entropy (Bridle 1990a b is the de facto loss function for multi-class classification in deep learning. It can be written as:
K
cx Σ et,
K -
Looftmax (l,t) = Σ ty log
tule
log Σ
(I)
k=1
k=1
The loss consists of term that maximizes the dot product between the logits and targets, as well as contractive term that minimizes the LogSumExp of the logits.
Label smoothing (Szegedy al. 2016) "smooths" the targets for softmax cross-entropy loss. The new targets are given by mixing the original targets with a uniform distribution over all labels, t' t'=tx x (1 (1-0) a) + a/ K, where a determines the weighting of the original and uniform targets. In order to maintain the same scale for the gradient with respect to the positive logit, in our experiments,
we scale the label smoothing loss by 1/(1 a). The resulting loss
cla
Somooth (L, t; =
Σ
E(1
a.tt +
log
a
K
Σ
K
el
K
K
tuly +
log
4
K
k=
k=1
Compared softmax cross- cross-entropy loss, label smoothing adds an additional term that encourages the MMlliir et al. (2019) previously showed that label smoothing improves calibration
logits to be positive. Muüller and encourages class centroids to lie at the vertices of a regular simplex.
Dropout (Srivastava et al.) 2014) is among the most prominent regularizers in the deep learning literature. We consider dropout applied to the penultimate layer of the neural network, i.e., when inputs to the final layer randomly kept with some probability p. When employing dropout, we replace the penultimate layer activations x with ã = x @ E/p where & Bernoullipp.. Writing the dropped out logits as L. W + b, the dropout loss is:
Caropout (W, b. x. t:p) = Eg Cootimax (e,
(4)
Dropout produces both implicit regularization, by introducing noise into the optimization process, and explicit regularization, by altering the representation that minimizes the loss (Wei et al. 2222)).
Wager et al. (2013) have previously derived quadratic approximation to the explicit regularizer for logistic regression and other generalized linear models; this strategy can also be used to approximate the explicit regularization imposed by dropout the penultimate layer of a neural network with softmax loss. However, we observe that penultimate layer dropout has similar effects to extra final layer 2 regularization, suggesting that implicit regularization is the more important component.
Extra final layer L regularization: It is common to place the same L regularization on the final layer elsewhere in the network. However, we find that applying greater 1.2 regularization the
2

---

Under review as a conference paper at ICLR 2021
final layer improve performance. In architectures with batch normalization, adding additional L regularization has no explicit regularizing effect if the learnable scale (7) parameters that are unregularized, but still exerts implicit regularizing effect by altering optimization.
Logit penalty: Whereas label smoothing encourages logits not to be too negative, and dropout imposes penalty on the logits that depends the covariance of the weights, alternative possibility is simply to explicitly constrain logits to be small in I,² norm:
(5) Logit normalization: We consider the use of L normalization, rather than regularization, of the logits. Because the entropy of the output of the softmax function depends on the scale of the logits, which is lost after normalization, we introduce additional temperature parameter that controls the magnitude of the logit vector, and thus, indirectly, the minimum entropy of the output distribution:
Slogit_penatie (f, 3) Loottmax (f.t) )) 3||e||?
Llogit_norml (e, t; T) Lsoftmax (ofoftma((e/(Tlllle (f/(TI)e))),
Cosine softmax: We additionally consider 2 normalization of both the penultimate layer features and the final layer weights corresponding to each class. This loss equivalent to softmax cross- entropy loss if the logits given by cosine similarity sim(x,, y) x //( (1 weight vector and the penultimate layer plus a per-class bias:
between the
K
K
Fccos_ooftmmas W, b, x. x,))
t (sim Wk.: x3/T bb) + log
sim ().../)/++b
where a temperature parameter as above. Similar losses have appeared in previous literature Ran- Jan et al. 2017 Wojke & Bewley [2018] Wang ct al. 20188 b Deng et al. 2019; Liu et al. 2017). and variants have introduced explicit additive or multiplicative margins to this loss that we do not consider here (Liu et al.) 2017] Wang et al. 2018a (b] Deng et al. (2019). It is possible that performance could be enhanced by employing one of these margin schemes, although we observe that manipulating the temperature alone has large impact on observed class separation.
Sigmoid cross-entropy is the natural analog to softmax cross-entropy for multi-label classification problems. Although we investigate only single-label multi-class classification tasks, we train networks with sigmoid cross-entropy and evaluate accuracy by ranking the logits of the sigmoids. This approach is related to the one-versus-rest strategy for converting binary classifiers to multi-class classifiers. The sigmoid cross-entropy loss is:
e²* c+ +1
+ (1 - (1)) -14)log log (1 (1-
ch cix+i) -
el
Esigmold (f,ttolde,,))..)) =
Lx log
(8)
K
K
tule +
Σ
log(e +1).
(9)
A=
The Log term softmax loss is replaced with the sum the cetpllusttrasformee logits. We initialize the biases of the logits b to log(K) that the initial output probabilities are approx- imately 1/K. Beyer et al. (2020) have previously shown that sigmoid cross-entropy loss leads to improved accuracy on ImageNet relative to softmax cross-entropy.
Squared error: Finally, we investigate squared error loss, as formulated by Hui & Belkin 2020
Eaquared_errod (L, t; K; M)
K Σ (kla(le M)² + ())
(10)
where x and M are hyperparameters. A sets the strength of the loss for the correct class relative to incorrect classes, whereas M controls the magnitude of the correct class target. When x M = 1, the loss is simply the mean squared error between & and t. Like Hui & Belkin (2020), we find that placing greater weight on the correct class slightly improves ImageNet accuracy.
RESULTS
For each loss, trained 8 ResNet-50 (He et 2016 Gross & Wilber 2016) models on ImageNet. al. To tune loss hyperparameters and the epoch for early stopping, we performed 3 training runs per hyperparameter configuration where we held out validation set of 50,046 ImageNet training example.
3

---

Under review as a conference paper at ICLR 2021
Table 1: Regularizers and alternative losses improve ImageNet accuracy. Accuracy of models trained with different losses/regularizers on the ImageNet validation (mean + standard error of models) and CIFAR- and CIFAR-100 test sets (mean + standard error of 25 models). Losses are sorted from lowest to highest ImageNet top-1 accuracy. Accuracy values not significantly different from the hest (p > 0.05, t-test) are bold-faced.
ImageNNct (ResNct-50)
CIFAR-10
CIFAR- 100
Loss/regularizer
Top-1 Acc. (%)
Top-5 Acc. (%)
(AIl-CNN-C BN)
(WRN 16-8)
Softmax
77.0 0.06
93.40 + 0.02
93.49 +: 0.03
79.7 + 0.04
Squared error
77.2 0.04
92.79 + 0.02
93.31 + 0.02
79.4 = 0.05
Dropout
77.5 0.04
93.62 0.02
98.74 0.08
79.5 0.06
Label smoothing
77.6
98.78
93.79 0.03
8.05
Extra final layer Z
77.7
93.79
93.63 1 0.03
80.2 00220000
Logit penalty
93.83 0.02
93.84
80.2 0.05
Logit normalization
0.02
08.71 0.02
93.55
78.9 0.05
Cosine softmax
77.9 7.092
93.86 0.01
80.1 - 0.06
Sigmoid
77.9 0.05
93.50 + 0.02
93.79 0.04
80.0 = 0.05
We also trained 25 batch-normallied All-CNN-C (Springenberg al. 2014) models for each loss
on CIFAR-I0 (Krizhevsky & Hinton 2009), where we performed extensive hyperparameter tuning for learning rate and weight decay addition loss hyperparameters. We provide further details regarding training and hyperparameter selection in Appendix A.I
3.1 REGULARIZERS AND ALTERNATIVE LOSSES ENIIANCE ACCURACY
We found that, when properly tuned, many investigated objectives often provide statistically significant improvement over softmax cross-entropy, as shown in Table The range of improvements was small, but meaningful, with sigmoid cross-entropy and cosine softmax both leading to an improvement of 0.9% in top-1 accuracy over the baseline for ResNet-50 on ImageNet. No single loss performed best across all benchmarks, although cosine softmax, logit penalty, and sigmoid were frequently among the top-performing losses.
Losses that yielded large improvements top-1 accuracy on ImageNet did not necessarily improve top-5 accuracy. For ResNet-50, sigmoid cross-entropy led to large (0.9%) improvement in top-1 accuracy over vanilla softmax cross-entropy, but only a small (0.1%) improvement top-5 accuracy. Cosine softmax performed comparably to sigmoid cross-entropy in terms of top-1 accuracy, but better in top-5 accuracy, with a 0.4% improvement over the baseline. Similar patterns were observed for Inception v3 (Table B.I). where sigmoid cross-entropy top-1 accuracy but performed worse than the softmax baseline in terms of top-5 accuracy.
the rest-performing model in terms of
Losses also differed in out-of-distribution robustness, and in the calibration of the resulting predictions. Table|B.2|shows results on the out-of of-distribution test sets ImageNet-v2 (Recht et al. 2019). ImageNet-
A (Hendrycks et al. 2019), ImageNet-Skette Sketch (Wang et al., 2019), ImageNet-R R (Hendrycks et al.) 2020), and ImageNet-CC (Hendrycks & Dietterich 2019). In almost all cases, alternative loss functions outperformed softmax cross-entropy, with logit normalization and cosine softmax typically performing slightly better than alternatives. Effects on calibration, shown in Table B.3 were mixed. Label smoothing substantially reduced expected calibration error (Guo et al.) 2017). ), previously shown by Müller et al.) (2019). although cosine softmax achieved a lower negative log likelihood. However, there was no clear relationship between calibration and accuracy. Although logit penalty performed well in terms of accuracy, it provided the worst calibration of any objective investigated.
Our attempts achieve higher accuracy by combining objectives were unsuccessful. As described in Appendix C. adding additional regularization did not improve performance of well-tuned loss functions, and normalized variants of sigmoid cross-entropy loss failed to improve accuracy on ImageNet. However, it was still possible to improve networks' performance substantially using AutoAAument (Cubuk et al. 2019) or Mixup (Zhang et al.) 2017). and gains from improved losses and these data augmentation strategies were approximately additive (Table C.2) With longer training, both sigmoid cross-entropy and cosine softmax achieve state-of of-te-of-th-arr accuracy among ResNet-50 networks trained with Autmmnt (Table C.3). matching outperforming supervised contrastive learning (Khosla et al.) 2020). Combining cosine softmax loss, AutoAugment, and Mixup, we achieve 79.1% top-1 accuracy and 94.5% top-5 accuracy, which is to our knowledge the best reported 224 x 224 pixel single-crop accuracy with an unmodified ResNet-50 architecture trained from scratch.
4

---

Under review as a conference paper at ICLR 2021
Softmax
&
Percent matching pradetions
Label Smosthing
Sigmold
&
More Final Layer
Drapout
R
Logit Penalty
Loglt Normalization
Cosine Softmax
Squared Error
Softma
Label Smoothing
sigmolo
More Final Layer LB
Dropout
Logit Penalty
Logtt Loght Nomalization
Cosine Softmoo
Squared Erar
Squared Emor
Cosine Softman
Logt Normelllaaen lzztiozzioo
Softman
Logit Penally
Dropout
More Final Layer La
Sigmald
Label Smoothing
Figure Different losses produce different predictions. a: Percentages of ImageNet validation set examples
for which models assign the same top-1 l predictions, for seeds of ResNet-50 models. b: Dendrogram based on similarity of predictions. All models naturally cluster according to loss, except for "Dropout" and "More Final Layer L2" models. See also Figure D.1
Percent activations
100
56x56
28x28
14x14
Pool
Softmax
Label Smoothing
Logit Norm
80
Squared Error
More L2
Cosine Soltmax
Drapout
Logit Penalty
Sigmoid
40
20
ResNet Block
Figure 2: Loss functions affect sparsity of later layer representations. Plot shows the average % non-zero activations for each ResNet-50 block, after the residual connection and subsequent nonlinearity, on the ImageNet validation set. Dashed lines indicate boundaries between stages.
3.2 DIFFERENT LOSSES PRODUCE DIFFERENT PREDICTIONS
Given that effects of regularization were non-additive, we sought to determine whether different regularizers and losses had similar effects on network predictions. For each pair of models, we measured the percentage of images in the ImageNet validation set where both models predicted the same class. The results are shown in Figure We also examined the percentage images that where both models either correct incorrect, and the agreement examples that both models get incorrect (Figure D.I). All ways of measuring similarity of predictions yielded similar results.
Models' predictions clustered into distinct groups according their loss functions. Models trained from different initializations with the same loss function were more similar than models trained with different loss functions. However, all models trained with (regularized) softmax loss sigmoid loss were more similar to each other than they were to models trained with logit or feature + weight normalization. Networks trained with squared error were dissimilar all others examined.
Variability in predictions of models trained with the same loss but different random initializations was large. Although standard deviations in top-1 accuracy similar pair of models disagreed on 13.9% of test set examples. When ensembling the 8 models trained with the same loss but different random initializations, the least similar losses (softmax and squared error) disagreed on only 11.5% of examples (Figure D.2). The accuracy of ensembles of models trained with different losses was closely related to the accuracies of the constituent models; ensembling models trained with the two best losses yielded only modest accuracy improvements over ensembles trained with either loss alone (Figure D33,
<0.2% for all losses, the most
3.3
LOSSES PRIMARILY AFFECT HIDDEN REPRESENTATIONS CLOSE TO THE OUTPUT
Loss functions differ not only in their predictions, but also in their effects on internal representations of neural networks. In Figure 2 we show the sparsity of the activations of layers of networks trained with different loss functions. In all networks, the percentage of non-zero ReLU activations decreased with depth, attaining its minimum the last convolutional layer. In the first three ResNet stages, activation sparsity was broadly similar regardless of the loss. However, in the final stage and penultimate average pooling layer, there were substantial differences.
5

---

Under review as a conference paper at ICLR 2021
Softmax Softmax
Label Smoothing vs. Softmax
Squared Error Softmax
1.0
Softmex Res Block
Block
0.8
Similarit (CKA))
6
Re
0.8
10
10
12
12
14
14
16
16
Avg.
Logis
0.0
6 8
10 12
2
10 12 14 16
2 6 10
12
14 16
ResNet Block
Label Smoothing ResNet Black
Squared ResNet Block
b
Block
Block 4
Block 7
Block
Block 13
Label Smoothing
More Final Layer L2
Dropout
0.8
Logit Penalty
Logit Normalizaiinn
Similariy (CKAA)
Cosine Saftmax
Squared Emart
0.6
Block 14
Block 15
Block 16
Avg. Pool
Logits
Softmax
Labal Smopthing
Sigmoid
More Final Layer L2
Dropout
Logit Peralty
0.2
Logit Normalizaion
Cosine Saftmax
Squared Eror
Softmo
Label Smaothig
Sigmok
More Final Lay
Dropsut
Logit Panally
Lagit Normaliz at
Cosine Softmas Squaree redd Error
Soft
Labal smo sttting
Sigmor
More Findl Layrer L2 Dropsun
Logit Normeliz melization Softmas Eiror
Softme
Label Sme Sooktin
More Final Soare
Drapout
Loght Peñalty
Logit Normalt ptormllrattinn attonn
Cosind Softmas
Squared Erro
Labet Seechi Sigmor
More Final Lay e L2 Diopoun
Logit Panally Logit Nor malization Cosine Softmex
Squarez Emor
Softmit
Labal smo gitting
Sigmoo
More Layer
Logit Dropade
Normelt stlizzttiion
Softmal
Cossine Softmise
Squared Enor
0.0
Figure 3: The loss function has little impact on representations in early network layers. All plots show linear centered kernel aligmment (CKA) between representations computed the ImageNet validation set. CKA between network layers, for pairs of networks trained from different initializations. b: CKA between rep- resentations extracted from architecturally corresponding layers networks trained with different loss functions. Diagonal reflects similarity of networks with the same loss function trained from diffcrent initalizations.
Given that these observations, we wondered whether the choice of loss had any effect on representa- tions in these layers at all. We used linear centered kernel alignment (CKA) (Kornblith al. 2019a
Cortes et al. 2012; Cristianini et al.) 2002) to measure the similarity between networks' hidden representations. As shown in Figure representations of corresponding early, but not late, network layers were highly similar regardless of loss function. These results provide further confirmation that effects of the loss function are limited later network layers.
3.4 REGULARIZATION IMPROVES CLASS SEPARATION
there a feature of the investigated regularizers that can potentially explain their beneficial effect on accuracy? We demonstrate that all investigated regularizers and alternative losses force the network to shrink or eliminate directions the penultimate layer representation space that are not aligned with weight vectors. The universality of this finding suggests may relate the accuracy-enhancing properties of these losses.
The ratio of the average within-class cosine distance the overall average cosine distance provides measure of how distributed examples within a class are that is between 0 and 1. We take one minus this quantity get a closed-form measure of class separation:
ΣK No No (1 sim( Fk,n)) /N
R2 =
(11)
Σ EK EN
Ne m= (1 sim(x xk,n)) (N)Ngs)
where Db.m is the embedding of example m in class k, N₁ the number of examples in class k. and sim (x, y) = x' y/ ( [|>||])
(|]) is cosine similarity between vectors. If the embeddings are first L2 normalized, then 1 R² is the ratio of the average within-class variance to the weighted total variance, where the weights are inversely proportional to the number of examples in each class. For a balanced dataset, R² is also equivalent to centered kernel alignment (Cortes et al. 2012 Cristianini ct al., 2002) between the embeddings and the one-hot label matrix, with a cosine kernel. We also examined alternative class separation metrics (Appendix)) B results were similar.

---

Under review as a conference paper at ICLR 2021
Table 2: Regularization and alternative losses improve class separation the penultimate layer. Results averaged over 8 ResNet-50 models per loss on the Image- Net training set.
56x58
28x28
14x14
7x7
Pool
1.0
Logit Penalty
Squared Error
Logit Normalization
Dropout
Cosine Softmax
Clase saparation (R
Label Smoothing
Sigmoid
More Final Layer L2
Loss/regularizer Class separation (H²")
Softmax
0.3494 0.0002
Squared error
0.8452 + 0.0002
Dropout
0.4606 + 0.0003
Label smoothing
0.4197 + 0.0003
0.2
Extra -
0.5718 0.0006
Logit penalty
0.6012 = 0.0004
0.0
2
10
12
14
Logit norm
0.5167 + 0.0002
ResNet Block
Cosine softmax
0.6406 = 0.0003
Figure 4: Class separation in different layers of ResNet-50 models, the ImageNet training set.
Sigmoid
0.4267 0.0003
Table 3: Regularized networks learn features specialized to ImageNet. Accuracy of linear classifiers (L²- regularized multinomial logistic regression) trained to classify different datasets using fixed penultimate layer features. IN(50k) reflects accuracy of classifier trained 50,046 examples from the ImageNet training set and tested on the validation set. See Appendix A.2 for training details.
Pretraining loss
Food
CIFAR1O CIFAR100 Birdsnap SUN397
Cars
Pets
Flowers
IN(50k)
Softmax
74.6
92.4
76.9
55.4
62.0
60.3
92.0
94.0
71.1
Squared error
39.8
82.2
56.3
21.8
39.9
15.3
84.7
46.7
76.7
Dropout
72.6
91.4
75.0
53.6
61.2
54.7
92.6
92.1
74,8
Label smoothing
72.7
91.6
75.2
53.6
61.6
54.8
92.9
91.9
74.5
Extra Z
91.0
73.7
60.1
50.3
92.4
89.8
75.9
Logit penalty
68.1
90.2
72.3
48.1
59.0
48.3
92.3
86.6
76.4
Logit norm
66.3
90.5
72.9
50.7
58.1
45.4
92.0
82.9
75.1
Cosine softmax
62.0
89.9
71.3
45.4
55.0
36.7
91.1
75.3
76.9
Sigmoid
73.4
91.7
75.7
52.3
62.0
92.5
92.9
74.3
As shown in Table 2 and Figure 4 all regularizers and alternative loss functions resulted in greater class separation in penultimate (average pooling) layer representations compared to softmax loss. Whereas additional final layer T.2, logit penalty, and squared error also produced greater class separation before the penultimate layer, other losses did not.
Although losses that improve class separation also improve accuracy on the ImageNet validation set, they result in penultimate layer features that are substantially less useful for other tasks. Kornblith al. (2019b) previously showed that networks trained with label smoothing and dropout learn less transferable features. As in this work, trained logistic regression classifiers classify
selection transfer datasets (Bossard et al.) 2014:) Krizhevsky & Hinton 2009 Berg et al.) 2014
Xiao al. 2010 Krause al. [2013] Parkhi et al. Nilsbjk & Zisserman 2008 using fixed features from networks trained with different losses. As shown in Table features from networks
trained with vanilla softmax loss yield the highest transfer accuracy. However, when attempted to relearn the original 1000-way ImageNet classifier using 50,046 training set examples, features from networks trained with vanilla softmax loss performed worst. Thus, the ease with which ImageNet classifier weights can be relearned from representations is inversely related to the performance of these representations when they are used to classify other datasets (Figure
To confirm this relationship between class separation, ImageNet accuracy, and transfer, we trained models with cosine softmax with varying values of the temperature parameter Table 4 lower temperatures resulted in lower top-1 accuracies and worse class separation, and made the ImageNet classifier weights more difficult to recover. However, even though the lowest temperatuu temperature achieved 2.7% lower accuracy ImageNet compared higher temperatures, this lowest temperature yielded the better features for nearly all transfer datasets. Thus, controls tradeoff between the generalizability of penultimate-lay features and the accuracy the target dataset.
shown in
'Training at low temperatures unstable, scaled the loss by the temperature, which slightly worsened overall ImageNet accuracy. Relationships for temperatures 0.05 remain consistent without loss scaling.
7

---

Under review as a conference paper at ICLR 2021
Figure Transfer accuracy and accuracy relearned ImageNet weights are nega- tively related. a: Average transfer task ac- curacy versus accuracy of a classifier trained 50,046 ImageNet training set examples and tested on the validation set for different objectives. b: Relationship of transfer accu- racy and relearned ImageNet accuracy with cosine softmax temperature.
Mean Trarefer Acc. (M4)
@Softmax Sigmooid abe Smooth
b
Layer L
75
70-
Logit Normlk
Toatt Panant
Accuracy (86)
Gosine Softriaxe
70
60-
65
Moan Transfer Acc.
50-
Squared Eyooe
60
Welghts
76
0,02
0.04
0.06
0.08
of Releamed
Temperature
ImageNet Weights (9))
Table 4: Temperature of cosine softmax loss controls ImageNet top-1 accuracy, class separation (R²), and linear transfer accuracy.
NetgeNet
Temp.
Top-1
R²
Fond
CI CIFAR100 Birdsnap
SUN397
Cars
Pets
Flowers
IN(50k)
0.01
74.9
0.236
73.4
91.9
76.5
57.2
60.5
62.9
91.7
93.6
67.0
0.02
77.0
0.358
72.1
91.8
76.2
56.5
60.4
58.5
92.2
91.2
70.8
0.03
77.5
0.475
69.1
91.5
74.9
53.7
59.1
51.8
92.3
87.4
74.3
0.04
77.6
0.562
66.0
90.7
73.8
50.3
57.4
45.1
91.7
82.2
75.7
0.05
77.6
0.634
62.8
90.4
72.2
47.6
55.4
38.6
91.0
78.3
76.4
0.06
77.5
0.693
60.3
89.3
69.8
43.3
53.8
33.3
91.0
72.7
76.6
0.07
77.5
0.738
88.7
68.6
39.6
51.4
29.1
90.2
67.9
0.08
77.6
0.770
53.7
87.7
66.5
35.5
49.4
25.7
89.3
63.2
77.0
4 RELATED WORK
Theoretical analysis of loss functions is challenging; most cases, solutions cannot be expressed in closed form even when the predictor is linear. However, Soudryy et al. (2018) have previously shown that, on linearly separable data, gradient descent on the unregularized logistic or multinomial logistic regression objectives (i.e., linear models with sigmoid or softmax cross-entropy loss) eventually converges to the minimum norm solution. These results can be extended to neural networks in certain
restricted settings (Soudry al. 2018 Gunasekar et al. 2018 Wei et al. 2019).
Our study of class separation in penultimate layers neural networks related to work investigating angular visual hardness (Chen et al.) 2019)), which measures the arccosine-transformee similarity between the weight vectors and examples. This metric is similar to the class separation metric we apply (Eq. ITI. but fails to differentiate between networks trained with softmax and sigmoid cross-entropy; see Appendix Figure E. Other work has investigated how class information evolves through the hidden layers of neural networks, using linear classifiers (Alain & Bengio 2016), binning estimators of mutual information (Shwartz-Ziv & Tishby| 2017: Saxe et 2019 Goldfeld et al. 2018), Euclidean distances (Schilling et al., 2018), and manifold geometry (Cohen et al. 20200). However, this previous work has not analyzed how training objectives affect these measures.
cosine
The loss functions we investigate are only a subset those explored in past literature. We have excluded loss functions that require specially constructed batches from the current investigation (Snell
et al. 2017 Khosla et al. 2020). as well as losses designed for situations with high label noise (Jindal
al. 2016 Ghosh et al. 2017| Patrini et al.) [2017] Amid et al. 2019: Lukasik et al. (2020). Other
work has investigated replacing the softmax function with other functions that lead to normalized
class probabilities (de Brébisson & Vincent] 2015 Laha et al. (208).. Our approach is related to previous studies of metric learning (Musgrave et al. 2020) and optimizers (Choi et al. 2019).
5 CONCLUSION
Our study identifies many similarities among networks trained with different objectives. On CIFAR- 10, CIFAR-100. and ImageNet, different losses and regularizers achieve broadly similar accuracies. Although the accuracy differences are large enough to be meaningful in some contexts, the largest is still <1.5%. Representational similarity analysis using centered kernel alignment indicates that the choice of loss function affects representations in only the last few layers of the network, suggesting inherent limitations to what can be achieved by manipulating the loss. However, we also show that different objectives lead to substantially different penultimate layer representations. We find that class separation an important factor that distinguishes these different penultimate layer representations, and show that it is inversely related to transferability of representations to other tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: vrXiv:1610.01644 2016.
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. In Advances Neural Information Processing Systems, pp. 15013-15022, 2019.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine- lie-grained visual categorization of birds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2019-2026. IEEE, 2014.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? arXiv preprint arXiv:2006.07159, 2020.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference Computer Vision, pp. 446 Springer, 2014.
John S Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227-236. Springer, 1990a.
John S Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In Advances neural information processing systems, pp. 211-217, 1990b.
Beidi Chen, Weiyang Liu Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, and Anima Anandkumar. Angular visual hardness. arXiv preprint arXiv: 19/2.0229, 2019.
Dami Choi. Christopher J Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arX1v:1910.05446, 2019.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):955-828, 2012.
Nello Cristianini, John Shawe- Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. In Advances in neural information processing systems, pp. 367-373, 2002.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 113-123, 2019.
Alexandre de Brébisson and Pascal Vincent. An exploration of softmax alternatives belonging to the spherical loss family. arXiv preprint arXiv: 1,50042, 2015.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
Thomas G Dietterich and Ghulum Bakiri. Solving multiclass learning problems via crror-correcting output codes. Journal of artificial intelligence research, 2:263-286, 1994.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. arXiv preprint arXiv: 1712.0982.. 2017.
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. arXiv preprint arXiv: 1810.00700522 2018.

---

Under review as a conference paper at ICLR 2021
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint wtXiv:/706.026// 2017.
Sam Gross and Michael Wilber. Training and investigating residual nets. The Torch Blog. http://torch. wttccchcccco/ococccc oo0o 6/ 02/ hwtrr///rresssee///eee html, 2016.
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp. 9461-9471, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07111 07799077077744... 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of ditto---istribttion generalization. arXiv preprint arXiv:2006. 16241, 2020.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy classification tasks. arXiv preprint arXiv:2006.07322 2020.
Ishan Jindal, Matthew Nokleby, and Xuewen Chen. Learning deep networks from noisy labels with dropout regularization. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 967-972. IEEE, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 1162,, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint 105.0:1905.00414. 2019a.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019b.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. In Second Workshop on Fine-Grained Visual Categorization, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https:
cs http://wwwww..tooooocooo .orunto.edu/
#kriz/learning wtw.eaannnninnaeeeeeeeeeeeeeeeee
features-220 2009- TR.
Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankara-
narayanan, and Harish G Ramaswamy. On controllable sparse alternatives to softmax. In Advances in Neural Information Processing Systems, pp. 6422-6432, 2018.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, volume 2, pp. 7, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu. Ming Li. Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
10

---

Under review as a conference paper at ICLR 2021
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.028// 2020.
Rafael Müller, Simon Korblithh, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. arXiv preprint arXiv:2003.08505, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Computer Vision, Graphics & Image Processing, 2008. ICVGIP'08. Sixth Indian Conference on, pp. 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3498-3505. IEEE, 2012.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 1944 1952, 2017.
Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L.2-constrained softmax loss for discrimina- tive face verification. arXiv preprint arXiv:1703.09507, 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv: 2019.
Michael D Richard and Richard P Lippmann. Neural network classifiers estimate bayesian posteriori probabilities. Neural computation, 3(4):461 -483, 1991.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Achim Schilling, Claus Metzner, Jonas Rietsch, Richard Gerum, Holger Schulze, and Patrick Krauss. How deep is deep quught?-quantfyiing class separability in the hidden layers of deep neural networks. arXiv preprint arXiv:1811.0175: 2018.
Ravid Shwartz--Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1tXXiv:/03..00 770.0008/0. 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1): 2822-2876 2018
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox. and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6886, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In Advances in neural information processing systems, pp. 351-359, 2013.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926-935 2018a.

---

Under review as a conference paper at ICLR 2021
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5265-5274, 2018b.
Haohan Wang, Songwei Ge, Eric Xing, and Zachary C. Lipton. Learning robust global representa- tions by penalizing local predictive power, 2019.
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, pp. 9712-9724, 2019.
Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. arXiv preprint 12Xiv:200../2915, 2020.
Andreas Weigend. On overfitting and the effective number of hidden units. In Proceedings of the 1993 connectionist models summer school, volume 1, pp. 335-342, 1993.
Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re-identification. 2018 IEEE winter conference applications of computer vision (WACV), pp. 748-756. IEEE, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and IEEE, 2010.
Pattern Recognition (CVPR), pp.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, 2016. ISBN 1-901725-59-6.
doi: 10.5244/C.30.87. URL https?
77dx. doi
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
12

---

Under review as a conference paper at ICLR 2021
Appendix
A DETAILS OF TRAINING AND HYPERPARAMETER TUNING
A.1 TRAINING AND TUNING NEURAL NETWORKS
ImageNet. We trained ImageNet models (ResNet-50 (He et al. 2016 Gross & Wilber 2016; Goyal
al. 2017 .v..5 0 and Inception v3 SZzeedy et al. 2016) models with SGD with Nesterov momentum of 0.9 and a batch size 4096 and weight decay of 8 x 10 (applied to the weights but not batch norm parameters). After 10 epochs of linear warmup
maximum learning rate of 1.6, we decayed the learning rate by a factor of 0.975 per epoch. We took an exponential moving average of the weights over training in (nZeeedy al. 2016) with a momentum factor of 0,9999. We used standard data augmentation comprising random crops of 10-100% of the image with aspect ratios of 0.75 to 1.33 and random horizontal flips. At test time, we resized images to 256 pixels on their shortest side and took a 224 x 224 center crop.
To tune hyperparameters, we initially performed a set of training runs with wide range of different parameters, and then narrowed the hyperparameter range to the range shown in Table A.1 To further tune the hyperparameters and the epoch for early stopping, we performed 3 training runs per configuration where we held out validation set of approximately 50,000 ImageNet training examples, We tuned loss hyperparameters for ResNet-50 only. For Inception v3, we used the same loss hyperparameters for ResNet-50, but we still performed 3 training runs with the held out validation set to select the point at which to stop for each loss.
Table A.1: Hyperparameters for ImageNet.
Loss/regularizer
Hyperparameters
Epochs
Softmax
N/A
146
Squared error
9, M 60, loss scale 10
196
Dropout
P (0.6, 0.65, 0.7, 0.75, 0.8, 0.85]
172
Label smoothing
Q = {0.08, 0.09, 0.1, 0.100.1001
180
Extra final layer 7,2
Aimal = (4e-4, 6e-4, 8e-4, 1c-3)
168
Logit penalty
3 (5e-5, 1e-4, 2e-4, 4e-4, 6e-4, 8e-4]
180
Logit normalization
{0.03, 0.04, 0.05, 0.06)
152
Cosine softmax
0..04, 0.05, 0.06, 0.07, 0.08)
158
Sigmoid
N/A
166
CIFAR. We trained CIFAR-10 and CIFAR- models using SGD with Nesterov momentum of 0.9 and a cosine learning rate decay schedule without restarts, and without weight averaging. For CIFAR-10, we used batch size of 128; for CIFAR-100, we used batch size of 256. For these networks, we performed hyperparameter tuning to select the learning rate and weight decay parameters. We started by selecting the learning rate from {10-2, 2 10 1.5 10 10 0.5 1.0, 100.5] and the weight decay from (10-1. 10-4, 10 3.5 10-3), where we parameterize the weight de- 10-3.5
cay so that it is divided by the learning rate. We manually inspected hyperparameter grids and expanded the learning rate and weight decay ranges when the best accuracy was on the edge of the searched grid. After finding the best hyperparameters in this coarse search, we performed a finer search in the vicinity of the best coarse hyperparameters with double the granularity, e.g., for an optimal learning rate of 10 in the coarse search, our fine grid would include learning rates of {10 0.5 10 0.75 10 10 1.25 10 1.53 }. All results show optimal hyperparameters from this finer
"The terchvis s si ResNet-50 model and the "official" TensorFlow ResNet both implement this architec- ture, which was first proposed by Gross & Wilber (2016) and differs from ResNet described by He et al.) (2016) in performing strided convolution in the first 3 x convolution in each stage rather than the first 1 x convolution. Our implementation initializes the parameters of the last batch normalization layer in each block
0, in Goyal et al. (2017).
2 "Due to the large number of hyperparameter configurations, for squared error, we performed only I run per configuration to select hyperparameters, but 3 to select the epoch at which to stop. We manually narrowed the hyperparameter search range until all trained networks achieved similar accuracy. The resulting hyperparaameters
performed better than those suggested by Hui & Belkin 20020
13

---

Under review as a conference paper at ICLR 2021
grid. During both coarse and fine hyperparameter tuning, computed accuracies averaged over different initializations for each configuration to reduce the bias toward selecting veriivarancc hy- perparameter combinations when searching are shown Table A.2
large number of configurations. Hyperparameters
The architecture used for CIFAR-10 experiments based AlI-CNN-C architecture of Springenberg et al. (2014), with batch normalization added between layers and the global average pooling operation moved before the final convolutional layer. On CIFAR-100, we used the Wide
ResNet 16-8 architecture from Zagoruyko & Komodakis (016). Our CIFAR-100 architecture applied weight decay batch normalization parameters, but our CIFAR- architecture did not.
Table A.2: Hyperparameters CIFAR. 7) is learning rate and is product of the learning rate and the weight decay added the loss, i... the weight decay loss is dacay
2ny ||w
Loss/regularizer
CIFAR-IO (AII-CNN-C BN)
CIFAR- 100 (WRN 16-8)
Squared error
" 0.1, 10-3.0 , 8, M 0.83
0.1,
10 3.75
= 6, M 12
Softmax
" 10-0.75 10
7 0.1,
10
Logit normalization
0.01, == 0.14
10-2.25
=
0.11
Extra final layer L
" 0.1,A=10 11005...,, Atnal
10 1.3
= 0.1,
-1 3.75
Amml = 10 3333
3333
Cosine softmax
" 10-2.25 1=22255=11 10 10
0.08
0.01, Á 10
T 0.1
Dropout
" 0.1, À = 10 3.70
/ 0.65
10 10 1.25 ; ;}= 10
3.75 0.75
Sigmoid
" = L, 10 3.76
= 0.1, = 10 3.75
Label smoothing
7 0.1, X 10
0.04
0.1, À 10
0.18
Logit penalty
10 0.75
Ā 10
3.76
= - 2.83
10 1.25 x = 10 3.75
10
2.83
A.2 TRAINING AND TUNING MULTINOMIAL LOGISTIC REGRESSION CLASSIFIERS
To train multinomial logistic regression classifiers on fixed features, we follow a similar approach to Kornblith et al. (2019b). We first extracted features for every image in the training set, by resizing them to 224 pixels on the shortest side and taking a 224 x 224 pixel center crop. We held out validation set from the training set, and used this validation
select the 1 regularization hyperparameter, which we selected from 45 logarithmically spaced values between 10-6 and 105, applied to the sum of the per-example losses. Because the optimization problem convex, we used the previous weights as a warm start as we increased the L regularization hyperparameter. After finding the optimal hyperparameter on this validation set, we retrained on the entire training set and evaluated accuracy on the test set.
B ADDITIONAL EVALUATION OF PERFORMANCE OF REGULARIZERS AND LOSSES
Table B.1: Regularizers and alternative losses improve Inception v3 accuracy on ImageNet. Accuracy (mean + standard error of models) with different losses/regularizers on the ImageNet validation set. Losses are sorted from lowest highest top-1 accuracy. Accuracy values not significantly different from the best 0.05, t-test) bold-faced.
Loss/regularizer
Top-1 1 Acc. (%)
Top-5 Acc. (%)
Squared error Softmax
77.7 + 0.03
93.28 + 0.01
78.6 + 0.03
94.24 + 0.03
Logit normalization
78.8 + 0.11
94.34 0.04
Label smoothing
78.8 + 0.03
94.60 0.03
Cosine softmax
78.9 + 0.06
94.38 0.03
Logit penalty
78.9 + 0.06
94.63 0.02
Dropout
79.0 + 0.02
94.50 0.04
Extra final layer L
79.0 + 0.03
94.52 # 0.01
Sigmoid
79.1 + 0.07
94.17 0.02
14

---

Under review as a conference paper at ICLR 2021
Table B.2: Regularizers and alternative losses improve performance on out-of-distribution test sets. Ac- curacy averaged over ResNet-50 models per loss.
Loss/regularizer
ImageNet-v2 (%)
ImageNet-A
IN-Sketch
ImageNet-R (%)
ImageNet-C (mCE)
(%)
(%)
Softmax
65.0 0.1
2.7 0.0
21.8 + 0.1
36.8 + 0.1
75.9 + 0.1
Squared emror
65.3 1 0.1
1.5
22.4 0.1
36.3 + 0.1
74.6 0.1
Dropout
65.4 0.0
3.1
2.1
37.2
74.5 0.1
Label smoothing
65.7 0.1
3.1
2.1
37.
75.2 0.1
Extra final layer L
65.8
3.0
23.1
3.1
74.1
Logit penalty
6.0
4.5
2..8
38.1
7.1
Logit normalization
65.8 65.8
4.8
23.
39.2
0.1
Cosine softmax
65.8 #01
4.6
24.8
38.7
7.1
Sigmoid
0.1
3.0
2.1
3..6
0.1
Table B.3: Regularizers and alternative losses may or may not improve calibration. We report negative log likelihood (NLL) and expected calibration error (ECE) for each loss on the ImageNet validation set, before and after scaling the temperature of the probability of the distribution to minimize NLL, in Guo al. 2017) ECE computed with evenly spaced bins. For networks trained with sigmoid loss, we normalize probability distribution by summing probabilities over all classes.
Uncalibrated
With temperature scaling
Loss/regularizer
NLL
ECE
NLL
ECE
Softmax
0.981 0.002
0.073 = 0.0001
0.917 = 0.002
0.027 + 0.0004
Dropout
0.971 = 0.002
0.074 0.0009 0.0009
0.905 = 0.002
0.031 + 0.0002
Label smoothing
0.947 0.001
0.016 0.0007
0.941 = 0.001
0.044 + 0.0004
Extra final layer 2
0.976 0.002
0.081 1081=0000 0.0003
0.908 = 0.002
0.038 + 0.0006
Logit penalty
1.041 0.001
0.995 = 0.001
0.055 0.0004
Logit normalization
0.965 0.001
0.069 + 0.0002
0.949 + 0.001
0.049 +.00000
Cosine softmax
0.912 0.002
0.066
+ 0.0006
0.895 = 0.002
0.043 0.0008
Sigmoid
0.944 0.002
0.044 + 0.0003
0.914 = 0.002
0.019 0.0002
Table B.4: Training accuracy of ResNet-50 models.
Loss/regularizer
Top-1 Acc. (%)
Top-5 Acc. (%)
Sofumax
93.61 + 0.01
99.33 + 0.002
Squared error
91.65 + 0.01
98.59 + 0.002
Dropout
92.25 0.01
99.03 = 0.003
Label smoothing
93.62
99.43 = 0.007
Extra final layer 2
91.62
98.85 = 0.003
Logit penalty
0..01
99.13 = 0.002
Logit normalization
92.86 0.01
99.01 + 0.003
Cosine softmax
02.47+000
98.75 + 0.004
Sigmoid
93.22 + 0.01
99.19 + 0.002
15

---

Under review as a conference paper at ICLR 2021
C RESULTS OF COMBINING REGULARIZER//LOSEE
Table C.I: Combining final-layer regularizers and/or improved losses does not enhance performance. ImageNet holdout set accuracy of ResNet-50 models when combining losses and regularizers between models. All results reflect the maximum accuracy on the holdout set at any point during training, averaged across training runs. Accuracy numbers higher the holdout set than the official ImageNet validation set. This difference accuracy likely due difference image distributions between the ImageNet training and validation sets, as previously noted Section C.3.1 of Recht al. (2019).
Baseline
Label smoothing (a 0.1)
Sigmoid
Cosine softmax (T = 0.05)
Bascline
79.9
80.4
80.6
80.6
Dropout (3 = 0.7)
80.3
80.3
80.3
80.2
Dropout (3 0.8)
80.2
80.4
80.4
80.4
Dropout (3 = 0.9)
80.3
80.5
80.6
Dropout (3 = 0.95)
80.4
80.6
80.7
Logit penalty (7 = 5 x 10 6)
80.4
80.3
80.5
80.6
Logit penalty (? x 10
80.4
80.3
80.5
80.5
Logit penalty (? x 10
80.4
80.3
80.4
80.5
Logit penalty (7 4 x 10 10-4)
80.4
80.2
80.3
80.5
Logit penalty (7 = x 10 10-4)
80.5
80.2
80.3
80.5
Logit normalization (r 0.02)
80.4
Logit normalization (T = 0.03)
80.3
80.6
Logit normalization (r 0.04)
80.4
80.6
Logit normalization (r = 0.05)
80.3
80.5
Logit normalization (T -..66)
80.3
80.5
Cosine normalization (r 0.045)
80.6
80.5
Cosine normalization (T 0.05)
80.6
80.6
Cosine normalization (T 0.06)
80.4
75.3
Table C.2: AutoAugment and Mixup provide consistent accuracy gains beyond well-tuned losses and regularizers. Top-1 accuracy of ResNet-50 models trained and without AutoAugment, averaged over 3 (with AutoAugment) or 8 (without AutoAugment) runs. Models trained with AutoAugment use the loss hyperparameters chosen for models trained without AutoAugment, but the point at which to stop training was chosen independently our holdout set. For modcls trained with Mixup, the mixing parameter a is chosen from [0.1, 0.2, 0.3, 0.4] holdout set. Best results in each column, as well as results insignificantly different from best (p > 0.05, t-test), are bold-faced.
Standard Augmentation
AutoAugment
Mixup
Losshregularizer
Top-1 1 (%)
Top-5 (%)
Top-1 (%)
Top-S (%)
Top-1 (%)
Top-5 (%)
Softmax
77.0 1 0.06
93.40 1 0.02
77.7 1 0.05
93.74 1 0.05
78.0 0.05
93.98
0.03
Sigmoid
77.9 + 0.05
93.50 0.02
78.5 1 0.04
93.82 1: 0.02
78.5 1 0.07
93.94 1 0.04
Logit penalty
77.7 + 0.02
93.83
78.3 0.05
94.10 0.03
78.0
93.95 + 0.05
Cosine softmax
77.9 = 0.02
03.86
18.3 0.02
94.12 0.04
78.4 0.04
94.14 0.02
16

---

Under review as a conference paper at ICLR 2021
Table C.3: Comparison with state-of-the-art. All results are for ResNet-50 models trained with Augonugment. Loss hyperparameters are same as Table C22 the learning schedule decays exponentially at rate of 0.985 per epoch, rather than 0.975 per epoch. This learning rate schedule takes approximately 2x as many epochs before it reaches peak accuracy, and provides a ~~0.4% improvement in top-1 accuracy across settings.
Loss
Epochs
Top-1 (%)
Top-5 (%)
Softmax CCbuuk et al. 2019) Supervised contrastive
270
77.6
93.8
Khosla al. 2020)
700
78.8
93.9
Ours:
Softmax
306
77.9 + 0.02
93.77 + 0.03
Sigmoid
324
78. 0.04
93.96 = 0.06
Logit penalty
346
78.6 0.07
94.30 0.01
Cosine softmax
308
78.7 0.04
94.24 = 0.02
Ours (with Mixup):
Sigmoid
384
79.1 1 0.06
94.28 + 0.03
Cosine softmax
348
79.1 + 0.09
94.49 + 0.01
D SIMILARITY OF MODEL PREDICTIONS
8 Percent examples where both
Softmax
8
Percent matchingg predictions
Age durcementt on examples both both models get incoreett
Label Smoothing
Sigmold
Final Layer
Dropout
Logit Penaly
Normalization
Coelne Softmax
8
Squared
Softmax
Label smoothing
Sigmold
More Finell Layer L22
Dropout
Logit Perally
Loght Nomallzatio malizattion
Cosine Softmax
Squared Emoi
Softmax
Label Smooiting
More Sigmoid
Dropout
Logit Penalt
Lagit Nomalizztion
Cosine Softmax
Softmax
Labd Smootting
Sigmoid
Dropout
Logit Penall
Logit Nomalzzatinn
Cosine Softmax
Squared Enoi
More Finel Layer
Figure D.I: Different ways of measuring similarity of single-model ResNet-50 predictions yield similar
qualitative results. See also Figure
both
Softmax
Agreement on examples both models ged lineerrect
Label Smoothing
82.5
Sigmeid
95
More Final Layer L2
91
Percent maiching
Dropout
Logit Penalty
94
Logit Normaizztiio
77.5
Nor
Percent
Squared Eroi
75.0
Softmea
Label Smosthing
Sigmoid
Dropout
Logit Penally
Logit Normelization
Feature Wt Norm Normi
Squared Error
Softmen
Label Smoathing
Sigmold
More Final Layer L2
Dropous
Penalty
Logit Normaltaation
Festure Wt Norm
Squuared Enor
Softmexi
Label Smoothing
Sigmoid
More Final Layer L.2
Dropout
Logh Panalty
Logit Nornalization
Feature Wi Norm)
Squaared Ererr
Layer
More
Figure D.2: Ensemble predictions are substantially more similar than single-model predictions. Predic-
tions of the ensemble were computed by taking ResNet-50 models trained from different random initializations
with the same loss and picking the most common top-1 prediction for cach example.
17

---

Under review as a conference paper at ICLR 2021
Softmax
Label Smoothing
80.0
Sigmoid
More Layer L2
79.6
Dropout
Logit Penalty
79.4
Logit Normalization
79.2
Cosine Softmax
Squared Emor
79.0
Softmox
Label Smoothing
Sigmalo
More Final Layer L2
Drapout
Logit Penalty
Logit Lagit Normalzation
Cosine Softmax
Squared Emor
Figure D.3: Ensembling models trained with different losses provides only modest performance benefits. Ensembles consist of ResNet-50 models, half of which are trained with the objective on the x-axis, the other half with the objective the y-axis. The ensemble prediction is the modal class prediction of the 8 models.
OTHER CLASS SEPARATION METRICS
Table E.1: Comparison of class separation under different distance metries. Cosine (mean-subtracted) subtracts the mean the activations before computing the cosine distance. All results reported Re on the ImageNet training set.
Loss/regularizer
Cosine
Cosine (mean- subtracted)
Euclidean distance
Softmax
0.3494 + 0.0002
0.3472 0.0002
0.3366 0.0002
Squared error
0.8452 + 0.0002
0.8450 + 0.0002
0.8421 + 0.0007
Dropout
0.4606 + 0.0003
0.4559 + 0.0002
0.4524 + 0.0003
Label smoothing
0.4197 = 0.0003
0.4124 + 0.0004
0.3662 + 0.0005
Extra final layer 2
0.5718 + 0.0006
0.5629 + 0.0005
0.5561 + 0.0005
Logit penalty
0.6012 = 0.0004
0.5950 0.0004
0.5672 + 0.0004
Logit normalization
0,5167 = 0.0002
0.5157 = 0.0002
0.5326 + 0.0002
Cosine softmax
0.6406 = 0.0003
0.6389 0.0003
0.6406 + 0.0003
Sigmoid
0.4267 + 0.0003
0.4315 + 0.0003
0.4272 + 0.0003
Softmax (77.0%)
0.7
Squared (77.2%)
Dropout (77.5%
Label Smoothing
More Final Layer L2 (77.7%)
Density
0.5
Logit Penalty (77.7%)
0.4
Logit Normalizaiion (77.8%)
Cosine Softmax (77.9%)
Sigmoid (77.9%)
0.2
0.0
8
Angular Visual Hardness (x110 '
Figure F.1: Angular visual hardness of different loss functions. Keree density estimate of the angular visual hardness (Chen et al. 2019) scores of the 50,000 examples the ImageNet validation set, computed with a Gaussian kernel of bandwidth 5 x 10 for ResNet-50 networks trained with different losses. Legend shows ImageNet top-1 accuracy for each loss function in parentheses. Although alternative loss functions generally reduce angular visual hardness softmax loss, sigmoid loss does not, yet is tied for the highest accuracy of any loss function.
18

---

Under review as a conference paper at ICLR 2021
Penultimate Layer Activations
1.0
Cunulative Varlance Explaihed
0.8
Softmax (77.0%)
Squared Error(77.2%)
Dropout (77.5%)
0.6
Label Smoothhng (77.8%)
More Finsl Layer L2 (77.7%)
0.4
Lagit Penalty (77.7%)
Logit Nomalization (77.8%) (77.8%)
Cosine Softmax (77.9%)
0.2
Sigmoid (77. (77.9%)
0.0
500
1000
1500
2000
Number Dimensions
Final Layer Weights
Class Centroids
1.0
1.0
Cunulaaivv Variance Explained
Cumuulatvv Variance Explained
0.8
0.8
0.6
0.8
0.4
0.4
0.2
0.2
0.0
0.0
200
400
600
800
1000
200
400
600
800
1000
Number Dimensions
Number of Dimensions
Figure E.2: Singular value spectra of activations and weights learned by different losses. Singular value spectra computed for penultimate layer activations, final layer weights, and class centroids of ResNet-50 models on the ImageNet training set. Penultimate layer activations and final layer weights fail differentiate sigmoid cross-entropy from softmax cross-entropy. By contrast, the singular value spectrum the class centroids clearly distinguishes these losses.
40
Softmax (77.0%))
Squared (77.2%))
30
11
Dropout (77.5%)
Label Smoothing (77.6%)
Denatty
1A
More Final Layer (77.7%)
20
115
11h
Logit Penalty (77.7%)
Logit Normalization (77)8%)
Casine Softmax (77.9%)
Sigmoid (77.9%)
10
3.0
2.5
11c
111
111
Softmax (77.0%)
Squared Error (77.2%)
111
Dropout (77.5%)
2.0
111
111
More Final Layer L. (77.7%)
Label Smoothing (77.8%)
Densty
711
Logit Penalty (77.7%)
113
Logit Normaliaation (77.8%)
Cosine (77.9%)
Sigmoid (77.9%)
0.5
0.0
0.0
0.2
0.4
0.8
0.8
1.0
1.2
1.4
Cosine Distance
Figure E.3: The distribution of cosine distance between examples. Kerel density estimate of the cosine distance hetween examples of the same class (solid lines) and of different classes (dashed lines), for penultimate layer embeddings of 10,000 training set examples from ResNet-50 on ImageNet. Top and bottom plots show the same data with different y scales.
19


Reviews:
[{"review_id": "xl9SqmH0b0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a study on different loss functions and regularizers that apply on the penultimate layer and show how the choice of the loss function/regularizer can affect the performance of classifiers. The authors experiment with image datasets and study multiple parameters and side effects of the loss functions/regularizers.\n\nThis is an interesting paper with multiple experiments. It is a practical paper and provides multiple results on experiments for image classification. It is easy to follow and flows well.\n\nSome weak points are the following;\n\n- I find the datasets a bit limited and I think it would be nice if the authors could add extra datasets that go also beyond image classification.\n- In the same spirit what it misses is an extension on a large-scale scenario. Many practical applications have very large output space. Experiments on such scenarios would help to better understand benefits/limitations of the different loss functions. Also, the authors could extend to other objectives like negative sampling etc.\n- Some analysis on the individual classes is missing also. So, what do you observe when you have imbalanced classes?\n\nAs most of the impact is on the penultimate layers would it make sense to have a couple of experiments where you tune a model on another task. If I am not mistaken you trained from scratch.  It would be nice for example to have some results from another domain also like an NLP task.\n\nOverall I think is a good paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lx_jopuSfZN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to analyze the softmax cross entropy loss and discuss useful properties it has in an intuitive and accessible way. The paper mainly presents comparisons between this loss function and two other (very standard as well) loss functions that are the \"sigmoid cross entropy\" and the \"squared error\" loss functions. \n\nIt is very interesting to read the paper and realize that the topics discussed here have been written in well cited and well established statistics and ML books for decades. The softmax cross entropy loss is the multinomial regression loss while the sigmoid cross entropy loss is no other than the standard binary  logistic regression loss and finally the \"squared error\" is the least squared loss. \n\nAuthors compare the performance of these three on a few benchmarks. \n\nI think it's good to read the \"Elements of Statistical Learning\" and get a clear exposition to what science has been up to for decades before getting too deep into deep learning. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "A7RZgS4TDD4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors give a pretty thorough investigation of several existed losses of training deep networks in the task of supervised classification. The differences objectives are compared in terms of accuracy, calibration, out-of-distribution robustness, and predictions. \nSeveral discoveries are presented from the experiments:\n1) Regularizers and alternative losses did improve over vanilla softmax loss. But the effect of improvement is not additive.\n2) Different objectives mainly affect the last few layers while the layers not close to the output seems similar.\n3) All objectives that improve over vanilla softmax loss produce greater class separation in the penultimate layer of the network.\n\nPros:\n+ A comprehensive analysis of different behaviors of different objectives.\n+ The experiments setup seems fair.\n\nCons:\n- The authors claimed \n'''Our goal is instead to understand when one might want to use one loss function or regularizer over another and, more broadly, to understand the extent to which neural network performance and representations can be manipulated through the choice of objective alone.'''\nI wonder if the study in this paper realizes the goal that the authors claimed. It seems it is hard to make a conclusion to decide which single objective performs better than others, even in the task of classification, with the existing methods and optimization methods. It seems to have little insight for future works that using novel methods or optimization frameworks.\n\nI respect the efforts that the authors made to give a thorough analysis of existing common-used objectives. But since little novel technical contribution is made and the results seem not surprising, I am not sure whether such work is above the threshold of ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7-HHpm6Ojaw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper benchmarks various  training objectives, including loss functions and regularization schemes and measures accuracy, calibration and out-of-distribution robustness. The paper also studies how these objectives affect representations in various layers of the network.\n\n\nPros\n+ Paper is well written, and the narrative is coherent.\n+ Informative plots for the most part.\n+ Numerous experiments.\n\nCons\n- Unfortunately, novelty is very limited and this is mostly a restating/confirmation of results of earlier work.\n- Examples:  Sec 3.1 \"Regularizers and alternate losses increase accuracy\" -- what is new here? We already know this, otherwise these alternatives to vanilla cross-entropy wouldn't exist.\n- Sec 3.2 \"Different losses produce different predictions\" : again, why is this surprising? If the losses produced the same predictions, then they would all have the same accuracy. And we know this is not the case.\n- \"Mixup and auto-augment improve performance \" : already known.  \n- \"label smoothing improves calibration\" : already known\n\n The only possibly novel contribution is that losses behave similarly in the earlier layers, with most of the separation happening at the final layers. But this line --i.e., entanglement or separation behaving differently  in earlier vs. later blocks -- was already explored in Frosst et al 2019, so again here the novelty impact is limited. \n\n\nminor\n- dendrogram isn’t very intuitive. can the authors show a 2-d cluster representation of the model predictions (for all seeds, and for all losses?)\n\nSuggestions on improving novelty:\n- For a conference devoted to learning representations, the authors should investigated  into why, for example,  different random seeds of squared error loss disagree with each other so much, while some other losses are more self-consistent.\n\n- Different losses produce different predictions. but given that top-1 accuracies are nearly the same (table 1), the difference in figure 2 must  mostly be contained in the predictions where the classifiers were wrong (respect to ground truth). in other words, there shouldn’t be a significant difference in the cases where the classifiers agree with ground truth -- can the authors look into this?\n\n- At the start of the paper it is stated that  \"our goal is instead to understand when one might want to use\none loss function or regularizer over another\" : however, I do not see the paper providing any additional insights into this beyond what is already known.\n\nIn summary, lot of space is devoted to confirming earlier results, and while the  benchmarking is appreciated, a paper purely devoted to benchmarking but that only ends up confirming earlier results does not meet the bar.\n\n\nReferences:\n\n+ Frosst et al .Analyzing and Improving Representations with the Soft Nearest Neighbor Loss, ICML 2019", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an extensive empirical evaluation of several loss functions and regularization techniques used in deep networks. The authors conclude that the classical softmax is significantly outperformed by the other approaches, but there is no clear winner among them. Moreover, the authors have noticed two interesting facts, (1) the choice of loss function affects only upper layers of neural networks with the lower layers being very similar to each other, (2) losses that result in greater class separation also result in higher accuracy, but their features are less transferable to other tasks.

I agree with the authors that the comments of Reviewer 2 are shallow and not informative. Therefore, they were not taken into account in making the final decision and, as AC, I read the paper very carefully. Regarding Reviewer 4, however, I found his comments to be valid. There is a message that the authors want to communicate, and the reader that needs to decode this message using a noisy channel. Therefore, I encourage the authors to accordingly revise the paper to make the message much clearer. 

The experimental papers that compare a wide spectrum of methods are always hard to judge and this judgement is often very subjective. There are several seminal papers of this type, but not so many for several reasons. I agree with the authors that such studies are very valuable and give evaluation being not biased by authors of a given method. They are also very time- and resource-consuming. But there should be a general consensus how such an experiment should be performed. The authors of the particular methods should also be able to give right feedback to make their methods to be run appropriately. Therefore, there exist several websites and initiatives that try to fulfill these requirements. As said above, any paper of this type will be judged very subjectively.  

The discoveries made by authors should also be presented in a different way. One should start with a hypothesis that, for example, the lower layers are not affected by the loss function and then perform appropriate theoretical and empirical studies to verify the hypothesis. The same applies to the other discovery. In that way the message of the paper would be much clearer. I suppose that analysis of each of the discoveries deserves its own paper. 


Decision: Reject