Why Do Better Loss Functions Lead to Less Transferable Features?
Authors: Simon Kornblith
Ting Chen
Honglak Lee
Mohammad Norouzi
Venue: NeurIPS 2021 Poster (2021)
Keywords: transfer learning
representational similarity
class separation
training objectives
classification
supervised learning
label smoothing
dropout
centered kernel alignment

Abstract: Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.

Why Do Better Loss Functions Lead to Less Transferable Features?
Simon Kornblith', Ting Chen', Honglak Lee"t, Mohammad Norouzi' 'Google Rescarch, Toronto University of Michigan
Abstract
Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden repre- sentations of convolutional neural networks trained on ImageNet. We show that many objectives lead statistically significant improvements in ImageNet accu- racy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.
1
Introduction
Features learned by deep neural networks on ImageNet transfer effectively to wide range of computer
vision tasks 223, 60 34. These networks are often pretrained using vanilla softmax cross-entropy [IO]
[1], but recent work reports that other loss functions such as label smoothing (66) and sigmoid cross-entropy [8] outperform softmax cross-entropy on ImageNet. Although a previous investigation suggested that label smoothing and dropout can hurt transfer learning (34), the vast majority of work studying alternatives to softmax cross-entropy has restricted its empirical investigation to the accuracy of the trained networks on the original datasct. While it is valuable to study the impact objective functions classification accuracy, improving transfer accuracy of learned representations can have a more significant practical impact for many applications.
This paper takes comparative approach to understand the effects of different training objectives for image classification through the lens of neural network representations and their transferability. We carefully tune hyperparameters of each loss function and confirm that several loss functions outperform softmax cross-entropy by a statistically significant margin on ImageNet. However, we find that these improvements do not transfer to other tasks (see Table and Figure D We delve deeper and explain these empirical findings terms effects of different objectives on neural network representations. Our key findings are as follows:
We analyze the performance of different objectives on ImageNet and in transfer settings. Although many loss functions and regularizers lead to statistically significant improvements over vanilla
"Correspondence to: skormblith@@oggle.com
Work performed while at Google.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

softmax cross-entropy on ImageNet, these gains do not transfer. These alternative loss functions produce fixed feature extractors that transfer substantially worse to other tasks, in terms both linear and k-nearest neighhors classification accuracy, and provide no benefit when representations are fully fine-tuned.
The choice of objective primarily affects representations in network layers close to the output. Centered kernel aligmment (CKA) reveals large differences in representations of the last few layers of the network, whereas earlier layers are similar regardless of which training objective is used. This helps explain why the choice of objective has little impact on fine-tuning transfer accuracy.
All objectives that improve accuracy over softmax cross-entropy also lead to greater separation between representations of different classes in the penultimate layer features. These alternative objectives appear to collapse within-class variability representations, which accounts for both the improvement in accuracy the original task and the reduction in the quality of the features on downstream tasks.
Loss Functions and Output Layer Regularizers
We investigate loss functions and output layer regularizers. Let RK denote the network's ' output ("logit") vector, and let E 10 denote a one-hot vector of targets. Let æ E M denote the vector of penultimate layer activations, which gives rise to the output vector where W E KR is the matrix of final layer weights, and is a vector of biases.
Wx b.
All investigated loss functions include term that encourages z to have a high dot product with t. To avoid solutions that make this dot product large simply by increasing the scale of z, these loss functions must also include one more contractive terms and/or normalize z. Many "regularizers" correspond to additional contractive terms added to the loss, so we do not draw firm distinction between loss functions and regularizers. We describe each objective in detail below, and provide hyperparameters in Appendix A.
Softmax cross-entropy [10 is the de facto loss function for multi-class classification in deep learning:
K
=k
Lsoftmax (z, 1)
b log
L 2 + log
e
Σラ=1
K
k=1
The loss consists of a term that maximizes the product between the logits and targets, as well as contractive term that minimizes the LogSunExp of the logits.
Label smoothing [66] "smooths" the targets for softmax cross-entropy. The new targets are given by mixing the original targets with uniform distribution over all labels, t' = t x (1-a) +a// K, where determines the weighting of the original and uniform targets. order maintain the same scale for the gradient with respect to the positive logit, in our experiments, we scale the label smoothing loss by 1/(1 ax. The resulting loss
a/K,
K Σ
ck
(2,
a1-
+
log
K
レ== e²s
Müller et al. [43 previously showed that label smoothing improves calibration and encourages class centroids to lie at the vertices of regular simplex.
Dropout on penultimate layer: Dropout [64 is among the most prominent regularizers in the deep learning literature. We consider dropout applied to the penultimate layer of the neural network, i.c., when inputs to the final layer are randomly kept with some probability p. When employing dropout, we replace the penultimate layer activations æ with ē x ④ ε/p where & ~ Bernoull(p). Writing the dropped out logits as Σ Wa a + b. the dropout loss
K
Caropout W. b. x. t: = Eg [eottmax (ž, =
Σ
tb2k + Eg
log
(3)
Dropout produces both implicit regularization, by introducing noise into the optimization process and explicit regularization, by changing the parameters that minimize the loss [69].
75

---

Extra final layer L regularization: It is common to place the same L regularization on the final layer as elsewhere the network. However, we find that applying greater L regularization to the final layer can improve performance. The corresponding loss is:
Le W, t,t; Ammal Loltimu (z, t) + Ainaa WlI.
(4)
In architectures with batch normalization, adding additional L regularization has no explicit regu- larizing effect if the learnable scale (7) parameters are unregularized, but it still exerts an implicit regularizing effect by altering optimization.
Logit penalty: Whereas label smoothing encourages logits not to be too negative, and dropout imposes penalty on the logits that depends the covariance of the weights, alternative possibility is simply to explicitly constrain logits to be small in I norm:
Leenit_ (z, B) Lonttmax (z, t) -|1=11
Dauphin and Cubuk dropout.
19 showed that this regularizer yields accuracy improvements comparable to
We consider the use of L normalization, rather than regularization, of the logits. Because the entropy of the output of the softmax function depends on the scale of the logits, which is lost after normalization, we introduce an additional temperature parameter that controls the magnitude of the logit vector, and thus, indirectly, the minimum entropy of the output distribution:
Logit normalization:
K
K
Clogit porm (z, t; T) = Lsottmax (=/(F|== = T |||
t + log
(6)
k=1
k=1
Cosine softmax: We additionally consider 1.2 normalization of both the penultimate layer features and the final layer weights corresponding each class. This loss is equivalent softmax cross- entropy loss the logits are given by cosine similarity sim(x, y) = x y y/([=>]]ll)) between the weight vector and the penultimate layer plus a per-class bias:
K
Less (W, b, x. t;T)
4 (sim( Wk.: x//T + bk) + log
sim ()...
where T is a temperature parameter as above. Similar losses have appeared in previous literature [34 76 72 7 21 38 15], and variants have introduced explicit additive or multiplicative margins We observe that, even without an explicit margin, manipulating the temperature has large impact observed class separation.
to this loss that we do not consider here 33 72 73 21.
Sigmoid cross-entropy, also known as binary cross-entropy, is the natural analog to softmax cross- entropy for multi-label classification problems. Although we investigate only single-label multi-class classification tasks, we train networks with sigmoid cross-entropy and evaluate accuracy by ranking the logits of the sigmoids. This approach is related to the one-versus-rest strategy for converting binary classifiers to multi-class classifiers. The sigmoid cross-entropy loss is:
Luce (z, t) = --EK log
A (ETI) + +(1 (1-1)) (x) (1))log (1-0444) +
=-EK k== 1E22 + Ek_ log(e + 1).
(8)
The LogSum Exp term of softmax loss is replaced with the sum of the patuuuuutaansfommee log- its. We initialize the biases of the logits to - Glog K)) so that the initial output probabilities are approximately 1/K /KK. Beyer al. ( have previously shown that sigmoid cross-entropy loss leads to improved accuracy on ImageNet relative to softmax cross-entropy.
Squared error: Finally, we investigate squared error loss, formulated by Hui and Belkin 30
K
Emse t; M) = K
(ktx(zz M)² (1 t2)
-
k=
where x and M are hyperparameters. & sets the strength of the loss for the correct class relative to incorrect classes, whereas M controls the magnitude of the correct class target. When M = 1, the loss is simply the mean squared error between z and t. Like Hui and Belkin [00, we find that placing greater weight on the correct class slightly improves ImageNet accuracy.
3

---

Results
For each loss, we trained ResNet-50 [29 To tune loss hyperparameters and the epoch for early stopping, we performed 3 training runs per hyperparameter configuration where we held out validation set of 50,046 ImageNet training examples. We provide further details regarding the experimental setup in Appendix A. We also confirm that our main findings hold for Inception v3 666 models in Appendix B.
models on the ImageNet ILSVRC 2012 dataset [20
3.1 Better objectives improve accuracy, but do not transfer better
Table I: Objectives that produce higher ImageNet accuracy lead to less transferable fixed features. "Im- ageNet" columns reflect accuracy of ResNet-50 models the ImageNet validation set. "Transfer" columns reflect accuracy of L2 rregularized multinomial logistic regression k-nearest neighbors classifiers trained classify different transfer datasets using the fixed penultimate layer features of the ImageNet-traiee terarnnee networks. Numbers are averaged over 8 different pretraining runs; values not significantly different than the best are hold-faced (p 0.05, f-test). The strength of T,2 regularization is selected on a holdout set, and & is selected using leave-one-out cross-validation the training See Appendix Table I for similar table for Inception models, Appendix for linear transfer accuracy evaluated different epochs ImageNet pretraining, and Appendix D.3 for similar findings on the Chexpert chest X-ray dataset [3]].
ImageNet
Transfer
Pretraining loss
Top-1
Top-5
Food
CIFARIO
CIFAR100
Birdsnap SUN397
Cars
Pets
Flowers
Linear transfer:
Softmax
77.0
93.40
74.6
92.4
76.9
55.4
62.0
60.3
92.0
94.0
Label smoothing
77.6
93.78
72.7
91.6
53.6
61.6
54.8
92.9
91.9
Sigmoid
77.9
93.50
73.4
91.7
52.3
62.0
56.1
92.5
92.9
More linal layer 2
77.7
93.79
70.6
91.0
73.7
51.5
60.1
50.3
92.4
89.8
Dropout
77.5
93.62
72.6
91.4
75.0
53.6
61.2
54.7
92.6
92.1
Logit penalty
77.7
93.83
68.1
90.2
72.3
48.1
59.0
48.3
92.3
86.6
Logit nonnalization
77.8
93.71
66.3
90.5
72.9
50.7
58.1
45.4
82.9
Cosine softmax
77.9
93.86
62.0
89.9
71.3
55.0
36.7
91.1
75.3
Squared error
77.2
92.79
39.8
82.2
56.3
21.8
39.9
15.3
84.7
46.7
K-nearest neighbors:
Softmax
77.0
93.40
60.9
88.8
67.4
38.4
53.0
28.9
88.8
83.6
Label smoothing
77.6
93.78
59.2
88.3
66.3
39.2
53.5
27.3
91.5
80.3
Sigmoid
77.9
93.50
58.5
88.2
66.6
34.4
52.8
26.7
90.8
81.5
More final layer 2
77.7
93.79
55.0
87.7
36.2
51.1
24.4
90.9
75.6
Dropout
77.5
93.62
58.4
88.2
66.4
38.1
52.4
27.8
91.1
80.1
Logil penalty
77.7
93.83
52.5
86.5
62.9
31.0
49.3
22.4
90.8
68.1
Logit nomnalizaiion
77.8
93.71
50.9
86.6
63.1
35.1
45.8
24.1
88.2
63.1
Cosine softmax
45.5
86.0
41.8
52.7
Squared error
77.2
92.79
27.7
74.5
44.3
13.8
28.6
9.1
82.6
28.2
We found that, when properly tuned, all investigated objectives ex- cept squared error provide statistically significant improvement over softmax cross-entropy, as shown in the left two columns of Table The gains are small but meaningful, with sigmoid cross- entropy and cosine softmax both leading to improvement of 0.9% in top-1 accuracy over the baseline. For further discussion of differences in the training curves, robustness, calibration, and predictions of these models, see Appendix C.
Average Transfer Accuracy
70-
60
Logi Nom
* Sigmold
Cas mox
More 12
Seuared Broo
5g
Drapout
77.0
77.5
76.0
76.5
Although networks trained with softmax cross-entropy attain lower ImageNet top-1 accuracy than any other loss function, they nonetheless provide the most transferable features. We evaluated the transferability of the fixed features of our ImageNett models by training linear or k-nearest neighbors (kNN) classi-
ImagaNet Top-1 Accuracy
Figure Higher ImageNet accu- racy is not associated with higher linear transfer accuracy. Points represent individual training See Appendix Figure D. for simi- lar plots for individual datasets.
fiers to classify 8 different natural image datasets: Food-101 [9]
CIFAR-10 10 and CIFAR-10 100 (36,, Birdsnap [6]. SUN397 [77]
Stanford Cars (35 Oxford-IIIT Pets 511. and Oxford Flow-
ers The results of these experiments are shown in Table Figure In both linear and kNN settings, representations learned with vanilla softmax cross-entropy perform best for most tasks.

---

Table 2: The training objective has little impact on the performance of fine-tuned networks, Accuracy of fine-tuning pretrained networks on transfer datasets, averaged over 3 different pretraining initializations. We tune hyperparameters separately
objective and dataset. Numbers not significantly different than the best are bold-faced (p < 0.05, t-test for individual datasets, two-way ANOVA for average). See Appendix A.3 for training details.
Pretraining loss
Food
CIFARIO C0FAR100 Birdsnap
SUN397
Cars
Pets
Flowers
Avg.
Softmax
88.2
96.9
84.1
76.2
63.4
91.3
93.1
96.7
86.2
Label smoothing
88.3
96.7
84.0
76.3
63.6
91.2
93.7
96.3
86.3
Sigmoid
88.3
96.9
83.6
76.2
63.5
91.8
93.7
96.3
86.3
More final layer L
88.1
96.9
84.4
75.9
64.5
91.5
93.8
96.2
86.4
Dropout
88.3
96.7
84.2
76.5
63.9
91.2
96.3
Logit penalty
88.4
96.9
83.9
76.4
63.3
91.2
93.4
96.0
86.2
Logit normalization
87.9
96.9
82.9
76.0
58.3
91.2
92.7
95.9
85.2
Cosine softmax
88.3
96.9
83.2
75.6
56.9
91.3
92.5
95.9
85.1
Squared error
87.8
96.9
84.0
75.7
61.0
91.4
93.0
95.2
85.6
As shown in Table when networks are fully fine-tuned downstream tasks, the pretraining objective has little effect on the resulting accuracy. When averaging across all tasks, the best loss provides only 0.2% improvement over softmax cross-entropy, which does not reach statistical significance and is much smaller than the 0.9% difference in ImageNet top-1 accuracy. Thus, using different loss function for pretraining can improve accuracy on the pretraining task, but this improvement does not appear to transfer to downstream tasks.
3.2 The choice of objective primarily affects hidden representations close the output
Our observation that "improved" objectives improve only on ImageNet and not on transfer tasks raises questions about which representations, exactly, they affect. We use two tools to investigate differences in the hidden representations of networks trained with different loss functions. First, we use centered kernel alignment [33] [7 [8] directly measure the similarity of hidden representations across networks trained with different loss functions. Second, we measure the sparsity of the ReLU activations each layer. Both analyses suggest that all loss functions learn similar representations throughout the majority of the network, and differences are present only in the last few ResNet blocks.
Linear centered kernel alignment (CKA) provides a way measure similarity of neural network representations that is invariant to rotation and isotropic scaling in representation space [33] [17 [8]. Unlike other ways of measuring representational similarity between neural networks, lincar CKA can identify architectural correspondences between layers of networks trained from different initializations (33, a prerequisite for comparing networks trained with different objectives. Given two matrices X E Raxpi and Y E Raxp2 containing activations to the same examples, linear CKA computes the cosine similarity between the reshaped n x n covariance matrices between examples:
vec(cov (X)) vec(cov(Y))
CKAtinner X, Y)-
(10)
||oov( X F|]lo//oo(() [[F
We measured CKA between all possible pairings of ResNet blocks from 18 networks (2 different initializations for each objective). To reduce memory requirements, we used minibatch CKA [47
with minibatches of size 1500 and processed the ImageNet validation set for 10 epochs.
As shown in Figure 2 representations of the majority of network layers are highly similar regardless of loss function, but late layers differ substantially. Figure - shows similarity between all pairs of blocks from pairs of networks, where one network trained with vanilla softmax cross-entropy and the other is trained with either softmax or a different loss function. The diagonals of these plots indicate the similarity between architecturally corresponding layers. For all network pairs, the diagonals are stronger than the off- diaggonassss indicating that architecturally corresponding layers are more similar than non-corresponding layers. However, in the last few layers of the network. the diagonals are substantially brighter when both networks are trained with softmax than when the second network is trained with a different loss, indicating representational differences these layers. Figure 2 shows similarity of representations among all loss functions for a subset of blocks. Consistent differences among loss functions are present only in the last third of the network, starting around block 13.
5

---

a
Softmax Softmax
Softmax ve.
Softmax vs.
Saftmmx vs. More Final Layer L2
Softmax
1.0
Label Simoothing
Sigmoid
Dropout
Softmox Net Block
Softmax Net Block
16
16
18
18
Net Block
14
14
14
14
Softmax Ag 1 14
12
10
12
10
Net
10
12
12
10
0.8
4 8 8 1012144818
2 4 6 1012 141818
8 11111141818
2 4 8 61012141816
2.4 8 1012141818
0.6
Net Block
Label Smoothing Block
Sigmold Block
More Final Layer L2 Block
Dropout Block
Similaniy
Softmax vs.
Softmrx vs.
Saftmax vs.
Softmax vs.
Softmax vs.
Logit Penalty
Logit Normalization
Cosine Softmax
Squared Emar
Untrained
Softmax Net Blook
18
Saftmax Net Blook
18
Softmax Net Block
Softmax Net Blook
Softmax Net Block
0.4
13
2.15
c
m
01
1012141618
1012141115
1012141615
Logit Penalty Block
Logit Nomalization Block
Cosine Softmax Block
Squared Erao Block
Untrained Block
0.0
b
Softmax
11
1.0
Smosthing
More Final L2
Dropout
0.8
Logit Panelly
Logit Normalization
Cosine Softmax
0.6
Avg.
Logits
Label Smopthing
0.4
Sigmoid
More Final Layer L2
Logit Panalty
Dropout
0.2
Coeine Softmax
Squared Eror
Label Softmax
Sigmod
Dropour pouu
Logkt Logk Normalien Penally
Softmaa
Sotmer
Drepon uppout
Penalty
Logs Logk-anntt
Logit Normallzan dermallzationn
Cosine Softmal
Softmux
Label Smoothh Sigmoo
More Final Layer L.2
Logk Leoppn
Normall Normallcatt inliicatioon
Coeine Softmex
Squared Enror
softma Smom
max
Dropsa oopout
Logh Petially eeaally
Loggt Nomalization Cosine Squared miia
Softmax soceting
sayer L2
Logk Deeros
Logit Normallzan comallaationn
Crosie Softmax
Squared Erro
0.0
More Final Layer
Layer
Squared
Squared
More Final
More Final
More Final
Label
Label
Label
Figure 2: The choice of loss function affects representations only later network layers. All plots show linear centered kernel alignment (CKA) between representations computed on the ImageNet validation set. CKA between network layers, for pairs of ResNet-SN 50 models trained from different initializations with different losses. As controls, the top-right plot shows CKA between two networks trained with softmax. and the hottom-right plot shows CKA.A between model trained with softmax loss and model initialization where batch norm moments have been computed corresponding layers of networks trained with different loss functions. Diagonal reflects similarity of networks with the same loss function trained from different initalizations. See Appendix Figure B.1 for similar figure for Inception v3 models.
training set. b: CKA between representations extracted from
Percent acthvatons
100
56x56
28x28
14x1
7x7
Pool
Softmax
Label Smoothing
Logit Norm
80
Squared Error
More L2
Cosine Softmax
Drapout
Lagit Penalty
Sigmoid
20
12
ResNet Block
Figure 3: Loss functions affect sparsity later layer representations. Plot shows the average non-zero activations for each ResNet-50 block, after the residual connection and subsequent nonlinearity, on the ImageNet validation set. Dashed lines indicate boundaries between stages.
The sparsity of activations reveals a similar pattern of layer-wise differences between networks trained with different loss functions. Figure Sequre shows the proportion non-zero activations in different layers. In all networks, the percentage of non-zero ReLU activations decreases with depth, attaining its minimum the last convolutional layer. the first three ResNet stages, activation sparsity is broadly similar regardless of the loss. However, in the final stage and penultimate average pooling layer, the

---

Table 3: Regularization and alternative losses im-
56x56
28x28
14x14
7x7
Pod
prove class separation in the penultimate layer. Re-
Class separation (R3
Softrix
Logit Penally
sults are averaged over ResNet-50 models per loss on the ImageNet training set.
Squared Error
Logit Normalizaion
Dropout
Casine Softmax
Label Smoothing
Sigmoid
Loss/regularizer
ImageNet top-1
Class sep. (R²)
More Final Layer L2
Softmax
77.0 0.06 0.349 +: 0.0002
Label smooth.
77.6 : 0.03
0,420 + 0.0003
Sigmoid
77.9:0000
0,427 = 0.0003
2
16
Extra L
77.7
0.572 = 0.0006
ResNet Block
Dropout
7.04
0.461 = 0.0003
Figure 4: Class separation in different layers of ResNet-
Logit penalty
77.7 + 0.04
0.601 + 0.0004
50, averaged over models per loss on the ImageNet
Logit norm
77.8 0.02
0.517 + 0.0002
training set. For convolutional layers, we compute co-
Cosine softmax
77.9 0.02
0.641 + 0.0003
sine distances by flattening the representations of exam- ples across spatial dimensions.
Squared error
77.2 0.04
0.845 + 0.0002
degree of sparsity depends greatly on the loss. Penultimate layer representations of vanilla softmax networks are the least sparse, with 92.8% non-zero activations. Logit normalization, cosine softmax, and squared error all result in much sparser (<25% non-zero) activations.
These results immediately suggest an explanation for the limited effect of the training objective on networks' fine-tuning performance. We observe differences among objectives only in later network layers, but previous work has found that these layers change substantially during fine- tuning 77 53 46], an observation we replicate in Appendix D.4. Thus, the choice of training objective appears to affect parts of the network that are specific to the pretraining task, and do not transfer when the network is fine-tuned on other tasks.
3.3 Regularization and alternative losses increase class separation
The previous section suggests that different loss functions learn very different penultimate layer representations, even when their overall accuracy is similar. However, as shown Appendix C.4, combining different losses and penultimate layer regularizers provides no accuracy improvements over training with only one, suggesting that they share similar mechanisms. In this section, we demonstrate that simple property of networks' penultimate layer representations can explain their beneficial effect on accuracy relative to vanilla softmax cross-entropy, as well as their harmful effect on fixed features. Specifically, compared to vanilla softmax cross-entropy, all investigated losses cause the network to reduce the relative within-class variance in the penultimate layer representation space. This reduction in within-class variance corresponds to increased separation between classes, and is harmful to linear transfer.
The ratio of the average within-class cosine distance to the overall average cosine distance measures the dispersion of representations of examples belonging to the same class relative to the overall dispersion of embeddings. We take one minus this quantity to get closed-form index of class separation that is between and 1:
R² 1
dwvthhii/ / dhotal
(11)
N
N₂
sim kk Fk dyotall
K
K
N sim (2 jm
Bk.n
diittin
KN
K2N,N
1=1
J=1 k=
where æk,m is the embedding of example in class & E {1...., K}. N₂ is the number of examples in class k. and sim(æ, y) x T //(l [||x|||]]]] | is cosine similarity between vectors. As we show in Appendix I. if the embeddings are first L2 normalized, then 1 R² is the ratio of the average within-class variance the weighted total variance, where the weights are inversely proportional to the number of examples in each class. For a balanced dataset, R² is equivalent to centered kernel alignment [17 [8] between the embeddings and the one-hot label matrix, with cosine kernel. See Appendix E.2 for results with other distance metrics.
As shown in Table and Figure all regularizers and alternative loss functions we investigate produce greater class separation penultimate layer representations as compared to vanilla softmax loss. Importantly, this increase in class separation is specific to forms of regularization that affect networks'
7

---

Figure 5: Class separation negatively correlates with lin- ear transfer accuracy. Mean linear transfer accuracy across all tasks vs, class separa- tion on ImageNet, for different loss functions. b: For cosine softmax loss, training at higher temperature produces greater class separation but worse lin- transfer accuracy.
Pigmoid
b
Mean Trannfer Anc. (sg)
Smodhing
Loght Norm
@Cropoul
LLogh Penalty
Mean Transfer Acc. (%)
75
0.7
Cosine Softmake
70
0.6
S890 dogandes
0.5
60
65
60
0.3
50
Enjre
0.2
0.4
0.8
0.8
0.02
0.0c
0.06
0.08
Class Separation (R
Temperature
Table 4: Temperature of cosine softmax loss controls ImageNet accuracy, class separation (R²), ) and linear transfer accuracy.
ImageNet
Transfer
Temp.
Top-1
R²
Food
CIFARIO
CIFARI00
Birdsnap
SUN397
Cars
Pets
Flowers
0.01
74.9
0.236
73.4
91.9
76.5
57.2
60.5
62.9
91.7
93.6
0.02
77.0
0.358
72.1
91.8
76.2
56.5
60.4
58.5
92.2
91.2
0.03
77.5
0.475
69.1
91.5
74.9
53.7
59.1
51.8
92.3
87.4
0.04
77.6
0.562
66.0
90.7
73.8
50.3
57.4
45.1
91.7
82.2
0.05
77.6
0.634
62.8
90.4
72.2
47.6
55.4
38.6
91.0
78.3
0.06
77.5
0.693
60.3
89.3
69.8
43.3
53.8
33.3
91.0
72.7
0.07
77.5
0.738
57.1
88.7
68.6
39.6
51.4
29.1
90.2
67.9
0.08
77.6
0.770
53.7
87.7
66.5
35.5
49.4
25.7
89.3
63.2
final layers. In Appendix E.3, we show that adding regularization through data augmentation improves accuracy without a substantial change in class separation. We further investigate the training dynamics of class separation in Appendix E.4, finding that, for softmax cross-entropy, class separation peaks early in training and then falls, whereas for other objectives, class separation either saturates or continues rise training progresses.
3.4 Greater class separation is associated with less transferable features
Although losses that improve class separation lead to higher accuracy on the ImageNet validation set, the feature extractors they learn transfer worse other tasks. Figure 5h plots mean linear transfer accuracy versus class separation for each of the losses we investigate. We observe a significant negative correlation (Spearman's 's -0.93, p 0.002). Notably, vanilla softmax cross-entropy produces the least class separation and the most transferable features, whereas squared error produces much greater class separation than other losses and leads much lower transfer performance.
To confirm this relationship between class separation, ImageNet accuracy, and transfer, we trained models with cosine softmax with varying values of the temperature parameter 7 [|] As shown in Table 4 lower temperatures yield lower top-1 accuracies and worse class separation. However, even though the lowest temperature of 5 0.01 achieved 2.7% lower accuracy ImageNet than the best temperature, this lowest temperature gave the best features for nearly all transfer datasets. Thus, т controls tradeoff between the generalizability of penultimate-layer features and the accuracy the target dataset. For 2 0.04, ImageNet accuracy saturates, but, as shown in Figure 5b, class separation continues to increase and transfer accuracy continues to decrease.
there any situation where features with greater class separation could beneficial for a downstream task? In F we L'-ueuulrr logistic regression relearn the original 1000-way ImageNet classification head from penultimate layer representations of 40,000 examples from the ImageNet validation set ? We find that features from networks trained with vanilla softmax loss perform worst, whereas features from networks with greater class separation perform substantially better. Thus,
'Training < 0.05 was unstable; we scale the loss by the to reduce the effect of on the size of the gradient WRT the correct class logit. Relationships for 2 0.05 remain consistent without loss scaling.
We use this experimental setup to avoid training linear classifiers on examples that were also seen during pretraining. However, results are qualitatively similar if we train the linear classifier on a 50,046 example subset of the training set and test the full validation set.

---

Figure 6: Class separation positively correlates with accuracy when relearning how to classify ImageNet classes from limited data. All accuracy numbers are computed on a 10,000 example subset of the ImageNet validation set (10 examples per class). Blue dots indicate accuracy of the weights of the original ImageNet- trained model. Orange dots indicate accuracy of a linear classifier trained the other 40,000 examples from the ImageNet validation set. Lines connect blue and orange dots corresponding to the same objective. The gap between the accuracies of the original and relearned classifiers narrows as class separation increases, Class separation is measured the ImageNet training set. All numbers are averaged over models.
Top-1 Accurecy (K0)
78-
L Penally
77-
à Extra L2
Squared Enor
og Norm
76-
"Drapout"
Sigmoid Label
75-
Preirained model
claesi
40,000 cxamples
&oflmax 0.4
0.5
0.6
0.7
0.8
Class Separation (R
seems that representations with greater class separation are "overfit," not to the pretraining datapoints, but to the pretraining classes- they perform better for classifying these classes, but worse when the downstream task requires classifying different classes.
Our results above provide some further evidence that greater class separation can be beneficial in real-world scenarios where downstream datasets share classes with the pretraining dataset. In [and representations with the lowest class separation perform best on all datasets except for Oxford-IIIT Pets, where representations with slightly greater class separation consistently perform slightly better. We thus explore class overlap between transfer datasets and ImageNet in Appendix F, and find that, of the 37 cat and dog breeds in Oxford-III Pets, 25 correspond directly to ImageNet classes. Furthermore, is possible to achieve 71.2% accuracy on Oxford-IIIT Pets simply by taking the top-1 predictions of an ImageNet classifier and and mapping them directly to its classes, but this strategy achieves much lower accuracy on other datasets.
4 Related work
Understanding training objectives. There is a great deal of previous work that investigates why some objectives perform better than others, using both theoretical and empirical approuches. On linearly separable data, theoretical analysis shows that gradient descent on both unregularized logistic or multinomial logistic regression objectives (i.c., linear models with sigmoid or softmax cross- entropy loss) eventually converges to the minimum norm solution 663. These results can be extended to neural networks in certain restricted settings [28 74. However, the convergence to this solution very slow. Theoretical analysis of dropout has bounded the excess risk of single-layer (70 and two-layer [42 networks. Empirical studies have attempted to understand regularization in more realistic settings. Label smoothing has been explained in terms of mitigation of label noise [39 [3], entropy regularization 52 41) and accelerated convergence (78) Other work has shown that dropout on intermediate layers has both implicit and explicit effects, and both are important to accuracy 77 Janocha and Czarnecki [32) previously studied both empirical and mathematical similarities and differences among loss functions for supervised classification in the context of deep learning.
Training objectives for transfer. Our study of the transferability of networks trained with different objectives extends the previous investigation of Kornblith et al. 34 which showed that two of the regularizers considered here (label smoothing and dropout) lead to features that transfer worse. Similar results have been reported for suf-suuere learning, where the loss parameters that maximize accuracy on the contrastive task do not provide the best features for linear classification [14]. Our work demonstrates that this phenomenon representations. By contrast, work that explores -utto--dist (OOD) generalization, where the training and evaluation datasets consist of the same classes but with some degree distribution shift, finds that ImageNet accuracy is highly predictive of OOD accuracy across many pretrained models [55] 67 In Appendix Table C.2, we show that, across loss functions, ImageNet validation set accuracy is not entirely correlated with OOD accuracy, but different losses produce much smaller differences in OOD accuracy than linear transfer accuracy.
pervasive, and connects it to properties of hidden
Other related work has attempted to devise loss functions to learn more transferable embeddings for few-shot learning (62 65 15 49 but embeddings of models trained with softmax cross-entropy often perform on par with these more sophisticated techniques (68). Doersch et al. 22 motivate their few-shot learning method as way to mitigate "supervision collapse," which they do not directly quantify, but is closely related to our notion of class separation. While paper was under review,

---

two studies reported advantages for increased final layer L regularization when performing few-shot linear transfer from very large pretraining datasets some, but not all. downstream tasks [80] I Our findings suggest that these advantages may arise because the pretraining and downstream tasks share classes, but further investigation is needed to confirm this hypothesis.
Class separation. Prior work has investigated class separation in neural networks in a variety of different ways. Theoretical work connects various concepts of the normalized margin between classes to the generalization properties of neural networks H S 45
Empirically, Chen et al. [12] measure class separation via "angular visual hardness," " the arccosine tranosnn-rrnssoommed cosine similarity between the weight vectors and examples, and suggested an association between this metric and accuracy. However, as shown in Appendix Figure E. this metric unable to differentiate between networks trained with softmax and sigmoid cross-entropy. Papyan et al. [50] describe phenomenon they call "neural collapse," which occurs after training error vanishes and is associated with the collapse of the class means, classifier, and activations to the vertices of an equiangular tight frame, implying maximal class separation. We relate our observations to theirs in Appendix Table E.2. More generally, studies have investigated how class information evolves through the hidden layers of neural networks using linear classifiers [2]. binning estimators of mutual information [6] [88 26]. Euclidean distances [59], the soft nearest neighbor loss 24,, and manifold geometry K [6. In the context of study of representational consistency across networks trained from different initializations, Mehrer et al. [40] previously reported increased class separation in CIFAR- 10 networks trained with dropout.
Other work has explored relationships between measures of class separation in embedding spaces and accuracy in few-shot and deep metric learning settings. In deep metric learning, Roth et al. [56] show that objectives that lead to greater class separation generally produce higher recall, and find a similar relationship for measure of the uniformity of the singular value spectrum of the representation space. In few-shot learning, different work has reached different conclusions: Goldblum et al. (25) find that regularization that increases class separation improves few-shot classification performance on mini Imag-ee and CIFAR-FS, whereas Liu et al. (37 find that introducing a negative margin into a cosine softmax loss yields both lower class separation and higher accuracy.
5 Limitations
Although we investigate many losses and multiple architectures, our experiments are limited to moderately sized datasets with moderately sized models, and our conclusions are limited to supervised classification settings. ResNet-50 trained ImageNet is realistic transfer scenario where our analysis is computationally tractable, but bigger models trained on bigger datasets with more classes achieve better performance. These bigger datasets with richer label spaces could potentially help to mitigate the trade-off between pretraining accuracy and feature quality, and may also have lleel overlap with many downstream tasks, making greater class separation helpful rather than harmful.
Our results establish that a wide variety of losses that improve over vanilla softmax cross- cross-entropy lead to greater class separation, but this observation does not immediately lead to recipe for higher pretraining accuracy. The relationship between class separation and pretraining accuracy is non- monotonic: squared error achieves the highest class separation but does not significantly outperform vanilla softmax cross-entropy.
6 Conclusion
In this study, we find that the properties of loss that yields good performance on the pretraining task are different from the properties of a loss that learns good generic features. Training objectives that lead to better performance on the pretraining task learn more invariant representations with greater class separation. However, these same properties are detrimental when fixed features are transferred to other tasks. Moreover, because different loss functions produce different representations only in later network layers, which change substantially during fine-tuning, gains in pretraining accuracy do not lead to gains when models are fine-tuned on downstream tasks.
Our work suggests opportunities for improving fixed feature representations in deep neural networks. We see no inherent reason that features learned by softmax cross-entropy should be optimal for transfer, but previous work has optimized for pretraining accuracy, rather than transferable features. With the increasing importance of transfer learning in deep learning, we believe that future research into loss functions should explicitly target and evaluate performance under transfer settings.
10

---

Acknowledgements
We thank Pieter-Jan Kindermans for comments on the manuseript, and Geoffrey Hinton and David Fleet for useful discussions.
References
[1] S. Abnar, M. Dehghani, B. Neyshabur, and H. Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:21 2021.
[2] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 2016.
[3] A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan. Intrinsic dimension of data representations
in deep neural networks. In Advances in Neural Information Processing Systems, 2019.
[4] P. L.. Bartlett. For valid generalization, the size of the weights is more important than the size of the network. Advances in Neural Information Processing Systems, pages 134-140, 1997.
[5] L. Bartlett, D. J. Foster, and M. Telgarsky. Spectraaly-nnormaaizee
networks. Advances in Neural Information Processing Systems, margin bounds for neural 2017.
[6] T. Berg, J. Liu, S. W. Lee, M. L. Alexander, D. W. Jacobs, and P. N. Belhumeur. Birdsnap: Large-scale scale fine-grained visual categorization of birds. In IEEE Conference on Computer Vision and Puttern Recognition, pages 2019-2026. IEEE, 2014.
[7] L. Bertinetto, F. Henriques, P. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference Learning Representations, 2019.
[8] L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. d. Oord. Are we done with ImageNet? arXiv preprint arXiv:2006.07159 2020.
[9] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision, pages 446 461. Springer, 2014.
[10] S. Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pages 227-236. Springer, 1990.
[11] S. Bridle. Training stochastic model recognition algorithms as networks lead to maximum mutual information estimation of parameters, In Advances in Neural Information Processing Systems, pages 211 211-2 1-1217. 1990.
[12] B. Chen, W. L. A. Garg, Z. Yu, A. Shrivastava, J. Kautz, and A. Anandkumar. Angular visual hardness. In International Conference on Machine Learning, 2020.
[13] B. Chen, L. Ziyin, Z. Wang, and Liang. An investigation of how label smoothing affects generalization. arXiv preprint arXiv:2010. 12644, 2020.
[14] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[15] W.- Chen, Y.-C. Liu, Z. Kira, Y.-. F. Wang, and J.-B. Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
[16) U. Cohen, S. Chung, D. D. Lee, and Sompolinsky. Separability and geometry of object
manifolds in deep neural networks. Nature Communications, 1 111::11-3,, 2020.
[17] C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
[18] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and S. Kandola. On kernel-target alignment. In Advances in Neural Information Processing Systems, pages 367-373, 2002.
[19] Y. Dauphin and E. D. Cubuk. Deconstructing the regularization batchnorm. In International Conference Learning Representations, 2021.
[20] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. Icce, 2009.

---

[21] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
[22] C. Doersch, A. Gupta, and A. Zisserman. Crosstransformers: spatially aware few-shot transfer. arXiv preprint arXiv:2007. 11440,,,,1449,, 2020.
[23] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang. E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generie visual recognition. In International Conference on Machine Learning, 2014.
[24] N. Frosst, N. Papernot, and G. Hinton. Analyzing and improving representations with the soft nearest neighbor loss. In International Conference on Machine Learning, pages 2012-2020. PMLR, 2019.
[25] M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein. Unraveling meta- learning: Understanding feature representations for few-shot tasks. International Conference on Machine Learning, pages 3607-3616. PMLR, 2020.
[26] Z. Goldfeld, E. d. Berg, K. Greenewald, I. Melnyk, N. Nguyen, B. Kingsbury, and Y. Polyan- skiy. Estimating information flow in deep neural networks. In International Conference on Machine Learning, 2018.
[27] S. Gross and M. Wilber. Training and investigating residual nets. In The Torch Blog. http: //torch.ch/ blog/ wwwpg/////////////////////////// 21/22221102222 20016/02/04/reseeeee
html, 2016.
[28] S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pages 9461-99 1111-9471,222 2018.
[29] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[30] L. Hui and M. Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks. In International Conference on Learning Representations, 2021.
[31] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurca-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 990-597,10 2019.
[32] K. Janocha and W. M. Czarnecki. On loss functions for deep neural networks in classification. Schedae Informaticae, 25, 2016.
[33] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, 2019.
[34] S. Kornblith, J. Shlens, and Q V. Le. Do better ImageNet models transfer better? In IEEE Conference Computer Vision and Pattern Recognition, pages 2661-2671, 2019.
[35] Krause, J. Deng, M. Stark, and L. Fei-Fei. Collecting large-scale dataset of fine-grained cars. In Second Workshop on Fine-Grained Visual Categorization, 2013.
[36] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[37] B. Liu, Y. Cao, Y. Lin, Q. Li, Z. Zhang, M. Long, and H. Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision, pages 438 455. Springer, 2020.
[38] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 212-220, 2017.
[39] M. Lukasik, S. Bhojanapalli, A. Menon, and S. Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pages 6448 6458. PMLR, 2020.
[40] Mehrer, C. J. Spoerer, N. Kriegeskorte, and T. C. Kietzmann. Individual differences among deep neural network models. Nature Communications, 11(1):1-12, 2020.
12

---

[41] C. Meisterr, E. Salesky, and R. Cotterell. Generalized entropy regularization or: There's 's nothing special about label smoothing. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870-6886 2020.
[42] P. Mianjy and R. Arora. On convergence and generalization of dropout training. Advances in Neural Information Processing Systems, 33, 2020.
[43] R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pages 4696-4705, 2019.
[44] B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Conference on Leurning Theory, pages 1376-1401. PMLR, 2015.
[45] B. Neyshabur, S. Bhojanapalli, and N. Srebro. A pac-bayesian approach to eeectrally-normalize margin bounds for neural networks. In International Conference on Learning Representations, 2018.
[46] B. Neyshabur, H. Sedghi, and C. Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems, 2020.
[47] T. Nguyen, M. Raghu, and S. Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference Learning Representations, 2021.
[48] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Computer Vision, Graphics & Image Processing, 2008. ICVGIP'08. '08. Sixth Indian Conference on, pages 722-729. IEEE, 2008.
[49] B. N. Oreshkin, Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. arXiv preprint 1r05 10123, 2018.
[50] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652-24663, 2020.
[51] O. M. Parkhi, Vedaldi, A. Zisserman, and Jawahar. Cats and dogs. IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-35411 -0088-355.-11 IEEE, 2012.
[52] G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton. Regularizing neural networks 066448, 2017.
by penalizing confident output distributions. arXiv preprint
[53] M. Raghu, C. Zhang. J. Kleinberg, and S. Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
[54] R. Ranjan, C. D. Castillo, and R. Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv: 1rXXi:1703.009: 700770500090 2017.
[55] B. Recht, R. Roelofs, L.. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning, 2019.
(56) K. Roth, T. Milbich, S. Sinha, P. Gupta, B. Ommer, and J. P. Cohen. Revisiting training strategies and generalization performance deep metric learning. In International Conference on Machine Learning, pages 8242-8252. PMLR, 2020.
[57] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
[58] A. M. Saxe, Y. Bansal. J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, (019(12))124021) 2019.
[59] A. Schilling, C. Metzner, J. Rietsch, R. Gerum, H. Schulze, and P. Krauss. How deep is deep class separability in the hidden layers of deep neural networks. arXiv 2018
preprint 1rXiv:
[60] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014.
[61] R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1703: 00010, 2017.
13

---

[62] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087, 2017.
[63] D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822 28778, 2018.
[64] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-198)) 2014.
[65] Sung. Y. Yang. L. Zhang. T. Xiang, P. H. Torr, and M. Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1199-1208 1208, 2018.
[66] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In JEEE Conference on Computer Vision and Pattern Recognition. pages 2818-12811118 18 28262, 2016.
[67] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 2020.
[68] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola. Rethinking few-shot image classification: good embedding is all you need? In European Conference on Computer Vision, 2020.
[69] Wager, Wang, and P. S. Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems, pages 351-359, 2013.
[70] Wager, W. Fithian, S. Wang. and P. S. Liang. Altitude training: Strong bounds for single-layer dropout. In Advances in Neural Information Processing Systems, 2014.
[71] Wang, X. Xiang. J. Cheng, and L. Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international Conference on Multimedia, pages 1041-1049, 2017.
[72] F. Wang. J. Cheng, W. Liu, and H. Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7))926-95) 2018
[73] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. IEEE Conference on Computer Vision and Pattern Recognition, pages
5274, 2018.
[74] C. Wei, J. D. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. In Advances in Neural Information Processing Systems, pages 9712-9724, 2019.
[75] C. Wei, S. Kakade, and T. Ma. The implicit and explicit regularization effects of dropout. In International Conference on Machine Learning. 2020.
[76] N. Wojke and A. Bewley. Deep cosine metric learning for person re-identification. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 748-756. IEEE, 2018.
[77] Xiao, Hays, Ehinger, Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3492. IEEE, 2010.
[78] Y. Xu, Y. Xu, Q. Qian, H. Li, and R. Jin. Towards understanding label smoothing. arXiv preprint 2rXiv:2006. / 1653, 2020.
[79] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[80] X. Zhai, A. Kolesnikov. N. Houlsby, and Beyer. Scaling vision transformers. arXiv preprint 21Ki::2006.05660 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss limitations in Sec-
tion 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide training details Appendix A.I.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We do not report the standard deviations across seeds in all tables because it is not possible to do while keeping the text at readable size. However, we perform appropriate statistical testing based on the standard deviations across seeds and bold-face numbers in each table that are not statistically significant from the best.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "vxlUpt0eyG6", "reviewer": "Reviewer_qwG6", "summary": "The development of new loss functions and regularizers is a major domain of ML research, but researchers often prioritize accuracy on the original dataset over transfer learning. The authors of the present work wanted to understand the effects of different training objectives on transferability in image classification tasks. They find that many loss functions outperform the traditional softmax cross-entropy but transfer worse, and provide insight into this phenomenon by examining hidden layer representations: higher original-dataset accuracy collapses within-class variability in deeper layer representations, leading to less generalizable features. ", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors comprehensively characterize the effects of different loss functions empirically, not just on transfer learning, but also many performance metrics of interest including accuracy, out-of-distribution/naturalistic robustness, and calibration. I am also convinced by their analysis-based approach to understanding the mechanisms underlying the observed effects of different loss functions on different performance metrics. The work is broadly relevant beyond simply understanding and improving transfer learning. And while I would like to see this work expanded beyond the ResNet50 architecture, I understand that replicating the entirety of this work on larger models would be very resource-intensive.\n\nThis is a very thorough paper. \n\nI have a few minor comments:\n\nOptimal validation accuracy is typically achieved after 140-200 epochs of training, which leads to the question of whether the results are consistent across the duration of training. The answer to this is interesting from a scientific perspective, but there is also a practical consideration: it would be very helpful for other ML practitioners to know whether they can expect to see benefits from alternative loss functions at shorter training durations. At the very least, I think the authors should include a plot of validation accuracy as a function of epoch for the different loss functions.\n\nI think it would be helpful to have a version of Figure 1 with the full transfer learning results and/or some measure of variability across different transfer datasets.\n\nThe authors state “The relationship between class separation and pretraining accuracy is non-monotonic; squared error achieves the highest class separation but does not significantly outperform vanilla softmax cross-entropy.” I would appreciate seeing some discussion (and ideally analysis) of the mechanisms by which accuracy can be improved other than by increasing class separation in the penultimate layer. Also, perhaps it was cut for space, but there is no motivation provided for why the authors chose to examine class separation in the penultimate layer. It seems like a perfectly sensible property to examine, but the transition to this analysis feels a bit abrupt.\n\nThe manuscript could be enriched with some discussion about the very interesting result that none of the loss functions have worse original-dataset accuracy or better transfer accuracy than softmax-cross-entropy, though I understand space is at a premium.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately address the limitations of their work in the space provided.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "hZwx3zc-iC", "reviewer": "Reviewer_a9W9", "summary": "This paper studies the impact of common supervised objectives and regularization on transfer learning. With extensive experiments, the authors find out that although these objectives (or regularizers) lead to better performance on the pre-training dataset, their performance on the downstream tasks are always worse. This complements the results in Kornblith et al.. Furthermore, the authors try to probe into the reason of this phenomenon, and find out that these objectives induce better class separation, which is associated with less transferable features.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies a very important problem in transfer learning. The manuscript is well written. The claims are stated clearly in each section and supported with thorough experiments. However, I believe several aspects of the manuscript can be improved in its current form.\n\n1)\tAlthough the conclusion of the paper is interesting and solid, I can be limited to the current setup. When the pretraining and target tasks are very similar (in extreme terms, the same), better performance on the pretraining task can lead to better transfer learning. As the similarity decreases, the performance of the pretraining and target tasks are less correlated, and we may enter the regime of the current paper. \n2)\tThe findings of section 3.2 is not surprising as it is common sense that neural networks tend to learn similar “low-level” features in lower layers, and lower layers are more important in fine-tuning. In addition, the accuracies of different objectives vary only by ~1%, which can be too small for an informative CKA visualization.\n3)\tAs to the class separation, many pre-training tasks are not supervised classification, which do not involve it. Even if greater class separation is associated with less transferable features, it is hard to say class separation determines transferability, which cast doubt on the value of this claim.\n\nPost-rebuttal:  \nAfter reading the rebuttal and other reviews carefully, I found out some of my concerns addressed. On the other hand, I still believe some part of the experiments and the conclusion of the paper are limited:  \n1. The downstream tasks are limited to classification. It is fine to me to study only the computer vision setup, but it is necessary to include other type of downstream tasks such as object detection. Even for classification tasks, there exists datasets which are very different from those chosen in this paper. I don't think the 8 datasets are representative of downstream tasks in practical use.  \n2. As to the concrete value of the paper, an interesting direction would be a loss function which decreases the separation for more transferability.  \n\nI would like to increase my score to 5, and I strongly recommend the authors to incorporate these points into future versions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "alY1_OqqKaK", "reviewer": "Reviewer_HjSp", "summary": "This paper studies an interesting problem of how the representation features learned by different loss functions affect the performance on downstream tasks. They reveal some interesting patterns, such as better class separation on pre-training dataset does not guarantee better performance on downstream tasks and it can be caused by “overfitting” on original task. Overall, this paper is clearly written and easy to follow. The numerical results are also well-presented.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) The authors present detailed studies on the choice of loss functions, comparison of transferability, etc., and draw several interesting conclusions. \n2) Overall, I find this paper is clear and well-presented. \n\nCons:\n1) ResNet is a sophisticated architecture, however, it needs more evidence to draw general conclusions about transferability, especially for section 3.2. For example, if we don’t use the residual connection, can we observe similar patterns?\n\n2) The choice of downstream tasks is somewhat limited. To better demonstrate the transferability, different types of downstream tasks should be considered, e.g., object detection.\n\n3) In line 224-230, the authors illustrate that representations with better class separation can perform better on downstream tasks if the classes are overlapped. So, I wonder how many overlapped classes are in these 8 selected datasets. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This has been addressed in the paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5MJIriNnKoJ", "reviewer": "Reviewer_xVq5", "summary": "The paper studies the role of downstream performance on the given task and its role on transferring to a novel data distribution. The paper rigorously studies the role of different loss functions and its effect on transfer learning (with or without finetuning) while also investigating other learned features of the NN. The paper concludes that learning better downstream discriminative features may not lead to better transfer learning models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well studied and a clear investigation into learning transferrable features and how learning better downstream classifiers might lead to slightly worsening performance of the models. The study investigates various properties of the learned network such as sparsity, early layer representations, class separation over 7 different chosen loss function and variants. The paper delivers a very clear message despite being limited in some ways such as:\n\n- experiments with non-resnet CNNs\n- experiments with non-CNN models, such as VIT\n- analysis of similar papers (for example big transfer)\n- the role of self-supervised pretraining and if better self-supervised models lead to better transfer over all domains (but that might be a whole research project in itself)\n\nThe main role of Figure 2 is also a bit confusing to me, as in what is it trying to show. If I understand the plot correctly, is it suggesting that similar features are learned despite the choice of the loss function? Please correct me if I am mistaken. In lines 167-170 the paper suggests that there are a lot of similarities in the early layers but not as many in the later layers, but I am unsure if that is the case since the diagonal appears to be somewhat similar except in certain cases (softmax vs squared error for example), and the differences in the later layers are somewhat expected.\n\nThe paper is missing a couple citations in the form of [1] which reaches a somewhat different conclusion to the paper and [2] which also talks about how tightly the clustering is in the representation space and how tighter clusters hurt the downstream transfer properties (very similar to the class separability conclusions). It might be also useful for the paper to separate itself more from Kornblith et al. 2018. \n\n[1] http://proceedings.mlr.press/v97/recht19a.html\n[2] https://arxiv.org/abs/2002.08473", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is an experimental study in transfer learning and therefore may have societal impact in the form of harmful features being learned during training. The paper does not discuss this directly, however the experimental study is somewhat orthogonal to the societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "4nx8nlOlxQ4", "reviewer": "Reviewer_MRUN", "summary": "The authors study how different loss functions and regularizers affect the learned representation of a ResNet-50. They find that although sigmoid, label smoothing, etc. can improve ImageNet performance, the learned representations do not transfer as well as when training with softmax cross-entropy. The main hypothesis is that softmax attains embeddings with lower inter-class separation and higher intra-class variance, which is verified empirically by computing the $\\frac{d_{intraclass}}{d_{total}}$. In addition, they also use centered kernel alignment (CKA) to compare representations learned with different loss functions at different depths of the neural networks. They observe that most of the representational differences between models trained with different loss functions happen closer to the output. As a result, fine-tuning the models on new data is enough to eliminate the performance discrepancies observed when linear probing.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Review\nI found this study interesting and necessary in order to better understand how different design choices affect the performance of deep learning models. The text is well-written, the experiments are sound, and the conclusions are instructive. I have some questions regarding the experimental setup (see below), particularly on the choice of loss functions and optimizer. For the moment I think that the strengths outweigh the weaknesses of this work and thus I recommend its acceptance.\n\n## Strengths\n* Understanding how the choice of loss functions affect the learning of deep representations is an important topic.\n* The authors explore the problem from different perspectives like CKA, class separation, simplex ETF measurements, angular visual hardness, and representation sparsity.  \n* The text is clear and well-written.\n\n## Weaknesses\n* The choice of optimizer and loss function is not clear (see detailed comments). \n* Transfer results are only shown from ImageNet to other datasets. It would have been interesting to see what happens when trying to transfer from another dataset (see detailed comments).\n \n\n# Detailed comments\n## Originality\n* To the best of my knowledge this is the first work exploring transferability of different pre-existing loss functions. There are previous works on meta-larning more transferable functions [B, C], previous studies on different loss functions [D], and specific loss functions that address the intraclass compactness problem [E] but they are orthogonal to the work presented in this paper. \n\n[B] Li, Chuming, et al. \"Am-lfs: Automl for loss function search.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n\n[C] Huang, Chen, et al. \"Addressing the loss-metric mismatch with adaptive loss alignment.\" International Conference on Machine Learning. PMLR, 2019.\n\n[D] Janocha, Katarzyna, and Wojciech Marian Czarnecki. \"On Loss Functions for Deep Neural Networks in Classification.\" Schedae Informaticae 25 (2016).\n\n[E] Luo, Yan, et al. \"$\\mathcal {G} $-softmax: improving intraclass compactness and interclass separability of features.\" IEEE transactions on neural networks and learning systems 31.2 (2019): 685-699.\n\n## Quality\n* The overall quality is good, in particular on the experimental part with with experiments on 9 different loss functions over 8 different datasets (+ImageNet versions), and the evaluation over multiple metrics. For instance, given the current interest in self-supervised learning, it would have been interesting to try a contrastive loss such as the one described in [F]\n* However it is not clear why you chose those particular loss functions. For instance, benchmarking a meta-learned loss function could have provided more insight on whether meta-learning-based loss functions overfit the meta-training set.  Could you comment on this point? \n* SGD is sensitive to the magnitude of the loss, thus, different loss functions should have a different learning rate. Do you account for this fact? One way to alleviate this problem is using Adam.\n\n[F] Khosla, Prannay, et al. \"Supervised Contrastive Learning.\" Advances in Neural Information Processing Systems 33 (2020).\n\n## Clarity\n* In general, the text is well-written and easy to follow. \n* There is a minor typo in the supplementary material in the caption of Table B3 (\"see Table ??\").\n\n## Significance\n* The proposed study is sound and interesting for the research community. The experiments are extensive and insightful. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations but not the negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper performed an empirical study on the impact of loss functions to the transferability of deep features for downstream tasks, and pinpointed several interesting findings which are previously unknown. Reviewers generally believed that this study is novel and significant to the community, the experimentation with several quantitative measures is solid, and the findings are conclusive and inspiring. Reviewers also raised some relevant concerns on the limited scope of this study, such as the limitation to ImageNet pre-training and less interesting downstream datasets, and to vision classification backbones and SGD optimizers. While these concerns are way of exhaustivity, authors are encouraged to extend the study to at least more interesting downstream datasets (such as Chest X-ray in the transfusion work, NeurIPS 2019) embodying a larger diversity of domain shifts, because this is the most relevant perspective to make their findings more conclusive for practical problems.
Decision: Accept (Poster)