Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies an interesting problem of how the representation features learned by different loss functions affect the performance on downstream tasks. They reveal some interesting patterns, such as better class separation on pre-training dataset does not guarantee better performance on downstream tasks and it can be caused by “overfitting” on original task. Overall, this paper is clearly written and easy to follow. The numerical results are also well-presented.
main_review: Pros:
1) The authors present detailed studies on the choice of loss functions, comparison of transferability, etc., and draw several interesting conclusions. 
2) Overall, I find this paper is clear and well-presented. 

Cons:
1) ResNet is a sophisticated architecture, however, it needs more evidence to draw general conclusions about transferability, especially for section 3.2. For example, if we don’t use the residual connection, can we observe similar patterns?

2) The choice of downstream tasks is somewhat limited. To better demonstrate the transferability, different types of downstream tasks should be considered, e.g., object detection.

3) In line 224-230, the authors illustrate that representations with better class separation can perform better on downstream tasks if the classes are overlapped. So, I wonder how many overlapped classes are in these 8 selected datasets. 
limitations_and_societal_impact: This has been addressed in the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the impact of common supervised objectives and regularization on transfer learning. With extensive experiments, the authors find out that although these objectives (or regularizers) lead to better performance on the pre-training dataset, their performance on the downstream tasks are always worse. This complements the results in Kornblith et al.. Furthermore, the authors try to probe into the reason of this phenomenon, and find out that these objectives induce better class separation, which is associated with less transferable features.
main_review: This paper studies a very important problem in transfer learning. The manuscript is well written. The claims are stated clearly in each section and supported with thorough experiments. However, I believe several aspects of the manuscript can be improved in its current form.

1)	Although the conclusion of the paper is interesting and solid, I can be limited to the current setup. When the pretraining and target tasks are very similar (in extreme terms, the same), better performance on the pretraining task can lead to better transfer learning. As the similarity decreases, the performance of the pretraining and target tasks are less correlated, and we may enter the regime of the current paper. 
2)	The findings of section 3.2 is not surprising as it is common sense that neural networks tend to learn similar “low-level” features in lower layers, and lower layers are more important in fine-tuning. In addition, the accuracies of different objectives vary only by ~1%, which can be too small for an informative CKA visualization.
3)	As to the class separation, many pre-training tasks are not supervised classification, which do not involve it. Even if greater class separation is associated with less transferable features, it is hard to say class separation determines transferability, which cast doubt on the value of this claim.

Post-rebuttal:  
After reading the rebuttal and other reviews carefully, I found out some of my concerns addressed. On the other hand, I still believe some part of the experiments and the conclusion of the paper are limited:  
1. The downstream tasks are limited to classification. It is fine to me to study only the computer vision setup, but it is necessary to include other type of downstream tasks such as object detection. Even for classification tasks, there exists datasets which are very different from those chosen in this paper. I don't think the 8 datasets are representative of downstream tasks in practical use.  
2. As to the concrete value of the paper, an interesting direction would be a loss function which decreases the separation for more transferability.  

I would like to increase my score to 5, and I strongly recommend the authors to incorporate these points into future versions.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The development of new loss functions and regularizers is a major domain of ML research, but researchers often prioritize accuracy on the original dataset over transfer learning. The authors of the present work wanted to understand the effects of different training objectives on transferability in image classification tasks. They find that many loss functions outperform the traditional softmax cross-entropy but transfer worse, and provide insight into this phenomenon by examining hidden layer representations: higher original-dataset accuracy collapses within-class variability in deeper layer representations, leading to less generalizable features. 
main_review: The authors comprehensively characterize the effects of different loss functions empirically, not just on transfer learning, but also many performance metrics of interest including accuracy, out-of-distribution/naturalistic robustness, and calibration. I am also convinced by their analysis-based approach to understanding the mechanisms underlying the observed effects of different loss functions on different performance metrics. The work is broadly relevant beyond simply understanding and improving transfer learning. And while I would like to see this work expanded beyond the ResNet50 architecture, I understand that replicating the entirety of this work on larger models would be very resource-intensive.

This is a very thorough paper. 

I have a few minor comments:

Optimal validation accuracy is typically achieved after 140-200 epochs of training, which leads to the question of whether the results are consistent across the duration of training. The answer to this is interesting from a scientific perspective, but there is also a practical consideration: it would be very helpful for other ML practitioners to know whether they can expect to see benefits from alternative loss functions at shorter training durations. At the very least, I think the authors should include a plot of validation accuracy as a function of epoch for the different loss functions.

I think it would be helpful to have a version of Figure 1 with the full transfer learning results and/or some measure of variability across different transfer datasets.

The authors state “The relationship between class separation and pretraining accuracy is non-monotonic; squared error achieves the highest class separation but does not significantly outperform vanilla softmax cross-entropy.” I would appreciate seeing some discussion (and ideally analysis) of the mechanisms by which accuracy can be improved other than by increasing class separation in the penultimate layer. Also, perhaps it was cut for space, but there is no motivation provided for why the authors chose to examine class separation in the penultimate layer. It seems like a perfectly sensible property to examine, but the transition to this analysis feels a bit abrupt.

The manuscript could be enriched with some discussion about the very interesting result that none of the loss functions have worse original-dataset accuracy or better transfer accuracy than softmax-cross-entropy, though I understand space is at a premium.

limitations_and_societal_impact: The authors adequately address the limitations of their work in the space provided.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors study how different loss functions and regularizers affect the learned representation of a ResNet-50. They find that although sigmoid, label smoothing, etc. can improve ImageNet performance, the learned representations do not transfer as well as when training with softmax cross-entropy. The main hypothesis is that softmax attains embeddings with lower inter-class separation and higher intra-class variance, which is verified empirically by computing the $\frac{d_{intraclass}}{d_{total}}$. In addition, they also use centered kernel alignment (CKA) to compare representations learned with different loss functions at different depths of the neural networks. They observe that most of the representational differences between models trained with different loss functions happen closer to the output. As a result, fine-tuning the models on new data is enough to eliminate the performance discrepancies observed when linear probing.
main_review: # Overall Review
I found this study interesting and necessary in order to better understand how different design choices affect the performance of deep learning models. The text is well-written, the experiments are sound, and the conclusions are instructive. I have some questions regarding the experimental setup (see below), particularly on the choice of loss functions and optimizer. For the moment I think that the strengths outweigh the weaknesses of this work and thus I recommend its acceptance.

## Strengths
* Understanding how the choice of loss functions affect the learning of deep representations is an important topic.
* The authors explore the problem from different perspectives like CKA, class separation, simplex ETF measurements, angular visual hardness, and representation sparsity.  
* The text is clear and well-written.

## Weaknesses
* The choice of optimizer and loss function is not clear (see detailed comments). 
* Transfer results are only shown from ImageNet to other datasets. It would have been interesting to see what happens when trying to transfer from another dataset (see detailed comments).
 

# Detailed comments
## Originality
* To the best of my knowledge this is the first work exploring transferability of different pre-existing loss functions. There are previous works on meta-larning more transferable functions [B, C], previous studies on different loss functions [D], and specific loss functions that address the intraclass compactness problem [E] but they are orthogonal to the work presented in this paper. 

[B] Li, Chuming, et al. "Am-lfs: Automl for loss function search." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

[C] Huang, Chen, et al. "Addressing the loss-metric mismatch with adaptive loss alignment." International Conference on Machine Learning. PMLR, 2019.

[D] Janocha, Katarzyna, and Wojciech Marian Czarnecki. "On Loss Functions for Deep Neural Networks in Classification." Schedae Informaticae 25 (2016).

[E] Luo, Yan, et al. "$\mathcal {G} $-softmax: improving intraclass compactness and interclass separability of features." IEEE transactions on neural networks and learning systems 31.2 (2019): 685-699.

## Quality
* The overall quality is good, in particular on the experimental part with with experiments on 9 different loss functions over 8 different datasets (+ImageNet versions), and the evaluation over multiple metrics. For instance, given the current interest in self-supervised learning, it would have been interesting to try a contrastive loss such as the one described in [F]
* However it is not clear why you chose those particular loss functions. For instance, benchmarking a meta-learned loss function could have provided more insight on whether meta-learning-based loss functions overfit the meta-training set.  Could you comment on this point? 
* SGD is sensitive to the magnitude of the loss, thus, different loss functions should have a different learning rate. Do you account for this fact? One way to alleviate this problem is using Adam.

[F] Khosla, Prannay, et al. "Supervised Contrastive Learning." Advances in Neural Information Processing Systems 33 (2020).

## Clarity
* In general, the text is well-written and easy to follow. 
* There is a minor typo in the supplementary material in the caption of Table B3 ("see Table ??").

## Significance
* The proposed study is sound and interesting for the research community. The experiments are extensive and insightful. 

limitations_and_societal_impact: The authors have addressed the limitations but not the negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the role of downstream performance on the given task and its role on transferring to a novel data distribution. The paper rigorously studies the role of different loss functions and its effect on transfer learning (with or without finetuning) while also investigating other learned features of the NN. The paper concludes that learning better downstream discriminative features may not lead to better transfer learning models.
main_review: The paper is well studied and a clear investigation into learning transferrable features and how learning better downstream classifiers might lead to slightly worsening performance of the models. The study investigates various properties of the learned network such as sparsity, early layer representations, class separation over 7 different chosen loss function and variants. The paper delivers a very clear message despite being limited in some ways such as:

- experiments with non-resnet CNNs
- experiments with non-CNN models, such as VIT
- analysis of similar papers (for example big transfer)
- the role of self-supervised pretraining and if better self-supervised models lead to better transfer over all domains (but that might be a whole research project in itself)

The main role of Figure 2 is also a bit confusing to me, as in what is it trying to show. If I understand the plot correctly, is it suggesting that similar features are learned despite the choice of the loss function? Please correct me if I am mistaken. In lines 167-170 the paper suggests that there are a lot of similarities in the early layers but not as many in the later layers, but I am unsure if that is the case since the diagonal appears to be somewhat similar except in certain cases (softmax vs squared error for example), and the differences in the later layers are somewhat expected.

The paper is missing a couple citations in the form of [1] which reaches a somewhat different conclusion to the paper and [2] which also talks about how tightly the clustering is in the representation space and how tighter clusters hurt the downstream transfer properties (very similar to the class separability conclusions). It might be also useful for the paper to separate itself more from Kornblith et al. 2018. 

[1] http://proceedings.mlr.press/v97/recht19a.html
[2] https://arxiv.org/abs/2002.08473
limitations_and_societal_impact: The paper is an experimental study in transfer learning and therefore may have societal impact in the form of harmful features being learned during training. The paper does not discuss this directly, however the experimental study is somewhat orthogonal to the societal impact.
review_title: null

Meta-review: This paper performed an empirical study on the impact of loss functions to the transferability of deep features for downstream tasks, and pinpointed several interesting findings which are previously unknown. Reviewers generally believed that this study is novel and significant to the community, the experimentation with several quantitative measures is solid, and the findings are conclusive and inspiring. Reviewers also raised some relevant concerns on the limited scope of this study, such as the limitation to ImageNet pre-training and less interesting downstream datasets, and to vision classification backbones and SGD optimizers. While these concerns are way of exhaustivity, authors are encouraged to extend the study to at least more interesting downstream datasets (such as Chest X-ray in the transfusion work, NeurIPS 2019) embodying a larger diversity of domain shifts, because this is the most relevant perspective to make their findings more conclusive for practical problems.
Decision: Accept (Poster)