Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper benchmarks various  training objectives, including loss functions and regularization schemes and measures accuracy, calibration and out-of-distribution robustness. The paper also studies how these objectives affect representations in various layers of the network.


Pros
+ Paper is well written, and the narrative is coherent.
+ Informative plots for the most part.
+ Numerous experiments.

Cons
- Unfortunately, novelty is very limited and this is mostly a restating/confirmation of results of earlier work.
- Examples:  Sec 3.1 "Regularizers and alternate losses increase accuracy" -- what is new here? We already know this, otherwise these alternatives to vanilla cross-entropy wouldn't exist.
- Sec 3.2 "Different losses produce different predictions" : again, why is this surprising? If the losses produced the same predictions, then they would all have the same accuracy. And we know this is not the case.
- "Mixup and auto-augment improve performance " : already known.  
- "label smoothing improves calibration" : already known

 The only possibly novel contribution is that losses behave similarly in the earlier layers, with most of the separation happening at the final layers. But this line --i.e., entanglement or separation behaving differently  in earlier vs. later blocks -- was already explored in Frosst et al 2019, so again here the novelty impact is limited. 


minor
- dendrogram isn’t very intuitive. can the authors show a 2-d cluster representation of the model predictions (for all seeds, and for all losses?)

Suggestions on improving novelty:
- For a conference devoted to learning representations, the authors should investigated  into why, for example,  different random seeds of squared error loss disagree with each other so much, while some other losses are more self-consistent.

- Different losses produce different predictions. but given that top-1 accuracies are nearly the same (table 1), the difference in figure 2 must  mostly be contained in the predictions where the classifiers were wrong (respect to ground truth). in other words, there shouldn’t be a significant difference in the cases where the classifiers agree with ground truth -- can the authors look into this?

- At the start of the paper it is stated that  "our goal is instead to understand when one might want to use
one loss function or regularizer over another" : however, I do not see the paper providing any additional insights into this beyond what is already known.

In summary, lot of space is devoted to confirming earlier results, and while the  benchmarking is appreciated, a paper purely devoted to benchmarking but that only ends up confirming earlier results does not meet the bar.


References:

+ Frosst et al .Analyzing and Improving Representations with the Soft Nearest Neighbor Loss, ICML 2019
review_title: Well written paper with comprehensive experiments, but lacking novelty of results

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a study on different loss functions and regularizers that apply on the penultimate layer and show how the choice of the loss function/regularizer can affect the performance of classifiers. The authors experiment with image datasets and study multiple parameters and side effects of the loss functions/regularizers.

This is an interesting paper with multiple experiments. It is a practical paper and provides multiple results on experiments for image classification. It is easy to follow and flows well.

Some weak points are the following;

- I find the datasets a bit limited and I think it would be nice if the authors could add extra datasets that go also beyond image classification.
- In the same spirit what it misses is an extension on a large-scale scenario. Many practical applications have very large output space. Experiments on such scenarios would help to better understand benefits/limitations of the different loss functions. Also, the authors could extend to other objectives like negative sampling etc.
- Some analysis on the individual classes is missing also. So, what do you observe when you have imbalanced classes?

As most of the impact is on the penultimate layers would it make sense to have a couple of experiments where you tune a model on another task. If I am not mistaken you trained from scratch.  It would be nice for example to have some results from another domain also like an NLP task.

Overall I think is a good paper.
review_title: Very interesting work on practical aspects for training DNN models.

final_score: rating: 3: Clear rejection
review_content: The paper attempts to analyze the softmax cross entropy loss and discuss useful properties it has in an intuitive and accessible way. The paper mainly presents comparisons between this loss function and two other (very standard as well) loss functions that are the "sigmoid cross entropy" and the "squared error" loss functions. 

It is very interesting to read the paper and realize that the topics discussed here have been written in well cited and well established statistics and ML books for decades. The softmax cross entropy loss is the multinomial regression loss while the sigmoid cross entropy loss is no other than the standard binary  logistic regression loss and finally the "squared error" is the least squared loss. 

Authors compare the performance of these three on a few benchmarks. 

I think it's good to read the "Elements of Statistical Learning" and get a clear exposition to what science has been up to for decades before getting too deep into deep learning. 
review_title: review of "DEMYSTIFYING LOSS FUNCTIONS FOR CLASSIFICATION"

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors give a pretty thorough investigation of several existed losses of training deep networks in the task of supervised classification. The differences objectives are compared in terms of accuracy, calibration, out-of-distribution robustness, and predictions. 
Several discoveries are presented from the experiments:
1) Regularizers and alternative losses did improve over vanilla softmax loss. But the effect of improvement is not additive.
2) Different objectives mainly affect the last few layers while the layers not close to the output seems similar.
3) All objectives that improve over vanilla softmax loss produce greater class separation in the penultimate layer of the network.

Pros:
+ A comprehensive analysis of different behaviors of different objectives.
+ The experiments setup seems fair.

Cons:
- The authors claimed 
'''Our goal is instead to understand when one might want to use one loss function or regularizer over another and, more broadly, to understand the extent to which neural network performance and representations can be manipulated through the choice of objective alone.'''
I wonder if the study in this paper realizes the goal that the authors claimed. It seems it is hard to make a conclusion to decide which single objective performs better than others, even in the task of classification, with the existing methods and optimization methods. It seems to have little insight for future works that using novel methods or optimization frameworks.

I respect the efforts that the authors made to give a thorough analysis of existing common-used objectives. But since little novel technical contribution is made and the results seem not surprising, I am not sure whether such work is above the threshold of ICLR.
review_title: An investigation paper of existed loss functions. Seems no surprising results.

Meta-review: The paper presents an extensive empirical evaluation of several loss functions and regularization techniques used in deep networks. The authors conclude that the classical softmax is significantly outperformed by the other approaches, but there is no clear winner among them. Moreover, the authors have noticed two interesting facts, (1) the choice of loss function affects only upper layers of neural networks with the lower layers being very similar to each other, (2) losses that result in greater class separation also result in higher accuracy, but their features are less transferable to other tasks.

I agree with the authors that the comments of Reviewer 2 are shallow and not informative. Therefore, they were not taken into account in making the final decision and, as AC, I read the paper very carefully. Regarding Reviewer 4, however, I found his comments to be valid. There is a message that the authors want to communicate, and the reader that needs to decode this message using a noisy channel. Therefore, I encourage the authors to accordingly revise the paper to make the message much clearer. 

The experimental papers that compare a wide spectrum of methods are always hard to judge and this judgement is often very subjective. There are several seminal papers of this type, but not so many for several reasons. I agree with the authors that such studies are very valuable and give evaluation being not biased by authors of a given method. They are also very time- and resource-consuming. But there should be a general consensus how such an experiment should be performed. The authors of the particular methods should also be able to give right feedback to make their methods to be run appropriately. Therefore, there exist several websites and initiatives that try to fulfill these requirements. As said above, any paper of this type will be judged very subjectively.  

The discoveries made by authors should also be presented in a different way. One should start with a hypothesis that, for example, the lower layers are not affected by the loss function and then perform appropriate theoretical and empirical studies to verify the hypothesis. The same applies to the other discovery. In that way the message of the paper would be much clearer. I suppose that analysis of each of the discoveries deserves its own paper. 


Decision: Reject