Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors theoretically investigate the square loss in classification for over-parametrized neural networks in NTK regime. They reveal properties regarding the generalization error, robustness and calibration error. Separable and non-separable cases are considered. In the non-separable case, they establish fast convergence rate  for misclassification rate and calibration error. While in separable case, the misclassification rate improves to be exponentially fast. 

Further, they prove a margin lower bounded away from zero, and claim theoretical guarantees for robustness. They also conduct experiments with practical neural networks to support and extend the theoretical claims.

The main conclusion is that compared with the cross-entropy loss, the square loss has comparable generalization error while noticeable advantages in robustness and model calibration.
strengths_and_weaknesses: Strengths:

1) This paper extends the study of square loss in classification with more theoretical understanding, with systematical investigations on several import properties: generalization error, robustness and model calibration.

2) The main conclusion is well-supported by theoretical proofs and some empirical results.

3) The paper is well-written and easy to follow.

Weaknesses:

The experimental results is not very convincing as it mainly considers CIFAR-10 dataset.
questions: 1) It was widely believed that training with the square loss in classification suffer from the 'outliers' problem, and it was not robust. 
How do your claim on robustness solve the concerns on the 'outlier'  issue?

2) Are the results of the square loss in Table 1 with simplex coding? As in large-class number case, previous works show that the original square loss with one-hot encoding may not work well, it seems in Table 1 the square loss results on CIFAR-100 are way better than the cross-entropy. How simplex encoding make a difference with the one-hot encoding? 
limitations: This paper extends the study of the square loss in classification by providing systematic theoretical study, which is import for people to consider other possible alternatives as training objectives in classification.

The assumptions are a little bit strong which may not hold in practice, and the experiments are a little bit limited for practitioners to be convinced. 

Overall, the paper indeed provides more theoretical understanding which is an important contribution to the study of square loss in classification.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper conducted comprehensive review in research, and compare the loss function between Cross Entropy (CE) loss and Square Loss(SL).  The analytical results show that compared to CE, SL has comparable generalization error, but noticeable advantage in robustness and calibration. Experimental results from synthetic data and real data further demonstrate the claim of the paper.
strengths_and_weaknesses: The paper comprehensive summarized the areas and related results. The paper is well written and clearly organized. The theoretical results of CE calibration issue aligned with the observations and the learning from the theoretical results helps to understand the gaps in practice. 

Some hypothesis and learning from the theorem might need more clarification. Experimental results might need further improvement to demonstrate the claim made in the theorem. More details can be found from questions sections. 
questions: [1] Theorem 3.1 shows the gap between loss with neural networks and loss with minimize 0-1 loss. As mentioned in equation 2.1, one minimizer is L*., which might not be the optimal minimizer. Could the author help to clarify if L* is bounded with optimal loss with limited error?

[2] From line 62 in page 2, it shows the mis-calibration of CE in prediction region close to 0 or 1 has been shown in paper [9]. Similar results are also covered in Theorem 3.4, between line 248 and 256 in page 6. Could the author help to confirm the originality of the results and if any gaps or difference between this paper and paper [9]?

[3] From the theory analysis, it seems CE tends to weakly calibrated in the high predicted or low predicted region (Theorem 3.4). However, from the experimental results in Fig 2, it seems the mis-calibration happens in low prediction region (confidence < 0.1) and normal prediction region (confidence between 0.5 and 0.8). Does the experimental result align with the theorem?

[4] From the experimental results from Table 1, it seems SL is not always better than CE, although SL seems to be better than CE when under attack. Are there any hypothesis or interpretation to show the situation when CE will outperform than SL?


limitations: The paper shows the advantage of SL over CE, but not the dis-advantage of SL compared to CE. Might need more insights to compare the disadvantage of SL, and explain the the performance gap between two loss function in experimental results from Table 1.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors explore the use of square loss in classification settings, with both one-hot and scaled simplex features. They motivate the study by invoking improvements in model calibration and feature selection that have been theorized/observed about the use of square loss in classification settings.

The authors then study the generalization error and model calibration in the large width Neural Tangent Kernel (NTK) setting. Under a particular set of conditions on the dataset, they derive non-trivial bounds for the generalization error and the calibration as a function of the number of data points n.

They conclude with experiments showing that using squared loss with scaled simplex features improves performance on CIFAR. They also show that squared loss provides additional robustness to adversarial examples.
strengths_and_weaknesses: The analysis of the generalization error and calibration is novel and interesting. I believe that it is correct, though the arguments are long and it is possible I've missed an issue there.

The authors mention that not much work exists in terms of generalization bounds for wide neural networks under squared loss; however, there has been much recent work in this area in the context of the simultaneous scaling of width and data (http://proceedings.mlr.press/v119/huang20l.html, https://proceedings.mlr.press/v119/adlam20a.html). Comparing the author's results to these scenarios would be interesting and useful to position the paper in the literature.

Overall the empirical results seemed the most promising. The fact that both training and adversarial attacks were studied helped solidify the claim that both generalization and calibration can be improved by using square loss. Overall I find the empirical results even more compelling than the theoretical ones due to their careful presentation. I think the experiments could be strengthened by exploring the efficacy of squared loss on datasets with a large number of classes (Imagenet, language modeling), where the SOTA models are far from the NTK regime studied in the theory section of the paper.
questions: How close do the bounds in equations 3.1 and 3.4 compare to empirical behavior? I would be curious to know how tight they are even in toy examples. I'd also be curious about the relationship between the bounds in this paper and those in http://proceedings.mlr.press/v119/huang20l.html and https://proceedings.mlr.press/v119/adlam20a.html.

What do results of squared loss look like on tasks with a large number of classes, like imagenet or language modeling?

Is there any way to visualize/quantify the difference in decision boundaries between the cross entropy and squared loss cases? This seems particularly interesting in the adversarial setting.
limitations: Adequately addressed
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provides solid arguments in favor of using MSE loss instead of the most common CE loss. From several aspects, showing that:
- MSE is better calibrated.
- It has a better generalization bound.
- Is more robust to adversarial perturbations.
strengths_and_weaknesses: ** Strengths**:
- I loved the structure of this work. Well-motivated, properly stated, and provide a good balance between the theory and intuition.
- I found the results to be novel and significant.
- The experiments appear sufficient to support the claims.
- The authors are fair in their evaluations. For example, I enjoyed reading your argument on why white-box attacks appear to fail with MSE.

** Weaknesses**:
- There are a few citations missing. I know at least the following three which advocate using MSE instead of CE:
    + "Excessive invariance causes adversarial vulnerability" by Jacobsen et al
    + "Cross-entropy loss and low-rank features have responsibility for adversarial examples" by Nar et al.
    + "Gradient starvation: A learning proclivity in neural networks" by Pezeshki et al.
- The writing can benefit from a grammar check. For example, "it's" --> "it is".
- The use of separable / non-separable terminology was initially very confusing for me. Generally "separable" refers to a case that data is linearly separable or the model has enough capacity to fully classify the data. However, here, authors refer to label ambiguity.
- The extension to multi-class using simplex coding is rather difficult to follow.
- The code for the experiments is not provided.

questions: - CE is shown to converge to the max-margin solution. So I don't fully understand Thm 3.3. Yes, MSE's margin is bounded from below but is still worse to equal to margin of CE. I am right?
- I do not fully understand Fig. 2. How do you measure the confidence? The reason I'm asking this is because the output with MSE is real valued and not a probability. No?
limitations: N/A
review_title: null

Meta-review: metareview: This paper provides a theoretical investigation of square loss in the NTK regime due to a recent observation: empirical evidence seems to promote square loss other than cross-entropy loss. The authors provide generalization error (in this work it refers to population risk) bound, calibration error, and robustness (in terms of lower bounds of the margin). The theoretical analysis justifies the benefits of square loss. Reviewers all agree that the analysis is novel. Three reviewers think the experimental results might need further improvement. I suggest the authors to strengthen the experiments in the final version. Overall, I recommend accept. 

confidence: Certain
Decision: Accept