Understanding Square Loss in Training Overparametrized Neural Network Classifiers
Authors: Tianyang Hu
Jun Wang
Wenjia Wang
Zhenguo Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Nonparametric classification
square loss
generalization error
robustness
calibration error
neural tangent kernel

Abstract: Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.

Understanding Square Loss in Training
Overparameterized Neural Network Classifiers
Tianyang Hu" Huawei Noah's Ark Lab hutianyangl www.anynnnnguuwwi....m.m @huawei.
Jun Wang" HKUST j jwangfxccomect. ust.hk
com
Wenjia Wang" HKUST (GZ) and HKUST
Zhenguo
Huawei Noah's s Ark Lab 1i. zhenguo@huaweii. com
senjiawang@ust. hk
Abstract
Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but theoretical justification is still lacking. In this work, we contribute the theoretical un- derstanding of square loss in classification by systematically investigating how it performs for overparameterized neural networks the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robust- ness, and calibration error are revealed. We consider two cases, according whether classes are separable or not. In the general non-separable case, fast con- vergence rate established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies practical neural networks, demonstrating the effec- tiveness of square loss both synthetic low-dimensional data and real image data. Comparing cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration. Code will be available at http://tcccooomooccceeeeeeeeeeee
Introduction
The pursuit of better classifiers has fueled the progress of machine learning and deep learning research. The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provide test fields for all kinds of classification models, especially those based deep neural networks (DNN). With the introduction of CNN, ResNets, and transformers, DNN classifiers are constantly improving and catching up to the human- performance. In contrast the active innovations in model architecture, the training objective remains largely stagnant, with cross-entropy loss being the default choice. Despite its popularity, cross-entropy has been shown to problematic in some applications. Among others, M argued that features learned from cross-entropy lack interpretability and proposed new loss aiming for maximum coding rate reduction. linked the use of cross-entropy adversarial
'Thsse authors contributed equally to this work. Corresponding author: Wenjia Wang.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

vulnerability and proposed new classification loss based on latent space matching. B discovered that the confidence of most DNN classifiers trained with cross-entropy is not well-calibrated.
Recently, several alternative losses have seen revived interests for deep classifiers. In particular, many existing works have presented empirical evidence promoting the of square loss over cross- entropy. A conducted large-scale experiments comparing the two and found that square loss tends to perform better in natural language processing related tasks while cross-entropy usually yields slightly better accuracy in image classification. Similar comparisons are also made in [ 7 M 99 compared variety of loss functions and output layer regularization strategies on the accuracy and out-of-distribution robustness, and found that square loss has greater class separation and better performance.
In comparison to the empirical investigation, theoretical understanding of square loss in training deep learning classifiers is still lacking. Through our lens, square loss has its uniqueness among classic classification losses, and we argue that it has great potentials for modern classification tasks. Below we list our motivations and reasons why.
Explicit feature modeling Deep learning's s success can be largely attributed to its superior ability as feature extractors. For classification, the ideal features should be separated between classes and concentrated within classes. However, when optimizing cross-entropy loss, it is not obvious what the learned features should look like [T. In the terminal stage of training, [0 proved that when cross-entropy is sufficiently minimized, the penultimate layers features will collapse to the scaled simplex structure. Such phenomenon is referred to as "neural collapse". Knowing this, would it be better to directly enforce the terminal solution by using square loss with the simplex coding ΑاЛ. as Euclidean distance is probably most natural to measure the distance between samples' features and class maasmmeans? Unlike cross-entropy, square loss uses the label codings (one-hot, simplex etc.) as features, which can explicitly control class separations.
Model calibration An ideal classifier should not only give the correct class prediction, but also with the correct confidence. Calibration error measures the closeness of the predicted confidence to the underlying conditional probability y). Using square loss in classification can be essentially viewed as regression where treats discrete labels as continuous code vectors. can be shown that the optimal classifier under square loss is 2n- linear with the ground truth. This distinguishing property allows it to easily recover r7. In comparison, the optimal classifiers under the hinge loss and cross-entropy are sign(21) 1) and log( respectvelly [12]. Therefore, hinge loss doesn't 't provide reliable information on the prediction confidence, and cross-entropy be problematic when is close to 0 or 1, as stated in [12.. Hence, in terms of model calibration, square loss is a natural choice.
Connections to popular approaches Mixup B3 is a data augmentation technique where aug- mented data are constructed via convex combinations of imputs and their labels. Like in square loss, mixup treats labels as continuous and is shown to improve the generalization of DNN classifiers. In knowledge distillation [14], where student classifier is trying learn from trained teacher, [I5] proved that the "best" teacher with the ground truth conditional probabilities provides the lowest variance in student learning. Since classifiers trained using square loss is a consistent estimator of n, one can argue that it is a better teacher. In supervised contrastive learning [16. the optimal features are the same as those from square loss with simplex label coding (details in Section 3.4).
Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked. In this work, we systematically investigate from statistical estimation perspective, the properties of deep learning classifiers trained using square loss. Comparing to cross entropy, the square loss is much more theoretically tractable, which allows us develop sharper results on the training process and generalization performance. The neural networks in our analysis are required to be sufficiently overparameterized in the neural tangent kernel (NTK) regime. Even though this restricts the implication of our results, it is a necessary first step towards a deeper understanding. In summary, our main contributions are:
Generalization error bound: We consider two cases, according to whether classes are separable or not. In the general non-separable case, we adopt the classical binary classi- fication setting with smooth conditional probability. Fast convergence rate is established for overparameterized neural network classifiers with Tsybakov's noise condition. If two

---

classes are separable with positive margin, we show that overparameterized neural network classifiers can provably reach zero misclassification error with probability exponentially tending to one. To the best of our knowledge, this is the first such result for separable but not linear separable classes. Furthermore, we bridge these two cases and offer unifted view by considering auxiliary random noise injection.
Adversarial robustness (margin property): When two classes are separable, the decision boundary is not unique and large-margin classifiers are preferred. The margin is naturally connected to adversarial robustness [8. the separable case, we show that the decision boundary of overparameterized neural network classifiers trained by square loss cannot be too close to the data support and the resulting margin is lower bounded away from zero, providing theoretical guarantees for adversarial robustness.
We show that classifiers trained using square loss are inherently well- calibrated, i.e., the trained classifier provides consistent estimation of the ground-truth conditional probability in Lx norm. Such property doesn' hold for cross-entropy.
Calibration error:
Empirical evaluation: We corroborate our theoretical findings with empirical experiments in both synthetic low www-dimensional data and real image data. Comparing to cross-entropy. square loss has comparable generalization error but noticeable advantages in robustness and model calibration.
This work contributes to the theoretical understanding of deep classifiers from a nonparametric estimation point of view, which has been classic topic in statistics literature. Among others, [9 established the optimal convergence rate for 0-1 loss excess risk when the decision boundary is
smooth. 12 20 extended the analysis to various surrogate losses. [2] 22 studied the convergence
rates for plug-in classifiers from local averaging estimators. 23 investigated the convergence
rate for support vector machine using Gaussian kernels. We build on and extend classic results to neural networks in the NTK regime. There exist nonparametric results on deep classifiers, e.g., fast convergence rates have been derived for DNN classifiers that minimize the empirical 0-1 loss 24 25
2 etc. Unlike aforementioned works that only concern the existence of a good classifier (with theoretical worst-case guarantee), ignorance of the tremendous difficulty of neural network optimization, our results further incorporate the training algorithm and apply to trained classifiers, which relates better to practice. To the best of the authors' knowledge, similar attainable fast rates (faster than n have never been established for neural network classifiers.
hinge loss [26]] and cross-entropy loss
We require the neural network to be overparameterizee, which has been extensively studied recently via NTK. Most such results are in the regression setting with a handful of exceptions. [2 showed that only polylogarithmic width is sufficient for gradient descent to overfit the training data using logistic loss. provided optimization and generalization guarantees for overparameterized network trained with cross-entropy. In comparison, our results are sharper in the sense that take the ground truth data assumptions into consideration. This allows a faster convergence rate, especially when the classes are separable, where the exponential convergence rate is attainable. The NTK framework greatly reduces the technical difficulty for our theoretical analysis. However, our results are mainly due to properties of the square loss itself and we expect them to hold for wide range of classifiers.
20 proved generalization error bound for regularized NTK in classification. BC 3l
There are other works investigating the use of square loss for training (deep) classifiers. uncovered that the "neural collapse" phenomenon also occurs under the square loss. 33 compared classification and regression tasks in the overparameterized linear model with Gaussian features, illustrating different roles and properties of loss functions used at the training and testing phases. иd made interesting observations on effects of popular regularization techniques such as batch normalization and weight decay on the gradient flow dynamics under square loss. These findings support our theoretical results' implication, which further strengthens our beliefs that the essence comes from the square loss and our analysis can go beyond NTK regime.
32
The rest of this paper is arranged as follows. Section 2 presents some preliminaries. Main theoretical results are in Section 3 The simplex label coding is discussed in Section 3.4 followed by numerical studies in Section and conclusions in Section 5 Technical proofs and details of the numerical
studies can be found in the Appendix.
3

---

2 Preliminaries
Notation For a function f: Ω2 + E. let If sup [f(z) and II Un (dx)) For vector æ, ||æ||, denotes its p-norm, for 1 S p p oo. Lp and lp are used to distinguish function norms and vector norms. For two positive sequences {an heN and (bn we write an X bn there exists constant C > 0 such that On I Cb, for all sufficiently large We write On x b,, if an X bn and bn S On Let [N] = (1.....
.......T N] for N E N, be the indicator function, and Ia be the d x d identity matrix. (1, Σ) represents Gaussian distribution with mean μ and covariance E.
Classification problem settings Let P be an underlying probability measure on x Y, where Let (X, () be random variable with respect to P. Suppose we have observations {.,, y: } C (Ω x Y)" ii.d. sampled according to P. The classification task is to predict the unobserved label y given new input
Ω C TRd is compact and Y = (1.-1). 13.
Ω. Let 7) defined on Ω denote the conditional probability, i.e., 17(x)) = P(y = 1 Let Px be the marginal distribution of P on X.
Remark 1. In this work, we consider the fixed dimension settings, i.e., d is fixed. As pointed by a reviewer, 35 36 discussed overparameterizee neural networks in high dimensional settings under regression. The study in the high dimensional settings will be pursued in the future.
According to whether labels are deterministic, there are two scenarios of interest. ry only takes values from {0, 1) i.e., labels are deterministic, we call this case the separable case. Note that in the separable case we consider, the classes are not limited to linearly separable but can be arbitrarily complicated, long as the label is deterministic. Let 22 { [<x)) = 1}. f2 {œ|n(x) = 0} and = Q U Q2.. If the probability measure of {æc (x)) E (0, 1)} is non-zero, i.c., the labels contain randomness, assume that there exists a positive margin, i.e., dist 02) 2 27 > 0, where y is constant, and dist(S4, 12) = inf wnf w EQ, | x'1l2- In the non-separable case, to quantify the difficulty of classification, we adopt the well-established Tsybakov's 's noise condition [1], which measures how large the "difficult region" is where ()x))) 1/2.
call this case the non-separable case. In the separable case, further
Definition 2.1 (Tsybakov's noise condition). Let 6. E [0, oo). We say P has Tsybakov noise exponent i: if there exists a constant C, T > 0 such that for all < t < T, Px (X) 1| < S t*.
large value of i implies the difficult region to small. is expected that a larger x: leads to a faster convergence rate of neural network classifier. This intuition is verified for the overparameterized neural network classifier trained by square loss and lg regularization. See Section 3 for details.
The key quantity of interest is the misclassification error, i.e., 0-1 loss. In the population level, the 0-1 loss be written
T(S)- = ((x.r)-p((iign((()))) #Y)
-Ex~Px (1- (1 2 +(X))x)))(( < 0H}.
(2.1)
One optimal minimizer of [(f) is 21p 1, and we define I," L(2n) - 1). fact, as long as the classifier has the same sign with 27(x) 1 for all E Ω. is optimal minimizer and achieves the minimum 0-1 loss.
Neural network setup We mainly focus on the one-hidden-layer ReLU neural network family F with m nodes in the hidden layer, denoted by fw (x) = m 1/2 Σm= a, x), where x E Ω W = (WI.... Wm)) E ddxm is the weight matrix in the hidden layer, = (a1. ... .mm) E R the weight vector in the output layer, (2) max(0., 2} the rectified linear unit (ReLU). The initial values of the weights are independently generated from (0) ~ (O, a, ~ unif(-1, 1}, Vr E [m]. Based on the observations ((x,, y1) } the goal of training neural network is to find solution to
min W
Σ 1( (2x), v.) + +R(W a),
(2.2)
where is the loss function, R is the regularization, and μ 2 the regularization parameter. Note in Equation 2.2 that we only consider training the weights W. This is because a-a(2) = sign(a)-o((al)))

---

which allows us to reparametrize the network to have all a, 's to be either or -1. In this work, we consider square loss associated with &g regularization, i.e., lffwa (x;), yi) (fw,a(2) - y,) and R(W, a) |
A popular way to train the neural network is via gradient based methods. It has been shown that the training process of DNNs can be characterized by the neural tangent kernel (NTK) K7 As is usually assumed in the NTK literature E8
39 40 we consider data the unit sphere Sd-1, .ic., و vi E [n]. and the neural network is highly overparameterized (m > ni) and trained by gradient descent (GD). For details about NTK and GD in one- ane-hidden-layer ReLU neural networks, we refer to Appendix A In the rest of this work, fW(K).a denotes the GDD-trained neural network classifier under square loss associated with & regularization, where l is the iteration number satisfying Assumption D.1 and W (k) is the weight matrix after k-th iteration.
29
3 Theoretical results
In this section, we present our main theoretical results, which consist of three parts: generalization erro, robustness, and calibration error. Throughout the analysis, we make the following assumptions on the data and the estimation model. Due to the page limit, we move the technical specification of the assumptions to Appendix D with detailed discussions.
Data assumptions We assume the ground-truth (z)) to be well-behaved (Ae (Assumption D.2 Op-
tionally, the marginal density is assumed to be upper bounded (Assumptions D.4p or both upper and lower bounded (Assumptions D.5). Assumptions D.4 and D.5 are standard in classical analysis
of classification in statisties literature 21 22
which covers large class of density functions.
Model assumptions We require the ReLU neural network to be sufficiently overparameterized (with finite width), and imposes conditions on the learning rate and iteration number (Assumption D.I:; similar settings have been adopted by 38 40). We also assume that the solution to (2. is well-behaved, i.e., the complexity of the neural network estimator generated by the GD training is controlled (Assumption D.3
3.1 Generalization error bound
In classification, the generalization error is typically referred to as the misclassification error, which can be quantified by L(J) defined Equation 2.1 In the non-separable case, the excess risk, defined by L(J)- L", is used to evaluate the quality of classifier S. where L L(2n) 1), which is the
minimum 0-1 loss, as stated after Equation 2.T Theorem 3.T states that the overparameterized neural
network with GD and le regularization
achieve a small excess risk in the non-separable case.
Theorem 3.1 (Excess risk in the non-separable case). Suppose Assumptions D.I
D.2 and D.4 hold.
Assume the conditional probability 7(x) satisfies Tsybakov's noise condition with component k. Let
11 X
Then
L(fw())a)= = L' +0 (21-1](043))
(3.1)
From Theorem 3.1 we can see that as i becomes larger, the convergence rate becomes faster, which is intuitively true. Generalization error bounds in this setting is scarce. To the best of the authors' knowledge, [29 is the closest work (the labels are randomly flipped), where the bound is in the order of Vn) Our bound is faster, especially with larger x. It is known that the optimal convergence rate
dín (1)
under Assumptions D.2 (with slightly different notion of the smoothness) and D.4 Op(n
21.. The differences between 3.1) and the optimal convergence rate is the extra (d 1)x in the d(s+1)
Y2 [a-1)44a=+1a-3 ). If the conditional probability vi has bounded I. Lipschitz constant, (22 showed that the convergence rate
denominator of the convergence rate in (3.1) (since n. [2
based on the plug-in kernel estimate is (n w(stz), which is slower than the rate in Equation 3.1 d is large.
Now we turn to the separable case. Since n only takes value from {0, 1) the separable case, v is bounded away from 1/2. Therefore, one trivially take x - co in Equation 3.1 and obtain the
5

---

convergence rate (e(n=d/(2d-1)) However, this rate can be significantly improved in the separable case, as stated in the following theorem.
Theorem 3.2 (Generalization error the separable case). Suppose Assumptions D.I D33 and D.5
hold. Let μr o(1). There exist positive constants C1. C2 such that the misclassification rate is 0% with probability least
C1 exp(-C2n)) and 5 can be arbitrarily small by enlarging the
neural network's width.
In Theorem 3.2 the regularization parameter can take any rate that converges to zero. In particular, can be zero, and the corresponding classifier overfits the training data. Theorem B.2 states that the convergence rate in the separable case is exponential, if sufficiently wide neural network is applied. This is because the observed labels are not corrupted by noise, i.e., PPy = [ is either one zero. Therefore, is easier to classify separable data, which is intuitively true.
3.2 Robustness and calibration error
If two classes are separable with positive margin, the decision boundary is not unique. Practitioners often prefer the decision boundary with large margins, which are robust against possible perturbation on input points 41 42] The following theorem states that the square loss trained margin can be lower bounded by positive constant. Recall that in the separable case, Ω = U2 U f2,, where Ω {x|n(x)) = 1} and f2 (xx) 0}.
Theorem 3.3 (Robustness in the separable case). Suppose the assumptions of Theorem 3.2 are satisfied. Let μ = o11. Then there exist positive constants C, C1, C2 such that Daanaeee,,'',,uuul - x' lll 2 C, and the misclassification rate is 0% with probability least 1 1- expeeppppp C2n) for all n, where Dr is the decision boundary, and d is as in Theorem 3.2
Theorem states that the square loss trained margin is robust the sense that the predicted label will not change in case of any noise whose la norm smaller than C. Since || 1zz-20 x z'1l 2 /d ||x x 2'112- Theorem 3.3 also indicates loc robustness. To the best of our knowledge, similar theoretical robustness guarantee has not been provided for any other loss functions. The most relevant work is 43), but their result is not on the population and doesn't apply to ReI.U networks trained via GD.
In the non-separable case, n)(=) varies within (0,1) and practitioners may only want clas- sifier with small excess risk, but also want to recover the underlying conditional probability Therefore, square loss is naturally preferred since it treats the classification problem regression problem. The following theorem states that, one can recover the conditional probability ry by using an overparameterized neural network with & regularization and GD training.
Theorem 3.4 (Calibration emror), Suppose Assumptions D.I D.4 are fulfilled. Let p x + Then
(fw()), + 1)/2 allm_ Ox(n 00=(n-1(Ad22)
Theorem 3.4 states that the underlying conditional probability in the non-separable case can be recovered by (fw(k).. + 1)/2. The form (Jw(s)) + 1)/2 is to account for the 1, label coding. Under [0.1) coding, the estimator would be /w(k), itself. The Lox consistency doesn't hold for cross-entropy trained neural networks, due to the form of the optimal solution log( 1-m)
-7)
12 With
limited capacity, the network's 's confidence prediction is bounded away from 0 and 1. In practice, we control the complexity of the neural network thus it is usually the that f xo < C
for some constant C. Hence, following the results in 12), direct calculation shows that such a cc
neural network with limited complexity cannot accurately estimate 77(x) when r7(x) > T+ or (xx) < 1 T which makes the calibration error under the cross-entropy loss always bounded away from zero. However, square loss does not have such a problem.
Notice that the calibration error bound in Theorem 3.4 does not depend on the Tsybakov's noise condition, and is slower than the excess risk. This is because small calibration error is much stronger than a small excess risk, since the former requires the conditional probability estimation to
"The term å only depends on the width of the neural network. A smaller If 8 then the number of nodes in the hidden layer is infinity.
å requires a wider neural network.

---

be uniformly accurate, not just matching the sign of 17 1/2. To be more specific, a good estimated f)
always lead to low risk plug-in classifier f f 27 1, but not vice versa.
Remark (Technical challenge). Despite the similar forms of regression and classification using square loss, most of the regression analysis techniques cannot be directly applied to the classification problem, even if the supports of two classes are non-separable. Moreover, it is clear that classification problems in the separable case are completely different with regression problems.
3.3 Transition from separable to non-separable
be connected via Gaussian noise injection. In practice, data augmentation an effective way to improve robustness and the simplest way is Gaussian noise injection [|]]] In this section, we only consider it an auxiliary tool for theoretical analysis purpose and not for actual robust training. Injecting Gaussian noise amounts to convoluting a Gaussian distribution [(0, v² I2) to the marginal distribution Px, which enlarges both Q and S2g to I and unique decision boundary D, can be induced. Correspondingly, the "noisy" conditional probability, denoted as Tju, is also smoothed to be continuous on Red. As v - 0, 1177 >/ll0 - 0 on f and f2g and the limiting 7jo is piecewise constant function with discontinuity at the induced decision boundary.
The general non-separable case and the special separable
Lemma 3.5 (Tsybakov's noise condition under Gaussian noises). Let the margin be 2y > 0, the noise be (0, Ia). Then there exist some constants T, C > such that
Px (x - 1| < t) = (Cv² //) (appp->2/(22)) )), Vt E (0,
Theorem 3.6 (Exponential convergence rate). Suppose the classes are separable with margin 27 >> 0. No matter how complicated [2 U.2, are, the excess risk of the over parameterized neural network classifier satisfying Assumptions D.1 and D.4 has the rate Op(e mn/7).
Although Theorem 3.6 similar that in 45 the techniques are different. Our analysis is directly from the general non-separable case (Theorem 3.1. by the addition of auxiliary noises (Lemma B.5) The proof of Theorem 3.6 involves taking the auxiliary noise to zero, e.g.. V' 2n X 1/ The exponential convergence rate is direct outcome of Lemma 3.5 and Theorem B.1 Note that our exponential convergence rate much faster than existing ones under the similar separable setting which are all polynomial with n, e.g., Oz(1/vn).
28
30
31
Remark 3 (Extension NTK). Although our analysis only concerns overparameterizeed one-hidden- layer ReLU neural networks, it can potentially apply to other types of neural networks. Recently, it has been shown that overparameterized multi-layer networks correspond the Laplace kernel As long as the trained neural networks can approximate the classifier induced by the NTK, our results can be naturally extended.
46 47
Remark 4. Theorems 3.4 and 3.6 share the same gist: the overparameterized neural network classifiers can have exponential convergence rate when data separable with positive margin. The result of Theorem 3.6 is weaker than that of Theorem 3.4. but with milder conditions. Technically, Theorem 3.6 also bridges the non-separable case and separable case through auxiliary noise injection.
3.4 Multiclass Classification
In binary classification, the labels are usually encoded as -1 and When there are K classes, the default label coding is one-hot. However, it is empirically observed that this vanilla square loss struggles when the number of classes are large, for which scaling tricks have been proposed 4 / Another popular coding scheme is the simplex coding [II), which takes maximally separated K points on the sphere as label features. When K 2, this reduces to the typical -1, 1 coding. Many advantages of the simplex coding have been discussed, including its relationship with cross-entropy loss and supervised contrastive learning IC 32 may have significant impact on optimization, it does not significantly affect our analysis of square loss from a theoretical perspective. In this work, we adopt the simplex coding
40 Even though the choice of label coding
Given the label coding, one can easily generalize the theoretical development in Section by employing the following objective function
Mooe discussion and empirical comparison about the coding choices can be found in Appendix G.2
7

---

Jroest
Jibran))
Figure 1: Test misclassification rates and decision boundaries predicted by: SL-ONN l (Left); CE-ONN + &g (Center); CE-ONN (Right) in the separable case.
minw Σ El= (WW, 3h.) W
where fw,a and y: (yt.1, ..., Vt,k) is the label of i-th observation.
Next proposition states relationship between the simplex coding and the conditional probability, which is generalization of the binary classification case.
Proposition 3.7 (Conditional probability). Let f - RK minimize the mean square error (x) - By)? where Vy is the simplex coding vector of label Then
myk (z) = (y = k) = ((K - (x) on + +1)/K. /K.
(3.2)
one can show that the calibration error can be small in the multi-class classification case. Unlike the softmax function when using cross entropy, the estimated conditional probability using square loss is not guaranteed to be within 0 and This will cause issues for adversarial attacks; see more diseussions in Appendix G.2
Based on the relationship established by Proposition 3.7
Numerical experiments
Although theoretical results are for overparameterized neural network in the NTK regime, expect our conclusions to generalize to practical network architectures, The focus of this section is on improving the state-of-the-ard performance for deep classifiers, but illustrate the difference between cross-entropy and square loss. We provide experiment results on both synthetic and real data, support our theoretical findings and illustrate the practical benefits of square loss in training overparameterized DNN classifiers. Compared with cross- rross-entrppy, the square loss has comparable generalization performance, but with stronger robustness and smaller calibration error.
4.1 Synthetic data
We consider the square loss based, cross-entropy based overparameterized neural networks (ONN) with f2 regularization, and the cross-entropy based ONN without & regularization. denoted as SL-ONN &,, CE-ONN + fz, and CE-ONN respectively. For the separable case, we consider two separated classes with spiral curve like supports. Figure [ shows one instance of the test misclassification rate and decision boundaries attained by three methods, where we can see that SL-ONN + lg has smaller test misclassification rate and much smoother decision boundary. In particular, in the red region, where the training data are sparse. SL-ONN + &y fits the correct data distribution best. For the non-separable case, we consider the calibration performance of SL-ONN + l2 and CE-ONN + L, where the classifiers are denoted by fiz and F.e. respectively. The results presented in Figure G.9 in the Appendix shows that Fiz has the smaller mean and standard deviation than foe. More implementation details are in Appendix G.1

---

4.2
Real data
To make a fair comparison, we adopt popular architectures, ResNet 150 and Wide ResNet [1 and evaluate them on the CIFAR image classification datasets [52], with only the training loss function changed, from cross-entropy (CE) to square loss with simplex coding (SL). Further, we don't employ any large scale nyper-paramettr tuning and all the parameters are kept as default except for the learning rate (Ir) and batch size (bs), where we are choosing from the better of (Ir=0.01, bs=32) and (1r0.11, bs=128). Each experiment setting is replicated 5 times and we report the average performance followed by its standard deviation in the parenthesis. (Ir=0.01, bs=32) works better for the most cases except for square loss trained WRN-16-10 on CIFAR- 100. More experiment details and additional results can be found in Appendix
Generalization In both CIFAR-10 and 100, the performance of cross- enosoyy and square loss with simplex coding are quite comparable, as observed in 4 Cross-entropy tends to perform slightly better for ResNet, especially on CIFAR-100 with an advantage of less than 1 1%. There is a more significant gap with Wide ResNet where square loss outperforms cross-entropy by more than 1% on both CIFAR-10 and CIFAR-100. The details can be found in Table II
Table 1: Test accuracy on CIFAR datasets. Average accuracy larger than 0 but less than 0.1 is denoted as O without standard deviation.
PGD- 100 -trrength)
AuterAtack (la -trengghh)
Dataset
Network
Loss
Clean acc %
2/255
4/255
8/255
2/255
4/255
8/255
CE
95.15 (0.11)
8.81 (1.61)
0.65 (0.24)
0
2.74 (0.09)
0
0
ResNet-18
SL
95.04 (0.07)
30.53 (0.92)
6.64 (0.67)
0.86 (0.24)
4.10 (0.50)
0"
0
CIFAR-10
CE
93,94 (0.16)
(0.10)
0.33 (0.06)
0
0
WRN-16-10
SL
95.02 (0.11)
37.47 (0.61)
23.16 (1.28)
7,88 (0.72)
5.37 (0.50)
0
CF:
79.82 (0.14)
2.31 (0.07)
0.99 (0.10)
R5
SL
78.91 (0.14)
13.76 (1.30)
4.63 (1.20)
1.21 (0.80)
3.67 (0.60)
0.16 (0.05)
0
CIFAR-100
CE
77.89 (0.21)
0.83 (0.07)
0"
0.42 (0.07)
0
0
WRN- 16 10
SL.
79.65 (0.15)
6.48 (0.40)
0.42 (0.04)
0"
2.73 (0.20)
0
Adversarial robustness Naturally trained deep classifiers are found be adversarially vulner- able and adversarial attacks provide a powerful tool to evaluate classification robustness. For our experiment, we consider the black-box Gaussian noise attack, the classic white-box PGD attack [53 and the state-of-the-art Auto/ttack [54], with attack strength level 2/255, 4/255, 8/255 in loc norm. Auto.Attack contains both white-box and black-box attacks and offers a more comprehensive evaluation of adversarial robustness. The Gaussian noises results are presented in Table G.3 in the Appendix. At different noise levels, square loss consistently outperforms cross-entropy. especially for WRN-16-10 10, with around 2-4% accuracy improvement. More details can be found in Appendix The PGD and Attoct results are reported Table T Even though classifiers trained with square loss is far away from ad adversarially robust, it consistently gives significantly higher adversarial accuracy. The same margin can be carried over to standard adversarial training as well. Table 2 lists results from standard PGD adversarial training with CE and SL. By substituting cross-entropy loss to square loss, the robust accuracy increased around 3% while maintaining higher clean accuracy.
G.2
One thing to notice is that when constructing white-box attacks, square loss will not work well since it doesn't 't directly reflect the classification accuracy. More specifically, for a correctly classified image (ar, y). maximizing the square loss may result in linear scaling of the classifier f(æ)), which doesn't 't change the predicted class (see Appendix G.2 for more discussion). To this end, we consider a special attack for classifiers trained by square loss by maximizing the cosine similarity between / f((r) and By. We call this angle attack and also utilize it for the PGD adversarial training paired with square loss in Table g In our experiments, this special attack rarely outperforms the standard PGD with cross-entropy and the reported PGD accuracy are from the latter settings. This property of square loss may be an advantage in defending adversaria attacks.
The predicted class probabilities for square loss can be obtained from Equa- tion 2 Expected calibration error (ECE) measures the absolute difference between predicted confidence and actual accuracy. Deep classifiers are usually over-confident (55) Using ResNet as an example, we report the typical reliability diagram in Figure 2 On CIFAR- 10 with R8.N
Model calibration

---

Table 2: Performance on CIFAR-10 dataset for ResNet-18 18 under standard PGD adversarial training.
Loss
Acc (%)
PGD steps
Strength(/
AutoAttack
86.87
3
8/255
37.08
CE
84.30
7
8/255
41.88
87.31
3
8/255
40.46
SL
84.52
7
8/255
44.76
the average ECE for cross-entropy is 0.028 (0.002) while that for square loss is 0.0097 (0.001). On CIFAR-I00 with ResNet-50, the average ECE for cross-entropy is 0.094 (0.005) while that for square loss is 0.068 (0.005). Square loss results are much more calibrated with significantly smaller ECE.
CIFAR-
CIFAR-10 ResNet-19
CIFAR-100 ResNet-50
CIFAR- 100 ResNet-50
Orp
000
04
Confidence
Confidence
Confidence
Confidence
Figure 2: Reliability diagrams of ResNet- 18 on CIFAR- 10 and ResNet-50 CIFAR-100. Square loss trained models behave more well-calibrated while cross-entropy trained ones tend to be visibly more over-confident.
5 Conclusions
Classification problems ubiquitous in deep learning. As fundamental problem, any progress in classification can potentially benefit numerous relevant tasks. Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked. Through both theoretical analysis and empirical studies, we identify several ideal properties of using square loss in training neural network classifiers, including provable fast convergence rates, strong robustness, and small calibration error. We encourage readers to try square loss in your own application scenarios.
6
Acknowledgements
We would like to thank reviewers for their valuable comments the manuscript. Wenjia Wang and Jun Wang were supported by Foshan HKUST Projects FSUST20-FYTRIO3B. We gratefully acknowledge the support of MindSpore for this research.
References
[1] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33, 2020.
[2] Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross-entropy loss for adversarial robustness. arXiv preprint arXiv: 1oo:::005..1200 2019.
[3] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[4] Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. arXiv preprint 2rXiv:2006.07322 2020.
10

---

[5] Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invari- ance causes adversarial vulnerability. In International Conference on Learning Representations. 2018.
[6] Kamil Nar, Orhan Ocal, S Shankar Sastry, and Kannan Ramchandran. Cross-entropy loss and low-rank features have responsibility for adversarial examples. arXiv preprini FrrXiv::901.0..00 2019.
[7] Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 2020 54th Annual Conference on Information Sciences and Systems (CISS), pages 1-5. IEEE, 2020.
[8] Mohammad Pezeshki. Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1256-1272, 2021.
[9] Simon Kornblith, Honglak Lee, Ting Chen, and Mohammad Norouzi. Demystifying loss functions for classification. 2020.
[10] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652-24663, 2020.
[11] Youssef Mroueh, Tomaso Poggio, Lorenzo Rosasco, and Jean-Jacques Slotine. Multiclass learning with simplex coding. arXiv preprint arXiv:1209. 2012.
[12] Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, pages 56-85, 2004.
[13] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 150.:02503303 2015.
[15] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. International Conference on Machine Learning, pages 7632-7642. PMLR, 2021.
[16] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1 1362, 2020.
[17] Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In International Conference Machine Learning, pages 3821-3830.
PMLR, 2021.
[18] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Di- reet input space margin maximization through adversarial training. In International Conference on Learning Representations, 2020.
[19] Enno Mammen and Alexandre B Tsybakov. Smooth discrimination analysis. The Annals of Statistics, 27(6):1808 1000-1882221 1829, 1999.
[20] Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155 2006.
[21] Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of Statisties, 35(2):608-66 633, 2007.
[22] Michael Kohler and Adam Krzyzak. On the rate of convergence of local averaging plug-in classification rules under margin condition. IEEE Transactions on Information Theory, 53(5): 1735-1742, 2007.

---

[23] Ingo Steinwart, Clint Scovel, et al. Fast rates for support vector machines using Gaussian kernels. The Annals of Statistics. 35(2):575-607, 2007.
[24] Tianyang Hu, Zuofeng Shang, and Guang Cheng. Sharp rate of convergence for deep neural network classifiers under the teacher-student setting. arXiv preprint Stti::2001006892 2020.
[25] Tianyang Hu, Ruiqi Liu, Zuofeng Shang, and Guang Cheng. Minimax optimal deep neural network classifiers under smooth decision boundary. arXiv preprint vrXiv:2207.01000 2022.
[26] Yongdai Kim, Ilsang Ohn, and Dongha Kim. Fast convergence rates of deep neural networks for classification, arXiv preprint arXiv:/812.03599, 2018.
[27] Michael Kohler and Sophie Langer. Statistical theory for image classification using deep convolutional neural networks with cross-entropy loss. arXiv preprint arXiv:2011.33602 2020.
[28] Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks. arXiv preprint arXiv: 1909. 12292, 2019.
[29] Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations, 2020.
[30] Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances Neural Information Processing Systems, 32:10836-10846, 2019.
[31] Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over-parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3349-3356, 2020.
[32] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under msse loss: Proximity to and dynamics the central path. arXiv preprint arXiv: 222.0200002255 2021.
[33] Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification regression in overparameterizee regimes: Does the loss function matter? arXiv preprint arXiv:2005.08054 2020.
[34] Tomaso Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square loss. Technical report, CBMM Memo No, 2019.
[35] Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference Machine Learning, pages 74844. PMLR, 2020.
[36] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4))667-766, 2022.
[37] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, pages 8571-8588 2018.
[38] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.0858/ 2019.
[39] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. arXiv preprint arXiv:/908 12173, 2019.
[40] Tianyang Hu, Wenjia Wang. Cong Lin, and Guang Cheng. Regularization matters: A non- parametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics, pages 829-837. PMLR, 2021.
12

---

[41] Gamaleldin F Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. arXiv preprint arXiv:1803.05595 2018.
[42] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma train- ing: Direct input space margin maximization through adversarial training. arXiv preprint 1.Xiv:1812.02637. 2018.
[43] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint vrXiiv/906.000 2019.
[44] Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable random- ness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3888557, 2019.
[45] Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. arXiv preprint arXiv:2006. 12297, 2020.
[46] Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Ronen Basri. On the similarity between the Laplace and neural tangent kernels. arXiv preprint arXiv:2000.01550,
2020.
[47] Lin Chen and Sheng Xu. Deep neural tangent kernel and Laplace kernel have the same RKHS. arXiv preprint arXiv:2009. 10683, 2020.
[48] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021.
[49] Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. arXiv preprint arXiv:2205. 14814, 2022.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[51] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks. arXiv preprint
[52] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[53] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1eXiiv:706..060 06083, 2017.
[54] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, pages 2206-2216. PMLR, 2020.
[55] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference Artificial Intelligence and Statistics, pages 3459-3467. PMLR, 2019.
[56) Simon $ Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.0205: 2018
[57] David Eric Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators, volume 120. Cambridge University Press, 2008
[58] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 1874)11875-1897, 2020.
[59] Max Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89()):81-213, 2021.
13

---

[60] Atsushi Nitanda, Geoffrey Chinot, and Taiji Suzuki. Gradient descent can learn less over-parameterized two-layer neural networks on classification problems. arXiv preprint nete:/905000777 2019.
[61] Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pages 8157-8166, 2018.
[62] N. Aronszajn. Theory of reproducing kemels. Transactions of the American Mathematical Society, 68(3):337-404 1950.
[63] Colin Bennett and Robert C Sharpley. Interpolation of Operators. Academic press, 1988.
[64] Holger Wendland. Scattered Data Approximation. Cambridge University Press, 2004.
14


Reviews:
[{"review_id": "eBRSqD_H1hY", "reviewer": "Reviewer_rLs9", "summary": "The authors explore the use of square loss in classification settings, with both one-hot and scaled simplex features. They motivate the study by invoking improvements in model calibration and feature selection that have been theorized/observed about the use of square loss in classification settings.\n\nThe authors then study the generalization error and model calibration in the large width Neural Tangent Kernel (NTK) setting. Under a particular set of conditions on the dataset, they derive non-trivial bounds for the generalization error and the calibration as a function of the number of data points n.\n\nThey conclude with experiments showing that using squared loss with scaled simplex features improves performance on CIFAR. They also show that squared loss provides additional robustness to adversarial examples.", "questions": "How close do the bounds in equations 3.1 and 3.4 compare to empirical behavior? I would be curious to know how tight they are even in toy examples. I'd also be curious about the relationship between the bounds in this paper and those in http://proceedings.mlr.press/v119/huang20l.html and https://proceedings.mlr.press/v119/adlam20a.html.\n\nWhat do results of squared loss look like on tasks with a large number of classes, like imagenet or language modeling?\n\nIs there any way to visualize/quantify the difference in decision boundaries between the cross entropy and squared loss cases? This seems particularly interesting in the adversarial setting.", "limitations": "Adequately addressed", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The analysis of the generalization error and calibration is novel and interesting. I believe that it is correct, though the arguments are long and it is possible I've missed an issue there.\n\nThe authors mention that not much work exists in terms of generalization bounds for wide neural networks under squared loss; however, there has been much recent work in this area in the context of the simultaneous scaling of width and data (http://proceedings.mlr.press/v119/huang20l.html, https://proceedings.mlr.press/v119/adlam20a.html). Comparing the author's results to these scenarios would be interesting and useful to position the paper in the literature.\n\nOverall the empirical results seemed the most promising. The fact that both training and adversarial attacks were studied helped solidify the claim that both generalization and calibration can be improved by using square loss. Overall I find the empirical results even more compelling than the theoretical ones due to their careful presentation. I think the experiments could be strengthened by exploring the efficacy of squared loss on datasets with a large number of classes (Imagenet, language modeling), where the SOTA models are far from the NTK regime studied in the theory section of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F_Jfyx8mma1", "reviewer": "Reviewer_zn8W", "summary": "The paper conducted comprehensive review in research, and compare the loss function between Cross Entropy (CE) loss and Square Loss(SL).  The analytical results show that compared to CE, SL has comparable generalization error, but noticeable advantage in robustness and calibration. Experimental results from synthetic data and real data further demonstrate the claim of the paper.", "questions": "[1] Theorem 3.1 shows the gap between loss with neural networks and loss with minimize 0-1 loss. As mentioned in equation 2.1, one minimizer is L*., which might not be the optimal minimizer. Could the author help to clarify if L* is bounded with optimal loss with limited error?\n\n[2] From line 62 in page 2, it shows the mis-calibration of CE in prediction region close to 0 or 1 has been shown in paper [9]. Similar results are also covered in Theorem 3.4, between line 248 and 256 in page 6. Could the author help to confirm the originality of the results and if any gaps or difference between this paper and paper [9]?\n\n[3] From the theory analysis, it seems CE tends to weakly calibrated in the high predicted or low predicted region (Theorem 3.4). However, from the experimental results in Fig 2, it seems the mis-calibration happens in low prediction region (confidence < 0.1) and normal prediction region (confidence between 0.5 and 0.8). Does the experimental result align with the theorem?\n\n[4] From the experimental results from Table 1, it seems SL is not always better than CE, although SL seems to be better than CE when under attack. Are there any hypothesis or interpretation to show the situation when CE will outperform than SL?\n\n", "limitations": "The paper shows the advantage of SL over CE, but not the dis-advantage of SL compared to CE. Might need more insights to compare the disadvantage of SL, and explain the the performance gap between two loss function in experimental results from Table 1.  ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper comprehensive summarized the areas and related results. The paper is well written and clearly organized. The theoretical results of CE calibration issue aligned with the observations and the learning from the theoretical results helps to understand the gaps in practice. \n\nSome hypothesis and learning from the theorem might need more clarification. Experimental results might need further improvement to demonstrate the claim made in the theorem. More details can be found from questions sections. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "7F8-KolBwCP", "reviewer": "Reviewer_rSfq", "summary": "The authors theoretically investigate the square loss in classification for over-parametrized neural networks in NTK regime. They reveal properties regarding the generalization error, robustness and calibration error. Separable and non-separable cases are considered. In the non-separable case, they establish fast convergence rate  for misclassification rate and calibration error. While in separable case, the misclassification rate improves to be exponentially fast. \n\nFurther, they prove a margin lower bounded away from zero, and claim theoretical guarantees for robustness. They also conduct experiments with practical neural networks to support and extend the theoretical claims.\n\nThe main conclusion is that compared with the cross-entropy loss, the square loss has comparable generalization error while noticeable advantages in robustness and model calibration.", "questions": "1) It was widely believed that training with the square loss in classification suffer from the 'outliers' problem, and it was not robust. \nHow do your claim on robustness solve the concerns on the 'outlier'  issue?\n\n2) Are the results of the square loss in Table 1 with simplex coding? As in large-class number case, previous works show that the original square loss with one-hot encoding may not work well, it seems in Table 1 the square loss results on CIFAR-100 are way better than the cross-entropy. How simplex encoding make a difference with the one-hot encoding? ", "limitations": "This paper extends the study of the square loss in classification by providing systematic theoretical study, which is import for people to consider other possible alternatives as training objectives in classification.\n\nThe assumptions are a little bit strong which may not hold in practice, and the experiments are a little bit limited for practitioners to be convinced. \n\nOverall, the paper indeed provides more theoretical understanding which is an important contribution to the study of square loss in classification.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) This paper extends the study of square loss in classification with more theoretical understanding, with systematical investigations on several import properties: generalization error, robustness and model calibration.\n\n2) The main conclusion is well-supported by theoretical proofs and some empirical results.\n\n3) The paper is well-written and easy to follow.\n\nWeaknesses:\n\nThe experimental results is not very convincing as it mainly considers CIFAR-10 dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1VRNSye6Alb", "reviewer": "Reviewer_bBhR", "summary": "This paper provides solid arguments in favor of using MSE loss instead of the most common CE loss. From several aspects, showing that:\n- MSE is better calibrated.\n- It has a better generalization bound.\n- Is more robust to adversarial perturbations.", "questions": "- CE is shown to converge to the max-margin solution. So I don't fully understand Thm 3.3. Yes, MSE's margin is bounded from below but is still worse to equal to margin of CE. I am right?\n- I do not fully understand Fig. 2. How do you measure the confidence? The reason I'm asking this is because the output with MSE is real valued and not a probability. No?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths**:\n- I loved the structure of this work. Well-motivated, properly stated, and provide a good balance between the theory and intuition.\n- I found the results to be novel and significant.\n- The experiments appear sufficient to support the claims.\n- The authors are fair in their evaluations. For example, I enjoyed reading your argument on why white-box attacks appear to fail with MSE.\n\n** Weaknesses**:\n- There are a few citations missing. I know at least the following three which advocate using MSE instead of CE:\n    + \"Excessive invariance causes adversarial vulnerability\" by Jacobsen et al\n    + \"Cross-entropy loss and low-rank features have responsibility for adversarial examples\" by Nar et al.\n    + \"Gradient starvation: A learning proclivity in neural networks\" by Pezeshki et al.\n- The writing can benefit from a grammar check. For example, \"it's\" --> \"it is\".\n- The use of separable / non-separable terminology was initially very confusing for me. Generally \"separable\" refers to a case that data is linearly separable or the model has enough capacity to fully classify the data. However, here, authors refer to label ambiguity.\n- The extension to multi-class using simplex coding is rather difficult to follow.\n- The code for the experiments is not provided.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept