Understanding Square Loss in Training Overparametrized Neural Network Classifiers
Authors: Tianyang Hu
Jun Wang
Wenjia Wang
Zhenguo Li
Venue: ICLR 2022 Submitted (2022)
Keywords: classification
square loss
neural tangent kernel
convergence rate

Abstract: Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.

Under review as a conference paper at ICLR 2022
UNDERSTANDING SQUARE Loss IN TRAINING OVER- PARAMETRIZED NEURAL NETWORK CLASSIFIERS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Re- cently, several alternative losses have seen revived interests for deep classifiers. particular, empirical evidence seems to promote square loss but theoretical justi- fication still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. In- teresting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are sepa- rable not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration eπror. When classes are separable, the misclassification rate improves to exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guar- antees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical stud- ies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensions data and real image data. Comparing to cross- entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.
INTRODUCTION
The pursuit of better classifiers has fueled the progress of machine learning and deep learning re- search. The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provides test fields for all kinds of new classification models, especially those based on deep neural networks (DNN). With the introduction of CNN, ResNets, and transformers, DNN classifiers are constantly improving and catching up to the human-level performance. In contrast the active innovations in model architecture, the training objective remains largely stagnant, with cross-entropy loss being the default choice. Despite its popularity, cross-entropy has been shown to be problematic in some applications. Among others, Yu et al. (2020) argued that features Iearned from cross-entropy lack interpretability and proposed a new loss aiming for maximum coding rate reduction. Pang et al. 2019) linked the use of cross-entropy to ad vdersariaal vulnerability and proposed a new classification loss based on latent space matching. Guo et al. (017)) discovered that the confidence of most DNN classifiers trained with cross-entropy is not well- well-calibrated.
Recently, several alternative losses have seen revived interests for deep classifiers. In particular, many existing works have presented empirical evidence promoting the use of square loss over cross- entropy. Hui & Belkin 2020) conducted large-scale experiments comparing the two and found that square loss tends to perform better in natural language processing related tasks while cross-entropy usually yields slightly better accuracy in image classification. Similar comparisons are also made in Demirkaya et al. (2020). Kornblith al. (2020) compared a variety of loss functions and output layer regularization strategies on the accuracy and of-distribution robustness, and found that square loss has greater class separation and better oat-of-distribution robustness.
In comparison to the empirical investigation, theoretical understanding of square loss in training deep learning classifiers is still lacking. Through our lens, square loss has its uniqueness among

---

Under review as a conference paper at ICLR 2022
classic classification losses, and we argue that it has great potentials for modern classification tasks. Below we list our motivations and reasons why.
Explicit feature modeling Deep learning's 's success can be largely attributed to its superior ability as feature extractors. For classification, the ideal features should be separated between classes and concentrated within classes. However, when optimizing cross-entropy loss, it's not clear what the learned features should look like (Yu et al. 2020p. In comparison, square loss uses the label codings (one-hot, simplex etc.) as features, which can be modeled explicitly to control class separations.
Model Calibration An ideal classifier should not only give the correct class prediction, but also with the correct confidence. Calibration error measures the closeness of the predicted confidence to the underlying conditional probability 1). Using square loss in classification can be essentially viewed as regression where treats discrete labels as continuous code vectors, can be shown that the optimal classifier under square loss is 2n 1, linear with the ground truth. This distinguishing property allows it to easily recover 17. In comparison, the optimal classifiers under the hinge loss and cross-entropy are sign(2n) (2x) - 1) and (og(r",)), respectively. Therefore, hinge loss doesn't 't provide reliable information on the prediction confidence, and cross-entropy can be problematic when is close to 0 or 1 Zhang 2004). Hence, in terms of model calibration, square loss is a natural choice.
Connections to popular approaches Mixup Zhang et al. 2017) is popular data augmentation technique where augmented data are constructed via convex combinations of inputs and their labels. Like in square loss, mixup treats labels continuous and is shown to improve the generalization of DNN classifiers. In knowledge distillation (Hinton et al.] (013). where a student classifier is trying learn from trained teacher. Menon al. (2021) proved that the "optimal" teacher with the ground truth conditional probabilities provides the lowest variance in student learning. Since classifiers trained using square loss is a natural consistent estimator of 17, one can argue that it is a better teacher. In supervised contrastive learning (Khosla et al.) (2020), the optimal features are the same as those from square loss with simplex label coding (Graf et 202T (details in Section F..
Despite its lack of popularity in practice, square loss has many advantages that can be easily over- looked. In this work, we systematically investigate from a statistical estimation perspective, the properties of deep learning classifiers trained using square loss. The neural networks in our anal- ysis are required to be sufficiently overparametrized in the neural tangent kernel (NTK) regime. Even though this restricts the implication of our results, it is a necessary first step towards deeper understanding. In summary, our main contributions are:
Generalization error bound: We consider two cases, according whether classes are separable or not. In the general non-separable case, we adopt the classical binary classification setting with smooth conditional probability. Fast rate of convergence is established for overparametrized neural network classifiers with Tsybakov's noise condition. If two classes are separable with positive margins, we show that overparametrized neural network classifiers can provably reach zero misclassification error with probability exponentially tending to one. To the best of our knowledge, this the first such result for separable but not linear separable classes. Furthermore, we bridge these two cases and offer unified view by considering auxiliary random noise injection.
Robustness (margin property): When two classes are separable, the decision boundary is not unique and large-margin classifiers are preferred. In the separable case, we further show that the decision boundary of overparametrized neural network classifiers trained by square loss can- not be too close to the data support and the resulting margin is lower bounded away from zero, providing theoretical guarantees for robustness.
Calibration error: We show that classifiers trained using square loss are inherently well-calibrated, i.e., the trained classifier provides consistent estimation of the ground-truth conditional probability in Lox norm. Such property doesn't it hold for cross-entropy.
Empirical evaluation: We corroborate our theoretical findings with empirical experiments in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.
This work contributes towards the theoretical understanding of deep classifiers, from an estimation point of view, which has been classic topic in statistics literature. Among others, Mammen & Tsybakov (1999) established the optimal convergence rate for 0-1 loss excess risk when the decision boundary smooth. ZZhnn 2004): BBrrlett et al. (0066 extended the analysis to various surrogate
2

---

Under review as a conference paper at ICLR 2022
losses. Audibert & Tsybakov (2007) Kohler & Krzyzak (2007) studied the convergence rates for plug-in classifiers from local averaging estimators. [Steinwart et al. (2077) investigated the conver- gence rate for support vector machine using Gaussian kernels. We build on and extend classic results to neural networks in the NTK regime. Comparing to existing works on deep learning classification,
e-g.. Kiim et al. (2018) derived fast convergence rates of ReI.U DNN classifiers that minimize the empirical hinge loss, our results incorporate the training algorithm and apply to trained classifiers.
We require the neural network be overparametrized, which has been extensively studied recently, under the umbrella term NTK. Most such results are in the regression setting with handful of ex- ceptions. Di & Telgarsky (2199) showed that only polylogarithmic width is sufficient for gradient descent to overfit the training data using logistic loss. Hu et al. (2020) proved generalization error bound for regularized NTK in classification. Cao Gu 2019 2020) provided optimization and generalization guarantees for overparametrized network trained with cross-entropy. In comparison, our results are sharper the sense that we take the ground truth data assumptions into consideration. This allows faster convergence rate, especially when the classes are separable, where the exponen- tial convergence rate is attainable. The NTK framework greatly reduces the technical difficulty for our theoretical analysis. However, our results mainly due to properties of the square loss itself and we expect them to hold for a wide range of classifiers.
There are other works investigating the of square loss for training (deep) classifiers. Han et al. 2021 uncovered that the "neural collapse" phenomenon also occurs under square loss where the last-layer features eventually collapse to their simplex-style class-meant /Muthukumar et al. (2020) compared classification and regression tasks in the overparameterized linear model with Gaussian features, illustrating different roles and properties of loss functions used the training and test- ing phases. PPoggio & Liao (0199) made interesting observations on effects of popular regulariza- tion techniques such as batch normalization and weight decay on the gradient flow dynamics under square loss. These findings support our theoretical results' implication, which further strengthens our beliefs that the essence comes from the square loss and our analysis can go beyond NTK regime.
The rest of this paper is arranged as follows. Section 2aroomp@rreee presents some preliminaries. Main theoretical results are in Section 5] The simplex label coding discussed in Section g followed by numerical studies in Section 5 and conclusions in Section 6 Technical proofs and details of the numerical studies can be found in the Appendix.
2 PRELIMINARIES
suPaea [xx) and IlI - (Sa ( For vector ær ||>eIIp denotes its p-norm, for 1 S S co. Tap and I, are used to distinguish function norms and vector norms. For two positive sequences {az (unhnen and (b, JnEN, we write a, S bn there exists a constant C 0 such that an < Cb, for all sufficiently large n. We write an x 6n if an S by and bn S an Let [N] = [I....,N] for N E N, 1 be the indicator function, and Ia be the x d identity matrix. N (μ, E) represents Gaussian distribution with mean l and covariance Σ.
Notation For function f - let ||
Classification problem settings Let P be an underlying probability measure on Ω x Y, where Q C IR is compact and Y = 11.. Let (X, Y) be random variable with respect to P. Suppose we have observations { y1) }))) C (52 x Y)" i.idd. sampled according to P. The classification task is to predict the unobserved label y given new input x E Ω. Let y defined on 52 denote the conditional probability, i.e., v)(x)) = (P(y 1 |ar). Let Px be the marginal distribution of P on X. The key quantity of interest is the misclassification error, i... 0-1 loss. In the population level, the 0-1 loss can be written as
L(F) Eei.. FY3 -Ex ((X)) 2 0} + (X)LL((X OH,,
(2.1)
where the expectation taken with respect to the probability measure P. Clearly, an optimal classi- fier with the minimal (-1 loss is 2n 1.
According to whether labels are deterministic, there are two scenarios of interest. If n only takes values from {0, 1J. i.e., labels are deterministic, we call this case the separable case Let Ω
In the separable case we consider, the classes are not limited to linearly separable but can be arbitrarily complicated.
3

---

Under review as a conference paper at ICLR 2022
92 {x|n(()) = 0} and Ω = 2 U U2g If the probability measure of {œ|n(æ) {a/n(x) = 13. =
(0, 1)} is non-zero, i.e., the labels contain randomness, we call this case the non-se non-separable case. In the separable case, further assume that there exists positive margin, i..., dist(S?,, 2 2 >> 0, where 7 is a constant, and dist(S2, Ω2) = inf , adopt the well-established Tsybakov's noise condition
|| - x' x21/2. In the non-separable case, to
quantify the difficulty of classification, (Audibert & Tsybakov 2007). which measures how large the "difficult region" is where ny(x) R 1/2.
Definition 2.1 (Tsybakov's 's noise condition). Let K E [0, oo). We say P has Tsybakov noise expo- nent A if there exists a constant C, T > O such that for all 0 < t < T. (x - < t) s C'-t".
large value of implies the difficult region to be small. is expected that a larger leads a faster convergence rate of a neural network classifier. This intuition is verified for the overparametrized neural network classifier trained by square loss and &2 regularization. See Section S for more details.
Neural network setup We mainly focus on the one-hidden-layer ReLU neural network family F with m nodes in the hidden layer, denoted by
1 Vm Σ a, (W, x),
=
where E Ω W (W1, Wm) E Rdxm is the weight matrix in the hidden layer, a =
(a), am) T RT is the weight vector in the output layer, oz)) = max =} is the rectified linear unit (ReLU). The initial values of the weights are independently generated from
(0) NO (10, Im), ar unif(-1,1], Vr [m/]
Based on the observations {(x,, u-))i=n, the goal of training neural network is to find solution to
min Σ
WR(W, a):
(2.2)
where / is the loss function, is the regularization, and μ 2 is the regularization parameter. Note in Equation 2.2 that we only consider training the weights W. This is because a(2) sign(a)- (Ia 2). which allows us to reparametrize the network to have all a 's to be either or -1. In this work, we consider square loss associated with & regularization, i.e., ((W,aa(a)) Us) (fw,a(x,) - y and R(W, a) = W
A popular way train the neural network is via gradient based methods. It has been shown that the training process of DNNs can be characterized by the neural tangent kernel (NTK) (Jacol et al.
2018). As is usually assumed in the NTK literature (Arora al. 2019 Hu et al. 2020, Bietti & Mairal 2019 Hu et al. 2021). we consider data on the unit sphere S ..e., T I, vi [n], and the neural network is highly overparametrized (m >> 11) and trained by gradient descent (GD).
For details about NTK and GD in one-hidden-layer ReL.U neural networks, we refer to Appendix
A In the rest of this work, we use fW(R), to denote the GD-trained neural network classifier under
square loss associated with & regularization, where i the iteration number satisfying Assumption D.I and W(k) is the weight matrix after k-th iteration.
THEORETICAL RESULTS
In this section, we present our main theoretical results. Throughout the analysis, we assume that the overparametrized neural network fw.a and the training process via GD satisfy Assumption D.I Appendix D). which essentially requires the neural network to be sufficiently overparametrized (with (with a finite width). and imposes conditions on the learning rate and iteration number. Our theoretical results consist of three parts: generalization error, robustness, and calibration error.
3.1 GENERALIZATION ERROR BOUND
In classification, the generalization error is typically referred to as the misclassification error, which can be quantified by L(f) defined in Equation 2.1 In the non-separable case, the excess risk, defined by L(f) L" is used evaluate the quality of classifier f. where L L(2n 1). which minimizes the 0-1 loss. The following theorem states that the overparametrized neural network with GD and &g regularization can achieve small excess risk in the non-separable case.
4

---

Under review as a conference paper at ICLR 2022
Theorem 3.1 (Excess risk in the non-separable case). Suppose Assumptions D.I
D.2 and D.4
hold. Assume the conditional probability n)))) satisfies Tsybakov's 's noise condition with component
R. Let y 20-1 Then
d(m+d)
((fw()),))= - Op(n (2d 12
(3.1)
From Theorem 3.1 we can see that becomes larger, the convergence rate becomes faster, which is intuitively true. Generalization error bounds in this setting is scarce. To the best the au- thors' knowledge, Hu al. (2020) is the closest work (the labels are randomly flipped), where the bound is in the order of Op [1/ vn). Our bound is faster, especially with larger K. It is known
that the optimal convergence rate under Assumptions D.2 and D.4 is Or-(n an d(n+1) (Audibert
& Tsybakov] 2007). The differences between Equation 3.1 and the optimal convergence rate that there is an extra (d 1)xs in the denominator of the convergence rate in Equation 3.1 (since d(n+1)
d(e+4) Y 12 T
ta 13 dic
If the conditional probability has a bounded Lipschítz con-
stant, then Kohler & Krzyzak (007 showed that the convergence rate based on the plug-in kernel
estimate is Op(n *+3+4 ), which is slower than the rate in Equation 3.1 if dl is large.
Now turn the separable case. Since 7 only takes value from {0, 1} in the separable case, is bounded away from 1/2. Therefore, one can trivially take k + oo in Equation [3.] and obtain the convergence rate (=(n-d((d-1)) However, this rate can be significantly improved the separable case, as stated in the following theorem.
Theorem 3.2 (Generalization error in the separable case). Suppose Assumptions D.I D.3 and D.5
hold. Let μ o1). There exist positive constants C₁, C2 such that the misclassincation rate is 0% with probability at least - C1 exp( - -2n), and can be arbitrarily smal by enlarging the neural network's 's width.
Note that in Theorem 3.2] the regularization parameter can take any rate that converges to zero. In particular, can be zero, and the corresponding classifier overfits the training data. Theorem 3.2 states that the convergence rate in the separable case is exponential, if a sufficiently wide neural network is applied. This is because the observed labels are not corrupted by noise, i.e., P(y 1/x)) either one or zero. Therefore, it is easier to classify separable data, which intuitively true.
3.2 ROBUSTNESS AND CALIBRATION ERROR
If two classes separable with positive margin, the decision boundary is not unique. Practitioners often prefer the decision boundary with large margins, which are robust against possible perturbation on imput points (Elsayed al., 2018, Ding et al. 20180. The following theorem states that the square loss trained margin can be lower bounded by positive constant. Recall that in the separable case, Q = 22 12, where Q2 (z) 1} and f2g {>|n(x)) 0}.
Theorem 3.3 (Robustness in the separable case). Suppose the assumptions of Theorem 3.2 are satisfied. Let μ = o(1). Then there exist positive constants C, C1, C2 such that for all n.
min #EDr "EDrr
||
x ||2 2 C,
and the misclassification rate is 0% with probability at least 1 - d 8- -1 exp(-C2n), where Dr is the decision boundary, and is as in Theorem 3.2
Remark 1. Note that [ 221 2 vdll x' [|2, thus Theorem 3.3 also indicates 7o robustness.
In the non-separable case, 77(x) varies within (0,1) and practitioners may not only want clas- sifier with small excess risk, but also want to recover the underlying conditional probability 17. Therefore, square loss is naturally preferred since it treats the classification problem as a regression problem. The following theorem states that, can recover the conditional probability by using an overparametrized neural network with lg regularization and GD training.
Theorem 3.4 (Calibration error). Suppose the conditions in Theorem are fulfilled. Let μ x n 2 Then
3.1 Assumption
D.3 and D.4
(l( 1)/2 m/1... = Or(n mج=
(3.2)
2The term 5 only depends the width of the neural network. A smaller requires wider neural network. If 5 then the number of nodes in the hidden layer is infinity.
5

---

Under review as a conference paper at ICLR 2022
Theorem states that the underlying conditional probability in the non-separable case can be recovered by (fw(k).a + 1)/2. The form (fw(()). + 1)/2 is to account for the {-1, 1 label coding. Under {0.1] coding, the estimator would be fW((). itself. The T consistency doesn't hold for cross-entropy trained neural networks, due to the form of the optimal solution log( 1-y With limited capacity, the network's confidence prediction bounded away from and (Zhang 2004]. In practice, we want to control the complexity of the neural network thus it is usually the case that (k) < for some constant C. Hence, it camnot accurately estimate r7(x) when ec
7(x)) I+ee or 71x)) I+ which makes the calibration error under the cross-entropy loss always bounded away from zero. However, square loss does not have such a problem.
Notice that the calibration error bound in Theorem 3.4 does not depend on the Tsybakov's noise condition, and is slower than the excess risk. This because, small calibration error is much stronger than a small excess risk, since the former requires the conditional probability estimation be uniformly accurate, not just matching the sign of n(Φ) 1/2. To be more specific, a good estimated f) can always result in a low risk plug-in in classifier f(xx = 2f)(z) - 1, but not vice versa.
Remark (Technical challenge). Despite the similar forms of regression and classification using square loss, most of the regression analysis techniques cannot be directly applied to the classifi- cation problem, even the supports of two classes are non-separable. Moreover, it is clear that classification problems in the separable case are completely different with regression problems.
Remark 3 (Extension on NTK). Although our analysis only concerns overparametrized one- hidden-layer ReLUU neural networks, can potentially apply to other types of neural networks in the NTK regime. Recently, it has been shown that overparametrized multi-layer networks corre- spond to the Laplace kernel (Geifiman et al. 2020 Chen & Xu 2020). As long as the trained neural networks can approximate the classifier induced by the NTK, results can be naturally extended.
3.3 TRANSITION FROM SEPARABLE TO NON-SEPARABLE
The general non-separable case and the special separable case can be connected via Gaussian noise injection. In practice, data augmentation an effective way to improve robustness and the simplest way is Gaussian noise injection (He al. 2019). In this section, we only consider it as auxiliary tool for theoretical analysis purpose and not for actual robust training. Injecting Gaussian noise amounts convoluting Gaussian distribution N(0, v² La) the marginal distribution Px, which enlarges both 22 and f2 to IRd and a unique decision boundary D, can be induced. Correspondingly, the "noisy" conditional probability, denoted as Nu is also smoothed be continuous on Rd. As + 0, 117 1//loe + 0 on 22 and f2 and the limiting tio is a piecewise constant function with discontinuity at the induced decision boundary.
Lemma 3.5 (Tsybakov's noise condition under Gaussian noises). Let the margin be 2y 0, the noise be (0, Ia). Then there exist some constants T, 0 such that for any < < <
Cv² 7
Px((27, - <
<
exp
2x
L.
Theorem 3.6 (Exponential convergence rate). Suppose the classes are separable with margin 2 > No matter how complicated Q, UΩ2 are, excess risk of the over parameterized neural network classifier satisfying Assumptions D.I and D.4 has the rate Op(e-
The proof of Theorem 3.6 involves taking the auxiliary noise zero, e.g., Up
1/v7. The
exponential convergence rate a direct outcome of Lemma 3.5 and Theorem 3.1 Note that our exponential convergence rate is much faster than existing ones under the similar separable setting Jj & Telgarsky 2019 Cao & Gu
2019
2020 which are all polynomial with n, e.g., Op(1//))
Remark 4. Theorems 3.4 and 3.6 share the same gist that the over parameterized neural network classifiers have exponential convergence rate when data are separable with positive margin, while the result of Theorem 3.6 is weaker than that of Theorem B.4, but with milder conditions. Nevertheless, Theorem 3.6 bridges the non-separable case and separable case.

---

Under review as a conference paper at ICLR 2022
4 MULTICLASS CLASSIHICATION
In binary classification, the labels are usually encoded as -1 and 1. When there are > 2 classes, the default label coding is one-hot. However, it empirically observed that this vanilla square loss struggles when the number of classes are large, for which scaling tricks have been proposed (Hui al, 2020). Another popular coding scheme is the simplex coding (Mrouch et al. 2012), which takes maximally separated K points on the sphere as label features. When K 2, this reduces to the typical -1, coding. Many advantages of the simplex coding
& Belkin 2020 Demirkaya
have been discussed, including its relationship with cross-entropy loss and supervised contrastive
learning (Papyan et al. 2020 Han et al. 2021 Graf et al. 2021, Fang et al. 202T
In this work, we adopt the simplex coding. More discussion and empirical comparison about the
coding choices can be found in Appendix G.2 Given the label coding, one can easily generalize the
theoretical development in Section 3 by employing the following objective function
min
Uj.waa (x:)) yi.)² + M1|| WI
where fw.a f + RK and y: (v4,1, v1,k) T is the label of i-th observation.
The following proposition states a relationship between the simplex coding scheme and the condi- tional probability.
Proposition 4.1 (Conditional probability). Let f Ω R minimize the mean square error (x) - vy)², where Vy is the simplex coding vector of label y. Then we have
(x) == P (y = k|x) = ((K - 1)f (x) un + +1) /K.
(4.1)
Unlike the softmax function when using cross entropy, the estimated conditional probability using square loss is not guaranteed to be within 0 and 1. This will cause issues for adversarial attacks, which will be discussed in detail in Appendix G.2
5 NUMERICAL STUDIES
Although our theoretical results are for overparametrized neural network in the NTK regime, we expect our conclusions to generalize practical network architectures. The focus of this section is not on improving the state-of-the-art performance for deep classifiers, but to illustrate the difference between cross-entropy and square loss. We provide experiment results on both synthetic and real data, to support our theoretical findings and illustrate the practical benefits of square loss in training overparametrized DNN classifiers. Compared with cross-entropy, the square loss has comparable generalization performance, but with stronger robustness and smaller calibration error.
5.1 SYNTHETIC DATA
We consider the square loss based and cross-entropy based overparametrized neural networks (ONN) with & regularization, denoted SL-ONN la and CE-ONN + f2, respectively. The chosen ONNs are wwo-hidden-layer ReLU neural networks with 500 neurons for each layer, and the parameter y is selected via validation set. More implementation details are in Appendix G.1
Separable case
We consider two separated classes with spiral curve like supports. We also present the performance of the cross-entropy based ONN without & regularization (CE-ONN). Figure] shows one instance of the test misclassification rate and decision boundaries attained by SL-ONN le (Left), CE-ONN + f2 (Center), and CE-ONN (Right). From this example and other examples in Appendix G.I it can be seen that SL-ONN + l has smaller test misclassification rate and a much smoother decision boundary. In particular, the red region, where the training data are sparse, SL-ONN + & fits the correct data distribution best.
Non-separable case We consider the conditional probability Tπ(x) sin 12), [-,, 1] and the calibration performance of SL-ONN Eg and CE-ONN + b,, where the classifiers
denoted by fiz and Fre. respectively. The results are presented in Figure G.8 in the Appendix.
7

---

Under review as a conference paper at ICLR 2022
Square scouracy 100.00
COOSS entropyimm accurccy 99
oreropytim test scoursty 95,99998.8
Librake) 1[lran)
&
of Li
Figure 1: Test misclassification rates and decision boundaries predicted by: SL.-NNN l2 (Left);;
CE-ONN + l2 (Center); CE-ONN (Right) for the separable case.
The error bar plot of the test calibration error shows that fi2 has the smaller mean and standard deviation than See- It suggests that square loss generally outperforms cross entropy in calibration. The histogram and kernel density estimation of the test calibration errors for one case show that the pointwise calibration errors on the test points of fiz are more concentrated around zero than those of fee. Moreover, despite comparable misclassification rate with fce. fi2 has smaller calibration error. Figure G.8 demonstrates that SL.-ONN + & recovers r much hetter than CE-ONN + f2.
5.2 REAL DATA
To make a fair comparison, we adopt popular architectures, ResNet (He et al. 2016) and Wide ResNet (Zagoruyko & Komodakis 2016) and evaluate them on the CIFAR image classification datasets, with only the training loss function changed, from cross-entropy (CE) to square loss with simplex coding (SL). Further, we don't i employ any large scale hyper-paramete tuning and all the parameters are kept as default except for the learning rate (Ir) and batch size (bs), where we are choosing from the better of (Ir=0.01, bs=32) and (Ir=0.1, bs=128). Each experiment setting repli- cated 5 times and we report the average performance followed by its standard deviation in the paren- thesis. (Ir=0.01, bs=32) works better for the most cases except for square loss trained WRN-16-N 10 on CIFAR-100. More experiment details and additional results can be found in Appendix 1.
Generalization In both CIFAR-10 10 and CIFAR- 100, the performance of cross-entropy and square loss with simplex coding are quite comparable, observed in Hui & Belkin (2020). Cross-entropy tends to perform slightly better for ResNet, especially on CIFAR- -00 with an advantage of less than 1%. There is more significant gap with Wide ResNet where square loss outperforms cross-entropy by more than %% on both CIFAR-10 10 and CIFAR-100. The details can found in Table [
Table 1: Test accuracy on CIFAR datasets. Average accuracy larger than O but less than 0.1 is denoted as 0 without standard deviation.
PGD- 100 (la-sstrenghh)
AutoAAtuck (1 sttrenggh))))
Dataset
Network
Loss
Clean ace %
2/255
4/256
8/255
2/255
4/255
8/255
CE
95.15 (0.11)
8.81 (1.61)
0.65 (0.24)
0
2.74 (0.09)
18
SI.
95.04 (0.07)
30.53 (0.92)
6.64 (0.67)
0.86 (0.24)
4.10 (0.50)
0*
0
CIFAR-10
CE
93.94 (0.16)
1.04 (0)
0
0
0.33 (0.06)
0
WRN-16-10
SL
95.02 (0.11)
37.47 (0.61)
23.16 (1.28)
7.88 (0.72)
5.37 (0.50)
0'
CE
79.82 (0.14)
2.31 (0.07)
0'
0
0.99 (0.10)
0"
0
ResNet-50
S1.
78.91 (0.14)
13.76 (1.30)
4.63 (1.20)
1.21 (0.80)
3.67 (0.60)
0.16 (0.05)
0
CIFAR-100
CE
77.89 (0.21)
0.83 (0.07)
0
0.42 (0.07)
WRN-16-10 10
SI.
79.65 (0.15)
6.48 (0.40)
0.42 (0.04)
0"
2.73 (0.20)
0"
0
Adversarial robustness Normally trained deep classifiers are found to be adversarially vulnerable and adversarial attacks provide a powerful tool to evaluate classification robustness. For our exper- iment, we consider the black-bor Gaussian noise attack, the classic white-box PGD attack (Madry et al. 2017) and the of-te-of-the-an AutoAttack (Croce & Hein 2020), with attack strength level 2/255, 4/255, 8/255 in lxx norm. AutoAttack contains both white-box and black-box attacks and offers more comprehensive evaluation of adversarial robustness. The Gaussian noises results
8

---

Under review as a conference paper at ICLR 2022
Table Performance on CIFAR-10 dataset for ResNet-18 under standard PGD adversarial training.
CIFARIO
Loss
Acc (%)
PGD steps
Strength(/wo)
Autoattack
86.87
3
8/255
37.08
ResNet-18
CE
84.50
7
8/255
41.88
87.31
3
8/255
40,46
ResNet-18
SL
84.52
7
8/255
44.76
presented in Table G.3 in the Appendix. At different noise levels, square loss consistently outper- forms cross-entropy, especially for WRN-16-10, with around 2-4% accuracy improvement. More details can be found in Appendix G.2] The PGD and AutoAAtack results are reported in Table ! Even though classifiers trained with square loss is far away from adversarially robust, consistently gives significantly higher adversarial accuracy. The same margin can be carried over to standard adversarial training as well. Table 2
lists results from standard PGD adversarial training with CE and SL. By substituting cross-entropy loss to square loss, the robust accuracy increased around 3% while maintaining higher clean accuracy.
One thing to notice is that when constructing white-box attacks, square loss will not work well since doesn't directly reflect the classification accuracy. More specifically, for a correctly classified image (x, y). maximizing the square loss may result in linear scaling of the classifier f(x). which doesn't change the predicted class (see Appendix G.2 for more discussion). To this end, we consider special attack for classifiers trained by square loss by maximizing the cosine similarity between (x) and Vy. We call this angle attack and also utilize it for the PGD adversarial training paired with square loss in Table In our experiments, this special attack rarely outperforms the standard PGD with cross-entropy and the reported PGD accuracy are from the latter settings. This property of square loss may be an advantage in defending adversarial attacks.
The predicted class probabilities for square loss can be obtained from Equa- tion 4.1 Expected calibration error (ECE) measures the absolute difference between predicted confi- dence and the actual accuracy. Deep classifiers are usually found to be over-confident (Vaicenavicius al.
Model calibration
2019p. Using ResNet as an example, we report the typical reliability diagram in Figure CIFAR-10 10 with ResNet-18 18, the average ECE for cross-entropy is 0.028 (0.002) while that for square loss 0.0097 (0.001). On CIFAR- 100 with ResNet-50, the average ECE for cross-entropy is 0.094 (0.005) while that for square loss is 0.068 (0.005). Square loss results are much more calibrated with significantly smaller ECE.
CIFAR-10 ResNet-18
CIFAR-10 ResNet-18
CIFAR-100 ResNet-50 CE
CIFAR-100 ResNet-50
Sap
Gap
6.1
40
04
16
64
Lo
00
04
64
50
04
46
Confidence
Confidence
Confidence
Confidence
Figure 2: Reliability diagrams of ResNet- 18 on CIFAR-10 and ResNet-50 on CIFAR-100. Square loss trained models behave more well-calibrated while cross-entropy trained ones tend be visibly more over-confident.
CONCLUSIONS
Classification problems are ubiquitous in deep learning. As fundamental problem, any progress in classification can potentially benefit numerous relevant tasks. Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked. Through both theoretical analysis and empirical studies, identify several ideal properties of using square loss in training neural network classifiers, including provable fast convergence rates, strong robustness, and small calibration error. We encourage readers try square loss in your application scenarios.

---

Under review as a conference paper at ICLR 2022
Ethnics Statement We acknowledge the ICLR Code of Ethics. This submission is mostly theo- retical and the authors could not think of any potential violations of them this submission.
Reproducibility Statement For theoretical results, explanations of assumptions can be found in Appendix D and complete proof of the claims can be found in the Appendix [ and Appendix Our experiment details can be found in Appendix G with both data and training descriptions.
REFERENCES
N. Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc., (8(3):337-400) 1950.
Sanjeev Arora, Simon Du. Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- sine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584 2019.
Jean- Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of statistics, 35(2):608-63) 633, 2007.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds.
Journal of the American Statistical Association, 101 136-166,,6661 2006.
Colin Bennett and Robert C Sharpley. Interpolation of Operators. Academic press, 1988.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. arXiv preprint 1rXi::195S.12133 2019.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks, Advances in Neural Information Processing Systems, 32:10836-10846, 2019.
Yuan Cao and Quanquan Gu. Generalization error bounds gradient descent for learning over- parameterized deep relu networks. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 34, pp. 3359-3356, 2020.
Lin Chen and Sheng Xu. Deep neural tangent kernel and Laplace kernel have the same RKHS. arXiv preprint arXiv:2009. /0683, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- fereeter--feeeeeee attacks. In International Conference on Machine Learning, pp. 2206- 2216. PMLR, 2020.
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp- 1-5. IEEE, 2020.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv: 18/.0278/, 2018.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint 111.0:1810.02054, 2018.
David Eric Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators, volume 120. Cambridge University Press, 2008
Gamaleldin F Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. arXiv preprint 18x.v:/803.05555 2018.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43), 2021.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and infer- ence. Econometrica, 89(1):181-21 2021.
10

---

Under review as a conference paper at ICLR 2022
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Ronen Basri. On the similarity between the Laplace and neural tangent kernels. arXiv preprint arXiv:2007.01550, 2020.
Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised con- strastive learning. In International Conference on Machine Learning, pp. 3821-3830. PMLR, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint arXiv:2106.02073, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Zhezhi He. Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp- 518-588877777 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 150.00333,,,,,,, 2015.
Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A nonparamet- ric perspective overparametrized neural network. In International Conference Artificial Intelligence and Statistics, pp. 829-837. PMLR, 2021.
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. International Conference on Learning Rep- resentations, 2020.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross- entropy classification tasks. arXiv preprint arXiv:2006.07322. 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. In Advances in Neural Information Processing Systems, pp. 8571- 1- 8580, 2010.
Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbi- trarily small test error with shallow ReLU networks. arXiv preprint arXiv:1909.12299 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1 / 1362, 2020.
Yongdai Kim, Ilsang Ohn, and Dongha Kim. Fast convergence rates of deep neural networks for classification. arXiv preprint arXin:1812.03500 2018.
Michael Kohler and Adam Krzyzak. On the rate of convergence of local averaging plug-in classifica- tion rules under a margin condition. IEEE Transactions on Information (eeeoy,53(5):73551111 2007.
Simon Kornblith, Honglak Lee, Ting Chen, and Mohammad Norouzi. Demystifying loss functions for classification. 2020.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157- 8166, 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 17X6::77060668. 2017.
11

---

Under review as a conference paper at ICLR 2022
Enno Mammen and Alexandre B Tsybakov. Smooth discrimination analysis. The Annals of Statis- tics, (7(6):1808 (0000:8888-8228 1829, 1999.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statisti- cal perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Youssef Mroueh, Tomaso Poggio, Lorenzo Rosasco, and Jean-Jacques Slotine. Multiclass learning with simplex coding. arXiv preprint arXiv:/209./360 2012.
Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification regression in overparameterized regimes: Does the loss function matter? arXiv preprint www.xx2005.080555 2020.
Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. arXiv preprint arXiv:2006. 12297, 2020.
Atsushi Nitanda, Geoffrey Chinot, and Taiji Suzuki. Gradient descent can learn less over-parameterized two-layer neural networks on classification problems. arXiv preprint arXiv:1905.008800. 2019.
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross-entropy loss for adversarial robustness. arXiv preprint 1005 2019.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 14652-24663, 2020.
Tomaso Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square loss. Technical report, CBMM Memo No, 2019,
Johannes Schmidt-Hieber Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(44)1875-118) 2020.
Ingo Steinwart, Clint Scovel, et al. Fast rates for support vector machines using Gaussian kernels. The Annals of Statistics. 35(2):575-607, 2007.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 3459-3467. PMLR, 2019.
Holger Wendland. Scattered Data Approximation. Cambridge University Press, 2004.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arX1v:1605.07146. 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: arXiv: 17140.942. 2017.
Tong Zhang. Statistical behavior and consistency of classification methods based convex risk minimization. The Annals of Statistics, pp. 56-85, 2004.
12


Reviews:
[{"review_id": "o-SJwGo7tEr", "reviewer": "Reviewer_MNxz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n+ The paper improves on previous results on the convergence of neural networks, by deriving a faster convergence rate (however under stronger assumptions). \n\n+ In contrast to prior work, which assumes data to be linearly separable, the paper studies the more general and much more realistic setting of separable data (and also of unseparable data).\n\n\n**Cons** (see also detailed comments below)\n- The presentation of the theoretical results is superficial and crucial information can only be found in the appendix (see detailed comments below)\n\n- The content and setting of the experiments differs from the theoretical contributions and thus they do not corroborate them.\n\n- There are some claims in the paper, which need more backing.\n\n\n**Detailed comments:**\n- Presentation of the results:\n\n  - Crucial information is missing from the main text and can only be found in the supplementary material, most importantly, the assumptions under which the theorems hold. Furthermore, these assumptions are only stated, but not discussed. In consequence, it remains unclear, whether these assumptions are, for example, fulfilled in practice, might be loosened, or and where they come from.\n\n  - The theoretical results are not properly stated as they do not specify what the respective variables mean. This becomes a real issue, when the variables have not been defined in the text so far (or at any later point).  \nFor example, Theorem 3.1 is a asymptotic equality for some function $\\hat f$, which is not specified in the main text. In fact, $\\hat f$ is only defined in the passing, somewhere in the supplementary material as a solution of some minimization problem stated one page earlier. However, then one needs to look up the function space on which the minimization occurs, which in turn is specified two pages earlier. But this is only helpful, if the reader already knows, what the \"reproducing hilbert space generated by the neural tangent kernel on $\\mathbb S^{d-1}$\" is.  This is clearly suboptimal.  \nSimilarly, Theorem 3.2 contains a summand $\\delta$, which is not defined anywhere, and appears to can be chosen $=0$ anyway.\nTheorem 3.6 contains  the imprecise formulation \"no matter how complicated $\\Omega_1 \\cup \\Omega_2$ are\". Does this mean for any $\\Omega_1 \\cup \\Omega_2 \\subset \\Omega$?. It would also improve the readability, if the definitions of $\\Omega_1$ and $\\Omega_2$ are restated in the theorem or in its proximity.\n\n- Experiments:\n  - The experiments are somewhat disconnected from the theoretical part. In the experiments, a comparison between the squared loss and the cross entropy loss is made, similar to the empirical study [1]. However, the theory exclusively covers properties of the square loss, and thus does not make any statement or prediction whether training with either of these loss functions is preferable. From my perspective, a comprehensive on the extent to which the theoretical predictions hold would have been more valuable.\n\n  - Similarly, the experiments are not designed according to the presented theory. The theory mainly treats  1-hidden-layer networks, whereas experiments use either 2-hidden layer experiments (5.1) or ResNets (5.2). Furthermore, the 500 hidden neurons in (5.1) do not seem to be wide enough, such that the assumptions of the theorem are fulfilled.\n\n- Not sufficiently supported claims:\n  - Remark following Theorem 3.4  \n\"Under {0,1} coding, the estimator would be $f_{W,A}$ itself. The ${L_\\infty}$ consistency doesn’t hold for cross-entropy trained neural networks, due to the form of the optimal solution $\\log(\\frac\\eta{1-\\eta})$. With limited capacity, the network’s confidence prediction is bounded away from 0 and 1.\"\nThis needs to be discussed in more detail, as Theorem 3.4 is of asymptotic nature (infinite width), but the presented argument only rules out networks with limited capacity.\n\n  - Regarding the simplex label coding  \n  \"The one-hot coding forces the features for each class to be orthogonal in the unit sphere, which is problematic for large K since the representations are all cramped into the corner (positive direction in each dimension) taking only $2^{-K}$ of the entire sphere.\nIn comparison, maximally separated K points on the unit sphere is the better choice, which coincides\nwith the vertices of a (K-1)-simplex in K-1 dimensions.\"  \nFrom my understanding, this not an issue. As has already been shown in the neural collapse literature, the one-hot encoding leads (at optimum) to a simplex coding in the penultimate layer. See for example [2].\n\n\n- Misc.\n  - Proposition 4.2 has already been proven in [3] and in a more general case in [4]. Furthermore, here the proof appears to only cover the special case of 2 instances per class.\n\n  - The train and test data in Figure 1 are very difficult to distinguish.\n\nReferences:  \n[1] Like Hui and Mikhail Belkin, Evaluation of neural architectures trained with square loss vs cross-\nentropy in classification task, ICLR 2021  \n[2] DG Mixon, H Parshall, J Pi, Neural collapse with unconstrained features. arXiv:2011.11619  \n[3] C. Fang, H. He, Q. Long, and W. Su, Exploring deep neural networks via layer-peeled model: Minority  \ncollapse in imbalanced training, Proceedings of the National Academy of Sciences, 2021  \n[4] Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt, Dissecting supervised contrastive\nlearning, ICML 2021  ", "summary_of_the_paper": "The paper studies wide neural networks trained with the square loss in the neural tangent kernel regime from a theoretical perspective. In particular, it provides generalization error bounds, robustness and calibration results and makes a connection to supervised contrastive learning. Furthermore, an empirical comparison between cross entropy and square loss is provided.", "summary_of_the_review": "I am not very familiar with the related work and thus cannot really evaluate the papers significance and potential impact. That being said, the paper is difficult to navigate, as crucial information lies in the appendix and the experiments do, from my perspective, not really corroborate the theoretical contribution. Therefore I cautiously give a score of 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Ush70O7c8t", "reviewer": "Reviewer_Eo56", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThe theoretical analysis of this paper is rich, including generalization and robustness theory.\n2.\tIt fully verifies the theoretical advantages of square loss in several perspectives and provides the corresponding experimental verification for theoretical results in the neural tangent kernel (NTK) regime.\n\nWeaknesses:\n1.\tWhen introducing the theoretical results, we should make a detailed comparison with the existing cross-entropy loss results. The current writing method cannot reflect the advantages of square loss.\n2.\tThe synthetic experiment in a non-separable case seems to be a problem. Considering the nonlinear expression ability of neural networks, how to explain that the data distribution illustrated in Figure 1 is inseparable from the network model?\n3.\tThis paper presents that the loss functions like hinge loss don’t provide reliable information on the prediction confidence. In this regard, there is a lack of references to some relevant literature. [Gao, 2013] has given a detailed analysis of the advantages and disadvantages between the entire margin distribution and the minimum margin. Based on this, [Lyu, 2018] designed a square-type margin distribution loss to improve the generalization ability of DNN.\n\n[Gao, 2013] W. Gao and Z.-H. Zhou. On the doubt about margin explanation of boosting. Artificial Intelligence 203:1-18 2013.\n\n[Lyu, 2018] Shen-Huan Lyu, Lu Wang, and Zhi-Hua Zhou. Improving Generalization of Neural Networks by Leveraging Margin Distribution. http://arxiv.org/abs/1812.10761\n\n", "summary_of_the_paper": "This paper provides a theoretical guarantee for square loss in the neural tangent kernel (NTK) regime. Whether classes are separable or not, the convergence rate is proved to be improved. ", "summary_of_the_review": "In a word, I agree with the argument put forward in this paper and think its theoretical conclusion is correct. If the author can provide the theoretical comparison between SL and CL and supplement some missing citations of related work, I will more recognize the integrity of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SlEN_-9Gtk", "reviewer": "Reviewer_Yckn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern about this submission is its relevance and importance. Most realistic experiments in this paper is on CIFAR-10/100, however I am not sure if the experiments included in the submission are convincing. Let me first describe my concern about the robustness experiments. \n\nAuthors measure the robustness of the square loss and cross entropy models using two kinds of adversarial attacks, PGD and AutoAttack. In general results using white-box adversarial attacks can be hard to interpret, as the performance of white-box attacks depend heavily on implementation details and attack hyperparameters. This was one reason historically that adversarial defense papers would get published in leading conferences only to be broken in less than a month. For the purposes of this submission, adversarial attacks are further concerning since the hyperparameters of these attacks were designed and optimized for cross-entropy loss, but the authors apply them to square loss here without re-optimizing them. I acknowledge that AutoAttack contains black-box attacks, such as SquareAttack, which may not suffer as much from this problem. However, it is not clear to me how much of the improvement of square loss on AutoAttack robustness is due to its improvement of SquareAttack. Furthermore, it is not clear to me if an improvement from 1% to 3% accuracy is significant, both of them seem low enough to be effectively useless. For robustness evaluation, I would recommend a simpler black-box noise such as Gaussian noise on the input with several noise scales. Another interesting experiment would be to evaluate CIFAR-10-C performance, which contains lots of noise-based corruptions that relate to the definition of robustness used in this submission.   \n\nNext concern is about the calibration results. It is nice to see that on the models tried by the authors, SL leads to an improved ECE. To evaluate the significance of this improvement, I would suggest including comparisons to ECE measures reported by published papers for the same model. This way the reader can see if the SL can lead to an improvement on ECE relative to the implementation of the same neural networks implemented by others. I think it would also be more helpful to train a more conventional and performant model such as WRN-28-10, for which there are plenty of available comparisons in literature.    ", "summary_of_the_paper": "This paper studies square loss theoretically in the NTK regime. In this particular regime, authors find indications that square loss might lead to (adversarially) robust models and good calibration. Authors then run experiments outside of the NTK regime, on CIFAR-10 and CIFAR-100, to compare the robustness and calibration performance of square loss with cross-entropy.   ", "summary_of_the_review": "Theoretical analysis is limited to toy models. Empirical studies should be improved demonstrate that square loss leads to more robust and calibrated models. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DJouix8wry", "reviewer": "Reviewer_2eZX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ New convergence results and insights of squared loss for classification. \n+ The results hold for a non-separable case using the Tsybakov noise condition.\n+ Additional insights on the robustness and model calibration.\n\nCons:\n- The analysis under the NTK regime is somewhat standard and may not be practical. For example, when mu=0, GD operates under the kernel regime, which do not explain some phenomenon observed in practice such implicit bias.\n- There are a number of restrictive assumptions, for example, C.1, C.2 and C.3. In linear regression (Du et al., 2019), C.1 is sufficient, and C.2 seems to be sufficient for logistic regression (Ji et al. 2020). \n- The experimental results on real data reinforce but perhaps repeat previous work.\n\n\nQuestions:\n1. When is the definition of separability (when \\eta only takes either 0 or 1 in two separate sets) more general than the linear separable case?\n2. How does this result compare to other results of implicit biases of gradient descent for separable and non-separable data? Similarly, how does the convergence rate compare to that of Ji et al., 2020, which uses logistic loss and seems to assume less restrictive assumptions? A discussion would be helpful for readers.\n3. It looks like the regularization term is a crucial part in the analysis important for this analysis. Does the analysis carry over if \\mu is exactly zero?\n4. Paragraph under Theorem 3.1: where is the (d-1)\\kappa that you’re mentioning?\n5. Assumption C.1: why does delta_n depend on n? When n=1, Eq. (A.3) implies that it is positive, right? Also, \\lambda_0 scales with poly(1/n), when n -> infty, is \\lambda_0 = 0?\n6. Can you say anything about the robustness of squared loss to outliers? Empirical evidence is sufficient. Assumption C.3 seems to be related to that.\n\nMinor:\n\n1. Equation (2.1): the first expectation is taken over both (X, Y) ~ P? What does expectation over f(X) >= 1 mean? It can be re-written over X in a more precise way.\n2. Please consider using \\mathbf{a} instead of capital A for the second layer weight because the latter can be easily confused with a matrix. The objective in (2.2) gives the impression that A is optimized, but it is not. The analysis would be very different with a joint optimization over both W and A.\n3. The color bar in Figure 1: how probability P[y=1] can be negative? I don’t see any color difference in each of the plots.", "summary_of_the_paper": "This paper provides a theoretical analysis of the squared loss for classification where cross-entropy loss is often the standard choice in both theory and practice. The authors show under some assumption of the data and the NTK regime, GD on squared loss enjoys fast convergence, strong robustness and good calibration. They also provide empirical results.\n", "summary_of_the_review": "See above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8mTW4TAWXxa", "reviewer": "Reviewer_RzXw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The non-parametric fast rate under the Tsybakov noise condition in the NTK Regime.\n- Results of model calibration are impressive.\n- some theory and experiment have gap\n\nWeakness:\n- No comparison between L2 and cross-entropy loss\n- lack of reference\n\nDetailed comments:\n\n**In terms of theory** The paper investigated standard setting of low noise condition (theorem 3.1) and achieves the exponential convergence rate for the separable case. The most interesting part is Theorem 3.4 (Calibration error). However, Theorem 3.4 doesn't have the tsybakov's noise condition kappa in the final convergence bound. Is it standard in this setting? Is it tight? It's slower than the convergence rate of the Excess risk?\n\n**Simplex label coding** I guess most of the prove have been done by the neural collapse literature [7,8] , and it's also true for the CE loss [9].\n\nThe paper gives positive results for L2 losses,however, the paper didn't show any results that the cross-entropy loss is not good. The convergence of CE loss can be slow in the terminal phase of deep learning training [6] (O(1/log t)), but if you train longer, maybe all the properties you proved can appear. I hope the author can provide that why CE loss is not good enough.\n\n**Adversarial robustness** The gain of SL interms of the adversarial loss is actually not significant, I would to suggest the author to try more norms(l2 norm robustness as example) can be tried in this setting. In the experiment I can't see what's the norm the author is using for PGD.(The number here looks like l_inf norm, but in theory theorem 3.3. it's l2 norm. I see a mismatch here.) For l2 robustness, you can further try randomized smoothing [5].\n\n\n**Missing Reference** Optimal rate for NTK for regression [1,2] and outof NTK regime (but no optimization)[3,4], both of the papers using l2 loss.\n\n[1] Nitanda A, Suzuki T. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime[J]. arXiv preprint arXiv:2006.12297, 2020.\n\n[2] Hu T, Wang W, Lin C, et al. Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2021: 829-837.\n\n[3] Schmidt-Hieber J. Nonparametric regression using deep neural networks with ReLU activation function[J]. The Annals of Statistics, 2020, 48(4): 1875-1897.\n\n[4] Farrell M H, Liang T, Misra S. Deep neural networks for estimation and inference[J]. Econometrica, 2021, 89(1): 181-213.\n\n[5] Cohen J, Rosenfeld E, Kolter Z. Certified adversarial robustness via randomized smoothing International Conference on Machine Learning. PMLR, 2019: 1310-1320.\n\n[6] Lyu K, Li J. Gradient descent maximizes the margin of homogeneous neural networks[J]. arXiv preprint arXiv:1906.05890, 2019.\n\n[7] Fang C, He H, Long Q, et al. Layer-peeled model: Toward understanding well-trained deep neural networks[J]. arXiv preprint arXiv:2101.12699, 2021.\n\n[8] Zhu Z, Ding T, Zhou J, et al. A Geometric Analysis of Neural Collapse with Unconstrained Features[J]. arXiv preprint arXiv:2105.02375, 2021.\n\n[9] Ji W, Lu Y, Zhang Y, et al. An Unconstrained Layer-Peeled Perspective on Neural Collapse[J]. arXiv preprint arXiv:2110.02796, 2021.\n\n", "summary_of_the_paper": "This paper considers the non-parametric convergence rate of neural network (NTK regime) under tsybakov's noise condition (low noise with large margin) and discussed the potential application of the theoretical result on the robustness of neural network and calibration results. ", "summary_of_the_review": "my main two concern:\n- missing discussion of the relationship between line of research of using NN as a non-parametric estimator.\n- I don't see any discussion of the specialty of the l2 loss. The paper don't answer the question of how l2 loss and CE loss is different. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes a theoretical understanding of training over-parametrized deep neural networks using gradient descent with respect to square loss in the NTK regime. Besides giving guarantees on the classification accuracy using square loss, authors reveal several interesting properties in this regime including robustness and calibration. 

The problem studied here is exciting and very relevant. The current version, unfortunately, has some shortcomings. For example, under a margin assumption, the authors show that the least-squares solution finds something with the margin and, therefore, it yields “robustness.” There is no quantification of how “robust” is the trained model, what is the threat model, what if the noise budget is larger than the attained margin. In general, the analysis lacks any careful finer characterization or quantification of the claimed properties. Besides, as was pointed out, the setting of the neural tangent kernel regime is somewhat limited and to some extent impractical. The assumptions under which the results hold further make the setting of the paper significantly restrictive. 

The writing can be improved with more emphasis on the novelty and significance of the contributions. Currently, all of the assumptions are buried in the appendix and the main paper is not even self-contained. I believe the comments from the reviewers have already helped improve the quality of the paper. I encourage the authors to further incorporate the feedback and work towards a stronger submission.
Decision: Reject