EVALUATION OF NEURAL ARCHITECTURES TRAINED WITH SQUARE LOSS VS CROSS-ENTROPY IN CLASSIFICATION TASKS
Authors: Like Hui
Mikhail Belkin
Venue: ICLR 2021 Poster (2021)
Keywords: large scale learning
square loss vs cross-entropy
classification
experimental evaluation

Abstract: Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be well-founded. 
We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources.
Indeed, we observe that the square loss produces better results in the dominant majority of NLP and ASR experiments. Cross-entropy appears to have a slight edge on computer vision tasks.

We argue that there is little compelling empirical or theoretical evidence indicating a clear-cut advantage to the cross-entropy loss. Indeed, in our experiments, performance on nearly all non-vision tasks  can be improved, sometimes significantly, by switching to the square loss. Furthermore, training with square loss appears to be less sensitive to the randomness in initialization. We posit that
training using the square loss for classification needs to be a part of best practices of modern deep learning on equal footing with cross-entropy. 

Published as a conference paper at ICLR 2021
EVALUATION OF NEURAL ARCHITECTURES TRAINED IN CLASSI-
WITH SQUARE Loss VS FICATION TASKS
Like Hui
Mikhail Belkin
Computer Science and Engineering University of California, San Diego San Diego, CA 92093 1uu facsd.edu
Hahcıoğlu Data Science Institute University of California, San Diego San Diego, CA 92093 mbe 1 k kikkkuccddeee.
ABSTRACT
Modern neural architectures for classification tasks trained using the cross- entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be well- founded. We explore several major neural architectures and a range standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter set- tings as reported in the literature, perform comparably better when trained with the square loss, even after equalizing computational resources. Indeed, we ob- serve that the square loss produces better results in the dominant majority of NL.P and ASR experiments. Cross-entropy appears to have slight edge on computer vision tasks.
We argue that there is little compelling empirical or theoretical evidence indicating a clear-cut advantage to the cross-entropy loss. Indeed, in our experiments, per- formance nearly all non-vision tasks be improved, sometimes significantly, by switching to the square loss. Furthermore, training with square loss appears to be less sensitive the randomness in initialization. We posit that training using the square loss for classification needs to be part of best practices of modern deep learning on equal footing with cross-entropy.
INTRODUCTION
Modern deep neural networks are nearly universally trained with cross-entropy loss classification tasks. To illustrate, cross-entropy is the only loss function specifically discussed in connection with training neural networks for classification in popular references (Goodfellow et al. 2016 Zhang al. 2020). It is the default for classification in widely used packages such NLP implementation Hugging Face Transformers (Wolf et al.) 2019). speech classification by ESPnet (Watanabe et al.) 2018) and image classification implemented by torchvision (Marcel & Rodriguez 2010). Yet know of few empirical evaluations or compelling theoretical analyses to justify the predominance of cross-entropy in practice. In what follows, we use a number of modern deep learning architectures and standard datasets across the range of tasks of natural language processing, speech recognition and computer vision domains as basis for a systematic comparison between the cross-entropy and square losses. The square loss (also known as the Brier score (Brier) 1950) in the classification context) is a particularly useful basis for comparison since it is nearly universally used for regression tasks and is available in all major software packages. To ensure fair evaluation, for the square loss we use hyper- hyper-parameter parameter settings and architectures exactly as reported in the literature for cross- entropy, with the exception of the learning rate, which needs to be increased in comparison with cross-entropy and, for problems with a large number of classes (42 or more in our experiments), loss function rescaling (see Section

---

Published as a conference paper at ICLR 2021
Our evaluation includes 20 separate learning tasks" (neural model/dataset combinations) evaluated in terms of the error rate or, equivalently, accuracy (depending on the prevalent domain conven- tions). We also provide some additional domain-specifie evaluation metries Fl for NLP tasks, and Top-5 accuracy for ImageNet. Training with the square loss provides accuracy better or equal to that of cross-entropy in 17 out of 20 tasks. These results are for averages over multiple random initalizations, results for each individual initialization are similar. Furthermore, we find that training with the square loss has smaller variance with respect to the randomness of the initialization in the majority of our experiments.
Our results indicate that the models trained using the square loss are not just competitive with same models trained with cross-entropy across nearly all tasks and settings but, indeed, provide better classification results in the majority of our experiments. The performance advantage persists even when we equalize the amount of computation by choosing the number of epochs for training the square loss to be the same as the optimal (based on validation) number of epochs for eross-entropy, setting favorable to cross-entropy.
Note that with the exception of the learning rate, we utilized hyper-parameters reported in the lit- erature, originally optimized for the cross-entropy loss. This suggests that further improvements in performance for the square loss can potentially be obtained by hyper-paramette tuning.
Based on our results, we believe that the performance of modern architectures sification tasks may be improved by using the square loss in training. We conclude that the choice between the cross-entropy and the square loss for training needs to be important aspect of model selection, in addition to the standard considerations of optimization methods and tuning.
range of clas-
A historical note. The modern ubiquity of cross-entropy loss is reminiscent of the predominance of the hinge loss in the era of the Support Vector Machines (SVM), At the time, the prevailing in- tuition had been that the hinge loss was preferable the square loss for training classifiers. Yet, the empirical evidence had been decidedly mixed. In his remarkable thesis (Rifkin) (202)). Ryan Rifkin conducted an extensive empirical evaluation and concluded that "the performance of the RLSC [square loss] is essentially equivalent to that of the SVM [hinge loss] across a wide range of problems, and the choice belween the two should be based on computational tractability con- siderations". More recently, the experimental results in (Que & Belkin 2016) show an advantage to training with the square loss over the hinge loss across the majority of the tasks, paralleling our results in this paper. We note that conceptual or historical reasons for the current prevalence of cross-entropy in training neural networks are not entirely clear.
Theoretical considerations. The accepted justification of cross-entropy and hinge loss for clas- sification is that they are better "surrogates" for the 0-1 classification loss than the square loss, e.g. (Goodfellow et al. 2016). Section 8.2. There is little theoretical analysis supporting this point of view. To the contrary, the recent work (Muthukumar et al. 2020) proves that in certain over- parameterized regimes, the classifiers obtained by minimizing the hinge loss and the square loss in fact the same. While the hinge loss is different from cross-entropy, these losses are closely related in certain settings (Ji & Telgarsky 2019: Soudry et al., 2018). See (Muthukumar et al., 2020) for a more in-depth theoretical discussion of Toss functions and the related literature.
Probability interpretation of neural network output and calibration. An argument for using the cross-entropy loss function is sometimes based on the idea that networks trained with cross- entropy are able to output probability of a new data point belonging to given class. For linear models in the classical analysis of logistic regression, minimizing cross-entropy (logistic loss) in- deed yields the maximum likelihood estimator for the model c....(Harree) Jr 2015), Section 10.5). Yet, the relevance of that analysis to modern highly non- non-linear and often over seee-parameterizer neural networks is questionable. For example, in (Gal & Ghahramani 2016) the authors state that "In classification, predictive probabilities obtained at the end of the pipeline (the softmax output) are often erroneously interpreted as model confidence". Similarly, the work (Xing al., 2019) asserts that "for DNNs with conventional (also referred as 'vanilla") ') training to minimize the softmax cross- entropy loss, the outputs do not contain sufficient information for well-calibrated confidence estima- tion". Thus, accurate class probability estimation cannot be considered an unambiguous advantage
We note WSJ and Librispeech datasets have two separate classification tasks in terms of the evaluation metrics, based on the same learned acoustic model. We choose to count them as separate tasks.
2

---

Published as a conference paper at ICLR 2021
of neural networks trained with cross-entropy. While the analysis of calibration for different loss functions is beyond the scope of this paper, we note that in many practical settings accurate classifi- cation, the primary evaluation metric of this work, takes precedence over the probability estimation.
Domain applicability. It is interesting to note that in our experiments the square loss generally performs better on NLP and ASR tasks, while cross- crosspyntropy has a slight edge on computer vision. is tempting to infer that the square loss is suitable for NLP and speech, while cross-entropy may be more appropriate for training vision architectures. Yet we are wary of over-interpreting the evidence. In particular, we observe that the cross-entropy has significant performance advantage single vision architecture (EfficientNet (Tan & Le 2019) trained on ImageNet). The rest of the vision results quite similar between square loss and cross-entropy and are likely to be sensitive to the specifics of optimization and parameter tuning. Understanding whether specific loss functions are better suited for certain domain will require more in-depth experimental work.
just
Related work. The choice of loss function is an integral and essential aspect of training neural networks. Yet we are aware of few comparative analyses of loss functions and no other systematic studies of modern architectures across range of datasets. Kline & Berardi (2005) compared the effectiveness of squared-error versus cross-entropy in estimating posterior probabilities with small neural networks, five or less nodes in each layer, and argued that cross-entropy had performance advantage. Golik et al. (2013) provided a comparison of cross-entropy and squared error training for a hybrid HMM/neural net model for one ASR and one handwriting recognition datasets. The authors observed that with a good initialization by pre-training, training with the squared error had better performance than the cross-entropy. [Sangari & Sethares (2015) analyzed the convergence of mean squared error (MSE) and cross-entropy under the normalized logistic regression model (Soft-Max) setting, and indicated the MSE loss function is robust to the true model parameter val- ues and can converge to the same parameter estimation variance of the cross cross-entropy loss function with half the number of gradient descent iterations. Janocha & Czarnecki (2017) compared several different loss functions on MNIST and CIFAR-10 datasets concluding that "depending the ap- plication of the deep model- losses other than log loss [cross-entropy] are preferable". recent work (Demirkaya et al., 2020) provided a theoretical comparison of square and cross-entropy losses for training mixture models. The authors argued that the cross-entropy loss has more favorable opti- mization landscapes in multiclass settings. To alleviate that issue, they proposed rescaling of the loss function equivalent to choosing parameter ls in Section The authors showed that rescaling allowed the square loss to become competitive with cross-entropy on CIFAR- 100,, a finding that aligns with the results in our paper.
EXPERIMENTS
We conducted experiments on number of benchmark datasets for NLP, ASR and computer vision, following the standard recipes given in recent papers of cach domain. Four NLP datasets are MRPC, SST-2, QNLI and QQP. TIMIT, WSJ and Librispeech are three standard datasets used for training ASR systems. For vision experiments, we choose MNIST, CIFAR-10, and ImageNet. To the best of our knowledge, we are the first to experimentally compare the square loss and the cross-entropy on a wide range of datasets with different size, dimensionality (number of features) and the number of classes (up to 1000 class numbers). See Appendix [A] for references and description. Architectures. In what follows we explore several widely used modern neural architectures. For NLP tasks, we implement classifiers with a fine-tuned BERT (Devlin et al. 2018). LSTM+Attention model (Chen et al.) 2017). and a LSTM+CNN model (He & Lin 2016). Joint CTC-Attention based model (Kim et al. [017]). triggered attention model with VGG and BLSTM modules (Moritz et al. 2019) are used for ASR tasks. Note that for the CTC-Attention based model, the original loss function is a weighted sum of the cross- cross-entropy and the CTC loss. When training with the square loss, we only replace the cross-entropy to be the square loss, and keep the CTC loss untouched. For vision tasks, we use TCNN (Bai et al.) 2018). Wide ResNet (Zagoruyko & Komodakis 2016), ResNet (He et al. (2016) and EfficientNet (Tan & Le 2019) architectures.
Experimental protocols. For training with the cross-entropy loss, we use a standard protocol, which is to stop training after the validation accuracy does not improve for five consecutive epochs. For the square loss we use two protocols. The first one is the same as for cross-entroph The second protocol is to train the square loss using the number of epochs selected when training the cross-entropy loss with the first protocol. The second protocol is designed to equalize the usage of computational resources between the square loss and entss-yn and is favorable cross-entropy.
3

---

Published as a conference paper at ICLR 2021
Following the ryper-parameter settings of the architectures in the literature, re-implement the models trained with the cross-entropy loss keeping the same architecture and hyper-panee parameter set- tings. We train the same models using the square loss, employing The only alteration to the parameters of the network reported in the literature adjustment of the learning rate. For datasets with large number of labels (42 more in our experiments) apply loss function rescaling (see Section 5).
two experimental protocols.
The key points for the implementation specific hyper-parameter settings are given in Appendix [B] See Appendix [ for a summary of com- parisons between the original results and our i-impmmmmttatios. Additionally, we report the results on validation sets and training sets in Appendix C
described in Section The implementation details and
The results presented below are average results of runs corresponding to 5 different random inital- izations for each task. The result across initializations are given in Section
2.1 NLP EXPERIMENTS
We conduct 2-class classification tasks from NLP domain. The datasets information is summarized Table As in (Wang al. 2018), we report accuracy and F1 scores for MRPC and QQP datasets, and report accuracy for SST-2 and QNLI
Table I: NLP task statistics and descriptions
Corpus
Train
Test
#classes
Metric
Domain
MRPC (Dolan & Brockett 2005
3.7K
1.7K
2
acc./F1
news
SST-2 (ocher el al. 2013
67K
1.8K
acc.
movie reviews
QNLI (Rajpurkar al. 2016
105K
5.4K
2
acc.
Wikipedia
QQP (Iyer al. 2017)
364K
391K
acc./F1
social QA questions
Table 2
gives the accuracy and Table gives the F1 scores of the neural models on NLP tasks. As
Table NLP results, accuracy
train with
train with cross-entropy (%)
square loss w/ same epochs as CE (%)
Model
Task
square loss (%)
MRPC
83.8
82.1
83.6
BERT
SST-2
94.0
93.9
93.9
Devlin et al.
2018
QNLI
90.6
90.6
90.6
QQP
88.9
88.9
88.8
MRPC
71.7
70.9
71.5
LSTM+ +ITM+Atton Attention
QNLI
79.3
79.0
79.3
Chen et al. 2017
QQP
83.4
83.1
83.4
MRPC
73.2
69.4
72.5
LSTM+CNN
QNLI
76.0
76.0
76.0
HH & Lin 2016)
QQP
84.3
84.4
84.3
can be seen in Table 9 out of 10 tasks using the square loss has better/equal accuracy compared
with using the cross-entropy, and in terms of F1 score (see 3)) 5 out of 6 tasks training with
the square loss outperform training with the cross-entropy loss. Hven with same epochs, i.e. with same computation cost, using the square loss has equal/better accuracy in 8 out of 10 tasks and has higher Fl score out of tasks.
Table 3: NLP results, F1 scores
train with
train with cross-entropy (%)
square loss w/ same epochs as CE (%)
Model
Task
square loss (%)
BERT
MRPC
88.1
86.7
88.0
Devlin et al.
2018
QQP
70.9
70.7
70.7
Attentnn
MRPC
80.9
80.6
80.7
Chen al. 2017
QQP
62.6
62.3
62.6
LSTM NN
MRPC
81.0
78.2
81.0
He & Lin 2016
QQP
60.3
60.5
60.3
We observe the relative improvements brought by training with the square loss vary with different model architectures, and other than LSTM+CNN model QQP dataset, all architectures trained
4

---

Published as a conference paper at ICLR 2021
with the square loss have better/equal accuracy and F1 score. The performance of loss functions also varies with data size, especially for MRPC, which is a relatively small dataset, all model archi- tectures trained with the square loss gives significantly better results than the cross-entropy.
2.2 AUTOMATIC SPEECH RECOGNITION (ASR) EXPERIMENTS
We consider three datasets, TIMIT, WSJ and Librispeech, and all are ASR tasks. For Librispeech, we choose its train-clean-10 as training set, dev-clean and test-clean as validation and test set. We report phone error rate (PER) and character emror rate (CER) for TIMIT, word emror rate (WER) and CER for both WSJ and Librispeech. A brief description the datasets used in our ASR experiments given in Table Note that only alter the training loss of the acoustic model, while keeping
Table 4: ASR task statistics and descriptions
Corpus
Train
Test
#classes
Metric
Domain
TIMIT
42
PER
3.2 hours (training set)
1.15M
54K
Garofolo et al.
1993
27
CER
telephone English
WSJ
WER
80 hours (training set) read newspapers
28.8M
252K
52"
(Paul & Baker
1992)
CER
Librispeech
WER
100 hours (training set)
36M
IM
1000
Panayotov et al. 2015)
CER
audio books
This is the number of classes used for training the acoustic model.
the language model and decoding part the same as described in the literature. The acoustic model is classifier with the dictionary size the class number. For TIMIT, getting PER and CER needs two different acoustic models, i.e. they are two separate classification tasks, 42-class classification for PER, and 27-class classification for CER. For WSJ, the size of dictionary used for acoustic model is 52. WER and CER of WSJ are calculated with one acoustic model. Hence for WSJ it is 52-class classification task for both WER and CER. Acoustic model of Librispeech is a 1000-class classifier for both WER and CER, as we use 1000 unigram (Jurafsky 2000 based dictionary. The results are in 5
Table 5: ASR results, error rate
train with square loss (%)
train with cross-entropy (%)
square loss w/ same epochs as CE (%)
Model
Task
Attention+CTTC
TIMIT (PER)
20.8
20.8
20.8
Kim et al. 2017]
TIMIT (CER)
32.5
33.4
32.5
VGG+BLS STMP
WSJ (WER)
5.1
5.3
5.1
Moritz et al.
2019
WSJ (CER)
2.4
2.4
VGG+BLS TM
Librispeech (WER)
9.8
10.6
10.3
Moritz ct al. 2019
Librispeech (CER)
9.7
10.7
10.2
We see that the square loss performs better (equal for TIMIT PER result) in all of our tasks. It is interesting to observe that the performance advantage of the square loss reported in Jinleeses with dataset size. In particular, the relative advantage of the square loss (9.3% relative improvement on CER, and 7.5% on WER, respectively) largest for the biggest dataset, Librispeech. On WSJ, using the square loss has ~4 relative improvement on both CER and WER, while the results on TIMIT for the square loss and cross-entropy are very similar. The question whether this dependence between the data size and the relative advantage of the square loss cross-entropy is a coincidence or a recurring pattern requires further investigation.
For TIMIT and WSJ, observed that training with both the square loss and the cross-entropy need same epochs to converge. The two training protocols for training with the square loss have same performance, and both comparable/better than training with the cross-entropy. On Librispeech, the square loss needs more epochs, but provides better performance.
2.3 COMPUTER VISION EXPERIMENTS
For vision tasks we conduct experiments on MNIST, CIFAR-IO and ImageNet, as in Table
2 We measure the data size in terms of frame numbers, i.c. data samples. As we take frame shift to be 10ms, 1 hour data 360k frames.
5

---

Published as a conference paper at ICLR 2021
Table 6: Vision task statistics and descriptions
Corpus
Train
Test
#classes
Metric
Domain
MNIST LeCun al. 1998
60K
10K
10
acc.
x 28
CIFAR-10
Krizhevsky & Hinton 2009
50K
10K
10
acc.
32 x 32
ImageNet
1.28MM
50K
1000
acc.
224 x 224
[Russakovsky al.
2015
Top-5 acc.
As in Table 7 on MNIST and CIFAR-10, training with the square loss and the cross-entropy have comparable accuracy. much larger ImageNet, with ResNet-50 architecture, the accuracy and Top-5 accuracy of using the square loss are comparable with the ones got by using the cross-entropy loss. While with EfficientNet, using the cross-entropy shows better results. The performance of different loss functions varies among different architectures. On MNIST and CIFAR-10, we use exactly the same syper-parameters well-selected for the eross-y loss. For ImageNet, we adjust the learning rate and add simple rescaling scheme (see Section 5 all other the same as for the cross-entropy loss. The performance of using the square loss can improve with more hyper-paramettr tuning.
Table 7: Vision results, accuracy
Model
Task
with square loss (%)
train with cross-entropy (56)
square loss same epochs as CE (%)
TCNN (Bai et al. 2018
MNIST (acc.)
97.7
97.7
97.7
W-Resnet Zagordyko Komoudaki 2016
CIFAR-10 (acc.)
95.9
96.3
95.9
ResNee-SO
ImageeNet (acc.)
76.2
76.1
76.0
et 2016
ImageNet (Top-5 acc.)
93.0
92.9
Elleteetig
ImageNNt (acc.)
74.6
77.0
74.6
Image (Top-5 acc.)
92.7
92.7
For all three datasets, training with the square loss converges as fast training with the cross- entropy, and our two experimental protocols for the square loss result in same accuracy performance (except ImageNet with ResNet-50 model).
PERFORMANCE ACROSS DIFFERENT INITIALIZATIONS
Non (1E SCULD dioeeiz
intidlicaion
mean
mean
mean
Numoo Dre
Noan
MRPC
SST-2
ONLI
COP
MRPC
ONLI
QOP
MRPC
SST-2
ONU
(a) NLP: BERT
(b) NLP: LttMiAttentin
(c) NLP: LSTM+CNN
PR02I1 the Ste Reroonss 3
05
0%
T147 PORTMITI WEA Will COI LE: MEX CE:
TOW Whitee Top-itan Taad Top laps
(d) ASR: Error rate
(c) Vision: Accuracy
Figure 1: Difference between accuracy (or error rate) between square loss and CE for each initial- ization. (Square loss acc. CE acc.) is shown for accuracy, (CE Sqqare loss) for error rate.
We report validation set size, as results are on validation, following the papers for ImageNet tasks.

---

Published as a conference paper at ICLR 2021
To evaluate the stability of the results with respect to the randomness of model initialization we analyze the results for each random seed initialization. For each random seed, we calculate the difference between the the accuracy (or the error) of networks trained with the square loss and the cross-entropy respectively. We present the results with error bars for one standard deviation in Figure [ Absolute eтror and accuracy results for each run and the corresponding standard deviations are given in Appendix
Table 8: Standard deviation of test aceu- racy/error. Smaller number is bolded.
Model
Datasel
Square loss
CE
MRPC
0.484
0.766
SST-2
0.279
0.173
BERT
QNLI
0.241
0.205
QQP
0,045
0.063
MRPC
0,484
0.786
LSTM
QNLI
0.210
0.371
Attention
QQP
0,566
0.352
MRPC
0.322
0.383
LSTM
QNLI
0.173
0,286
+CNN
Table (Libri short for Librispeech and I-Net is short for ImageNet) shows the standard deviation of test accuracy/error for training with the square loss and cross-entropy. Square loss has smaller variance in 15 out of 20 tasks, which indicates that training with the square loss is less sensitive to the randomness in the training process.
QQP
0.458
0.161
Attention
TIMIT (PER)
0.508
0,249
+CTC
TIMIT (CER)
0.361
0.873
VGG+
WST (WER)
0,184
BLSTMP
WSJ (CER)
0.077
0.118
VGG+
Libri (WER)
0,126
0.257
BLSTM
Libri (CER)
0.148
0.316
TCNN
MNIST
0.161
0,173
W-ResNet
CIFAR-10
0.184
0,481
Neett (Top- 1)
0.032
0.045
ResNct-50
I-Net (Top-5)
0.126
0.045
4 OBSERVATIONS DURING TRAINING
1-Net (Top-1)
0.138
0.122
EfficientNet
1-Net (Top-5)
0.089
0.089
There are several interesting observations in terms of the optimization speed comparing training with the square loss and the cross-entropy loss. We
give the experimental observations for the cases when the class number is small, as for our NLP tasks, which are all 2-class classification tasks, and when the class number is relatively large, for Libripseech and ImageNet (both have 1000 classes).
NUP
Librispecch
Imagefict
Accunty Aosuette mode
With so are fram DEIT use an fum bedies ua ve LSTM-13U
02
LETU
Roshia wite erte ay (Tep-))
madia automebuuy bock
04
0.
Spoche
Epache
Trainig siepe
100
NLP tasks
(b) ASR tasks
(c) Vision tasks
Figure 2: Training curves
We compare the convergence speed in terms of accuracy, and find that for 2-class NLP classification tasks, the training curves of training with the square loss and the cross-entropy are quite similar. Fig- ure 2 (a) gives the accuracy of three model architectures trained with the square loss and the cross- entropy along differrntt epochs for QNLI dataset. For all three models, BERT, LSTM+Altention, and LSTM+CNN, using the square loss converges as fast as cross-entropy loss, and achieves bet- ter/comparable accuracy to training with the cross-entropy.
Convergence speed when class number is large When the class number becomes large, as on speech dataset Librispeech and vision dataset ImageNet, training with the square loss may need more epochs to converge. Figure (b) gives the classification accuracy of acoustic model along different epochs, and Figure 2 (c) gives the accuracy (Top-l) and Top-5 accuracy along different training steps of ResNet on ImageNet. Training with the square loss converges slower but reaches similar/better accuracy.
IMPLEMENTATION
We summarize the key points of implementation in this section. Full details and the exact param- eters are given in Appendix B Two important pieces of the implementation are (1) no softmax for training with the square loss and (2) loss rescaling for datasets with large number of classes.
7

---

Published as a conference paper at ICLR 2021
Table 9; Rescaling parameters
No softmax. The widely accepted pipeline for mod- neural classification tasks trained with the cross- entropy loss contains the last softmax layer before calculating the loss. When training with the square loss that layer needs to be removed as appears to impede optimization.
Dataset
#classes
k
M
MRPC
2
SST-2
2
1
1
QNLI
2
1
1
QQP
2
1
1
TIMIT (CER)
27
1
1
Loss rescaling mechanism. For datasets with small number of classes, do not use any additional mechanisms. For datasets with a large number of employ
TIMIT (WER)
42
1
15
WSJ
52
1
15
Librispeech
1000
15
30
output classes (2 42 our experiments) loss rescaling which helps to accelerate training. Let (x, y) denote single labeled point, where æ E Rd is the feature vector, and IR. Here C' is the number of output labels and y = [0,
MNIST
10
1
1
CIFAR-I0
10
1
1
ImageNet
1000
15
30
0, .,....,,,
,0 is the corresponding one-hot encoding vector
label We denote our model by f F. RC.
The standard square loss for the one-hot encoded label vector can be written (at single point) as
1 - (fc(a)) 1))+
Σ
fi(x)?
(1)
1,igle
For a large number of classes, we use the rescaled square loss defined by two parameters, & and M, follows:
1 (x k* (f((z)) ( L. -
My²
E f((z)) L=
The parameter rescales the loss value at the label, while AM rescales the one-hot encoding (the one-hot vector is multiplied by M). Note that when k = M = 1, the rescaled square loss is same the standard square loss in Eq. The values of l and M for all experiments are given in Table 9 As in (Demirkaya et al. 2020). the parameter is used to increase the emphasis on the correct class in multiclass classification, and this paper proves how adding can simplify the optimization landscape. We find that for very large class numbers additional parameter M further improves performance.
SUMMARY AND DISCUSSION
In this work provided empirical comparison of training with the cross-entropy and square loss functions for classification tasks range of datasets and architectures. We observe that the square loss outperforms cross-entropy across the majority of datasets and architectures, sometimes by a significant margin. No additional parameter modification except for adjusting the learning rate was necessary for most datasets. For datasets with large number classes (42 or more) we used additional loss rescaling to accelerate training. We note that all models used in our experiments were originally designed and tuned for training with the cross-entropy loss. We conjecture that the neural architectures were selected and tuned for the square loss, performance would be further improved and no extra loss rescaling parameters would be necessary. Another important observation is that the final softmax layer, commonly used with cross-entropy, needs to be removed during training with the square loss.
While we could only explore small sample of modern models and learning tasks, we believe that the scope of our experiments - ten different neural architectures and ten different datasets across three major application domains is broad enough to be indicative of the wide spectrum of neural models and datasets. Our empirical results suggest amending best practices of deep learning to include training with square loss for classification problems equal footing with cross-entropy even as a preferred option. They also suggest that new theoretical analyses and intuitions need to be developed understand the important question of training loss function selection.
8

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
The authors acknowledge support from NSF (IIS-1815697) and NIH (R01EB022899) and Google Faculty Research Award. We thank Nvidia for the donation of GPUs and Google for the free access to the cloud TPUs provided by the TFRC program. LH thanks Wuwei Lan for helpful discussions on NLP experiments and Peidong Wang for discussions on ASR experiments. MB thanks his co-authors on (Muthukumar et al., 2020), D. Hsu, V. Multukumar, A. Narang, A. Sahai and V. Subramanian, for insightful discussions related loss functions and the Simons Institute for the Theory of Computing, where the initial discussions took place. We thank Ryan Rifkin for valuable feedback.
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271, 2018.
Glenn W Brier. Verification forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1668, 2017.
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp- 1-5. IEEE, 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1059, 2016.
John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93, 1993.
Pavel Golik, Patrick Doetsch, and Hermann Ney. Cross-entropy vs. squared error training: a theo- retical and experimental comparison. In Interspeech, volume 13, pp. 1756-1760, 2013.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
/www. addeppeea inw.onnnnnoocoom org
Frank E Harrell Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer, 2015.
Hua He and Jimmy Lin. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 937-948, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. First Quora Dataset Release: Question Pairs. In https://data.quorra www.i.uuooooriaaa.. www.//ooot---oeeeeeeeee wor.roooo///otorooaaa--oo
Questuun-Paainii 2017.
Katarzyna Janocha and Wojciech Marian Czarnecki. On loss functions for deep neural networks in classification. arXiv preprint 17Xiv:100. 2017.

---

Published as a conference paper at ICLR 2021
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent nonseparable data. In Conference on Learning Theory, pp. 1772-1798, 2019.
Dan Jurafsky. Speech & language processing. Pearson Education India, 2000.
Suyoun Kim, Takaaki Hori, and Shinji Watanabe. Joint etc-attention based end-to-end speech recog- nition using multi-task learning. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4835-4839, IEEE, 2017.
Douglas M Kline and Victor L Berardi. Revisiting squared-error and cross-entropy functions for training neural network classifiers. Neural Computing & Applications, 14(4):310-315 2005.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Wuwei Lan and Wei Xu. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, 3890-3902, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11))2278-23244 1998.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Pro- ceedings of the 18th ACM international conference on Multimedia, pp. 1485-1488, 2010.
Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Triggered attention for end-to-end speech recog- nition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5666-5670. IEEE, 2019.
Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai, Classification vs regression in overparameterized regimes: Does the loss function matter?, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 5206-5210. IEEE, 2015.
Douglas B Paul and Janet M Baker. The design for the wall street journal-based based esr corpus. In Proceedings of the workshop on Speech and Natural Language, pp. 357-362. Association for Computational Linguistics, 1992.
Qichao Que and Mikhail Belkin. Back to the future: Radial basis function networks revisited. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statisties, volume 51 of Proceedings of Machine Learning Research, pp. 1375-1383, Cadiz, Spain, 2016. PMLR.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: arXiv:1606.00 1606.55250, 2016.
Ryan Michael Rifkin. Everything old is new again: a fresh look at historical approuches in machine learning. PhD thesis, MaSSachuSettS InStitute of Technology, 2002.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-2555 2015.
Arash Sangari and William Sethares. Convergence analysis of two loss functions in soft-max re- gression. IEEE Transactions on Signal Processing, 1280 2015.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
10

---

Published as a conference paper at ICLR 2021
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822- 2878, 2018.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprini arXiv: 1946, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.00 074411, 2018.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nel- son Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduch- intala, and Tsubasa Ochiai. Espnet: End-to-end speech processing toolkit. In Interspeech, pp- 2207-2211, 2018. doi: 10/21437/meerppechh2113.
URL http: doi org/1C.
21437/ (tt333/Intersppeeee 2018 1456
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' e'mi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface' trans- formers: offee-of-the-are art natural language processing. ArXiv, abs/1910.03771, 2019.
Chen Xing, Sercan Arik, Zizhao Zhang, and Tomas Pfister. Distance-based learning from emrors for confidence calibration. arXiv preprint OrXiv:/912.0/733 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1rXiv:1605.07146, 2016.
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. Dive into Deep Learning. 2020. https://d21.ai
APPENDICES
A DATASETS AND TASKS
Below provide summary of datasets used in the experiments.
NLP tasks
MRPC (Microsoft Research Paraphrase Corpus) (Dolan Brockett 2005) is a corpus of sentence pairs extracted from online news sources. Human annotation indicates whether the sentences in the pair are semantically equivalent. We report accuracy and F1 score.
SST-2 (The Stanford Sentiment Treebank) (Socher al. 2013) task determine the sentiment of given sentence. This corpus contains sentences from movie reviews and their sentiment given by human annotations. We use only sentence-level labels, and predict positive or negative sentiment.
QNLI is converted dataset from the Stanford Question Answering Dataset (Rajpurkar et al. 2016) which consists of crestion-paragrappe pairs. As in (Wang al. 2018), ) this task is to predict whether the context sentence selected from the paragraph contains the answer to the question.
QQP (Quora Question Pairs dataset) (Iyer al. 2017) contains question pairs from the question-answering website Quora. Similar to MRPC, this task is to determine whether a pair of questions are semantically equivalent. We report accuracy and Fl score.
ASR tasks
TIMIT (Garofolo et al. 1993) consists of speech from American English speakers, along with the corresponding phonemical and lexical transcription. It is widely used for acoustic- phonetic classification and ASR tasks. Its training set, validation set and test set are 3.2 hours, 0.15 hours, 0.15 hours long, respectively.

---

Published as a conference paper at ICLR 2021
WSJ (Wall Street Journal corpus) (Paul & Baker 1992) contains read articles from the Wall Street Journal newspaper. Its training, validation and test set are 80 hours, 1.1 hours and 0.7 hours long, respectively.
Librispeech (Panayotov et al. 2015) is large-scale (1000 hours in total) corpus of 16 kHz English speech derived from audiobooks. We choose the subset train-clean-100 (100 hours) our training data, dev-clean (2.8 hours) our validation set and test-clean (2.8 hours) as our test set.
Vision tasks
MNIST LeCun et al., 1998) contains 60, 000 training images and 10, 000 testing 28 x 28 pixel images hand written digits. It is a 10-class image classification task.
CIFAR-10 (Krizhevsky Hinton, 2009) consists 50, 000 32 32 pixel training images and 10, 000 32 x 32 pixel test images in 10 different classes. It is balanced dataset with 6, 000 images of each class.
ImageNet (Russakovsky et al. 2015) is an image dataset with 1000 classes, and about 1.28 million images training set. The sizes of its validation and test set are 50, 000 and 10, 000, respectively. All images we use are 224 224 pixels.
B HYPER-PARAMETER SETTINGS
We give the implementation toolkits and specific results, and list the epochs needed for training with the square loss and the cross-entropy (CE) loss. The data processing following the standard methods. For NLP tasks, is the same as in (Wang al. 2018), and for ASR tasks, it is the same in (Watanabe et al. 2018). For vision tasks, following the default ones given in the implementation of the corresponding papers.
settings to help reproduce our
B. HYPER-PARAMAE PARAMETERS FOR NLP TASKS
The implementation BERT based on the PyTorch toolkit (Wolf et al. 2019). The specific script run is http://gttutubte/ htt.:/ttww....c.....tt..e.aa http:///////ggggn//////aaaaaaaae examppes//extttcllsssffiiiii
http://uunnguuuueee glue and we use the bert-base-casee model implemented based on the toolkit released by (Lan & Xu 2018). The specific hyper-parameters used in the experiments are in Table [T0] As there are many hyper-parameters, we only list the key ones, and all other parameters are the default in the scripts.
for fine-tuning. LSTM+Attention and LSTM+CNN
Table 10: Hyper-parameters for NLP tasks
Model
Task
Batch
max_seq
Learning rate w/
Epochs training w/
size
length
square loss
CE
square loss
CE
MRPC
32
128
5e-5
2e-5
5
3
SST-2
32
128
2c-5
2e-5
3
BERT
QNLI
32
128
2c-5
2e-5
3
3
QQP
32
128
2e-5
2e-5
MRPC
64
80
2e-4
le-4
25
20
LSTM+Attention
QNLI
32
sent. kentlen"
le-4
Te-4
20
20
QQP
64
120
le-4
le-4
30
30
MRPC
64
80
2c-4
le-4
20
20
LSTM+CNN
QNLI
32
sent len"
8e-5
lc-4
20
20
QQP
32
120
1e-3
Te-3
20
20
The max sequence length equals the max sentence length of the training set.
B.2 HYPER-PARAMETER FOR ASR TASKS
The implementation of ASR tasks is based on the ESPnet (Watanabe et al. 2018 toolkit, and the specific code use is the run.sh script under the base folder of each task, which is https:
12

---

Published as a conference paper at ICLR 2021
com/ http:///ww..ppppe/pp////t///aaaa
where "?" can be 'timit',
wsj', and 'librispeech'. The specific ryper-parameters are following the ones in the configuration file of each task, which is under the base folder. We list the files which give the hyper-parameter settings for acoustic model training in Table
Table 11: Hyper-parameters for ASR tasks
Epochs training w/
Model
Task
Hyper-parametten
square loss
CE
Attention+CTC
TIMIT
conf/train.
20
20
VGG+BLSTMP
WSJ
conf/unning/rrii__nng yan.
15
15
VGG+BLSTM
Librispeech
conf/unning/raiinrnnn, rnt.:/aiiinnnnmmmmmm
30
20
For WSJ, we use the language model given by ps: ie googlo le. com/
open?id=1Az- Sa2SuunnFa411aaame www.:eeaaaala--a-a EKi www.aaaa.a.....a..ac...kkkkkkkka jeJp We set mtlalpha=0.3,
batch-size=30.
We set elayers=4, as we use 100 hours training data.
B.3 HYPER-PARAMENT PARAMETERS FOR VISION TASKS
The implementation of these models are based on the open source toolkits. For TCNN and Effi cientNet, we the open source implementation given by (Bai al. 2018) and (Tan & Le 2019), respectively, For Wide ResNet, we are based on the open source PyTorch implementation ps //github. com wtapm/xtennaall. t (W-ResNett. For ResNet-50, our ex- periments are based on the Tensorflow toolkit httts:://qgtht:i leseemmattermodee httpooeeelooofccaa//eeeeaae
cww/tensor fttpoorrlowwp///// (ResNet) implemented on TPU. The hyper-
parameter settings for our vision experiments are in Table 12
Table 12: Hyper-parameters for vision tasks
Epochs training w/
Model
Task
Hyper-parameters
square loss
CE
TCNN
MNIST
the default (Bai et al. 2018
20
20
the default NetesNet,
Wide-ResNet
CIFAR-10
200
200
except wide-factor=20
the default ResNet, for square loss, learning rate=0.3
ResNet-50
ImageNet
168885*
112590*
the default EfficientNer-B0 of (Tan & Le 2019
EfficientNet
ImageNet
218949*
218949"
We are doing the permuted MNIST task as in Bai et al. (2018).
We give the training steps as the original implementations.
C
EXPERIMENTAL RESULTS ON VALIDATION AND TRAINING SETS
Table 13: NLP results validation set, accuracy
train with
train with cross-entropy (%)
square loss w/ same epochs CE (%)
Model
Task
square loss (%)
MRPC
85.3
85.0
85.3
BERT Devlin et al.
SST-2
91.2
91.5
91.2
2018
ONLI
90.8
90.7
90.8
QQP
90.8
90.7
90.6
MRPC
76.5
74.8
75.3
LStMtittnntion Chen et al. 2017)
QNLI
79.7
79.7
79.7
QQP
86.0
85.5
86.0
MRPC
76.0
73.3
76.0
LSTM+CNN
QNLI
76.8
76.8
76.8
He & Lin 2016)
QQP
84.0
85.3
84.0
13

---

Published as a conference paper at ICLR 2021
Table 14: NLP results on validation set, F1 scores
train with
train with cross-entropy (%)
square loss w/ same epochs CE (%)
Model
Task
square loss (%)
BERT
MRPC
89.5
89.6
89.5
Devlin et al.
2018
QQP
87.5
87.4
87.4
LStMMttttnttion
MRPC
83.7
83.3
83.55
Chen et al.
2017)
QQP
82.1
81.7
82.1
LSTM+CNI
MRPC
82.6
81.4
82.6
(He Lin 2016
QQP
77.4
80.2
77.4
We report the results for validation set of NLP tasks in Table 13 for accuracy and Table [4 for FI scores.
The validation set results of the ASR tasks are in Table 15
Table 15: ASR results on validation set, error rate
train with
train with cross-entropy (%)
square loss w/ same
Model
Task
square loss (%)
epochs as CE (%)
Allention+CTC
TIMIT (PER)
18.1
18.3
18.1
Kim et 2017
TIMIT (CER)
30.4
VGG+BLS TMP
WSJ (WER)
8.5
8.8
8.5
Moritz et al.
2019
WSJ (CER)
3.9
4.0
3.9
VGG+BLS IM
Librispeech (WER)
9.3
10.7
9.9
Moritz al. 2019
Librispeech (CER)
9.4
11.1
10.2
We report the training result for NLP tasks in Table I6 for accuracy and Fl score in Table I7 The training results for ASR tasks and vision tasks are in Table and and Table 19 respectively.
Table 16: NLP results on training and test set, accuracy
train with
train with
square loss w/ same epochs CE (%)
Model
Task
square loss (%)
cross-entropy (%)
Train
Test
Train
Test
Train
Test
MRPC
99.7
83.8
99.9
82.1
99.6
83.6
BERT
SST-2
98.6
94.0
99.2
93.9
98.6
93.9
Devlin et al. 2018
QNLI
98.0
90.6
97.5
90.6
98.0
90.6
QQP
96.2
88.9
98.0
88.9
96.2
88.8
MRPC
94.6
71.7
84.9
70.9
93.2
71.5
Ltteniitention Chen al. 2017)
QNLI
87.7
79.3
90.8
79.0
87.7
79.3
QQP
93.7
83.4
91.5
83.1
93.7
83.4
MRPC
98.3
73.2
92.5
69.4
98.3
72.5
LSTM+CNN
ONLI
92.8
76.0
90.7
76.0
92.8
76.0
He & Lin 2016)
QQP 91.3
84.3
95.7
84.4
91.3
84.3
Table 17: NLP results on training and test set, F1 scores
train with
train with
square loss w/ same
Model
Task
square loss (%)
cross-entropy (%)
epochs CE (%)
Train
Test
Train
Test
Train
Test
BERT
MRPC
99.8
88.1
99.9
86.7
99.7
88.0
Devlin et al.
2018
QQP
94.5
70.9
97.2
70.7
94.5
70.7
LSTM4 Attention
MRPC
96.1
80.9
89.5
80.6
94.7
80.7
Chen al.
2017)
QQP
91.9
62.6
89.2
62.3
91.9
62.6
LSTM +
CNN
MRPC
98.8
81.0
94.5
78.2
98.8
81.0
(He Lin
2016
QQP
88.0
60.3
94.2
60.5
88.0
60.3
14

---

Published as a conference paper at ICLR 2021
Table 18: ASR results training and test set, rate
train with
train with
square loss w/ same
Model
Task
square loss (%)
cross-entropy (%)
epochs as CE (%)
Train
Test
Train
Test
Train
Test
Attention+CTT
TIMIT (PER)
0.9
20.8
4.8
20.8
0.9
20.8
Kim et al. 2017]
TIMIT (CER)
4.5
32.5
11.6
33,4
4.5
32.5
VGG++BI TMMP
WSJ (WER)"
0.7
5.1
0.3
5.3
0.7
5.1
Moritz ct al.
2019
WSJ (CER)"
0.3
2.4
0.1
2.5
0.3
2.4
VGG+BLS TM
Librispeech (WER)*
0.8
9.8
0.4
10.6
0.8
10.3
Moritz ct al.
2019
Librispeech (CER)"
0.6
9.7
0.3
10.7
0.6
10.2
For WSJ and Librispeech, take 10% of the training set for the evaluation of the training rate.
Table 19: Vision results on training and test set, accuracy
train with
train with
square loss w/ same
Model
Task
square loss (%)
cross-entropy (%)
epochs CE (%)
Train
Test
Train
Test
Train
Test
TCNN (Bai et
MNIST (acc.)
98.3
97.7
98.3
97.7
W-Resnet ZagordyKo & Komodaaks 2016
CIFAR-10 (acc.)
100.0
95.9
100.0
96.3
100.0
95.9
Kesine-SUU
ImageNet (acc.)
77.7
76.2
80.5
76.1
77.7
76.0
He et al. 2016
ImageNet 5Top-5 acc.)
93.2
93.0
93.4
93.0
93.2
92.9
Eineerent Ner
Image Net (acc.)
75.1
74.6
81.4
77.0
75.1
74.6
2019
ImageNet (Top-5 acc.)
92.7
92.7
D OUR RESULTS COMPARED WITH THE ORIGINAL WORK
We list our results for the models trained with the cross-entropy (CE) loss and compare them the results reported in the literature or the toolkits in Table As we observe, our results are comparable the original reported results.
Table 20: Training with the cross-entropy loss, our results and the reported ones
Model
Task
Our CE result
CF result the literature
MRPC (ace./F1)
85 85.0/89.6
85. 85.29/89.47 (Wolf et al. 2019
SST-2 (acc.)
91.5
91.97 (Woll et al. 2019
BERT
QNLI (acc.)
90.7
87 $7.46 Wolf ct al.,
2019
QQP (acc./FI)
90.7/87.4
88.40/84.31 Woll el al. N/A
2019
LSTM+Attention
LSTM+CNN
N/A
TIMIT (PER)
20.7
20.5
Watanabe et al.
2018
Attention+CTC
TIMIT (CER)
32.7
33.7
Watanabe et al.
2018
WSJ (WER)
5.4
5.3
Watanabe et al.
2018
VGG+BLSTMP
WSJ (CER)
2.6
2.4
Watanahe et al.
2018
Librispeech (WHR)
10.8
N/A
VGG+BL
Librispeech (CER)
11.0
N/A
TCNN
MNIST (acc.)
98.0
97.2 (Bai et al. 2018
Wride-ResNel
CIFAR-IO (acc.)
96.5
96.11
[Zagoruyko & Komodakis 2016
ResNet-50
ImageNet (ace:/Top-S acc.)
76.1/9
76.0/93.0 (Tn & 2019
EllicientNet ImageNet (acc. /TopS acc.)
77.2/93.4
77.3/93.5 lan & Le 2019)
'The implementation in (Wolf et al. [2019] is using bert- sart-bas-uuncsed model, we are using bert-base-can cast-bas--csed, which will result in little difference. Also, they didn't give test set results, here for BERT, we give the results of validation set.
The models marked with 'N/A in Table 20 do not have comparable results reported in the literature. Specifically, LSTM*Attentin and LSTM+CNN models for NLP tasks are implemented based on the toolkit released by (Lan & Xu 2018), where they did not show results MRPC and QNLI. The QQP results are not comparable with ours as they were using a different test set, while we are using the standard test set same in (Wang et 2018). The VGG+BLSTM model for Librispeech dataset is based on ESPnet toolkit (Watanabe et al.] 2018). Due to computational resources limita- tions, only use train-clean- 100 (100 hours) as training data and 1000 unigram based dictionary for acoustic model training, while they use 1000 hours of training data with at least 2000 unigram dictionary.
15

---

Published as a conference paper at ICLR 2021
E REGULARIZATION TERMS
We give the regularization term of each task in Table O means we didn't add regularization term. For WSJ, check the details line 306 of httpttss//gtttttttt wtt.cccc/cspppe ccmm///spnettassst//boooo master http://eeeeetttttrrrrorooaaaaaee
hetpn//dccosss.see py
Table 21: Regularization term for each task
Model
Task
dropout
batch norm
Regularization Term
BERT
MIRPC/SST-2/ONLIIOON
0.1
N
Lttentotn
MRPC/ONLI/QOO
0.5
N
LSTM+CNN
MRPC/QNLI/QOP
0.0
N
Allention+CTC
TIMIT
0.0
N
VGG+BLSTMP
WSJ
0.0
N
label smoothing based
VGG+BLSTM
Librispeech
0.0
N
TCN Wide-ResNet
MNIST
0.05
N
CIFAR-10
0.0
N
0
ResNet-50
ImageNet
0.0
Y
10
EfficientNet
ImageNet
0..
Y
10
w
For dropout, 0.0 means have not apply dropout.
F VARIANCE OF ACCURACY AMONG DIFFERENT RANDOM SEEDS
Figure gives the error bar of 5 runs corresponding to 5 different random seeds, along with the results each inidividual run. In the left of each subfigure the result of training with the square loss, while in the right is result of the cross-entropy. As can be seen in Figure 3 using the square loss better accuray/error rate and smaller variance NLP and ASR tasks, which indicates that training with the square loss for those classification tasks is statistically better.
16

---

Published as a conference paper at ICLR 2021
Accuracy among
Accuracy among
loss
squate
73
CE
square
CE
lsa
64
84
I
83-
MPPC
SST-2
Altention: MAPC
Attention: QNU
Altention: QQP
lows
CE
square
loss
89.1
square
squara
CE
bass
89.3
88.9
90.5
88.8
70
B4
QNU
QQP
CNN: MRPC
CNN: QNU
CNN: QOP
NLP: BERT
(b) NLP: LStMtAttentin & LSTM+CNN
Error rate among results of random seeds
Accuracy among results random seeds
CE
Square
21.5
loss
kiss
97
SE
5.50
87.8
33
20.5
20.0
5.00
97.6
76.1
TIMIT: CER
TIMIT: PER
WSJ: WER
TCN
Wid-ResNet less
ResNet: Top-1
1805
loss
11.0
loss
11
squore
91.4
3?
CE
square loss
93.5
+
10.5
76
30.0
91.0
+
WSJ: CER
Lib: WER
Lib: CER
ResNet: Top-5
Efii: Top-1
Effi: Top-5
(c) ASR
Vision
Figure 3: Accuracy/error rate variance of results among random seeds
17


Reviews:
[{"review_id": "iEfwhZDxnMi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper compares the more popular cross entropy loss function to the squared loss function for classification tasks. The authors look at NLP, ASR and and CV tasks, keeping the architecture the same (as much as possible) and varying the loss functions. The authors demonstrate that although cross entropy is more commonly used in state-of-the-art architectures and is standard across tutorial code, square loss functions are often times advantageous. \n\nStrengths: \nThis paper highlights an ongoing issue in deep learning -- we often take for granted what 'best practices' are and do not sufficiently investigate the best loss function, etc. when starting a new problem. \n\nThe study covers a reasonable amount of types of problems and architectures and conducts 5 random initializations which is rare in many ML papers. \n\nWeaknesses:\nWhile this work is highly important and the study was well-done, the novelty is a little low. Many papers have done variations of this same type of work to answer this question. \n\nWhile I understand there are only so many experiments that a group can do, there are some limitations to this study -- including not performing hyper-parameterization for architectures comparing the two loss functions which may reveal some important distinctions and use-cases. \n\nA minor comment is that the average accuracies are reported but not standard deviation to get a sense of the variation across performance for both loss functions. Statistical significance calculations would also be helpful to interpret results. \n\nQuestion for authors:\nWhat other areas do you find people are using the wrong hyper-parameter choice? This could be an interesting discussion write-up for researchers to re-consider how they select their training paradigm. \n\nI recommend accepting this work. The study was well-done for answering this question and more thorough than related work. My reason for not giving a higher score is that the work is not highly novel. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UtutCgLBgXe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nI think this a very good contribution to ICLR given the topic and the quality of the submission (originality, contribution to the stare of the art, experimental evidence, etc) \n\n Some of the strong points of the submission are summarized as follows:\n\n1.\tThe paper tackles a very interesting subject, questioning the conventional wisdom the CE is superior to MSE loss in a wide range of machine learning problems. \n2.\tVery good introduction and motivations sections. The hypothesis, as well as the main motivations are discussed succinctly but in a very logical manner including historical aspects leading to the current state of affairs (in terms of the manner in which models are trained) that might be helpful for interested readers not sufficiently familiar with the aspects discussed in the article.\n3.\tThe state of the art (despite the previous comment) contextualizes the subject matter in a succinct but comprehensive manner. \n4.\tI have read people making similar claims in other forums and articles, but the authors here provide a very thorough and careful experimental design, which helps to validate their hypothesis. However, it would be interesting to see how these experiments generalize to problems (in particular in computer vision) where datasets are noisier or where the image quality/resolution are lower.\n5.\tThe experimental design is good, showing a careful analysis to validate the proposal and several ablation studies to confirm that the hypothesis holds for various machine learning domains, as well as several datasets.\n6.\tThe foundations for the method are presented in great detail in a formalized manner and provides sufficient to assess the validity of the proposed experimental design.\n\nHowever, there are certain things that in my opinion could be improved:\n\n1.\tFuture work could be further elaborated and discussion in specific domains (medical imaging, for instance) could be further discussed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FM-XMSDc2rT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper questions the omnipresence of cross-entropy loss for classification tasks while showing that square loss also yields competitive results. The experimental section spans a wide variety of tasks and architectures covering NLP, ASR, and Vision. The authors find that the model performance is more stable w.r.t. the random initialization of parameters when trained using the square loss. For NLP and ASR datasets, the authors find that the square loss yields slightly better results. The authors also report that the square loss usually takes more time to converge and requires rescaling with a larger number of classes. \nThe authors do not dwell on the fundamental reasons behind the observations made in this paper. However, I believe that these observations are indeed useful to advance further research for better loss functions. \n\nStrong Points:\n1. Diverse experiments and insightful results\n\nWeak Points:\n1. The paper does not make an effort to provide well-grounded explanations for the experimental observations. \n2. It is unclear how the CTC-based architectures for ASR were modified for square-loss. An explanation in the paper would be helpful. \n\nQuestions:\n 1. How were the CTC-based architectures for ASR were modified for square-loss. ?\n 2. What is the opinion of authors about the trade-off between the hyperparameter associated with the loss-scaling Vs. the simplicity of the cross-entropy loss for a larger number of classes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "EM6koFBcsuU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper compares cross-entropy and squared losses on a wide range of tasks. The focus primarily is on thorough empirical evaluation of these two losses on NLP, Vision and Speech tasks. The paper shows that squared loss is better or competitive with cross-entropy loss in most cases. Most of the experiments and comparisons seem to be well done; parameters, setups etc. are well explained. \n\nI believe a few additional tasks and settings would have helped put a better picture of the comparison. \nFor speech, the application of the two losses in tasks beyond ASR might give more insights. Similar for vision, classification tasks other than image classification (on MNIST/CIFAR and Imagenet) might be useful. \n\nAdditional learning settings might also be useful. For example, there is considerable amount of work (e.g R1 and R2 below) on noisy label learning, where the loss function (often cross entropy losses) is modified for noisy label condition. What do we expect for these two losses  in this noisy label setting ? Essentially what can we expect in learning settings beyond the supervised training in vanilla form. \n\nFor square loss, scaling is done for a large number of classes. The motivation behind it is not very clear. Also, it is perhaps worth showing results for squared loss with softmax outputs. Especially for the large number of classes. Can softmax be advantageous in this case for performance, even though computationally it will be worse than the scaling mechanism applied. \n\nR1. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels\nR2. Normalized Loss Functions for Deep Learning with Noisy Labels\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper questions the use of cross-entropy loss for classification tasks and shows that using squared error loss can work just as well for deep neural networks. The authors conduct extensive experiments across ASR, NLP, and CV tasks. Comparing cross-entropy to squared error loss is certainly not novel, but the conclusions of the paper, backed by a lot of experimental evidence, are certainly thought-provoking. 

I would have liked to see a bit more analysis into the results of the paper, and perhaps a bit more theoretical justification. That said, the paper will be of interest to the community, given the ubiquity of classification tasks.

Decision: Accept (Poster)