PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions
Authors: Zhaoqi Leng
Mingxing Tan
Chenxi Liu
Ekin Dogus Cubuk
Jay Shi
Shuyang Cheng
Dragomir Anguelov
Venue: ICLR 2022 Poster (2022)
Keywords: classification
computer vision
loss

Abstract: Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classification problems. Generally speaking, however, a good loss function can take on much more flexible forms, and should be tailored for different tasks and datasets. Motivated by how functions can be approximated via Taylor expansion, we propose a simple framework, named PolyLoss, to view and design loss functions as a linear combination of polynomial functions. Our PolyLoss allows the importance of different polynomial bases to be easily adjusted depending on the targeting tasks and datasets, while naturally subsuming the aforementioned cross-entropy loss and focal loss as special cases. Extensive experimental results show that the optimal choice within the PolyLoss is indeed dependent on the task and dataset. Simply by introducing one extra hyperparameter and adding one line of code, our Poly-1 formulation outperforms the cross-entropy loss and focal loss on 2D image classification, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin.

Published as a conference paper at ICLR 2022
POLYLOSS: A POLYNOMIAL EXPANSION PERSPEC- TIVE OF CLASSIFICATION LOss FUNCTIONS
Zhaoqi Leng' Mingxing Tan', Chenxi Liu', Ekin Dogus Cubuk?, Xiaojie Shi?,
Shuyang Cheng's Dragomir Anguelov!
WWamo LLC "Google LLC
(lengzhaogi, tanmingxing, cxliu, shuyangcheng, dragomirJ@waymon (cubuk, xiaojies)@@ooogle.com
ABSTRACT
Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classification problems. Generally speaking, however, a good loss function can take on much more flexible forms, and should tailored for different tasks and datasets. Motivated by how functions can be approximated via Taylor expansion, we propose a simple framework, named PolyLoss, to view and design loss functions linear combination of polynomial functions. Our PolyLoss allows the importance of different polynomial bases to be easily ad- justed depending on the targeting tasks and datasets, while naturally subsuming the aforeementoned cross-entropy loss and focal loss as special cases. Extensive experimental results show that the optimal choice within the PolyLoss is indeed dependent on the task and dataset. Simply by introducing one extra hyperparam- eter and adding one line of code, our Poly-/ formulation outperforms the cross- entropy loss and focal loss on 2D image classification, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin.
Tack
ImageNer classification Crass-entropy
and sep. Cross- cross-entropy
Waymo Open Dataser 3D detection
Defonlt lass
Focal lass
Model
ENaV2.122K)
ENerV2-L(1K)
Mask R-CNN
PointPillars Car
PointPillass Pod
RSN Car
RSN Ped
Baseline
45.8
$6.8
47.2
42.3
63.3
68.9
78.4
79.4
Polyt oss
46.4 (14.6)
N7.2 (41.4)
(+2.5)
(42.1)
(+1..4
69.6 (41.7)
(40.5)
M0.2 (+0.8)
Table 1: PolyLoss outperforms cross-entropy and focal loss various models and tasks. Re-
sults are for the simplest Poly-1, which has only a single hyperparameter. On ImageNet (Deng et al.] 2009), our PolyLoss improves both pretraining and finetuning for the recent EfficientNetV2 (Tan & Le][2021p on COCO (Lin et al.) | 2014), PolyLoss improves both 2D detection and segmentation AR for Mask-RCNN (He al. 2017]; 7); on Waymo Open Dataset (WOD) (Sun et al. 2020)) PolyLoss improves 3D detection AP for the widely used PointPillars (Lang et al. 2019) and the very recent
Range Sparse Net (RSN) (Sun al. 202T). Details in Table
INTRODUCTION
Loss functions are important in training neural networks. In principle, a loss function could be any (differentiable) function that maps predictions and labels to a scalar. Therefore, designing a good loss function is generally challenging due to its large design space, and designing a universal loss function that works across different tasks and datasets even more challenging: for example, LI / 1.2 losses are commonly used for regression tasks, but they are rarely used for classification tasks; focal loss is often used to alleviate the overfitting issue of cross-entropy loss for imbalanced object detection datasets (Lin et al. 2017). but it is not shown to consistently help other tasks. Many recent works have also explored new loss functions via meta-lcarning, ensembling or compositing different
losses (Hajiabdd al. 2017 Xu et al. 2018 Gonzalez & Miikkulainen 2020b a Li et al. 2019).
In this paper, we propose PolyLoss: a novel framework for understanding and designing loss func- tions. Our key insight is to decompose commonly used classification loss functions, such cross- entropy loss and focal loss, into a series weighted polynomial bases. They are decomposed in the p,),, where 0₁- E RR+ is the polynomial coefficient and P, is the prediction
form of Σド= probability of the target class label. Each polynomial base (1 - P.). is weighted by a corresponding polynomial coefficient ay, which enables us to easily adjust the importance of different bases for different applications. When a = 1/j for all j. our PolyLoss becomes equivalent to the commonly used cross-entropy loss, but this coefficient assignment may not be optimal.

---

Published as a conference paper at ICLR 2022
Our study shows that, in order to achieve better results, it is necessary to adjust polynomial coef- ficients ay for different tasks and datasets. Since it is impossible to adjust an infinite number of aj, explore various strategies with a small degree of freedom. Perhaps surprisingly, we observe that simply adjusting the single polynomial coefficient for the leading polynomial, which we denote P 1, is sufficient to achieve significant improvements over the commonly used cross-entropy loss and focal loss. Overall, our contribution can be summarized as:
We propose a unified framework, named PolyLoss, rethink and redesign loss functions. This framework helps to explain cross-entropy loss and focal loss as two special cases of the PolyLoss family (by horizontally shifting polynomial coefficients), which was not recognized before. This new finding motivates us to investigate new loss functions that vertically adjust polynomial coefficients, shown in Figuure
Insights on common losses:
New loss formulation: We evaluate different ways of vertically manipulating polynomial coef- ficients to simplify the hyperparameters search space. We propose a simple and effective Poly-1 loss formulation which only introduces one hyperparameter and one line of code.
New findings: We identify that focal loss, though effective for many detection tasks, is suboptimal for the imbalanced ImageNet-21K, We find the leading polynomial contributes of the gradient during training, and its coefficient correlates to the prediction confidence Pt. In addition, we provide an intuitive explanation PolyLoss tailored to imbalanced datasets.
large portion
how to leverage this correlation design good
Extensive experiments: We evaluate our PolyLoss on different tasks, models, and datasets. Re- sults show PolyLoss consistently improves the performance on all fronts, summarized in Table I which includes the state-of-the-ar classifiers EfficientNetV2 and detectors RSN.
2 RELATED WORK
Cross-entropy loss is used in popular and current state-of-the- art models for perception tasks such as classification, detection and semantic segmentation (Tan & Le 2021 He et al., 2017 Zoph et al.
2020 Tao al. 2020). Various losses are proposed to improve cross-entropy loss (Lin al. 2017
Law & Deng 2018:(Cui et al. 2019; Zhao et al. 202T). Unlike prior works, the goal of this paper is to provide a unified framework for systematically designing a better classification loss function.
Loss for class imbalance Training detection models, especially single-stage detectors, is difficult due class imbalance. Common approaches such as hard example mining and reweighing are developed to address the class imbalance issue (Sung) 1996; Viola & Jones 2001: Felzenszwalb al. 2010 SShrvastaaa et al., [016 [iu et al. 2016 Bulo et al. (017). D. As one of these approaches, Tocal loss is designed to mitigate the class imbalance issue by focusing on the hard examples and used to train state-of-the-
2D and 3D detectors (Lin et al.) 2017 Tan et al.) 2020 Du et al.) 2020 Shi et al. 2020; Sun et al.) (021). In our work, we found that focal loss is suboptimal for the imbalanced ImageNet-21X Using the PolyLoss framework, we discover a better loss function, which performs the opposite role of focal loss. We further provide intuitive understanding of why it is important to design different loss functions tailored to different imbalanced datasets using the PolyLoss framework.
Robust loss to label noise to label noise (Ghosh al.) 2015 2017 Zhang & Sabuneu 2018 Wang et al., 2019 Oksuz al. 2020 Menon et al. 2019). A commonly used approach is to incorporate noise robust loss function such Mean Absolute Error (MAE) into cross-entropy loss. In particular, Taylor cross entropy loss is proposed to unify MAE and cross- cross-entropy loss by expanding the cross-entropy loss in (1 P.Y polynomial bases (Feng al. 2020). By truncating the higher-order polynomials, they show trun- cated cross-entropy loss function is closer to MAE, which is more robust to label noise on datasets with synthetic label noise. In contrast, our PolyLoss provides a more general framework to design loss functions for different datasets by manipulating polynomial coefficients, which includes drop- ping higher-order polynomials proposed in Feng et al.) (2020). Our experiments in (susecction 4. show the loss proposed in Feng et al. (2020) performs worse than cross-entropy loss on the clean ImageNet dataset.
Another direction of research is to design loss functions that are robust
Learned loss functions Several recent works demonstrate learning the loss function during train- ing via gradient descent or meta learning (Hajiabadi et al. 2017 Xu et al. 2018 Gonzalez & Mi- ikkulainen 2020a; Li et al. 2019 2020 Notably, TaylorGLO utilizes CMA-ES to optimize multi- variate Taylor parameterization of a loss function and learning rate schedule during training (Hansen
& Ostermeier 1996. Gonzalez. & Miikkulainen 2020b). Due to the search space scale with the order
2

---

Published as a conference paper at ICLR 2022
3-
Poly coefficients
PolyLoss allows verbical adjustment
Food (L-Pi
Pacd las 4 CI-PP
Focal loss only allows
Chosi enthas bis
horizonial
0.6
1.0
P, P.
PJ P, P/111 Polynomial terms
P.41 P.Y
Prediction
Figure 1: Unified view cross-entropy loss, focal loss, and PolyLoss. PolyLoss Σj=1 ag(1
P.P is more general framework, where Pe stands for prediction probability of the target class. Left: Polyloss is more flexible: it can be steeper (deep red) than cross-entropy loss (black) or flatter (light red) than focal loss (green). Right: Polynomial coefficients of different loss functions in the bases of (1 - P.) where j E Z+ Black dash lines are drawn to show the trend of polynomial coefficients. In the PolyL framework, focal loss can only shift the polynomial coefficients horizonially (green arrow), see [Equation 2] whereas the proposed PolyLoss framework more general, which also allows vertical adjustment (red arrows) of the polynomial coefficient for each polynomial term.
of polynomials, the paper demonstrates that using the third-order parameterization (8 parameters), the learned loss function schedule outperforms gross-entropy loss on 10-class classification prob- lems. Our paper (Figure 2a). on the other hand, shows for 1000-class classification tasks, hundreds of polynomials are needed. This results in prohibitively large search space. Our proposed Poly-1 formulation mitigates the challenge of the large search space and do not rely on advanced black-bor optimization algorithms. Instead, show simple grid search over one hyperparameter can lead to significant improvement on all tasks that we investigate.
3 POLYLOSS
PolyLoss provides framework for understanding and improving the commonly used cross-entropy loss and focal loss, visualized in Figure 1 It is inspired from the Taylor expansion of cross-entropy loss (Equation 1) and focal loss Equation in the bases of P.).:
LCE = log(P:) =
Σ
1/j(1 - P.) = (1- Pb) + 1/2(1 - P.)
(1)
j=1
LFL. = - log(P.) =
Σ 1/j(1-P))** = -1- P.) P.) 1/2(1 2.)
where P. is the model's prediction probability of the target ground-truth class.
Cross-entropy loss as PolyLoss Using the gradient descent method to optimize the cross-entropy loss requires taking the gradient with respect to Pt. In the PolyLoss framework, an interesting observation is that the coefficients 1/j exactly cancel the jth power of the polynomial bases, see Equation 1 Thus, the gradient of cross-entropy loss is simply the sum of polynomials (1 P.Y', shown in Eqquation 3
dLeB
E1- = (1- P.) dP.
30
j=1
The polynomial terms in the gradient expansion capture different sensitivity with respect to Pt. The leading gradient term is 1, which provides constant gradient regardless of the value of P.. On the contrary, when j > 1, the jth gradient term is strongly suppressed when P. gets closer 1.
Focal loss PolyLoss In the PolyLoss framework, [Equation 2) it is apparent that the focal loss simply shifts the power j by the power of a modulating factor 7. This is equivalent to horizontally shifting all the polynomial coefficients by ? shown in Figure 1] To understand the focal loss from a gradient prospective, we take the gradient of the focal loss (Equation 2) with respect to Pt:
dLa. dP.
- -2 +1 - PT (4)
For positive 7, the gradient of focal loss drops the constant leading gradient term, in the cross- entropy loss, see Eqqution 3) As discussed the previous paragraph, this constant gradient term causes the model to emphasize the majority class, since its gradient is simply the total number of
3

---

Published as a conference paper at ICLR 2022
Polynomial expansion in the hosis of P.)
Loss
Crossentropp loss
(1 P2) 1/201 fjP 1/N(1 R2) + 1/(N + 11(1 R2) +...
Let logit))
Drop poly. (Sec 4.1)
(1 Pi) 1/2/1 RJ++ +.. 1/8(11 P.) (drop remaining temmss
Lrwy Lip -, 1/5(1 By
Poly-N (Sec 4.2)
(e + 1)01 - (8) + (1w) 1/N) + 11[[N + +11 -10
Inea = Lax Ez 500 Mf
Poly-1 (Sec 4.3)
< + 1)(1 Rdy 1/2(1 RJ) + 1/2(1 RJ) + 1/CV + 1101 P.N+ 4...
Ioaas Lex cx(l P.)
Table 2: Comparing different losses in the PolyLoss framework. Dropping higher order poly- nomial, proposed in prior works, truncates all higher order (N + + oo) polynomial terms. We propose Poly-N loss, which perturbs the leading N polynomial coefficients. Poly-1 is the final loss formulation, which further simplifies Poly-N and only requires simple grid search over one hyper- parameter. The differences compared to cross-entropy loss are highlighted in red.
examples for each class. By shifting the power of all the polynomial terms by 7, the first term then becomes (1 Pa)", which is suppressed by the power of y avoid overfitting to the already confident (meaning Pt close to 1) majority class. More details are shown in section 12
Connection to regression and general form Representing the loss function in the PolyLoss framework provides an intuitive connection regression. For classification tasks where y is the effective probability of the ground-truth label, the polynomial bases (1 - P.Y can be ex- pressed as (y - P.Y. Thus both cross-entropy loss and focal loss can be interpreted as a weighted ensemble of distances between the prediction and label to the jth power. However, fundamental question in those losses: Are the coefficients in front of the regression terms optimal?
In general, PolyLoss is a monotone decreasing function" on [0, 1] which can be expressed as Σj=1 E (1(1 P.) and provides flexible framework to adjust each coefficien e be generalized to non-integer, j. but for simplicity we only focus on integer power (j E 2+) in this paper. In the next section, we investigate several strategies designing better loss functions in the PolyLoss framework via manipulating ajj
PolyLoss can
UNDERSTANDING THE EFFECT OF POLYNOMIAL COFFFICIENTS
In the previous section, we established the PolyLoss framework and showed that cross-entropy loss and focal loss simply correspond to different polynomial coefficients, where focal loss horizontally shifts the polynomial coefficients of cross-entropy loss.
In this section, we propose the final loss formulation Poly-1. We study in depth how vertically adjusting polynomial coefficients, shown in Figure 1, may affect training. Specifically, we explore three different strategies in assigning polynomial coefficients: dropping higher-order terms; adjust- ing multiple leading polynomial coefficients; and adjusting the first polynomial coefficient, summa- rized in Table 2 We find adjusting the first polynomial coefficient (Poly-1 formulation) leads to maximal gain while requiring minimal code change and hyperparameter tuning.
In these explorations, we experiment with 1000-class ImageNet (Deng et al. 2009) classification. We abbreviate it as ImageNet- K to differentiate it from the full version, which contains 21K classes. We use ResNet-50) (He et al. 2016) and its training hyperparameters without modification
4.1 Drop:: REVISITING DROPPING NIGHER-ORDEE POLYNOMIAL TERMS
Prior works (Feng et al. 2020 Gonzalez & Miikkulainen 2020b) have shown dropping the higher- order polynomials and tuning the leading polynomials can improve model robustness and perfor- mance. We adopt the same loss formulation Lioo Σド= 1/j(1 - P.Y', as in Feng et al. 2020 and compare their performance with the bascline cross- cross-entropy loss on ImageNet- IK. As shown in Figure 2a we need to sum up more than 600 polynomial terms to match the accuracy of cross- entropy loss. Notably, removing higher-order polynomials cannot simply be interpreted adjusting the learning rate. To verify this, FFiuure 2b compares the performance for different learning rates with various cutoffs: no matter we increase or decrease the learning rate from the original value of 0.1, the accuracy worsens. Additional hyperparameter tuning is shown in section 9
I 'e only consider the case all o, 0 in this paper for simplicity. There exist monotone decreasing functions on [0, 1] with some a, negative, for example sin(1 - P.) -, /(j + 1)!(1 - 2t)
"To ensure series converges, we require s/limsup, - V/ aa 2 for P, in (0, 1). For P. = we don't 't.
require point-wise convergence; in fact cross-cntropy and focal loss both go 4

---

Published as a conference paper at ICLR 2022
75
F-252
Accuracy
50-
Accuracy
2-2.001
Lomp 2 PP
25
2
100
200
300
400
500
600
Cutoff index
Cutoff
(a) Truncating the infinite of polynomials in cross-entropy loss N reduces accuracy.
(b) Adjusting the learning rate (default 0.1)
Loep does not improve the classification accuracy.
Figure 2: Training ResNet-50 on ImageNet-1K requires hundreds of polynomial terms re- produce the same accuracy as cross-entropy loss.
To understand why higher-order terms are important, we consider the residual sum after removing the first N polynomial terms from cross-entropy loss: RN LCF LDro Lj=N+1 1/j(1-P3
Theorem 1. For any small Ç > 0, 5 > 0 N > log1 (S 6), then for any p E (ō, 1]. we have [Rx(p)] < < and RRay (p) Ç. (Proof section 7|
Hence, taking a large N is necessary to ensure D.mp is uniformly close to LCE in the perspectives of loss and loss derivative on [6, 1]. For a fixed Ç. as approaches 0, N grows rapidly, Our experimental results align with the theorem. The higher-order (j > N + 1) polynomials play an important role during the early stages of training, where Pt is typically close to zero. For example, when Pt ~ 0.001, according to Equation 3] the coefficient of the 500th term's gradient is 0.999499 0.6, which is fairly large. Different from aforementioned prior works, our results show that we cannot easily reduce the number of polynomial coefficients Og by excluding the higher-order polynomials.
Dropping higher order polynomials equivalent to pushing all the higher order (j > N+1 1) polyno- mial coefficients aj vertically to zero in the PolyLoss framework. Since simply setting coefficients to zero is suboptimal for training ImageNet-IK, in the following sections, we investigate how to manipulate polynomial coefficient beyond setting them to zero in the PolyLoss framework. In par- ticular, we aim to propose simple and effective loss function that requires minimal tuning.
4.2 LPOLY-N: PERTURBING LEADING POLYNOMIAL COEFFICIENTS
In this paper, we propose an alternative way of designing new loss function in the PolyLoss frame- work, where we adjust the coefficients of each polynomial. In general, there are infinitely many polynomial coefficients a, need be tuned. Thus, is infeasible optimize the most general loss:
Poly a1 P.) a2 P.)2 + aa P.) + Σ a
(5)
j=1
The previous section (subsection 4.1) has shown that hundreds of polynomials are required in train- ing do well tasks such as ImageNet-IK classification. If we naively truncate the infinite sum in Equaaiion 5) to the first few hundreds terms, tuning coefficients for so many polynomials still results prohibitively large search space. In addition, collectively tuning many coefficients also does not outperform cross-entropy loss, details in section 10)
To tackle this challenge, propose perturb the leading polynomial coefficients in cross-entropy loss, while keeping the rest the same. We denote the proposed loss formulation as Poly-N, where N stands for the number of leading coefficients that will be tuned.
Lpoly.N = (61 + 1)(1 Pt) + ... + (EN 1/N)(1 -P))PP) N.) N + +1/(N + 1)(11 P.) N+1
perturbed by 5
same as Len.
= = Pt) +
(1- - P.Y j=1
Σ
(6)
Here, we replace the jth polynomial coefficient in cross- entropy loss 1/j with 1/j + Ej. where Ej E [-1/j, xs)
CE loss
N=1
N=2
N=3
N-dim, grid search Gready grid search
76.3
76.7
76.8
the perturbation term. This allows us to pinpoint the
76.3
76.7
76.7
76.7
first N polynomials without the need to worry about the
infinitely many higher-order (j > N + coefficients, as in Equation 5
Table 3: LPoly-N outperforms cross- entropy loss on ImageNet-IK.
5

---

Published as a conference paper at ICLR 2022
Gradient contriburtion
1.0
Accuracy
76.5
76.0
0.5
Loay = log(P/++ f1 6/(1-P)
75.5
Loe
First polynomial
Rest
75.0
0.0
Perturbation 61
0
20
Training step (k)
60
80
100
(a) PosyLoss family Lpaty-
(e
(b) Percentage of gradient from the first polyno-
Pt). where
(-1,0,
mial versus the rest (infinitely many) polynomials.
Figure 3: The first polynomial plays an important role for training ResNet-50 on ImageNet-
1K. (a) Increasing the coefficient of the first polynomial term (<1 > 0) consistently improves the ResNet50 prediction accuracy. Red dash line shows the accuracy when using cross-entropy loss. Mean and stdev of three runs are plotted. (b) The first polynomial (1 - P.) contributes more than half of the cross-entropy gradient at the last 65% of the training steps, which highlights the importance of tuning the first polynomial. The red dash line shows the crossover.
Table 3 shows L outperforms the baseline cross-entropy loss accuracy. We explore N- dimensional grid search and greedy grid search of <j in Lpayy-N up to N = 3 and find that simply adjusting the coefficient of the first polynomial (N = 1) leads to better classification accuracy. Per- forming 2D grid search (N = 2) can further boost the accuracy. However, the additional gain is small (+0.1) compared to adjusting only the first polynomial (+0.4).
4.3
LPOLY-I: : SIMPLE AND EFFECTIVE
As shown in the previous section, we find tuning the first polynomial term leads to the most sig- nificant gain. In this section, further simplify the Poly-N formulation and focus evaluating Poly-1, where only the first polynomial coefficient in cross-entropy loss is modified.
Lpoly-1 = (1 + (1 Pt) + 1/2(1 Pt) + +... log P:) G1 P()
We study the effect of different first term scaling on the accuracy and observe that increasing the first polynomial coefficient can systematically increase the ResNet-50 accuracy, as shown in Figure 3a This result suggests that the cross-entropy loss is suboptimal in terms of polynomial coefficient values, and increasing the first polynomial coefficient leads consistent improvement, which is comparable to other training techniques (section
Figure 3b) shows the leading polynomial contributes more than half of the cross-entropy gradient during training for the majority of the time, which highlights the significance of the first polynomial term (1 Pt) compared to the rest of the infinite many terms. Therefore, in the remaining of the paper, we adopt the form of Lpaly-1 and primarily focus on adjusting the leading polynomial coefficient. As is evident from Equation 7] it only modifies the original loss implementation by single line of code (adding e P.) term on top of cross-entropy loss).
Note that, all the training hyperparameters are optimized for cross-entropy loss. Even so, a simple grid search on the first polynomial coefficients in the Poly-1 formulation significantly increases the classification accuracy. We find optimizing other hyperparameters for Lpoly-1 leads to higher accuracy, and show more details in section 8
EXPERIMENTAL RESULTS
In this section, we compare our PolyLoss against the commonly used cross-entropy loss and focal loss various tasks, models, and datasets. For the following experiments, adopt the default training hyperparameters in the public repositories without any tuning. Nevertheless, Poly-1 formu- lation leads to consistent advantage over default loss functions at the cost of simple grid search.
5.1
LPOLY-1 IMPROVES 2D IMAGE CLASSIFICATION ON IMAGENET
Image classification is fundamental problem computer vision, and progress on image classifica- tion has led to progress many related computer vision tasks. In terms of the network architecture, in addition to the ResNet-50 already used in sectionn 4 we also experiment with the state-of-the- EfficientNetV2 (Tan & Le 2021). We use the ImageNet settings in (Tan & Le 2021) except for replacing the original cross cross-entropy loss with our PolyLoss LPoly-1 with different values of 61. In terms of the dataset. in addition to the ImageNet- 1K dataset already used in |section 4 we also con- sider ImageNet-21KK which has about 13M training images with 21,841 classes. We will study both the ImageNet-211 pretraining results and the ImageNet- I finetuning results.

---

Published as a conference paper at ICLR 2022
Pretraining EfficientNetV2-LL ImageNet-211 21K, then finetuning it ImageNet-IK IK can improve 2021p. Here, we follow the same pretraining and finetuning schedule as reported in Tan & Le 2021 without modification but replace the cross- eross-entropy loss
classification accuracy (Tan Le.
with P.aay-1 = log( Pt) 61 to search the optimal 61-
Pt). We reserve 25,000 images from the training set as minival
Pretraining on ImageNet-21K Figure highlights the importance of using tailored loss function when pretraining model on ImageNet- 21K dataset. A simple grid search over 6j e {0, 1, ......................... 7} other default hyperparameters leads to around 1% accuracy gain for all SOTA EfficientNetV2 models with different sizes. The accuracy im- provement of using a better loss function nearly matches the improvement of scaling up the model architecture (S to M and M to L).
Accuracy
45.7
X2 faster
45
in L without changing
44
log(P)
42.5
4.3
ENetvZ-M
ENatv2-L
8.88
FLOPs
Figure 4: PolyL.oss improves EfficientNetV2 family on the speed-accuracy Pareto curve. Validation accuracy of EfficientNetV2 models pretrained on ImageNet-21K are plotted. Poly- Loss outperforms cross- ensropy loss with about x2 speed-up.
Surprisingly, see Figure 5a increasing the weight of the leading polynomial coeffi- cient improves the accuracy of pretraining on ImageNet-21 (+0.6), whereas reducing it low- ers the accuracy (-0.9), Setting E = -1 truncates the leading polynomial term in the cross-entropy loss [Equation Tp. which is similar to having focal loss with = Lqquatio 2p. However, the opposite change, where 61 > 0, improves the accuracy on the imbalanced ImageNet-2IK.
We hypothesize the prediction of the imbalanced ImageNet-2IK is not confident enough (P: is small), and using positive E PolyLoss leads to more confident predictions. To validate our hypoth- esis, we plot P. function of training steps in Figure 5b) We observe that 6 directly controls the mean P. over all classes. Using positive 61 PolyLoss leads to more confident prediction (higher Pt). On the other hand, negative e PolyLoss lowers the confidence.
46.5
0.4
-hgp23+11
Accuracy
46.0
Mean Pr
0.2
45.5
Lavy -log(Pa) (111-P)
45.0
0.0
50
100
150
Perturbation
Training (k)
(a) Validation accuracy of EfficientNerV--L. ImageNee-21KK, PolyLoss with positive e outper- forms baseline cross-entropy loss (red dash line).
(b) Positive e1 (dark) increases the predic- tion confidence, while negative é -1 (light)
decreases the prediction confidence.
Figure 5: PolyLoss improves EfficientNerV2-.1 by increasing prediction confidence P..
Fine tuning ImageNet-1K After pretraining ImageNet-21K K, we take the EfficientNetV2-L.cc PifciienNeet22...ccccpooon and finetune it ImageNet-1K, using the same proce- dure as Tan & Le (2021) except for replacing the original cross-entropy loss with the Poly-1 formulation. PolyLoss improves the finetuning accuracy by 0.4%, advancing the ImageNet- 1K 1 accuracy from 86.8% to 87.2%.
EffiiemNNeiV2-L.
LCE
Ipaa
Improv.
checkpoint
ImageNet-21K
45.8
46.4
+0.6
ImageNet-IK
86.8
87.2
+0.4
Table 4: PolyLoss improves classifica- tion accuracy ImageNet validation set. We set Ej = 2 for both.
5.2 LPOLY-I IMPROVES 2D INSTANCE SEGMENTATION AND OBJECT DETECTION COCO
Instance segmentation and object detection require localizing objects in image in addition to recognizing them: the former in the form of arbitrary shapes and the latter in the form of bounding boxes. For both instance segmentation and object detection, we use the popular COCO LLi al. 2014] dataset, which contains 80 object classes. We choose Mask R-CNN (He et al. 2017) as the representative model for instance segmentation and object detection. These models optimize multiple losses, e.g. LMaaRRRNN Lets Lbox + Lmuas- For the following experiments, we only replace the Les with PolyLoss and leave other losses intact. Results are summarized in Table
Code http:////tttt.ccccccoo/////////t

---

Published as a conference paper at ICLR 2022
Loss
Box
Mask
AP
AR
AP
AR
Mask R-CNN LCE Mask R-CNN Envly-
- (Pg
35.0= 0.09
47.2+t
0.16
31.3 + 0.09
=
(
35,3 0.12
49.7: 0.07
31.6 4 0.11
44.4 0.07
Improvement
+0.3
+2.5
+0,3
+2.1
Table 5: PosyLoss improves detection results on COCO validation set. Bounding box and in- stance segmentation mask average-precision (AP) and average-recall (AR) are reported for Mask R-CNN model with ResNet-50 backbone. Mean and stdev of three runs are reported.
Reducing the leading polynomial coefficient improves Mask R-CNN AP and AR. In training Mask R-CNN, we use the training schedule optimized for cross-entropy loss and replace the cross-
entropy loss with Poly-1 = (o)( + e1 - P2)) for the classification loss Lots. where E1 E --1.0.00.3. 00.6, -0.4, -0.2, 0, 0.5, 1.0). We ensure the leading coefficient is positive, i.e. é1 2 -1. Our results in Figure show systematic improvements of box AP, box AR, mask AP, and mask
AR reduce the weight of the first polynomial by using negative 6 values. Note that Poly-1 1) not only improves AP but also significantly increases AR, shown in Table 5
(6 =
Box AP
Mask AP
1.0
35
0.9
34
1.0
0.5
0.0
0.5
1.0
50
1.0
0.5
0.0
0.5
1.0
Mean
Porturbation
Porturbation FL
Box AR
Mask AR
login)
RogiR) (1-P)
-8
log(8) -P3
82
46
50
100
0.5
-100
-0.5
0.0
Training step (k)
Perturbation
Perturbation &
(a) Bound box AP, AR and Mask AP, AR increase as é decreases, Negative E outperforms eross-entropy loss (red dash line).
(b) Negative e -1 (light) reduces the overconfident prediction Pt.
Figure 6: PolyLoss improves Mask R-CNN by lowering overconfident predictions. Mean and stdev of three runs are plotted.
Tailoring loss function to datasets and tasks is important. ImageNet-21 KK and COCO are both imbalanced but the optimal e for PolyLoss are opposite in sign, i.. e = 2 for ImageNet-21KK classi- fication and = -1 for Mask R-CNN detection. We plot the P. of the Mask R-CNN classification head and found the original prediction is overly confident (P; is close to 1) on the imbalanced COCO dataset, thus using negative lowers the prediction confidence, shown in Figure 6b| This effect is similar to label smoothing (Szegedy et al. 2016)) and confidence penalty (Pereyra et al.) 2017). but unlike those methods, as long as e > 11, PolyLoss lowers the gradients of overconfident predictions but will not encourage incorrect predictions or directly penalize prediction confidence.
5.3 LPOLY-I IMPROVES 3D OBJECT DETECTION ON WAYMO OPEN DATASET
Polynomial expansion in the basis of (1 (1-F) P.)
Loss
Focal
(1 B3)+1 1/2(1 13)+2 1/3(11 15)++ +...
Ln. (1 11)7 log(/1) Tibi In telll P.y+
Poly-1 (PointPillars)
(6] + 1]11 P, 1/2(1 p,y+* + 1/3(1 Poptt +..
Poly-1" (RSN)
(drop first) (1/2 + ra X1 P2)+ 4 1/301
P.y
Biiil Fa (1- RY- a0l R.)+t
Table 6: Losy vs. focal loss for 3D detection models. Differences are highlighted in red. We found the best Poly-1 1 for PointPillars is 6 -1, which is equivalent to dropping the first term. Therefore, for RSN, we drop the first term and tune the new leading polynomial (1 (1)
Detecting 3D objects from LiDAR point clouds is an important topic and can directly benefit au- tonomous driving applications. We conduct these experiments on the Waymo Open Dataset (Sun) al. 2020). Similar to 2D detectors. 3D detection models are commonly based on single-stage and two-stage architectures. Here, we evaluate our PolyLoss on two models: a popular single-stage PointPillars model (Lang et al. 2019); and state-of-the-ar two-stage Range Sparse Net (RSN) model (Sun et al. 2021). Both models rely on multi-task loss functions during training. Here, we focus on improving the classification focal loss by replacing it with PolyLoss. Similar to the 2D perception cases, we adopt the Poly-1 formulation to improve upon focal loss, shown in Table 6
PolyLoss improves single-stage PointPillars model. The PointPillars model converts the raw 3D point cloud to 2D top-down pseudo image, and then detect 3D bounding boxes from the 2D image in a similar way to RetinaNet (Lin ct al. 2017). Here, we replace the classification
8

---

Published as a conference paper at ICLR 2022
Loss
BEV
3D
APAPHLI
AP/APH
APIAPH
AP/APH 1.2
Vehicle (loU=0.7)
PointPillars LeL
-11 logt -(1 P.) (Pg - PQ
82.58815
73.9/72.9
63.3/62.7
55.2/54.7
PointPillars L
83.6/82.5
74.8/73.7
63.7/63.1
55.5/55.0
Improvement
+1.1/+1.0
+0.9//0.8
+0.4/+0.7
+0.3/+0,3
RSN LFL
-(1 p.3 Pp) -(1 - Pi) (og( R.)
91.3/90.8
82.6/82.2
78.4/78.1
69.5/69.1
RSN Leei-1"
0.1 R)*
91.5/90.9
82.7/82.1
78.9/78.4
69.9/69.5
Improvement
+0.2/+0.1
+0.1/0.1
+0.5/40.3
+0.4/+0.4
Pedestrian (loU=0.5)
PointPillars La
(1 R1 log(/t)
76.0/62.0
67.2/54.6
68.9/56.6
60.0/49.1
PointPillars L Pebys
13) (Pt
77.1/62.9
67.7/55,1
69.6/57.1
60,2/49.3
Improvement RSN Lm
+1.1/+0,9
+0,5//0,5
+0.7/+0.5
+0.2+0.2
-11 P3 log(P))
85.0/81.4
75.5/72.2
79.4/76.2
69.9/67.0
RSN --h--
P.) log(?)) P3) 0.201 A)"
85.4/81.8
75.8/2.55
80.2/77.0
70.6/67.7
Improvement
+0.4/+0.4
+0.3/+0.3
+0.8/+0.8
+0.7/+0.7
Table 7: PolyLoss improves detection results Waymo Open Dataset validation set. Two detection models: single-stage PointPillars (Lang et al. al.] (219) and two-stage SOTA RSN (Sun et al.] 2021] are evaluated. Bird's eye view (BEV) and 3D detection average precision (AP) and average precision with heading (APH) at Level 1 (LI) and Level 2 (L2) difficulties are reported. The IoU threshold is set to 0.7 for vehicle detection and 0.5 for pedestrian detection.
focal loss (7 = 2) with Poly.1 = -(1 - P log Pt + (1 (1
P:) and adopt the same train- ing schedule optimized for focal loss without any modification 6 Table 7 shows that L with -1 leads to significant improvement on all the metries for both vehicle and pedestrian models.
Poly coefficients
Advancing the state-of-the-art with RSN. RSN seg- ments foreground points from the 3D point cloud in the first stage, and then applies sparse convolution to pre- dict 3D bounding boxes from the selected foreground points. RSN uses the same focal loss as the PointPillars model, i.e., Lr. (1 (1- P3) log Pt. Since the optimal Poly-1 for PointPillars (e1 = -1) is equivalent to drop- ping the first polynomial, we adapt the same loss for- mulation for RSN and tune the new leading polynomial
1.0
Co Pilars
0.5
-It -nt-P
0.0
p.)
(1 (1-PJ4
(1 P.)
(1 P,)6
Polynomial terms
Poly coefficients
1.0
-1"
2 by defining IFL Puuly-1 = -(1 P₂) log(P,) P,)
0.5
+0201 -A
-0.411
(1 P2) + c2 P2) shown in Figure 7 We fol- low the same training schedule optimized for focal loss
0.0
(1 Pa)
(1 PM) (1 (1-Pa)
(1 Pp)s
described in (Sun et al. (202T) without adjustment. Our results, in TTbll 7] show that tuning the new leading polynomial improves all metrics (except vehicle detec- tion BEV APH L2) for the SOTA 3D detector.
Polynomial terms
Figure 7: Visualizing L and - in the PolyLoss framework.
CONCLUSION
In this paper, we propose the PolyLoss framework, which provides unified view on common loss functions for classification problems. We recognize that, under polynomial expansion, focal loss is a horizontal shift of the polynomial coefficients compared to the cross-entropy loss. This new insight motivates us to explore an alternative dimension. i.c. vertically modify the polynomial coefficients.
Our PolyLoss framework provides flexible ways of changing the loss function shape by adjusting the polynomial coefficients. In this framework, we propose a simple and effective Poly-/ formula- tion. By simply adjusting the coefficient of the leading polynomial coefficient with just one extra hyperparameter é1, we show our simple Poly-/ improves a variety of models across multiple tasks and datasets. We hope Poly-1 formulation's 's simplicity (one extra line of code) and effectiveness will lead to adoption in more applications of classification than the ones we have managed to explore.
More importantly, our work highlights the limitation of common loss functions, and simple modifi- cation could lead to improvements even on well established statc-of-the-art models. We hope these findings will encourage exploring and rethinking the loss function design beyond the commonly used cross-entrope and focal loss, as well as the simplest Poly-1 loss proposed in this work.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank James Philbin, Doug Eck, Tsung-Yi Lin and the rest of Waymo Research and Google Brain teams for valuable feedback.
REPRODUCIBILITY STATEMENT
Our experiments based public datasets and open source code repositories, shown in footnote 3-6, We do not tune any default training hyperparameters and only modify the loss functions, which are shown Table 2-7. The proposed final formulation Loay-1 requires one line of code change. Example code for L with softmax activation shown below.
def polyl_cress crr___cros__etrppyy entropy (logits, labels, epsilon)
epsilon >=-1.
pt, CE, and Polyl have shape [batch]. pt tf.reduce_sum (laabbels (labels CE Polyl cE epsilon (1 pt)
(logits), axis=-1)
cectemaaa_rosss_ee____ cross_sssscccceeeeeese httpa/orosssssssseeeeeeee wntropy_with (labels, logits)
return Polyl
Example code for LCE with label smoothing is shown below.
def polyl_cross_entree (logits, labels, epsilon, alpha - 0.1):
epsilen >=-1,
one.minus.pt, CE, Polyl have shape [batch].
num classes labels. getells.et_shapeee () ass_s_____ () [-1]
smooth. labels labels = labels (1-alpha) alpha/num.clasee cla.ssssscccssss
one one_minus_pt = tf. reduce_suu?
smooth.labeil. ott.::/w....ccgeeccccoooooooeeeo
(logitts)s axis=-1)
loss kera loweesssossssss..
from_logits label_smoct Label_emocthinn-alplaa reduction=' none')
CR CF_loss (F_loss(labels,, logits)
Polyl CE epsilon one minus pt return Polyl
Example code for Poly-1 with sigmoid activation is shown below.
def Dolyy_focaa__i llss (logits, labels, epsilon, gamma=2.0):
epsilon >=-1.
P, pt, FL, and Polyl have shape [batch, num of classes]
p tf. sigoid
pt labels * + (1 labels) * (1 - p)
FL focal_loss (pt, gamma)
Polyl FL epsilon tf.math. pow pt, gamma
return Polyl
Example code for Poly-1 with balance is shown below.
def polyl_focal_loss (logits, labels, epsilon, gamma-2.00 alpha=0.255
epsilon >=-1.
P, pt, FL, weight, and Polyl have shape [batch, num of classes].
p tf. t.f.math.sigmo......... sigoid
pt labels * + (1 labels) * (1 - p)
FL focal_loss (pt, gamma, alpha)
weight labels alpha + labels) (1 alpha)
Poly1 FL epsilon
smathhppwwd. pt, gamma 1) weight
return Polyl
10

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
7 PROOF OF THEOREM
Theorem 1. For any small Ç > 0, 6 N log1-6(5 ( -0), then for any p 10, 11, we have
[Rx( < and (p)( <.
Proof.
(1 - p)¹ (1 -1-P1 (1-p)
(1 8)
(1-
[Rx(p]] = Σ 1/j(1 - p)¹ & Σ
5
/=N+1
j=N+1
(P)( = Σ (1 p)² (1 p)
(1- 8)N
p 6
N
8 ADJUSTING OTHER TRAINING HYPERPARAMETERS LEADS TO HIGHER GAIN.
All the experiments shown in the main text are based hyperparameters optimized for the baseline loss function, which actually puts PolyLoss at disadvantage. Here we use weight decay rate for ResNet50 as an example. The default weight decay (le-4) is optimized for cross-entropy loss. Adjusting the decay rate may reduce the model performance of cross cross-entropy loss but leads to much higher gain for PolyLoss (+0.8%). which is better than the best accuracy (76.3%) trained using cross-entropy loss (+0.8%).
Weight decay
le-4t
2e-4
9e-5
Cross-entropy
76.3
76.3
76.1
PolyLoss
76.7
77.1
76.7
Improv. @ the same weight decay
+0.4
+0.8
+0.6
Improv. compared to the hest LCE (76.3%)
+0.4
+0.8
+0.4
Table ResNet50 performances on ImageNet-IK using different weight decays. 'The default weight decay value is le-4.
Here, we add additional ablation studies on COCO detection using RetinaNet. The optimal 7 and balance values for Focal loss (2.0, 0.25) (Lin al. 2017). Since all the hyperparameters optimized with respect to the optimal (y, a) values, we observe no improvement when tuning the leading polynomial term. We suspect the detection AP is 'local maximum' of hyperparameters. By adjusting (7, a) values, we show PolyLoss consistently outperforms the best Focal Loss AP (33.4), i.e, adjusting only y value (column 4) both 7 and values (column 5, 6).
Focal loss (7,a) a))
(2.0, 0.25)
(1.5, 0.25)
(2.5,
(1.5, 0.3)
(2.5, 0.15)
Focal loss
33.4
33.4
33.2
33.2
32.9
PolyLoss
33.4
33.6
33.7
33.8
33.8
Improv. @ same (7.a)
+0.2
+0.5
+0.6
+0.9
Improv. compared the best LFL (33.4)
+0.2
+0.3
+0.4
+0.4
Table 9: RetinaNet (ResNet50 backbone) performances on COCO using different Focal loss (1,a). 'hhe default (7, a) used in Focal loss is (2.0, 0.25).
LIROP WITH MORE HYPERPARAMETER TUNING
For Lrup (N 2), besides adjusting the learning rate, we further tune the coefficient (a) of the second polynomial, similar to prior work (Gonzalez & Müikkulainen 2020b), and weight decay.
D (1 P.) + a(1 - 2
Unlike Feng et al. 2020), where a = 0.5 after dropping all higher-order polynomial, we find the optimal 8, while the optimal learning rate is the same the default setting (0.1). This alone
11

---

Published as a conference paper at ICLR 2022
increases the accuracy to 70.9, which shows simply dropping polynomial terms is not enough and adjusting the polynomial coefficients is critical. Further tuning weight decay leads to less than 0. % model quality improvement.
Comparing to prior works (Gonzalez Mikkulainen 2020b Feng et al. 2020). Poly-1 more
effective and only contains one hyperparameter. Tuning weight decay of Poly- further increases the accuracy while having less hyperparameters compared to Drop' shown in Table 10
Cross-entropy
Poly-1
Poly-1 (weight decay)
D
Accuracy
76.3
76.7
77.1
70.9
Num. of parameters
1
3
Table 10: Poly-1 outperforms Drrop with hyperparameter tuning. Accuracy of ResNet50 on ImageNet-18 IK is reported.
10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COBFFICIENTS
Besides adjusting individual polynomial coefficients, in this section, explore collectively tuning multiple polynomial coefficients in the PolyLoss framwork. In particular, we change the coefficients in the original cross-entropy loss from 1/j (Equation 1) to exponential decay. Here, we define
2N
I Txx
Σ /=1
e 1--1- P.Y
where we cut off the infinite sum at twice the decay factor N. We performed 2D grid search on
N E (5. 20, 80, 320) and learning rate E f0.1,0.4.. 1.6, 6.4]. The best accuracy 72.3, where
N 80 and learning rate 1.6, shown in Table 11
Cross-entropy
Poly-1
Lesp
Accuracy
76.3
76.7
72.3
Num. of parameters
1
2
Table 11: Comparing Poly-1 with exponential decay coefficients. Accuracy of ResNet50 on ImageNet- 1K is reported.
Though Poly-1 is better than using Lexp- there are a lot more possibilities besides using exponential decay. We believe understanding how collectively tuning multiple coefficients affects the training is important topic.
11 COMPARING TO OTHER TRAINING TECHNIQUES
As shown in recent works (He al. 2019 Bello et al. 2021 Wightman al. 2021). though independent novel training techniques often lead to sub 1% improvement, combining them could lead to significant overall improvements. To put things into perspective, Poly-1 achieves similar improvements as other commonly used training techniques, such as label smoothing and dropout on FC, shown in Table 12
Cross-entropy
Poly-1
Label smoothing
Dropout on FC
Accuracy
76.3
76.7
76.7 1
76.4
Num. of parameters
Table 12: Comparing Poly-1 with common training techniques. ImageNet- 1K is reported.
Accuracy ResNet50 on
12

---

Published as a conference paper at ICLR 2022
12 REDISCOVERING FOCAL LOSS FROM POLYLOSS
Focal loss was first developed for single-stage detector RetinaNet address strong class imbalance presented in object detection (Lin et al., 2017). Here, we provide an additional ablation study on how to systemically discover focal loss in the PolyLoss framework and investigate how the leading terms affect training the presence of class imbalance.
0.32
Box AR
Box
0.45
0..30
AR
0.40
3 Drop first N polynomials
2
Rediscovering the concept of focal loss from cross- entropy loss. Here, we take step back and attempt to systematically rediscover the concept of focal loss via our PolyLoss framework. Focal loss is commonly used for training detection models. Coming up with such an insight to address the class imbalance issue in detection requires strong domain expertise. We start with the PolyLoss representation of cross- entropy loss and improve it from the PolyLoss gradient perspective.
Figure 8:
Dropping leading polyno- mial terms can improve RetinaNet.
We start with the cross-entropy loss and define PolyLoss by dropping the first N polynomials in cross-entropy loss. i.c. Drop--tontt Ej=N+1 1/j(1 P.) = LCE
1.000
0.995
-lug!!)
-lag(n) -11-P)
Σ/=1 1/j(1 P.). Dropping the first two polynomial terms (1 - P:) significantly improves both the detection AP and AR, see FFigure 8) Dropping the first two polyno- mials (N = 2) leads to the best RetinaNet performance, which is similar to setting 7 2 in focal loss, i.e. focal
0.990
-log/9)
-11 11- PPP
0.985
0
10
20
30
40
Training step (k)
1.000
background
0.4
0.985
object
loss ? = pushes all the polynomial coefficients to the
0.2
right by 2, shown in Figuure 1] right, which is similar to truncating the first two polynomial terms.
0.985
25
25
Training step
Training step (k)
Leading polynomials cause overfitting the major- ity class. In the PolyLoss framework, the leading polynomial of cross-entropy loss constant, shown Equation 3 For binary classification, the leading gradi- ent for each class is simply Nhackground - -oojecct where Naackgound and Nobject are the counts of background and object instances in the training mini-batch. When the class counts are extremely imbalanced, the majority class will dominate the gradient which will lead to sig- nificant bias towards optimizing the majority class.
Figure 9: Dropping leading polynomi- als reduces overfitting to the major- ity class. P. during RetinaNet training are plotted. Top: overall. Bottom left: background. Bottom right: foreground object. Dark blue curves represents Pt for cross-entropy loss. Blue curves rep- resents dropping the first polynomial in the cross-entropy loss. Light blue curves represents dropping both the first and second polynomials in the cross-entropy loss.
Dropping polynomials reduces the extremely confident prediction Pt, see Figure 9 To examine the composi- tion of the overall prediction confidence, we also plot the P/ for background only and P. for object only. Due to the extreme imbalance between the background and the object class, the overall P3 dominated by the background only Pt. So reducing the overall P decreases the background P,. On the other hand, reducing overfitting to the majority background class leads to more confident prediction P, on the object class.
13

---

Published as a conference paper at ICLR 2022
REFERENCES
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv:2103.07579, 2021.
Samuel Rota Bulo, Gerhard Neuhold, and Peter Kontschieder. Loss max- max-pooling for semantic image segmentation. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7082-7091. IEEE, 2017.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268 927799777722 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scabe scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Xianzhi Du, Tsung-Yi Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song. Spinenet: Learning scale-permute backbone for recognition and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11592-11601, 2020.
Pedro F Felzenszwalb, Ross Girshick, and David McAllester. Cascade object detection with deformable part models. In 2010 IEEE Computer society conference computer vision and pattern recognition, pp. 2241-2248. IEEE, 2010.
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise. In Proceedings of the 29th International Joint Conferences on Artificial Intelligence, pp. 2206-2212, 2020.
Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Neurocomputing. 160:93-1017 2015.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2020a.
Santiago Gonzalez and Risto Miikkulainen. Optimizing loss functions through multivariate taylor polynomial parameterization. arXiv preprint 00X0v:2002.00059, 2020b.
Hamideh Hajiabadi, Diego Molla-Aliod, and Reza Monsefi. On extending neural networks with loss ensembles for text classification. arXiv preprint arXiv:/711.05170, 2017.
Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE international conference on evolutionary compulation, pp. 312-317. IEEE, 1996.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp- 770-778, 2016.
Kaiming He,, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-emn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 558-567, 2019.
14

---

Published as a conference paper at ICLR 2022
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12697-12705, 2019.
Hei Law and Jia Deng. Cornemet: Detecting objects as paired keypoints. Proceedings of the European conference computer vision (ECCV), pp. 734-750, 2018.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs:
Automl for loss function search. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 8410-8419, 2019.
Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang, Gao Huang, and Jifeng Dai. Auto seg-loss: Searching metric surrogates for semantic segmentation. arXiv preprint arXiv:2010.07930, 2020.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiy Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2019.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: review. IEEE transactions on pattern analysis and machine intelligence, 2020.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 2017.
Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Poxnt-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529-10553 2020.
Abhinay Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- beseeddbased object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 761-769, 2016.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 2446 2454, 2020.
Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Christian Sminchisescu, and Dragomir Anguelov. Rsn: Range sparse net for efficient, accurate lidar 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Kah-Kay Sung. Learning and example selection for object and pattern detection. 1996.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe V Le. Efficientnetv2: Smaller models and faster training. In International Conference Machine Learning, 2021.
15

---

Published as a conference paper at ICLR 2022
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10781-10790, 2020.
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005. 1000: 2020.
Paul Viola and Michael Jones. Rapid object detection using boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, volume pp. I-I. IEEE, 2001.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 322-330- 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
Haowen Xu, Hao Zhang, Zhiting Hu, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Au- toloss: Learning discrete schedules for alternate optimization. arXiv preprint arXiv: 1010::8810.02442, 2018.
Zhilu Zhang and Mert R Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv: 180..0780 2018.
Guangxiang Zhao, Wenkai Yang, Xuancheng Ren, Lei Li, and Xu Sun. Well-classified examples are underestimated in classification with deep neural networks. arXiv preprint arX0v:2110.065337, 2021.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. Rethinking pre-training and self-training. arXiv preprint arXiv:2006.0683 06882, 2020.
16


Reviews:
[{"review_id": "rDpGGkRMmz_", "reviewer": "Reviewer_kfqh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- (a) Paper is well-written, especially the main part of PolyLoss, which has a clear logic. \n- (b) The paper has an interesting idea that shows cross-entropy and focal loss can be expressed as a linear combination of polynomial functions. Then it unifies these two loss functions as a part of the general polynomial form. The final PolyLoss is simple and effective.\n- (c) The experiments are solid enough to support the main idea.", "summary_of_the_paper": "- **Motivation**.\nThe paper argues that the original cross-entropy loss and focal loss are primarily used in the classification task. \nA general loss function should get rid of constraints of learning tasks and datasets. \n\n\n- **Method**. Motivated by this insight, the paper expresses the loss function as a linear combination of polynomial functions and shows that cross-entropy loss and focal loss are special cases. \nThen inspired by Taylor expansion, the paper proposes a polynomial loss function, called PolyLoss.\nThe final version $\\text{Poly}-1$ is with first-order. \n\n\n- **Experiments**. The paper verifies the proposed loss on ImageNet for classification, MS-COCO for detection and instance segmentation, and Waymo Open dataset for 3D detection.", "summary_of_the_review": "I like this paper. The idea is new, and the discussion is deep. I have no comments on how to improve the paper. Although partial experimental improvements are marginal, I think the novelty is promised. It deserves to expose to the community to inspire future works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "loWRJqKF4Cl", "reviewer": "Reviewer_Zps3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\nThe paper reads well and has almost no typos. It provides an interesting observation showing how different losses could all be expressed in a common polynomial form where only coefficients differ.\n\n# Weaknesses\n\nThe motivation is not entirely clear. While loss functions could be expressed in the form of Taylor expansions, the fact that the polynomial coefficients would have to be fine-tuned in order to obtain further performance improvements introduces even more hyper-parameters for in model giving even more room for possible overfitting. Moreover, the claimed improvements are only marginally higher than the chosen baselines. It is not entirely clear if the added complexity (in both having to fine-tune the coefficients as well as determining where the expansions should be cut) would be justifiable.\n\n## Minor\n\n - \"polynmoial\" (p.9, conclusion)", "summary_of_the_paper": "The paper proposes a framework for creating loss functions based on the polynomial expansion of known loss functions. The authors show that, by fine-tuning the polynomial coefficients of those expansions can bring improvements in multiple computer vision tasks. The authors experiment with image classification (ImageNet-21k), instance segmentation and object detection (COCO), 3D object detection (Waymo), showing small improvements over the chosen baselines. \n\n", "summary_of_the_review": "The paper is interesting, but it is unclear to me whether replacing losses with their polynomial expansions while bringing only small performance improvements to existing methods would suffice for a publication in ICLR. \n\nAfter rebuttal, most of my concerns have been addressed. I am therefore increasing my rating to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3NTyhRDg4k", "reviewer": "Reviewer_Vgzk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Clear and easy to follow text.\n- An application of the Taylor expansion of cross entropy loss and focal loss on different classification problems.\n- Experiments on different classification problems.\n\nWeaknesses:\n- The major problem that I had with the first version, which persists with the current version, is that some results are not significant. The authors obtain very minor improvements on several models and tasks. The large values for AR are actually not impressive since many simple tricks can provide similar (even better) improvements for AR. And the reported improvements for all other measurements and tasks are rather minimal.\n\nMinor comment:\n- \"can improves\" => \"can improve\".", "summary_of_the_paper": "\nIn this paper, the authors introduce a new loss function for classification problems. To be specific, the authors introduce Taylor expansion of cross-entropy loss + focal loss and show that various subsets of this expansion can improve the models on image classification, 2D object detection and 3D object detection problems.\n\nI've reviewed a previous version of this paper at NeurIPS2021. Compared to the previous version, I see that the authors improved the paper significantly, mainly in terms of structure and presentation.", "summary_of_the_review": "In general, I am positive about the paper, though I need to be convinced about the significance of the results and the importance of the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "133fsHzSjds", "reviewer": "Reviewer_KpJa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n(1) The polynomial expansion perspective is interesting and the PloyLoss makes sense.\n\n(2) Extensive experiments are provided and further produce some findings. \n\n(3) The proposed loss function can improve various tasks: image classification, instance segmentation, and 3D object detection.\n\n## Weaknesses\n(1) surprisingly, the final version of PloyLoss is Ploy-1. I think this final version is trivial. Does this mean the high-order items are useless? I understand the results of $L_{Drop}$ in Figure 2(a), but I still think the poor results of the low-order loss function are because of the bad coefficient tuning. So can you provide the following versions for comparison?\n- L1 loss with hyper-parameter tuning. (Drop the CE part in the Ploy-1.)\n- L1 and L2 loss with the hyper-parameter tuning.\n\n(2) Because this paper proposes a general loss family, the exploration is still insufficient. There are many variables of the PloyLoss family, such as the coefficient (particularly analysis in this paper), the start order ($\\gamma$ in Focal loss), and the coefficient decay strategy.\n- CE and focal loss only provides one kind of coefficient organization strategy as the logarithmic function. What about others? For example, the exponential decay for the coefficients.\n\n(3) The name is misleading for $L_{ploy-1}$ in image classification and $L_{ploy-1}$ in object detection since they use different base loss functions (CE and focal respectively). \n\n(4) It seems like some errors in the label of the y-axis in Figure 2(b).", "summary_of_the_paper": "This paper analyzes the cross-entropy loss and focal loss through the polynomial expansion perspective. Further, this paper proposes a family of loss functions called PloyLoss. Three instances of PloyLoss are analyzed and obtain the final version as ploy-1. Detailed and extensive experiments are provided.", "summary_of_the_review": "This paper starts from an interesting loss family as PloyLoss, but the final version is quite simple and trivial. Considering both the experiments, motivation, and the final method, I would like to rank this paper as 5. I am positive about this paper, but more experiments and clarification are expected.\n\nAfter rebuttal, most of my concerns are addressed. Thus, I raise the rank to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Realizing the fact that cross-entropy loss and focal loss are widely used for training deep learning models but mathematical understanding and exploration for such losses are lacking, the authors propose a simple framework named PolyLoss to express the loss function as a linear combination of polynomial functions. 

In this framework, the aforementioned cross-entropy loss and focal loss are the special cases of PolyLoss by easily adjusting the importance of different polynomial bases depending on the targeting tasks and datasets. The final version of PolyLoss, Poly-1 formulation, is simple with one line of code and an extra hyperparameter but outperforms the cross-entropy loss and focal loss on 2D image classification, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin.

This paper is well-motivated by a novel perspective of polynomial expansion. The proposed method is novel, simple to implement, and effective in practice. The authors have a deep and thorough discussion with reviewers, and most concerns were well addressed. After rebuttal and discussion, reviewers increased their scores, and all agreed with acceptance. AC checked the paper and all relevant information, and found sufficient ground for acceptance.
Decision: Accept (Poster)