POLYLOSS: A POLYNOMIAL EXPANSION PERSPEC-
Year: 2021

Abstract:
Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classification problems. Generally speaking, however, a good loss function can take on much more flexible forms, and should be tailored for different tasks and datasets. Motivated by how functions can be approximated via Taylor expansion, we propose a simple framework, named PolyLoss, to view and design loss functions as a linear combination of polynomial functions. Our PolyLoss allows the importance of different polynomial bases to be easily adjusted depending on the targeting tasks and datasets, while naturally subsuming the aforementioned cross-entropy loss and focal loss as special cases. Extensive experimental results show that the optimal choice within the PolyLoss is indeed dependent on the task and dataset. Simply by introducing one extra hyperparameter and adding one line of code, our Poly-1 formulation outperforms the crossentropy loss and focal loss on 2D image classification, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin. Task ImageNet classification COCO det. and seg. Waymo Open Dataset 3D detection Default loss Cross-entropy Cross-entropy Focal loss Model ENetV2-L(21K) ENetV2-L(1K) Mask R-CNN PointPillars Car PointPillars Ped RSN Car RSN Ped Baseline 45.8 86.8 47.2 42.3 63.3 68.9 78.4 79.4 PolyLoss 46.4 (+0.6) 87.2 (+0.4) 49.7 (+2.5) 44.4 (+2.1) 63.7 (+0.4) 69.6 (+0.7) 78.9 (+0.5) 80.2 (+0.8) Table 1: Polyloss outperforms cross-entropy and focal loss on various models and tasks. Results are for the simplest Poly-1, which has only one single hyparameter 1. On ImageNet (Deng et al., 2009), our PolyLoss, improves both pretraining and finetuning for the recent EfficientNetV2 (Tan & Le, 2021);On COCO (Lin et al., 2014), Polyloss improves both 2D detection and segmentation AR for Mask-RCNN (He et al., 2017); on Waymo Open Dataset (WOD), Polyloss improves 3D detection AP for the widely used PointPillars (Lang et al., 2019) and the very recent Range Sparse Net (RSN) (Sun et al., 2021). Details are in Table 4, 5, 7.

heading: 1 INTRODUCTION
text: Loss functions are important in training neural networks. In principle, a loss function could be any (differentiable) function that maps predictions and labels to a scalar. Therefore, designing a good loss function is generally challenging due to its large design space, and designing a universal loss function that works across different tasks and datasets is even more challenging: for example, L1 / L2 losses are commonly used for regression tasks, but they are rarely used for classification tasks; focal loss is often used to alleviate the overfitting issue of cross-entropy loss for imbalanced object detection datasets (Lin et al., 2017), but it is not shown to consistently help other tasks. Many recent works have also explored new loss functions via meta-learning, ensembling or compositing different losses (Hajiabadi et al., 2017; Xu et al., 2018; Gonzalez & Miikkulainen, 2020b;a; Li et al., 2019).
In this paper, we propose PolyLoss: a novel framework for understanding and designing loss functions. Our key insight is to decompose commonly used classification loss functions, such as crossentropy loss and focal loss, into a series of weighted polynomial bases. They are decomposed in the form of ∑∞ j=1 αj(1 − Pt)j , where αj ∈ R+ is the polynomial coefficient and Pt is the prediction probability of the target class label. Each polynomial base (1−Pt)j is weighted by a corresponding polynomial coefficient αj , which enables us to easily adjust the importance of different bases for different applications. When αj = 1/j for all j, our PolyLoss becomes equivalent to the commonly used cross-entropy loss, but this coefficient assignment may not be optimal.
Our study shows that, in order to achieve better results, it is necessary to adjust polynomial coefficients αj for different tasks and datasets. Since it is impossible to adjust an infinite number of αj , we explore various strategies with a small degree of freedom. Perhaps surprisingly, we observe that simply adjusting the single polynomial coefficient for the leading polynomial, which we denote LPoly-1, is sufficient to achieve significant improvements over the commonly used cross-entropy loss and focal loss. Overall, our contribution can be summarized as:
• Insights on common losses: We propose a unified framework, named PolyLoss, to rethink and redesign loss functions. This framework helps to explain cross-entropy loss and focal loss as two special cases of the PolyLoss family (by horizontally shifting polynomial coefficients), which was not recognized before. This new finding motivates us to investigate new loss functions that vertically adjust polynomial coefficients, shown in Figure 1. • New loss formulation: We evaluate different ways of vertically manipulating polynomial coefficients to simplify the hyperparameters search space. We propose a simple and effective Poly-1 loss formulation which only introduces one hyperparameter and one line of code. • New findings: We identify that focal loss, though effective for many detection tasks, is suboptimal for the imbalanced ImageNet-21K. We find the leading polynomial contributes to a large portion of the gradient during training, and its coefficient correlates to the prediction confidence Pt. In addition, we provide an intuitive explanation on how to leverage this correlation to design good PolyLoss tailored to imbalanced datasets. • Extensive experiments: We evaluate our PolyLoss on different tasks, models, and datasets. Results show PolyLoss consistently improves the performance on all fronts, summarized in Table 1, which includes the state-of-the-art classifiers EfficientNetV2 and detectors RSN.

heading: 2 RELATED WORK
text: Cross-entropy loss is used in many popular and current state-of-the-art models for perception tasks such as classification, detection and semantic segmentation (Tan & Le, 2021; He et al., 2017; Zoph et al., 2020; Tao et al., 2020). Instead of trying to improve cross-entropy loss for specific tasks like many other prior works, the goal of this paper is to provide a unified framework for systematically designing a better classification loss function. Loss for class imbalance Training detection models, especially single-stage detectors, is difficult due to class imbalance. Common approaches such as hard example mining and reweighing are developed to address the class imbalance issue (Sung, 1996; Viola & Jones, 2001; Felzenszwalb et al., 2010; Shrivastava et al., 2016; Liu et al., 2016; Bulo et al., 2017). As one of these approaches, focal loss is designed to mitigate the class imbalance issue by focusing on the hard examples and is used to train state-of-the-art 2D and 3D detectors (Lin et al., 2017; Tan et al., 2020; Du et al., 2020; Shi et al., 2020; Sun et al., 2021). In our work, we found that focal loss is suboptimal for the imbalanced ImageNet-21K. Using the PolyLoss framework, we discover a better loss function, which performs the opposite role of focal loss. We further provide intuitive understanding of why it is important to design different loss functions tailored to different imbalanced datasets using the PolyLoss framework. Robust loss to label noise Another direction of research is to design loss functions that are robust to label noise (Ghosh et al., 2015; 2017; Zhang & Sabuncu, 2018; Wang et al., 2019; Oksuz et al., 2020; Menon et al., 2020). A commonly used approach is to incorporate noise robust loss function such as Mean Absolute Error (MAE) into cross-entropy loss. In particular, Taylor cross entropy loss is proposed to unify MAE and cross-entropy loss by expanding the cross-entropy loss in (1 − Pt)j polynomial bases (Feng et al., 2020). By truncating the higher-order polynomials, they show truncated cross-entropy loss function is closer to MAE, which is more robust to label noise on datasets with synthetic label noise. In contrast, our PolyLoss provides a more general framework to design loss functions for different datasets by manipulating polynomial coefficients, which includes dropping higher-order polynomials proposed in (Feng et al., 2020). Our experiments in subsection 4.1 show the loss proposed in (Feng et al., 2020) performs worse than cross-entropy loss on the clean ImageNet dataset. Learned loss functions Several recent works demonstrate learning the loss function during training via gradient descent or meta learning (Hajiabadi et al., 2017; Xu et al., 2018; Gonzalez & Miikkulainen, 2020a; Li et al., 2019; 2020). Notably, TaylorGLO utilizes CMA-ES to optimize multivariate Taylor parameterization of a loss function and learning rate schedule during training (Hansen & Ostermeier, 1996; Gonzalez & Miikkulainen, 2020b). Due to the search space scale with the order
of polynomials, the paper demonstrates that using the third-order parameterization (8 parameters), the learned loss function schedule outperforms cross-entropy loss on 10-class classification problems. Our paper Figure 2a, on the other hand, shows for 1000-class classification tasks, hundreds of polynomials are needed. This results in a prohibitively large search space. Our proposed Poly-1 formulation mitigates the challenge of the large search space and do not rely on advanced black-box optimization algorithms. Instead, we show a simple grid search over one hyperparameter can lead to significant improvement on all tasks that we investigate.

heading: 3 POLYLOSS
text: PolyLoss provides a framework for understanding and improving the commonly used cross-entropy loss and focal loss, visualized in Figure 1. It is inspired from the Taylor expansion of cross-entropy loss (Equation 1) and focal loss (Equation 2) in the bases of (1− Pt)j :
LCE = − log(Pt) = ∞∑ j=1 1/j(1− Pt)j = (1− Pt) + 1/2(1− Pt)2... (1)
LFL = −(1− Pt)γ log(Pt) = ∞∑ j=1 1/j(1− Pt)j+γ = (1− Pt)1+γ + 1/2(1− Pt)2+γ ... (2)
where Pt is the model’s prediction probability of the target ground-truth class.
Cross-entropy loss as PolyLoss Using the gradient descent method to optimize the cross-entropy loss requires taking the gradient with respect to Pt. In the PolyLoss framework, an interesting observation is that the coefficients 1/j exactly cancel the jth power of the polynomial bases, see Equation 1. Thus, the gradient of cross-entropy loss is simply the sum of polynomials (1 − Pt)j , shown in Equation 3.
−dLCE dPt = ∞∑ j=1 (1− Pt)j−1 = 1 + (1− Pt) + (1− Pt)2... (3)
The polynomial terms in the gradient expansion capture different sensitivity with respect to Pt. The leading gradient term is 1, which provides a constant gradient regardless of the value of Pt. On the contrary, when j 1, the jth gradient term is strongly suppressed when Pt gets closer to 1.
Focal loss as PolyLoss In the PolyLoss framework, Equation 2, it is apparent that the focal loss simply shifts the power j by the power of a modulating factor γ. This is equivalent to horizontally shifting all the polynomial coefficients by γ as shown in Figure 1. To understand the focal loss from a gradient prospective, we take the gradient of the focal loss (Equation 2) with respect to Pt:
−dLFL dPt = ∞∑ j=1 (1 + γ/j)(1− Pt)j+γ−1 = (1 + γ)(1− Pt)γ + (1 + γ/2)(1− Pt)1+γ ... (4)
For a positive γ, the gradient of focal loss drops the constant leading gradient term, 1, in the crossentropy loss, see Equation 3. As discussed in the previous paragraph, this constant gradient term causes the model to emphasize the majority class, since its gradient is simply the total number of
examples for each class. By shifting the power of all the polynomial terms by γ, the first term then becomes (1 − Pt)γ , which is suppressed by the power of γ to avoid overfitting to the already confident (meaning Pt close to 1) majority class. More details shown in Appendix F.
Connection to regression and general form Representing the loss function in the PolyLoss framework provides an intuitive connection to regression. For classification tasks where y = 1 is the effective probability of the ground-truth label, the polynomial bases (1 − Pt)j can be expressed as (y − Pt)j . Thus both cross-entropy loss and focal loss can be interpreted as a weighted ensemble of distances between the prediction and label to the jth power. However, a fundamental question in those losses: Are the coefficients in front of the regression terms optimal? In general, PolyLoss is a monotone decreasing function1 on [0, 1] which can be expressed as∑∞ j=1 αj(1 − Pt)j , which provide a flexible framework to adjust each coefficient2. PolyLoss can be generalized to non-integer j, but for simplicity we only focus on integer power (j ∈ Z+) in this paper. In the next section, we investigate several strategies on designing better loss functions in the PolyLoss framework via manipulating αj .

heading: 4 UNDERSTANDING THE EFFECT OF POLYNOMIAL COEFFICIENTS
text: In the previous section, we established the PolyLoss framework and showed that cross-entropy loss and focal loss simply correspond to different polynomial coefficients, where focal loss horizontally shifts the polynomial coefficients of cross-entropy loss.
In this section, we propose the final loss formulation Poly-1. We study in depth how vertically adjusting polynomial coefficients, shown in Figure 1, may affect training. Specifically, we explore three different strategies in assigning polynomial coefficients: dropping higher-order terms; adjusting multiple leading polynomial coefficients; and adjusting the first polynomial coefficient, summarized in Table 2. We find adjusting the first polynomial coefficient (Poly-1 formulation) leads to maximal gain while requiring minimal code change and hyperparameter tuning.
In these explorations, we experiment with 1000-class ImageNet (Deng et al., 2009) classification. We abbreviate it as ImageNet-1K to differentiate it from the full version, which contains 21K classes. We use ResNet-50 (He et al., 2016) and its training hyperparameters without modification.3
4.1 LDrop: REVISITING DROPPING HIGHER-ORDER POLYNOMIAL TERMS
Prior works (Feng et al., 2020; Gonzalez & Miikkulainen, 2020b) have shown dropping the higherorder polynomials and tuning the leading polynomials can improve model robustness and performance. We adopt the same loss formulation LDrop = ∑N j=1 1/j(1 − Pt)j , as in Feng et al. (2020), and compare their performance with the baseline cross-entropy loss on ImageNet-1K. As shown in Figure 2a, we need to sum up more than 600 polynomial terms to match the accuracy of crossentropy loss. Notably, removing higher-order polynomials cannot simply be interpreted as adjusting the learning rate. To verify this, Figure 2b compares the performance for different learning rates with various cutoffs: no matter we increase or decrease the learning rate from the original value of 0.1, the accuracy worsens. Additional hyperparameter tuning is shown in Appendix C.
1We only consider the case all αj ≥ 0 in this paper for simplicity. There exist monotone decreasing functions on [0, 1] with some αj negative, for example sin(1− Pt) = ∑∞ j=0(−1)
j/(2j + 1)!(1− Pt)2j+1. 2To ensure series converges, we require 1/ lim supj→∞ j √ |αj | ≥ 1 for Pt in (0, 1]. For Pt = 0 we don’t
require point-wise convergence; in fact cross-entropy and focal loss both go to +∞. 3Code at https://github.com/tensorflow/tpu/tree/master/models/official/
To understand why higher-order terms are important, we consider the residual sum after removing the firstN polynomial terms from cross-entropy loss: RN = LCE−LDrop = ∑∞ j=N+1 1/j(1−Pt)j .
Theorem 1. For any small ζ > 0, δ > 0 if N > log1−δ (ζ · δ), then for any p ∈ [δ, 1], we have |RN (p)| < ζ and |R′N (p)| < ζ. (Proof in Appendix A)
Hence, taking a large N is necessary to ensure LDrop is uniformly close to LCE in the perspectives of loss and loss derivative on [δ, 1]. For a fixed ζ, as δ approaches 0,N grows rapidly. Our experimental results align with the theorem. The higher-order (j > N + 1) polynomials play an important role during the early stages of training, where Pt is typically close to zero. For example, when Pt ∼ 0.001, according to Equation 3, the coefficient of the 500th term’s gradient is 0.999499 ∼ 0.6, which is fairly large. Different from aforementioned prior works, our results show that we cannot easily reduce the number of polynomial coefficients αj by excluding the higher-order polynomials.
Dropping higher order polynomials is equivalent to pushing all the higher order (j > N+1) polynomial coefficients αj vertically to zero in the PolyLoss framework. Since simply setting coefficients to zero is suboptimal for training ImageNet-1K, in the following sections, we investigate how to manipulate polynomial coefficient beyond setting them to zero in the PolyLoss framework. In particular, we aim to propose a simple and effective loss function that requires minimal tuning.
4.2 LPOLY-N : PERTURBING LEADING POLYNOMIAL COEFFICIENTS
In this paper, we propose an alternative way of designing a new loss function in the PolyLoss framework, where we adjust the coefficients of each polynomial. In general, there are infinitely many polynomial coefficients αj need to be tuned. Thus, it is infeasible to optimize the most general loss:
LPoly = α1(1− Pt) + α2(1− Pt)2 + ...+ αN (1− Pt)N + ... = ∞∑ j=1 αj(1− Pt)j (5)
The previous section (subsection 4.1) has shown that hundreds of polynomials are required in training to do well on tasks such as ImageNet-1K classification. If we naively truncate the infinite sum in Equation 5 to the first few hundreds terms, tuning coefficients for so many polynomials still results in a prohibitively large search space. In addition, collectively tuning many coefficients also does not outperform cross-entropy loss, details in Appendix D.
To tackle this challenge, we propose to perturb the leading polynomial coefficients in cross-entropy loss, while keeping the rest the same. We denote the proposed loss formulation as Poly-N, where N stands for the number of leading coefficients that will be tuned.
LPoly-N = ( 1 + 1)(1− Pt) + ...+ ( N + 1/N)(1− Pt)N︸ ︷︷ ︸ perturbed by j +1/(N + 1)(1− Pt)N+1 + ...︸ ︷︷ ︸ same as LCE
= − log(Pt) + N∑ j=1 j(1− Pt)j (6)
CE loss N=1 N=2 N=3 N-dim. grid search 76.3 76.7 76.8 – Greedy grid search 76.3 76.7 76.7 76.7
Table 3: LPoly-N outperforms crossentropy loss on ImageNet-1K.
Here, we replace the jth polynomial coefficient in crossentropy loss 1/j with 1/j + j , where j ∈ [−1/j,∞) is the perturbation term. This allows us to pinpoint the first N polynomials without the need to worry about the infinitely many higher-order (j > N + 1) coefficients, as in Equation 5.
Table 3 shows LPoly-N outperforms the baseline cross-entropy loss accuracy. We explore Ndimensional grid search and greedy grid search of j in LPoly-N up to N = 3 and find that simply adjusting the coefficient of the first polynomial (N = 1) leads to better classification accuracy. Performing 2D grid search (N = 2) can further boost the accuracy. However, the additional gain is small (+0.1) compared to adjusting only the first polynomial (+0.4).
4.3 LPOLY-1 : SIMPLE AND EFFECTIVE
As shown in the previous section, we find tuning the first polynomial term leads to the most significant gain. In this section, we further simplify the Poly-N formulation and focus on evaluating Poly-1, where only the first polynomial coefficient in cross-entropy loss is modified.
LPoly-1 = (1 + 1)(1− Pt) + 1/2(1− Pt)2 + ... = − log(Pt) + 1(1− Pt) (7)
We study the effect of different first term scaling on the accuracy and observe that increasing the first polynomial coefficient can systematically increase the ResNet-50 accuracy, as shown in Figure 3a. This result suggests that the cross-entropy loss is suboptimal in terms of polynomial coefficient values, and increasing the first polynomial coefficient leads to consistent improvement, which is comparable to other training techniques (Appendix E).
Figure 3b shows the leading polynomial contributes to more than half of the cross-entropy gradient during training for the majority of the time, which highlights the significance of the first polynomial term (1 − Pt) compared to the rest of the infinite many terms. Therefore, in the remaining of the paper, we adopt the form of LPoly-1 and primarily focus on adjusting the leading polynomial coefficient. As is evident from Equation 7, it only modifies the original loss implementation by a single line of code (adding a 1(1− Pt) term on top of cross-entropy loss). Note that, all the training hyperparameters are optimized for cross-entropy loss. Even so, a simple grid search on the first polynomial coefficients in the Poly-1 formulation significantly increases the classification accuracy. We find optimizing other hyperparameters for LPoly-1 leads to higher accuracy, and show more details in Appendix B.

heading: 5 EXPERIMENTAL RESULTS
text: In this section, we compare our PolyLoss against the commonly used cross-entropy loss and focal loss on various tasks, models, and datasets. For the following experiments, we adopt the default training hyperparameters in the public repositories without any tuning. Nevertheless, Poly-1 formulation leads to consistent advantage over default loss functions at the cost of a simple grid search.
5.1 LPOLY-1 IMPROVES 2D IMAGE CLASSIFICATION ON IMAGENET
Image classification is a fundamental problem in computer vision, and progress on image classification has led to progress on many related computer vision tasks. In terms of the network architecture, in addition to the ResNet-50 already used in section 4, we also experiment with the state-of-the-art EfficientNetV2 (Tan & Le, 2021). We follow the ImageNet settings in Tan & Le (2021) except for replacing the original cross-entropy loss with our PolyLoss LPoly−1 with different values of 1. In terms of the dataset, in addition to the ImageNet-1K dataset already used in section 4, we also consider ImageNet-21K, which has about 13M training images with 21,841 classes. We will study both the ImageNet-21K pretraining results and the ImageNet-1K finetuning results.
Pretraining EfficientNetV2-L on ImageNet-21K, then finetuning it on ImageNet-1K can improve classification accuracy (Tan & Le, 2021). Here, we follow the same pretraining and finetuning schedule as reported in Tan & Le (2021) without modification4 but replace the cross-entropy loss with LPoly-1 = − log(Pt) + 1(1 − Pt). We reserve 25,000 images from the training set as minival to search the optimal 1.
Pretraining on ImageNet-21K Figures 4 highlights the importance of using tailored loss function when pretraining model on ImageNet21K dataset. A simple grid search over 1 ∈ [0, 1, 2, . . . , 7] in LPoly-1 without changing other default hyperparameters leads to around 1% accuracy gain for all SOTA EfficientNetV2 models with different sizes. The accuracy improvement of using a better loss function nearly matches the improvement of scaling up the model architecture (S to M and M to L).
Surprisingly, see Figure 5a, increasing the weight of the leading polynomial coefficient improves the accuracy of pretraining on ImageNet-21K (+0.6), whereas reducing it low-
ers the accuracy (-0.9). Setting 1 = −1 truncates the leading polynomial term in the cross-entropy loss (Equation 1), which is similar to having a focal loss with γ = 1 (Equation 2). However, the opposite change, where 1 > 0, improves the accuracy on the imbalanced ImageNet-21K.
We hypothesize the prediction of the imbalanced ImageNet-21K is not confident enough (Pt is small), and using positive 1 PolyLoss leads to more confident predictions. To validate our hypothesis, we plot Pt as a function of training steps in Figure 5b. We observe that 1 directly controls the mean Pt over all classes. Using positive 1 PolyLoss leads to more confident prediction (higher Pt). On the other hand, negative 1 PolyLoss lowers the confidence.
Fine tuning on ImageNet-1K After pretraining on ImageNet-21K, we take the EfficientNetV2-L checkpoint and finetune it on ImageNet-1K, using the same procedure as (Tan & Le, 2021) except for replacing the original cross-entropy loss with the Poly-1 formulation. PolyLoss improves the finetuning accuracy by 0.4%, advancing the ImageNet-1K top-1 accuracy from 86.8% to 87.2%.
5.2 LPOLY-1 IMPROVES 2D INSTANCE SEGMENTATION AND OBJECT DETECTION ON COCO
Instance segmentation and object detection require localizing objects in an image in addition to recognizing them: the former in the form of arbitrary shapes and the latter in the form of bounding boxes. For both instance segmentation and object detection, we use the popular COCO (Lin et al., 2014) dataset, which contains 80 object classes. We choose Mask R-CNN (He et al., 2017) as the representative model for instance segmentation and object detection. These models optimize multiple losses, e.g. LMaskRCNN = Lcls + Lbox + Lmask. For the following experiments, we only replace the Lcls with PolyLoss and leave other losses intact. Results are summarized in Table 5.
4 Code at https://github.com/google/automl/tree/master/efficientnetv2
Reducing the leading polynomial coefficient improves Mask R-CNN AP and AR. In training Mask R-CNN, we use the training schedule optimized for cross-entropy loss,5 and replace the crossentropy loss with LPoly−1 = − log(Pt) + 1(1 − Pt) for the classification loss Lcls, where 1 ∈ {−1.0,−0.8,−0.6,−0.4,−0.2, 0, 0.5, 1.0}. We ensure the leading coefficient is positive, i.e. 1 ≥ −1. Our results in Figure 6a show systematic improvements of box AP, box AR, mask AP, and mask AR as we reduce the weight of the first polynomial by using negative 1 values. Note that Poly-1 ( = −1) not only improves AP but also significantly increases AR, shown in Table 5.
Tailoring loss function to datasets and tasks is important. ImageNet-21K and COCO are both imbalanced but the optimal for PolyLoss are opposite in sign, i.e. = 2 for ImageNet-21K classification and = −1 for Mask R-CNN detection. We plot the Pt of the Mask R-CNN classification head and found the original prediction is overly confident (Pt is close to 1) on the imbalanced COCO dataset, thus using a negative lowers the prediction confidence, as shown in Figure 6b. This effect is similar to label smoothing, but unlike label smoothing, as long as > −1, PolyLoss reduces the gradient but will not penalize over confident prediction.
5.3 LPOLY-1 IMPROVES 3D OBJECT DETECTION ON WAYMO OPEN DATASET
Detecting 3D objects from LiDAR point clouds is an important topic and can directly benefit autonomous driving applications. We conduct these experiments on the Waymo Open Dataset (Sun et al., 2020). Similar to 2D detectors, 3D detection models are commonly based on single-stage and two-stage architectures. Here, we evaluate our PolyLoss on two models: a popular single-stage PointPillars model (Lang et al., 2019); and a state-of-the-art two-stage Range Sparse Net (RSN) model (Sun et al., 2021). Both models rely on multi-task loss functions during training. Here, we focus on improving the classification focal loss by replacing it with PolyLoss. Similar to the 2D perception cases, we adopt the Poly-1 formulation to improve upon focal loss, shown in Table 6.
PolyLoss improves single-stage PointPillars model. The PointPillars model converts the raw 3D point cloud to a 2D top-down pseudo image, and then detect 3D bounding boxes from the 2D image in a similar way to RetinaNet (Lin et al., 2017). Here, we replace the classification
5Code at https://github.com/tensorflow/tpu/tree/master/models/official
focal loss (γ = 2) with LFLPoly-1 = −(1 − Pt)2 logPt + 1(1 − Pt)3 and adopt the same training schedule optimized for focal loss without any modification6. Table 7 shows that LFLPoly-1 with = −1 leads to significant improvement on all the metrics for both vehicle and pedestrian models.
Advancing the state-of-the-art with RSN. RSN segments foreground points from the 3D point cloud in the first stage, and then applies sparse convolution to predict 3D bounding boxes from the selected foreground points. RSN uses the same focal loss as the PointPillars model, i.e., LFL = −(1−Pt)2 logPt. Since the optimal LFLPoly-1 for PointPillars ( 1 = −1) is equivalent to dropping the first polynomial, we adapt the same loss formulation for RSN and tune the new leading polynomial (1−Pt)4 by defining LFLPoly-1∗ = −(1−Pt)2 log(Pt)− (1 − Pt)3 + 2(1 − Pt)4, shown in Figure 7. We follow the same training schedule optimized for focal loss described in (Sun et al., 2021) without adjustment. Our results, in Table 7, show that tuning the new leading polynomial improves all metrics (except vehicle detection BEV APH L2) for the SOTA 3D detector.

heading: 6 CONCLUSION
text: In this paper, we propose the PolyLoss framework, which provides a unified view on common loss functions for classification problems. We recognize that, under polynomial expansion, focal loss is a horizontal shift of the polynomial coefficients compared to the cross-entropy loss. This new insight motivates us to explore an alternative dimension. i.e. vertically modify the polynomial coefficients.
Our PolyLoss framework provides flexible ways of changing the loss function shape by adjusting the polynomial coefficients. In this framework, we propose a simple and effective Poly-1 formulation. By simply adjusting the coefficient of the leading polynomial coefficient with just one extra hyperparameter 1, we show our simple Poly-1 improves a variety of models across multiple tasks and datasets. We hope Poly-1 formulation’s simplicity (one extra line of code) and effectiveness will lead to adoption in more applications of classification than the ones we have managed to explore.
More importantly, our work highlights the limitation of common loss functions, and simple modification could lead to improvements even on well established state-of-the-art models. We hope these findings will encourage exploring and rethinking the loss function design beyond the commonly used cross-entropy and focal loss, as well as the simplest Poly-1 loss proposed in this work.
6Code at https://github.com/tensorflow/lingvo/tree/master/lingvo/tasks/car
REPRODUCIBILITY STATEMENT
Our experiments are based on public datasets and open source code repositories, shown in footnote 3-6. We do not tune any default training hyperparameters and only modify the loss functions, which are shown in Table 2-7. The proposed final formulation LPoly-1 requires one line of code modification, i.e. LOSS = CE + epsilon * (1-Pt) and LOSS = FL + epsilon * tf.math.pow((1-Pt), 3), where LOSS, CE, FL, and Pt are float tensor with shape [batch size].

heading: A PROOF OF THEOREM 1
text: Theorem 1. For any small ζ > 0, δ > 0 if N > log1−δ (ζ · δ), then for any p ∈ [δ, 1], we have |RN (p)| < ζ and |R′N (p)| < ζ.
Proof.
|RN (p)| = ∞∑
j=N+1
1/j(1− p)j ≤ ∞∑
j=N+1
(1− p)j = (1− p) N+1 p ≤ (1− δ) N+1 δ ≤ (1− δ) N δ
|R′N (p)| = ∞∑ j=N (1− p)j = (1− p) N p ≤ (1− δ) N δ

heading: B ADJUSTING OTHER TRAINING HYPERPARAMETERS LEADS TO HIGHER GAIN.
text: All the experiments shown in the main text are based on hyperparameters optimized for the baseline loss function, which actually puts PolyLoss at a disadvantage. Here we use weight decay rate for ResNet50 as an example. The default weight decay (1e-4) is optimized for cross-entropy loss. Adjusting the decay rate may reduce the model performance of cross-entropy loss but leads to much higher gain for PolyLoss (+0.8%), which is better than the best accuracy (76.3%) trained using cross-entropy loss (+0.8%).
C LDROP WITH MORE HYPERPARAMETER TUNING
For LDrop (N = 2), besides adjusting the learning rate, we further tune the coefficient (α) of the second polynomial, similar to a prior work (Gonzalez & Miikkulainen, 2020b), and weight decay.
LDrop* = (1− Pt) + α(1− Pt)2 (8)
Unlike (Feng et al., 2020), where α = 0.5 after dropping all higher-order polynomial, we find the optimal α = 8, while the optimal learning rate is the same as the default setting (0.1). This alone increases the accuracy to 70.9, which shows simply dropping polynomial terms is not enough and adjusting the polynomial coefficients is critical. Further tuning weight decay leads to less than 0.1% model quality improvement.
Comparing to prior works (Gonzalez & Miikkulainen, 2020b; Feng et al., 2020), Poly-1 is more effective and only contains one hyperparameter. Tuning weight decay of Poly-1 further increases the accuracy while having less hyperparameters compared to LDrop∗ , shown in Table 9.

heading: D COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS
text: Besides adjusting individual polynomial coefficients, in this section, we explore collectively tuning multiple polynomial coefficients in the PolyLoss framwork. In particular, we change the coefficients in the original cross-entropy loss from 1/j (Equation 1) to exponential decay. Here, we define
Lexp = 2N∑ j=1 e−(j−1)/N (1− Pt)j (9)
where we cut off the infinite sum at twice the decay factor N . We performed 2D grid search on N ∈ {5, 20, 80, 320} and learning rate ∈ {0.1, 0.4, 1.6, 6.4}. The best accuracy is 72.3, where N = 80 and learning rate = 1.6, shown in Table 10.
Though Poly-1 is better than using Lexp, there are a lot more possibilities besides using exponential decay. We believe understanding how collectively tuning multiple coefficients affects the training is an important topic.

heading: E COMPARING TO OTHER TRAINING TECHNIQUES
text: As shown in recent works (He et al., 2019; Bello et al., 2021; Wightman et al., 2021), though independent novel training techniques often lead to sub 1% improvement, combining them could lead to significant overall improvements. To put things into perspective, Poly-1 achieves similar improvements as other commonly used training techniques, such as label smoothing and dropout on FC, shown in Table 11.

heading: F REDISCOVERING FOCAL LOSS FROM POLYLOSS
text: Focal loss was first developed for single-stage detector RetinaNet to address strong class imbalance presented in object detection (Lin et al., 2017). Here, we provide an additional ablation study on how to systemically discover focal loss in the PolyLoss framework and investigate how the leading terms affect training in the presence of class imbalance.
Rediscovering the concept of focal loss from crossentropy loss. Here, we take a step back and attempt to systematically rediscover the concept of focal loss via our PolyLoss framework. Focal loss is commonly used for training detection models. Coming up with such an insight to address the class imbalance issue in detection requires strong domain expertise. We start with the PolyLoss representation of cross-entropy loss and improve it from the PolyLoss gradient perspective.
We start with the cross-entropy loss and define PolyLoss by dropping the first N polynomials in cross-entropy loss, i.e. LDrop-front = ∑∞ j=N+1 1/j(1 − Pt)j = LCE −∑N
j=1 1/j(1 − Pt)j . Dropping the first two polynomial terms (1− Pt) significantly improves both the detection AP and AR, see Figure 8. Dropping the first two polynomials (N = 2) leads to the best RetinaNet performance, which is similar to setting γ = 2 in focal loss, i.e. focal loss γ = 2 pushes all the polynomial coefficients to the right by 2, shown in Figure 1 right, which is similar to truncating the first two polynomial terms.
Leading polynomials cause overfitting to the majority class. In the PolyLoss framework, the leading polynomial of cross-entropy loss is a constant, shown in Equation 3. For binary classification, the leading gradient for each class is simplyNbackground−Nobject, where Nbackground and Nobject are the counts of background and object instances in the training mini-batch. When the class counts are extremely imbalanced, the majority class will dominate the gradient which will lead to significant bias towards optimizing the majority class.
Dropping polynomials reduces the extremely confident prediction Pt, see Figure 9. To examine the composition of the overall prediction confidence, we also plot the Pt
for background only and Pt for object only. Due to the extreme imbalance between the background and the object class, the overall Pt is dominated by the background only Pt. So reducing the overall Pt decreases the background Pt. On the other hand, reducing overfitting to the majority background class leads to more confident prediction Pt on the object class.

Review summaries:
The paper proposes a framework for creating loss functions based on the polynomial expansion of known loss functions. The authors show that, by fine-tuning the polynomial coefficients of those expansions can bring improvements in multiple computer vision tasks. The authors experiment with image classification (ImageNet-21k), instance segmentation and object detection (COCO), 3D object detection (Waymo), showing small improvements over the chosen baselines.   

This paper analyzes the cross-entropy loss and focal loss through the polynomial expansion perspective. Further, this paper proposes a family of loss functions called PloyLoss. Three instances of PloyLoss are analyzed and obtain the final version as ploy-1. Detailed and extensive experiments are provided.

- **Motivation**. The paper argues that the original cross-entropy loss and focal loss are primarily used in the classification task.  A general loss function should get rid of constraints of learning tasks and datasets.    - **Method**. Motivated by this insight, the paper expresses the loss function as a linear combination of polynomial functions and shows that cross-entropy loss and focal loss are special cases.  Then inspired by Taylor expansion, the paper proposes a polynomial loss function, called PolyLoss. The final version $\text{Poly}-1$ is with first-order.    - **Experiments**. The paper verifies the proposed loss on ImageNet for classification, MS-COCO for detection and instance segmentation, and Waymo Open dataset for 3D detection.

 In this paper, the authors introduce a new loss function for classification problems. To be specific, the authors introduce Taylor expansion of cross-entropy loss + focal loss and show that various subsets of this expansion can improve the models on image classification, 2D object detection and 3D object detection problems.  I've reviewed a previous version of this paper at NeurIPS2021. Compared to the previous version, I see that the authors improved the paper significantly, mainly in terms of structure and presentation.