PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions

1 INTRODUCTION . Loss functions are important in training neural networks . In principle , a loss function could be any ( differentiable ) function that maps predictions and labels to a scalar . Therefore , designing a good loss function is generally challenging due to its large design space , and designing a universal loss function that works across different tasks and datasets is even more challenging : for example , L1 / L2 losses are commonly used for regression tasks , but they are rarely used for classification tasks ; focal loss is often used to alleviate the overfitting issue of cross-entropy loss for imbalanced object detection datasets ( Lin et al. , 2017 ) , but it is not shown to consistently help other tasks . Many recent works have also explored new loss functions via meta-learning , ensembling or compositing different losses ( Hajiabadi et al. , 2017 ; Xu et al. , 2018 ; Gonzalez & Miikkulainen , 2020b ; a ; Li et al. , 2019 ) . In this paper , we propose PolyLoss : a novel framework for understanding and designing loss functions . Our key insight is to decompose commonly used classification loss functions , such as crossentropy loss and focal loss , into a series of weighted polynomial bases . They are decomposed in the form of ∑∞ j=1 αj ( 1 − Pt ) j , where αj ∈ R+ is the polynomial coefficient and Pt is the prediction probability of the target class label . Each polynomial base ( 1−Pt ) j is weighted by a corresponding polynomial coefficient αj , which enables us to easily adjust the importance of different bases for different applications . When αj = 1/j for all j , our PolyLoss becomes equivalent to the commonly used cross-entropy loss , but this coefficient assignment may not be optimal . Our study shows that , in order to achieve better results , it is necessary to adjust polynomial coefficients αj for different tasks and datasets . Since it is impossible to adjust an infinite number of αj , we explore various strategies with a small degree of freedom . Perhaps surprisingly , we observe that simply adjusting the single polynomial coefficient for the leading polynomial , which we denote LPoly-1 , is sufficient to achieve significant improvements over the commonly used cross-entropy loss and focal loss . Overall , our contribution can be summarized as : • Insights on common losses : We propose a unified framework , named PolyLoss , to rethink and redesign loss functions . This framework helps to explain cross-entropy loss and focal loss as two special cases of the PolyLoss family ( by horizontally shifting polynomial coefficients ) , which was not recognized before . This new finding motivates us to investigate new loss functions that vertically adjust polynomial coefficients , shown in Figure 1 . • New loss formulation : We evaluate different ways of vertically manipulating polynomial coefficients to simplify the hyperparameters search space . We propose a simple and effective Poly-1 loss formulation which only introduces one hyperparameter and one line of code . • New findings : We identify that focal loss , though effective for many detection tasks , is suboptimal for the imbalanced ImageNet-21K . We find the leading polynomial contributes to a large portion of the gradient during training , and its coefficient correlates to the prediction confidence Pt . In addition , we provide an intuitive explanation on how to leverage this correlation to design good PolyLoss tailored to imbalanced datasets . • Extensive experiments : We evaluate our PolyLoss on different tasks , models , and datasets . Results show PolyLoss consistently improves the performance on all fronts , summarized in Table 1 , which includes the state-of-the-art classifiers EfficientNetV2 and detectors RSN . 2 RELATED WORK . Cross-entropy loss is used in many popular and current state-of-the-art models for perception tasks such as classification , detection and semantic segmentation ( Tan & Le , 2021 ; He et al. , 2017 ; Zoph et al. , 2020 ; Tao et al. , 2020 ) . Instead of trying to improve cross-entropy loss for specific tasks like many other prior works , the goal of this paper is to provide a unified framework for systematically designing a better classification loss function . Loss for class imbalance Training detection models , especially single-stage detectors , is difficult due to class imbalance . Common approaches such as hard example mining and reweighing are developed to address the class imbalance issue ( Sung , 1996 ; Viola & Jones , 2001 ; Felzenszwalb et al. , 2010 ; Shrivastava et al. , 2016 ; Liu et al. , 2016 ; Bulo et al. , 2017 ) . As one of these approaches , focal loss is designed to mitigate the class imbalance issue by focusing on the hard examples and is used to train state-of-the-art 2D and 3D detectors ( Lin et al. , 2017 ; Tan et al. , 2020 ; Du et al. , 2020 ; Shi et al. , 2020 ; Sun et al. , 2021 ) . In our work , we found that focal loss is suboptimal for the imbalanced ImageNet-21K . Using the PolyLoss framework , we discover a better loss function , which performs the opposite role of focal loss . We further provide intuitive understanding of why it is important to design different loss functions tailored to different imbalanced datasets using the PolyLoss framework . Robust loss to label noise Another direction of research is to design loss functions that are robust to label noise ( Ghosh et al. , 2015 ; 2017 ; Zhang & Sabuncu , 2018 ; Wang et al. , 2019 ; Oksuz et al. , 2020 ; Menon et al. , 2020 ) . A commonly used approach is to incorporate noise robust loss function such as Mean Absolute Error ( MAE ) into cross-entropy loss . In particular , Taylor cross entropy loss is proposed to unify MAE and cross-entropy loss by expanding the cross-entropy loss in ( 1 − Pt ) j polynomial bases ( Feng et al. , 2020 ) . By truncating the higher-order polynomials , they show truncated cross-entropy loss function is closer to MAE , which is more robust to label noise on datasets with synthetic label noise . In contrast , our PolyLoss provides a more general framework to design loss functions for different datasets by manipulating polynomial coefficients , which includes dropping higher-order polynomials proposed in ( Feng et al. , 2020 ) . Our experiments in subsection 4.1 show the loss proposed in ( Feng et al. , 2020 ) performs worse than cross-entropy loss on the clean ImageNet dataset . Learned loss functions Several recent works demonstrate learning the loss function during training via gradient descent or meta learning ( Hajiabadi et al. , 2017 ; Xu et al. , 2018 ; Gonzalez & Miikkulainen , 2020a ; Li et al. , 2019 ; 2020 ) . Notably , TaylorGLO utilizes CMA-ES to optimize multivariate Taylor parameterization of a loss function and learning rate schedule during training ( Hansen & Ostermeier , 1996 ; Gonzalez & Miikkulainen , 2020b ) . Due to the search space scale with the order of polynomials , the paper demonstrates that using the third-order parameterization ( 8 parameters ) , the learned loss function schedule outperforms cross-entropy loss on 10-class classification problems . Our paper Figure 2a , on the other hand , shows for 1000-class classification tasks , hundreds of polynomials are needed . This results in a prohibitively large search space . Our proposed Poly-1 formulation mitigates the challenge of the large search space and do not rely on advanced black-box optimization algorithms . Instead , we show a simple grid search over one hyperparameter can lead to significant improvement on all tasks that we investigate . 3 POLYLOSS . PolyLoss provides a framework for understanding and improving the commonly used cross-entropy loss and focal loss , visualized in Figure 1 . It is inspired from the Taylor expansion of cross-entropy loss ( Equation 1 ) and focal loss ( Equation 2 ) in the bases of ( 1− Pt ) j : LCE = − log ( Pt ) = ∞∑ j=1 1/j ( 1− Pt ) j = ( 1− Pt ) + 1/2 ( 1− Pt ) 2 ... ( 1 ) LFL = − ( 1− Pt ) γ log ( Pt ) = ∞∑ j=1 1/j ( 1− Pt ) j+γ = ( 1− Pt ) 1+γ + 1/2 ( 1− Pt ) 2+γ ... ( 2 ) where Pt is the model ’ s prediction probability of the target ground-truth class . Cross-entropy loss as PolyLoss Using the gradient descent method to optimize the cross-entropy loss requires taking the gradient with respect to Pt . In the PolyLoss framework , an interesting observation is that the coefficients 1/j exactly cancel the jth power of the polynomial bases , see Equation 1 . Thus , the gradient of cross-entropy loss is simply the sum of polynomials ( 1 − Pt ) j , shown in Equation 3 . −dLCE dPt = ∞∑ j=1 ( 1− Pt ) j−1 = 1 + ( 1− Pt ) + ( 1− Pt ) 2 ... ( 3 ) The polynomial terms in the gradient expansion capture different sensitivity with respect to Pt . The leading gradient term is 1 , which provides a constant gradient regardless of the value of Pt . On the contrary , when j 1 , the jth gradient term is strongly suppressed when Pt gets closer to 1 . Focal loss as PolyLoss In the PolyLoss framework , Equation 2 , it is apparent that the focal loss simply shifts the power j by the power of a modulating factor γ . This is equivalent to horizontally shifting all the polynomial coefficients by γ as shown in Figure 1 . To understand the focal loss from a gradient prospective , we take the gradient of the focal loss ( Equation 2 ) with respect to Pt : −dLFL dPt = ∞∑ j=1 ( 1 + γ/j ) ( 1− Pt ) j+γ−1 = ( 1 + γ ) ( 1− Pt ) γ + ( 1 + γ/2 ) ( 1− Pt ) 1+γ ... ( 4 ) For a positive γ , the gradient of focal loss drops the constant leading gradient term , 1 , in the crossentropy loss , see Equation 3 . As discussed in the previous paragraph , this constant gradient term causes the model to emphasize the majority class , since its gradient is simply the total number of examples for each class . By shifting the power of all the polynomial terms by γ , the first term then becomes ( 1 − Pt ) γ , which is suppressed by the power of γ to avoid overfitting to the already confident ( meaning Pt close to 1 ) majority class . More details shown in Appendix F. Connection to regression and general form Representing the loss function in the PolyLoss framework provides an intuitive connection to regression . For classification tasks where y = 1 is the effective probability of the ground-truth label , the polynomial bases ( 1 − Pt ) j can be expressed as ( y − Pt ) j . Thus both cross-entropy loss and focal loss can be interpreted as a weighted ensemble of distances between the prediction and label to the jth power . However , a fundamental question in those losses : Are the coefficients in front of the regression terms optimal ? In general , PolyLoss is a monotone decreasing function1 on [ 0 , 1 ] which can be expressed as∑∞ j=1 αj ( 1 − Pt ) j , which provide a flexible framework to adjust each coefficient2 . PolyLoss can be generalized to non-integer j , but for simplicity we only focus on integer power ( j ∈ Z+ ) in this paper . In the next section , we investigate several strategies on designing better loss functions in the PolyLoss framework via manipulating αj .

Summary:  In this paper, the authors introduce a new loss function for classification problems. To be specific, the authors introduce Taylor expansion of cross-entropy loss + focal loss and show that various subsets of this expansion can improve the models on image classification, 2D object detection and 3D object detection problems.  I've reviewed a previous version of this paper at NeurIPS2021. Compared to the previous version, I see that the authors improved the paper significantly, mainly in terms of structure and presentation.