Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search

1 INTRODUCTION . The computer vision community has witnessed substantial progress in object detection in recent years . The advances for the architecture design , e.g . two-stage detectors ( Ren et al. , 2015 ; Cai & Vasconcelos , 2018 ) and one-stage detectors ( Lin et al. , 2017b ; Tian et al. , 2019 ) , have remarkably ∗Equal Contribution . Work done when the first author ( Peidong Liu ) interns at Huawei Noah ’ s Ark Lab . †Correspondence to : Xiaodan Liang ( xdliang328 @ gmail.com ) , Yong Jiang ( jiangy @ sz.tsinghua.edu.cn ) . pushed forward the state of the art . The success can not be separated from the sophisticated design for training objective , i.e . loss function . Traditionally , two-stage detectors equip the combination of Cross-entropy loss ( CE ) and L1 loss/Smooth L1 loss ( Girshick , 2015 ) for bounding box classification and regression respectively . In contrast , one-stage detectors , suffering from the severe positive-negative sample imbalance due to dense sampling of possible object locations , introduce Focal loss ( FL ) ( Lin et al. , 2017b ) to alleviate the imbalance issue . However , optimizing object detectors with traditional hand-crafted loss functions may lead to sub-optimal solutions due to the limited connection with the evaluation metric ( e.g . AP ) . Therefore , IoU-Net ( Jiang et al. , 2018 ) proposes to jointly predict Intersection over Union ( IoU ) during training . IoU loss series , including IoU loss ( Yu et al. , 2016 ) , Bounded IoU loss ( Tychsen-Smith & Petersson , 2018 ) , Generalized IoU loss ( GIoU ) ( Rezatofighi et al. , 2019 ) , Distance IoU loss ( DIoU ) , and Complete IoU loss ( CIoU ) ( Zheng et al. , 2020 ) , optimize IoU between predicted and target directly . These works manifest the necessity of developing effective loss functions towards better alignment with evaluation metric for object detection , while they heavily rely on careful design and expertise experience . In this work , we aim to discover novel loss functions for object detection automatically to reduce human burden , inspired by the recent progress in network architecture search ( NAS ) and automated machine learning ( AutoML ) ( Cai et al. , 2019 ; Liu et al. , 2020a ) . Different from Wang et al . ( 2020 ) and Li et al . ( 2019b ) that only search for particular hyper-parameters within the fixed loss formula , we steer towards finding new forms of the loss function . Notably , AutoML-Zero ( Real et al. , 2020 ) proposes a framework to construct ML algorithm from simple mathematical operations , which motivates us to design loss functions from primitive mathematical operations with evolutionary algorithm . However , it encounters a severe issue that a slight variation of operations would lead to a huge performance drop , which is attributed to the sparse action space . Therefore , we propose a novel Convergence-Simulation driven Evolutionary search algorithm , named CSE-Autoloss , to alleviate the sparsity issue . Benefit from the flexibility and effectiveness of the search space , as Figure 1 shows , CSE-Autoloss discovers distinct loss formulas with comparable performance with the Crossentropy loss , such as ( b ) and ( c ) in the figure . Moreover , the best-searched loss function ( d ) , named CSE-Autoloss-Acls , outperformed CE loss by a large margin . Specifically , to get preferable loss functions , CSE-Autoloss contains a well-designed search space , including 20 primitive mathematical operations , 3 constant-type inputs , and 3 variable-type inputs , which can cover a wide range of existing popular hand-crafted loss functions . Besides , to tackle the sparsity issue , CSE-Autoloss puts forward progressive convergence-simulation modules , which verify the evolved loss functions from two aspects , including mathematical convergence property and optimization behavior , facilitating the efficiency of the vanilla evolution algorithm for loss function search without compromising the accuracy of the best-discovered loss . For different types of detector , CSE-Autoloss is capable of designing appropriate loss functions automatically without onerous human works . In summary , our main contributions are as follows : 1 ) We put forward CSE-Autoloss , an end-to-end pipeline , which makes the first study to search insightful loss functions towards aligning the evaluation metric for object detection . 2 ) A well-designed search space , consisting of various primitive operations and inputs , is proposed as the foundation of the search algorithm to explore novel loss functions . 3 ) To handle the inefficiency issue caused by the sparsity of action space , innovative convergence-simulation modules are raised , which significantly reduces the search overhead with promising discovered loss functions . 4 ) Extensive evaluation experiments on various detectors and different detection benchmarks including COCO , VOC2017 , and BDD , demonstrate the effectiveness and generalization of both CSE-Autoloss and the discovered loss functions . 2 RELATED WORK . Loss Functions in Object Detection . In object detection frameworks , CE loss dominates the twostage detectors ( Ren et al. , 2015 ; Cai & Vasconcelos , 2018 ) for classification purpose , while Focal loss ( Lin et al. , 2017b ) and GHM loss ( Li et al. , 2019a ) are widely equipped in one-stage detectors ( Lin et al. , 2017b ; Tian et al. , 2019 ) for solving imbalance issues between positive and negative samples . Chen et al . ( 2019b ) ; Qian et al . ( 2020 ) attempt to handle the sample-imbalance problem from the ranking perspective . However , these works are sensitive to hyper-parameters hence they can not generalize well . To further improve classification and localization quality , Jiang et al . ( 2018 ) ; Tian et al . ( 2019 ) ; Wu et al . ( 2020 ) introduce quality estimation , and GFocal ( Li et al. , 2020 ) unifies the quality estimation with classification towards consistent prediction . With regard to regression loss functions , Smooth L1 Loss ( Girshick , 2015 ) has been commonly performed in the past years until IoU-based loss series ( Yu et al. , 2016 ; Rezatofighi et al. , 2019 ; Zheng et al. , 2020 ) gradually occupy regression branch for their effectiveness in bounding box distance representation due to the direct optimization of the evaluation metric . However , these works rely on expert experience on loss formula construction , which limits the development of the loss function design in object detection . Automated Loss Function Design . By introducing a unified formulation of loss function , Li et al . ( 2019b ) ; Wang et al . ( 2020 ) raise automated techniques to adjust loss functions for face recognition . However , they only search for specific hyper-parameters in the fixed loss formulas , which limit the search space and fail to discover new forms of loss functions . Real et al . ( 2020 ) ; Liu et al . ( 2020b ) ; Ramachandran et al . ( 2018 ) attempt to design formulations from basic mathematical operations to construct building blocks in machine learning such as normalization layers and activation functions . However , these works can not directly apply to loss function search since their search space and search strategies are specialized . Gonzalez & Miikkulainen ( 2020 ) proposes a framework for searching classification loss functions but the searched loss poorly generalizes to large-scale datasets . Besides , all these works are not suitable for the challenging object detection task due to the sparsity of the action space and the heavy evaluation cost for training object detectors . Instead , in this work , we make the first attempt to search for loss function formulations directly on the largescale detection dataset via an effective pipeline with novel convergence-simulation modules . 3 METHOD . In this section , we present the CSE-Autoloss pipeline for discovering novel loss functions towards aligning the evaluation metric for object detection . We first introduce the well-designed search space in Section 3.1 . The detailed CSE-Autoloss pipeline is elaborated in Section 3.2 . 3.1 SEARCH SPACE DESIGN . Input Nodes . In object detection , default classification losses ( i.e . CE loss and FL loss ) , take prediction and label as inputs . Inspired by GFocal loss ( Li et al. , 2020 ) , to better motivate the loss to align with the evaluation metric ( i.e . AP ) , we introduce IoU between ground truth and prediction into the loss formula , where prediction , label , IoU are notated as x , y , w respectively . For the regression branch , to cover the mainstream IoU loss series ( i.e . IoU and GIoU ) , we take the intersection i , union u , and enclosing area e between the predicted and target box as input tensors . Besides , 3 constant-type inputs ( i.e . 1 , 2 , 3 ) are brought in to improve the flexibility of the search space . As mentioned above , for consistent prediction of localization quality and classification score between training and testing , we simply introduce IoU into the CE and FL loss to get two nuanced loss variants , namely CEI and FLI respectively : CEI ( x , y , w ) = −dot ( wy , log ( softmax ( x ) ) ) , FLI ( x′ , y′ , w ) = −w [ y′ ( 1− σ ( x′ ) ) log σ ( x′ ) + ( 1− y′ ) σ ( x′ ) log ( 1− σ ( x′ ) ) ] , where x , y ∈ Rc+1 , x′ ∈ R , c is the number of class . x and x′ are the prediction , y is a one-hot vector of ground truth class , y′ ∈ { 0 , 1 } is the binary target , and w ∈ R is the IoU value between ground truth and prediction . As Table 4 shows , CEI and FLI outperform CE and FL by a small margin in terms of AP , which verifies the effectiveness of introducing IoU into the classification branch . Note that we apply CEI and FLI loss as the predefined initial loss LI in the search experiment for two-stage detectors and one-stage detectors , respectively . Primitive Operators . The primitive operators , including element-wise and aggregation functions that enable interactions between tensors with different dimensions , are listed as below : • Unary functions : −x , ex , log ( x ) , |x| , √ x , softmax ( x ) , softplus ( x ) , σ ( x ) , gd ( x ) , alf ( x ) , erf ( x ) , erfc ( x ) , tanh ( x ) , relu ( x ) , sin ( x ) , cos ( x ) • Binary functions : x1 + x2 , x1 − x2 , x1 × x2 , x1x2+ , dot ( x1 , x2 ) is a small value to avoid zero division , softplus ( x ) = ln ( 1 + ex ) , σ ( x ) = 11+e−x is the sigmoid function . To enlarge the search space , more S-type curves and their variants∗ are included , i.e . gd ( x ) = 2 arctan ( tanh ( x2 ) ) , alf ( x ) = x√ 1+x2 , erf ( x ) = 2√ π ∫ x 0 e−t 2 dt is the error function , erfc ( x ) = 1 − erf ( x ) . For binary operations , broadcasting is introduced to ensure the assembling of inputs with different dimensions . dot ( x1 , x2 ) is the class-wise dot product of two matrix x1 , x2 ∈ RB×C , where B and C indicate batch size and number of categories respectively . In practice , regression branch contains fewer primitive operators than classification branch because introducing more primitive functions for regression branch does not bring performance gain . ∗In the implementation , the output of these functions are re-scaled to obtain the same range as sigmoid function , i.e . from 0 to 1 . Loss Function as a Computation Graph . As Figure 1 illustrates , we represent the loss function as a directed acyclic graph ( DAG ) computation graph that transforms input nodes into the scalar output o ( i.e . loss value ) with multiple primitive operators in the intermediate . Sparsity of the Action Space . We perform random search for Faster R-CNN classification branch on COCO and find only one acceptable loss every 105 , which indicates the sparsity of the action space and the challenge for searching the loss function formulations .

Summary: This paper proposes to use an evolutionary search algorithm to search for better loss functions for the classification and regression branch of an object detector. The algorithm starts with 20 primitive mathematical operations. Due to the highly sparse action space, the vanilla evolutionary algorithm would take a long time to converge. Then the authors propose two ways to reduce the search space. First, they filter out loss functions which generates gradients of large magnitude to well-classified samples and do not converge to zero. Second, they construct a very small dataset by sampling only one image randomly from each category and evaluate the loss function on it to quickly filter out bad loss candidates.