Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search
Authors: Peidong Liu
Gengwei Zhang
Bochao Wang
Hang Xu
Xiaodan Liang
Yong Jiang
Zhenguo Li
Venue: ICLR 2021 Poster (2021)
Keywords: Object detection
AutoML
Evolutionary algorithm
Loss function search

Abstract: Designing proper loss functions for vision tasks has been a long-standing research direction to advance the capability of existing models. For object detection, the well-established classification and regression loss functions have been carefully designed by considering diverse learning challenges (e.g. class imbalance, hard negative samples, and scale variances). Inspired by the recent progress in network architecture search, it is interesting to explore the possibility of discovering new loss function formulations via directly searching the primitive operation combinations. So that the learned losses not only fit for diverse object detection challenges to alleviate huge human efforts, but also have better alignment with evaluation metric and good mathematical convergence property. Beyond the previous auto-loss works on face recognition and image classification, our work makes the first attempt to discover new loss functions for the challenging object detection from primitive operation levels and finds the searched losses are insightful. We propose an effective convergence-simulation driven evolutionary search algorithm, called CSE-Autoloss, for speeding up the search progress by regularizing the mathematical rationality of loss candidates via two progressive convergence simulation modules: convergence property verification and model optimization simulation. CSE-Autoloss involves the search space (i.e. 21 mathematical operators, 3 constant-type inputs, and 3 variable-type inputs) that cover a wide range of the possible variants of existing losses and discovers best-searched loss function combination within a short time (around 1.5 wall-clock days with 20x speedup in comparison to the vanilla evolutionary algorithm). We conduct extensive evaluations of loss function search on popular detectors and validate the good generalization capability of searched losses across diverse architectures and various datasets. Our experiments show that the best-discovered loss function combinations outperform default combinations (Cross-entropy/Focal loss for classification and L1 loss for regression) by 1.1% and 0.8% in terms of mAP for two-stage and one-stage detectors on COCO respectively. Our searched losses are available at https://github.com/PerdonLiu/CSE-Autoloss.

Published as a conference paper at ICLR 2021
Loss FUNCTION DISCOVERY FOR OBJECT DE- TECTION VIA CONVERGENCE--SMULLTTON DRIVEN
SEARCH
Peidong Liu', Gengwei Zhang", Bochao Wang', Hang Xu²
Xiaodan Liang", Yong Jiang', Zhenguo Li
'Tsinguua Shenzhen International Graduate School, Tsinghua University
SSnn Yat-Sen University
Huawei Noah's Ark Lab
1 pd 90ma il wwl.ssssiinaaaeuu.....u
f 1ggddvvid, sergeywong, enromexbjxb, xdliangg388egmaiil com cerryyezz.tinngua.ccom
1i. w...hnnuuuhuawci... com
ABSTRACT
Designing proper loss functions for vision tasks has been a long-standing research direction to advance the capability of existing models. For object detection, the well-established classification and regression loss functions have been carefully designed by considering diverse learning challenges (e.g. class imbalance, hard negative samples, and scale variances). Inspired by the recent progress in network architecture search, it is interesting to explore the possibility of discovering new loss function formulations via directly searching the primitive operation combi- nations. So that the learned losses not only fit for diverse object detection chal- lenges alleviate huge human efforts, but also have better alignment with evalu- ation metric and good mathematical convergence property. Beyond the previous auto-loss works on face recognition and image classification, our work makes the first attempt to discover new loss functions for the challenging object detec- tion from primitive operation levels and finds the searched losses are insightful. We propose an effective rithm, called CSE-Autoloss, for speeding up the search progress by regularizing the mathematical rationality of loss candidates via two progressive convergence simulation modules: convergence property verification and model optimization simulation. CSE-Autoloss involves the search space (i.c. 21 mathematical opera- tors, 3 constant-type inputs, and variable-type inputs) that cover a wide range the possible variants of existing losses and discovers best-searched loss function combination within a short time (around 1.5 wall-clock days with 20x speedup in comparison the vanilla evolutionary algorithm). We conduct extensive eval- uations of loss function search on popular detectors and validate the good gen- eralization capability of searched losses across diverse architectures and various datasets. Our experiments show that the best-discovered loss function combina- tions outperform default combinations (Cross-entropy/Foer) loss for classification and I.1 loss for regression) by 1.1 % and 0.8% in terms of mAP for two-stage and one-stage detectors on COCO respectively. Our searched losses are available http:///iiiiaaacooooeeeeroo/o//s Autoloss
driven evolutionary search algo-
INTRODUCTION
The computer vision community has witnessed substantial progress in object detection in recent years. The advances for the architecture design, e.g. two-stage detectors (Ren et al., [015]] Cai & Vasconcelos 2018) and one-stage detectors (Lin et al. 2017b Tian al. 2019), have remarkably
"Equal Contribution. Work done when the first author (Peidong Liu) interns at Huawei Noah's Ark Lab. 1 "Coresponndecce Xiaodan Liang (xdliang328 (esillmmmmlmmmm Yong Jiang (jiangy @ www.ssinguaaedu.crr

---

Published as a conference paper at ICLR 2021
Variable
Operation
x
x
W
y
w
softmax
softmax erfc
add
softmax sin
log
log
Neg
softmax
log
mul
add
neg
sin
log
neg
dot
dot
erfc
neg
mul
mul
dot
dot
AP: 37.6%
AP: 37.7%
AP: 37.8% (c) Candidate2
AP: 38.2% (d) CSE-Autoloss-Ag
(a) Cross-entropy loss
(b) Candidate1
Figure I:
Computation graphs of loss function examples: (a) Cross-entropy loss; (b, c) Searched loss candidates with comparable performance to Cross-entropy loss; (d) Searched best-performed loss named Autolosoloss-Als
pushed forward the state of the art. The success cannot be separated from the sophisticated design for training objective, i.c. loss function.
Traditionally, two-stage detectors equip the combination of Cross-entrops loss (CE) and LI loss/Smooth LI loss (Girshick] 2013) for bounding box classification and regression respectively. In contrast, one-stage detectors, suffering from the severe positive-negative negative sample imbalance due to dense sampling of possible object locations, introduce Focal loss (FL) (Lin et al.) (0077)) to alle- viate the imbalance issue. However, optimizing object detectors with traditional hand-crafted loss functions may lead to sub-optimal solutions due to the limited connection with the evaluation met- ric (e.g. AP). Therefore, IoU-Net (Jiang al.] (2018) proposes to jointly predict Intersection over Union (IoU) during training. IoU loss series, including IoU loss (Yu et al.) 2016)), Bounded IoU loss Tychhse-Smmth
Petersson, 2018), Generalized loU loss (GloU) (Rezatofighi et al. 2019). Distance IoU loss (DloU), and Complete loU loss (CloU) (Zheng et al.) 2020), optimize loU be- tween predicted and target directly. These works manifest the necessity developing effective loss functions towards better alignment with evaluation metric for object detection, while they heavily rely on careful design and expertise experience.
In this work, we aim to discover novel loss functions for object detection automatically reduce human burden, inspired by the recent progress in network architecture search (NAS) and automated machine learning (AutoML) (Cai al. 2019 Liu et al., 2020a). Different from Wang et al. (2020) and Li et al. (2019b) that only search for particular hyper-parameters within the fixed loss formula, steer towards finding forms of the loss function. Notably, AutoML-Zero (Real et al.) 2020)
proposes framework to construct ML algorithm from simple mathematical operations, which mo- tivates us design loss functions from primitive mathematical operations with evolutionary algo- rithm. However, it encounters a severe issue that a slight variation of operations would lead to huge performance drop, which is attributed the sparse action space. Therefore, we propose novel Convergence- BnnvereeneeSSimulatoon driven Evolutionary search algorithm, named CSE-Autoloss, to alle- viate the sparsity issue. Benefit from the flexibility and effectiveness of the search space, Figure [ shows, CSE-Autoloss discovers distinct loss formulas with comparable performance with the Cross- entropy loss, such as (b) and (c) the figure. Moreover, the best-searched loss function (d), named CSE-Autoloss-Aaa outperformed CE loss by large margin. Specifically, to get preferable loss functions, CSE-Autoloss contains a well-designed search space, including 20 primitive mathemat- ical operations, 3 constant-type inputs, and 3 variable-type inputs, which can cover wide range of existing popular hand-crafted loss functions. Besides, to tackle the sparsity issue, CSE-Autoloss puts forward progressive convergence-simulati modules, which verify the evolved loss functions
2

---

Published as a conference paper at ICLR 2021
from two aspects, including mathematical convergence property and optimization behavior, facilitat- ing the efficiency of the vanilla evolution algorithm for loss function search without compromising the accuracy of the best-discovered loss. For different types of detector, CSE-Autoloss is capable of designing appropriate loss functions automatically without onerous human works.
In summary, our main contributions are as follows: 1) We put forward CSE-Autoloss, an end-to-end pipeline, which makes the first study to search insightful loss functions towards aligning the evalu- ation metric for object detection. 2) A well-designed search space, consisting of various primitive operations and inputs, is proposed as the foundation of the search algorithm to explore novel loss functions. 3) To handle the inefficiency issue caused by the sparsity of action space, innovative simmulaion modules are raised, which significantly reduces the search overhead with promising discovered loss functions. 4) Extensive evaluation experiments on various detectors and different detection benchmarks including COCO, VOC2017, and BDD, demonstrate the effective- ness and generalization of both CSE-Autoloss and the discovered loss functions.
2 RELATED WORK
Loss Functions in Object Detection. In object detection frameworks, CE loss dominates the two- stage detectors (Ren et al. 2015 Cai & Vasconcelos] 2018) for classification purpose, while Focal loss (Lin et al. 2017b) and GHM loss (Li et al., 20.
2019a) 9a) are widely equipped in one-stage detec- tors (Lin et al., 2017b Tian et al. 2019) for solving imbalance issues between positive and negative samples. Chen et al. (201 2b); Qian et al. (2020) attempt to handle the sample- immalaee problem from the ranking perspective. However, these works are sensitive to aper-parameters hence they cannot generalize well. To further improve classification and localization quality, Diang al. (2018); Tian et al. (2019); [Wu et al. (2020) introduce quality estimation, and GFocal (Li et al. (2220) unifies the quality estimation with classification towards consistent prediction. With regard to regression loss functions, Smooth L1 Loss (Girshick, 2015) has been commonly performed in the past years until IoU-based loss series YYu et al.) 2016 Rezatofighi et al., 2019, Zheng et al.) 2020) gradually occupy regression branch for their effectiveness in bounding box distance representation due to the direct optimization of the evaluation metric. However, these works rely on expert experience on loss formula construction, which limits the development of the loss function design in object detection.
Automated Loss Function Design. By introducing a unified formulation of loss function, Li et al.) (2019)); Wang et al.] (2020) raise automated techniques to adjust loss functions for face recogni- tion. However, they only search for specific hyper-parameters in the fixed loss formulas, which limit the search space and fail to discover new forms of loss functions. Real al. 2020 Liu et al. (2020b): Raaachannran et al. (2018) attempt to design formulations from basic mathematical oper- ations construct building blocks in machine learning such as normalization layers and activation functions. However, these works cannot directly apply to loss function search since their search space and search strategies are specialized. Gonzalez & Miikkulainen (2020) proposes a framework for searching classification loss functions but the searched loss poorly generalizes to large-scale datasets. Besides, all these works are not suitable for the challenging object detection task due to the sparsity of the action space and the heavy evaluation cost for training object detectors. Instead, in this work, we make the first attempt search for loss function formulations directly on the large- scale detection dataset via an effective pipeline with novel convergence-simulatin modules.
METHOD
In this section, we present the CSE-Autoloss pipeline for discovering novel loss functions towards aligning the evaluation metric for object detection. We first introduce the well-designed search space in Section 3.1 The detailed CSE-Autoloss pipeline is elaborated in Section 3.2
3.1 SEARCH SPACE DESIGN
Input Nodes. In object detection, default classification losses (i.e. CE loss and FL loss), take prediction and label as inputs. Inspired by GFocal loss eLi al. 2020). better motivate the loss to align with the evaluation metric (i.e. AP), we introduce LoU between ground truth and prediction into the loss formula, where prediction, label, IoU are notated as x. y, w respectively. For the regression
3

---

Published as a conference paper at ICLR 2021
Conctant
thrsshole
Varseble
Isg
segioole
Povlive dass score
m
Convergence property verification
Model optimization simulation
Loss function
Search
Loss function population
evaluation
Parent,
Parent,
Offspring: Offsprings
Offspring,
Evolution (offspring generation)
Figure 2:
An overview of the proposed CSE-Autoloss pipeline. Our CSE-AutoLove first generates large amount of candidate loss functions by assembling formulations based on the well-designed search space. Then CSE-Autoloss filters out more than 99% divergent, Taiiing-uuffrien performed loss candidates by evaluating them with the proposed convergence-simee simuulltionn modules, iec. convergence property verification and model optimization simulation, which verifies mathemat- ical property and optimization quality. After that, K top-performing loss functions are selected to evaluate on the proxy task to further obtain top-P loss candidates parents to derive offspring for the next generation. Please see more details about CSE-Autoloss in Algorithm
or poor-
branch, cover the mainstream IoU loss series (i.e. IoU and GloU), we take the intersection i, union u, and enclosing area e between the predicted and target box input tensors. Besides, constant-type inputs (i.e. 2, 3) are brought in improve the flexibility of the search space.
As mentioned above, for consistent prediction of localization quality and classification score be- tween training and testing, we simply introduce loU into the CE and FL loss to get two nuanced loss variants, namely CEI and FLI respectively:
CEI(r, y, w) = dot dot(ww, lo(ssotmxx(()x))))
FLI(2', v/ () wly o logo
(oo))ooo)) o
where x, E RC+1, E R. is the number of class. x and x' are the prediction, y is one-hot vector of ground truth class, y {0, is the binary target, and R is the IoU value between ground truth and prediction. As Table 4 shows, CEI and FL.I outperform CE and FI. by small margin in terms of AP, which verifies the effectiveness of introducing IoU into the classification branch. Note that we apply CEI and FLI loss as the predefined initial loss L, in the search experiment for two-stage detectors and one-stage detectors, respectively.
Primitive Operators. The primitive operators, including element-wise and aggregation functions that enable interactions between tensors with different dimensions, are listed as below:
Unary functions: -x, e", log(z), |z], vz, softmax(r), softplus(z), a(x), gd(x). alf(x), er((r), erf((x), tanh(x). relu(r)) sin(x), cos(x)
Binary functions: I1 + F2, 21 - 22, F1 x 22, mgte. dot(x1,
is a small value avoid zero division, softplus(x) moid function. To enlarge the search space, more S-type curves and their variants are included, ie. gd(x) arctanntanh((
In(1 + e*), (xx) 1to-o is the sig-
alf(x) = vT+E. crf(x) = / Ie e-t" dt the error function, erfo(x) = 1 - (r:) For binary operations. broadcasting is introduced to ensure the assem- bling of inputs with different dimensions. dot 22) the class-wise dot product of two matrix 11.22 E RBxC, where B and C' indicate batch size and number of categories respectively. In practice, regression branch contains fewer primitive operators than classification branch because introducing more primitive functions for regression branch does not bring performance gain.
"In the implementation, the output of these functions re-scaled to obtain the same range sigmoid function, i.c. from 0
4

---

Published as a conference paper at ICLR 2021
Loss Function as a Computation Graph. As Figure jillustrates, represent the loss function as a directed acyclic graph (DAG) computation graph that transforms input nodes into the scalar output (i.e. loss value) with multiple primitive operators in the intermediate.
Sparsity of the Action Space. We perform random search for Faster R-CNN classification branch on COCO and find only acceptable loss every 105, which indicates the sparsity of the action space and the challenge for searching the loss function formulations.
3.2 CSE- AUTOLOSS PIPELINE
We propose novel CSE-Autoloss pipeline with progressive convergence-simulatin modules, con- sisting of two perspectives: 1) convergence property verification module verifies the mathematical property of the loss candidates. 2) model optimization simulation module estimates the optimiza- tion quality to further filter out divergent, training-unfrienden
and poor-performed loss functions, An overview of CSE-Autoloss is illustrated in Figure and the details are shown in Algorithm [] Our CSE-Autoloss achieves more than 20x speedup in comparison to the vanilla evolutionary algorithm.
3.2.1
EVOLUTIONARY ALGORITHM
Here we give a short overview of vanilla evolutionary algorithm, which refers to the traditional tournament selection (Goldberg & Deb) 1991]. We first generate N loss functions from predefined initial loss Cy as the first generation, where loss function indicates an individual in the population. Then these individuals perform cross-over randomly pair by pair with probability pı and conduct mutation independently with probability p2- We evaluate all the individuals on the proxy task and select top-P of them serve parents for the next generation. The above process is repeated for E generations to get the best-searched loss.
The bottleneck of the evolutionary process lies at tons of loss evaluations the proxy due to the highly sparse action space, and it costs approximately half an hour for each evaluation. How to estimate the quality of individuals in the early stage is crucial for efficiency. Therefore, we apply computational validness check for the population. Specifically, invalid values (i.e. NaN, +o0, -co) are not allowed. This simple verification accelerates the search process by 10x. But a great number of divergent, training- training-unfrient unfriendly. and poor-performed loss functions still remain, which yet require a great amount of computation cost.
3.2.2 CONVERGENCE PROPERTY VERIFICATION
As stated above, the vanilla evolution search suffers from inefficiency issues due to the high sparsity of the action space and the unawareness of loss mathematical property. Therefore, we introduce the convergence property verification module into the evolutionary algorithm, which filters out either non-monotonic or non-convergent loss candidates within a short time.
Classification Loss Function We analyze the property of cxisting popular classification loss func- tion (i.e. CE loss). For simplicity, we only consider binary classification scenario, which is known binary Cross-entropy loss (BCE):
@BCE(2) Or
BCE(*) IIn( Ite-
I
where æ indicates positive class score. Further analysis of BCE inspires us that binary classification loss should meet basic and general mathematical properties to guarantee validness and convergence in challenging object detection. We summarize these properties as follows:
Monotonicity. The loss value should monotonically decrease
positive class score and increase w.r.t negative class score because large positive score or small negative score im- plies well-performed classifier, which should have small loss penalty.
Convergence. When the positive class score tends to be + the loss gradient magnitude should converge to zero ensure model convergence.
Regression Loss Function For the regression branch, consistency of loss value and distance be- tween the predicted and target bounding box, named distance-loss consistency, should be confirmed. Distance-loss consistency claims that loss value is supposed to change consistently with distance, where loss value increases when the prediction moves away from the ground truth.
5

---

Published as a conference paper at ICLR 2021
3.2.3 MODEL OPTIMIZATION SIMULATION
Although the convergence property verification module filters out either non-monotonic or non- convergent losses, it does not guarantee that the loss candidates are suitable for optimization. The sparse action space demands that there should be other techniques to ensure the quality of optimiza- tion. To address this issue, we propose the model optimization simulation module that inspects the optimization capability of the loss.
Specifically, we train and test the loss candidates on a small verification dataset Dverify- which is constructed by sampling only one image randomly from each category on benchmark dataset like COCO, to estimate the optimization quality. Then apply AP performance on D Dverify of the top loss in the previous generation as a threshold to filter out divergent, training-unfriendy unfriendly, and poor-performed loss candidates.
Algorithm CSE-Autoloss algorithm.
Input: Search Space S. Number of Generations E. Population Size N, Number of Parents P. Verification Set yy Proxy Training Set Dirain, Proxy Validation Set Deat, Initial Predefined Loss Function Lt, Number of Loss to Evaluate on Verification Set
1: Evolving N loss functions to generate loss function population from Cr
2: for <- 1. E do
3: Check loss function properties with convergence-simulation modules
4: Train and test on Dversfy and keep top-K loss candidates
5: Evaluate on Dtrasn and D.at, and keep top-P" losses as parents
6: Generate N offisring from the parents for the next generation
7: end for
Output: Best-discovered loss function Loest
4 EXPERIMENTS
Datasets We conduct loss search on large-scale object detection dataset COCO (Lin et al. 2014) and further evaluate the best-searched loss combinations on datasets with different distribution and domain, iee PASCAL VOC (VOC) (Everingham et al., 2015) and Berkeley Deep Drive (BDD) Yu al. 2020). to verify the generalization of the searched loss functions across datasets. COCO is common dataset with 80 object categories for object detection, containing 118K images for training and 5K minival for validation. In the search experiment, we randomly sample 10k images from the training set for validation purpose. VOC contains 20 object categories. We use the union of VOC 2007 trainval and VOC 2012 trainval for training and VOC 2007 test for validation and report mAP using IoU at 0.5. BDD is autonomous driving dataset with 10 object classes, in which 70k images are for training and 10k images are for validation.
Experimental Setup We apply Faster R-CNN and FCOS the representative detectors for two- stage and ote-stage, respectively, in the loss search experiments on COCO for object detection. We apply ResNet-50 (He al., 2016) and Feature Pyramid Network (Lin et al., (01 as feature tractor. For FCOS, we employ common tricks such as normalization on bounding box, centerness on regression, and center sampling. Besides that, we replace centerness branch with the IoU scores as the target instead of the original design for FCOS and ATSS to better utilize the IoU information, which has slight AP improvement. Note that loss weights are set default as MMDetection (Chen) et al. 2019a) but the regression weight for ATSS sets to 1 for the search loss combinations. To be consistent with the authors' implementation. we use GPUs with 4 images/GPU and GPUs with 2 images/GPU for FCOS and Paster-RCNN, Concerning the proxy task for loss function evaluation, we perform training on the whole COCO benchmark for only one epoch to trade off performance and efficiency. Our code for object detection and evolutionary algorithm are based MMDetec- tion (Chen et al. 2019a) and DEAP (Fortin et al. 2012).
In the search experiments, regression and classification loss functions are searched independently. For regression loss search, we take GloU loss (Rezatofighi et al. 2019) as predefined initial loss C to generate the first loss function population, with CE and FL serving the fixed classification loss for Faster R- R-CNN and FCOS, respectively. While for classification branch search, CEI and FLI loss served Lr, with GloU serving the fixed regression loss.

---

Published as a conference paper at ICLR 2021
4.1 RESULTS ON TWO-STAGE AND ONE-STAGE DETECTORS
We name the best loss combination searched with Faster R-CNN R50 (Ren et al. 2013) and FCOS R50 (Tian et al. 2019) as CSE-Autoloss- and CSE-Autoloss-B respectively, with the subscript cls and reg indicating classification and regression branch. The searched formulations are as follows:
CSE-Autoloss-Acule w) - dot + sin(w) (n)) log(softmax())),,
(1 2
CSE-Autoloss-Am((1 u. (1
CSE-Autoloss Bus (2 w. = - -ww((1 + (1 g))) + (gd(x)
wy)(o(()) wy))og(1 dx))]
3eu + 12e + 3i + 3u + 18
CSE-Autoloss-B.i Bugg u. = -3eu + iu + 15e + 56 + 5u
Table 1: Detection results on the searched loss combination and default loss combination for mul- tiple popular two-stage and one-stage detectors on COCO val. Column Loss indicates loss for classification and regression branch in sequence.
Detector
Loss
AP (%)
AP %)
AP75(
APs
AP APM (%)
AP
CE CE+L) L1
37.4
58.1
40.4
21.2
41.0
48.1
Faster R-CNN R50
CE GloU
37.6+0.2
58.2+0.1
41.0-0.6
21.5+0.3
41.1+0.1
48.9+0.8
CSE-Autoloss-A
38.5+
58.6+0.5
41.8-14 1.4
22.0+0.8
42.2+1.2
50.2
CE + L1
39.4
60.1
43.1
22.4
43.7
51.1
Faster R-CNN R101
CE GloU
39.6
59.2 0.9
42.9 0.2
22.6+0.2
43.5
51.5+0.
CSE-Autoloss-A A
40.2
60.1
43.7
22.6
44.3+0.6
52.7+16
CE + Smooth L.
40.3
58.6
44.0
22.5
43.8
52.9
Cascade R-CNN R50
CE GloU
40.2-0.1
58.0
0.6
43.6
22.4
43.6
52.6 0.3
CSE-Autoloss-A
40.5
58.8
44.1 0.11
22.8
+0.3
43.9
53.3
CE + Smooth LI
38.2
58.8
41.4
21.9
40.9
49.5
Mask R-CNN R50
CE + GloU
38.5+0.3
58,8+0.0
41.8
0.4
21.9+0.0
42.1+12
49.7+0.2
CSE-Autoloss-A
39.1 +0.9
59.3+0.5
42.4-1.0
22.4
43.0+2.
51.4+15 +1.9
FL GloU
38.8
56.8
42.2
22.4
42.6
51.1
FCOS R50
CSE-Autolosss 39.6+0.88
57.5+0.7
43.1-0.9
22.7+0.3
43.7+1.
52.6+1.5
FL GloU
40.0
57.9
43.3
23.8
43.7
51.3
ATSS R50
CSE-Autoloss-B 40.50
40.5
58.3
43.9-0.6
43.9
0.6
23.3-0.5
23.3
0.5
44.3+0.6
52.5
The quantitative results about the gain brought by the searched loss under the same hyper-parameters for multiple popular two-stage and one-stage detectors shown in Table [ Results Faster R-CNN R50 and FCOS R50 indicate the generalization of CSE-Autoloss across detectors and the best-searched loss combination is capable of stimulating the potential of detectors by a large margin. Take Faster R-CNN R50 for example, CSE-Autolos- outperforms the baseline by 1. 1% in terms of AP, which is a great improvement for that two years of expert effort into designing hand-crafted loss function yields only 0.2% AP improvement from Bounded IoU loss to GloU loss.
To verify the searched loss can generalize to different detectors, we respectively apply CSE- Autoloss-A on other two-stage models (i.e. Faster R-CNN R101, Cascade R-CNN R50 CCii & Vasconcelos 2018) and Mask R-CNN R50 (He al. (017)), and CSF-Autoloss-B on ATSS (Zang et 2020). Note that only replace the classification and regression branch of Mask R-CNN with CSE-Autoloss-A A. Results Table represent the consistent gain brought by the effective searched loss without additional overhead.
We further conduct best-searched loss evaluation experiments for Faster R-CNN R50 on VOC and BDD to validate the loss transferability across datasets. Results displayed in Table Bl which indicate the best-searched loss combination enables the detectors to converge well on datasets with different object distributions and domains.
7

---

Published as a conference paper at ICLR 2021
Table Efficiency improvement with progressive convergence-simulaten modules on searching for Faster R-CNN classification branch. The proposed modules filter out 99% loss candidates, which enhances the efficiency a large extent.
Convergence property verification Model optimization simulation #Evaluated loss
5 10
7 x 102
50
Table 3: Detection results searched loss combination and default loss combination with Faster R-CNN on PASCAL VOC 2007 test and BDD val.
Table 4: Detection results on predefined ini- tial loss and default loss for Faster R-CNN and FCOS classification branch on COCO val.
Detector
Loss
AP (%)
Loss
VOC mAP (%)
BDD AP (%)
CE GloU
37.6
Faster R-CNN R50
CE+
79.5
36.5
CEI + GloU
37.7
CE + GloU
79.6+
36.6+0.1
FL GloU
38.8
FCOS R50
CSE-Autoloss-A
80.4+ +..9
37.3 +0.8
FLI GloU
39.0+0.2
Table Comparison on the efficiency search algorithms and distinct branches for Faster R-CNN.
Search algorithm
Search branch
Another branch #Evaluated loss
Wall-clock hours
AP (%)
Random search
Classification Classification Classification
GloU
10a
100
37.8
Evolution search
GloU
5 x 103
5 x 102
38.2
CSE-Autoloss CSE-Autoloss
GloU
50
26
38.2
Regression
CE
15
8 37.9
4.2 ABLATION STUDY
Effectiveness of Sanvergence-Simulatinn Modules. Efficiency improvement with progressive simuulaioe-s modules is shown in Table 2 on searching for Faster R-CNN classifica- tion branch. With our proposed modules, CSE-Autoloss filters out 99% loss candidates with bad mathematical property and poor performance, which largely increases the efficiency without com- promising the accuracy the best-discovered loss.
Individual Loss Contribution. As shown in Table 6 and Table 7 our searched losses match or outperform the existing popular losses consistently for Faster R-CNN and FCOS. Figure illlstrrates illustaass the convergence behaviors of popular loss combinations and our CSE-Autoloss-A.
Table 6: Comparison on different loss combina- tions for Faster R-CNN R50 on COCO val.
Table 7: Comparison on different loss combina- tions for FCOS R50 on COCO val.
Loss
AP (%)
CE + L1
37.4
Loss
AP (%)
CE + IoU
37.9
FL GloU
38.8
CE + Bounded loU
37.4+0.0
FL. + DloU
DloU
38.7 38.7
CE + GloU
37.6+0.2
FL + CloU
+8.8
CE + DIoU
37.9+0.5
GHM et al. 2019a
38.6 0.2
CE + CloU
37.8+0.4
FL CSt-Autoloss B
39.1 +0.3
+ CSE-Autoloss-Ang 37.9+0.5
CSE-Autoloss-B. GloU
39.4 +0.6
CSE-Autllos-AAg + GloU
38.2+008
CSE-Autolos--B
39.6
CSE-Autoloss-A
38.5 +1.1
8

---

Published as a conference paper at ICLR 2021
0.23
0.22
34
0.21
bbox mAP
32
boox mAP
30-
CE+ Uo Loss
Bounded Loss
0.18
CE+ GloU Loss
CE+ CloU Loss
0.17
22
CE+ DioU Loss
20
CSE ostal
0.16
10 12
epochs
Evolutional Generations
Figure 3: Evaluation results of popular hand- crafted loss combinations and CSE-Autoloss- with Faster R-CNN R-50 on COCO val. Differ- ent loss functions have various convergence be- haviors, and CSE-Autoloss- achieves the best performance compared with other loss combi- nations by large margin.
Figure 4: Visualization of the CSE-Autoloss and vanilla evolution search for FCOS, where the evaluated loss individuals are scattered as orange and blue respectively. With our convergence-simulatinn modules, the number of evaluated loss candidates is largely reduced.
Search Algorithm. We compare the efficiency between different search algorithms including ran- dom search, vanilla evolution search, and CSE-Autoloss. Table [ shows that the evolution search is much better than random search. But due to the high sparsity of the action space, the vanilla evolution search requires hundreds of wall-clock hours for server with GPUs. CSE-Autoloss speeds up the search process by 20x because of the effective discovers well performed loss combinations in around 1.5 wall-clock days.
simullaions modules and
Search Complexity of Different Branches. In our search space. the input nodes of the regres- sion branch are scalars, while the classification branch contains vectors. Besides, the optimization of classification branch is more sensitive and difficult than regression, leading to sparse valid indi- viduals. Therefore, the evolution population required for classification is 10*, while only 10t for regression. The total number of evaluated loss and search overhead for discovering classification branch is much larger than that for regression as Table shows.
Evolution Behavior Analysis. To validate the effectiveness of CSE-Autoloss, we further compare the behaviors CSE-Autoloss and vamilla evolution search with FCOS. The vanilla evolution search evaluates around 1000 loss candidates in each generation, compared only about 10 individuals for CSE-Autoloss. The loss individuals are scattered in Figure 4 Note that we only plot top-100 well- performed losses in each generation in the vanilla evolutionary search for clear visualization. The vanilla evolution search is hard to converge because a discrete replacement of the operation could fail loss function. With our convergence-simulan modules, the efficiency largely improved.
5 CONCLUSION
In this work, we propose convergence--iimultion driven evolutionary search pipeline, named CSE- Autoloss, for loss function search on object detection. speeds up the search process largely by regularizing the mathematical property and optimization quality and discovers well-performed loss functions efficiently without compromising the accuracy. We conduct search experiments on both two-stage detectors and one-stage detectors and further empirically validate the best-searched loss functions on different architectures across datasets, which shows the effectiveness and generalization of both CSE-Autoloss and the best-discovered loss functions. We hope work could provide insights for researchers this field to design novel loss functions for object detection and more effective frameworks for automated loss function search.

---

Published as a conference paper at ICLR 2021
REFERENCES
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, pp. 6154-6162 2018
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint TTtV::906.077 2019a.
Kean Chen, Jianguo Li, Weiyao Lin, John See, Ji Wang, Lingyu Duan, Zhibo Chen, Changwei He, and Junni Zou. Towards accurate one-stage object detection with ap-loss. In CVPR, pp. 5119-5127, 2019b.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):8--13,, January 2015.
Félix-nntiine Fortin, François-Michel De Rainville, Marc-André Gardner, Marc Parizeau, and Christian Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171-2175, July 2012.
Ross Girshick. Fast r-cnn. In ICCV, pp. 1440-1448, 2015.
David E Goldberg and Kalyanmoy Deh. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of genetic algorithms, volume 1, pp. 99-93. Elsevier, 1991.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. In CEC, pp. 1-8, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961- 2969, 2017.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, pp. 784-799, 2018.
Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In AAAI, vol- ume 33, pp. 8577-8584, 2019a.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. In ICCV. pp. 8410-8419, 2019b.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang- Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detec- tion. In NeurlPS, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, pp. 2117-2125, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017b.
Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In ECCV, 2020a.
10

---

Published as a conference paper at ICLR 2021
Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoc V Le. Evolving normalization-activvaan activation layers. arXiv preprint arXiv:2004.02967 2020b.
Qi Qian, Lei Chen, Hao Li, and Rong Jin. Dr loss: Improving object detection by distributional ranking. In CVPR, pp. 12164-12172, 2020.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. In ICLR, 2018.
Esteban Real, Chen Liang, David R So, and Quoc V Le. Automl-zero: Evolving machine learning algorithms from scratch. ICML, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, pp. 91-99, 2015.
Hamid Rezatofighi, Nathan Tsoi, Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pp. 638-666, 2019.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In ICCV, pp. 9627-9636, 2019.
Lachlan Tychsen-Smith and Lars Petersson. Improving object localization with fitness nms and bounded iou loss. In CVPR, pp. 6877-6885, 2018.
Xiaobo Wang, Shuo Wang, Cheng Chi. Shifeng Zhang, and Tao Mei. Loss function search for face recognition. ICMI., 2020.
Shengkai Wu, Xiaoping Li, and Xinggang Wang. lou-aware single-stage object detector for accurate localization. Image and Vision Computing, pp. 103911, 2020.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pp. 2636-2645, 2020.
Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. In ACMM, pp. 516-520, 2016.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In CVPR, pp. 9759 9768, 2020.
Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. In AAAI, pp. 12993-13000, 2020.


Reviews:
[{"review_id": "rF9SacaL9RF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to automatically discover proper loss functions for object detection. It first designs some unit mathematical operations as search space, and then performs evolutionary algorithm to discover well-performed loss functions for the object detection tasks. Different from image classification, one needs to search both classification and localization losses in object detection. To accelerate the search, the paper proposes convergence property verification and model optimization simulation to effectively evaluate the searched loss and reduce the search space.\n\nOverall, this paper brings new ideas and experimental conclusions to the auto loss discovery in the object detection area. However, the proposed method is heavily based on the works of auto loss for image classification, i.e. \"Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization\". Some key differences lies on the acceleration by checking the Monotonicity and Convergence of the searched loss. Another issue I am curious about is how the method balances between classification and localization losses. Independently searching either cls or localization loss does not guarantee a good performance when combining them together for a high AP. Finally, the searched loss formulation seems complex and tricky, thus raising the doubt of its generalization across a wide range of neural networks and datasets. Do we need to re-search the loss function once we change the network architecture or the training data slightly?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "rDnT1FW45T", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes CSE-Autoloss to deal with searching loss function for object detection with an effective convergence-simulation driven evolutionary search algorithm. CSE-Autoloss outperforms both previous CE/FL loss for classification and L1 loss for regression.\n\nPros:\n1. Motivation is clear and good.\nLoss design is tricky in object detection, since for it needs to deal with class imbalance and fore/background imbalance for classification loss and regression loss. This paper motivated by network architecture search, proposes effective searching algorithm for loss function, which is impressive and novel.\n\n2. Convergence property verification and model optimization simulation are simple and effective.\nPrevious NAS papers usually require large computation resources and time to search the best model. CSE-Autoloss employs monotonicity and convergence of classification loss to and a small verity dataset to easily filter out loss candidate, it speedups the searching process by 10x and reaches similar results with random research in Table 5.\n\nCons:\n1. Some related works on improving Focal loss and GIoU [1][2] are missing, it is better to include them as parts of hand-crafted loss functions for comparisons.\n\n2.Is it possible to search classification and regression loss together? It seems for now classification and regression loss are searched independently.\n\n3. Different formulations of CSE-Autoloss-A_cls and CSE-Autoloss-B_cls.\nI understand that formulation of Autoloss_A_reg can be different from Autoloss_B_reg since their hand-crafted counterpart L1 & IoU loss are different, but for classification loss, FL loss is CE loss with a simple modulating factor, but it seems Autoloss-A_cls and Autoloss-B_cls design are quiet different. Any possible explanation for this difference?\n\n[1] Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression, in AAAI 2020\n[2] Gradient Harmonized Single-stage Detector, in AAAI 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "NQQKsOgvNdZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use an evolutionary search algorithm to search for better loss functions for the classification and regression branch of an object detector. The algorithm starts with 20 primitive mathematical operations. Due to the highly sparse action space, the vanilla evolutionary algorithm would take a long time to converge. Then the authors propose two ways to reduce the search space. First, they filter out loss functions which generates gradients of large magnitude to well-classified samples and do not converge to zero. Second, they construct a very small dataset by sampling only one image randomly from each category and evaluate the loss function on it to quickly filter out bad loss candidates.\n\nPros:\nThe idea of applying evolutionary algorithm to search for a loss function for training an object detector is interesting and, to the best of my knowledge, new. The proposed way to reduce the search space seems to be effective. The authors apply the new loss function to other detectors, and it seems to generalize well to different detectors.\n\nCons:\nThe baselines provided in this paper are little bit weak. They are pretty far behind from the current state-of-the-art detectors. Although the new loss function is able to improve the performance of all of the baselines, it is unclear if it would generalize well to state-of-the-art results. The authors should consider using baselines with better performance. I also wonder if the loss would generalize to different datasets, such as PASCAL VOC.\n\nIn a subsection of section 3.1, “Effectiveness of the Search Space”, the authors mention that they introduce IoU into the cross entropy and focal loss, and show that they outperform the conventional cross entropy and focal loss. They then said this verifies the effectiveness of the search space. I am not seeing how this is connected to the effectiveness of the search space. Can the authors further elaborate on this?\n\nThe authors use the modified version of the cross entropy or focal loss as the initial loss in the search experiments. The final version of the loss is still pretty much the same as the initial loss. It seems that the search is biased towards to the initial loss. I wonder if the authors have tried other initializations.\n\nThe authors said that the regression branch contains fewer primitive operators because of the simplicity of the regression task. It is unclear which task, classification or regression, is actually simpler. It seems that regression may actually be harder as we have achieved human level results on the task of image classification but not on the detection, which requires precise localization of objects.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hvAPw350MM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nIn this paper, the authors propose CSE-Autoloss to search loss functions for object detection. A well-designed search space is carefully designed to explore novel loss functions. In order to reduce the search overhead, convergence-simulation modules are raised.\n\nReasons for score:\n\nOverall, I vote for accepting. My major concern is about the clarity of the paper and some additional experiments (see cons below). Hopefully the authors can address my concern in the rebuttal period.\n\nPros:\n\n1.This paper proposes a new framework to search loss functions for object detection. For me, it is the first attempt to discover loss functions for object detection automatically.\n\n2.To search the novel loss functions for object detection, the authors propose a well-designed search space.\n\n3.To reduce the search overhead caused by evolutionary algorithm, this paper raises two modules: convergence property verification module and model optimization simulation module. \n\nCons:\n\n1.For the predefined initial loss, CEI/FLI loss and GioU loss are used for classification and regression, respectively. If using other losses as the predefined initial loss, how is the performance? Could you provide some experiments?\n\n2.In the search experiments, regression and classification loss functions are searched independently. For me, this can not result in the optimal loss functions. Why does not search jointly?\n\n3.In the evolutionary algorithm, how do the probabilities of cross-over and mutation affect the performance of searched loss?\n\n4.In the experiments, there are some results to verify the generalization of the searched loss on different detectors and datasets. If directly search loss functions on these detectors and datasets, how are the performances?\n\nQuestions during rebuttal period:\n\nPlease address and clarify the cons above", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline scores but overall lean positive. 

The reviewers point out that the paper presents interesting new ideas and an effective solution to the problem of automatically searching for loss functions. The empirical results are convincing, although the baselines are not the strongest possible in terms of absolute performance. Overall, the ACs find that the paper has sufficient novelty and technical contribution to be accepted. 
Decision: Accept (Poster)