Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation
Authors: Hao Li
Chenxin Tao
Xizhou Zhu
Xiaogang Wang
Gao Huang
Jifeng Dai
Venue: ICLR 2021 Poster (2021)
Keywords: Loss Function Search
Metric Surrogate
Semantic Segmentation

Abstract: Designing proper loss functions is essential in training deep networks. Especially in the field of semantic segmentation, various evaluation metrics have been proposed for diverse scenarios. Despite the success of the widely adopted cross-entropy loss and its variants, the mis-alignment between the loss functions and evaluation metrics degrades the network performance. Meanwhile, manually designing loss functions for each specific metric requires expertise and significant manpower. In this paper, we propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metric. We substitute the non-differentiable operations in the metrics with parameterized functions, and conduct parameter search to optimize the shape of loss surfaces. Two constraints are introduced to regularize the search space and make the search efficient. Extensive experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently. The searched losses can generalize well to other datasets and networks. Code shall be released at https://github.com/fundamentalvision/Auto-Seg-Loss.

Published as a conference paper at ICLR 2021
AUTO SEG-LOSS: SEARCHING METRIC SURROGATES FOR SEMANTIC SEGMENTATION
Hao Li Chenxin T Xizhou Zhu Xiaogang Wang Gao ,uang', Jifeng 4 The Chinese University of Hong Kong TTinnhua University
"SenseTime Research 'Qing Yuan Research Institute, Shanghai Jiao Tong University haoli@lingg cuhk.
(zhuwalter, edu. hk, tcx200ma aeeaemmall....m..aameeeaeealllll wwt.ams.ttiinhhaaaa....... edu. cn
com
xqwangßee cuhk. edu. hk, machuang@tsingheen edu.
ABSTRACT
Designing proper loss functions is essential in training deep networks. Especially in the field of semantic segmentation, various evaluation metrics have been pro- posed for diverse scenarios. Despite the success of the widely adopted cross- entropy loss and its variants, the mis-alignment between the loss functions and evaluation metrics degrades the network performance. Meanwhile, manually de- signing loss functions for each specific metric requires expertise and significant manpower. In this paper, we propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metrie. We substi- tute the non-differentiable operations in the metrics with parameterized functions, and conduct parameter search optimize the shape of loss surfaces. Two con- straints are introduced to regularize the search space and make the search efficient. Extensive experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions con- sistently. The searched losses can generalize well to other datasets and networks. Code shall be released at htttt:t///// //gi ttt.b thub. com fundamental leiintaiiiiiei liision on
Auto- Seg- Loss
INTRODUCTION
Loss functions are of indispensable components in training deep networks, as they drive the feature learning process for various applications with specific evaluation metrics. However, most metrics, like the commonly used 0-1 classification error, are non-differentiable in their original forms and cannot be directly optimized via gradient- based methods. Empirically, the cross-entropy loss serves well an effective surrogate objective function for variety of tasks concerning categorization. This phenomenon is especially prevailing image semantic segmentation, where various evaluation metrics have been designed to address the diverse task focusing on different scenarios. Some metrics measure the accuracy on the whole image, while others focus more on the segmentation boundaries. Although cross-entropy and its variants work well for many metrics, the mis-alignment between network training and evaluation still exist and inevitably leads to performance degradation.
Typically, there are ways for designing metric-specific loss functions in semantic segmenta- tion. The first is to modify the standard cross-entropy loss to meet the target metric (Ronneberger al. 2015 Wu al. 2016). The other is design other clever surrogate losses for specific eval- uation metrics (Rahman & Wang 2016 Milletari et al.) 2016). Despite the improvements, these handcrafted losses need expertise and are non-trivial to extend other evaluation metrics.
In contrast to designing loss functions manually, an alternative approach is to find a framework that design proper loss functions for different evaluation metrics in an automated manner, motivated
by recent progress in AutoML (Zoph & Le 2017] Pham et al.) 2018 Liu et al. 2018 Li et al. 2019). Although automating the design process for loss functions is attractive,
non non-trivial to apply an
Equual contribution. Thhi work is done when Hao Li and Chenxin Tao are interns at SenseTime Research. 'Corrssponiing author.

---

Published as a conference paper at ICLR 2021
AutoML framework loss functions. Typical AutoML algorithms require proper search space, in which some search algorithms are conducted. Previous search spaces are either unsuitable for loss design, or too general to be searched efficiently. Recently [i et all (2019) and Wang al. (2020) proposed search spaces based on existing handcrafted loss functions. And the algorithm searches for the best combination. However, these search spaces are still limited to the variants of cross-entropy loss, and thus do not address the mis-alignment problem well.
In this paper, propose a general framework for searching surrogate losses for mainstream non- differentiable segmentation metrics. The key idea is that we can build the search space according to the form of evaluation metries. In this way, the training criteria and evaluation metries are unified. Meanwhile, the search space is compact enough for efficient search. Specifically, the metrics are first relaxed to the continuous domain by substituting the one-hot prediction and logical operations, which are the non-differentiable parts in most metrics, with their differentiable approximations. Parameterized functions are introduced to approximate the logical operations, ensuring that the loss surfaces are smooth while effective for training. The loss parameterization functions can be of arbitrary families defined on [0, 1]. Parameter search is further conducted on the chosen family so to optimize the network performance
the validation set with the given evaluation metric. Two essential constraints are introduced to regularize the parameter search space. We find that the searched surrogate losses can effectively generalize to different networks and datasets. Extensive experiments on Pascal VOC (Everingham et al. 2015) and Cityscapes (Cordts et al.) (2016) show our approach delivers accuracy superior than the existing losses specifically designed for individual segmentation metrics with a mild computational overhead.
Our contributions can be summarized as follows: 1) Our approach is the first general framework of surrogate loss search for mainstream segmentation metrics. 2) We propose an effective parameter regularization and parameter search algorithm, which can find loss surrogates optimizing the target metric performance with mild computational overhead. 3) The surrogate losses obtained via the pro- posed searching framework promote our understandings on loss function design and by themselves are novel contributions, because they are different from existing loss functions specifically designed for individual metrics, and are transfcrable across different datasets and networks.
2 RELATED WORK
Loss function design is an active topic in deep network training Ma 2020). In the area of image semantic segmentation, cross-entropy loss is widely used (Ronneberger et al. 2015 Chen et al., 2018). But the cross-entropy loss is designed for optimizing the global accuracy measure (Rahman & WWang 2016: et al. 2020), which is not aligned with many other metrics. Numerous studies are conducted to design proper loss functions for the prevalent evaluation metrics. For the mloU metric, many works (Ronneberger et al., 2013: Wu et al. 2016) incorporate class frequency to mit- igate the class imbalance problem. For the boundary Fi score, the losses at boundary regions are up-weighted (Caliva et al. 2019 Qin et al. 2019). so as to deliver more accurate boundaries. These works carefully analyze the property of specific evaluation metrics, and design the loss functions in a fully handcrafted way, which needs expertise. By contrast, we propose a unified framework for deriving parameterized surrogate losses for various evaluation metrics. Wherein, the parameters are searched by reinforcement learning in an automatic way. The networks trained with the searched surrogate losses deliver accuracy on par or even superior than those with the best handcrafted losses.
Direct loss optimization for non-differentiable evaluation metrics has long been studied for struc- tural SVM models (Joachims] 2005 Yue et al., 2007; Ranjbar et al., 2012). However, the gradients w.r.t. features cannot be derived from these approaches. Therefore. they cannot drive the training of deep networks through back poepppropagation Hazan et al. (2010) proposes to optimize structural SVM with gradient descent, where aosg-nng inference is applied to get the gradients of the expectation of evaluation metrics. Song et al. (2016) further extends this approach to non-linear models (e.g., deep neural networks). However, the computational complexity is very high during each step in gradient descent. Although Song et al. (2016) and Mohapatra et al. (2018) have de- signed efficient algorithms for the Average Precision (AP) metric, other metrics still need specially designed efficient algorithms. Our method, by contrast, is general for the mainstream segmentation metrics. Thanks to the good generalizability, our method only needs perform the search process
2

---

Published as a conference paper at ICLR 2021
once for specifie metric, and the searched surrogate loss can be directly used henceforth. Applying the searched loss for training networks brings very little additional computational cost.
Surrogate loss is introduced to derive loss gradients for the non non-differentiable evaluation metries. There are usually two ways for designing surrogate losses. The first is to handcraft an approximated differentiable metric function. For the loU measure, |Rahman & Wang (2066) propose to approxi- mate the intersection and union seperately using the softmax probabilities in differentiable form, and show its effectiveness on binary segmentation tasks. [Berman et al. (2018) further deal with multi-class segmentation problems by extending mloU from binary inputs to the continuous domain with the convex Lovåsz extension, and their method outperforms standard cross entropy loss in multi-class segmentation tasks. For the Fl measure, dice loss is proposed by MMileetari et al. (2016) as a direct objective by substituting the binary prediction with the softmax probability. In spite of the success, they do not apply for other metrics.
The second solution is to train a network to approximate the target metrie. Nagendar et al. (2018) train network to approximate mloU. Patel et al. (2020) design neural network to learn embeddings for predictions and ground truths for tasks other than segmentation. This line of research focuses on minimizing the approximation error w.r.t the target metrics. But there is no guarantee that their approximations provide good loss signals for training. These approximated losses are just employed in post-tuning setup, still relying on cross-entropy pre-trained models. Our method significantly differs in that we search surrogate losses to directly optimize the evaluation metrics in applications.
target of machine learning (He al. 2019). Recently sub-field of AutoML. neural architecture search (NAS), has attracted much attention due to its success in au- tomating the process of neural network architecture design (Zoph & Le 2017 Pham et al. 2018 Liu et al. 2018). As an essential element, loss function has also raised the interest of researchers to automate its design process. Li ei al. (2019) and Wang et al. (2020) design search spaces based on ex- isting human-designer loss functions and search for the best combination parameters. There are two issues: a) the search process outputs whole network models rather than loss functions. For every new network or dataset, the expensive search procedure is conducted again, and b) the search space are filled with variants of cross-entropy, which cannot solve the mis-alignment between cross-entropy loss and many target metrics. By contrast, our method outputs the searched surrogate loss functions of close form with the target metrics, which are transferable between networks and datasets.
Auto.L is
3 REVISITING EVALUATION METRICS FOR SEMANTIC SEGMENTATION
Various evaluation metrics are defined for semantic segmentation, to address the diverse task focus- ing on different scenarios. Most of them are of three typical classes: Acc-based, loU-based, and F1-score-based. This section revisits the evaluation metries, under a unified notation set.
suppose the validation set is composed of N images, labeled with categories from C' classes (background in- cluded). Let In. n E {l...., N} be the n-th image, and Yn be the corresponding ground-truth segmentation mask. Here Y, = (yn.e.h.wJe.h.. Je.h, is a one-hot vector, where yn,e.h,w E (0, 1} indi- cates whether the pixel at spatial location (h, w) belongs to the e-th category (c {1..... C}). In evaluation, the ground-truth segmentation mask Yn is compared to the network prediction Y, = (fm,e,h.w )e, where ūn,c,h,w E {0, 1}. Dn,e,h,w is quantized from the continuous scores produced by the network (by argmax operation).
Table T summarizes the mainstream evaluation metrics. The notations are as follows:
Ace-based metries. The global accuracy measure (gAcc) counts the number of pixels correctly classified. It can be written with logical operator AND Eq. (I). The gAce metric counts each pixel equally, so the results of the long-tailed categories have little impact on the metric number. The mean accuracy (mAcc) metric mitigates this by normalizing within each category as in Eq. (2).
IoU-d metrics. The evaluation is on set similarity rather than pixel accuracy. The intersection- over-union (loU) score is evaluated between the prediction and the ground-truth mask of each cate- gory. The mean IoU (mloU) metric averages the loU scores of all categories, as in Eq. (3).
In the variants, the frequency weighted IoU (FWIoU) metric weighs each category IoU score by the category pixel number, as in Eq. (4). The boudary IoU (BloU) (Kohli et al., 2009) metric only cares about the segmentation quality around the boundary, it picks the boundary pixels out in evaluation
3

---

Published as a conference paper at ICLR 2021
Table 1: Revisiting mainstream metrics for semantic segmentation. The metrics with measure the segmentation accuracy on the whole image. The metrics with * focus on the boundary quality.
Type
Name
Formula
2moa a Baichor AND Voucha
Global Accuracy!
gAce
(1)
Enoh Mincchi
Acc-based
mAce 1
C /
den Muachio AND Mouchhe
Mean Accuracy"
Emdha Miuchho
mlol 1
Eadia Misactio AND Miladii
Mean
Emdra Briech.s OR Umiehss
(3)
Σ En Muaadwo Endri Brachio AND Baachao
IoU bused
Froquency Weighted IoU!
FWIoU
Ln Aise Bajo' Lnihess Ba bo OR Mac
(4)
1 7 Σ
Σn E.weRoIy, Amacha AND Bhuchao
BloU =
Boundary IoU*
En (AINEBDD Bainstor OR Mia
(5)
where BD(y) XOR Min-Poolingg))
BF1 Σ
2 prec, recall, (prec_ recall.)
where prec, > h. BD( AND Max-PPollng(DD((s)....
Fl-score-based Boundary Fl Score*
(6)
Σnh. HD
recall, Endrr Max (BDling AND(BD(),)..))))
En BD(ga)cn.
and ignores the rest pixels. It can be calculated with Eq. 5 in which (z().) denotes the boundary region in map y/- (D)),) derived by applying XOR operation on the min-pooled ground-truth mask. The stride of the Min-Poolingg()) is
Flsscore-based metrics. F1-score is criterion that takes both precision and recall into consider- ation. A well-known metric of this type is boundary Fl-score (BFI-score) (Csurka et al., 2013), which is widely used for evaluating boundary segmentation accuracy. The computation and recall in BF1-score is as in Eq. 6 where (0),,) and BD(,,) are derived from Eq. (5). Max pooling with stride 1, Max- Max-Pooling (alling(),, applied on the boundary regions to allow error tolerance.
precision
AUTO SEG-LOSS FRAMEWORK
In the Auto Seg-Loss framework, the evaluation metrics transferred into continuous surrogate losses with learnable parameters, which are further optimized. [ iillustrates our approach.
4.1 EXTENDING METRICS TO SURROGATES
As shown in Section most segmentation metrics are non-differentiable because they take hot prediction maps as input, and contain binary logical operations. We extend these metrics to be continuous loss surrogates by smoothing the non-differentiable operations within.
Extending One-hot Operation. The one-hot prediction map, Y, (ün.e,h.w be.h, w. is derived by picking the highest scoring category at each pixel, which is further turned into one-hot form. Here, we approximate the one-hot predictions with softmax probabilities, as,
Fn, Vn.e.h,w = Softmax, (Zn,o,h,ar)
where &n,c.h,w is the category score output by the network (without normalization). The approximated one-hot prediction is denoted by Fn.c,h,w-
Extending Logical Operations. As shown in Table [ the non-differentiable logical operations, fAND (31, V2). (3R and fxoR (v1, 32), of indispensable components in these metrics.
Because the XOR operation can be constructed by AND and OR, fxoR(1) y2) = fon(y), /2)
fAND (31, y2). focus extending FAND(y1. y2) and fon(ur Y2) to the continuous domain.
Following the common practice, the logical operators are substituted with arithmetic operators
(A W2) = V112, SOR (V11 M22 = 91 + 1/2 9/3/2
(8)
4

---

Published as a conference paper at ICLR 2021
Auto Seg-Loss Parameter Search
network
one-hot
(NN. Shatd-am))
Ahold-aat hold-out subset
P
target metric
rewards
category Score
one hot prediction
Surrogate Parameter Search Space
split
Po Bann
surrogate metric
praxy network
Softmax
(N S
Swain
N
gradients
training subset
category score
softmax probability
Optimal Parameterication
Searched parameters 6*1
segreentation network
Softmax
Eor (Nai 8)
N
f
gradients
surrogate
training
category
softmax probability
Fox (a.
Searched Surrogate Loss
Figure 1: Overview of the proposed Auto Leg-Loss framework. The surfaces of HAND and hon shown in the "Optimal Parameterization' illustrate the searched optimal parameterization for mloU.
where V1, 1/2 1]. Eq. (8) can be directly extended to take continuous //, V1 E [0, 1] as in- puts. By such extension, together with the approximated one-hot operation, naïve version of differentiable surrogate losses can be obtained. The strength such surrogates is that they are di- rectly derived from the metries, which significantly reduces the gap between training and evaluation. However, there is no guarantee that the loss surfaces formed by naïvely extending Eq. (8) provide accurate loss signals. To adjust the loss surfaces, we parameterize the AND and OR functions as
hAND y2; ( = ((N)))AND) {(2; AND)
hOR (V11 : 32; fOR) 9(y: (oR) + ( 0RR) - g(v:: POR) g(v2 0OR) where 0): [0, 1] R is scalar function parameterized by 0.
The parameterized function g () can be from arbitrary function families defined on [0, 1]. e.g., piecewise linear functions and piecewise Bézier curves. With chosen function family, the param- eters 0 control the shape of loss surfaces. We seek to search for the optimal parameters so as to maximize the given evaluation metric.
Meanwhile, optimal parameter search is non-trivial. With the introduced parameters, the plasticity of loss surfaces is strong. The parameterized loss surfaces may well be chaotic, or be far away from the target evaluation metric even at the binary inputs. For more effective parameter search, we regularize the loss surfaces by introducing two constraints on g(y;
Truth-table constraint is introduced to enforce the surrogate loss surfaces taking the same values as the evaluation metric score at binary inputs. This is applied by enforeing
g(0; 0) 0, g(1:0) = 1.
(10)
Thus, the parameterized functions hyn Y/; ()) preserve the behavior of the corresponding logical operations (1,, 1/2 on binary inputs y1, 1/2 {0, 1).
Monotonicity constraint is introduced based on the observation of monotonicity tendency in the truth tables of AND and OR. It pushes the loss surfaces towards benign landscape, avoiding dramatic non-smoothness. The monotonicity constraint is enforced on AAND (2), , y2) and (1R, 32). as
AhAND 2 0, Ohon/ /Oy; 2 O, Vy, E [0, 1), i 1,2.
Applying the chain rule and the truth table constraint, the monotonicity constraint implics
ag(y: 0yy 2 0, Vy E [0, [0. Empirically find it important to enforce these constraints in parameterization.
(11)
Extending Evaluation Metrics. Now we can extend the metrics to surrogate losses by a) replacing the one-hot predictions with softmax probabilities. and b) substituting the logical operations with parameterized functions. Note that if the metric contains several logical operations, their parameters will be shared. The collection of parameters in one metric denoted O For a segmentation network N and evaluation dataset S, the score of the evaluation metric is denoted as {), And the parameterized surrogate loss is denoted as (N:

---

Published as a conference paper at ICLR 2021
4.2 SURROGATE PARAMETERIZATION
The parameterized function can be from any function families defined on [0, IJ. such as picewise Bézier curve and piecewise linear functions. Here we choose the piecewise Bézier curve for param- eterizing ((;;0) which is widely used in computer graphics and is easy to enforce the constraints via its control points. We also verify the effectiieness of parameterizing gly: 0) by piecewise linear functions. See Fig. for visualization and Appendix [ for more details.
A piecewise Bézier curve consists of a series of quadratic Bézier curves, where the last control point of one curve segment coincides with the first control point of the next curve segment. If there are segments in a piecewise Bézier curve, the k-th segment defined as
B(k s) = - 5)8 Bas + 2s(1 8) B2x++1 + s² B2x+2; 0 11851
(12)
where s transverses the k-th segment, B2 = (1(2k+1). ((2x+)),.) (i 0, 1, denotes the i-th control point the k-th segment, in which u, index the 2-d plane axes. A piecewise Bézier curve with n segments has 2n + 1 control points in total. To parameterize gly 8), we assign
B2R a + 2s(1 5) + (22++)), y (1 =))²BaR.
(13a)
g 0) (1 s)² B2x,0 + +2s(1 9) B(2++11) s² ((22+2),
(13b)
s.t. B2 S y B(2x+2),ux
(13c)
where 0 is the control point set, B2 B(2x+1).. < B(2x+2). O k - Given input the segment index & and the transversal parameter s are derived from Eq. (13c) and Eq. (I3a), respectively. Then gly; is assigned arrange the control points in the u u-axis as, Bu., = 0. B and the last control points are 0 and respectively.
Eq. (l3)). Because gly; 0) is defined on y E [0, 1]. we where the u-coordinate of the first
The strength of the piecewise Bézier curve is that the curve shape is defined explicitly via the control points. Here we enforce the truth-table and the monotonicity constraints on the control points via, (truth-table constraint)
Bo,v 0, B2n,0 1; B2G, < B(28+1), I B(2k+2).r == 1,......, 1.
(monotonicity constraint)
To fulfill the above restrictions in optimization, the specific form of the parameters is given by Bi.a B(i-1),a Bi - B(1-1),v)
0 0= {
1 1.2..... ................................ 2n 1
B2n, B(i-1).a B2n,e Bii-1)ar
with Bo = (0,0) and B2., (1,1) fixed. So every 0; P0.u,0s.0 is in range [0, 1j2 and it is straight-forward to compute the actual coordinates of control points from this parameterized form. Such parameterization makes each O independent with each other, and thus simplifies the optimiza- tion. By default, we use piecewise Bézier curve with two segments to parameterize 0).
Algorithm 1:
Auto Seg-Loss Parameter Search
Input: Initialized network Nov- initialized distribution f41 and o², target metric Ç, training set Stain and hold-out training set Swotsoo
(p,)
Result: Obtained optimal parameters O
06-
18
1-
for to do do Sample parameter 6,0) Network training
for
PR Rd.
NNomeco, o² T);
02
w² w'()!") arg max,, Eg(e) Suain, with w initialized from wo: Compute the evaluation metric score {(e(")) {(N wt (o()) Smolscour;
Parameterization of g(v; 0) using Piecewise Bézier curve with four segments. The red points are control points. The purple point is on the curve, which shows the relationship among y. g(y; 0) and the transversal parameter s.
Figure 2:
end Update Me+1 arg max, M DM R(u, 0!
end return O- = arg max,,, ΣM ) vt "....,T+..

---

Published as a conference paper at ICLR 2021
4.3 SURROGATE PARAMETER OPTIMIZATION
Algorithm describes our parameter search algorithm. The training set is split into two subsets, Smain for training and Suu for evaluation in the search algorithm, respectively. Specifically, suppose we have a segmentation network N., with weights w, our search target is the parameters that maximize the evaluation metric on the hold-out training set E(N Shold-out)
max {( E(N (e) (e) Shold-our),
s.t. w w0(O) arg max Eo Strain.
(14)
θ
To optimize Eq. [14]. the segmentation network is trained with SGD as the inner-level problem. At the outer-level. we use reinforcement learning our searching algorithm, following the com- mon practice in AutoML. (Zoph & Lej 2017; Pham et al.] 2018). Other searching algorithms, such as evolutionary algorithm, may also be employed. Specifically, the surrogate parameters searched via the PPO2 algorithm (Schulman et al., (017). The process consists of T sam- pling steps. In the 1-th step, we aim to explore the search space around that from 1. Here M sets of parameters i Ji=l IM are sampled independently from truncated normal distribu- tion (Burkardi 2 as θ (1 T²T), with each variable in range [0. 1]. In it, Pt and o²7 denote the mean and covariance of the parent normal distribution (o is fixed as 0.2 in this paper). fe summarizes the information from the (t 1)-th step. M surrogate losses are con- structed with the sampled parameters, which drive the training of M segmentation networks sep- arately. To optimize the outer-level problem, we evaluate these models with the target metric and take the evaluation scores as rewards for PPO2. Following the PPO2 algorithm, Pt+1 is computed as 1++1 arg max M7 Ei= R(1,, M4, where the reward R(u, Mr, O:) is
R(1, Mb, Qi) min [((0,,,01)) (p(Or E(O.), CLIP
1 e, (11+0 )(0.)).
where min(..-) picks the smaller item from its inputs, CLIP 1 c, 1,11 c) clips to be within 1 c and 1 + c, and p( o²I) the PDF of the truncated normal distribution. Note that the mean reward of the M samples is subtracted when computing {(O;) for better convergence. After steps, the mean f4 with the highest average evaluation score is output as the final parameters O*.
Empirically we find the searched losses have good transferability, i.e., they can be applied for dif- ferent datasets and networks. Benefiting from this, we use a light proxy task for parameter search. In it, utilize a smaller image size, a shorter learning schedule and lightweight network. Thus, the whole search process is quite efficient (8 hours on PASCAL VOC with NVIDIA Tesla V100 GPUs). More details are in Appendix A In addition, the search process can be conducted only once for a specific metric and the resulting surrogate loss can be directly used for training henceforth.
5 EXPERIMENTS
We evaluate on the PASCAL VOC 2012 (Everingham et al. 2015) and the Cityscapes (Cordts et al.) 2016) datasets. We use Deeplabv3+ (Chen et al., 2018)) with ResNet-50/101 (He et al. 2016) the network model. During the surrogate parameter search, we randomly sample 1500 training images in PASCAL VOC and 500 training images in Cityscapes form the hold-out set Shold respectively. The remaining training images form the training set Suain in search. (Ao is set to make g 0) y- The backbone network is ResNet-50. The images are down-sampled to be of 128 x 128 resolution. SGD lasts only 1000 iterations with a mini-batch size of 32. After the search procedure, we re-train the segmentation networks with ResNet-101 using the searched losses on the full training set and evaluate them on the actual validation set. The re-train settings are the same as Deeplabv3+ (Chen) al., 2018), except that the loss function is substituted by the obtained surrogate loss. The search time is counted on 8 NVIDIA Tesla V100 GPUs. More details are in Appendix A
5.1 SEARCHING FOR DIFFERENT METRICS
In Table 2 we compare our searched surrogate losses against the widely-used cross-entropy loss and its variants, and some other metric-specific surrogate losses. We also seek to compare with the AutoML-based method in Li et al. (2019), which was originally designed for other tasks. But we cannot get reasonable results due convergence issues. The results show that searched losses
7

---

Published as a conference paper at ICLR 2021
par better the previous losses their target metries. It is interesting to note that the obtained surrogates for boundary metrics (such as BloU and BF1) only focus on the boundary areas, Appendix for further discussion. We also tried training segmentation networks driven by both searched mloU and BloU/BF1 surrogate losses. Such combined losses refine the boundaries while keeping reasonable global performance.
Table 2: Performance of different losses PASCAL VOC and Cityscapes segmentation. The results of each loss function's 's target metrics are underlined. The scores whose difference with the highest less than 0.3 are marked in bold.
Dataset
PASCAL VOC
Cityscapes
Loss Function
mIoU FWIoU
BloU
BF1
mAec
gAcc
mloU
FWIoU
BloU
BF1
mAce
gAcc
Cross Entropy
78.69
91.31
70.61
65.30
87.31
95.17
79.97
93.33
62.07
62.24
87.01
96.44
WCE Rouneberger
2015
69.60
85.64
61.80
37.59
92.61
91.11
73.01
90.51
53.07
51.19
89.22
94.56
DPCE Caliva al. 2019
79.82
91.76
71.87
66.54
87.76
95.45
80.27
93.38
62.57
65.99
86.99
96.46
SSIM
Ola cl. 20199
79,26
91.68
71.54
66.35
87.87
95.38
80.65
93.22
63.04
72.20
86.88
96.39
Dicel Los8 Millitan
77.78
91.34
69.85
64.38
87.47
95.11
79.30
93.25
60.93
59.94
86.38
96.39
Lovåsz
79.72
91.78
72.47
66.65
88.64
95.42
77.67
92.51
56.71
53.48
82.05
96.03
Scarched mIoU
80.97
92.09
73.44
68.86
88.23
95.68
80.67
93.30
63.05
67.97
87.20
96.44
Searched FWIoU
80.00
91.93
75.14
65.67
89.23
.4
79.42
93.33
61.71
59.68
87.96
96.37
Searched BloU
48.97
69.89
79.27
38.99
81.28
62.64
45.89
39.80
63.89
38
62.80
58.15
Scarched BFI
1.93
0.96
7.39
74.83
6.51
2.66
6.78
3.19
18.37
77.40
12.09
8.19
Searched mAccc
69,80
85.86
72.85
35.62
92.66
91.28
74.10
90.79
54,62
53 45
89.22
94.75
Searched gAce
79.73
91.76
74.09
64.41
88,95
95.47
79.41
93.30
61.65
62.04
87.08
96.51
Searched mloU BloU
81.19
92.19
76.89
69.56
88.36
95.75
80.43
93.34
63.88
65.87
87.03
96.45
Searched mloU + BFI
78.72
90.80
71.81
73.57
86.70
94.88
78.30
93.00
61.62
71.73
87.13
96.23
5.2 GENERALIZATION OF THE LOSS
Generalization among datasets. Table 3 evaluates the generalization ability of our searched loss surrogates among different datasets. Due to limited computational resource, we train networks only with the searched mloU, BF1 and mAcc surrogate losses. The results show that our searched surro- gate losses generalize well between these two datasets with quite different scenes and categories.
Table 3: Generalization of our searched surrogate losses between PASCAL VOC and Cityscapes.
Datasets
Cityscapes
VOC
VOC - Cityscapes
Loss Function
mloU
FWIoU
BloU
BF1
mAce
gAce
mloU
FWIoU
BloU
BF1
mAcc
gAcc
Cross Entropy
78.69
91.31
70.61
65.30
87.31
95.17
79.97
93.33
62.07
62.24
87.01
96.44
Searched mloU
80.05
91.72
73.97
67.61
88.01
95.45
80.67
93.31
62.96
66.48
87.36
96.44
Searched BF1
1.84
0.93
7.42
75.85
6.48
1.47
6.67
3.20
19.00
77.99
12.12
4.09
Searched mAcc
70.90
86.29
73.43
37.18
93.19
91.43
73.50
90.68
54.34
54.04
88.66
94.68
Generalization among segmentation networks. The surrogate losses are searched with ResNet- 50 + Daepp.abv3+ on PASCAL VOC. The searched losses drive the training of ResNet-101 DeepLabv3+, PSPNet (Zhao al., (017) and HRNet (Sun al., 2019) PASCAL VOC. Ta- ble 4 shows the results. The results demonstrate that our searched loss functions can be applied to various semantic segmentation networks.
5.3 ABLATION
Parameterization and constraints. Tabaees the parameterization and the search space con- straints. In it, a surrogate without parameters refers to Eq. (8), with the domain extended from dis- crete points {0. }} to continuous interval [0, 1]. This naive surrogate deliver much lower accuracy, indicating the essence of parameterization. Without the truth-table constraint, the training process diverges at the very beginning, where the loss gradients become "NaN". And the performance drops the monotonicity constraint is not enforced. The performance drops or even the algorithm fails without the constraints.
8

---

Published as a conference paper at ICLR 2021
Table 4: Generalization our searched surrogate losses among different network architectures on
PASCAL VOC. The losses are searched with ResNet-50+ DeepLabv3+ on PASCAL VOC.
Network
R50-DeepLabv3+
R101-DeepLab33+
R101-PSPNet
HRN W48
Loss Function
mloU
BF1
mAce
mioU
BF1
mAcc
mloU
BF1
mAcc
mloU
BF1
mAcc
Cross Entropy
76.22
61.75
85,43
78.69
65,30
87.31
77.91
64.70
85.71
76.35
61.19
85,12
Searched mloU
78.35
66.93
85.53
80.97
68.86
88.23
78.93
65.65
87.42
77.26
63.52
86.80
Searched BF1
1.35
70.81
1.93
74.83
6.51
1.62
71.84
6.33
1.34
68.41
5.99
Scarched mAce
69.82
36.92
91.61
69.80
35.62
92.66
71.66
39.44
92.06
68.22
35.90
91.46
Proxy tasks for parameter search. Table O ablates this. The bottom row is our default setting with light-weight backbone, down-sampled image size and shorter learning schedule. The default setting delivers on par accuracy with heavier settings. This is consistent with the generalization ability of our surrogate losses. Thus
improve the search efficiency via light proxy tasks.
Parameter search algorithm. Fig. compares the employed PPO2 (Schulman et al. 2017) gorithm with random search. The much better performance of PPO2 suggests that surrogate loss search non-trivial and reinforcement learning helps to improve the search efficiency.
Table 5: Ablation search space constraints.
Table 6: Ablation on search proxy tasks.
Parameter Truth-table
Monotonicity
VOC mIoU
Backbone Image Iterations
Time(hours)
VOC mloU
46.99
x
33.0
81.15
Fail
R50
128 x 128
2000
17.1
80.56
/
77.76
R101
128 x 128
1000
13,3
80.75
/
80.64
R50
128 x 128
1000
8.5
80.97
0.55
0,20
searshing DF1
searching midee
01677
0.45
0.40
0.50
0.30
PPO2
Random
Random
PPO2
0.30
Randam
PPO2
0.30
10
20
25
0060
30
0.50
10
15
20
25
30
35
sampling
sampling Stop
sampling stop
(a) search mIoU
(b) search for BF1
(c) search for mAce
Figure 3: Ablation on loss parameter search. Each curve presents the highest average evaluation
score up the f-th step in one search process. The search process is repeated four times.
6 CONCLUSION
The introduced Auto Seg-Loss is powerful framework to search for the parameterized surrogate losses for mainstream segmentation evalutation metrics. The non-differentiable operators are substi- tuted by their parameterized continuous counterparts. The parameters are optimized to improve the final evaluation metries with essential constraints. would be interesting to extend the framework to more tasks, like object detection, pose estimation and machine translation problems.
ACKNOWLEDGMENTS
The work is supported by the National Key R&D Program of China (2000AA00005000), Beijing Academy of Artificial Intelligence and the Institute for Guo Qiang of Tsinghua University.
REFERENCES
Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. The lovász-softman loss: tractable surrogate for the optimization of the intersection-over-uiioo measure in neural networks. In Pro-

---

Published as a conference paper at ICLR 2021
ceedings of IEEE Conference Computer Viston and Pattern Recognition (CVPR), pp. 4413- 4421, 2018.
John Burkardt. The truncated normal distribution. Department of Scientific Computing Website, Florida State University, pp. 1-35, 2014.
Francesco Caliva, Claudia Iriondo. Alejandro Morales Martinez, Sharmila Majumdar, and Valentina Pedoia. Distance map loss penalty term for semantic segmentation. In International Conference on Medical Imaging with Deep Learning-Extee Extended Abstract Track, 2019.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834 848, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213-3223, 2016.
Gabriela Csurka, Diane Larlus, Florent Perronnin, and France Meylan. What is good evaluation measure for semantic segmentation? In Proceedings of the British Machine Vision Conference (BMVC), volume 27, pp. 2013, 2013.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision, 111(1):98-118111111 136, 2015.
Bharath Hariharan, Pablo Arbeléez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In Proceedings of IEEE International Conference Computer Vision (ICCV), pp. 991-998, 2011.
Tamir Hazan, Joseph Keshet, and David A McAllester. Direct loss minimization for structured prediction. In Advances in Neural Information Processing Systems (NIPS), pp. 1594-1602, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: survey of the state-of-the-art arXiv preprint arXiv: 00709, 2019.
Thorsten Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd International Conference on Machine Learning (ICML), pp. 377-384, PMLR, 2005.
Pushmeet Kohli, Philip HS Torr, et al. Robust higher order potentials for enforcing label consistency. International Journal of Computer Vision, 82(3):302-33 324, 2009.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am- Ifs: Automl for loss function search. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pp. 8410-8419, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
Jun Ma. Segmentation loss odyssey. arXiv preprint 2mXiv:2005.13449 2020.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV), pp. 565-571. IEFF, 2016.
10

---

Published as a conference paper at ICLR 2021
Pritish Mohapatra, Michal Rolinek, CV Jawahar, Vladimir Kolmogorov, and M Pawan Kumar. Ef- ficient optimization for based loss functions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3693-3701, 2018.
Gattigorla Nagendar, Digvijay Singh, Vineeth N Balasubramanian, and CV Jawahar. Neuro-iou: Learning a surrogate loss for semantic segmentation. In Proceedings of the British Machine Vision Conference (BMVC). pp. 278, 2018.
Yash Patel, Tomas Hodan, and Jiri Matas. Learning surrogates via deep embedding. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learming (ICML), pp. 41044 PMLR, 2018.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary-aware salient object detection. In Proceedings of IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 7479- 2019.
Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-uiio in deep neural networks for image segmentation. In International Symposium on Visual Computing, pp. 234-244. Springer, 2016.
Mani Ranjbar, Tian Lan, Yang Wang, Steven N Robinovitch, Ze-Nian Li, and Greg Mori. Opti- mizing nondecomposable loss functions in structured prediction. IEEE Transactions Pattern Analysis and Machine Intelligence, (5(4):9119224 2012.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. Springer, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Yang Song, Alexander Schwing, Raquel Urtasun, et al. Training deep neural networks via direct loss minimization. In Proceedings of the 33rd International Conference on Machine Learning (ICML). pp. 2169-2177. PMLR, 2016.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5693-5703, 2019.
Xiaobo Wang, Shuo Wang, Cheng Chi, Shifeng Zhang, and Tao Mei. Loss function search for face recognition. In Proceedings of the 37th International Conference on Machine Learning (ICML). PMLR, 2020.
Zifeng Wu, Chunhua Shen, and Anton van den Hengel. Bridging category-level and instance-level semantic image segmentation. arXiv preprint arXiv: 1605.06888.. 2016.
Yisong Yue, Thomas Finley, Filip Radlinski, and Thorsten Joachims. A support vector method for optimizing average precision. In Proceedings of the 30th Annual International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 271-278, 2007.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene pars- ing network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2881-2890, 2017.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. Proceedings of the 5th International Conference Learning Representations (ICLR), 2017.
11


Reviews:
[{"review_id": "tM4WYz1Sfov", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors suggest using differentiable surrogate parameterized loss functions that more closely approximate some of the frequently used metrics for segmentation, including variations of accuracy, IoU, and F-score for the whole area and the boundaries, and use reinforcement learning to tune the parameters instantiating the surrogate loss functions.\nModerate improvement is shown through experimental evaluations compared to cross-entropy and other used loss functions.\n\nI think, overall, this is a good paper, the following elaborates it:\n\nStrengths:\n* I like the general idea of bridging the gap between the often-non-differentiable evaluation metrics and the optimization loss functions so that the model is more closely optimizing for what the task mostly cares about.\n* Despite the existence of several previous works on the taken general direction, there are still seemingly novel methodological contributions including the loss parameterization and its tuning process. The design of most of the components made intuitive sense.\n* The experimental setup including comparisons to different losses, ablation, and transferability studies are thorough. A moderate but consistent improvement is observable when compared to the frequently used loss functions, over two commonly used semantic segmentation datasets, and across various network architectures. \n* The paper is fairly well-written and easy to follow.\n\nWeak points:\n* Even though designing differentiable surrogate losses is conceptually simple and straight-forward, as shown in the experimental section, in practice it turns out that a relatively complicated set of tricks is needed to be leveraged before it shows improvements, including the parameterization and its two regularization schemes and the parameterization tuning using reinforcement learning. This in total adds significant complexity on top of a normal training scheme.\n* Some of the other distance-based metrics including the Hausdorff and Chamfer distance used to evaluate semantic segmentation have not been covered in this work.\n\nMinor comments:\n* In table 4, for R101-DeepLabv3+, R101-PSPNet and HRNetV2p-W48, I am missing comparisons to their ceiling, i.e. the values they could have obtained if the parameters were directly optimized for them.\n* Algorithm 1: \\mu_0, for t=1... and (\\mu_t, ...) seem not consistent. Please fix.\n* Equation (4) variable c is ambiguous in the left denominator as being the variable for both nested sigmas, please change the inner, e.g. to c'.\n* Equation (2) is not computationally stable as the denominator can turn to 0, in case a class c is not present in the dataset.\n* In Table 5, \"Fail\" under VOC mIoU is ambiguous; in case a very low performance is obtained, I think reporting the number would still help.\n* In the text of section 3, IoU-based metrics, please specify that Min-Pooling and Max-Pooling are with stride 1.\n* Typo: \"the our searched loss\" => \"that our searched loss\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "bypSdrBvo44", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to directly optimize the metrics of semantic segmentation tasks, such as mIoU, which is different from the most existing methods which minimize the cross-entropy as a proxy. The metrics typically contain one-hot labels and logical operations. In order to directly optimize them, the authors first relax the one-hot label/prediction by Softmax. Then the logical operations applied on the one-hot label are extended by a continuous parameter function which is Monotonical and has the same output as the logical operation with 0/1 input. Finally, the authors describe a reinforcement learning framework to optimize the metrics parameterization (i.e., the outer objective), while the inner objective (i.e., the segmentation network) is trained by standard SGD. The experiments have been performed on Pascal VOC 2012 and Cityscapes datasets, showing the searched loss outperformed the traditional ones such as cross-entropy.\n\nPros:\n1. The Bezier curves parameterization is used to guarantee a smooth relaxation, and two additional constraints are proposed to regularize the parameterization.\n2. Two-stage optimization is proposed to learn the network weights and the loss parameterization alternatively.\n3. The ablations on the generalization-ability is conducted.\n\nCons:\n1. My main concern is that this paper is poorly written and difficult to understand. The abstract and the introduction do not provide necessary information about the high-level design of the proposed algorithm.\n2. The method is also not well motivated. For example:\n2a. Why using reinforcement learning to optimize the outer objective? Is there any other option?\n2b. Are there any other alternatives to the piecewise Bezier curves? Is the algorithm sensitive to a different number of Bezier segments?\n3. Some necessary technical details are missing, for example, what does \"No Parameters\" mean in Table 5? Does it mean a single straight line from (0, 0) to (1, 1)?\n\nIn summary, this paper may contain interesting idea and algorithm implementation, I encourage the authors to further polish their presentation to better express them.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XZpPXv3v1xb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Major strengths:   \n   \n1.) In comparison with traditional loss function such as Cross-Entropy, WCE, DPCE, and SSIM, the proposed method achieves competitive performance. In addition, the authors also compare with the searched loss functions such as searched mIoU, searched FWIoU, etc. By combining the searched mIoU with the BIoU/BFI surrogate losses, the overall method achieves reasonable global performance, while refines the boundaries. \n  \n2.) Generalization experiments are conducted to demonstrate that the proposed surrogate losses are effectiveness under various scenes and categories. \n  \n3.) The overall paper is clearly presented and easy to follow.  \n   \n   \nMajor weaknesses:   \n\n1.) Lack of computational cost analysis. According to the implementational details, the overall training process has two steps, which is likely to increase the computational burden. To this end, the authors are suggested to conduct some analysis on this issue.  \n   \n2.) The comparison experiments with some recent and important methods are missing. For example, the following three papers also focus on Auto-ML based semantic segmentation:  \n  \n[1] FasterSeg: Searching for Faster Real-time Semantic Segmentation, ICLR 2020;  \n[2] Graph-guided Architecture Search for Real-time Semantic Segmentation, CVPR 2020;  \n[3] Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation, CVPR 2019.  \n\nAs the Auto Seg-Loss is particularly designed for Auto-ML based semantic segmentation, the comparison with the related methods is required. \n  \nIn addition, there are some other recent semantic segmentation methods (proposed in 2019 and 2020) focusing on the PASCAL VOC and Cityscapes datasets, although they are not related to Auto-ML. As an effective semantic segmentation method, the overall segmentation performance of the model is supposed to be competitive. To this end, the authors are also suggested to make comparisons with these SOTA semantic segmentation methods.  \n\n3.) The writing of the abstract needs to be improved. In the current manuscript, the abstract is too brief. To better attract the interests of the readers, the authors can first introduce the background of the problem studied in this paper. Next, some detailed motivations can also be included.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mjs8kwRjcof", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks at loss functions for semantic segmentation. Typical metrics are not easily differentiable w.r.t. the outputs of the DNNs that generate the labels. Instead, a proxy/surrogate loss function is learned jointly with the network in a two-level optimization.\n\nPros:\n\ni) Good accuracy results. Code will allow others to verify and build on these.\n\nii) High-quality ablation studies.\n\nAvoiding any parameterization (one of the the more surprising components of the method) is a good ablation to have. Also, comparison to random in Fig 3 demonstrates that the outer-level optimization is working as expected.\n\nQ1: What is the naive surrogate used in Table 5? Equation (8)?\n\nCons:\n\niii) Seems like an ad-hoc approach that doesn't incorporate classical techniques for optimizing \"hard\" loss functions.\n\nNon-differentiable loss functions, or losses over discrete variables, are less common when dealing with CNNs. But they can be included in deep learning with techniques like that in:\n\nChen et. al. \"Learning Deep Structured Models\"\n\nNotable citations that take this a \"classical\" approach are:\n\nRanjbar et. al., \"Optimizing Non-Decomposable Loss Functions in Structured Prediction\"\n\nThis kind of approach is an omission from related work that might be worth correcting. Especially because it is the \"direct\" approach, in that it is optimizing the original loss functions using known techniques, without additional levels of learning or approximatinos as in the submission.\n\nOverall, though the empirical results are strong enough that it is likely that the submission is doing something useful well. So the approach in the submission seems well-supported by that fact, even if previous literature makes it a non-obvious way to do things.\n\niv) Not much illustration or exploration of the learned loss surfaces.\n\nExperiments partially leave open the question: How closely is the surrogate loss matching the target metrics? Basically, is there more direct evidence that for the intuition that h_AND is approximately equivalent to f_AND? The networks trained on the surrogate losses do well on the original metrics. But it's not impossible that some trivial solution or unexpected loss function, that is not clearly similar to e.g. mIoU, can train a network to learn a segmentation model that produces segmentations with good mIoU.\n\nMinor comments:\n\nQ2: Reason for bolding in Table 2 is unclear, what is a \"(co)-highest result?\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Auto Seg-Loss uses a differentiable surrogate parameterized loss function that approximates using RL some of the non-differentiable metrics for segmentation. Auto Seg-Loss outperform cross-entropy and other loss functions through a great number of experiments. The main concerns rised by the reviewers (More clarity on the abstract and intro, extending the related work, and performance experiments) has been addressed. Accordingly I recommend the paper to be accepted at ICLR 2021.
Decision: Accept (Poster)