Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper looks at loss functions for semantic segmentation. Typical metrics are not easily differentiable w.r.t. the outputs of the DNNs that generate the labels. Instead, a proxy/surrogate loss function is learned jointly with the network in a two-level optimization.

Pros:

i) Good accuracy results. Code will allow others to verify and build on these.

ii) High-quality ablation studies.

Avoiding any parameterization (one of the the more surprising components of the method) is a good ablation to have. Also, comparison to random in Fig 3 demonstrates that the outer-level optimization is working as expected.

Q1: What is the naive surrogate used in Table 5? Equation (8)?

Cons:

iii) Seems like an ad-hoc approach that doesn't incorporate classical techniques for optimizing "hard" loss functions.

Non-differentiable loss functions, or losses over discrete variables, are less common when dealing with CNNs. But they can be included in deep learning with techniques like that in:

Chen et. al. "Learning Deep Structured Models"

Notable citations that take this a "classical" approach are:

Ranjbar et. al., "Optimizing Non-Decomposable Loss Functions in Structured Prediction"

This kind of approach is an omission from related work that might be worth correcting. Especially because it is the "direct" approach, in that it is optimizing the original loss functions using known techniques, without additional levels of learning or approximatinos as in the submission.

Overall, though the empirical results are strong enough that it is likely that the submission is doing something useful well. So the approach in the submission seems well-supported by that fact, even if previous literature makes it a non-obvious way to do things.

iv) Not much illustration or exploration of the learned loss surfaces.

Experiments partially leave open the question: How closely is the surrogate loss matching the target metrics? Basically, is there more direct evidence that for the intuition that h_AND is approximately equivalent to f_AND? The networks trained on the surrogate losses do well on the original metrics. But it's not impossible that some trivial solution or unexpected loss function, that is not clearly similar to e.g. mIoU, can train a network to learn a segmentation model that produces segmentations with good mIoU.

Minor comments:

Q2: Reason for bolding in Table 2 is unclear, what is a "(co)-highest result?"
review_title: Careful study of semantic segmentation proxy loss functions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to directly optimize the metrics of semantic segmentation tasks, such as mIoU, which is different from the most existing methods which minimize the cross-entropy as a proxy. The metrics typically contain one-hot labels and logical operations. In order to directly optimize them, the authors first relax the one-hot label/prediction by Softmax. Then the logical operations applied on the one-hot label are extended by a continuous parameter function which is Monotonical and has the same output as the logical operation with 0/1 input. Finally, the authors describe a reinforcement learning framework to optimize the metrics parameterization (i.e., the outer objective), while the inner objective (i.e., the segmentation network) is trained by standard SGD. The experiments have been performed on Pascal VOC 2012 and Cityscapes datasets, showing the searched loss outperformed the traditional ones such as cross-entropy.

Pros:
1. The Bezier curves parameterization is used to guarantee a smooth relaxation, and two additional constraints are proposed to regularize the parameterization.
2. Two-stage optimization is proposed to learn the network weights and the loss parameterization alternatively.
3. The ablations on the generalization-ability is conducted.

Cons:
1. My main concern is that this paper is poorly written and difficult to understand. The abstract and the introduction do not provide necessary information about the high-level design of the proposed algorithm.
2. The method is also not well motivated. For example:
2a. Why using reinforcement learning to optimize the outer objective? Is there any other option?
2b. Are there any other alternatives to the piecewise Bezier curves? Is the algorithm sensitive to a different number of Bezier segments?
3. Some necessary technical details are missing, for example, what does "No Parameters" mean in Table 5? Does it mean a single straight line from (0, 0) to (1, 1)?

In summary, this paper may contain interesting idea and algorithm implementation, I encourage the authors to further polish their presentation to better express them.
review_title: The formulation is interesting, but the paper is poorly written

final_score: rating: 5: Marginally below acceptance threshold
review_content: Major strengths:   
   
1.) In comparison with traditional loss function such as Cross-Entropy, WCE, DPCE, and SSIM, the proposed method achieves competitive performance. In addition, the authors also compare with the searched loss functions such as searched mIoU, searched FWIoU, etc. By combining the searched mIoU with the BIoU/BFI surrogate losses, the overall method achieves reasonable global performance, while refines the boundaries. 
  
2.) Generalization experiments are conducted to demonstrate that the proposed surrogate losses are effectiveness under various scenes and categories. 
  
3.) The overall paper is clearly presented and easy to follow.  
   
   
Major weaknesses:   

1.) Lack of computational cost analysis. According to the implementational details, the overall training process has two steps, which is likely to increase the computational burden. To this end, the authors are suggested to conduct some analysis on this issue.  
   
2.) The comparison experiments with some recent and important methods are missing. For example, the following three papers also focus on Auto-ML based semantic segmentation:  
  
[1] FasterSeg: Searching for Faster Real-time Semantic Segmentation, ICLR 2020;  
[2] Graph-guided Architecture Search for Real-time Semantic Segmentation, CVPR 2020;  
[3] Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation, CVPR 2019.  

As the Auto Seg-Loss is particularly designed for Auto-ML based semantic segmentation, the comparison with the related methods is required. 
  
In addition, there are some other recent semantic segmentation methods (proposed in 2019 and 2020) focusing on the PASCAL VOC and Cityscapes datasets, although they are not related to Auto-ML. As an effective semantic segmentation method, the overall segmentation performance of the model is supposed to be competitive. To this end, the authors are also suggested to make comparisons with these SOTA semantic segmentation methods.  

3.) The writing of the abstract needs to be improved. In the current manuscript, the abstract is too brief. To better attract the interests of the readers, the authors can first introduce the background of the problem studied in this paper. Next, some detailed motivations can also be included.  

review_title: In this work, the authors proposed an Auto Seg-Loss architecture for semantic segmentation. The authors first extended the traditional metrics to surrogates, by replacing the one-hot operation with the softmax one, and the non-differentiable logical operations with the arithmetic ones. Next, the authors employed the piecewise Bezier curve for surrogate parameterization. 

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors suggest using differentiable surrogate parameterized loss functions that more closely approximate some of the frequently used metrics for segmentation, including variations of accuracy, IoU, and F-score for the whole area and the boundaries, and use reinforcement learning to tune the parameters instantiating the surrogate loss functions.
Moderate improvement is shown through experimental evaluations compared to cross-entropy and other used loss functions.

I think, overall, this is a good paper, the following elaborates it:

Strengths:
* I like the general idea of bridging the gap between the often-non-differentiable evaluation metrics and the optimization loss functions so that the model is more closely optimizing for what the task mostly cares about.
* Despite the existence of several previous works on the taken general direction, there are still seemingly novel methodological contributions including the loss parameterization and its tuning process. The design of most of the components made intuitive sense.
* The experimental setup including comparisons to different losses, ablation, and transferability studies are thorough. A moderate but consistent improvement is observable when compared to the frequently used loss functions, over two commonly used semantic segmentation datasets, and across various network architectures. 
* The paper is fairly well-written and easy to follow.

Weak points:
* Even though designing differentiable surrogate losses is conceptually simple and straight-forward, as shown in the experimental section, in practice it turns out that a relatively complicated set of tricks is needed to be leveraged before it shows improvements, including the parameterization and its two regularization schemes and the parameterization tuning using reinforcement learning. This in total adds significant complexity on top of a normal training scheme.
* Some of the other distance-based metrics including the Hausdorff and Chamfer distance used to evaluate semantic segmentation have not been covered in this work.

Minor comments:
* In table 4, for R101-DeepLabv3+, R101-PSPNet and HRNetV2p-W48, I am missing comparisons to their ceiling, i.e. the values they could have obtained if the parameters were directly optimized for them.
* Algorithm 1: \mu_0, for t=1... and (\mu_t, ...) seem not consistent. Please fix.
* Equation (4) variable c is ambiguous in the left denominator as being the variable for both nested sigmas, please change the inner, e.g. to c'.
* Equation (2) is not computationally stable as the denominator can turn to 0, in case a class c is not present in the dataset.
* In Table 5, "Fail" under VOC mIoU is ambiguous; in case a very low performance is obtained, I think reporting the number would still help.
* In the text of section 3, IoU-based metrics, please specify that Min-Pooling and Max-Pooling are with stride 1.
* Typo: "the our searched loss" => "that our searched loss"
review_title:  

Meta-review: Auto Seg-Loss uses a differentiable surrogate parameterized loss function that approximates using RL some of the non-differentiable metrics for segmentation. Auto Seg-Loss outperform cross-entropy and other loss functions through a great number of experiments. The main concerns rised by the reviewers (More clarity on the abstract and intro, extending the related work, and performance experiments) has been addressed. Accordingly I recommend the paper to be accepted at ICLR 2021.
Decision: Accept (Poster)