Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
In this work, the authors seek to leverage external sources of data to improve the generalization of segmentation models. In particular, they seek to identify images which generate discordance among models, hypothesizing that they would be well-suited to improve model performance. Once selected, they leverage human annotators to first filter this image set and then segment the images, which are then used to retrain the model. They demonstrate improved performance relative to a batch of competing models which are not updated using this procedure.

------

Recommendation:
Given the lack of a competitive baseline, opaqueness around the impact of the algorithm's hyperparameters, and what's likely to be noisy estimates of improvement, I cannot recommend this paper for acceptance as is. See below for greater detail.

------

Positives:
* The authors recognize that not all images are created equal when looking to improve a model and attempt to tackle this challenging problem. This is particularly relevant in high-stakes environments when failure in rare cases can have a disproportionate impact (e.g., autonomous vehicles, healthcare, etc.).

* The authors identify that scaling human annotation, in particular for image segmentation, can be cost prohibitive and propose a method to optimize this process. If successful, such a method could have significant implications for industries where the cost of annotation is high (e.g., healthcare where highly paid experts are required).

---

Concerns:
* The authors propose a fairly complex (and costly) pipeline for improving generalization to the unseen dataset. However, they fail to compare against even a simple baseline such as random selection of images from this dataset. Comparing against models which are not updated, in particular when it's clear that none of them generalize, is a weak baseline that could likely be outperformed by far simpler uses of the external data. 

* T^(i+1) = 30 is quite small, especially given the number of classes. While it is understandable that such a sample cannot be extraordinarily large, by leveraging such a small value there's significant noise in the evaluation criteria. Given the previous concern, this likely would not affect the reported results in the paper. However, with a more competitive baseline, such noise may make it challenging to identify improvements in the selection of image for finetuning.

* There are a few magic constants throughout the paper. While conducting an ablation study may be cost-prohibitive given the sequential dependencies of the algorithm, it would be helpful to the reader to provide some means of estimating the impact of and sensitivity to these parameters.

---

Nits:
* The comment on maximum test set size on page 1 is a bit strong. Test set sizes are limited by financial incentives. For potentially lucrative endeavors, it would not be surprising to find a test set larger than 10k images.

* There is a typo on page 2: "not be[en] spotted beforehand"

* It would be nice to quantify the computational cost of constructing M in each iteration.

* Consider moving the superscript 4 after the period to make it clear that it is a footnote and not exponentiation.

* There is a typo on page 4:(ARC) -> (ACR)

* The F in "failure" is erroneously capitalized in the first paragraph of section 3.2

* You may wish to consider citing the field of computer-assisted annotation as relevant work. One such paper (among others) and open-source implementation include:
** Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++ (Acuna et al)
** https://developer.nvidia.com/blog/annotation-transfer-learning-clara-train/
review_title: Interesting Premise, Could Use Further Refinement

final_score: rating: 3: Clear rejection
review_content: Annotating images for training of segmentation models is time consuming and it can be difficult to annotate enough examples to ensure good performance on the rare difficult examples that often occur when methods are applied to real world data. These cases are referred to as corner-cases. The paper therefore proposes a measure based on the discrepancy of a group of segmentation models to identify more valuable images to annotate and add to the training data in a iterative fashion. The approach is tested on the PASCAL VOC database.

The problem is important and relevant. I find the motivation clear, however, I disagree somewhat with the reasoning in parts of the introduction (see detailed comments). The approach is reasonable, and there is some evidence that performance is improved on these specific corner-cases by the addition of similarly identified corner-cases to the training set (Table 1). I am not sure how the approach would generalize, however. As I understand it the test-sets where performance is improved consists entirely of corner-cases identified in a similar manner, so whether the approach will generalize, depends entirely on how representable corner-cases identified in this manner are. It would have been much better to use a independent dataset of corner-cases identified manually.


Detailed comments

- "First, segmentation benchmarks require pixel-level dense annotation", I do not believe this is necessarily true, and there is little need to state this. One could certainly think of useful benchmarks with hard examples only. There are also examples of benchmarks where the groundtruth consists of computer segmentations corrected by humans.

- "Second, it is much harder for segmentation data to be class-balanced in the pixel level, making highly skewed class distributions notoriously common for this particular task", "Besides, the “universal” background class (often set to cover the distracting or uninteresting classes (Everingham et al., 2010)) adds additional complicacy to image segmentation (Mostajabi et al., 2015).", while this may be true for training datasets, I do not see how this is a problem for benchmarks necessarily.

- I find the description of the construction of the test dataset used in the different iterations unclear. It is my understanding, but I am not actually sure so it would be good to have the approach clarified, that the test datasets ($T^{(1)}$, $T^{(2)}$, and $T^{(3)}$) of iteration 1, 2, and 3 are hard examples, and are thus biased towards the methods involved. That is, they consist of examples that the proposed segmentation model disagrees with the "competing" models the most on. It is clear from the Table that these images are selected both based on mistakes of the competing models and mistakes of the target model. After one and two iterations we see that the target model now does much better on the next iteration of hard examples, but we really do not know how representative these hard examples are. If the methods tend to disagree on a limit number of typical cases, then these cases will be added to the training set and it is not so surprising that improvements in the target model is seen. To evaluate how this approach generalizes to the real world, an independent dataset would have to be used.

- "This also provides direct evidence that existing segmentation models could be particularly weak at certain real-world generalization, which is not surprising because the 1,464 training images are deemed to be extremely sparsely distributed in the space of natural images." I am not sure I agree with the evidence part. The dataset in question is selected to be hard (as far as I understand), so it is not surprising that the methods perform worse on it and does not say much about generalization. Essentially this is just evidence that the selection procedure is working as intended.

Clarity
While I believe I understand most of the manuscript to an acceptable level, it contains quite a lot of sentences that would benefit from some editting. Some examples below.

- Introduction is wordy, with long and difficult to understand sentences. Words that mean different things than the authors probably intended are also used, see examples "boosting", "spot", "cover" and wrong words are often used.

- "While the performance of segmentation models, as measured by excessively reused test sets (Everingham et al., 2010; Lin et al., 2014), keeps boosting", keeps boosting is a bit of a weird phrase here, "keeps improving" perhaps?

- "suggesting their insufficiency to cover hard examples that may be encountered in the real world", maybe replace "insufficiency" and "cover" with "inability" and "handle".

- "such test sets may only spot an extremely small subset of possible mistakes that the model will make", "spot" is likely the wrong word to use here, maybe contain? But even so, test sets do not contain mistakes, the methods possibly make mistakes on the test set. Consider rewording.

- "The existence of natural adversarial examples (Hendrycks et al., 2019) also
echos such hidden fragility of the classifiers to unseen examples", while I could guess at what the sentence means, it does not really make sense.

- "which possess inherent transferability to falsify different image classifiers with the same type of errors", not sure what you mean by this.

- "It is clear that images in S are visually much harder.", something is missing.

- "weakly labelling method of filtering", what do you mean by this?

- "Specifically, given the target model $f_t$, we let it compete with a group of state-of-the-art segmentation models {g_j}^m_{j=1} by maximizing the discrepancy (Wang et al., 2020) between f_t and g_j on D." They are not really competing are they? The point, as I understand it, is not to select the best model, but to find the most "controversial" image.

- I would have prefered legends in each figure, as opposed to having to scroll up and down to find the relevant information.

- "indicating that many images in $T^{(1)}$ are able to falsify both the target model $f_t$...", the images are not really falsifying the model.

- "This suggests that the target model begins to introspect and learn from its counterexamples", the word introspect appears to be wrongly used here (and elsewhere).

- "Moreover, the top-1 model on $T^{(0)}$ does not necessarily perform the best on $T^{(1)}$ , conforming to the results in (Wang et al., 2020).", what results are you talking about specifically? And I guess it should be "confirming".

Originality
I don't find the work very original and it is not clear to me if the work is very novel. A lot of literature is referenced under related work, but I find it to be mostly tangentially related. It would be good if the authors could describe how this specific problem has been addressed before in image analysis and particularly image segmentation. A google search brings a number of works on hard negative mining. But "human-in-the-loop" techniques such as interactive training [1, 2] also enable annotators to focus more time on harder examples. The methodology itself is not groundbreaking. Multiple trained models have been used in combination to assess prediction certainty previously and uncertainty has also been used in active learning setups to focus annotations on difficult regions [3]. I am sure there are even more relevant links, but this is what a couple of minutes googling brought up.

Significance
While the problem is relevant and the method possibly useful, because of previously mentioned concerns with respect to novelty and generalizability of the results, I do not think it will have a wide ranging significance.

[1] Gonda, Felix, et al. "Icon: An interactive approach to train deep neural networks for segmentation of neuronal structures." 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). IEEE, 2017.
[2] Berg, Stuart, et al. "Ilastik: interactive machine learning for (bio) image analysis." Nature Methods (2019): 1-7.
[3] Casanova, Arantxa, et al. "Reinforced active learning for image segmentation." arXiv preprint arXiv:2002.06583 (2020).


review_title: The work promises more than it delivers. Approach could be useful, however, experiments and discussion of prior work appears to be lacking.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work used a variety of existing segmentation algorithms to discover most "controversial" samples from massive online unlabeled images. Those representative controversial samples were believed to have the best chance to confuse the algorithm being trained and to expose its weakness. They are rated by annotators on a spectrum from bad to excellent, and segmentation masks are collected from human annotators for the “worst” images. Several clever measures were taken to reduce human labor. 

The paper addressed an important and somewhat overlooked problem for segmentation and deep learning in the general. Leveraging those counterexamples to improve the segmentation models' generalization performance on unseen images seems to be novel in this field. This looks like a special case of finding natural adversarial examples from unlabeled data. The proposed solution is intuitive and logical, with lots of practical considerations made for its feasibility. The manuscript is also very well written, and the literature review is especially comprehensive. 

This work extends from MAD, ICLR 2020 (https://openreview.net/forum?id=rJehNT4YPr); but it also presents with two nontrivial and interesting innovations: (1) generalizing it to a dense prediction task, which requires revising the human labelling strategy in subjective experiments. Weakly-supervised labeling is more practical for segmentation; and (2) extending to active training/tuning, leveraging the selected hard examples to improve the segmentation model for multiple rounds. The tuned models were shown to improve their robustness remarkably on spotted catastrophic mistakes, while preserving their performance on canonical testing sets. 

This approach is also an instance of the basic active learning paradigm that iterates between spotting hard examples, labeling them, and tuning the model. The method of finding hard examples by competing with a human oracle seems to be novel though. 

I noticed in the Appendix, the authors also compared their method with entropy-based active learning – a vanilla baseline needing no competing model. Their model seems to achieve a good mIoU advantage, showing the new proposed way has better performance on open-world images than simple entropy-based fine-tuning. That is perhaps understandable, since the proposed new method also costs more human inspection efforts. However, I’d be interested to see the authors to compare the hard examples selected by their method and by entropy-based one: are there any distinct pattern or notable trend? Is there anything that the entropy-based method can clear miss but your method can pick up?

Besides, this paper uses different deep networks trained as competing models. Why wouldn’t those models more tend to make similar mistakes, due to same training dataset and model type? What if using learning-based but non-deep segmentation models to compete? What if using segmentation models trained on other datasets to compete?

review_title: Official review of "Efficiently Troubleshooting Image Segmentation Models with Human-In-The-Loop"

Meta-review: This paper studies how to efficiently expose failures of "top-performing" segmentation models in the real world and how to leverage such counterexamples to rectify the models. The key idea is to discover most "controversial" samples from massive online unlabeled images. The approach is sound, well grounded, and quite logical. Results demonstrate the effectiveness.

However, there exists some limitations coming from R2 and R3, for example, 1) Segmentation benchmarks may not require pixel-level dense annotation. There are also examples of benchmarks where the groundtruth consists of computer segmentations corrected by humans. 2) It is much harder for segmentation data to be class-balanced in the pixel level, making highly skewed class distributions common for this particular task. 3) Citing the field of computer-assisted annotation as relevant work.

In the end, I think that this paper may not be ready for publication at ICLR, but the next version must be a strong paper if above limitations can be well addressed.
Decision: Reject