Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors studied the "object-centric" robustness of various instance segmentation models. They used stylized objects/background to evaluate the performance which is adopted from a work by Geirhos et al. [1]. The authors performed large number of experiments to evaluate different aspects of instance segmentation models including framework, architecture, and pre-training.
main_review: Strengths:

The paper is a good read, and easy to follow. The paper also covered many of the popular instance segmentation frameworks and architectures. This is a good reference for researchers who address the object-centric robustness.

My main concerns are the following:

1- The paper is heavily influenced by the work in Geirhos et al. to the point that makes this paper a small incremental work.
2- The authors missed providing some explanation for the results sometimes. (e.g. yolact is consistently more robust against stylization compared to other architectures, yolov4-csp is more robust compared to rest of yolo models, and Swin being the most robust when combined with maskrcnn framework).
3- Since the paper is heavily relying on the stylized images for evaluation, and since the models are trained with different data augmentation strategies which might have influenced the robustness against stylization, I am surprised that the authors didn't study how data augmentation would influence the robustness metric. For instance, would we consider yolact more robust because of some data-augs used during training, or is it because of something else?



summary_of_the_review: The paper is a good study of existing deep-learning instance segmentation models, and frameworks. I enjoyed reading the paper. The authors put a lot of effort on performing large number of experiments to benchmark various models, however they put little effort on deeply studying and analyzing the results and they left it to the reader to interpret the results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a new setting for instance segmentation and compares a set of literature instance segmentation methods on this new setting. The new settings are: 1) use AdaIN (a style transfer method) to create stylized image, 2) only perform AdaIn on objects (use COCO's ground truth instance mask), 3) only perform AdaIN on background. This paper presents the instance segmentation AP numbers for many state-of-the-art methods (e.g. mask rcnn, swimtransformer, DETR) on the three settings with different ratio of the style transfer.
main_review: Strengths:
+ This paper conducts a large number of experiments and presents a comprehensive instance segmentation literature review.
+ The figures are very clear.

Weaknesses:
- I don't see any technical contribution for the method presented in the paper. It simply alters the dataset with existing style transfer method and perform other literature methods on the altered dataset.
- The main experiment that this paper presents is to use the style tranfered dataset. However, I don't see any user cases for this setting. Besides, the dataset is modified by using groundtruth instance labels, which makes the experiment make less sense.
summary_of_the_review: I prefer to reject this paper. My main concern of this paper is the lack of contribution to the community. To me, this paper mainly conducted a set of experiments of existing methods on a new setting. And the new data setting doesn't make a lot of sense since the groundtruth is encoded in the image and there's no user cases for such setting.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors study whether deep learning based techniques for instance segmentation are robust to changes in object texture or contour. This is inspired from a previous work in ICLR 2019 where the authors perform a similar study for image classification and show that existing CNN models learn classes based on texture and do not take the overall shape into account.

The authors take the task of instance segmentation where it is necessary for the model to learn the overall shape and measure the impact of texture changes. They use a stylized version of coco but also create two different types, one stylizing only the objects and the other stylizing only the background. Moreover, for each dataset type they create 10 variants by varying the blending weight. The evaluation is done on four dataset types, default coco val, stylized coco (everything stylized), stylized objects and stylized images. Except the default coco, every dataset type has 20 variants (different levels of blending plus pixel space vs feature space blending) which leads to a total of 61 datasets.
Following this, the authors perform an evaluation of various instance segmentation techniques. They take three paradigms - architectures (backbone / neck), frameworks and pre-training approaches and study the robustness of each to changes in texture.

main_review: Strengths:
The authors compare a variety of instance segmentation techniques along different axes like frameworks, architecture and pre-training

Concerns:
My concern with the approach is that because it uses ground truth masks to generate different versions of datasets, it creates a bias where the features are corrupted, and this affects the performance of detectors. The RGB features inside the object masks are not changed in the stylized object dataset (apart from the minor corruption of features from the background), so it is expected that detectors would perform best in this case. When we start changing the RGB features of the objects but keep the background as is, the performance degrades, although not entirely because object boundaries would still be visible to some extent. Finally, when the entire image is stylized, the boundaries get mixed up and there is very little signal in the image (as can be seen Figure 2), so the detector does not work in this case. Overall, it remains unclear to me if the method is adding any insights compared to what is not already expected. I expect similar problems if any other transformations (like gaussian noise, salt/pepper noise/blur etc.) are applied to COCO images on object masks, background and entire image. Also, its unclear if there is something special about using stylized transformations to make claims that object detectors are learning shape/contour specific features - the conclusion which I am taking here is “the more noise we add, the worse object detectors get”, more specifically, noise everywhere > noise on object > noise on background.

Other Comments:
Figure 6 is very dense and hard to read. What are the light/dark lines in stylized coco/ stylized objects?

The authors should include 20-30 examples of stylized images in the paper or the supplementary material. This will give better insights to the reader about how the dataset looks in different cases.

summary_of_the_review: There are gaps in the experimental design based on which we are concluding what features object detectors learn.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an evaluation on object-centric robustness of existing instance segmentation models, where an object-centric stylized coco val dataset is constructed. Then, a large number of experiments and evaluations are performed on this new dataset to study the impact of framework, architecture and pre-training.
main_review: Strength:
1. A new object-centric evaluation dataset based on COCO val is constructed for evaluating the robustness instance segmentation models.
2. The robustness evaluation is mainly evaluated from 3 impacted dimensions, which are well categorized. Also, a huge number of evaluations and comparisons are conducted.
3. The paper is presented in a clear structure and easy to understand.

Weakness:

1. The paper evaluates the robustness of instance segmentation, mainly from the image/feature corruptions perspective. However, in real life, the robustness for instance segmentation should be more about the adaptation to novel objects [a,b] in a long-tailed distribution world or how to handle overlapping objects with similar appearance in heavy occlusions [c,d]. There should be some relations discussion with existing instance segmentation works that also focus in robust instance segmentation but from different directions.

2. Instance segmentation models can be divided into backbone, neck and functional heads. The backbone and neck have been studied, but what’s the influence of the mask head design for instance segmentation? Some popular mask head designs, besides the Mask R-CNN, include BMask R-CNN [e] and BCNet [d], which also need to be evaluated and discussed.

    [a] Learning to Segment Every Thing. CVPR, 2018.

    [b] Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation. ECCV, 2020.

    [c] Multi-instance object segmentation with occlusion handling. CVPR, 2015.

    [d] Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers. CVPR, 2021.

    [e] Boundary-preserving mask r-cnn. ECCV, 2020.

3. The paper uses the image styling to simulate the image corruptions. How to guarantee it is realistic to real image corruptions with practical usages? For example, in Figure 1: right, a clear separation between background and foreground is observable in Stylized Object and Background but that is not common in real world, which may make the object detection and segmentation easier. I think some discussions on how to generalize the simulation to realistic world corruption cases should be added.

4. The ambiguous usage of term 'architecture' vs. 'framework'. The 'architecture' should be replaced with 'backbone' instead, as in the first column in Table 1, most variants are R50, R101, Swin-T, etc. Usually, the usages of 'architecture' and 'framework' are similar.

summary_of_the_review: My concerns are listed in the weakness section. If the authors can well address them. I would consider raising my score.

After reading the updated paper, my concern about the mask head influence is addressed. Also, I have read a new intro and abstract with better motivation connecting to the real-world instance segmentation challenges. Although image corruption using style transferring has no real-world cases, it does create novel object/background context. The experiments conducted are extensive and it provides some suggestions of instance segmentation model design for environment robustness. Thus, I decide to increase my rating to 6.
review_title: null

Meta-review: This paper finally received divergent and borderline reviews with two positive (6) and two negative (3) rates. After the thorough reviews by ACs ourselves, we would like to decide to reject this work at this time, even though this submission has a lot of potentials including intensive analyses on instance segmentation frameworks and architectures.

We first would like to appreciate comprehensive author’s responses and additional empirical results. They should be extremely helpful to make this submission stronger. Here are some of our suggested points for improvement: (i) The novelty, significance, and practical implications of this work (compared to previous analysis work) may need to be better presented in a more persuasive way. (ii) Nuance of stylization transformation can be better explained compared to other types of perturbations or transformations. (iii) Empirical fairness can be better justified. (iv) Since the paper is written in a highly condensed way, some of reduction may improve the readability. (v) Finally, given that this paper focuses on empirical study about instance segmentation, it may be more appreciated in a computer vision venue.
Decision: Reject