An object-centric sensitivity analysis of deep learning based instance segmentation
Authors: Johannes Theodoridis
Jessica Hofmann
Johannes Maucher
Andreas Schilling
Venue: ICLR 2022 Submitted (2022)
Keywords: robust vision
instance segmentation
deep learning
object-centric
robustness
sensitivity analysis

Abstract: In this study we establish a comprehensive baseline regarding the object-centric robustness of deep learning models for instance segmentation. Our approach is motivated by the work of Geirhos et al. (2019) on texture bias in CNNs. However, we do not compare against human performance but instead incorporate ideas from object-centric representation learning. In addition, we analyze and control the effect of strong stylization that can lead to disappearing objects. The result is a stylized and object-centric version of MS COCO on which we perform an extensive sensitivity analysis regarding visual feature corruptions. We evaluate a broad range of frameworks including Cascade and Mask R-CNN, Swin Transformer, YOLACT(++), DETR, SOTR and SOLOv2. We find that framework choice, data augmentation and dynamic architectures improve robustness whereas supervised and self supervised pre-training does surprisingly not. In summary we evaluate 63 models on 61 versions of COCO for a total of 3843 evaluations.

Under review as a conference paper at ICLR 2022
AN OBJECT-CENTRIC SENSITIVITY ANALYSIS OF DEEP LEARNING BASED INSTANCE SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding if and how instance segmentation models generalize to novel appearances of objects is crucial to many challenging real-world problems such as occlusion or in low-resource domains. In this study we contribute compre- hensive baseline regarding the object-centric robustness of deep learning models. Inspired by work on texture bias CNNs, we introduce a novel negative test and simulate the out-of-distribution appearance of familiar objects with stylized version of MS COCO and two counterfactual variants. In addition, we carefully control for potentially confounding corruptions that can lead to disappearing ob- jects in the extreme. The result is a causally well motivated sensitivity analysis. We evaluate broad range of frameworks including Cascade and Mask R-CNN, Swin Transformer, BMask, YOLACT(++), DETR, BCNet, SOTR and SOLOv2. We find that YOLACT(++), SOTR and SOLOv2 are significantly more robust corruptions and novel texture compared to competitive models (~ 10 - 30%). In general, we show that deeper and dynamic architectures improve robustmess whereas training schedules, data augmentation and pre-training have only minor impact. In summary we evaluate 68 models on 61 versions of COCO for a total of 4148 evaluations.
INTRODUCTION
In this study, we want to learn how deep learning models for instance segmentation generalize to novel appearances of familiar objects. Despite their remarkable success in computer vision, deep neural networks still struggle in challenging real-world scenarios ( (Yuille & Liu 2018 Michaelis al., 2019 Recht et al.) 2019 Madan al., 202T). Consider for instance a pedestrian with an unconventionally textured dress or rare horse statue made from bronze. The model might have seen many pedestrians or natural horses during training but still fails to detect these unseen and out of context examples, often with high confidence. Generalizing to such naturally adversarial objects typically described as -uttr--distribution robustness the literature (Hendrycks et al. 2021 Lau al. 202I). Interestingly. Hendrycks et al. (202I) suggest that improvements in this direction are more likely to come from computer vision architectures than from existing data augmentation or additional public datasets. With the ever increasing number of competitive models, our objective is therefore to contribute an extensive comparison of instance segmentation models to unveil promising future research directions. The perspective we take is inspired by the work of Baker et al. (2018) and Geirhos et al. (2019) on texture bias in classification models. More precisely. both groups found that when compared to humans, convolutional neural networks (CNNs) mostly ignore object shapes, i.e. ignore the defining structure of an object. In fact, Brendel & Bethge (019) have further shown that CNNs can robustly classify objects in texturized images where the global appearance of objects is fully mixed up. In summary, these findings indicate that revision of contemporary deep learning architectures is a much needed contribution on the pathway to more systematic generalization. To compare the robustness of instance segmentation models, the form of an object-centric sensitivity analysis. More precisely, we investigate the impact of increasingly novel object texture while controlling for the effect of corrupted color, shape and contour features. By object-centric, we mean that the appearance of visual objects is changed in a semantics preserving and causally plausible way, i.e. is controlled at the instance level.
introduce novel negative test in
Since we generally expect that segmentation performance will degrade under novel textures and increasing corruptions, our benchmark can be understood as a negative test, i.e. if a specifie model

---

Under review as a conference paper at ICLR 2022
content image (COCO)
styl image (Painter by Numbees))
style trsndfer (AlalN)
styloal image (Styliaed COCO)
mask ammctation
Stylized COCO
Stylized Objects
Stylized
COCO
Background
Figure 1: Left: Simplified creation process of the Stylized COCO dataset. Style images are randomly chosen from Kaggles Painter by Numhers dataset. Right: We use mask annotations to create counterfactual, object- centric versions of Stylized COCO. We append more examples of the creation process in the Appendix.
appears to be significantly more robust than others, we can consider it a promising candidate for more in depth research on generalization. To simulate novel appearances of familiar objects we utilize stylized version of MS COCO (Lin et al. 2014) as shown Figure left. As can be seen, AdaIN method (Huang Belongie 2017) effectively replaces local texture cues but preserves the global shape of objects. In addition, we create two object-centric versions of the stylized COCO dataset as displayed in Figure I righht. By comparing the performance on all three datasets we can control for spurious correlations that might have been introduced in the fully stylized dataset. Note that the style masking encodes ground truth information from the test set in the data which could potentially be exploited by the models. While this would be interesting finding in itself, do not see this to happen in practice. A limitation of our approach is the use of artistic style images. This leads novel object appearances that not be found in real-life directly. However, compromise on this choice since it ensures that textures are truly novel and not biased which could happen with natural style images. We discuss more alternatives our simulation method in the related work section.
A closely related perspective on segmentation robustness can be found in work that acknowledges the inherent long-tail distribution of real-world data. In such settings, the challenge is to become robust against the bias of extreme class imbalance in existing datsets. Common approaches re- solve this issue are re-sampling (Wang et al.] 2020b Chang et al. [202] 1) and regularization strategies (Tang et al. 2020; Pan et al. 2021| Hsieh et al., 2021 Wang et al.) 2021b). The long-tail problem can also be understood missing. In such cases, the challenge is to efficiently adapt to novel objects or uncommon appear- Fan Fan al. 2020). A particular instantiation of this problem is object occlusion. Since objects can occlude each other in almost infinitely different ways, common strategy is to model object features more explicitly, i.e. to decouple shape and appearance for instance (Chen et al. 2015 Cheng et al.) 2020: Ke et al. [022] Fan et al.) 2020). The idea is learn representations that generalize more systematically and we expect these methods to be strong contenders in our comparison. We hope that our comprehensive benchmark motivates more research in this direction which we believe, will lead to improvements in all of the related problem spaces.
low resource setting where data collection is expensive or class labels are
ances in semi-supervised manner (Hu et al. 2018b
METHODS
In this section we describe the datasets, frameworks and models that are used in this study. The code to reproduce our results, as well as Rs 1.5TB of detection and evaluation data can be found here: Link-to-project-PP0 -aapee-weeee-eeee-eeeeeeee
2.1 AN OBJECT-CENTRIC CAUSAL VERSION OF STYLIZED COCO
Stylized COCO as shown in Figure left is an adaptation of Stylized- ImageeNe by Geirhos et al. 2211)). It was first used by Michaelis et al. 2019 aa data augmentation technique during training to improve robustness of detection models against common corruption types such as gaussian noise motion blur. We instead tation models on this data directly. By manual inspection of Stylized COCO, we found that strong
stylized version of the val2017 subset test instance segmen-
Stylized Datasets potp L/ght it www...ooo/o//....oooooooo..//..a bet hige wlo/ wwb//ttty ine fine-dette wost
2

---

Under review as a conference paper at ICLR 2022
stylization can sometimes lead images where the object contour starts to vanish, up to the point where objects and their boundaries dissolve completely. This effect depends on the style image used in the creation process but affects objects of all scales alike. As shown in Figure 2] we resolve this issue by using the ground truth mask annotations to limit the style transfer to the actual objects or the background. This not only ensures that object contours preserved but also controls for global stylization as a confounding variable. By assuming an object-centric causal model, Stylized COCO allows us ask interventional questions regarding the original COCO dataset, e.g. "What happens if we change the texture of images?". By masking the style transfer to objects or background, we can also ask counterfactual questions such "Was actually the object that caused the change in performance?" " "What if we change the background instead?". We will refer to the different dataset versions as Stylized COCO Stylized Objects and Stylized Background
Figure 2: Depending the style image, object boundaries can van- ish due to strong stylization. The Stylized Objects and Background versions of Stylized COCO resolve this issue.
COCO instances
Stylized COCD
Stylized Objects
Stylined Background
related, second problem that remains after masking that shape information within the object can also be lost due strong stylization. We address this issue by controlling the strength of the AdaIN style transfer. This can be done with an a parameter that acts as a mixing coefficient between the content and style image. More precisely, AdaIN employs a pre-trained VGG encoder f on both images, performs an interpolation step between the resulting feature maps and produces the final output with learned decoder network g. In summary, a stylized image is produced by T(c, 8, a) g(1 - (c) + (dAa)N((((c) [())) where c and are the content and style images respectively. We will refer to this method as blending in feature space, The top row of Figure shows two examples of the extreme points a = 0 and a = 1. Note that at a O, the image colors are mostly preserved but the algorithm has already introduced artifacts the form of subtle texture and shape changes. In response we create control group where we perform alpha blending between the pixel values of the original content image and the fully stylized image fa=1: P(c, fa=1, a) a(1 + fa=1). We will refer to this method as blending pixel space. In contrast to the feature space sequence, the control group should preserve textures and object shape over longer range. The idea is to compare models on both sequences in order to attribute performance changes to either color and texture or shape, depending on the objects size. In contrast to Geirhos et al.) (2019) who used fixed style strength to modify ImageNet features (a 1), we produce the full alpha-range a E (0.0, 0.2, ..,, 1.0)
for both blending spaces. Note that every alpha value depicts separate and complete version of the accordingly styled COCO va 2017 subset. The qualitative differences can be inspected in Figure 3 bottom left (zoom in for better visibility).
coco image
sple 0
x
ooco imaaee
2-
styla-1 =1
Slanding siplas
Imagee gradicnts (Sube) grayazale) and RGB hiskggrm
Figure 3: Top row: Comparison of COCO and Stylized COCO at different alphas. The AdaIN method in- troduces subtle artifacts even at & = (no style). Bottom left: We control the style strength in feature space (yellow to pink) and pixel space (blue to pink). Note that every alpha value in these sequences depicts a separate and complete version of the accordingly styled COCO va 1207 subset. Bottom right: Comparison of image gradients and color histograms different alphas.
We also went for quantitive measures to validate our subjective impression of Stylized COCO. Figure 3) bottom right shows a comparison of image gradients and RGB histograms the extreme points. Compared to the original image we can see the subtle shape changes in the gradient map of 1. To describe this effect over the full
0 and significantly different color histogram
3

---

Under review as a conference paper at ICLR 2022
alpha range. we compute the structural similarity index (SSIM) (Wang et al. 2004 between gradi- ent images of corresponding image pairs. Between RGB histograms we compute the Wasserstein distance alike. We always compare against the original COCO data and report the mean distance averaged over the full dataset at a specific alpha. In addition to the image-to-image scores we also include an instance level comparison for the COCO scales S,M and I. Instances have been cropped based on bbox information. This addition was added after we observed that small objects appear to be more affected by the AdaIN artifacts compared medium and large instances. Figure 4 dis- plays the results and confirms our assumption. Structural similarity depends on object size and is in fact, almost constant over the full feature space range for small objects. Furthermore, the con- trol group preserves structural similarity over longer range as intended. Color distance in contrast converges at around a = 0.3. Based on these insights, we feel confident to better attribute potential performance dips and subsequently, determine the relative importance of each feature type.
Figure 4: Left: Average structural similarity between image gradients in relation to COCO score of means that there is no difference between images). Right: Wasser- stein distance between RGB his- tograms (reversed y-axis).
08-
dirising Space
65-
05-
Widersa
0.4
Mebic
02-
01
05
24
35
06
27
00 50
011
02
0.
04
05
07
00
Bundig apila
2.2 MODEL SELECTION
To contribute a comprehensive overview on model robustness, we opted for a broad comparison of popular frameworks and architectures. The dimensions we consider to be impactful are framework, architecture and pre-training. The selected models can be found in Table
Frameworks for instance segmentation can be categorized in different ways. first distinction can be made between methods that solve the detection problem as refinement process of box pro- posals (multi stage) and methods that predict bounding boxes directly (one stage). We include the popular multi-stage frameworks Mask R-CNN (He et al. 2017) and Cascade Mask R-CNN (Cai & Vasconcelos) 2018) that uses multiple refinement stages instead of one. Both frameworks formulate instance segmentation Faster R-CNN (Ren et al. 2015) can ignore object boundaries and shapes, we include the boundary- preserving mask head alternative (Cascade-) BMask (Cheng al., (020) for comparison. A remain- ing challenge to boundary detection are overlapping objects that occlude the ground truth contour of other instances. We therefore include the Bilayer Convolutional Network (BCNet) (Ke et al., 2021 as another mask head alternative. In BCNet, the occluded and occluding object are separately detected and modeled explicitly in a layered representation. The mask head can then "consider the interaction between [the decoupled boundaries] during mask regression (Ke et al. 2021)) ," A second distinction between frameworks concerns the use of predefined anchor boxes. Anchor based meth- ods predict relative transformations on these priors whereas anchor free methods predict absolute bounding boxes. We include YOLACT(++) (Bolya et al.) 2019 2020) as a one-stage, anchor based framework. YOLACT is real-time method that solves instance segmentation without explicit lo- calization (feature pooling). Instead, it generates prototype masks over the entire image which are combined with per-instance mask coefficients to form the final output. The (++) version improves by adding a mask re-scoring branch (Huang et al. 2019a) and deformable convolutions (v2) (Dai al.) 2017: Zhu al. 2019). We include DETR (Carion et al. 2020) as a one-stage, anchor free Tramework that formulates lates object detection as a set prediction problem over image features. Note that it was not primarily designed for instance segmentation but offers a corresponding extension that we use in our study. Based on model availability we include BCNet in the FCOS (Tian et al.) 2019] variant (F-BCNet). FCOS is fully convolutional, one-stage, anchor- frree alternative to Faster R-CNN that "solves object detection in a per-pixel prediction fashion, analogue to semantic segmen- tation (Tian et al. 2019) " Finally, we distinguish between top down frameworks where detection precedes segmentation and bottom up methods where bounding boxes are derived from mask pre- dictions. We include the bottom-up methods SOLOv2 (Wang et al. 2020d) and SOTR (Guo et al.) 2021). SOLO (Wang al., 2020c) divides the input into fixed grid and predicts semantic cat- egory and corresponding instance mask at each location. The final segmentation is obtained with non-maximun-suppeesioon the gathered grid results resolve similar predictions of adjacent
pixel-wise classification problem. Since this rather naive extension to
4

---

Under review as a conference paper at ICLR 2022
grid cells. SOLOv2 improves by introducing dynamic convolutions to the mask prediction branch, Le. an additional input dependent branch that dynamically predicts the convolution kernel weights. A similar idea was used by Tian et al. 2020b). SOTR uses a twin attention mechanism (Huang ee al., (019h) to model global and semantic dependencíes between encoded image patches. The final result is obtained by patch wise classification and a multi-level upsampling module with dy- namic convolution kernels for mask predictions, similar to SOLOv2. For completeness, we also include YOLO(V3,4 and scaled y4) our comparison since detection is vital sub-task of top down frameworks (Redmon & Farhadi 2018 BBohkovskiy et al. 2020 Wang et al., 2021a).
Table 1: Overview of frameworks, backbomes and neck methods. (*) Swin Transformer use hierarchical repre- sentations similar to FPN necks in CNNs. RegNetY is similar to RegNetX but implements the Squeeze-ann Excitation operation (Hu et al. 2018a). Yolo consists of darknet (D), spatial pyramid pooling (SSP) (He et al. 2014) and a Path Aggregation Network (PAN) (Liu et al. 2018 in varying combinations with CSPNet (C) (Wang et al. 2020a
Rackheer
16
RCSD
JOSS EUSH SATK
1:0
126
315
27N
1750
5wb-5
2750
Svb-3
Architectures used in instance segmentation can be divided into backbone, neck and functional heads. The latter output the final results and are framework specific. Backbones and necks however are typically chosen from a pool of established models which allows for a controlled comparison. The role of backbone networks is to extract meaningful feature representations from the input, i.c. to encode the input. The neck modules define which representations are available to the functional heads, i.e. define the information flow. We include the CNN backbones ResNet (He et al.. 2016), ResNext (Xie et al.] 2017) and RegNet (Radosavovic et al., 2020), a network found with meta ar- chitecture search that outperforms EfficientNet (Tan & Le 2019). Note that BCNet utilizes a Graph Convolutional Network (GCN) (Kipf & Welling (217)) within its mask heads to model long-range dependencies between pixels (to evade local occlusion). Furthermore, DETR and SOTR are hybrid frameworks that use transformer architectures to process the encoded backbone features. With Swin Transformer (Liu et al., 2021 we also include convolution free backhone alternative based on the Vision Transformer approach (ViT) (Dosovitskiy et al. 202T). D. The most popular neck choice is the Feature Pyramid Network (FPN) (Lin et al. 2017). It builds a hicrarchical feature representation from intermediate layers to improve performance at different scales, e.g. small objects. For compar- ison we also include a ResNet conv4 neck (C4) as used in Ren et al.) (2015) and a ResNet conv5 neck with dilated convolution (DC5) as used by Dai et al. (2017). Finally we abbreviate FPN models that use deformable convolutions as DCN (Dai et al. 2017 Zhu et al. 2019). Similar to dynamic con- volutions which predict kernel weights, DCNs learn to dynamically transform the sampling location of the otherwise fixed convolution filters.
Pre-training of backbone networks commonly done as supervised learning on ImageNet (IN). Due to the recent success of self supervised learning (SSL) in classification, we are interested in how these representations perform in terms of object-centric robustness. In particular we are in- terested in the contrastive learning framework that seeks to learn "representations with enough in- variance be robust to inconsequential variations (Tian et al., (020a))" Based on availability we include the methods InstDis (Wu et al. 2018). MoCo ( (He et al., 2020) Chen et al. (020), PIRI. (Misra & van der Maaten 2020) and InfoMin TTaa et al., 2020a). Note that pre-trained backhones were only used as initialization for supervised training on COCO. As a final comparison we include models that have been trained with random initialization and Large Scale Jittering (LSJ) (Ghiasi et al. 202T) data augmentation an alternative to pre-training.
The complete list of models can be inspected in Figure 5 From the overview in Table I we can see that selection allows for fair comparison of frameworks long as we fix the backbone and neck architecture to RPNN Vice versa we can compare backbone and neck combinations within fixed framework, in particular Mask R-CNN. Note that we also compared different learning
5

---

Under review as a conference paper at ICLR 2022
schedules but did not include this dimension more prominent in the categorization after finding it to be the least significant factor in our evaluation.
3 RELATED WORK
The classical understanding of robustness is concerned with corruptions that stem from signal processing errors such as gaussian or salt and pepper noise etc. (Hendrycks & Dietterich 2019
Michaelis ct al. 2019; Kamann & Rother 2021 Muummadi et al.
2021). A popular alternative is to understand robustness by comparing model performance and failure cases against humans (Geirhos et al. 2018 2019; 2020; 2021; (Shankar et al. 2020 Tuli et al.]
(02T). Geirhos et al. (2021) and
Tuli et al. (2021) for instance find that Transformer models perform more consistent to humans than CNNs. Madan et al. (202T) the other hand report that both model types are prone to small in- distribution changes in 3D perspective and lighting. We contribute to this matter by including both architectures in our comparison. A third perspective is given by work that aims to compensate for the long-tail distribution in world data. In this case, robustness can be understood as the ability to adapt novel object classes uncommon appearances in data efficient transfer learning process (Hu et al. 2018b Fan et al. 2020). The approach we take is more direct and measures robustness in a challenging zero-shot generalization setting. As an alternative to our sensitivity benchmark, Islam et al. (2021) analyze feature importance in latent representations and Cao & Johnson (2021) use feature visualization to understand object detectors. Both leverage style transfer to simulate novel object appearances, The closest real-life alternative is the Natural Adversarial Objects (NAO) dataset (Lau et al. 202T). poses a more realistic long-tail setting but does not allow to control for pose and perspective, i.e. to observe the exact same object with varying textures for instance. Other alternatives are the 3DB framework (Leclere et al., (02T), a rendering engine that enables artifact free texture transfer on synthetic objects and S1-Score (Yung et al. 2021). dataset for analyzing robustness to rotation, location and size.
4 RESULTS
In this section we present the evaluation results on Stylized COCO, Objects and Background. Each dataset version contains 20 copies of the accordingly styled COCO va 1 112017 subset. As a reference point, we first report the absolute Average Precision (AP) on the original va12017 subset for all models in Figure As can be seen, training schedule, data augmentation and architecture choice have the biggest impact within framework. Overall, RegNets trained with LSJ and Swin Trans- former models perform best. Note that SOTR and SOLO have worse APs but significantly better APm and API compared to other frameworks (see Appendix).
TAe
ER
20.5
NEOSOVECOLOR
65
9900 9XX
6:96 1068000000
25:9
REST SE
FRN
DCS
DCN
FPN
DON
FRN PAN GPAN
DON
DOS
FRN
FANDON
Figure 5: Absolute performances COCO va 12017. Training schedules in epochs have been appended to model names. Note that Yolo is bbox which is not directly comparable hut included for model completeness. Methods that do not report scores for va12017 have been validated on test rest-dev201' first.
We now present the results of our object-centric sensitivity analysis. We first provide a general overview of the full blending sequences and then focus on the extreme points a = 0 and a = 1 in (P on 61 datasets which sums up to 4148 subset evaluations. To select models for grouping in the overview and reporting in the de- tailed comparisons, we calculated the distance matrix between all models for each dataset. These can be found in the Appendix together with more metrics and the corresponding bounding box scores.
more detailed comparisons. In total we tested 68 pre-trained models
Appendix complete list code projects, weight sources.

---

Under review as a conference paper at ICLR 2022
Note that we focus primarily the AP score for segmentation and the scale dependent COCO met- rics APs, APm and API due to our object-centric perspective. To quantify object-centric robustness, report the relative performance in comparison the uncorrupted data. For every dataset version, blending space and alpha value we calculate: rPa Pa/ Pooeo; P E {AP, APs, A APm, API). Note that we release the exact numerical values together with our code since reporting this amount of data in the form of tables is not expedient.
Figure displays the average relative performance per framework and stylized COCO version for the feature space blending sequence (dark) and the pixel space control group (light, always starting from 100%). We can immediately see that YOLACT(++)) is consistently more robust (rAP) than all other frameworks due to its strong performance on small objects (rAPs). Similarly, the bottom-up frameworks SOTR and SOLOV2 are significantly more robust on medium (rAPm) and large objects (rAPI). Within the Cascade and Mask R-CNN frameworks, Swin Transformer are slightly more robust than their CNN based counterparts. To our surprise, the advanced BMask and BCNet models do not consistently improve robustness but rather closely follow the performance of the naive mask head designs. We investigate this not expectable ranking in more detail in Figure and provide some further explanation in our discussion.
Figure 6: General overview of model robustness (zoom in for better visibility). Masking the style transfer effect to ground truth objects results in im- proved relative performance without changing the overall trend. Structural similarity as displayed in Figure 4 seems highly predictive for the low al- pha range but can not explain the further decreas- ing robustness values. We also like to point out the small performance dip of the control group 0.1 that seems he induced by color shift. Surprisingly, increasing the style strength on back- ground fcatures appears to have a positive effect for large objects (rAPI). Swin Transformer are display as separate group due to their convolution free backbone architecture. Yolo bounding box rAP which is not directly comparable but included for model completeness. Note that absolute metrics re- sult in different ordering due varying base per- formance. We append this version together with the corresponding bhox variants the Appendix.
21
Note that our object-centric sensitivity approach allows to validate these findings in a causally rig- orous way for the first time. More precisely, it allows to distinguish between the impact of stylization artifacts (artificial signal noise) and novel object appearances (out of distribution tex- ture). This is not possible with only single style strength tests as typically done. In general, it can be expected that all models will degrade in performance with increasing style strength at some point. We can see however that after an initial performance loss a (subtle corruptions of original shapes and textures), models are fairly robust until actual out of distribution texture from the style image is introduced (starting around objects where the performance appears to be fully dominated by the artifacts of the AdaIN method. As a result, we can respect these insights in our conclusions. For instance, we now understand that for medium and large objects, texture is more important to instance segmentation models than ob- ject shape or subtle corruptions. From the gap between the pixel space control group and the feature space blending sequence we can further tell that subtle shape and texture corruptions are more severe than color changes. In addition, we can validate that our findings can be attributed to actual objects by comparing the performance relying on context information for small objects but not for medium and large instances. Again, this could be assumed but not be proven with the scientific evidence of only one, fully stylized dataset.
0.4). This effect applies to medium and large but not small
Stylized Objects and Background. In general, models are heavily
7

---

Under review as a conference paper at ICLR 2022
Finally, we like to point out that models perform significantly better on Stylized Objects than on the fully Stylized COCO. Within the limitation of our data creation process, we conclude that instance segmentation models do indeed consider object contour a predictive feature.
For a fair comparison between frameworks we fix the backbone and neck to ResNet-50 and FPN. Figure 7 displays the results from the feature blending sequence at again observe that object-centric stylization is important but does not change the general ranking. Similarly, small objects are more directly affected by shape and texture artifacts whereas medium and large objects degrade due to texture transfer. Between Cascade and Mask R-CNN as well as BMak and BCNet we find only minor differences. YOLACT(++) performs consistently better across all scales and SOLO and SOTR perform on par even better but struggle with small objects.
0 and 1. We can
DETR shows the opposite behavior.
MA
rAPs
rAPm
APP
Figure 7: Object-centric robustness by framework. Note that we compromised on R-101 for SOTR and F. BCNet, see Table for an overview of available backbone combinations.
To investigate the impact of different backbones we fix the neck method (FPN) and compare model pairs within a given framework. As can be seen from the rAP metric in Figure 8 (top), deeper models perform generally slightly better. The rAPm and rAPl metries follow this trend. We therefore append the results for rAPs were the behavior is less clear and surprisingly, often reversed. Note that Swin-T and S are comparable ResNet-50 and 101 in model complexity and perform best within the Cascade and Mask R-CNN framework respectively. BM and C-BM represent B Mask and Cascade Mask respectively.
SFTRALLO
OFTRAULO
Fasgrofg #30
raagrarg
raggrerg
Figure 8: Top: Object-centric robustness by backbone architecture (zoom in for better visibility). Models marked with * trained with LSJ. Bottom: Robustness by neck architecture. Deformable convolutions highlighted for better visibility. Note that Yolo scores are bbox and not directly comparable.
To investigate the impact of different neck methods we do the reverse and compare models by fix- ing backbones and framework. As can be seen from Figure 8) (botttom) FPN necks are typically the least robust option. However, FPN in combination with deformable convolutions (DCN) performs consistently best. Note that SOTR additionally implements dynamic convolution kernels. For small objects, DC5 necks improve the robustness on Stylized Objects (triangle). We also compare Yolo against itself and find that surprisingly. the simple v4-csp model (CPAN) is the most robust option. We provide hypothesis for these findings in our discussion.
8

---

Under review as a conference paper at ICLR 2022
Finally we investigate the impact of pre-training and strong data augmentation. As displayed in Figure 9 differ- ent types of initialization perform almost similar within few percent. This is surprising since we expected represen- tations perform more differently based on the objective used in pre-training, in particular from the contrastive learn- ing framework. On the other hand we can report that super- vised and unsupervised representations perform almost on par. Similarly, longer training schedules have little impact on model robustness but do not lead to strong overgeneral- ization either. comparison we find that random initial- ization with LSJ data augmentation performs best.
E.
Hacggedd
Figure 9: Object-centric robustness by pre-training and data augmentation.
DISCUSSION
The first key finding of our object-centric sensitivity analysis is that YOLACT and the bottom-up frameworks SOTR and SOLOv2 are significantly more robust than other frameworks. We suspect that the consideration of the entire image in combination with their mechanisms to represent mul- tiple mask prototypes to play a major role, though more in depth research is required to verify this hypothesis. To our surprise, the object specific mask heads BMask and BCNet did not improve bustness in a significant way. A naive explanation might be that their specialized representations do simply not generalize to out-of-distribution texture. However, it could also be the case that improved mask detections are just not well reflected in the averaged AP scores. Since we release all detection data, this hypothesis could be investigated by extensive result visualization in a follow-up work. In general we find that deep learning models for instance segmentation appear to be biased towards texture, similar to classification models (Geirhos al. 2019). More specifically, we show that mod- els are fairly robust to shape and texture corruptions after an initial performance dip. The remaining loss can then be attributed to actual texture transfer from an unknown domain. This effect appears to scale with instance size and we hypothesize that the difference between shape and texture simply collapses for (very) small objects in MS COCO. In contrast to Geirhos et al. (2019) who argue that this problem is induced by the ImageNet training data, we find no such correlation in terms of pre- training. We conclude that either a similar bias is induced by the COCO dataset or that architectures simply not flexible enough to cope with feature variability, independent of training regime. The latter hypothesis was formulated by Greff et al. (2020) as the binding problem and concerns "the inability [of neural networks] to dynamically and flexibly combine (bind) information [... which] limits their ability to [...] accommodate different patterns of generalization". Our comparison of fixed and dynamic architectures supports this hypothesis as we find that deformable and dynamic convolutions lead to improved robustness consistently. We consider this another promising research direction towards systematic generalization.
CONCLUSION
In this study we contribute a comprehensive baseline on the object-centric robustness of instance segmentation models. We first introduce novel negative test on Stylized COCO in the form of an counterfactual sensitivity analysis. Based on broad selection of frameworks and architectures, we test 68 models on 61 dataset versions for total of 4148 subset evaluations. We find non-trivial ranking of frameworks with YOLACT(++), SOTR and SOLOv2 performing best. In addition we show that dynamic and deeper architectures improve robustness. In contrast, training schedules, data augmentation and pre-traaee teetriiningg have only a minor impact in comparison. We discuss limitations of our approach and provide interesting future research directions in our discussion.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We want thank the authors of Detectron BMask, PyContrast, Swin, YOLO, YOLACT(++), DETR, BCNet, SOTR and SOLOv2 for their contribution of publicly available model weights. With- out their commitment to open research, this study would not have been possible. We follow their example and publish all code and data used in this study to ensure reproducibility and more research moving forward.
The code to reproduce results Stylized COCO, Stylized Objects and Stylized Background can be found here: of detection and evaluation data. We include the latter to minimize GPU availability factor for future research. To improve actual reproducibility, we also provide automated environment setups for the dataset creation and each of the instance segmentation frameworks.
pubbishhdd shed. We also provide z 1.5TB
limiting
REFERENCES
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J. Kellman. Deep convolutional networks do not classify based on global object shape. PLOS Computational Biology. 14(12): e1006613, December 2018. ISSN 1553-7358. doi: 10.137 /
Alexey Bochkovskiy, Chien Yao Wang, and Hong- Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. CoRR, abs/2004. 10934, 2020.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. YOLACT: Real-time instance segmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. YOLACT++: Better real-time instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020. doi: 10.1109//PAMI..000..00
Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of-local-Featuue models works surprisingly well on ImageNet. In International Conference on Learning Representations, 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Ang Cao and Justin Johnson. Inverting and understanding object detectors. CORR, abs/2106. 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, volume 12346, pp. 213-229. Springer International Publishing, Cham, 2020. ISBN 978- 978-3-030-58451-1 978-3-030-58452-88 doi: 10. 10110079978-3000-58852222
Nadine Chang. Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level or object-level? tale of two resampling strategies for long-tailed detection. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1463-1472. PMLR, July 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04497,, 2020.
Yi-Ting Chen, Xiaokai Liu, and Ming Hssan Yang. Multi-instance object segmentation with occlu- sion handling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion (CVPR), June 2015.
10

---

Under review as a conference paper at ICLR 2022
Tianheng Cheng, Xinggang Wang, Lichao Huang, and Wenyu Liu. Boundary-Present Mask R- CNN. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12359, pp. 660-676. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58566-99 978-3-030-58568-6
doi: 10.
39.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vi- sion (ICCV), October 2017.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. CoRR, 0bss2107.006633 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Qi Fan, Lei Ke, Wenjie Pei, Chi-Keung Tang, and Yu-Wing Tai. Commonality- Parsiin Network Across Shape and Appearance for Partially Supervised Instance Segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12353, pp. 379-396. Springer International Publishing, Cham, 2020. ISBN 978-3-030- 58597-6 778-3-030-58598-11 doi:
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Fe- lix A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 31. Curran Associates, Inc., 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and self-supervised mod- els. 2020.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. CoRR, abs/2106.0741 2021.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmenta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2918-2928, June 2021.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. CoRR, abs/2012.05221 2020.
Ruohao Guo, Dantong Niu, Liao Qu, and Zhenbo Li. SOTR: Segmenting objects with transformers. CoRR, abs/2108.061 067477, 2021.
Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Spatial Pyramid Pooling in Deep Con- volutional Networks for Visual Recognition. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, volume 8691, pp. 346-361. Springer International Publishing, Cham, 2014. ISBN 978-3-319-105772 978-3-319-105777
doi:
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
11

---

Under review as a conference paper at ICLR 2022
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), October 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference Learning Representations, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15262-15271, June 2021.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving zero-shot transfer in reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1480-1490, International Convention Centre, Sydney, Australia, August 2017. PMLR.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. CoRR, abs/1812.022300 2018.
Ting-1 I Hsich, Esther Robb, Hwann- Tzong Chen, and Jia-Bin Huang. DropLoss for long-tail instance segmentation. In Proceedings of the Workshop on Artificial Intelligence Safety 2021 (SafeA/ 2021) Co-Located with the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI 2021), Virtual, February 8, 2021, 2021.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-excitan networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018a.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning segment every thing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), October 2017.
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019a.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CC- Net: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019b.
Md Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Bjöm Ommer, Konstantinos G. Derpa- nis, and Neil Bruce Shape or texture: Understanding discriminative features in [CNN}s. In International Conference Learning Representations, 2021.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elemen- tary Visual Reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recogni- fion (CVPR), pp. 1988-1997, Honolulu, HI, July 2017. IEEE. ISBN 978-1-5386-0457-1. doi: 10. 00.1109/CVPR.2017.255
Christoph Kamann and Carsten Rother. Benchmarking the Robustness of Semantic Segmentation Models with Respect to Common Corruptions. International Journal of Computer Vision, 129 (2):462-455 February 2021. ISSN 0920-5691, 1573-1405. doi: 10.1007/s1 126-00001122--
12

---

Under review as a conference paper at ICLR 2022
Lei Ke, Yu-Wing Tai, and Chi- Tang. Deep occlusion- instance segmentation with overlapping BiLayers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4019-4028, June 2021.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Felix Lau, Nishant Subramani, Sasha Harrison, Aerin Kim, Elliot Branson, and Rosanne Liu, Nat- ural adversarial objects. arXiv preprint arXiv:21 /1.22///1..2244.. 2021.
Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vi- neet, Kai Yuanqing Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang. Ashish Kapoor, and Aleksander Madry. 3DB: A framework for debugging computer vision models. CoRR, abs/2106.03805. 2021.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, May 2015. ISSN 0028-0836, 1476-4687.
doi: 10. 1038/nature145533
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Ziticck. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, volume 8693, pp. 740-755. Springer International Publishing, Cham, 2014. ISBN 978-3-319-10601-4 978-3-319-10002-1 doi: (0.1007//78-3-33 10602- 1_48.
Tsung-Yi Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), July 2017.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. /4030, 2021.
Spandan Madan, Tomotake Sasaki, Tzu-Mao Li, Xavier Boix, and Hanspeter Pfister. Small in- distribution changes in 3D perspective and lighting fool both CNNs and Transformers. CoRR, abs/2106. 16198, 2021.
C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. Machine Learning for Autonomous Driving Workshop, NeurIPS 2019, volume 190707484, July 2019.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant represen- tations. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Chaithanya Kumar Mummadi, Ranjitha Subramaniam, Robin Hutmacher, Julien Vitay, Volker Fis- cher, and Jan Hendrik Metzen. Does enhanced shape bias improve neural network robustness to common corruptions? In International Conference Learning Representations, 2021.
Tai-Yu Pan, Cheng Zhang, YANDONG LI, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. On model calibration for long-tailed object detection and instance segmentation. In Thhirty-Fii -iitttyFifth Conference on Neural Information Processing Systems, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? volume 97 of Proceedings of Machine Learning Research, pp. 5389- 5400, Long Beach, California, USA, June 2019. PMLR
13

---

Under review as a conference paper at ICLR 2022
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. CoRR, abs/ 1804.02767 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE Advances in Machine Learning and Deep Neural Networks, 109(5):612-633 2021. doi: 10.110//PR0..2201333333
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on ImageNet. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 8634-8644, PMLR, July 2020.
Ron Sun. On Variable Binding in Connectionist Networks. Connection Science, 4(2):93-224, Jan- uary 1992. ISSN 0954-0091, 1360-0494. doi: 10. 10.1080/0554099224466611
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- lernational Conference on Machine Leurning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 PMLR, June 2019.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 1513-1524. Curran Associates, Inc., 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839. Curran Associates, Inc., 2020a.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), October 2019.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional Convolutions for Instance Segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020, volume 12346, pp. 282-298. Springer International Publishing, Cham, 2020b. ISBN 978-3-030-58451-1 978-3-030-58452-5 doi: 10. 0-11997788--00--8-552-- 17.
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas I.. Griffiths. Are convolutional neural net- works or transformers more like human vision? CoRR, abbs/1105.07197, 2021.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are Disen- tangled Representations Helpful for Abstract Visual Reasoning? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' texquuteesinge Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 32, pp. 14222-14235. Curran Associates, Inc., 2019.
Chien Yao Wang, Hong- Yuan Mark Liao, Yueh-Hua Wu, Ping- Yang Chen, Jun-Wer Wei Hsieh, and I- Hau Yeh. CSPNet: A New Backbone that can Enhance Learning Capability of CNN. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp- 1571-1580, Seattle, WA, USA, June 2020a. IEEE. ISBN CNPRW55099..222000000
doi: 10. 109/
Chien Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 13029-13038, June 2021a.
14

---

Under review as a conference paper at ICLR 2022
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Zi- wei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed tailed instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1994,9704, June 2021b.
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang. Steven Hoi, and Jiashi Feng. The Devil Is Classification: A Simple Framework for Long-Tail Instance Segmentation. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12359, pp. 728-744. Springer International Publishing, Cham, 2020b. ISBN 978-3-030-58857-99 978-3-030-58568 doi: 00007/978-3-00-58888 6-356..-43..
Xinlong Wang. Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. SOLO: Segmenting Objects by Locations. Andrea Vedaldi, Horst Bischof. Thomas Brox, and Jan- fan-Michael Frahm (eds.), Computer Vision ECCV 2020, volume 12363, pp. 649-665. Springer International Publishing. Cham, 2020c. ISBN 978-3-030-55852--8 978-9- -0088-5--30---53-5.5 38.
doi: 10.1007/977.330000-555222
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 17721-17732. Curran Associates, Inc., 2020d.
Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13(4):600-612, April 2004. ISSN 1057-7149. doi: 10. 109/TTP.2003.81998
Zhirong Wu, Yuanjun Xiong. Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Alan Yuille and Chenxi Liu. Deep Nets: What have they ever done for Vision? Technical Report 88, Center for Brains, Minds and Machines (CBMM), MIT CSAIL, May 2018.
Jessica Yung, Rob Romijnders, Alexander Kolesnikov, Lucas Beyer, Josip Djolonga, Neil Houlsby, Sylvain Gelly, Mario Lucic, and Xiaohua Zhai. SI-Score: An image dataset for fine-gred fine-grained analysis of robustness to object location, rotation and size. CoRR, abs/2104.04191. 2021.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable ConvNets V2: More deformable, better results. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
15

---

Under review as a conference paper at ICLR 2022
APPENDIX
DATASET CREATION
Figure 10 provides representative sample of images in Stylized COCO. Figure 11 displays the effect of controlling the style strength parameter in AdaIN. Figure 12 shows the masked versions of the last example image in Figure and its blending sequences for Stylized COCO. Finally we attached a complete example for one image in Figure 13
the mock welicii
MyT PAIT
200
2001 YULCE10230D
24043300
Figure 10: Creation process of Stylized COCO. We plot the mask annotations locate ground truth instance the stylized images. These are also used to create the masked version Stylized COCO.
16

---

Under review as a conference paper at ICLR 2022
stpl diga
COOO
10 aggna 2.
cEpIE
Figure 11: Comparison of AdaIn style transfer strength. Depending the style image, an alpha value of (pink) produce rather extreme versions where objects are almost eradicated from the scene. An alpha value of (yellow) corresponds style transfer the content image with itself. As can bee seen in the middle column, this variant already introduces subtle shape changes to the images.
103
Alpha blend
ploel space (frst row) and feature space (second row).
Figure Comparison Stylized COCO, Stylized Objects Background for the example Figure 11 Bottom rows show the pixel and feature space blending sequences for the stylized COCO version.
17

---

Under review as a conference paper at ICLR 2022
379533 D0900379533.PPC instancesi
d'elophant'
'clephant', 'zabra' )
Ubee mask arnoationn
14%% image
Mylleed coco
styfied
stylzed dhjecs
Aoha plell space space isesonc
0.2
Alpha piael space ran)
feature spacee (wexard row).
Full example for one image. Top row: Creation Process of Stylized-COCC Second row: Com- parison of AdaIn style strength. Third row: Comparison of the stylized datasets. Last three rows: Blending Sequences for Stylize--OCCO, Stylized-Objects and -arked--aakguuuddd We create these for every image which results in 60 modified copies of the COCO val2017 subset.
Figure 13:
The configuration to reproduce the exact same version of stylized COCO can be found in our code release.
18

---

Under review as a conference paper at ICLR 2022
A NOTE ON THE ROLE OF DATA AUGMENTATION
certainly under-explored dimension of our study is the impact of data augmentation for systematic generalization. It is known that specific augmentation techniques will improve the performance on certain problem domains. For instance, Geirhos et al. (2019) and Michaelis al.) (2019) have shown that the texture bias in some classification and detection models can be reduced when stylized images are used during training. is not know however if the learned representations generalize in more systematic way, e-g. improve the performance on occluded objects for instance. It is also not known if frameworks and architectures benefit equally from such augmentation or some models exploiting the regularized signal more efficiently. These certainly interesting questions but would require to re-train all models that are used in our study from scratch. As this would result in significant amount of compute,
decided against this experiment for the following reasons. First, our infrastructure is simply limited. Second, we consider the investment to disproportional to the expected result. As can be seen from Figure 9 die difference in robustness for ResNet-50 without and with Large Scale Jitter augmentation lies within few percent. In contrast, frameworks and neck architectures appeared have a much larger impact on robustness. In consequence, we opted to make these dimension priority in our comparison and to postpone more in depth analysis of data augmentation.
PRE-TRAINED MODEL WEIGHTS
We code and checkpoints from the following projects. Note that unfortunately, CPMask (Fan et al. 2020) was not fully released at the time of this publication but will be included in our code release the future. Before we model on Stylized COCO and its variants, we reproduced the reported score on COCO va12017. Models without reported metrics on val2017 7 have been validated on test-dev2017 before testing.
Detectron 2:
BMask R-CNN: Blt.kKckw...tssss//./...a///arrr
YOLO: X000/h/tge//ggtttto...///aa//aa
DETR: DDTTT//w.aaa/gggai..ccooo/cooeee
CPMask: http://///atthhh.cmm/aaaaaa/////
SOTR: S0000//arpp//ggthbbooo/////rrar1
SOLOv2: https://hhh.h.oooaaar//////11111
SOLOv2
19

---

Under review as a conference paper at ICLR 2022
ADDITIONAL RESULTS
We include additional results for comparison. In partieular, we append the equivalent bounding box versions of our main paper figures. Figure 14 compares the absolute performance on COCO va12017 between models. The model grouping and detailed results we present the main paper are based on the distance matrices in Figure 15] Figure 16 provides the bbox equivalent to the general overview on model robustness. We also include the absolute performance versions of those figures. Figure 17 compares the relative framework performance by bbox score. Figure 18 and Figure 19 display the comparisons backbone and neck architectures respectively.
(a) Instance segmentation scores (YOLO bbox)
(b) Bounding Box scores
Figure 14: Comparison of absolute model performance COCO va12017. Methods that do not report
for va12017 have been validated on test-deve Note that SOTR SOLO perform worse on small objects but exhibit significantly improved segmentation scores for medium and large ohjects
(a) Stylized COCO
(b) Stylized Objects
(c) Stylized Background
Figure 15: Euclidian distance between the relative model performances (over the full alpha range). We average
over AP, APs, APm and Yellow squares group same model with different learning schedule.
20

---

Under review as a conference paper at ICLR 2022
(a) Segmentation
(b) Bounding Box
Figure 16: General overview of model robustness. Top row shows relative performance in the main paper. Bottom displays absolute performance for comparison. Note that comparing relative absolute perfor- mance results different order of frameworks due to varying base performance.
21

---

Under review as a conference paper at ICLR 2022
APs
Figure 17: Object-centric robustness by framework (bbox). Note that we compromised on R-101 for SOTR.
Whiti 10
(a) Segmentation scores
(b) Bounding Box scores
Figure 18: Full comparison of object-centric robustness by backbone architecture. Models marked with are trained with LSJ.
3
(a) Segmentation scores (YOLO is bbox)
(b) Bounding Box scores
Figure 19: Full comparison of object-centric robustness by neck architecture. Models with dynamic compo- nents are highlighted.
22


Reviews:
[{"review_id": "lmeTKPx5niv", "reviewer": "Reviewer_GTkh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe authors compare a variety of instance segmentation techniques along different axes like frameworks, architecture and pre-training\n\nConcerns:\nMy concern with the approach is that because it uses ground truth masks to generate different versions of datasets, it creates a bias where the features are corrupted, and this affects the performance of detectors. The RGB features inside the object masks are not changed in the stylized object dataset (apart from the minor corruption of features from the background), so it is expected that detectors would perform best in this case. When we start changing the RGB features of the objects but keep the background as is, the performance degrades, although not entirely because object boundaries would still be visible to some extent. Finally, when the entire image is stylized, the boundaries get mixed up and there is very little signal in the image (as can be seen Figure 2), so the detector does not work in this case. Overall, it remains unclear to me if the method is adding any insights compared to what is not already expected. I expect similar problems if any other transformations (like gaussian noise, salt/pepper noise/blur etc.) are applied to COCO images on object masks, background and entire image. Also, its unclear if there is something special about using stylized transformations to make claims that object detectors are learning shape/contour specific features - the conclusion which I am taking here is “the more noise we add, the worse object detectors get”, more specifically, noise everywhere > noise on object > noise on background.\n\nOther Comments:\nFigure 6 is very dense and hard to read. What are the light/dark lines in stylized coco/ stylized objects?\n\nThe authors should include 20-30 examples of stylized images in the paper or the supplementary material. This will give better insights to the reader about how the dataset looks in different cases.\n", "summary_of_the_paper": "In this paper, the authors study whether deep learning based techniques for instance segmentation are robust to changes in object texture or contour. This is inspired from a previous work in ICLR 2019 where the authors perform a similar study for image classification and show that existing CNN models learn classes based on texture and do not take the overall shape into account.\n\nThe authors take the task of instance segmentation where it is necessary for the model to learn the overall shape and measure the impact of texture changes. They use a stylized version of coco but also create two different types, one stylizing only the objects and the other stylizing only the background. Moreover, for each dataset type they create 10 variants by varying the blending weight. The evaluation is done on four dataset types, default coco val, stylized coco (everything stylized), stylized objects and stylized images. Except the default coco, every dataset type has 20 variants (different levels of blending plus pixel space vs feature space blending) which leads to a total of 61 datasets.\nFollowing this, the authors perform an evaluation of various instance segmentation techniques. They take three paradigms - architectures (backbone / neck), frameworks and pre-training approaches and study the robustness of each to changes in texture.\n", "summary_of_the_review": "There are gaps in the experimental design based on which we are concluding what features object detectors learn.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "gYmAAaOwVeb", "reviewer": "Reviewer_kn4q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is a good read, and easy to follow. The paper also covered many of the popular instance segmentation frameworks and architectures. This is a good reference for researchers who address the object-centric robustness.\n\nMy main concerns are the following:\n\n1- The paper is heavily influenced by the work in Geirhos et al. to the point that makes this paper a small incremental work.\n2- The authors missed providing some explanation for the results sometimes. (e.g. yolact is consistently more robust against stylization compared to other architectures, yolov4-csp is more robust compared to rest of yolo models, and Swin being the most robust when combined with maskrcnn framework).\n3- Since the paper is heavily relying on the stylized images for evaluation, and since the models are trained with different data augmentation strategies which might have influenced the robustness against stylization, I am surprised that the authors didn't study how data augmentation would influence the robustness metric. For instance, would we consider yolact more robust because of some data-augs used during training, or is it because of something else?\n\n\n", "summary_of_the_paper": "The authors studied the \"object-centric\" robustness of various instance segmentation models. They used stylized objects/background to evaluate the performance which is adopted from a work by Geirhos et al. [1]. The authors performed large number of experiments to evaluate different aspects of instance segmentation models including framework, architecture, and pre-training.", "summary_of_the_review": "The paper is a good study of existing deep-learning instance segmentation models, and frameworks. I enjoyed reading the paper. The authors put a lot of effort on performing large number of experiments to benchmark various models, however they put little effort on deeply studying and analyzing the results and they left it to the reader to interpret the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5Erevp2izP", "reviewer": "Reviewer_Hfwx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. A new object-centric evaluation dataset based on COCO val is constructed for evaluating the robustness instance segmentation models.\n2. The robustness evaluation is mainly evaluated from 3 impacted dimensions, which are well categorized. Also, a huge number of evaluations and comparisons are conducted.\n3. The paper is presented in a clear structure and easy to understand.\n\nWeakness:\n\n1. The paper evaluates the robustness of instance segmentation, mainly from the image/feature corruptions perspective. However, in real life, the robustness for instance segmentation should be more about the adaptation to novel objects [a,b] in a long-tailed distribution world or how to handle overlapping objects with similar appearance in heavy occlusions [c,d]. There should be some relations discussion with existing instance segmentation works that also focus in robust instance segmentation but from different directions.\n\n2. Instance segmentation models can be divided into backbone, neck and functional heads. The backbone and neck have been studied, but what’s the influence of the mask head design for instance segmentation? Some popular mask head designs, besides the Mask R-CNN, include BMask R-CNN [e] and BCNet [d], which also need to be evaluated and discussed.\n\n    [a] Learning to Segment Every Thing. CVPR, 2018.\n\n    [b] Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation. ECCV, 2020.\n\n    [c] Multi-instance object segmentation with occlusion handling. CVPR, 2015.\n\n    [d] Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers. CVPR, 2021.\n\n    [e] Boundary-preserving mask r-cnn. ECCV, 2020.\n\n3. The paper uses the image styling to simulate the image corruptions. How to guarantee it is realistic to real image corruptions with practical usages? For example, in Figure 1: right, a clear separation between background and foreground is observable in Stylized Object and Background but that is not common in real world, which may make the object detection and segmentation easier. I think some discussions on how to generalize the simulation to realistic world corruption cases should be added.\n\n4. The ambiguous usage of term 'architecture' vs. 'framework'. The 'architecture' should be replaced with 'backbone' instead, as in the first column in Table 1, most variants are R50, R101, Swin-T, etc. Usually, the usages of 'architecture' and 'framework' are similar.\n", "summary_of_the_paper": "The paper presents an evaluation on object-centric robustness of existing instance segmentation models, where an object-centric stylized coco val dataset is constructed. Then, a large number of experiments and evaluations are performed on this new dataset to study the impact of framework, architecture and pre-training.", "summary_of_the_review": "My concerns are listed in the weakness section. If the authors can well address them. I would consider raising my score.\n\nAfter reading the updated paper, my concern about the mask head influence is addressed. Also, I have read a new intro and abstract with better motivation connecting to the real-world instance segmentation challenges. Although image corruption using style transferring has no real-world cases, it does create novel object/background context. The experiments conducted are extensive and it provides some suggestions of instance segmentation model design for environment robustness. Thus, I decide to increase my rating to 6. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2lWahrSb594", "reviewer": "Reviewer_uJDe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper conducts a large number of experiments and presents a comprehensive instance segmentation literature review.\n+ The figures are very clear.\n\nWeaknesses:\n- I don't see any technical contribution for the method presented in the paper. It simply alters the dataset with existing style transfer method and perform other literature methods on the altered dataset.\n- The main experiment that this paper presents is to use the style tranfered dataset. However, I don't see any user cases for this setting. Besides, the dataset is modified by using groundtruth instance labels, which makes the experiment make less sense.", "summary_of_the_paper": "This paper presents a new setting for instance segmentation and compares a set of literature instance segmentation methods on this new setting. The new settings are: 1) use AdaIN (a style transfer method) to create stylized image, 2) only perform AdaIn on objects (use COCO's ground truth instance mask), 3) only perform AdaIN on background. This paper presents the instance segmentation AP numbers for many state-of-the-art methods (e.g. mask rcnn, swimtransformer, DETR) on the three settings with different ratio of the style transfer.", "summary_of_the_review": "I prefer to reject this paper. My main concern of this paper is the lack of contribution to the community. To me, this paper mainly conducted a set of experiments of existing methods on a new setting. And the new data setting doesn't make a lot of sense since the groundtruth is encoded in the image and there's no user cases for such setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper finally received divergent and borderline reviews with two positive (6) and two negative (3) rates. After the thorough reviews by ACs ourselves, we would like to decide to reject this work at this time, even though this submission has a lot of potentials including intensive analyses on instance segmentation frameworks and architectures.

We first would like to appreciate comprehensive author’s responses and additional empirical results. They should be extremely helpful to make this submission stronger. Here are some of our suggested points for improvement: (i) The novelty, significance, and practical implications of this work (compared to previous analysis work) may need to be better presented in a more persuasive way. (ii) Nuance of stylization transformation can be better explained compared to other types of perturbations or transformations. (iii) Empirical fairness can be better justified. (iv) Since the paper is written in a highly condensed way, some of reduction may improve the readability. (v) Finally, given that this paper focuses on empirical study about instance segmentation, it may be more appreciated in a computer vision venue.
Decision: Reject