An object-centric sensitivity analysis of deep learning based instance segmentation

1 INTRODUCTION . In this study , we want to learn how deep learning models for instance segmentation generalize to novel appearances of familiar objects . Despite their remarkable success in computer vision , deep neural networks still struggle in challenging real-world scenarios ( Yuille & Liu , 2018 ; Michaelis et al. , 2019 ; Recht et al. , 2019 ; Madan et al. , 2021 ) . Consider for instance a pedestrian with an unconventionally textured dress or a rare horse statue made from bronze . The model might have seen many pedestrians or natural horses during training but still fails to detect these unseen and out of context examples , often with high confidence . Generalizing to such naturally adversarial objects is typically described as out-of-distribution robustness in the literature ( Hendrycks et al. , 2021 ; Lau et al. , 2021 ) . Interestingly , Hendrycks et al . ( 2021 ) suggest that improvements in this direction are more likely to come from computer vision architectures than from existing data augmentation or additional public datasets . With the ever increasing number of competitive models , our objective is therefore to contribute an extensive comparison of instance segmentation models to unveil promising future research directions . The perspective we take is inspired by the work of Baker et al . ( 2018 ) and Geirhos et al . ( 2019 ) on texture bias in classification models . More precisely , both groups found that when compared to humans , convolutional neural networks ( CNNs ) mostly ignore object shapes , i.e . ignore the defining structure of an object . In fact , Brendel & Bethge ( 2019 ) have further shown that CNNs can robustly classify objects in texturized images where the global appearance of objects is fully mixed up . In summary , these findings indicate that a revision of contemporary deep learning architectures is a much needed contribution on the pathway to more systematic generalization . To compare the robustness of instance segmentation models , we introduce a novel negative test in the form of an object-centric sensitivity analysis . More precisely , we investigate the impact of increasingly novel object texture while controlling for the effect of corrupted color , shape and contour features . By object-centric , we mean that the appearance of visual objects is changed in a semantics preserving and causally plausible way , i.e . is controlled at the instance level . Since we generally expect that segmentation performance will degrade under novel textures and increasing corruptions , our benchmark can be understood as a negative test , i.e . if a specific model appears to be significantly more robust than others , we can consider it a promising candidate for more in depth research on generalization . To simulate novel appearances of familiar objects we utilize a stylized version of MS COCO ( Lin et al. , 2014 ) as shown Figure 1 left . As can be seen , the AdaIN method ( Huang & Belongie , 2017 ) effectively replaces local texture cues but preserves the global shape of objects . In addition , we create two object-centric versions of the stylized COCO dataset as displayed in Figure 1 right . By comparing the performance on all three datasets we can control for spurious correlations that might have been introduced in the fully stylized dataset . Note that the style masking encodes ground truth information from the test set in the data which could potentially be exploited by the models . While this would be an interesting finding in itself , we do not see this to happen in practice . A limitation of our approach is the use of artistic style images . This leads to novel object appearances that can not be found in real-life directly . However , we compromise on this choice since it ensures that textures are truly novel and not biased which could happen with natural style images . We discuss more alternatives to our simulation method in the related work section . A closely related perspective on segmentation robustness can be found in work that acknowledges the inherent long-tail distribution of real-world data . In such settings , the challenge is to become robust against the bias of extreme class imbalance in existing datsets . Common approaches to resolve this issue are re-sampling ( Wang et al. , 2020b ; Chang et al. , 2021 ) and regularization strategies ( Tang et al. , 2020 ; Pan et al. , 2021 ; Hsieh et al. , 2021 ; Wang et al. , 2021b ) . The long-tail problem can also be understood as a low resource setting where data collection is expensive or class labels are missing . In such cases , the challenge is to efficiently adapt to novel objects or uncommon appearances in a semi-supervised manner ( Hu et al. , 2018b ; Fan et al. , 2020 ) . A particular instantiation of this problem is object occlusion . Since objects can occlude each other in almost infinitely different ways , a common strategy is to model object features more explicitly , i.e . to decouple shape and appearance for instance ( Chen et al. , 2015 ; Cheng et al. , 2020 ; Ke et al. , 2021 ; Fan et al. , 2020 ) . The idea is to learn representations that generalize more systematically and we expect these methods to be strong contenders in our comparison . We hope that our comprehensive benchmark motivates more research in this direction which we believe , will lead to improvements in all of the related problem spaces . 2 METHODS . In this section we describe the datasets , frameworks and models that are used in this study . The code to reproduce our results , as well as ≈ 1.5TB of detection and evaluation data can be found here : link-to-project-page-when-published . 2.1 AN OBJECT-CENTRIC CAUSAL VERSION OF STYLIZED COCO . Stylized COCO as shown in Figure 1 left is an adaptation of Stylized-ImageNet by Geirhos et al . ( 2019 ) . It was first used by Michaelis et al . ( 2019 ) 1 as data augmentation technique during training to improve robustness of detection models against common corruption types such as gaussian noise or motion blur . We instead use a stylized version of the val2017 subset to test instance segmentation models on this data directly . By manual inspection of Stylized COCO , we found that strong 1 Stylized Datasets : https : //github.com/bethgelab/stylize-datasets stylization can sometimes lead to images where the object contour starts to vanish , up to the point where objects and their boundaries dissolve completely . This effect depends on the style image used in the creation process but affects objects of all scales alike . As shown in Figure 2 , we resolve this issue by using the ground truth mask annotations to limit the style transfer to the actual objects or the background . This not only ensures that object contours are preserved but also controls for global stylization as a confounding variable . By assuming an object-centric causal model , Stylized COCO allows us to ask interventional questions regarding the original COCO dataset , e.g . “ What happens if we change the texture of images ? ” . By masking the style transfer to objects or background , we can also ask counterfactual questions such as “ Was it actually the object that caused the change in performance ? ” , “ What if we change the background instead ? ” . We will refer to the different dataset versions as Stylized COCO ( l ) , Stylized Objects ( s ) and Stylized Background ( n ) . A related , second problem that remains after masking is that shape information within the object can also be lost due to strong stylization . We address this issue by controlling the strength of the AdaIN style transfer . This can be done with an α parameter that acts as a mixing coefficient between the content and style image . More precisely , AdaIN employs a pre-trained VGG encoder f on both images , performs an interpolation step between the resulting feature maps and produces the final output with a learned decoder network g. In summary , a stylized image t is produced by T ( c , s , α ) = g ( ( 1− α ) f ( c ) + αAdaIN ( f ( c ) , f ( s ) ) ) where c and s are the content and style images respectively . We will refer to this method as blending in feature space . The top row of Figure 3 shows two examples of the extreme points α = 0 and α = 1 . Note that at α = 0 , the image colors are mostly preserved but the algorithm has already introduced artifacts in the form of subtle texture and shape changes . In response we create a control group where we perform alpha blending between the pixel values of the original content image c and the fully stylized image tα=1 : P ( c , tα=1 , α ) = ( ( 1− α ) ∗ c+ α ∗ tα=1 ) . We will refer to this method as blending in pixel space . In contrast to the feature space sequence , the control group should preserve textures and object shape over a longer range . The idea is to compare models on both sequences in order to attribute performance changes to either color and texture or shape , depending on the objects size . In contrast to Geirhos et al . ( 2019 ) who used a fixed style strength to modify ImageNet features ( α = 1 ) , we produce the full alpha-range α ∈ ( 0.0 , 0.1 , 0.2 , ... , 1.0 ) for both blending spaces . Note that every alpha value depicts a separate and complete version of the accordingly styled COCO val2017 subset . The qualitative differences can be inspected in Figure 3 bottom left ( zoom in for better visibility ) . We also went for quantitive measures to validate our subjective impression of Stylized COCO . Figure 3 bottom right shows a comparison of image gradients and RGB histograms at the extreme points . Compared to the original image we can see the subtle shape changes in the gradient map of α = 0 and a significantly different color histogram at α = 1 . To describe this effect over the full alpha range , we compute the structural similarity index ( SSIM ) ( Wang et al. , 2004 ) between gradient images of corresponding image pairs . Between RGB histograms we compute the Wasserstein distance alike . We always compare against the original COCO data and report the mean distance averaged over the full dataset at a specific alpha . In addition to the image-to-image scores we also include an instance level comparison for the COCO scales S , M and L. Instances have been cropped based on bbox information . This addition was added after we observed that small objects appear to be more affected by the AdaIN artifacts compared to medium and large instances . Figure 4 displays the results and confirms our assumption . Structural similarity depends on object size and is in fact , almost constant over the full feature space range for small objects . Furthermore , the control group preserves structural similarity over a longer range as intended . Color distance in contrast converges at around α = 0.3 . Based on these insights , we feel confident to better attribute potential performance dips and subsequently , determine the relative importance of each feature type . 2.2 MODEL SELECTION . To contribute a comprehensive overview on model robustness , we opted for a broad comparison of popular frameworks and architectures . The dimensions we consider to be impactful are framework , architecture and pre-training . The selected models can be found in Table 1 . Frameworks for instance segmentation can be categorized in different ways . A first distinction can be made between methods that solve the detection problem as a refinement process of box proposals ( multi stage ) and methods that predict bounding boxes directly ( one stage ) . We include the popular multi-stage frameworks Mask R-CNN ( He et al. , 2017 ) and Cascade Mask R-CNN ( Cai & Vasconcelos , 2018 ) that uses multiple refinement stages instead of one . Both frameworks formulate instance segmentation as a pixel-wise classification problem . Since this rather naive extension to Faster R-CNN ( Ren et al. , 2015 ) can ignore object boundaries and shapes , we include the boundarypreserving mask head alternative ( Cascade- ) BMask ( Cheng et al. , 2020 ) for comparison . A remaining challenge to boundary detection are overlapping objects that occlude the ground truth contour of other instances . We therefore include the Bilayer Convolutional Network ( BCNet ) ( Ke et al. , 2021 ) as another mask head alternative . In BCNet , the occluded and occluding object are separately detected and modeled explicitly in a layered representation . The mask head can then ” consider the interaction between [ the decoupled boundaries ] during mask regression ( Ke et al. , 2021 ) . ” A second distinction between frameworks concerns the use of predefined anchor boxes . Anchor based methods predict relative transformations on these priors whereas anchor free methods predict absolute bounding boxes . We include YOLACT ( ++ ) ( Bolya et al. , 2019 ; 2020 ) as a one-stage , anchor based framework . YOLACT is a real-time method that solves instance segmentation without explicit localization ( feature pooling ) . Instead , it generates prototype masks over the entire image which are combined with per-instance mask coefficients to form the final output . The ( ++ ) version improves by adding a mask re-scoring branch ( Huang et al. , 2019a ) and deformable convolutions ( v2 ) ( Dai et al. , 2017 ; Zhu et al. , 2019 ) . We include DETR ( Carion et al. , 2020 ) as a one-stage , anchor free framework that formulates object detection as a set prediction problem over image features . Note that it was not primarily designed for instance segmentation but offers a corresponding extension that we use in our study . Based on model availability we include BCNet in the FCOS ( Tian et al. , 2019 ) variant ( F-BCNet ) . FCOS is a fully convolutional , one-stage , anchor-free alternative to Faster R-CNN that ” solves object detection in a per-pixel prediction fashion , analogue to semantic segmentation ( Tian et al. , 2019 ) . ” Finally , we distinguish between top down frameworks where detection precedes segmentation and bottom up methods where bounding boxes are derived from mask predictions . We include the bottom-up methods SOLOv2 ( Wang et al. , 2020d ) and SOTR ( Guo et al. , 2021 ) . SOLO ( Wang et al. , 2020c ) divides the input into a fixed grid and predicts a semantic category and corresponding instance mask at each location . The final segmentation is obtained with non-maximum-suppression on the gathered grid results to resolve similar predictions of adjacent grid cells . SOLOv2 improves by introducing dynamic convolutions to the mask prediction branch , i.e . an additional input dependent branch that dynamically predicts the convolution kernel weights . A similar idea was used by Tian et al . ( 2020b ) . SOTR uses a twin attention mechanism ( Huang et al. , 2019b ) to model global and semantic dependencies between encoded image patches . The final result is obtained by patch wise classification and a multi-level upsampling module with dynamic convolution kernels for mask predictions , similar to SOLOv2 . For completeness , we also include YOLO ( v3,4 and scaled v4 ) to our comparison since detection is a vital sub-task of top down frameworks ( Redmon & Farhadi , 2018 ; Bochkovskiy et al. , 2020 ; Wang et al. , 2021a ) . Architectures used in instance segmentation can be divided into backbone , neck and functional heads . The latter output the final results and are framework specific . Backbones and necks however are typically chosen from a pool of established models which allows for a controlled comparison . The role of backbone networks is to extract meaningful feature representations from the input , i.e . to encode the input . The neck modules define which representations are available to the functional heads , i.e . define the information flow . We include the CNN backbones ResNet ( He et al. , 2016 ) , ResNext ( Xie et al. , 2017 ) and RegNet ( Radosavovic et al. , 2020 ) , a network found with meta architecture search that outperforms EfficientNet ( Tan & Le , 2019 ) . Note that BCNet utilizes a Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) within its mask heads to model long-range dependencies between pixels ( to evade local occlusion ) . Furthermore , DETR and SOTR are hybrid frameworks that use transformer architectures to process the encoded backbone features . With Swin Transformer ( Liu et al. , 2021 ) we also include a convolution free backbone alternative based on the Vision Transformer approach ( ViT ) ( Dosovitskiy et al. , 2021 ) . The most popular neck choice is the Feature Pyramid Network ( FPN ) ( Lin et al. , 2017 ) . It builds a hierarchical feature representation from intermediate layers to improve performance at different scales , e.g . small objects . For comparison we also include a ResNet conv4 neck ( C4 ) as used in Ren et al . ( 2015 ) and a ResNet conv5 neck with dilated convolution ( DC5 ) as used by Dai et al . ( 2017 ) . Finally we abbreviate FPN models that use deformable convolutions as DCN ( Dai et al. , 2017 ; Zhu et al. , 2019 ) . Similar to dynamic convolutions which predict kernel weights , DCNs learn to dynamically transform the sampling location of the otherwise fixed convolution filters . Pre-training of backbone networks is commonly done as supervised learning on ImageNet ( IN ) . Due to the recent success of self supervised learning ( SSL ) in classification , we are interested in how these representations perform in terms of object-centric robustness . In particular we are interested in the contrastive learning framework that seeks to learn “ representations with enough invariance to be robust to inconsequential variations ( Tian et al. , 2020a ) ” . Based on availability we include the methods InstDis ( Wu et al. , 2018 ) , MoCo ( He et al. , 2020 ; Chen et al. , 2020 ) , PIRL ( Misra & van der Maaten , 2020 ) and InfoMin ( Tian et al. , 2020a ) . Note that pre-trained backbones were only used as initialization for a supervised training on COCO . As a final comparison we include models that have been trained with random initialization and Large Scale Jittering ( LSJ ) ( Ghiasi et al. , 2021 ) data augmentation as an alternative to pre-training . The complete list of models can be inspected in Figure 5 . From the overview in Table 1 we can see that our selection allows for a fair comparison of frameworks as long as we fix the backbone and neck architecture to ResNet+FPN . Vice versa we can compare backbone and neck combinations within a fixed framework , in particular Mask R-CNN . Note that we also compared different learning schedules but did not include this dimension more prominent in the categorization after finding it to be the least significant factor in our evaluation .

Summary: This paper presents a new setting for instance segmentation and compares a set of literature instance segmentation methods on this new setting. The new settings are: 1) use AdaIN (a style transfer method) to create stylized image, 2) only perform AdaIn on objects (use COCO's ground truth instance mask), 3) only perform AdaIN on background. This paper presents the instance segmentation AP numbers for many state-of-the-art methods (e.g. mask rcnn, swimtransformer, DETR) on the three settings with different ratio of the style transfer.