Unsupervised Part Discovery from Contrastive Reconstruction

1 Introduction . Humans perceive the world as a collection of distinct objects . When we interact with an object , we naturally perceive the different parts it consists of . In visual scene understanding , parts provide intermediate representations that are more invariant to changes in object pose , orientation , camera view or lighting than the objects themselves . They are useful in analyzing objects for higher level tasks , such as fine-grained recognition , manipulation etc . However , supervised learning of parts requires manual annotations , which are slow , expensive and infeasible to collect for the almost unlimited variety of objects in the real world . Thus , unsupervised part discovery and segmentation has recently gained the interest of the community . We thus consider the problem of automatically discovering the parts of visual object classes : given a collection of images of a certain object category ( e.g. , birds ) and corresponding object masks , we want to learn to decompose an object into a collection of repeatable and informative parts . It is important to define and understand the nature of parts before we begin describing approaches to part discovery . While there is no universally accepted formal definition for what constitutes a “ part ” , the nature of objects and object parts is accepted as different . For example , for Gibson [ 26 ] , an object is “ detachable ” , i.e . something that , at least conceptually , can be picked up and moved to a different place irrespective of the rest of the scene . Parts , in contrast , are constituent elements of an object , and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can not be removed without breaking the object , i.e . they are essential to the object and occur across most instances of the same object category . Unsupervised part discovery requires suitable inductive principles and the choice of these principles defines the nature of the parts that will be discovered . Parts could , for example , be defined based on motion following the principle of common fate in Gestalt psychology ( i.e . what moves together belongs together ) [ 66 , 73 ] , or they could be defined based on visual appearance or function . Here , we are interested in semantic parts across different instances of an object category ( e.g. , birds , cars , etc . ) and combine three simple learning principles as “ part proxy ” : ( a ) consistency to transformation ( equivariance ) , ( b ) visual consistency ( or self-similarity ) , and ( c ) distinctiveness among different parts . Prior work has suggested that useful cues for part discovery can be obtained from pre-trained neural networks [ 2 , 28 ] . These networks can in fact be used as a dense feature extractors and the feature responses can be clustered or otherwise decomposed to identify parts [ 14 , 37 ] . In particular , [ 37 ] learn part prototypes , and make the latter orthogonal to avoid parts collapsing into a single one . In this paper , we revisit and improve such concepts . We make the following contributions . First , we show that contrastive learning can be employed as an effective tool to decompose objects into diverse and yet consistent parts . In particular , we seek parts whose feature responses are homogeneous within the same or different occurrences of the same part type , while at the same time being distinctive for different types of parts . A second contribution is to discuss whether clustering pre-trained features is indeed sufficient for part discovery . To this end , we show that simply clustering dense features sometimes captures obviously self-similar structures , such as image edges , rather than meaningful parts ( Section 3.2 ) . This is somewhat intrinsic to using pre-trained feed-forward local features , as these can only analyze a fixed image neighborhood and thus pick up the pattern which is most obvious within their aperture . As a complementary cue , we thus suggest to look at the visual consistency of parts . The idea is that most parts are visually homogeneous , sharing a color or texture . A generative model of the part appearance may thus be able to detect part membership at the level of individual pixels . We show , in particular , that even very simple models that assume color consistency are complementary and beneficial when added to feature-based grouping . Finally , we consider the problem of assessing automated part discovery . An issue is the relative scarcity of data labelled with part segmentation . Another one , technically more challenging , is the fact that parts that are discovered without supervision may not necessarily correspond to the parts that a human annotator would assign to an image . This makes the use of manual part annotations for evaluation tricky . Prior work in the area has thus assessed the discovered parts via proxy tasks , such as learning keypoint predictors , using supervision . The idea is that , if parts are consistent , they should be good predictors of other geometric primitives . Unfortunately , as we show empirically , transferring parts to keypoints is unlikely to provide a meaningful metric for the quality of the part segments . We show , for instance , that knowledge of a single keypoint provides a better predictor of other keypoints than any of the previous unsupervised models . To address this issue , we propose a new evaluation protocol . We still use keypoints as they are readily available , or ground-truth part segmentation when possible ; however , instead of learning to regress such ground truth annotations , we simply measure the co-occurrence statistics of the predicted parts and these annotations using Normalized Mutual Information and Adjusted Rand Index . The latter require the learned parts to be geometrically consistent and distinctive regardless of whether they are in one-to-one correspondence with manually provided labels and results in a more meaningful measure for this task . Empirically , we demonstrate that these improvements lead to stronger automated part discovery than prior work on standard benchmarks . 2 Related work . There exists a vast amount of literature that studies the problem of decomposing a scene into objects and objects into parts , with or without supervision . Next , we discuss these lines of work with a focus on unsupervised approaches . Unsupervised scene decomposition . Unsupervised scene decomposition methods aim to spatially decompose a scene into a variable number of components ( segments ) , e.g. , individual objects and background . This is typically achieved by encoding scenes into object-centric representations which are then decoded against a reconstruction objective , often in a variational framework [ 44 ] . Representative methods leverage sequential attention [ 5 , 20 ] , iterative inference [ 19 , 29 , 55 ] , spatial attention [ 16 , 52 ] and physical properties [ 3 ] , or extend towards temporal sequences [ 3 , 17 , 40 , 41 , 46 ] . Discriminative approaches [ 45 ] also exist , using an object-level contrastive objective . It has been shown that , currently , most of these models perform well on simple , synthetic scenes but struggle with higher visual complexity [ 43 ] . There are key conceptual differences between decomposing a scene into objects and decomposing an object into its parts . First , the objects of a scene typically appear in an arbitrary arrangement , whereas the presence and the layout of object parts is generally constrained ( e.g. , the arms of a human connect to the torso ) . As such , parts are constituent elements of an object and they occur consistently across most instances of an object category . Moreover , the segments obtained by the systems described above are orderless and do not have a “ type ” assigned to them , in contrast to parts which usually refer to specific , nameable entities ( e.g. , head vs. beak ) . These differences lead to sufficiently different statistics and technical constraints , which make it difficult to envision methods that can do well both at scene-level and object-level . Part discovery and segmentation . Prior to deep learning , part-based models [ 15 , 21–23 ] played a major role in problems such as object detection and recognition . In the deep learning era , part discovery remains an integral part of fine-grained recognition , where it acts as an intermediary step with or without part-level supervision [ 9 , 25 , 36 , 48 , 49 , 51 , 67 , 77 , 79 , 81–83 , 85 , 89 , 90 ] . However , all these methods require joint training with image labels and focus mostly on discovering the most informative ( discriminative ) regions to ultimately help with the classification task . Unlike previous methods as well as existing supervised methods that learn from annotated part segments [ 38 , 50 , 70 , 75 ] , our goal is to discover independent and semantically consistent parts without image-level or part-level labels . Bau et al . [ 2 ] and Gonzalez-Garcia et al . [ 28 ] inspect the hidden units of convolutional neural networks ( CNNs ) trained with image-level supervision ( e.g. , on ImageNet [ 61 ] ) to understand whether part detectors emerge in them systematically . This is done by measuring the alignment between each unit and a set of dense part labels , and as such , the availability of manual annotations is required for interpretation . Most related to our work , however , are approaches for unsupervised part segmentation [ 4 , 14 , 37 , 47 , 64 ] . Based on the observation of [ 2 , 28 ] that semantic parts do indeed emerge in deep features , Collins et al . [ 14 ] propose to use non-negative matrix factorization to decompose a pre-trained CNN ’ s activations into parts . Similar observations had been previously discussed in [ 64 ] for constructing part constellations models and in [ 76 ] for part detection via feature clustering . To learn part segmentations in an unsupervised manner , Braun et al . [ 4 ] propose a probabilistic generative model to disentangle shape and appearance , but focus mostly on human body parts . Lastly , closest to our work is SCOPS by Hung et al . [ 37 ] ; SCOPS is a self-supervised approach for object part segmentation from an image collection of the same coarse category , e.g. , birds . The authors propose a set of loss functions to train a model to output part segments that adhere to semantic , geometric and equivariance constraints . Other recent methods [ 69 , 87 ] use generative adversarial networks for few-shot part segmentation , while [ 24 ] discover parts without supervision by interacting with articulated objects , and [ 53 , 62 , 63 , 80 ] from motion in videos . Self-supervised and contrastive learning . In self-supervised learning , one typically aims to design pretext tasks [ 18 , 27 , 58 , 59 , 84 ] for pre-training neural networks ; these tasks are constructed such that the model has to capture useful information about the data that leads to learning useful features . Contrastive learning has recently emerged as a promising paradigm in self-supervised learning in computer vision , with several methods [ 7 , 11 , 32–35 , 42 , 57 , 71 , 78 ] learning strong image representations that transfer to downstream tasks . The key idea in contrastive learning is to encode two similar data points with similar embeddings , while pushing the embeddings of dissimilar data further apart [ 30 ] . In absence of labels , most contrastive methods use heavy data augmentations to create different views of the same image to use as a positive pair and are trained to minimize different variants of the InfoNCE loss [ 71 ] . We instead follow an approach to contrastive learning that is more tailored to semantic part segmentation , i.e . taking into consideration the dense nature of this problem . Our method is thus also related to self-supervised learning of dense representations [ 39 , 60 , 72 , 88 ] . As these methods learn an embedding for every pixel , they can not be directly applied for part segmentation and need either fine-tuning or a clustering step to produce part masks .

Summary: The authors develop a method for segmenting an object into parts, given a dataset of images of that object category and foreground masks for each object. This method is a variant of contrastive self-supervised learning, which aims to make learned features within positive sample pairs more similar and features within negative sample pairs more disparate. In this case, the samples are feature vectors averaged over proposed part segmentation masks within each image, rather than feature vectors corresponding to an entire image as in now-standard approaches. Applying this principle, along with several additional contraints (auxiliary losses), leads to state-of-the-art performance on three semantic part segmentation benchmarks. The authors also introduce two new metrics for semantic part segmentation that they argue better capture the nature of the problem.