Unsupervised Part Discovery from Contrastive Reconstruction

1 Introduction . Humans perceive the world as a collection of distinct objects . When we interact with an object , we naturally perceive the different parts it consists of . In visual scene understanding , parts provide intermediate representations that are more invariant to changes in object pose , orientation , camera view or lighting than the objects themselves . They are useful in analyzing objects for higher level tasks , such as fine-grained recognition , manipulation etc . However , supervised learning of parts requires manual annotations , which are slow , expensive and infeasible to collect for the almost unlimited variety of objects in the real world . Thus , unsupervised part discovery and segmentation has recently gained the interest of the community . We thus consider the problem of automatically discovering the parts of visual object classes : given a collection of images of a certain object category ( e.g. , birds ) and corresponding object masks , we want to learn to decompose an object into a collection of repeatable and informative parts . It is important to define and understand the nature of parts before we begin describing approaches to part discovery . While there is no universally accepted formal definition for what constitutes a “ part ” , the nature of objects and object parts is accepted as different . For example , for Gibson [ 26 ] , an object is “ detachable ” , i.e . something that , at least conceptually , can be picked up and moved to a different place irrespective of the rest of the scene . Parts , in contrast , are constituent elements of an object , and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can not be removed without breaking the object , i.e . they are essential to the object and occur across most instances of the same object category . Unsupervised part discovery requires suitable inductive principles and the choice of these principles defines the nature of the parts that will be discovered . Parts could , for example , be defined based on motion following the principle of common fate in Gestalt psychology ( i.e . what moves together belongs together ) [ 66 , 73 ] , or they could be defined based on visual appearance or function . Here , we are interested in semantic parts across different instances of an object category ( e.g. , birds , cars , etc . ) and combine three simple learning principles as “ part proxy ” : ( a ) consistency to transformation ( equivariance ) , ( b ) visual consistency ( or self-similarity ) , and ( c ) distinctiveness among different parts . Prior work has suggested that useful cues for part discovery can be obtained from pre-trained neural networks [ 2 , 28 ] . These networks can in fact be used as a dense feature extractors and the feature responses can be clustered or otherwise decomposed to identify parts [ 14 , 37 ] . In particular , [ 37 ] learn part prototypes , and make the latter orthogonal to avoid parts collapsing into a single one . In this paper , we revisit and improve such concepts . We make the following contributions . First , we show that contrastive learning can be employed as an effective tool to decompose objects into diverse and yet consistent parts . In particular , we seek parts whose feature responses are homogeneous within the same or different occurrences of the same part type , while at the same time being distinctive for different types of parts . A second contribution is to discuss whether clustering pre-trained features is indeed sufficient for part discovery . To this end , we show that simply clustering dense features sometimes captures obviously self-similar structures , such as image edges , rather than meaningful parts ( Section 3.2 ) . This is somewhat intrinsic to using pre-trained feed-forward local features , as these can only analyze a fixed image neighborhood and thus pick up the pattern which is most obvious within their aperture . As a complementary cue , we thus suggest to look at the visual consistency of parts . The idea is that most parts are visually homogeneous , sharing a color or texture . A generative model of the part appearance may thus be able to detect part membership at the level of individual pixels . We show , in particular , that even very simple models that assume color consistency are complementary and beneficial when added to feature-based grouping . Finally , we consider the problem of assessing automated part discovery . An issue is the relative scarcity of data labelled with part segmentation . Another one , technically more challenging , is the fact that parts that are discovered without supervision may not necessarily correspond to the parts that a human annotator would assign to an image . This makes the use of manual part annotations for evaluation tricky . Prior work in the area has thus assessed the discovered parts via proxy tasks , such as learning keypoint predictors , using supervision . The idea is that , if parts are consistent , they should be good predictors of other geometric primitives . Unfortunately , as we show empirically , transferring parts to keypoints is unlikely to provide a meaningful metric for the quality of the part segments . We show , for instance , that knowledge of a single keypoint provides a better predictor of other keypoints than any of the previous unsupervised models . To address this issue , we propose a new evaluation protocol . We still use keypoints as they are readily available , or ground-truth part segmentation when possible ; however , instead of learning to regress such ground truth annotations , we simply measure the co-occurrence statistics of the predicted parts and these annotations using Normalized Mutual Information and Adjusted Rand Index . The latter require the learned parts to be geometrically consistent and distinctive regardless of whether they are in one-to-one correspondence with manually provided labels and results in a more meaningful measure for this task . Empirically , we demonstrate that these improvements lead to stronger automated part discovery than prior work on standard benchmarks . 2 Related work . There exists a vast amount of literature that studies the problem of decomposing a scene into objects and objects into parts , with or without supervision . Next , we discuss these lines of work with a focus on unsupervised approaches . Unsupervised scene decomposition . Unsupervised scene decomposition methods aim to spatially decompose a scene into a variable number of components ( segments ) , e.g. , individual objects and background . This is typically achieved by encoding scenes into object-centric representations which are then decoded against a reconstruction objective , often in a variational framework [ 44 ] . Representative methods leverage sequential attention [ 5 , 20 ] , iterative inference [ 19 , 29 , 55 ] , spatial attention [ 16 , 52 ] and physical properties [ 3 ] , or extend towards temporal sequences [ 3 , 17 , 40 , 41 , 46 ] . Discriminative approaches [ 45 ] also exist , using an object-level contrastive objective . It has been shown that , currently , most of these models perform well on simple , synthetic scenes but struggle with higher visual complexity [ 43 ] . There are key conceptual differences between decomposing a scene into objects and decomposing an object into its parts . First , the objects of a scene typically appear in an arbitrary arrangement , whereas the presence and the layout of object parts is generally constrained ( e.g. , the arms of a human connect to the torso ) . As such , parts are constituent elements of an object and they occur consistently across most instances of an object category . Moreover , the segments obtained by the systems described above are orderless and do not have a “ type ” assigned to them , in contrast to parts which usually refer to specific , nameable entities ( e.g. , head vs. beak ) . These differences lead to sufficiently different statistics and technical constraints , which make it difficult to envision methods that can do well both at scene-level and object-level . Part discovery and segmentation . Prior to deep learning , part-based models [ 15 , 21–23 ] played a major role in problems such as object detection and recognition . In the deep learning era , part discovery remains an integral part of fine-grained recognition , where it acts as an intermediary step with or without part-level supervision [ 9 , 25 , 36 , 48 , 49 , 51 , 67 , 77 , 79 , 81–83 , 85 , 89 , 90 ] . However , all these methods require joint training with image labels and focus mostly on discovering the most informative ( discriminative ) regions to ultimately help with the classification task . Unlike previous methods as well as existing supervised methods that learn from annotated part segments [ 38 , 50 , 70 , 75 ] , our goal is to discover independent and semantically consistent parts without image-level or part-level labels . Bau et al . [ 2 ] and Gonzalez-Garcia et al . [ 28 ] inspect the hidden units of convolutional neural networks ( CNNs ) trained with image-level supervision ( e.g. , on ImageNet [ 61 ] ) to understand whether part detectors emerge in them systematically . This is done by measuring the alignment between each unit and a set of dense part labels , and as such , the availability of manual annotations is required for interpretation . Most related to our work , however , are approaches for unsupervised part segmentation [ 4 , 14 , 37 , 47 , 64 ] . Based on the observation of [ 2 , 28 ] that semantic parts do indeed emerge in deep features , Collins et al . [ 14 ] propose to use non-negative matrix factorization to decompose a pre-trained CNN ’ s activations into parts . Similar observations had been previously discussed in [ 64 ] for constructing part constellations models and in [ 76 ] for part detection via feature clustering . To learn part segmentations in an unsupervised manner , Braun et al . [ 4 ] propose a probabilistic generative model to disentangle shape and appearance , but focus mostly on human body parts . Lastly , closest to our work is SCOPS by Hung et al . [ 37 ] ; SCOPS is a self-supervised approach for object part segmentation from an image collection of the same coarse category , e.g. , birds . The authors propose a set of loss functions to train a model to output part segments that adhere to semantic , geometric and equivariance constraints . Other recent methods [ 69 , 87 ] use generative adversarial networks for few-shot part segmentation , while [ 24 ] discover parts without supervision by interacting with articulated objects , and [ 53 , 62 , 63 , 80 ] from motion in videos . Self-supervised and contrastive learning . In self-supervised learning , one typically aims to design pretext tasks [ 18 , 27 , 58 , 59 , 84 ] for pre-training neural networks ; these tasks are constructed such that the model has to capture useful information about the data that leads to learning useful features . Contrastive learning has recently emerged as a promising paradigm in self-supervised learning in computer vision , with several methods [ 7 , 11 , 32–35 , 42 , 57 , 71 , 78 ] learning strong image representations that transfer to downstream tasks . The key idea in contrastive learning is to encode two similar data points with similar embeddings , while pushing the embeddings of dissimilar data further apart [ 30 ] . In absence of labels , most contrastive methods use heavy data augmentations to create different views of the same image to use as a positive pair and are trained to minimize different variants of the InfoNCE loss [ 71 ] . We instead follow an approach to contrastive learning that is more tailored to semantic part segmentation , i.e . taking into consideration the dense nature of this problem . Our method is thus also related to self-supervised learning of dense representations [ 39 , 60 , 72 , 88 ] . As these methods learn an embedding for every pixel , they can not be directly applied for part segmentation and need either fine-tuning or a clustering step to produce part masks .

Summary: The authors present a new self-supervised framework for object part discovery, where two networks are jointly trained to represent and segment the image into parts, respectively. The networks are trained with a linear combination of several losses.  The segmentation network outputs a set of masks for each image, which correspond to the object parts in the image. The first term in the objective encourages features within a mask to be similar to each other.   Despite being unsupervised, these parts are consistent across images. The second term in the objective leverages this finding by constructing a contrastive objective which uses the same part (across images) as positive pairs, and different parts as negative samples.   Two other terms are added to the loss: equivariance to affine image transformations, and a "visual consistency" loss which penalizes the pixel-domain variance within masks.    The authors evaluate their method on several standard dataset, including CUB and DeepFashion.