Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims to segment parts of visual object classes in an unsupervised manner. The segmented part should be consistent within the class. To enable the model, several loss functions are added including different consistency losses, contrastive loss across different images, and equivariance loss on the transformation of the current image. Quantitative and qualitative experiments are conducted to examine the effectiveness of the proposed model.
main_review: ********************Strengths********************

The proposed approach for solving the part discovery is reasonable. More specifically, i) a consistency loss to encourage the uniformity within each part/mask; ii) a contrastive loss to encourage the consistency of parts across different images; iii) a loss to encourage the visual consistency within each part; iv) a loss to maintain consistency of features after transformation.

The code is provided, which can benefit the research along this direction.


********************Weaknesses********************

1.	It is unclear to me what is the potential applications of the studied topic and why it is important for studying. The proposed method relies on self-altered parameters including weights for loss and K for the number of parts, which is rather heuristic and hard to use in the real case. That is, the parameters need to be tuned each time when changing the dataset or number of parts.
2.	The writing is hard to understand in most of the model parts, especially for those descriptions of equations. For example, what does m mean in Eq (3), which often makes me confusing. Some model choice also needs more explanations, such as why use KL for transformation loss but not generally used L2. It is suggested to show some qualitative results on ablation studies to demonstrate the function of each loss. 
3.	Why do the qualitative results only contain those compared with “SCOPS”, and how about the others listed in Table 1. Besides, the performance in Figure 4 is rather bad from my point of view, head of the dog, cat, and horse are even not in the same category of masks. This makes me wonder whether the problem has been solved properly.
4.	For the k-means baseline, why not try on some features learned by the general segmentation task to see how the model compared with them? Besides, it is suggested to put the ground truth used to calculate loss as an example for visualization, even this is an unsupervised method.


In summary, I am leaning towards rejection and do not think the paper meets the acceptance bar of NeurlPS. I suggest the author revise the paper to make it stronger. 

limitations_and_societal_impact: The paper has discussed limitations and social impart.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a new self-supervised framework for object part discovery, where two networks are jointly trained to represent and segment the image into parts, respectively. The networks are trained with a linear combination of several losses.

The segmentation network outputs a set of masks for each image, which correspond to the object parts in the image. The first term in the objective encourages features within a mask to be similar to each other. 

Despite being unsupervised, these parts are consistent across images. The second term in the objective leverages this finding by constructing a contrastive objective which uses the same part (across images) as positive pairs, and different parts as negative samples. 

Two other terms are added to the loss: equivariance to affine image transformations, and a "visual consistency" loss which penalizes the pixel-domain variance within masks.  

The authors evaluate their method on several standard dataset, including CUB and DeepFashion.
main_review: Although somewhat involved technically and conceptually (the final loss contains 4 different terms), the method is well motivated and easy to follow. The ablations section shows that each term is indeed necessary and leads to large gains in performance. The comparison to prior art is very encouraging, with strong results presented throughout. 

In addition to presenting an unsupervised method for part discovery, the authors investigate freeing themselves from supervised pretraining as well. There is indeed a large drop in performance, however this is an important baseline for future research in fully unsupervised part discovery.

One ablation is missing however: the authors define the positive pairs in their contrastive objective as the same part across images, rather than the same part across different views of the same image. How important is this modification? This finding could be useful beyond the specific problem considered here.
limitations_and_societal_impact: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors develop a method for segmenting an object into parts, given a dataset of images of that object category and foreground masks for each object. This method is a variant of contrastive self-supervised learning, which aims to make learned features within positive sample pairs more similar and features within negative sample pairs more disparate. In this case, the samples are feature vectors averaged over proposed part segmentation masks within each image, rather than feature vectors corresponding to an entire image as in now-standard approaches. Applying this principle, along with several additional contraints (auxiliary losses), leads to state-of-the-art performance on three semantic part segmentation benchmarks. The authors also introduce two new metrics for semantic part segmentation that they argue better capture the nature of the problem. 
main_review: Figuring out how to represent objects with substructure (i.e., parts) is an important and difficult challenge; many of the things we interact with and depend on in everyday life can be decomposed into meaningful parts (e.g that perform different functions), and understanding this fact is central to how we perceive and use those objects. It also makes sense, to me, to tackle this problem with a self-supervised approach, because collecting part supervision labels for a wide range of objects across many categories seems hopeless in the long run: there are too many categories with distinct substructures, not all exemplars of a category have the same substructure, and in the case of man-made objects, new types of substructure will be created. Perhaps most importantly, the notion of a "part" is highly ambiguous. 

My main reaction to this paper is surprise that the authors do not really consider the last of these difficulties -- neither in the introduction in talking about why their task is important and interesting, nor in the methods (which don't seem well-suited to handle this problem). They do get state-of-the-art results on what are apparently widely-used benchmarks, and they develop what seems to me a better set of metrics for assessing part-based segmentation given ground truth annotations. For these latter reasons I'd guess that this work might be valuable to people who work on the same problem, so I'm open to accepting the paper. But I hope that the authors can spend substantial space in the manuscript motivating their approach, addressing its limitations, and (ideally) modifying it to tackle the core problem I describe below -- or can explain why I'm wrong in thinking this is the core problem.

_The Core Problem: What makes a part a part?_ Based on the authors' introduction and explanation of their methods, the notion of "part" they seem to be going for here is some combination of "functional" (a part is something that performs a distinct function), "semantically conventional" (a part is something that most people agree is a part, regardless of the underlying reasons), and "appearance-based" (a part is something that has relatively uniform texture or color.) In particular, they seem to think that a learning method that implicitly uses the last of these notions (appearance-based) will suitably capture the first two (which is what they evaluate on.) 

But it seems to me that in most cases, relatively uniform appearance isn't a good way to discover or think about parts of objects. There's no underlying reason why parts of an object that perform a distinct function should have uniform appearance that's different across different parts, and it's easy to think of examples where they don't: wheels of a car, wings of a butterfly, organs on a face (eyes, ears, nose, mouth), and so on. It's possible that, by virtue of supervised or self-supervised training, deep ANNs do learn higher-level feature spaces in which features are uniform within each part and distinct across parts -- but I haven't seen good evidence of this, and that doesn't mean that parts will have uniform texture at the pixel level. Indeed, the visual examples in the work here don't often match up with what I think of as a part (e.g. segmenting people into "hair" and "skin" rather than "head" and "arms").

Instead, the most natural notion of a part that **could** be learned from visual observations is one based on motion: a part is something that can move partly independently of the rest of an object. A functional definition might be closer to the truth in some cases (e.g. the porch of a house doesn't move independently of the rest of the house), but understanding the distinct functions of different spatial components of an object is probably beyond the scope of current computer vision methods; in any case, function and independent movability are highly correlated, as in the examples I listed in the paragraph above. 

There are other recent works that _do_ recognize this notion of part as more fundamental than appearance, and develop methods that try to tackle it accordingly (whether not they succeed) -- for instance, [A,B]. In particular, the problem formulation in [A] is very similar to the one used here -- mainly the testing datasets are different.

Beyond the issue with the underlying notion of a part, I have another major concern with this approach: the number of parts will be different for different categories, and may also vary within a category, yet here the number of parts (K) is a fixed element of the architecture. This means that an architecture will have to be tailored for a specific category, depending on what the authors think is the "true" number of distinct parts. This goes against the principles of self-supervised learning, in which the designers of an algorithm should not need to know too much about the ground truth labels for a dataset to apply the approach. In other methods (like [A,B]), only the maximum number of parts is chosen in advance; different scenes may have different numbers of "occupied" slots. 

It's also worth remarking that the decomposition of an object into parts is not fundamentally different from decomposition of a scene into objects -- unless there is some strong philosophical reason to treat "parts of an object" differently from "parts of a scene." Because of this, I'm surprised that the authors don't relate their work to unsupervised object discovery methods like [B,C,D] or similar. 

Overall, I think this work would benefit from much stronger conceptual motivation and development, and ideally would attempt to tackle unsupervised part discovery in a more principled way -- I think this is possible with more or less the framework the authors use now if given some more thought. However, as far as I can tell, the quantitative improvements over competing methods are real and large, and it seems like it's more common in this research area to use the "conventional semantic" notion of an object part -- so addressing my conceptual concerns, even without running more experiments, would lead to a pretty strong submission overall.

[A]: Sabour, Sara, et al. "Unsupervised part representation by flow capsules." International Conference on Machine Learning. PMLR, 2021.

[B]: Bear, Daniel M., et al. "Learning physical graph representations from visual scenes." arXiv preprint arXiv:2006.12373 (2020).

[C]: Kipf, Thomas, Elise van der Pol, and Max Welling. "Contrastive learning of structured world models." arXiv preprint arXiv:1911.12247 (2019).

[D]: Hénaff, Olivier J., et al. "Efficient visual pretraining with contrastive detection." arXiv preprint arXiv:2103.10957 (2021).
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This submission initially received mixed reviews (two weak rejects, one weak accept). After the rebuttal, the reviews remain mixed (two weak accepts, one weak reject).  The AC has carefully read the paper, reviews, and rebuttals.  The AC agrees with the concerns raised by the reviewers, including the definition of parts, the lack of discussion on methods using motion, and the missing discussion on the number of parts (K). The AC also believes this submission has made enough contributions to warrant acceptance, given its technical innovations, results, and analyses. In the camera ready, the authors should definitely discuss the related work on using motion to discover object parts, in particular those raised by reviewer ogkV:

[A]: Sabour, Sara, et al. "Unsupervised part representation by flow capsules." International Conference on Machine Learning. PMLR, 2021.

[B]: Bear, Daniel M., et al. "Learning physical graph representations from visual scenes." arXiv preprint arXiv:2006.12373 (2020).
Decision: Accept (Poster)