Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes learning shape representation for the 3D shape classification task, 
by regularizing embeddings in hyperbolic space. The intuition on which this paper is based
is that complex shapes can be made by combining simpler parts and this composition can
be explained by a tree-like hierarchy. The paper proposes regularizing shape embeddings
such that simplest and most basic parts are embedded at the root level and entire shapes
are embedded at the leaf level, where embeddings are defined using hyperbolic space.
Specifically, a shape embedding is first mapped to hyperbolic space and then a mobius layer
is applied to project it to Poincare ball. The paper proposes two regularization, first is to 
encourage the whole shape embeddings to be close to leaf level and part level embeddings
are close to the root. The second regularization encourages a shape and its parts
to be closed to each other in embedding space and far from embeddings of parts from different
shapes.
This approach consistently improve the performance of shape classification task on several
neural network architectures.
strengths_and_weaknesses: **Strengths**
1. The paper is clearly written and explains the motivation of the proposed approach well.
2. The proposed approach is well described
3. All experiment details are provided ensuring reproducibility.
4. The proposed approach consistently improve the performance using several neural network architecture, ensuring generalizability.
5. All ablations are provided that all components are pertinent.
**Weakness**
1. The main weakness I see is that only shape classification is chosen to benchmark the approach. Specifically, since the embeddings are better aware of the composition the embeddings can shine in part segmentation task.
questions: 1. Is it easy to extend this approach to semantic segmentation task? If yes, why did authors not include those experiments? If not, can authors include some discussion on this matter.
limitations: Yes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposed to utilize hyperbolic space to learn part-whole hierarchy for 3D point clouds. The main idea is regularizing in the hyperbolic space if part or whole are representing same object, and if they are from different objects, the regularization is pushing them to larger distance (similar to triplet loss, or contrastive loss). The paper applied the proposed regularization to many different network architecture (e.g. DGCNN, PointNet++, PointMLP, etc) and improved the baseline across two datasets (ModelNet40 and ScanObjectNN). 
strengths_and_weaknesses: Strength:

1. Utilizing hyperbolic space for regularizing part-whole hierarchy is a new idea. The geodesic distance in the hyperbolic space naturally suitable for the tree structure of part-whole hierarchy and defining the regularization on the part-whole space makes more sense comparing with defining on the Euclidean space. (although I have some questions below)

2. The proposed regularization is agnostic to different network architectures, and the  paper experimented on multiple  different backbone for point cloud classification, each achieved improvement over compared baselines on two dataset. 

3. The paper is well written and easy to understand. 

Weakness:

1. (minor point) The definition of part-whole hierarchy seems controversial. In the paper, the part is defined as the ancestor of the whole object, and different object can share same ancestor. This is not intuitive, as one object is composed of multiple **different** parts, that means, if looking from the path from part to whole, it's not a tree structure  (one child node is strictly below one parent node). An intuitive way of defining part-whole hierarchy is the whole is the ancestor of the parts and parts can be share with many different objects. This is also how PartNet [a] (A dataset for part-whole hierarchy for 3D object) is created, and also the part-whole hierarchy mentioned in the seminal work [b]. 

2. Overclaiming. I don't agree with the author the paper is learning to "promotes the part-whole hierarchy of compositionality in the hyperbolic space" Specifically, the only thing the paper is proposing a regularization on the feature space, therefore how the object is composed with different part is not clear, there's no explicit way of representing object as a hierarchical tree using the proposed method, not mentioning the compositionality. Besides, the way the paper is defining "parts" is through subsampling a small local region of the object (Line 188), this is not a semantic part and is only a subgroup of the object, and also the definition of part-whole hierarchy is controversial as above

3. The comparison is not a fair comparison. I appreciate the paper compared with multiple network architectures and see the improvement over compared baselines. However, since the main idea is proposing a regularization loss in the hyperbolic space, a fair comparison should be apply the same regularization loss, but in the Euclidean space. The compared results is only using supervised loss for pretraining and fine-tuning with the classification loss, what about using both losses for fine-tuning? It was mentioned in Line 95 that "they are also mostly unable to improve upon state-of-the-art supervised methods when finetuned with full supervision", but there's no evidence to support this. 


4. Since the paper is proposing to learning compositionality with part-whole hierarchy, a more convincing experiment would be running with a dataset that contains part-whole hierarchy (e.g. PartNet [a]), and provide a quantitative analysis on the learnt hierarchy v.s. other baselines that learns part whole hierarchy. 


5. (minor point) In the Fig. 4 visualization, it seems the clustering is not great? If we only focus on the light green color, they are spread in many regions in the graph, does this mean the clustering of light green class is not great?

[a] PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding 
Kaichun Mo, Shilin Zhu, Angel X. Chang3, Li Yi, Subarna Tripathi, Leonidas J. Guibas, Hao Su

[b]How to represent part-whole hierarchies in a neural network
Geoffrey Hinton

questions: See Weakness section above.
limitations: See weakness section above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method for promoting the part-whole hierarchy in the learned feature space. In particular, it proposes to embed the features of a point cloud encoder into hyperbolic space. The part-whole hierarchy is enhanced through an explicit regularizer. Such a key idea is backed by the theory that the hyperbolic space (space with negative curvature) is the only space that can embed tree structures with low distortion. A regularizer layer is proposed for supervised training of point cloud classification models. It can be applied to existing architectures with a simple modification. Performance boost across a number of popular architectures is reported in the experiments. 
strengths_and_weaknesses: ### Strengths

*  The idea of promoting a part-whole hierarchy in the hyperbolic space provides a novel perspective for learning discriminative features of the point cloud, which I believe is beneficial to the community. Though it is only verified in the task of classification, I think it could be valuable for more tasks that require in-depth analysis of part hierarchies, e.g. matching incomplete point cloud, point cloud generation from parts, etc.  

* The experiments have shown steady performance improvement by applying the proposed method to existing mainstream backbones.

* Nice visualization of the formed feature space for a better understanding of the effectiveness of the proposed approach.

* Code is provided for reproduction. 

### Weaknesses

* Since hyperbolic learning is new to the point cloud analysis community, more insights on the technical part should be given to assist the audience to comprehend why the proposed method could work as expected. I would point out what should be improved in the questions section.

* The paper claims the proposed method can be applied to any existing architecture. However, the result section only shows it is applied to a small subset of methods that are compared. This makes me wonder if the results are cherry-picked. Can it provide a performance boost to arbitrary architecture?  
questions: * It is not clear to me why Equation (5) can promote part embeddings to lie closer to the center while whole embeddings to the edge. The main reason is that Equation (5) only specifies constraints on the distance between the embedding of whole and part. It only encourages the part and the whole feature to stay apart. Why the part feature (instead of the whole feature) is guaranteed to be pushed to the center? What is the technical insight behind it?

* Why hierarchy learning can be beneficial to the task of classification? In the end, only the whole shape is used for classification, and the parts are not used as the input. What is the advantage of the proposed method over a naive method that simply applies contrastive learning on the feature space of the whole shape?
limitations: No limitations or potential negative societal impact are provided in the paper. I would recommend the authors provide some failure cases or limitations in the rebuttal (if any).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to embed the features of a point cloud classifier into the hyperbolic space and explicitly regularize the space to account for the part-whole hierarchy. To do this, it employs a hyperbolic neural network [10] and introduces losses to regularize the part-whole hierarchy.

strengths_and_weaknesses: Strengths:
It seems the proposed use of hyperbolic neural network with regularization is able to improve accuracy of different point cloud classification backbones. The experiments demonstrated the benefit of the proposed work. 


Weakness:

The intuition of the proposed method looks handwavy to me. 

In literature, when talking about part-whole hierarchy, it almost always refers to a single object consists of multiple subparts, and the subparts further dissolve into smaller parts. The parts need not to be the same and most times they are assumed to be a mixture of different parts. However, in this work, it is in an unusual opposite direction, i.e. different objects share a single “atom” part and the full objects are “grown” piece by piece in sequential order (as shown in figure 2). Such definition of the hierarchy looks suspicious to me, as there shouldn’t be an unique sequential order / hierarchy to define how an object instance is composed in piecewise order, and it makes less sense to require different objects correspond to a single “common part ancestor”. This is in contrast to embedding WordNet (in the original hyperbolic neural network paper) where the word hierarchy is uniquely defined according to categorical relationships. 

Also, the way the author designed the contrastive and hierarchical loss (equation 5) is not fully justified by the author’s definition of part-whole hierarchy. The loss in equation 5 only enforces the embedding of the whole object to differ from the embedding of parts, in other words it only distinguishes the last level of hierarchy against the rest levels, but there is no regularization to distinguish subparts between intermediate levels.  Looking at the ablation in Table 4, it does seem that the hierarchical loss make little difference to baseline. Could the author also supply a version of DGCNN+HyCoRe without $R_\text{hier}$? The visualization in Figure 4 is not that helpful either, it is hard to tell the size of each points when everything is densely packed, and not straightforward to see the quality of the learned “part-whold hierarchy” other than the clustering.

Overall, I feel “hierarchy” may not be a good explanation to what the authors actually did. Probably just part-whole contrastive learning (without the hierarchical part) is more appropriate. 



questions: (1) further discussion of the definition of the part-whole hierarchy and its difference to object-part hierarchy usually defined in literature.

(2) comments on if $R_\text{hier}$ is sufficient to regularize hierchy with multiple levels, and also supply the result of DGCNN+HyCoRe without  $R_\text{hier}$

(3) improve visualization of figure 4, give more visual example of the pointcloud & embedding pairs. 

See discussions above for explanations.
limitations: I could not find a limitation section in the draft.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a novel representation for latent space of point clouds to capture and take advantage of the hierarchical nature of the underlying shapes. The proposed method encourages point clouds to be projected to a hyperbolic space, where common simple structures are forced to the center of the space, and the complex shapes are moved to the edge.

To achieve this they propose a module that can be added onto existing frameworks. The module takes the feature vector of a set of points representing a shape and projects it to a Poincarè ball. To achieve the proposed hierarchical representation, they also extract partial point cloud from the same shape and other shape as input and use it in the regularization term. The constraints push the partial point clouds to the center, while keeping the parts of the same point cloud close and pushing back parts from other point clouds. 

The regularization strategy establishes a more effective latent feature space, as seen from the classification accuracies on the benchmark datasets ModelNet40 and ScanObjectNN, as well as from the visualization of the resulting space, which shows features from different classes are kept apart.
strengths_and_weaknesses: ### Strengths

The representation is very novel and interesting, as it attempts to take advantage of the hyperbolic space in order to establish hierarchical relationships that starts from common simple parts to wholistic characteristic shapes. The authors thoroughly explain the geometric background of the proposal, and define a sound and practical constraints that drives data to be in the desired respective positions in the hierarchy.

The training procedure of preparing partial shapes to establish this hierarchy is also unique. The resulting feature space, as illustrated in Fig. 4, suggests that the constraints on the feature space have effectively mapped the latent features of even partial data to the desired locations.

The fact that the proposed module can be placed on top of existing point cloud analysis backbone networks is a huge advantage. By employing this representation, the conventional methods gain massive boosts in classification accuracy, which can be observed from Tables 1 and 2. 

### Weaknesses

The training procedure seems to require effort. From the text, the partial shapes have to be prepared at each epoch just to calculate the triplet loss defined in the paper. As the process is rather random, we can easily imagine the training process taking much longer than most existing frameworks. The burden of the training process seems to be excluded in the experimental section. 

It is doubtful whether the claimed hierarchy is actually achieved by the proposal. The colormap in Fig. 4 seems to be well-organized near the edge of the ball, but seems rather random near the middle. The interpolation results in Fig.2 of the appendix, despite the efforts to incorporate various partial shapes in the training phase, also isn't as convincing as desired. They are definitely smaller in size, but the claimed shape commonality is difficult to observe. 

Despite the efforts to consider parts of shapes, it is disappointing that the authors do not include any point segmentation task, which is deeply related to local information of shapes, in the evaluation. One can easily imagine using the $\mathbf{z}_{whole}$ feature as the global feature, which can be concatenated with conventional point-wise feature to conduct such task.
questions: I wondered if the partial point clouds would contaminate the feature space of the backbone network. Have the authors tried freezing the backbone networks so that the feature calculation backbone network is not affected by the triplet loss?

I also wondered what would happen if the partial point clouds created in each epoch for the constraints were centralized. It would seem to lead to a more robust common part ancestor, but leading to difficulty in establishing hierarchy between whole and partial shapes.

If segmentation was conducted in the suggested way in the section above, how would the results be? It is a shame that partial information is only utilized to map the point clouds in the hyperbolic space. All the efforts to include such data do not seem to be fully utilized.

Have the authors attempted to use other sets of data for visualization? Although slightly out of context, it may have been better to use a more simple set of targets for the visualization task, such as humans in different poses. Human models have parts that can be used as the partial data, and commonality and transformation are easier to observe.
limitations: The authors do mention the procedure of preparing the partial shapes to be the main limitation of the proposal. The method currently takes a random point and collects $N'$ nearest points to define one partial shape. Using segment data could improve the partial representation of the proposed feature space. 

Also, I believe lack of a framework to use the obtained feature for shape segmentation is also a drawback, as the authors go so far as to including parts of shapes as input to the proposed framework.
review_title: null

Meta-review: metareview: The paper presents a regularization for point cloud representation learning aiming to promote a part-whole hierarchy through a hyperbolic space. Most of the reviewers agree the idea of using the hyperbolic space is new and interesting. The experiment results seem to be sufficient. There was some confusion on how part is defined and compositionality in the paper. But the AC feels the paper has sufficient merit to be published It is required that the authors incorporate the reviewer feedbacks in the revised manuscripts. 
confidence: Certain
Decision: Accept