Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a way of learning a capsule-based representation of 3D point clouds. The representation decomposes an input point cloud into parts, each corresponding to a capsule. This decomposition is trained to be consistent across SE(3) transformations of the point cloud using loss terms that encourage feature invariance and pose equivariance across two random presentations of the input point cloud. Moreover, the model learns a canonical pose for each part that is tied to the identity of the object (not its presented pose).

The experiments show that
1. The reconstruction quality is high, showing that the model has capacity to fit point clouds from ShapeNet.
1. The canonical pose is indeed consistent (measured by low std deviation in relative angle of rotation).
1. The model can be used for pairwise registration (but DeepGMR does even better using RRI features, but doesn't do any of the other things this model can do.)
1. The learned features lend themselves well to unsupervised clustering/classification.
1. All the proposed loss terms are useful and contribute to better reconstruction and canonicalization.
main_review: Strengths
1. The model is able to find a part decomposition that is consistent across presented views, while retaining the ability to autoencode and classify.
1. The stability of the canonical pose is impressive.
1. Good performance on standard tasks and benchmarks.
1. Adequate comparisons to previous work.


Weakness
1. Capsules are not interchangeable. For instance, if I understood correctly, given an airplane point cloud, the encoder always assigns the 1st capsule to say, the left wing, 2nd capsule to the nose, etc. In other words, the capsule indices are tied to the entity they represent, as opposed to having an unordered *set* of capsules which can be constituted into different objects. This limits the ability of this model to reuse capsules across different object types and deal with the presence of multiple objects in the same scene.
1. The model has no way to "turn off" capsules, which means that when trained on multi-category data it is forced to use the same K capsules very differently depending on the object being encoded. It is surprising however that the model still works very well for pair-wise registration and classification. It would be useful to include a brief discussion on whether the presence probabilities can be modeled within the framework of this model (similar to the Stacked Capsule Autoencoders paper).
1. The "pose" of the part is really just the centroid. It does not capture the orientation of the part, which is something one typically associates with pose. It would be useful to include a comment on why orientation was not used (perhaps it is not helpful ?).
1. Removing canonicalization loss does not seem to have a big impact (Table 4). This is perhaps explained by the fact that the reconstruction loss already encourages canonicalization because it minimizes the CD between the canonicalized input point cloud and the reconstructed one ? It would help to clarify this.

Overall, this paper makes a significant contribution by proposing a model that learns canonical part decompositions in canonical pose, while retaining the ability to remember object shape and identity.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a capsule-based network architecture that allows self-supervised representation learning from 3D point clouds.
While existing methods rely heavily on pre-canonicalized (aligned) 3D models for training, this work proposes to automatically discover the "optimal" canonical transformation for alignment while learning to decompose and reconstruct the input point clouds. 
Specifically, it exploits the transformation invariance and equivariance properties of object parts (capsules) to learn a latent representation.
The experimental results on the (unaligned) ShapeNet dataset show that the method can effectively learn to decompose the 3D point cloud into consistent parts across objects and canonicalize unaligned data.
It outperforms the state-of-the-art methods in terms of reconstruction accuracy, canonicalization accuracy and stability, as well as unsupervised classification accuracy from latent representation.
It is very encouraging to see that the need of 3D pose/viewpoint annotations can be removed and such competitive performance is reached.
main_review: [Motivation and problem setting] \
The paper addresses the problem of 3D representation learning from point cloud data, where the 3D point clouds are not aligned in the canonical space.
Compared to most existing works that assumes the availability of aligned 3D data, the proposed scenario is more practical and crucial since such pose/viewpoint annotations of real-world 3D objects are usually hard to obtain.
By removing the need of pre-canonicalized 3D data for training, I believe this work makes a significant step towards learning from real-world 3D data with diverse transformations.


\
[Method] \
The network design, training strategies, and losses are well-justified and technically sound.
The proposed method follows an interesting line of work with part-based representation. 
Object parts are shown to be a good mid-level representation which is faithful to structural details, consistent across instances, and beneficial for pose/shape ambiguity.

From my understanding, equations (5) and (10) are individually solved in each branch in Figure 1 with distinct transformation ($T^a$ or $T^b$).
That is, $\theta_k$ and $\bar{\theta_k}$ are both from the top branch (${\theta_k}^a$ and ${\bar{\theta_k}}^a$) or both from the bottom branch  (${\theta_k}^b$ and ${\bar{\theta_k}}^b$).
If so, Figure 1 is a little confusing since $\theta_k$ and $\bar{\theta_k}$ for shape matching seem to come from the top and bottom branch, respectively.


\
[Experiments] \
The experimental settings with de-canonicalized (unaligned) ShapeNet data and mStd metric are reasonable. 
The random transformation includes rotation and translation within the range [0.2, 0.2].
I am curious if the authors have experimented with 1) a wider range of translation, 2) random scaling, and 3) decoupling the three transformations (e.g. only apply rotation).
Since addressing unaligned training data is a major contribution of this paper, it would be important investigate the effect of different transformations.

Both the quantitative and qualitative results show a significant and consistent improvement compared to the state-of-the-art methods on the synthetic ShapeNet dataset.
It is not necessary but would be better to compare with more existing methods and on multiple datasets.

For auto-encoding reconstruction, is it possible to also evaluate the performance of different part-based representations like multi-mesh and implicit functions?
It should be possible to sample 3D points from mesh or implicit functions and calculate Chamfer distance.
I believe this would make the proposed representation more convincing, and the results on unaligned data would be very helpful for follow-up research.

Table 7 in the supplemental document shows the ablation studies on the number of capsules. 
Are the results from multi-class training and evaluation?
It seems that there is an optimal number of capsules to best represent the 3D objects, and I suspect it varies between object classes.
I am also curious if the authors have tried fixing the descriptor dimension while increasing the number of capsule, since it is strange that more capsules don't lead to higher reconstruction accuracy.

I find it surprising that the network can figure out the optimal canonical pose and capsule configuration of multiple classes as the same time.
The detailed (per-class) quantitative results of multi-class training are missing, and the paper does not specified whether the qualitatively results are from single-class or multi-class training.
It is hard to observe the pros and cons of multi-class training.

Finally, I wonder if the authors have observed any failure cases of the proposed method.
I would like to see more discussion about the limitation of representation and more qualitative results of failure cases.


\
[Writing] \
The paper is well-written and easy to follow.

It is a little unclear to me which parts (losses/networks/representation) of the proposed method are from the baselines (AtlasNetV2 and 3D-PointCapsNet) and which are novel.
Maybe the authors can clarify it in the introduction or related work.

The word "object-centric" is mentioned multiple times.
Do "object-centric reference frame" in L116 and "object-centric coordinate frame" in L329 both mean the (optimal) canonical frame found by the network?


\
[Reference] \
Some missing reference I feel relative to the topic:
[1] Luo, Tiange, et al. "Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories." ICLR. 2019.
[2] Paschalidou, Despoina, Luc Van Gool, and Andreas Geiger. "Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image." CVPR. 2020.
[3] Kawana, Yuki, Yusuke Mukuta, and Tatsuya Harada. "Neural star domain as primitive representation." NeurIPS. 2020.

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work in Section 5.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes to learn a Canonical Capsule decomposition of shape point clouds in a self-supervised manner. 
Given an unaligned 3D point cloud dataset, the proposed method constructs pairwise data with random SE(3) transformations, and then extracts the per-capsule pose and a transformation-invariant descriptor.
The experiment results show the importance of the object-centric representation with capsule decomposition in many applications, including reconstruction, canonicalization, and classification.

main_review: The proposed pipeline learns, in an unsupervised manner, the canonicalization of 3D point clouds from aligned/unaligned point cloud datasets. The method is described clearly and could be reproduced.
The comparison, in each application, against 3D-PointCapNet, AtlasNetV2, and AtlasNetV2 w/ STN, adequately shows the significant improvement brought by canonicalization.

I have a few questions:
1. When training on 13 categories, is there any kind of consistency between the corresponding capsules for point clouds coming from different categories?
2. It's better to show the reconstruction results of AtlasNetV2 w/ STN in Figure 2.
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a self-supervised capsule architecture for 3D point clouds. In order to circumvent the customary need to train on pre-aligned datasets, the authors introduce the Canonical Capsules to compute the K-part decomposition of a point cloud that allows unsupervised "object-centric" learning of 3D representations without requiring a semantically aligned dataset. The main idea of the canonical capsules is to enforce invariance/equivariance by relating the decomposition result of two random rigid transformations.  The authors compare the proposed method against state-of-the-art methods on 3D point cloud reconstruction, canonicalization, and unsupervised classification.
main_review: ### Originality

The idea of the framework is basically enforcing the invariance/equivariance of the decomposition result of two random rigid transformations and use a canonicalizer to learn object-centric 3D representations. The idea is quite interesting and novel to some extent.


### Clarity

The paper is well written. The motivation and the idea of the proposed framework is well explained and could be followed even though I am not very familiar with this topic. 


### Quality and Significance

I am not an expert on this topic. Overall, the method looks reasonable to me and the authors conduct comprehensive experiments.  One small question: could you please give more explanation why the proposed method becomes unstable when using RRI features? 

limitations_and_societal_impact: Yes. The authors have addressed the limitation and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a self-supervised capsule based architecture for 3D point cloud. The main goal is to learn a  canonicalization operation that allows unbiased object-centric reasoning.  The idea is that an object is decomposed to K keypoints using canonical capsules, the whole system is trained in an unsupervised fashion based on pairs of randomly rotated/translated  of the same object. The capsule pose equivariance is enforced by requiring the equivariance of two keypoint sets based on the known relative transformation of the pair images and the capsule descriptor invariance is enforced by matching the pair keypoints. The proposed framework have been evaluated in 3D autoencoding and reconstruction, unsupervised classification and canonicalization on ShapeNet dataset.
main_review: Pros: 
- The paper is generally well written and easy to follow. 
- The unsupervised learning framework for 3D canonicalization is very novel and seems effective in classification, reconstruction and canonicalization tasks.  
- The proposed framework outperforms the baseline in all 3 tasks. 

Cons/ questions/ clarifications:
- Would you please clarify the encoder architecture? Does it only have one capsule layer with K capsule types as its last layer? 
- What is the effect of K on the performance and have you considered adding capsule layers as intermediate layers. 
- While the limited translation range is in favor of baseline methods in the de-canonicalizing dataset lines196-197, can you provide the performance of the proposed model for larger translation range? 
-One of the concerns in capsule based architectures are scalability of these architectures, can you provide the classification and reconstruction performance on the larger subset of  ShapeNet?  

limitations_and_societal_impact: The authors addressed the limitation and societal impact adequately in the future work and broader impact subsections.
review_title: null

Meta-review: The authors have successfully addressed the raised issues. All reviewers agree that this paper makes good contribution to the NeurIPS community. The area chairs agree and recommend this paper be accepted.
Decision: Accept (Poster)