Canonical Capsules: Self-Supervised Capsules in Canonical Pose
Authors: Weiwei Sun
Andrea Tagliasacchi
Boyang Deng
Sara Sabour
Soroosh Yazdani
Geoffrey Hinton
Kwang Moo Yi
Venue: NeurIPS 2021 Poster (2021)
Keywords: object-centric representation learning
capsules
primary capsules
unsupervised
self-supervised
3D point clouds

Abstract: We propose a self-supervised capsule architecture for 3D point clouds. We compute capsule decompositions of objects through permutation-equivariant attention, and self-supervise the process by training with pairs of randomly rotated objects. Our key idea is to aggregate the attention masks into semantic keypoints, and use these to supervise a decomposition that satisfies the capsule invariance/equivariance properties. This not only enables the training of a semantically consistent decomposition, but also allows us to learn a canonicalization operation that enables object-centric reasoning. To train our neural network we require neither classification labels nor manually-aligned training datasets. Yet, by learning an object-centric representation in a self-supervised manner, our method outperforms the state-of-the-art on 3D point cloud reconstruction, canonicalization, and unsupervised classification.

Canonical Capsules:
Self-Supervised Capsules in Canonical Pose
Weiwei I4n
1,,,+
Andrea Tagliasacchi2 2,3+
Boyang Deng
Sara Sabour
Soroosh Yazdani
Geoffrey Hinton 2.3
Kwang Moo Yi4.4
University of British Columbia, Unniversity of Toronto, GGoogle Research, *University Victoria, "equal contributions
htg..aaees..ppll.ggggg....bbbb io
Abstract
We propose self-superrised capsule architecture for 3D point clouds. We compute capsule decompositions of objects through permutation-eequiaaiin self- the process by training with pairs of randomly rotated objects. Our key idea to aggregate the attention masks into semantic keypoints, and use these supervise decomposition that satisfies the capsule invariance/equivariance prop- erties. This not only enables the training of a semantically consistent decomposition, but also allows reasoning. To train our neural network we require neither classification labels nor manually-alignn training datasets. Yet, by learning an object-centric representa- tion in a self-supervieed manner, our method outperforms the state-of-the art on 3D point cloud reconstruction, canonicalization, and unsupervised classification.
attention, and
learn canonicalization operation that enables object-centric
Introduction
While this task has traditionally relied on large annotated datasets 42 22 unsupervised approaches that utilize self-supervision [5] have emerged remove the need for labels. Recently, researchers have attempted to extend these methods to work on 3D point clouds [59] but the field of unsupervised 3D learning remains relatively uncharted. Conversely, researchers have been extensively investigating 3D deep representations for shape auto-encoding 61 19 33 16. making one wonder whether these discoveries can now benefit from unsupervised learning for tasks other than auto-encoding.
Understanding objects is one of the core problems of computer vision
32
14
Importantly, these recent methods for 3D deep representation learning are not entirely unsupervised. Whether using point clouds [61), meshes [19], implicits [3], they owe much of their success to the bias within the dataset that was used for training. Specifically, all 3D models in the popular ShapeNet 3 dataset are "object-centric" they are pre-canonicalized even more importantly. with an orientation that synchronizes object semantics to Euclidean frame axes (c.g. airplane cockpit is always along +y, car wheels always touch 2 0). Differentiable 3D decoders are heavily affected by the consistent alignment of their output with an Euclidean frame [6 as local-to-global transformations cannot be easily learnt by fully connected layers. As we will show in Section these methods fail in the absence of pre-alignment, even when data
unit bounding box, and,
Auto-encoding also at times referred to as "reconstruction" or "shape-space" learning.
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

augmentation is used. A concurrent work 45 also recognizes this problem and proposes separate learnt canonicalizer, which is shown helpful in downstream classification tasks.
In this work, we leverage the modeling paradigm of capsule networks [23] In capsule networks, scene is perceived via its decomposition into part hierarchies, and each part is represented with a (pose, descriptor) pair: I The capsule pose specifies the frame of reference of a part, and hence should be transformation equivariant: ( The capsule descriptor specifies the appearance of part, and hence should be transformation invariant. Thus, one does not have to worry how the data is oriented or translated, as these changes can be encoded within the capsule representation.
We introduce Canonical Capsules, a novel capsule architecture to compute a K-part decomposition of a point cloud. We train our network by feeding pairs of randomly rotated/tramslated copies of the same shape (i.e. siamese training) hence removing the requirement of pre-aligned training datasets. We then decompose the point cloud by assigning each point into one of the K parts via attention, which we aggregate into K keypoints. Equivariance is then enforced by requiring the two keypoint sets only differ by the known (relative) transformation (i.e. a form of selr-suueriision). For invariance, we simply ask that the descriptors of each keypoint of the two instances match.
With Canonical Capsules, we exploit our decomposition to recover canonical frame that allows unsupervised "object-centee" centricc"" learning of 3D deep representations without requiring a semantically aligned dataset. We achieve this task by regressing canonical capsule poses from capsule descriptors via a deep network, and computing a canonicalizing transformation by solving a problem [44] This not only allows more effective shape auto-encoding, but our experiments confirm this results in a latent representation that is more effective in unsupervised classification tasks. Note canonicalizing transformations are also learnt in self-supervised fashion, by only training on randomly transformed point clouds.
that, like our decomposition,
Contributions. In summary, in this paper we:
propose an architecture for 3D self-supervised learning based on capsule decomposition; enable object-centric unsupervised learning by introducing a learned canonical frame of reference; achieve state-of-the-ae performance 3D point cloud auto- auto--nncdingrreoossuuciiio, canonicalization, and unsupervised classification.
2
Related works
Convolutional Neural Networks lack equivariance to rigid transformations, despite their pivotal role describing the structure the 3D scene behind a 2D image. One promising approach to overcome this shortcoming is to add equivariance under a group action in each layer 49 9 In our work, we remove the need for a global (E(3)-equivarii
network by canonicalizing the input.
Capsule Networks. Capsule Networks 23 have been proposed to overcome this issue towards relational and hierarchical understanding of natural images. Of particular relevance our work, are methods that apply capsule networks to 3D input data 64
but note these methods are not unsupervised, as they either rely on classification supervision
65 46
or on datasets that present significant inductive bias in the form of pre-al mlignmenment [64] In this paper, we take inspiration from the recent Stacked Capsule Auto-Encoders [00, which shows how capsule-style reasoning can be effective as long as the primary capsules can be trained in a self-supervised fashion (i.e. via reconstruction losses). The natural question, which we answer in this paper, "How can we engineer networks that generate 3D primary capsules in an unsupervised fashion?"
Deep 3D representations. Reconstructing 3D objects requires effective inductive biases about 3D vision and 3D geometry. When the input is images, the core challenge is how to encode 3D projective geometry concepts into the model. This can be achieved by explicitly modeling multi-view geometry (28), by attempting to learn it [ or by hybrid solutions [60] But even when input is 3D, there are still significant challenges.
still not clear which is the 3D representation that is most
amenable to deep learning. Researchers proposed the use of meshes B1] voxels (54 55). surface
patches 19 10,, and implicit functions 35
3K 7 Unfortunately, the importance of geometric
structures (i.e. part-to-whole relationships) is often overlooked. Recent works have tried to close
this gap by using part decomposition consisting of oriented boxes [50] ellipsoids 17 I6, convex
polytopes [[]] and grids [2]. However, previously discussed, most of these still heavily rely

---

F.A
(..0..0)
Clensbeee......
(0x)
(0.,8))
[02.3.)
D2
C
Fetuanee
(02)
(0x,0,)
Dr
TP
((0L.8)) (.))
Figure 1: Framework- We learn capsule encoder for 3D point clouds by relating the decomposi- tion result of two random rigid transformations T and To, of given point cloud, i.e., a Siamese training setup. We learn the parameters of encoder &, a per-capsule decoder DE. well network that represents a learnt canonical frame K.. For illustrative purposes, we shade-out the outputs that do not flow forward, and with E summarize the aggregations in
pre-aligned training dataset; our paper attempts to bridge this gap, allowing learning of structured 3D representations without requiring pre-aligned data.
Canonicalization. One way to circumvent the requirement of pre-aligned datasets is to rely on meth- ods capable of registering point cloud into a canonical frame. The recently proposed CaSPR [39 fulfills this premise, but requires ground-truth canonical point clouds in the form of normalized object coordinate spaces 222 for supervision. Similarly, (20) regresses each view's pose relative to the canonical pose, but still requires weak annotations in the form multiple partial views. C3DPO 3 learns canonical 3D frame based on 2D input keypoints. In contrast to these methods, our solution is completely self-supervised. The concurrent Compass [45] also learns to canonicalize in self-supervised fashion, but as the process is not end-to-end, this results in worse performance than ours, as will be shown in Section
Registration. Besides canonicalization, many pairwise registration techniques based on deep learning have been proposed (e.g. [56, ( ), even using semantic keypoints and symmetry to perform
task [15]. These methods typically register pair of instances from the same class, but lack the ability to jointly and consistently register all instances shared canonical frame.
3 Method
Our network trains on unaligned point clouds as illustrated Figure | we train a network that through Siamese training
decomposes point clouds into parts, and enforce invariance/equivariaan setup [48]. We then canonicalize the point cloud to a learnt frame of reference, and perform auto- encoding in this coordinate space. The losses employed to train E, K. and D, will be covered in Section 3.1 while the details of their architecture are in Section 3.2
Decomposition. In more detail, given a point cloud P E PxD of P points in D dimensions (in our case D=3), we perturb with two random transformations T To E SE(D) (D) to produce point clouds pa ps, We then use shared permutation--quiiariiie attention map A E PxX for K capsules, as well as per-point feature map F E PxC with C channels:
capsule encoder
compute K-fold
F. ((F))(P).
(1)
where we drop the superscript indexing the Siamese branch for simplicity. From these attention masks, we then compute, for the k-th capsule its pose 0₂ R parameterized by its location in 3D space, and the corresponding capsule descriptor B C:C
0, ApAPP Ex Ap,
B₁ Ap Ep Ap.
Hence, as long as is invariant w.r.t. rigid transformations of the pose A will be transformation equivariant, and the descriptor B3 will be transformation invariant. Note that this simplifies the design (and training) of the encoder ε, which only needs to be invariant, rather than equivariant [9) 46
Canonicalization. Simply enforcing invariance and equivariance with the above framework is not enough learn 3D representations that are object-centric, as we lack an (unsupervised) mechanism to bring information into shared "object-centric" reference frame. Furthermore, the "right" canonical

---

frame is nothing but a convention, thus we need a mechanism that allows the network to make choice a choice, however, that must then be consistent across all objects. For example, a learnt canonical frame where the cockpit of airplanes is consistently positioned along + is just as good as a canonical frame where it is positioned along the +y axis. To address this, we propose to link the capsule deseriptors to the capsule poses in canonical space, that is, we ask that objects with similar appearance to be located in similar Euclidean neighborhoods in canonical space, We achieve this by regressing canonical capsules poses (i.e. canonical keypoints) 0 E IR K xx3 using the descriptors G E R RxxC via a fully connected deep network XC:
θ KC (3)
Because fully connected layers are biased towards learning low-frequency representations regressor also acts as a regularizer that enforces semantic locality.
e this
Auto-encoding. Finally, in the learnt canonical frame of reference, to train the capsule descriptors via auto-encoding, we reconstruct the point clouds with per-capsule decoders Dy:
P
Uw (Dx (ROx + (,,3,)
where U denotes the union operator. The canonicalizing transformation T (R, t) can be readily computed by solving a shape-mas matching problem [4), thanks to the property that our capsule poses and regressed keypoints are in one-to-one correspondence:
R. t arg min
1
Σ (RO, +t) 0xx
(5)
K
R.t
While the reconstruction in is in canonical frame, note it is trivial to transform the point cloud back to the original coordinate system after reconstruction, the transformation T is available.
3.1 Losses
As common in unsupervised methods, our framework relies on a number of losses that control the different characteristics we seek to obtain in our representation. Note none of these losses require ground truth labels. We organize the losses according to the portion of the network they supervise: decomposition, canonicalization, and reconstruction.
Decomposition. While a transformation invariant encoder architecture should be sufficient to achieve equivariannceiivarrancc this does not prevent the encoder from producing trivial solu- tions/decompositions once trained. As capsule poses should be transformation equivariant, the poses of the two rotation augmentations 0% and 0% should only differ by the (known) relative transformation:
C Cequivariance
K Σ (T)))) 0
Conversely, capsule descriptors should be transformation invariant, and as the two input points clouds are of the same object, the corresponding capsule descriptors /3 should identical:
Cinvariance
Σ K
1 - -3%
We further regularize the capsule decomposition to ensure each of the K heads roughly represent the same "amount" of the input point cloud, hence preventing degenerate (zero attention) capsules. This is achieved by penalizing the attention variance:
Σ K
1 lax K I
Lequilibrium
where Ep(Ap.)) denotes the total attention exerted by the k-th head on the point cloud.
Finally, to facilitate the training process, ask for capsules to learn localized representation of geometry. We express the spatial extent of a capsule by computing first-order moments of the represented points with respect to the capsule pose 0::
Eiocatizaion
Σ k
p.
0
-Ppll
(9)
KL
ak

---

Canonicalization. To train our canonicalizer K. we relate the predicted capsule poses to regressed canonical capsule poses via the optimal rigid transformation from (
Leanonical = K IIRO + +t) 1x1
Σ K
(10)
Recall that R. and T are obtained through a differentiable process. Thus, this loss is forcing the aggregated pose 0g to agree with the one that goes through the regression path, 0z. Now, since 03. is regressed solely from the set of capsule descriptors, similar shapes will result in similar canonical keypoints, and the coordinate system 0y is one that employs Euclidean space to encode semantics.
Reconstruction. To learn canonical capsule descriptors in an unsupervised fashion, we rely on an auto-encoding task. We train the decoders { by minimizing the Chamfer Distance (CD) between the (canonicalized) input point cloud and the reconstructed one, in 19
Lecon = CD RP + t, P)
(11)
3.2 Network Architectures
We briefly summarize implementation details, including the network architecture; for further details, please refer to the
material.
Encoder - 3. Our architecture is based on the one suggested in 47 pointnet-like architecture with residual connections and attentive context normalization. We utilize Batch Normalization 26 instead of the Group Normalization (58]. which trained faster in our experiments. We further extend their method to have multiple attention maps, where each attention map corresponds to a capsule.
Decoder- D. The decoder from 4 operates on a per-capsule basis. Our decoder architecture is similar AtlasNetV2 122 (with trainable grids). The difference is that translate the per-capsule decoded point cloud by the corresponding capsule pose.
Regressor K. We concatenate the descriptors and apply series of fully connected layers with ReLU activation to regress the P capsule locations. At the output layer we use a linear activation and subtract the mean of the outputs make regressed locations zero-centered in the canonical frame.
Canonicalizing the descriptors. As our descriptors are only approximately rotation invariant (via i) we found useful to re-extract the capsule descriptors 3g after canonicalization. Specifi- cally, we compute with the same encoder setup, but with P-RP++ +T instead of P and use it to compute Br we validate this empirically in the supplementare material.
4 Results
We first discuss the experimental setup, and then validate our method on a variety of tasks: auto- encoding, canonicalization, and unsupervised classification. While the task differs, learning process remains the same: we learn capsules by reconstructing objects a learnt canonical frame. We also provide ablation study, which is expanded in detail in the supplementary material, where we provide additional qualitative results.
4.1 Experimental setup
To evaluate our method, we rely the ShapeNet (Core) dataset 3 We follow the category choices from AtlasNetV2 [1],, using the airplane and chair classes for oigle-category experiments, while for multi-category experiments we use all 13 classes: airplane, bench, cabinet, car, chair, monitor, lamp, speaker, firearm, couch, table, cellphone, and watercraft. To make our results most compatible with those reported in the literature, the train, and 7943 shapes in the test set. Unless otherwise noted, we randomly sample 1024 points from the object surface for each shape create 3D point clouds.
also use the same splits in AtlasNetV2 [12] 31747 shapes in
De-canonicalizing the dataset. As discussed in the introduction, ShapeNet (Core) contains substan- tial inductive bias in the form of consistent semantic alignment. To remove this bias, we create random
Available researchers for non-commercial research and educational use.
5

---

SE(3) transformations. and apply them to each point cloud. We first generate uniformly sampled random rotations, and add uniformly sampled random translations within the range [-0.2, 0.2), where the bounding volume of the shape ranges in [-1, +1I. Note the relatively limited translation range chosen to give state-of-the-art methods a chance to compete with our solution. We then use the relative transformation between the point clouds extracted from this ground-truth transformation to evaluate our methods. We refer to this unaligned version of the ShapeNet Core dataset as the unaligned setup, and using the vanilla ShapeNet Core dataset as the aligned setup. For the aligned setup, as there is no need for equivariance adaptation, we simply train our method without the random transformations, and so Lequivariance and Linvariance not used. This setup is to simply demonstrate how Canonical Capsules would perform in the presence of a dataset bias.
We emphasize here that a proper generation of random rotation is important, While some existing works have generated them by uniformly sampling the degrees of freedom of an Euler-angle repre- sentation, this is known to be an incorrect way to sample random rotations [1], leading to biases in material.
the generated dataset; see the
Implementation details. For all our experiments we use the Adam optimizer 22 with an initial learning rate of 0.001 and decay rate of 0.1. We train for 325 epochs for the aligned setup match the V22 12 original setup. For the unaligned setting, as the problem is harder, we train for a longer number of 450 epochs. We use a batch size of 16. The training rate is 22. iters/sec. We train each model on single NVidia V100 GPU. Unless stated otherwise, we use k=10 capsules and capsule descriptors of dimension C=128. We train three models with our method: two that are single-catege engle-category (i.e., for airplane and chairs), and one that is multi-category (i.e., all 13 classes). To set the weights for each loss term, we rely on the reconstruction performance (CD) in the training set. We set weights to be one for all terms except for Lequivariance (5) and Lequiiibium (10-3), In the aligned case, because Lequivariance and Cinvariance are not needed (always zero), we reduce the weights for the other decomposition losses by 10 Liocalization to 10-3 and Lequilibrium to 10-6.
4.2 Auto-encoding- Figure 2 and Table
We evaluate the performance of our method for the task that was used to train the network - reconstruc- tion auto-encoding against three baselines (trained in both single-category and multi-category variants):
Table 1: Auto-encoding / quantitative - -Per formance in terms of Chamfer distance with 1024 points per point cloud - metric is multi- plied by 10 as in 12
I AD-PointCapsNet [44], an auto-encoder for 3D point clouds that utilize a capsule architecture; @ At- lasNetV2 [12, a state-of-the-art auto-encoder which utilizes multi-head patch-based decoder; ( Atlas- NetV2 I2 with ppatial-transfommet network (STN) aligner from PointNet [36], a baseline with canoni- calization. We do not compare against [46] as unfor- tunately source code is not publicly available.
Mothod
Airplane
Chair
Multi
Unaliqzed Allgned
3D PoiniCapcNet
1.94
3.30
2.49
AtlasNetV?
1.28
2.36
2.14
Our method
0.96
1.99
1.76
3D-PoinCaapSNet
111
5.58
7.57
4.66
AtlasNerV2 12]
2,80
3.98
3.08
AtlasNetV2
12
w/STN
1,90
2.99
2.60
Our method
1.11
2.58
2.22
Quantitative analysis - Table i
We achieve ofate-of-the-art performance in both the aligned and
unaligned settings. The wider margin in the unaligned setup indicates tackling this more challenging scenario damages the performance of AtlasNetV2 [12] and 3D-PointCapsNet [64] much more than
our method We also include a variant of AtlasNetV2 for which STN (Spatial Transformer Network) is used to pre-align the point clouds 33) demonstrating how the simplest form of pre- aligner/canonicalizer is not sufficient.
Qualitative analysis - Figure 2 We illustrate our decomposition- based reconstruction of 3D point clouds, as well as the reconstructions of 3DD-PointCapsNet [64] and AtlasNetV2 [[2]. As shown, even in the unaligned setup, our method is able to provide semantically consistent capsule decompositions - e.g. the wings of the airplane have consistent colours, and when aligned in the canonical frame, the different airplane instances are well-aligned. Compared to AtlasNetV2 [12] and DD-PointCapsNet [64], the reconstruction quality is also visibly improved: we better preserve details along the engines of the airplane, or the thin structures of the bench; note also that the decompositions
'Results in this table differ slightly from what is reported in the original papers as we use 1024 points to speed-up experiments throughout our paper. However, in the suppl Lementary material the same trend holds regardless of the number of points, and match with what is reported in the original papers with 2500 points.

---

Input
Our capsule decomposition
Our reconstruction in canonical frame
reconstruction in imput frume
Our
3D-PointCapsNet
AtlasNetV2
reconstruction
reconstruction
Example decomposition and reconstruction results setup using Canonical Capsules on several unaligned point cloud instances from the test We color each Canonical Capsule with unique colour, and similarly color "patches" from the reconstruction heads of SD-PoiitCapsset [64] and AtlasNetV2 [12]. Canonical Capsules provide semantically consistent decomposition that is aligned in canonical frame, leading to improved reconstruction quality.
Figure 2: Auto-encoding qualitative
are semantically consistent our examples. Results are better appreciated in our supplementary ma material, where we visualize the performance as we continuously traverse SE(3).
4.3 Canonicalization Table
We compare against three baselines: ! Deep Closest Points [56. deep learning- basedd pairwise point cloud registration method; @ DeepGMR 63 a state-of-the-art pairwise registration method that decomposes clouds into Gaussian mixtures and utilizes Rigorously Cotation-Invarian (RRI) features A:: ( Compass 451 a concurrent work on learnt aligament/canoiiaalzzttoo compared methods we use the official implementation. For DeepGMR we use both RRI and the typical XYZ coordinates as input. We also try our method with the RRI features, following DeepGMR'ss 's training protocol and train for 100 epochs.
For all
Metrics. To evaluate the canonicalization performance, we look into the stability of the canonicaliza- tion the shakiness shown in the videos in our supplementary material. repressentee the mean standard deviation of the rotations (mStd):
mStd = - - Σ
m (4 j=1 (4(R*) Ri R)pean))
(12)
where Z is the angular distance between two rotation matrices [8] of the j-th instance of the i-th object in canonical frame, and R is the mean rotation 21 of the i-th object. Note that with mStd we measure the stability canonicalization with respect to rotations to accommodate for methods that do not deal with translation pairwise registration methods, we also measure performance terms of the RMSE metric (3 66].
62 63
Rò is the rotation matrix
45 To allow for comparisons with
45 our method provides improved stability in canonicalization. This also provides an advantage in pairwise registration, delivering state-of-the-an results when XYZ-coordinates are used. Note that while pairwise methods can align
Quantitative analysis Table 2 Compared to Compass
7

---

Table 2: Canonicalization Quantitative evaluation for canonicalization, where we highlight signifi- cantly better performance than the concurrent work Compass [5. While pairwise registration is not our main objective, the global alignment frame created by our method still allows for effective registration (on par, or better) than the state-of-the-art.
Canonicalization (mStc) +
Pairwise registration (RMSB) L
Method
Airplane
Chair
Multi
Airplane
Chair
Multi
RRI XY7-20om
Deep Closest Points
56
0,318
0.160
0.131
DeepGMR [
0.079
0.082
0,077
Compass [5
19.105 8,278
19.508 5.730
$1.811 21.210
0,412
0.482
0.515
Our method
0.022
0.027
0.074
DeepGMM [3
0.0001
0.0001
0.0001
Our method
(anstable)
(unstable)
(unstable)
0.0006
0,0009
0.0016
two sets of given point clouds, creating canonical frame that simultaneously registers all point clouds is non-trivial extension to the problem.
When RRI is used input, method is on par with DeepGMR [63] up to level where registration
is near perfect alignment differences when errors are in the 10 ballpark are indiscernible. We note that the performance of Deep Closest Points [56] is not good as reported in the original paper, as we uniformly draw rotations from S3(3). When a sub-portion of SO(3) is used, e.g. a quarter of what are using, DCP performs relatively well (0.008 in the multi-class experiment). While curriculum learning could be used to enhance the performance of DCP, our technique does not need rely on these more complex training techniques.
We further note that, while RRI delivers good registration performance, using RRI features cause the learnt canonicalization to fail training becomes unstable. This hints that RRI features may be throwing away too much information to achieve transformation invariance. Our method using raw XYZ coordinates as input, on the other hand, provides comparable registration performance, and is able to do significantly more than just registration (i.e. classification, reconstruction).
4.4 Unsupervised classification Table 3
Beyond reconstruction and canonicalization, we evaluate the usefulness of our method via classifi- cation task that is not related any way to the losses used for training. We compute the features from the auto-encoding methods from Section 2 against those from our method (where we build features by combining pose with descriptors) to perform 13-way classification with two different techniques:
We train supervised linear Support Vector Ma- chine (SVM) on the extracted features [1 Ch. 7];
Table 3: Classification - Top-- accuracy(%)
We perform unsupervised K-Means clustering [I]
Method
SVM
K-Means
Ch. 9] and then label each cluster via bipartite matching with the actual labels through the Hun- garian algorithm.
Unaligned Aligned
3D-PointCapsNNtt
64
93.81
65.87
AtlasNetV2 [2
94.07
61.66
Our method
94.21
69.82
3D-PointCapsNee
64
71.13
14.59
Note the former provides upper bound for unsu- pervised classification, while better performance on the latter implies that the learnt features are able separate the classes into clusters that are compact (in an Euclidean sense).
AtlasNetV2 12
64.85
17.12
AtlasNerV2
12
w/ STN
78.55
20.03
Our method
87.33
43.04
Analysis of results - SVM. Note how our method provides best results in all cases, and when the dataset is not unaligned the difference is significant. This shows that, while 3D-PointCapsNet and AtlasNetV2 (with and without STN) are able to somewhat auto-encode point clouds in the unaligned setup, what they learn does not translate well to classification. However, the features learned with Canonical Capsules are more related to the semantics of the object, which helps classification.
Analysis of results K-Means. The performance gap becomes wider when K-Means is used - even in the aligned case. This could mean that the features extracted by Canonical Capsules are better suited for other unsupervised tasks, having a feature space that is close to being Euclidean in terms of semantics. The difference is striking in the wnaligned setup. We argue that these results emphasize the importance of the capsule framework jointly learning the invariances and equivariances in the
data is cardinal to unsupervised learning 25 !
8

---

Table 4: Effect of losses Reconstruction perfor- mance, and canonicalization performance when loss terms are removed.
Table 5: Backhone - Auto-encoding per- formance (Chamfer distance) when we use various permutation-invarianl backbones.
Full
-Cana
PointNet
PointNet++
DGCNN
ACNe
1.44
Std
8.278
9.983
110.174
8.411
113.204
92,970
1.28
1.34
1.21
1.11
4.5
Ablation study
To make the computational cost manageable, we perform all ablations with the airplane category (the category with most instances), and in the wnaligned setup (unless otherwise noted). Please also see supplementary material for more ablation studies.
Losses Table We analyze the importance of each loss term, with the exception of Lecon which necessary for training. losses beneficially contribute to reconstruction performance, but note how Lequiv. Llocalization and Lequilibrium affect it to a larger degree. By considering our canonicalization metric, we can motivate this outcome by observing that the method fails perform canonicalization when these losses are not employed (i.c. training collapses).
Encoder architecture Table Our method can be used with any backbone, as our main con-
tribution lies in the self-supervised canonicalization architecture. For completeness, we explore
variants of our method using different back-bonee bones: PointNet [36], PointNet++ 37. DGCNN (57 and ACNe [47]. Among these, the ACNe backbone performs best; note that all the variants in Table 5 regardless of backbone choice, significantly outperforms all other methods reported in Table !
5 Conclusions
In this paper, we provide self-supervised framework train primary capsule decompositions for 3D point clouds. We rely on a Siamese setup that allows self-supervision and auto-encoding in canonical space, circumventing the customary need train on pre-aligned datasets. Despite being trained in self-supervised fashion, our representation achieves sate--of-theann performance across auto-encoding, canonicalization and classification tasks. These results are made possible by allowing the network to learn a canonical frame of reference. We interpret this result as giving our neural networks a mechanism to construct "mental picture" of a given 3D object - so that downstream tasks exccuted within an object-centric coordinate frame.
Future work. As many objects have natural symmetries that we do consider at the moment (12 providing our canonicalizer way to encode such prior is likely to further improve the representation. We perform decomposition
single level, and it would be interesting to investigate how to effectively engineer multi-level decompositions [51]; one way could be to over-decompose the input redundant fashion (with large K), and use a downstream layers that "selects" the decomposition heads to be used (6) We would also like to extend our results to more "in-the-wild" 3D computer vision and understand whether learning object-centric representations is possible when incomplete (i.e., single view 355 occluded) data is given in input, when an entire scene with potentially multiple objects is given 43], or where our measurement of the 3D world is a single 2D image 48 or exploiting the persistence of objects in video 40
Broader impact. While our work is exclusively on 3D shapes, and thus not immediately subject to any societal concerns, it enhances how Artificial Intelligence (AI) can understand and model 3D geometry. Thus, similar to how image recognition could be misused, one should be careful when extending the use of our method. In addition, while not subject to how the data itself is aligned, the learnt canonical frame of our method is still data-driven, thus subject to any data collection biases that may exist canonicalization will favour shapes that appear more often. This should also be taken into account with care to prevent any biased decisions when utilizing our method within a decision making AI platform.

---

Acknowledgements
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google, Com- pute Canada, and Advanced Research Computing at the University of British Columbia.
References
[1] Christopher M Bishop. Pattern Recognition and Machine Learning. springer, 2006.
[2] Rohan Chabra, Jan Eric Lenssen, Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction. In European Conference on Computer Vision, 2020.
[3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An Information-Riee 3D Model Repository. arXiv Preprint, 2015.
[4] Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang. and Liang Lin. Clusternet: Deep Hicrarchical Cluster Network with Rigorously cotation-Invarian Representation for Point Cloud Analysis. Conference on Computer Vision and Pattern Recognition, 2019.
[5] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. International Conference Machine Learning, 2020.
[6] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Conference on Computer Vision and Pattern Recognition, 2020.
[7] Zhiqin Chen and Hao Zhang. Learning Implicit Fields for Generative Shape Modeling. In Conference on Computer Vision and Pattern Recognition, 2019.
[8] Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. NASA: Neural Articulated Shape Approximation. In European Conference on Computer Vision, 2020.
[9] Congyue Deng, Or Litany. Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas.
Vector neurons: general framework for -qquivrqui networks, 2021.
[10] Zhantao Deng, Jan Bednañík, Mathieu Salzmann, and Pascal Fua. Better Patch Stitching for Parametric Surface Reconstruction. arXiv Preprint, 2020.
[11] Deng, Boyang and Genova, Kyle and Yazdani, Soroosh and Bouaziz, Sofien and Hinton, Geoffrey and Tagliasacchi, Andrea. CvxNet: Learnable Convex Decomposition. In Conference on Computer Vision and Pattern Recognition, 2020.
[12] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry.
L Learning Elementary Structures for 3D Shape Generation and Matching. In Advances in Neural Information Processing Systems, 2019.
[13] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A Point Set Generation Network for 3D Object Recon- struction from Single Image. In Conference Computer Vision and Pattern Recognition, 2017.
[14] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A Discriminatively Trained, Multiscale, Deformable Part Model. In Conference on Computer Vision and Pattern Recognition, 2008.
[15] Clara Fernandez-Labradoe Ajad Chhatkuli, Danda Pani Paudel, Jose Guerrero, Cédric Demonceaux, and Luc Van Gool. Unsupervised Learning of Category-Specife, Symmetric 3D Keypoints from Point Sets. In European Conference on Computer Vision, 2020.
[16] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Deep Structured Implicit Functions. In Conference on Computer Vision and Pattern Recognition, 2020.
[17] Kyle Genova, Forrester Cole. Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning Shape Templates with Structured Implicit Functions. In International Conference Computer Vision, 2019.
[18] Zan Gojcic, Caifa Zhou, Jan D Wegner, Lconidas J Guibas, and Tolga Birdal. Learning multiview 3D point cloud registration. In Conference on Computer Vision and Pattern Recognition, 2020.
[19] Thihault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A Papier-Mäche Approach to Learning 3D Surface Generation. In Conference on Computer Vision and Puttern Recognition, 2018.
[20] Jiayuan Gu, Wei-Chiu Ma. Sivabalan Manivasagam. Wenyuan Zeng. Zihao Wang, Yuwen Xiong, Hao Su. and Raquel Urtasun. Wcakly-Supervisee 3D Shape Completion in the Wild. In European Conference on Computer Vision, 2020.
[21] Richard Hartley, Jochen Trumpf, Yuchao Dai, and Hongdong Li. Rotation averaging. International Journal of Computer Vision, 2013.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition.
In Conference on Computer Vision and Pottern Recognition, 2016.
10

---

[23] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming Auto-Encoders. In International Conference on Artificial Neural Networks, 2011.
[24] Geoffrey E Hinton and Kevin J Lang. Shape Recognition and Illusory Conjunctions. In International Joint Conference on Artificial Intelligence, 1985.
[25] Geoffrey Hinton. A Parallel Computation that Assigns Canonical Object-based Frames of Reference. In International Joint Conference on Artificial Intelligence, 1981.
[26] Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, 2015.
[27] Arthur Jacot, Franck Gahriel, and Clément Hongler. Neural Tangent Kemel:
Convergence and Generaliza-
tion in Neural Networks. Advances in Neural Information Processing Systems, 2018.
[28] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning Multi-View Stereo Machine. In Advances
Neural Information Processing Systems, 2017.
[29] D.P. Kingma and J. Ba. Adam: A Method for Stochastic Optimisation. In International Conference on I earning Representations, 2015.
[30] Adam Kosiorek. Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked Capsule Autoencoders. In Advances in Neural Information Processing Systems, 2019.
[31] Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. Deformable Shape Completion with Graph Convolutional Autoencoders. In Conference on Computer Vision and Pattern Recognition, 2018.
[32] David G Lowe. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60:91-110, 2004.
[33] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupaney Networks: Learning 3D Reconstruction in Function Space. Conference on Computer Vision and Pattern
Recognition, 2019.
[34] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi. C3dpo: Canonical 3d pose networks for non-rigid structure from motion. In Conference on Computer Vision and Pattern Recognition, 2019.
[35] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. Conference on Computer Vision and Pattern Recognition, 2019.
[36] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep Learning Point Sets for 3D
Classification and Segmentation. In Conference on Computer Vision and Pattern Recognition, 2017.
[37] Charles R Qi. Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. Advances in Neural Information Processing Systems, 2017.
[38] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You Only Look Once: Unified, Real-Time Object Detection. In Conference Computer Vision and Pattern Recognition, 2016.
[39] Davis Rempe Tolga Birdal, Yongheng Zhao, Zan Gojcie, Srinath Sridhar, and Leonidas J. Guibas. CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations. In Advances in Neural Information Processing Systems, 2020.
[40] Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E. Hinton, and David J. Fleet. Unsupervised
part representation by flow capsules. 2021.
[41] Ken Shoemake. Uniform random rotations. Graphics Gems III (IBM Version). pages 124-132, Elsevier. 1992.
[42] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations, 2015.
[43] Shuran Song. Fisher Yu, Andy Zeng. Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[44] Olga Sorkine- Sorkine-Hornung and Michael Rabinovich. Least-Squares Rigid Motion Using SVD. Computing. 2017.
[45] Riccardo Spezialetti, Federico Stella, Marlon Marcon, Luciano Silva, Samuele Salti, and Luigi Di Stefano. Learning Orient Surfaces by Self-supervised Spherical Advancdvs in Neural Information Processing
Systems, 2020.
[46] Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Geometric Capsule Autoencoders for 3D Point Clouds. arXiv Preprini, 2019.
[47] Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi. ACNe: Attentive Context Normalization for Robust Permutation-Fquivariai I Learning. In Conference Computer Vision and Pattern Recognition, 2020.
[48] Supasorn Suwajanakorn. Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discovery of
I Latent 3D Keypoints via End-to-Fnd Geometric Reasoning. 2018.
[49] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.
Tensor Field Networks: Rotation-and and Translation-Equiie Equivariant Neural Networks for 3D Point Clouds. arXiv Preprint, 2018
11

---

[50] Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape ab- stractions by assembling volumetric primitives. In Conference Computer Vision and Pattern Recognition. 2017.
[51] Oliver van Kaick, Kai Xu, Hao Zhang. Yanzhen Wang, Shuyang Sun, Ariel Shamir, and Daniel Cohen-Or. Co-hierarchical analysis of shape structures. ACM SIGGRAPH, 2013.
[52] He Wang. Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized Object Coordinate Space for Category-level 6d Object Pose and Size Estimation. In Conference on Computer Vision and Pattern Recognition, 2019.
[53] Nanyang Wang, Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision, 2018.
[54] Peng-Shune Saug-Shaii Wang. Yang Liu, Yu-Xiao Guo, Chun- Yu Sun, and Xin Tong. O-CNN: Octree-Based Convolu- tional Neural Networks for 3d Shape Analysis. ACM Transactions on Graphics. 36(4):1-11, 2017.
[55] Peng-Shure Leng-Shuai Wang. Chun- Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cin: A patch-based deep represen-
tation of 3d shapes. ACM Transactions on Graphics (TOG), 37(6):1-11, 2018
[56] Yue Wang and Justin M Solomon. Deep Closest Point: Learning Representations for Point Cloud Registration. In International Conference on Computer Vision, 2019.
[57] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.
Dynamcc graph can for learning on point clouds. ACM Transactions on Gruphics, 2019.
[58] Yuxin Wu and Kaiming He. Group Normalization. In European Conference on Computer Vision, 2018.
[59] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, and Or Litany. PointContrast:
Unsupervised Pre-training for 3D Point Cloud Understanding. In European Conference on Computer Vision,
2020.
[60] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective Transformer Nets: Learning Single- View 3D Object Reconstruction without 3D Supervision. In Advances in Neural Information Processing Systems, 2016.
[61]
Yaoging Yang. Chen Feng. Yiru Shen, and Dong Tian. FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation. In Conference Computer Vision and Pattern Recognition, 2018.
[62] Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Conference on
Computer Vision and Pattern Recognition, 2020.
[63] Wentao Yuan, Ben Eckart, Kihwan Kim, Varun Jampani, Dieter Fox, and Jan Kautz, DeepGMR: Learning
Latent Gaussian Mixture Models for Registration. In European Conference on Computer Vision, 2020.
[64] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3D Point Capsule Networks. In Conference on Computer Vision and Pattern Recognition, 2019.
[65] Yongheng Zhao, Tolga Birdal, Jan Fric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion Equivariant Capsule Networks for 3D Point Clouds. In European Conference on Computer Vision. 2020.
[66] Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In European Conference on Computer Vision 2016.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Abstract and Section !
(b) Did you describe the limitations of your work? [Yes] See Section
M
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] There are direct ethics concerns for the paper.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] In addition to the public code release that we will do, we have included the code in the supplementary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Due to the large amount of compute requirement for repeated experiments we have not been able include them the submission. How- ever, the results do not vary much from one run to the other in our experience, and we will release code to reproduce the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1 Note that to preserve anonymity we have decided not to include exact detail.
4. If you are using existing assets (e.g., code, data, models) or curating/releesing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Section
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "eYaf3YE9v8t", "reviewer": "Reviewer_jD2y", "summary": "The paper proposed a self-supervised capsule based architecture for 3D point cloud. The main goal is to learn a  canonicalization operation that allows unbiased object-centric reasoning.  The idea is that an object is decomposed to K keypoints using canonical capsules, the whole system is trained in an unsupervised fashion based on pairs of randomly rotated/translated  of the same object. The capsule pose equivariance is enforced by requiring the equivariance of two keypoint sets based on the known relative transformation of the pair images and the capsule descriptor invariance is enforced by matching the pair keypoints. The proposed framework have been evaluated in 3D autoencoding and reconstruction, unsupervised classification and canonicalization on ShapeNet dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n- The paper is generally well written and easy to follow. \n- The unsupervised learning framework for 3D canonicalization is very novel and seems effective in classification, reconstruction and canonicalization tasks.  \n- The proposed framework outperforms the baseline in all 3 tasks. \n\nCons/ questions/ clarifications:\n- Would you please clarify the encoder architecture? Does it only have one capsule layer with K capsule types as its last layer? \n- What is the effect of K on the performance and have you considered adding capsule layers as intermediate layers. \n- While the limited translation range is in favor of baseline methods in the de-canonicalizing dataset lines196-197, can you provide the performance of the proposed model for larger translation range? \n-One of the concerns in capsule based architectures are scalability of these architectures, can you provide the classification and reconstruction performance on the larger subset of  ShapeNet?  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitation and societal impact adequately in the future work and broader impact subsections. ", "review_text": "", "extra_scores": {"ethical_concerns": "I don't see any ethical issue with the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "btflviWySTN", "reviewer": "Reviewer_8HDQ", "summary": "The paper proposes a capsule-based network architecture that allows self-supervised representation learning from 3D point clouds.\nWhile existing methods rely heavily on pre-canonicalized (aligned) 3D models for training, this work proposes to automatically discover the \"optimal\" canonical transformation for alignment while learning to decompose and reconstruct the input point clouds. \nSpecifically, it exploits the transformation invariance and equivariance properties of object parts (capsules) to learn a latent representation.\nThe experimental results on the (unaligned) ShapeNet dataset show that the method can effectively learn to decompose the 3D point cloud into consistent parts across objects and canonicalize unaligned data.\nIt outperforms the state-of-the-art methods in terms of reconstruction accuracy, canonicalization accuracy and stability, as well as unsupervised classification accuracy from latent representation.\nIt is very encouraging to see that the need of 3D pose/viewpoint annotations can be removed and such competitive performance is reached.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Motivation and problem setting] \\\nThe paper addresses the problem of 3D representation learning from point cloud data, where the 3D point clouds are not aligned in the canonical space.\nCompared to most existing works that assumes the availability of aligned 3D data, the proposed scenario is more practical and crucial since such pose/viewpoint annotations of real-world 3D objects are usually hard to obtain.\nBy removing the need of pre-canonicalized 3D data for training, I believe this work makes a significant step towards learning from real-world 3D data with diverse transformations.\n\n\n\\\n[Method] \\\nThe network design, training strategies, and losses are well-justified and technically sound.\nThe proposed method follows an interesting line of work with part-based representation. \nObject parts are shown to be a good mid-level representation which is faithful to structural details, consistent across instances, and beneficial for pose/shape ambiguity.\n\nFrom my understanding, equations (5) and (10) are individually solved in each branch in Figure 1 with distinct transformation ($T^a$ or $T^b$).\nThat is, $\\theta_k$ and $\\bar{\\theta_k}$ are both from the top branch (${\\theta_k}^a$ and ${\\bar{\\theta_k}}^a$) or both from the bottom branch  (${\\theta_k}^b$ and ${\\bar{\\theta_k}}^b$).\nIf so, Figure 1 is a little confusing since $\\theta_k$ and $\\bar{\\theta_k}$ for shape matching seem to come from the top and bottom branch, respectively.\n\n\n\\\n[Experiments] \\\nThe experimental settings with de-canonicalized (unaligned) ShapeNet data and mStd metric are reasonable. \nThe random transformation includes rotation and translation within the range [\u00000.2, 0.2].\nI am curious if the authors have experimented with 1) a wider range of translation, 2) random scaling, and 3) decoupling the three transformations (e.g. only apply rotation).\nSince addressing unaligned training data is a major contribution of this paper, it would be important investigate the effect of different transformations.\n\nBoth the quantitative and qualitative results show a significant and consistent improvement compared to the state-of-the-art methods on the synthetic ShapeNet dataset.\nIt is not necessary but would be better to compare with more existing methods and on multiple datasets.\n\nFor auto-encoding reconstruction, is it possible to also evaluate the performance of different part-based representations like multi-mesh and implicit functions?\nIt should be possible to sample 3D points from mesh or implicit functions and calculate Chamfer distance.\nI believe this would make the proposed representation more convincing, and the results on unaligned data would be very helpful for follow-up research.\n\nTable 7 in the supplemental document shows the ablation studies on the number of capsules. \nAre the results from multi-class training and evaluation?\nIt seems that there is an optimal number of capsules to best represent the 3D objects, and I suspect it varies between object classes.\nI am also curious if the authors have tried fixing the descriptor dimension while increasing the number of capsule, since it is strange that more capsules don't lead to higher reconstruction accuracy.\n\nI find it surprising that the network can figure out the optimal canonical pose and capsule configuration of multiple classes as the same time.\nThe detailed (per-class) quantitative results of multi-class training are missing, and the paper does not specified whether the qualitatively results are from single-class or multi-class training.\nIt is hard to observe the pros and cons of multi-class training.\n\nFinally, I wonder if the authors have observed any failure cases of the proposed method.\nI would like to see more discussion about the limitation of representation and more qualitative results of failure cases.\n\n\n\\\n[Writing] \\\nThe paper is well-written and easy to follow.\n\nIt is a little unclear to me which parts (losses/networks/representation) of the proposed method are from the baselines (AtlasNetV2 and 3D-PointCapsNet) and which are novel.\nMaybe the authors can clarify it in the introduction or related work.\n\nThe word \"object-centric\" is mentioned multiple times.\nDo \"object-centric reference frame\" in L116 and \"object-centric coordinate frame\" in L329 both mean the (optimal) canonical frame found by the network?\n\n\n\\\n[Reference] \\\nSome missing reference I feel relative to the topic:\n[1] Luo, Tiange, et al. \"Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories.\" ICLR. 2019.\n[2] Paschalidou, Despoina, Luc Van Gool, and Andreas Geiger. \"Learning unsupervised hierarchical part decomposition of 3d objects from a single rgb image.\" CVPR. 2020.\n[3] Kawana, Yuki, Yusuke Mukuta, and Tatsuya Harada. \"Neural star domain as primitive representation.\" NeurIPS. 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work in Section 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "QUTwvEBd1C", "reviewer": "Reviewer_WfrE", "summary": "This paper proposes to learn a Canonical Capsule decomposition of shape point clouds in a self-supervised manner. \nGiven an unaligned 3D point cloud dataset, the proposed method constructs pairwise data with random SE(3) transformations, and then extracts the per-capsule pose and a transformation-invariant descriptor.\nThe experiment results show the importance of the object-centric representation with capsule decomposition in many applications, including reconstruction, canonicalization, and classification.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed pipeline learns, in an unsupervised manner, the canonicalization of 3D point clouds from aligned/unaligned point cloud datasets. The method is described clearly and could be reproduced.\nThe comparison, in each application, against 3D-PointCapNet, AtlasNetV2, and AtlasNetV2 w/ STN, adequately shows the significant improvement brought by canonicalization.\n\nI have a few questions:\n1. When training on 13 categories, is there any kind of consistency between the corresponding capsules for point clouds coming from different categories?\n2. It's better to show the reconstruction results of AtlasNetV2 w/ STN in Figure 2.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hrs"}, "extra_text": {}}, {"review_id": "M8OkQMk7IG", "reviewer": "Reviewer_WpqU", "summary": "This paper proposes a way of learning a capsule-based representation of 3D point clouds. The representation decomposes an input point cloud into parts, each corresponding to a capsule. This decomposition is trained to be consistent across SE(3) transformations of the point cloud using loss terms that encourage feature invariance and pose equivariance across two random presentations of the input point cloud. Moreover, the model learns a canonical pose for each part that is tied to the identity of the object (not its presented pose).\n\nThe experiments show that\n1. The reconstruction quality is high, showing that the model has capacity to fit point clouds from ShapeNet.\n1. The canonical pose is indeed consistent (measured by low std deviation in relative angle of rotation).\n1. The model can be used for pairwise registration (but DeepGMR does even better using RRI features, but doesn't do any of the other things this model can do.)\n1. The learned features lend themselves well to unsupervised clustering/classification.\n1. All the proposed loss terms are useful and contribute to better reconstruction and canonicalization.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The model is able to find a part decomposition that is consistent across presented views, while retaining the ability to autoencode and classify.\n1. The stability of the canonical pose is impressive.\n1. Good performance on standard tasks and benchmarks.\n1. Adequate comparisons to previous work.\n\n\nWeakness\n1. Capsules are not interchangeable. For instance, if I understood correctly, given an airplane point cloud, the encoder always assigns the 1st capsule to say, the left wing, 2nd capsule to the nose, etc. In other words, the capsule indices are tied to the entity they represent, as opposed to having an unordered *set* of capsules which can be constituted into different objects. This limits the ability of this model to reuse capsules across different object types and deal with the presence of multiple objects in the same scene.\n1. The model has no way to \"turn off\" capsules, which means that when trained on multi-category data it is forced to use the same K capsules very differently depending on the object being encoded. It is surprising however that the model still works very well for pair-wise registration and classification. It would be useful to include a brief discussion on whether the presence probabilities can be modeled within the framework of this model (similar to the Stacked Capsule Autoencoders paper).\n1. The \"pose\" of the part is really just the centroid. It does not capture the orientation of the part, which is something one typically associates with pose. It would be useful to include a comment on why orientation was not used (perhaps it is not helpful ?).\n1. Removing canonicalization loss does not seem to have a big impact (Table 4). This is perhaps explained by the fact that the reconstruction loss already encourages canonicalization because it minimizes the CD between the canonicalized input point cloud and the reconstructed one ? It would help to clarify this.\n\nOverall, this paper makes a significant contribution by proposing a model that learns canonical part decompositions in canonical pose, while retaining the ability to remember object shape and identity.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "H8JQENdxxej", "reviewer": "Reviewer_2DmA", "summary": "The paper proposes a self-supervised capsule architecture for 3D point clouds. In order to circumvent the customary need to train on pre-aligned datasets, the authors introduce the Canonical Capsules to compute the K-part decomposition of a point cloud that allows unsupervised \"object-centric\" learning of 3D representations without requiring a semantically aligned dataset. The main idea of the canonical capsules is to enforce invariance/equivariance by relating the decomposition result of two random rigid transformations.  The authors compare the proposed method against state-of-the-art methods on 3D point cloud reconstruction, canonicalization, and unsupervised classification.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe idea of the framework is basically enforcing the invariance/equivariance of the decomposition result of two random rigid transformations and use a canonicalizer to learn object-centric 3D representations. The idea is quite interesting and novel to some extent.\n\n\n### Clarity\n\nThe paper is well written. The motivation and the idea of the proposed framework is well explained and could be followed even though I am not very familiar with this topic. \n\n\n### Quality and Significance\n\nI am not an expert on this topic. Overall, the method looks reasonable to me and the authors conduct comprehensive experiments.  One small question: could you please give more explanation why the proposed method becomes unstable when using RRI features? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors have addressed the limitation and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have successfully addressed the raised issues. All reviewers agree that this paper makes good contribution to the NeurIPS community. The area chairs agree and recommend this paper be accepted.
Decision: Accept (Poster)