Learning 3D Dense Correspondence via Canonical Point Autoencoder
Authors: Anchieh Cheng
Xueting Li
Min Sun
Ming-Hsuan Yang
Sifei Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D dense correspondence
point autoencoder

Abstract: We propose a canonical point autoencoder (CPAE) that predicts dense correspondences between 3D shapes of the same category. The autoencoder performs two key functions: (a) encoding an arbitrarily ordered point cloud to a canonical primitive, e.g., a sphere, and (b) decoding the primitive back to the original input instance shape. As being placed in the bottleneck, this primitive plays a key role to map all the unordered point clouds on the canonical surface, and to be reconstructed in an ordered fashion. Once trained, points from different shape instances that are mapped to the same locations on the primitive surface are determined to be a pair of correspondence. Our method does not require any form of annotation or self-supervised part segmentation network and can handle unaligned input point clouds within a certain rotation range. Experimental results on 3D semantic keypoint transfer and part segmentation transfer show that our model performs favorably against state-of-the-art correspondence learning methods.

Learning 3D Dense Correspondence via Canonical Point Autoencoder
An-Chieh Cheng', Xueting Li?, Min Sun 13 Ming-Hsuan Y Sifei Liu National -ua University University of California, Merced, 3 Joint Research Center for AI Technologg and All Vista Healthcare, *Google Research, Yonsei University, "NVIDIA
Abstract
We propose a canonical point autoencoder (CPAE) that predicts dense correspon- dences between 3D shapes of the same category. The autoencoder performs two key functions: (a) encoding an arbitrarily ordered point cloud a canonical primitive, e.g., sphere, and (b) decoding the primitive back to the original input instance shape. being placed in the bottleneck, this primitive plays key role to map all the unordered point clouds on the canonical surface and to be reconstructed in an ordered fashion. Once trained, points from different shape instances that are mapped to the same locations on the primitive surface are determined to be pair correspondence. Our method does not require any form of annotation or self- supervised part segmentation network and can handle unaligned input point clouds within a certain rotation range. Experimental results on 3D semantic keypoint transfer and part segmentation transfer show that model performs favorably against state-of-the -ort---r-t-e-tttt correspondence learning methods. The source code and trained models can be found https//anjiichhnng. withtb.ii/chaaaecc
Introduction
With prior knowledge and experience, humans can easily perceive corresponding object parts (e.g.. the wings from two different airplanes). understand their shape and appearance variance, in order to distinguish different objects coming from the same category. computer vision, modeling dense correspondence between 3D shapes in one category is fundamental for numerous applications, such as robot grasping [1, 2], object manipulation [3] and texture mapping [2, 4J. However, existing 3D cameras typically capture raw point clouds of shape surfaces that arbitrarily-ordere and unstructured, in which correspondences are not established. 3D mesh representation, although is usually parameterized with UV maps that can indicate correspondences, cannot be directly obtained from sensors and needs to be reconstructed from other types of representations, c.g., 2D images [5, 6] 3D point clouds [7]. In this work, we focus on learning point cloud correspondences, which remains an open challenge since it infeasible to label ground truth correspondence annotations.
Without ground truth annotations, existing methods mainly discover shape correspondences via seeking a form of canonical space that can associate various instance shapes. For example, in particular shape domains such as human bodies [8] and human faces [9], parameterized shape primitives have been designed to fit the observed raw data and to obtain the correspondences. Such designs, however, cannot be generalized other categories, e.g., man man-made objects [10]. Recently, several part co-segmentation networks relax the requirements of specific parameterized primitives, but instead decompose input shapes into ordered group of simplest part constitutions [11, 12), in supervueer manner. These methods, however, require careful selection of the autoencoder architectures (i.e., they need to be considerably shallow to let the branches only able represent simple shapes), and the number part bases. Moreover, such part- representation does not explicitly provide fine-grained (e.g., peint-level) correspondences.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this work, we introduce a novel canonical space where dense (i.e., point-level) correspondences for all the shapes of a category can be explicitly obtained from. Inspired by 3D mesh representation [5. 13, 14, 15] where shapes from one category are represented as deformations on top of a shape primitive, in our work, we set the canonical space as 3D UV sphere. Our goal is to learn a "point cloud-to-sphere mapping" such that corresponding parts from different instances overlap when mapped onto the canonical sphere. In other words, similar to the mesh representation, a unique UV coordinate can represent the same semantic point/local region of shapes (e.g., the tip of an aeroplane's 's wing), regardless of shape variations. Towards this goal, we introduce the canonical point autoencoder (CPAE): place the sphere primitive at the bottleneck; the encoder non non-lineate con-linearly maps each individual input shape to the sphere primitive, where the decoder deforms the primitive back to match the original shape. We show that with several self-supervised objectives, this autoencoder architecture effectively (1) enforces the input points warped to the surface of the sphere primitives, and (2) encourages those corresponding points from different instances mapped to the same location on the sphere both guarantee that the network learns correct dense correspondences. Essentially, we do not assume all object shapes in one category having the same topology, e.g., an armchair does not have correspondence on its armrests, with another instance without an armrest. To introduce such uncertainty for correspondence matching, we propose an adaptive Chamfer loss on the bottleneck to allow customized primitive for each instance. As such, we are able to determine if a point on one instance has a correspondence in another point cloud.
One advantage of the proposed method compared to the recent work [12] is that we can learn correspondences even when instances in the training dataset are not aligned, i... our model is rotation- invariant within a certain rotation range and does not need to predict an additional rotation matrix as used in [12]. The main contributions of this work are:
We introduce a novel canonical space - a UV sphere, that explicitly represents dense correspondences of shapes from one category.
With the canonical space on the bottleneck, we design an autoencoder that learns such a "point cloud-to- -o-shee mapping" via group of self-supervised objectives. We apply the proposed method on various categories and quantitatively evaluate on the task of 3D semantic keypoint transfer and part segmentation label transfer, achieving comparable if not better performance than taae-of-the-as methods.
2 Related Work
Deep Learning on Point Clouds. As a flexible and memory efficient representation of 3D shapes, point cloud has been widely studied and combined with deep neural networks. The PointNet [16] solves point cloud classification and segmentation by using MLPs and a max pooling layer to aggregate 3D shape information. One crucial property of the PointNet is that it is able handle unordered input and is thus invariant to point permutations. In our work, we utilize the first few layers of the PointNet as our shape encoder. Another line of works [17, 18, 19, 20] focus on reconstructing or generating point clouds. The most representative and related to our work is the Folding [17], where point cloud is first encoded by a graph-based encoder and then reconstructed by sequentially applying the "folding operation" (instantiated by MLPs) to a 2D UV map. In this work, we reveal why the MLPs are able to preserve point order and utilize it as building blocks in our CPAE.
Primitive Fitting. Primitive fitting has a long history in computer vision [21]. Several approaches focus on fitting individual samples with shape priors, such blocks world [21], generalized cylin- ders [22], geons [23], Lego pieces [24], qualitative 3D blocks [25], and patches [26]. However, these primitives are analyzed per-instance without order, therefore, cannot be used for the correspondence. Some other approaches focus on fitting over a collection of shapes, which provide ordered set of primitives [11, 27, 28, 29, 30, 31, 32]. However, most of these approaches can only infer part-level correspondences and require additional supervision such as the number of primitives. In this work, we directly learns set-to-set alignment between point clouds, thus requires no shape prior.
3D Dense Correspondence. Given a pair of source and target instance in the same category, 3D dense correspondence learning targets at finding corresponding point in the target instance for each point from the source instance. Several approaches [33, 34, 35] resolve this task through point cloud registration, using labeled pairwise correspondence as supervision. To relax constraints on supervision, Bhatnagar et al. [36] predict part correspondences to a template via implicit functions. Unfortunately, they require part labels for training. To unsupervisedly learn 3D dense correspondence,

---

U₁ ACD
roplicate
PC
offlll 42)
PC))
ZA
0r v2))).
of at))
2
P Sa
UA
2,
Figure 1: Overview of the proposed method. Here, p = (x yi 2, 2 is a point on the input point cloud SA, u (x. yf ={() is a point on the primitive U.. The "Ø> sign indicates concatenation. (.) and (-) are MLPs as discussed in Section 3.1.
Chen al. [37] propose a method learn 3D structure points that are consistent across different instances. However, the model assumes structure similarity among different instances, which ignores intra-class variants and fails to detect non-existing correspondences between dissimilar shapes in the same category. Most relevant to our work, Liu et al. [12] introduce an unsupervised approach that leverages part features learned by the BAE-NET [11] to build dense correspondences. Their algorithm is able to calculate a confidence score representing the probability of correspondence. However, in order to train the implicit function, they require additional knowledge of object surface to compute the occupancy. Moreover, the correspondence learning in [12] heavily relies on the completeness of the part features from the BAF-NET [11], which can lead to incorrect correspondences for parts that BAE-NET cannot separate (e.g.. flat surfaces, objects with fine- sine-graned details), In contrast, approach directly learns dense correspondences from the point cloud with sulf-supervision.
Cycle Consistency. Our cross reconstruction loss is related to literature that uses cycle consistency as a supervisory signal to learn correspondence without ground truth annotations. Some of these works exploit the consistency among different modalities (e.g., 2D-3D). Specifically, [38] aim to learn cross-ines coss-instance pairwise correspondence for 2D images using cycle-consistency terms guided by 3D synthetic data. [39] and [40] both leverage cycle-consistency terms to enforce correspondence between a 2D image and 3D canonical template. Aside from these works that focus on cross- modality, work from [41] is mostly related to ours. They propose a "point-wise" cycle consistency loss that explicitly enforces a point deformed through any cycle deformations be mapped back to the origin location. Different from their approach, our consistency is instance- sisae and we do not assume bijection mapping among shapes.
3D Deformable Mesh Representations. Our algorithm is also related to methods [5, 13, 14, 15] that represent instance shapes as mesh deformations of mesh primitive, i.c. sphere template. Vertices on each instance surface that are mapped to the same locations on the shape primitive are discovered as correspondences. One obvious limitation is that only genus 0 shapes, e.g., birds [5, 13] can be deformed from a mesh sphere. In contrast, ours does not have such restriction thanks to the proposed non-linear mappings, instead of explicit deformation.
3 Proposed Method
In this section, we introduce our end-to-end CPAE for learning dense correspondences from point clouds without ground truth annotation. Given point cloud SA with individual point p R (see Figure 1), our model: (1) predicts its canonical primitive U (a "deformed" point cloud with the same number of points SA). which is supposed to be close as possible to the canonical sphere in the bottleneck; (2) reconstructs the original input point cloud back from UA. We show that while the first ensures each input point cloud to be warped to the surface of the sphere primitive, the second indirectly encourages the corresponding points from different point cloud instances to overlap during mapping the primitive. In the following, first describe network architecture, i.e., the encoder and decoder modules. We then introduce the adaptive Chamfer loss and the reconstruction losses that are applied each individual module. Finally, show that our decoder reconstructs ordered point clouds, i.e., different point cloud instances can fetch their correspondences directly via the point indices, which provides more accurate inference.
3

---

3.1 Network Architecture
Our method learns two mapping functions: one maps each individual 3D point p E RR in the world coordinates to the canonical space ((p) = u, E R3, while the other conducts the inverse mapping 1 = p. Instead of using a reversible network, we instantiate (.)) and $ (-) with two MLPs respectively, and cascade them to construct an autoencoder (i.e., (-)) is the encoder and ((() R & (-) is the decoder). However, this mapping function cannot be generalized to different point clouds if it only receives the coordinate of a single point p as the input. Therefore, we formulate them as conditional mapping functions by concatenating the input with a shape latent code Z that represents a unique input shape SA. e.g., (p, Za), in order to generalize the mapping functions to all instances in the same category.
There are three key modules in the proposed method: (1) PointNet encoder that produces the shape latent code; (2) an encoder MLP (denoted as canonical mapping) that maps a point cloud to a primitive sphere, and (3) decoder MLP (denoted as inverse mapping) that deforms the primitive sphere back to the input point cloud.
PointNet Encoder. As shown in Figure 1, given point cloud SA E Rexs with k points, we encode using a PointNet [16] encoder in a way similar to [8. 18, 19]. Each 3D point of the input point cloud is represented as a 512 dimensional vector using an MLP with 3 hidden layers of 64, 128, 512 neurons and ReLU activations. We then aggregate all point features with max-pooling and a linear layer to generate a global latent code A R512 We use PointNet because it produces robust latent code representing the global shape, which is also invariant to input permutation.
Canonical Mapping (.)). To learn the canonical mapping function, we concatenate each point with the global latent code [p, Z] E R255 as input to the MLP, which then outputs a 3D coordinate RD3 in the canonical space. We construct sphere point cloud by uniformly sampling large number of points from a standard sphere mesh and place them as the canonical primitive at the bottleneck. A Chamfer loss is used to measure the difference between the outputted point and primitive to encourage the mapped point adhering to the surface (see Eq. (1)).
Ideally, all the mapped instances from the same category should be close as possible to the canonical sphere primitive. However, an instance may include parts/regions that do not exist in other instances due to intra-class variation, e.g., not all chairs include armrests. With conventional Chamfer loss, the shapes for objects with rare components do not converge during training, i.e., those rare components are mapped to locations that are far away from the primitive surface. Thus, we relax the bidirectional constraint of the Chamfer loss and allow each instance to produce its own "instance primitive" to some extent (see UA in Figure 1) ). This is formulated via adaptive Chamfer loss LACD:
LAAD(UA, = UUA
Σ PEUA
min lpp - alla + a GEU
Σ min llq - -Pl2 9EU
(1)
T
PEUA
where a [0, 1] is an adaptive parameter indicating to what extent the predicted instance primitive should match a canonical sphere, UA and U are the instance primitive and the canonical UV primitive (c.g.. a 3D UV sphere), respectively. When a = 1, LAD is equivalent to calculating the conventional Chamfer distance between the unfolded primitive and the canonical UV primitive. During training, is initialized to I and gradually decreased to O. This allows the canonical mapping to predict instance- awkaece-awwae primitive since the second term of the Chamfer loss no longer constraints the primitive to be consistent with canonical sphere. In the experiments (see Section 4.5), we show that this design allows us to infer rare object components that have no correspondences in other instances since non- con-corresponiing regions usually occupy a distinct area in the canonical space.
Inverse Mapping (-). Similarly, we utilize another MLP in the decoder, which receives the concatenation of one point from the bottleneck and the global shape code, i.e., [u, Z] E RR515 as the input. The function learns to map back to its original world coordinate p, which can be fulfilled via a point-to-poing -oinn reconstruction loss. We leverage an MSE loss LMSE, a Chamfer distance LED and an Earth Mover Distance (EMD) LEMD between the reconstruction Š and input point cloud P:
Lree $) = μn LMsE (P, p2LCD (P, 5) p (P.S)
where in, /22 (A3 are the weights, and empirically determined, μa 1e:,, μ 10, and pa 1.
The inverse mapping bears some resemblance to the principle of FoldingNet [17], which adopts an MLP architecture to map one 2D coordinate together with a global shape representation the 3D

---

world'. However, since FoldingNet does not have the forward mapping function as ours and the imput points are untraceable, it needs to use Chamfer loss, instead of point- wise loss (Eq. (2)) as ours.
Align Canonical Mappings via Cross- Reconstruction. With the encoder and objective function, the canonical mapping is able to map the input points to the surface of the canonical primitive. However, even with the reconstruction loss (Eq. (2)), there is no guarantee that the corresponding points from different shape instances can be overlapped on the sphere which is the key learn dense correspondences. For instance, in Figure 1, while SA is mapped to the frontal half of the sphere, it is likely that another shape SB will be mapped to the rear half.
B.)
VC
Sx
Za
Ux
S
E.0(.)
48
VC--
Z
S
Figure 2: Cross-rccosttuuction. The F., (()), (.) are identical as in Figure 1.
To better align the mapped points, we introduce a cross cross-structured decoder that leverages: (1) a sefftrecontttuttioon branch presented in Eq. (2), and (2) roosssrccosttttttoon branch. As shown in Figure 2, we feed the combination of the predicted instance primitive UA of point cloud SA, with the shape latent vector z of another randomly sampled point cloud SB the folding decoder. We then minimize the Chamfer distance between the output shape S and Sp. The cross-reconstruucion loss between the point cloud SA and SB is:
Leross (SA, SB) = LCD(SA-B,
The coos-reconstrucctonn branch ensures even by swapping the predicted primitives UA and UB, the inverse mapping can still reconstruct their own shapes, conditioned on their own shape latent codes. That is, the decoder encourages UA and Un to overlap on the canonical primitive, and thus ensures the network learns correct correspondences.
Relation to Implicit Function. We note that since both mapping functions 4--) and (-) process each input point independently, they can be interpreted as conditional implicit functions, which are widely applied for 3D shape reconstruction [42, 43), view synthesis [44, 45], and recently for self-supervised 3D correspondence [12]. Such an interpretation reveals that the MLPs indeed learn continuous mappings that are feasible for interpolations, e.g., once the decoder MLP is learned, is able to map any point on continues sphere surface that is not necessarily among the mapped input u, to the world coordinate, E.g. one can sample points densely from the surface of the sphere to reconstruct continuous surface or a 3D mesh
3.2 Finding Correspondences from Ordered Point Clouds
In this section, we show that the CPAE is able to generate ordered point clouds (see the output point clouds in Figure 1). Compared to the methods obtaining correspondences by directly tracing overlapped points from the canonical space, our model that makes use of ordered output point cloud generates more accurate results during the inference stage.
Reconstruction of Ordered Point Clouds. We show that an MLP [17] decoder preserves the order of points in the canonical primitives (shown as colored points within the sphere), when mapping them back to individual shapes. Given two adjacent points UA and UB from the sphere, we denote PA V (U Z) and PB (UB, Z) their mapped points in the output space, where Z is the latent shape code. Since (-) contains only linear projections and ReLUs (i.e., piece-wise linear), it continuous function. As such, PA and pa will also be close each other, e.g., in Figure I, the output shape exhibits a smooth transition of color map similar to the colored canonical shape on the sphere. Thus, the order is maintained as opposed to being shuffled in the output space.
easy to be proved
Inference via CPAE. Given a source shape SA and query point PA E SA. we target at searching the correspondence of the query point in a target shape SB. We first compute the shape latent vectors for the two shapes as εл and æp. The query point is then mapped to a point UA in the canonical space by the canonical mapping encoder. By feeding the concatenation of u and zB to the inverse
Unlike our method, it utilizes and samples points from standard 2D UV map instead of 3D sphere.
5

---

Airglane
Bathtub
Cap
150
101
160
100
2.70
2270
899
150
800
323
500
0.
0.15
0:85
0.15
0.35
Eicisben Dubend
Dtatiin Distace
Chaii
G uuttar
Hlemee
Laptop
180
180
93
80
93
10
270
2.70
100
100
0.10
0.
6.13
0.2
0.00
0.1
0.35
0.10
0.
0.15
0.2
0.20
0.1
0.35
022
Eudidion
Dudidkon
Excidica
Motorcycle
Mug
Table
Vessel
120
100
120
100
8720
711
(1)
3.53
310
Eudidon EufidonniDtttt Distnco
Extl Disaace
Budidden Distaoe
Exciiixa Dissace
Ours
Liu al.
AlasNeeV2
FoldingNot
Figure 3: Correspondence accuracy for 12 categories in the KeypointNet dataset.
mapping decoder, we further map u to a point PA-BB on the reconstructed target shape S The correspondence of PA (denoted as pæ SB) is thus the closet point on SB to PA-+B-
Confidence of Correspondence. For each point and its correspondence pair (e.g. PB)), we can also compute confidence score Cpp PB) to measure the confidence of this mapping:
C(pA PB) D(PA. PB)
where D refers the normalized Fuclidean distance between PA and pn in the 3D world coordinate. Similar to [12), if C is lower than pre-defined threshold т, we conclude that point PA does not have correspondence on shape Sg.
Experiments
In this section, we present evaluations of the proposed dense correspondence learning approach. To the best of our knowledge, there is no benchmark that provides ground- ground-truth dense correspondences for general objects, which is exactly our motivation learn dense correspondence with self-supervision. However, thanks to the learned dense correspondences across pairs of instances, we are able to carry the task of 3D semantic keypoint transfer and part segmentation label transfer evaluate the proposed method as in [12]. In the following, we first introduce our experimental setup as well as baselines. We then report quantitative and qualitative comparisons with these baselines for semantic keypoint transfer and part segmentation label transfer. We further present ablation studies to demonstrate the contribution of each component in the proposed model. Finally, we show that our model generalized to real scanned point clouds unseen in the training stage. More results can be found in the appendix.
4.1 Experimental Setup
Dataset. We carry out the semantic keypoint transfer task on the KeypointNet dataset [46] as the BHCP benchmark used in Liu et al. [12] is not publicly available. Compared to the BHCP benchmark, the KeypointNet dataset is more challenging because: (a) contains diverse objects and comes with large-scale annotations, (b) it is template-free and annotated by large group of people, thus is less biased compared to the keypoints in the BHCP benchmark, which are from predefined templates. For the part segmentation label transfer task, we use the ShapeNet part dataset [47] as in [12]. For both datasets, we use the split provided in the original paper, and generate all pairs of shapes the testing set our testing pairs. To avoid interference from non-existing correspondences, we leave

---

Shupe
Shape
Shupe
Shape
Shape A
Shape
NoldingNet
AtlosNetV2
Liu et al.
Ours
Not
AtlasNetV2
Liu et
Ours
Figure 4: Keypoint transfer results for five categories: airplane, bathtub, chair, motorcycle, mug, and vessel. Each row contains two shape each with ground-trune keypoints and its pairwise transfer result.
pi
bug
cha
ear
gun
km
Jam.
lpp
bi..
mug
me
skn.
Wh
Liu et 00.1
56.3
59.7
12.2
45.3
61.5
06.4
43.6
88.5
10.5
873
66.4
37.2
30.7
70.4
61.7
55.
53.2
65.8
Table Part label transfer results for 15 categories in the ShapeNet part dataset. Number measured with average IOU(%).
instance pairs that do not share the same keypoint or part label. In all experiments, including our method and the baselines, we use validation set for model selection.
Baselines. We evaluate the proposed method against state-of of-the-art learning-bast -baiid 3D dense correspondence prediction approaches, including AtlasNetV2 [18], FoldingNet [17], and Liu et al. [12]. Specifically, FoldingNet deforms from fixed UV grid, while AtlasNetV2 explicitly allows the shape to be deformed from learnable elementary 3D structures. Neither of them estimates the confidence of correspondences. Liu et al. [12] propose method that utilizes part features learned by the BAE-NET [11] learn dense correspondence, including mechanism that estimates the correspondence confidence.
Implementation Details. For both the point canonical mapping encoder and point inverse mapping decoder (see Figure 1), we follow [18] and use three-layer MIP with ReLU activations, BatchNorm layers except for the last layer, where we use a hyperbolic tangent activation to obtain the final output. The training phase of our approach consists of two stages: (1) pre-training stage trained with LACD (Eq. 1) and Lyec (Eq. 2) using
1 for L LAD (2) A fine-tuning stage trained with LACD. Lree. and Leross (Eq. 3) where we set a = for LACD- The total loss is formulated as Liotal W| LACD + Wree + WyLooss where W| 10 for the pre-training stage; 20 for fine-tuning stage, We 1, wy 10. For all experiments, we set 2048, 7 = 0.9 (see Section 3.2), and the parameters of the network are optimized using the Adam [48] optimizer, with constant learning rate of le
4.2 3D Semantic Keypoints Transfer
For fair comparisons, we follow [12, 37] and compute the distances from transferred keypoints to ground truth keypoints and report the percentage of testing pairs where the distances below a given threshold in Figure 3. We demonstrate that for 11 out of 12 categories (e.g. airplane, chair, mug, etc.), keypoints transferred via our learned correspondence
more accurate than other methods [12, 17, 18]. At the distance threshold 0.05, our method performs 1 1..2% more accurately than Liu al. [12] on average of all categories. Figure 3 demonstrates the qualitative results of the keypoint transfer task. Even for categories with large intra-class variation, e.g. bathtub, motorcycle, or vessel, method is able to transfer keypoints accurately thanks to the learned dense correspondence.
7

---

Source
Target
Liu et al.
Ours
1
Liu et al.
Ours
Figure 5: Qualitative results of part label transfer. The 1st row of the target indicates the ground truth instance part labels, while the rest shows the label transferring results via learned correspondences.
4.3 Part Segmentation Label Transfer
We further validate our approach on the part label transfer task and present quantitative results in Table and qualitative results in Figure 5. Note that the settings are different from that of Liu al. [12]. which directly utilizes the branched co-segmentation results for evaluation (thus different quantitative results between the Table and that in their paper). In Table over-union (loU) between transferred and ground truth part labels. Our method performs better than Liu et [12] in 12 of 15 categories and has higher average IoU. For categories with large intra-class variations, such as lamps, motorbikes, our approach significantly outperforms Liu et al. [12]. The performance difference most likely due the branched co-segmentation [11] adopted by Liu et al. [12]. Branched co-segmentation relies heavily on the size of training data available. Such an approach is vulnerable large shape variations [11], and is unable segment flat surfaces. Moreover, its performance is sensitive to the pre-defined number of branches. Instead, our method naturally links different instances through canonical primitive and thus more robust to large shape variations. We show the qualitative results in Figure 5. Thanks to the dense correspondence learned by the proposed method, we are able to transfer small parts more accurately such as seat of motorcycle (the green points) and the pipe of lamp (the purple points) comparing to Liu et al. [12].
show the intersection-
4.4 Correspondence Confidence
Given a source shape SA and target shape Sg, for every point Sg. our model computes confidence score indicating whether corresponding point exists SA. as discussed in Section 3.2. As annotations can be biased by the annotator's definitions on keypoints, there is no absolute ground- truth for non-existence label between shape pair. Thus, we visualize the confidence score as heatmaps for multiple target shapes in the airplane category in Figure 6 and compare with Liu et al. [12]. As shown Figure 6, the proposed method able to produce more fine-grained confidence score compared to Liu et al. [12]. This is because approach explicitly evaluates the confidence of

---

Liu et al.
Ours
Liu et al.
Ours
Level of Confidence
20%
40%
60%
80%
Figure 6: Heatmaps representing the correspondence confidence generated by network. Source shape is the leftmost of each row and dark color in each heatmap refers to low confidence in existing correspondence.
correspondences at more fine- fine-grained level the distance between points, instead of a distance at the part-level, proposed in [12].
q-!
a-1
web L
2D UV
Fail Model
100
Full Model
90
80
w/a Korms
Comespoalanss (%)
70
Full Model
a =1
60
-w/l
50
40
W/ -0-0
30
20
10
0.05
0.1
0.15
0.2
Euclidean Distance
(a)
(b)
Figure 7: Qualitative (a) and quantitative (b) results of ablation studies on (i) the crosstreconttructioo loss term (ii) the adaptive Chamfer loss term (iii) different types of canonical UV primitive using the chair category in the KeypointNet dataset. Note that the t sign indicates methods trained with an un-aligned setting.
4.5 Ablation Studies
Effectiveness of cross-reconstruction. The cross cross-eeconstruction architecture is designed to ensure that corresponding points overlapping much as possible the canonical space. In exper- iments, we find that with the datasets containing shapes with aligned 6D poses [10, 46), a single branch encoder-decoder framework (i.e., w/o Teross)) already produces reasonable prediction of correspondences. However, as shown in Figure 7 (red vs blue), our cross-reconstruction framework significantly improves the performance with large margin.
In addition, we validate the effectiveness of the framework on un-aligned shapes, by rotating the input point cloud with radian noise (O, 0.5?). We note that all co-segmentation approaches assume that the pose of input shapes are consistently aligned [1 1, 12]. Consistent with the assumption, such rotation severely degrades the performance of Liu al. [12] (brown line). In contrast, with the help of cross-reconstruuttion loss (orange line), our model is rotation-invariant to a certain degree.
Effectiveness of adaptive Chamfer loss. To show that the adaptive Chamfer loss is necessary, we train our model with constant = and a = 1, respectively. The results are shown in Figure When is set to zero consistently, the primitive would eventually condense to single point in the canonical space, therefore, significantly hurts the performance (pink line). the other hand, 1, it is equivalent to enforcing single canonical primitive for all instances by encouraging the

---

Source
Targets
(Keppaletlie)
(conNet)
AA
Figure 8: Keypoint transfer results from synthesized data to real scanned objects. We show results on the chair, bathtub, and table category ScanNet categories that intersect with the KeypointNet dataset. Source shape is in the leftmost of each row.
primitive cover the entire canonical space. This ignores the intra-class variation and deteriorates the performance shown in Figure 7 (green line).
Using a 2D UV grid vs. 3D UV sphere as primitive. To analyze the effect of different canonical UV primitives, further train model using 2D UV grid instead of 3D UV sphere. Figure 7 shows that parameterizing the canonical space with a 3D sphere quantitatively outperforms a 2D) grid (grey line). The main reason is that the sphere is continuous at any point while 2D grid is discontinuous at boundaries.
4.6 Testing with Real Scanned Data
To demonstrate that our model robust to the domain gap between real and synthesized 3D objects, apply model to the real scanned objects. Specifically, train our network the KeypointNet dataset and test the trained model on real scanned point clouds from the ScanNet dataset [49]. As the ground truth keypoints the qualitative results in Figure 8. Although our model is only trained using synthesized data, successfully predicts the correspondence and transfers keypoints from the source to the target shape.
semantic part annotations in ScanNet are unavailable, we show
Conclusions
In this work, propose self-supervised model, CPAE, that learns dense correspondence between 3D shapes in the same category. We introduce a canonical UV sphere, where dense correspondence for all the shapes can be explicitly obtained from. The key learn 3D world coordinate to canonical space mapping, so that points from different instances are regarded as a pair of correspondences if they overlap on the sphere. We fulfill it through an autoencoder equipped with an adaptive Chamfer loss in the bottleneck, and cross-reconse reoosstocontt structure in the decoder. Experimental results validate the proposed method performs favorably against state-of-the-art schemes in various tasks and ablations. We show that our model is much more robust to rotation than the existing approaches that are non-rotation invariant by design. The task learning -nation-invariant representations for 3d dense correspondence is challenging and still far from being solved. In future, we plan to extend our work to handling larger rotations.
Acknowledgements
This work was supported in part by the MOST, Taiwan under Grants 110-26344F-007-0166 MOST Joint Research Center for AI Technology, All Vista Healthcare, and NSF CAREER grant 1 1149783. We thank National Center for High-performance Computing (NCHC) for providing computational and storage resources.
10

---

References
[1] Ashutosh Saxena, Justin Driemeyer, Justin Kearns, and Andrew Y Ng. Robotic grasping of novel objects. In NIPS, pages 1209-1216, 2007.
[2] Andrew T Miller, Steffen Knoop, Henrik Christensen, and Peter K Allen. Automatic grasp planning using shape primitives. In ICRA, pages 1824-1829, 2003. 1
[3] Qiang Hao, Rui Cai, Zhiwei Li, Lei Zhang, Yanwei Pang, Feng Wu, and Yong Rui. Efficient 2d-to-3d correspondence filtering for scalable 3d object recognition. In CVPR, pages 899-906, 2013.
[4] Wang Zeng, Wanli Ouyang, Ping Luo, Wentao Liu, and Xiaogang Wang. 3d human mesh regression with dense correspondence. In CVPR, pages 7054-7063, 2020.
[5] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In ECCV, 2020. 2,
[6] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In ECCV, 2018.
[7] Abhishek Badki, Orazio Gallo, Jan Kautz, and Pradeep Sen. Meshlet priors for 3d mesh reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2849-2858, 2020. 1
[8] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d-coded: 3d correspondences by deep deformation. In ECCV, pages 230-246, 2018. 1,4
[9] Syed Zulqarnain Gilani, Ajmal Mian, Faisal Shafait, and Ian Reid. Dense 3d face correspon- dence. IEEE TPAMI, 40(7):1584-1598 2017.
[10] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information -ich 3d model repository. arXiv preprint arXiv: 2015. 1,9
[11] Zhiqin Chen, Kangxue Yin. Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bac-net: Branched autoencoder for shape co-segmentation. In CVPR, pages 8490-8499, 2019. 1, 2, 3, 7,
[12] Feng Liu and Xiaoming Liu, Learning implicit functions for topology-varying dense 3d shape correspondence. In NeurlPS, 2020. 2, 3, 6, 8,
[13] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In ECCV, 2018. 2,
[14] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoints without keypoints. In ECCV, 2020. 2,3
[15] Shubham Tulsiani, Nilesh Kulkami, and Abhinav Gupta. Implicit mesh reconstruction from unannotated image collections. arXiv preprint arXiv:2007.08504 2020.
[16] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660) 2017. 2, 4
[17] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In CVPR, pages
2018. 2, 4,5,7
[18] Theo Deprelle, Thibault Groueix, Matheww Fisher, Vladimir Kim, Bryan Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. In NeurIPS, pages 7433-7443, 2019. 2, 4,
[19] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. papier-måché approach to learning 3d surface generation. In CVPR, pages 216-224, 2018. 2,

---

[20] Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. ICCV, 2019. 2
[21] Lawrence Roberts. Machine Perception of Sinee-Dimnsional Solids. PhD thesis, Mas- sachusetts Institute of Technology, 1963. 2
[22] I Binford. Visual perception by computer. In IEEE Conference of Systems and Control, 1971.
[23] Irving Biederman. pooggittionby--omoore Psychological Review, 94(2):115, 1987.
a theory of human image understanding.
[24] Anton van den Hengel, Chris Russell, Anthony Dick, John Bastian, Daniel Pooley, Lachlan Fleming, and Lourdes Agapito. Part-based modelling of compound scenes from images. In CVPR, pages 878-886, 2015.
[25] Abhinav Gupta, Alexei A Efros, and Martial Hebert. Blocks world revisited: Image under- standing using qualitative geometry and mechanics. In ECCV, pages 482-496. Springer, 2010.
2
[26] Gopal Sharma, Difan Liu, Subhransu Maji, Evangelos Kalogerakis, Siddhartha Chaudhuri, and Radomir Mëch. Parsenet: A parametric surface fitting network for 3d point clouds. ECCV, pages 261-276. Springer, 2020. 2
[27] Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In CVPR, pages 2635-2644 2017. 2
[28] Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 3d-prnn: Generating shape primitives with recurrent neural networks. In ICCV, pages 900-909, 2017.
[29] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnet: Learnable convex decomposition. In CVPR, pages 31-44, 2020. 2
[30] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadries revisited: Learning 3d shape parsing beyond cuboids. In CVPR, pages 10344-10353, 2019. 2
[31] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, pages 7154-7164, 2019. 2
[32] Dmitriy Smirnov, Matthew Fisher, Vladimir G Kim, Richard Zhang, and Justin Solomon. Deep parametric shape predictions using distance fields. In CVPR, pages 561-570, 2020. 2
[33] Mingjia Chen, Qianfang Zou, Changbo Wang, and Ligang Liu. Edgenet: Deep metric learning for 3d shapes. Computer Aided Geometric Design, 72:19-33, 2019.
[34] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In ICCV, pages 8958-8966, 2019. 2
[35] Zan Gojeic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In CVPR, pages 5545-5554, 2019. 2
[36] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Combining implicit function learning and parametric models for 3d human reconstruction. In ECCV, 2020.
[37] Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In CVPR, pages 9121-9130, 2020. 3,7
[38] Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In CVPR, pages 117-126, 2016.
[39] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In CVPR, pages 2202-2211, 2019. 3
12

---

[40] Tristan Aumentado-Armstrone Alex Levinshtein, Stavros Tsogkas, Konstantinos G Derpanis, and Allan D Jepson. Cycle-consistent generative rendering for 2d-3d modality translation. In 3DV, pages 230-240, 2020. 3
[41] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. Unsupervised cycle-consistent deformation for shape matching. In Computer Graphics Forum, volume 38, pages 123-133, 2019. 3
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[43] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, pages 4460-4470, 2019.
[44] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In NeurIPS, 2019. 5
[45] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
[46] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. In CVPR, pages 13647-13656, 2020. 6, 9
[47] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation 3d shape collections. ACM TOG, 35(6):1-12, 2016.
[48] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013. 7
[49] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: annottead 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5839, 2017. 10
13


Reviews:
[{"review_id": "t6dW4prsvlq", "reviewer": "Reviewer_kS8b", "summary": "This paper presents a novel solution for learning dense correspondence between un-aligned 3D shapes in an self-supervised manner. The key contributions this paper is to design a novel canonical space for representing all 3D shapes and learning an auto-encoder to learn a mapping to and from an input point cloud to this canonical space. The effectiveness of this approach is shown for the 3D semantic keypoint transfer and part segmentation transfer tasks across different categories.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The paper proposes a novel canonical space for representing 3D shape, which helps to learn correspondence of shapes. \n2. The paper proposes a way to learn an auto-encoder, which can learn how to map to and from the input point cloud and the canonical space. Besides the loss functions, used in [9], they use a modified version of Chamfer loss at the canonical representation space as well as a cross reconstruction loss where the encoded features from a particular shape instance is transfered to another shape instance.\n3. The paper shows the effectivess of this approach on the 3D semantic keypoint transfer and part segmentation transfer tasks where they show results across different categories. \n\n## Weaknesses\n1. **Use of sphere as the canonical representation**: There are several aspects that are unclear about using sphere as the canonical representation   \n  a. *Motivation*: The paper does not talk about the motivation behind using sphere as a primitive for the canonical representation. Shouldn't projecting a 3D shape onto the surface of the sphere lose the convexity and concavity of the different 3D shapes? The authors should also cite papers from the rich literatures (including recent literature on 3D shape reconstruction) that has explored different kinds of primitives.    \n  b. *Empirical Results* From the emprical perspective, the fact that the yellow (w/o $ L_{cross}^{\\dagger} $) and brown (Liu et al) lines are close by in Fig 7(b),  show that this just using this representation is not significantly useful. Moreover, Fig 9 shows that $ \\alpha = 0 $, does not work at all where the representation in the primitive space collapses to a single point. Isn't this an indication that this representation is not powerful enough to capture a good canonical representation?    \n c. *Loss at the encoded representation layer*: Why is Chamfer loss used at the encoder space? This is not supposed to be exact input 3D shape. What is the motivation of using such a loss function instead of loss functions which only looks at local surfaces or neighborhoods? Some recent papers that come to mind are as follows:   \n    - Smirnov, Dmitriy, et al. \"Deep parametric shape predictions using distance fields.\" CVPR. 2020.  \n    - Sharma, Gopal, et al. \"Parsenet: A parametric surface fitting network for 3d point clouds.\" ECCV 2020.  \n d. It is not clear why it is expected that the rare components would appear far away from the primitive. Is this infered emprically and/or is there an high level intuition why this would happen?  \n\n2. **Training details**: It would be good to mention more details about the training approach:  \n  a. *Training data*: Which dataset do you train on? How many categories are there? How many shape instances are there per category?  \n  b. *Cross-reconstruction loss*: It is not clear whether the other point cloud is sampled randomly from the same category or from a different category. During training, in a min-batch, how is the batch setup from this perspective?  \n  c. Do you perform data augmentation on the input point cloud, especially for the cross representation loss part? If no, why would this not help?  \n  d. Why do you rotate by only sampling from $ \\mathcal{N} (0, 0.5^2) $? Is that parameter chosen randomly or there is some empirical justification?  \n\n\n3. **Relation to cycle consistency**: The cross-reconstruction loss has similarities to cycle consistency. This is definitely a novel formulation for this task. But it would be interesting to cite related works on this topic and discuss what are the differences of this loss compared to others. Somne of the papers related on this part are:  \n  i. Kulkarni, Nilesh, Abhinav Gupta, and Shubham Tulsiani. \"Canonical surface mapping via geometric cycle consistency.\" ICCV 2019.  \n  ii. Zhou, Tinghui, et al. \"Learning dense correspondence via 3d-guided cycle consistency.\" CVPR 2016  \n  iii. Aumentado-Armstrong, Tristan, et al. \"Cycle-Consistent Generative Rendering for 2D-3D Modality Translation.\" 3DV 2020.  \n  iv. Groueix, Thibault, et al. \"Unsupervised cycle‐consistent deformation for shape matching.\" Eurographics 2019.  \n\n\n\n\n\n  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do mention certain limitations of their approach. However, it would be good if they discussed why intuition behind why some of the categories do not perform well for part segmentation label transfer task.", "review_text": "", "extra_scores": {"ethical_concerns": "They havent been discussed.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~5"}, "extra_text": {}}, {"review_id": "o7LOf2v5tom", "reviewer": "Reviewer_DYH7", "summary": "The paper proposes an autoencoder architecture that decouples point cloud’s representation into the global latent vector and spherical canonical embedding (in $\\mathbb{S}^2$ space) for each of its points. It is trained in the unsupervised way on the category-specific point clouds. This canonical map turns out to be robust to missing parts due to a specially crafted regulariser. What’s new related to the prior art is that the method is somewhat robust to initial orientation of the shape. The formulation is simple and elegant, however, the method is tested only on synthetic ShapeNet data, so it is unclear whether it would work well in a more practical setting. There are also other methodological concerns and clarity problems.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance / Experiments\n\n(+) the ability to ignore parts of $\\mathbb{S}^2$ embedding space with asymmetric Chamfer regulariser is novel and useful;\n\n(+) most methods in this area train on ShapeNet with aligned cameras; such a setting would be difficult to achieve with the real data; this method instead claims to be somewhat invariant to rotations: if this useful property generalises to real data, it can be used for global rigid alignment, e.g. to initialise ICP or to fit a parametric model;\n\n(−) experiments on rotation invariance are somewhat limited though: rotations are sampled from Normal distribution with STD=0.5 radians, which is less than 30˚ , so it hardly tests the ability to perform global alignment; a proper test would be to take random rotations; even given that limited setup, the method is not very accurate: from Fig. 7b, at the distance = 0.05, it gets 30% correspondences correctly vs 60% of the aligned case;\n\n(−) all experiments are performed on the synthetic ShapeNet dataset; there can be a significant domain gap with the real data that would be obtained by a LiDAR or dense stereo;\n\n(−) the performance is measured on sparse keypoints, which are usually defined in distinctive points; this measurement can be biased; I suggest to evaluate the method on datasets of SMPL human body models in the variety of poses, where the correspondences are given by mesh structure.\n\nMethodology\n\n(=) the 3D point coordinates are concatenated to the 512D latent code to be fed into the MLP; I would expect the contribution of the first part to be saturated; would some other way of conditioning be more reasonable?\n\n(−) in eq. (2), why are the CD and EMD terms needed? When MSE is satisfied, the other two should be as well;\n\n(−) line 253: do I understand correctly that tanh is used in the end of the MLP that generates points? Its output is bounded to (−1, 1) then; what if the original point cloud would not fit a unit cube? Regardless, it seems like a strange inductive bias to include in the model. (Generating points in the Fourier space might be a good idea)\n\nOriginality\n\n(+) asymmetric Chamfer distance loss to the unit sphere is a new idea which appears to be crucial for being able to learn the spherical embedding;\n\n(−) on the other hand, I don’t see why the first term in ACD, which makes the points to lie on the unit sphere, is needed; why not just normalise the embeddings? (I have a hypothesis but would be good to confirm that it does not work) The weight $\\alpha$ will then balance half of the CD vs MSE. Also, can the second term be replaced by entropy to avoid sampling/discretisation artefacts? Comparing to those simpler baselines would shed the light on the main contribution.\n\n(=) all other components have been used in point cloud autoencoders (including cross-reconstruction loss and PointNet encoder).\n\nClarity\n\n(−) The paper provides loss formulations (1–3) but does not say how they are combined, e.g. with what weights;\n\n(−) is $\\alpha$ *gradually* decreased during training (line 151) or in one step (line 255)?\n\n(−) I don’t understand the theoretical argument in Section 3.2. Also, in eq. (5), I don’t understand the idea of using Euclidean distance in the world space to determine the confidence? What if $S_A$ and $S_B$ are rotated w.r.t. each other?\n\nNotes to the authors (no effect on rating):\n* I would not call such method self-supervised; the terminology is vague here but I think autoencoders that match the output to the input are unsupervised methods (self-supervision usually assumes some modification of the input for supervision);\n* line 48: the sentence is not in grammatical agreement;\n* line 89: the algorithm is feasible to calculate ← is able to calculate;\n* line 146 refers to the pink box in Fig. 1; I don’t see it;\n* line 159: MES loss;\n* line 161: 1e1 ← 10;\n* line 165: a Chamfer loss ← Chamfer loss;\n* line 310: equivalent to enforce ← equivalent to enforcing.\n\n=========\n\nUPD. Thank you for engaging in a discussion and diligently evaluating new baselines. My main concerns are now resolved, to an extent. For rotation invariance the method indeed beats the weak baseline (non-rotation invariant method by design). Please follow-up on the promises to re-position the paper, i.e. claim only invariance within small angles and acknowledge that the problem is far from being solved even for small angles. The new experiments on the ShapeNet crops and nearest-neighbour baseline are steps in the right direction; I encourage you to find ways to evaluate quantitatively. The clarity will hopefully be sorted out with the help of our comments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no dedicated limitation section. What should be discussed:\n* applicability to real-world data;\n* the paper is over-claiming that it is robust to misaligned input (e.g. line 10); in fact, the paper experiments only with a limited angle variation and the accuracy still degrades quite a bit;\n* the paragraph starting with line 189 claims that the shape can be generated by sampling the sphere; however some regions of the sphere can be optional, e.g. a bicycle can have either disk brakes or V-brakes, while the generated point cloud will have both, which is not realistic.", "review_text": "", "extra_scores": {"ethical_concerns": "Re 4b in the checklist: strictly speaking, you not legally allowed to use data if you don’t have a license for it; I’d recommend ask the owners to issue you a license.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "U3YBnm38iwZ", "reviewer": "Reviewer_xuCY", "summary": "The paper proposes a canonical point autoencoder for the purpose of unsupervised learning dense correspondences between 3D shapes of the same category. The canonical space is defined on a 3D sphere, but with extra flexibility that the template needs not to fully cover the full sphere. This gives the flexibility to generalize to non genus 0 shapes. The proposed method seems to give noticeable improvement over existing ones.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general the idea of learning a point cloud autoencoder with the bottleneck defined as the projection onto a 3D sphere seems to be effective, as demonstrated by the authors in the experiments. The authors did a pretty good job in explaining their methods,  and the experiments are thorough. \n\nThough I do have several concerns / comments for the paper.\n(1) The intuition of why the proposed 3D sphere canonical space can handle non genus 0 shapes is provided very late in the paper, not until page 4 when “relax the bidirectional constraint of the Chamfer loss” is mentioned. The authors should give a clearer explanation about this in their intro, otherwise, a spherical canonical mapping is not enough detail to convince the readers that the proposed method is going to work for non genus 0 shapes. In regard to this, on page 3 Ln 99, “non-linear mappings” alone is too ambiguous and not enough to explain why the method is able to lift “such restriction”.\n\n(2) In the intro, the author claimed the method works “.. even. when instances in the training dataset are not aligned, …, does not need to predict an additional rotation matrix ...”  I am not fully convinced by this claim. I fail to see why the proposed method is “rotational invariant”, and how the method is able to solve the ambiguity between shape variation and rotation. In addition, the way it computes confidence of correspondences are clearly not rotational invariant, as it directly evaluates the euclidean distance between two points in the 3D world coordinates. If the source & target point cloud differs by a significant amount of rotation, such confidence score would not make any sense.\n\n(3) I find the “reconstruction of ordered point clouds” section on page 5 difficult to understand. I would suggest the author to make a more formal mathematical proposition. Moreover, the authors seem to try to prove something w.r.t. affine transformation, but why does that applies to MLP which is non-linear? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I don't find any negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "3GXkuCYkVSu", "reviewer": "Reviewer_yaub", "summary": "The paper suggests an end-to-end learnable method to find dense 3D point correspondences of point clouds. The authors first project the input point clouds onto a canonical spherical UV space, where corresponding points of different shape instances map to similar coordinates. Then, an instance-based feature vector is concatenated with the UV coordinates and fed to a decoder to retrieve the original point cloud. Since the UV coordinates hold correspondence information, the output point cloud is, in fact, an ordered one.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well written. The problem setting is well-motivated, and the paper addresses an important research problem. \n\nThe model achieves state-of-the-art performance.\n\nThe purpose of each component of the model is well discussed. \nThe ablation study is insightful.\n\nConcerns:\n\nThe biggest concern I have with the proposed approach is with the combination of loss components. I feel that the ideal combination of these different loss components might be too sensitive to the dataset bias, and the network might not be very stable. See below for detailed comments.\n\n1) The whole methodology of this paper depends on the fact that corresponding points of the input point cloud instances (of the same class) get mapped onto similar UV coordinates. The loss which encourages this is $L_{cross}$. However, this loss is utilized using randomly chosen instances from the same class. What if the chosen random samples are part-missing etc. samples? If the authors specifically avoid such samples, they should mention this. I understand that other related works also do not address this problem. But this paper argues explicitly that finding different intra-class point correspondence (e.g., missing parts) is the main virtue of their work. Hence, I put more weight on this fact. \n\n2) The authors claim that by reducing \\alpha in $L_{ACD}$ gradually allows each shape to have its own UV coordinates to some \"extent\". Now, \"extent\" is a highly ambiguous term. For instance, $L_{ACD}$ can easily force the model to find different UV coordinates for similar instances also (not only part missing instances). There is no direct way to control this. Again, the authors can argue that $L_{cross}$ prevents this by forcing similar instances to have similar UV coordinates. But again, $L_{ACD}$ and $L_{cross}$ have different goals, and hence, I feel the model might be too sensitive to the competition of these loss components. Similarly, the model can be too sensitive and dataset dependent for the ideal gradual decreasing of \\alpha. If alpha stays close to 1 too much, the model might not work well with part-missing instances. On the other hand, if \\alpha stays close to 0 too much, even similar instances might have different UV coordinates.\n\n3) Let's assume that by exhaustive parameter tuning, we find the ideal hyperparameters to avoid the above issues. Still, there is an inherent mismatch between the goals of $L_{ACD}$ and $L_{cross}$. The authors argue that somehow they can find an ideal point where the end-goal is satisfied. However, it is only guaranteed if the decoder is a linear transformation (Eq. 4). This is a very strong assumption that is clearly violated since the model uses a non-linear MLP. For example, a non-linear MLP can easily find a solution that satisfies both $L_{ACD}$ and $L_{cross}$ by projecting two similar instances onto different sets of UV coordinates and still reconstructing a similar output point cloud. Since the MLP is non-linear, it does not need to have similar UV coordinates to reconstruct similar outputs. \n\nIn summary, I feel that although the results are impressive, the model contains too many moving parts and hence is not a robust approach. However, I may be missing some crucial facts, and in that case, I invite the authors to enlighten me.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations not discussed. A (possible) critical limitation of the model is the instability and the high sensitivity to the hyper-parameters. The authors should discuss these aspects in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 positive final ratings: 6, 6, 7, 6.
On the positive side, reviewers acknowledged importance of the problem, originality of contributions (asymmetric Chamfer distance loss) an overall meaningful and well motivated approach, well executed ablation studies and strong performance. At the same time, they expressed concerns with the method's complexity (multiple loss terms that need balancing), requested additional clarifications (on training details, rotational invariance) and suggested to test the method on a real-world dataset. These concerns were mostly addressed in the rebuttal.
The final recommendation is therefore to accept for poster presentation.
Decision: Accept (Poster)