Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a novel solution for learning dense correspondence between un-aligned 3D shapes in an self-supervised manner. The key contributions this paper is to design a novel canonical space for representing all 3D shapes and learning an auto-encoder to learn a mapping to and from an input point cloud to this canonical space. The effectiveness of this approach is shown for the 3D semantic keypoint transfer and part segmentation transfer tasks across different categories.
main_review: ## Strengths

1. The paper proposes a novel canonical space for representing 3D shape, which helps to learn correspondence of shapes. 
2. The paper proposes a way to learn an auto-encoder, which can learn how to map to and from the input point cloud and the canonical space. Besides the loss functions, used in [9], they use a modified version of Chamfer loss at the canonical representation space as well as a cross reconstruction loss where the encoded features from a particular shape instance is transfered to another shape instance.
3. The paper shows the effectivess of this approach on the 3D semantic keypoint transfer and part segmentation transfer tasks where they show results across different categories. 

## Weaknesses
1. **Use of sphere as the canonical representation**: There are several aspects that are unclear about using sphere as the canonical representation   
  a. *Motivation*: The paper does not talk about the motivation behind using sphere as a primitive for the canonical representation. Shouldn't projecting a 3D shape onto the surface of the sphere lose the convexity and concavity of the different 3D shapes? The authors should also cite papers from the rich literatures (including recent literature on 3D shape reconstruction) that has explored different kinds of primitives.    
  b. *Empirical Results* From the emprical perspective, the fact that the yellow (w/o $ L_{cross}^{\dagger} $) and brown (Liu et al) lines are close by in Fig 7(b),  show that this just using this representation is not significantly useful. Moreover, Fig 9 shows that $ \alpha = 0 $, does not work at all where the representation in the primitive space collapses to a single point. Isn't this an indication that this representation is not powerful enough to capture a good canonical representation?    
 c. *Loss at the encoded representation layer*: Why is Chamfer loss used at the encoder space? This is not supposed to be exact input 3D shape. What is the motivation of using such a loss function instead of loss functions which only looks at local surfaces or neighborhoods? Some recent papers that come to mind are as follows:   
    - Smirnov, Dmitriy, et al. "Deep parametric shape predictions using distance fields." CVPR. 2020.  
    - Sharma, Gopal, et al. "Parsenet: A parametric surface fitting network for 3d point clouds." ECCV 2020.  
 d. It is not clear why it is expected that the rare components would appear far away from the primitive. Is this infered emprically and/or is there an high level intuition why this would happen?  

2. **Training details**: It would be good to mention more details about the training approach:  
  a. *Training data*: Which dataset do you train on? How many categories are there? How many shape instances are there per category?  
  b. *Cross-reconstruction loss*: It is not clear whether the other point cloud is sampled randomly from the same category or from a different category. During training, in a min-batch, how is the batch setup from this perspective?  
  c. Do you perform data augmentation on the input point cloud, especially for the cross representation loss part? If no, why would this not help?  
  d. Why do you rotate by only sampling from $ \mathcal{N} (0, 0.5^2) $? Is that parameter chosen randomly or there is some empirical justification?  


3. **Relation to cycle consistency**: The cross-reconstruction loss has similarities to cycle consistency. This is definitely a novel formulation for this task. But it would be interesting to cite related works on this topic and discuss what are the differences of this loss compared to others. Somne of the papers related on this part are:  
  i. Kulkarni, Nilesh, Abhinav Gupta, and Shubham Tulsiani. "Canonical surface mapping via geometric cycle consistency." ICCV 2019.  
  ii. Zhou, Tinghui, et al. "Learning dense correspondence via 3d-guided cycle consistency." CVPR 2016  
  iii. Aumentado-Armstrong, Tristan, et al. "Cycle-Consistent Generative Rendering for 2D-3D Modality Translation." 3DV 2020.  
  iv. Groueix, Thibault, et al. "Unsupervised cycle‐consistent deformation for shape matching." Eurographics 2019.  





  

limitations_and_societal_impact: The authors do mention certain limitations of their approach. However, it would be good if they discussed why intuition behind why some of the categories do not perform well for part segmentation label transfer task.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper suggests an end-to-end learnable method to find dense 3D point correspondences of point clouds. The authors first project the input point clouds onto a canonical spherical UV space, where corresponding points of different shape instances map to similar coordinates. Then, an instance-based feature vector is concatenated with the UV coordinates and fed to a decoder to retrieve the original point cloud. Since the UV coordinates hold correspondence information, the output point cloud is, in fact, an ordered one.
main_review: Strengths:
The paper is well written. The problem setting is well-motivated, and the paper addresses an important research problem. 

The model achieves state-of-the-art performance.

The purpose of each component of the model is well discussed. 
The ablation study is insightful.

Concerns:

The biggest concern I have with the proposed approach is with the combination of loss components. I feel that the ideal combination of these different loss components might be too sensitive to the dataset bias, and the network might not be very stable. See below for detailed comments.

1) The whole methodology of this paper depends on the fact that corresponding points of the input point cloud instances (of the same class) get mapped onto similar UV coordinates. The loss which encourages this is $L_{cross}$. However, this loss is utilized using randomly chosen instances from the same class. What if the chosen random samples are part-missing etc. samples? If the authors specifically avoid such samples, they should mention this. I understand that other related works also do not address this problem. But this paper argues explicitly that finding different intra-class point correspondence (e.g., missing parts) is the main virtue of their work. Hence, I put more weight on this fact. 

2) The authors claim that by reducing \alpha in $L_{ACD}$ gradually allows each shape to have its own UV coordinates to some "extent". Now, "extent" is a highly ambiguous term. For instance, $L_{ACD}$ can easily force the model to find different UV coordinates for similar instances also (not only part missing instances). There is no direct way to control this. Again, the authors can argue that $L_{cross}$ prevents this by forcing similar instances to have similar UV coordinates. But again, $L_{ACD}$ and $L_{cross}$ have different goals, and hence, I feel the model might be too sensitive to the competition of these loss components. Similarly, the model can be too sensitive and dataset dependent for the ideal gradual decreasing of \alpha. If alpha stays close to 1 too much, the model might not work well with part-missing instances. On the other hand, if \alpha stays close to 0 too much, even similar instances might have different UV coordinates.

3) Let's assume that by exhaustive parameter tuning, we find the ideal hyperparameters to avoid the above issues. Still, there is an inherent mismatch between the goals of $L_{ACD}$ and $L_{cross}$. The authors argue that somehow they can find an ideal point where the end-goal is satisfied. However, it is only guaranteed if the decoder is a linear transformation (Eq. 4). This is a very strong assumption that is clearly violated since the model uses a non-linear MLP. For example, a non-linear MLP can easily find a solution that satisfies both $L_{ACD}$ and $L_{cross}$ by projecting two similar instances onto different sets of UV coordinates and still reconstructing a similar output point cloud. Since the MLP is non-linear, it does not need to have similar UV coordinates to reconstruct similar outputs. 

In summary, I feel that although the results are impressive, the model contains too many moving parts and hence is not a robust approach. However, I may be missing some crucial facts, and in that case, I invite the authors to enlighten me.
limitations_and_societal_impact: Limitations not discussed. A (possible) critical limitation of the model is the instability and the high sensitivity to the hyper-parameters. The authors should discuss these aspects in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a canonical point autoencoder for the purpose of unsupervised learning dense correspondences between 3D shapes of the same category. The canonical space is defined on a 3D sphere, but with extra flexibility that the template needs not to fully cover the full sphere. This gives the flexibility to generalize to non genus 0 shapes. The proposed method seems to give noticeable improvement over existing ones.
main_review: In general the idea of learning a point cloud autoencoder with the bottleneck defined as the projection onto a 3D sphere seems to be effective, as demonstrated by the authors in the experiments. The authors did a pretty good job in explaining their methods,  and the experiments are thorough. 

Though I do have several concerns / comments for the paper.
(1) The intuition of why the proposed 3D sphere canonical space can handle non genus 0 shapes is provided very late in the paper, not until page 4 when “relax the bidirectional constraint of the Chamfer loss” is mentioned. The authors should give a clearer explanation about this in their intro, otherwise, a spherical canonical mapping is not enough detail to convince the readers that the proposed method is going to work for non genus 0 shapes. In regard to this, on page 3 Ln 99, “non-linear mappings” alone is too ambiguous and not enough to explain why the method is able to lift “such restriction”.

(2) In the intro, the author claimed the method works “.. even. when instances in the training dataset are not aligned, …, does not need to predict an additional rotation matrix ...”  I am not fully convinced by this claim. I fail to see why the proposed method is “rotational invariant”, and how the method is able to solve the ambiguity between shape variation and rotation. In addition, the way it computes confidence of correspondences are clearly not rotational invariant, as it directly evaluates the euclidean distance between two points in the 3D world coordinates. If the source & target point cloud differs by a significant amount of rotation, such confidence score would not make any sense.

(3) I find the “reconstruction of ordered point clouds” section on page 5 difficult to understand. I would suggest the author to make a more formal mathematical proposition. Moreover, the authors seem to try to prove something w.r.t. affine transformation, but why does that applies to MLP which is non-linear? 

limitations_and_societal_impact: Yes, I don't find any negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an autoencoder architecture that decouples point cloud’s representation into the global latent vector and spherical canonical embedding (in $\mathbb{S}^2$ space) for each of its points. It is trained in the unsupervised way on the category-specific point clouds. This canonical map turns out to be robust to missing parts due to a specially crafted regulariser. What’s new related to the prior art is that the method is somewhat robust to initial orientation of the shape. The formulation is simple and elegant, however, the method is tested only on synthetic ShapeNet data, so it is unclear whether it would work well in a more practical setting. There are also other methodological concerns and clarity problems.
main_review: Significance / Experiments

(+) the ability to ignore parts of $\mathbb{S}^2$ embedding space with asymmetric Chamfer regulariser is novel and useful;

(+) most methods in this area train on ShapeNet with aligned cameras; such a setting would be difficult to achieve with the real data; this method instead claims to be somewhat invariant to rotations: if this useful property generalises to real data, it can be used for global rigid alignment, e.g. to initialise ICP or to fit a parametric model;

(−) experiments on rotation invariance are somewhat limited though: rotations are sampled from Normal distribution with STD=0.5 radians, which is less than 30˚ , so it hardly tests the ability to perform global alignment; a proper test would be to take random rotations; even given that limited setup, the method is not very accurate: from Fig. 7b, at the distance = 0.05, it gets 30% correspondences correctly vs 60% of the aligned case;

(−) all experiments are performed on the synthetic ShapeNet dataset; there can be a significant domain gap with the real data that would be obtained by a LiDAR or dense stereo;

(−) the performance is measured on sparse keypoints, which are usually defined in distinctive points; this measurement can be biased; I suggest to evaluate the method on datasets of SMPL human body models in the variety of poses, where the correspondences are given by mesh structure.

Methodology

(=) the 3D point coordinates are concatenated to the 512D latent code to be fed into the MLP; I would expect the contribution of the first part to be saturated; would some other way of conditioning be more reasonable?

(−) in eq. (2), why are the CD and EMD terms needed? When MSE is satisfied, the other two should be as well;

(−) line 253: do I understand correctly that tanh is used in the end of the MLP that generates points? Its output is bounded to (−1, 1) then; what if the original point cloud would not fit a unit cube? Regardless, it seems like a strange inductive bias to include in the model. (Generating points in the Fourier space might be a good idea)

Originality

(+) asymmetric Chamfer distance loss to the unit sphere is a new idea which appears to be crucial for being able to learn the spherical embedding;

(−) on the other hand, I don’t see why the first term in ACD, which makes the points to lie on the unit sphere, is needed; why not just normalise the embeddings? (I have a hypothesis but would be good to confirm that it does not work) The weight $\alpha$ will then balance half of the CD vs MSE. Also, can the second term be replaced by entropy to avoid sampling/discretisation artefacts? Comparing to those simpler baselines would shed the light on the main contribution.

(=) all other components have been used in point cloud autoencoders (including cross-reconstruction loss and PointNet encoder).

Clarity

(−) The paper provides loss formulations (1–3) but does not say how they are combined, e.g. with what weights;

(−) is $\alpha$ *gradually* decreased during training (line 151) or in one step (line 255)?

(−) I don’t understand the theoretical argument in Section 3.2. Also, in eq. (5), I don’t understand the idea of using Euclidean distance in the world space to determine the confidence? What if $S_A$ and $S_B$ are rotated w.r.t. each other?

Notes to the authors (no effect on rating):
* I would not call such method self-supervised; the terminology is vague here but I think autoencoders that match the output to the input are unsupervised methods (self-supervision usually assumes some modification of the input for supervision);
* line 48: the sentence is not in grammatical agreement;
* line 89: the algorithm is feasible to calculate ← is able to calculate;
* line 146 refers to the pink box in Fig. 1; I don’t see it;
* line 159: MES loss;
* line 161: 1e1 ← 10;
* line 165: a Chamfer loss ← Chamfer loss;
* line 310: equivalent to enforce ← equivalent to enforcing.

=========

UPD. Thank you for engaging in a discussion and diligently evaluating new baselines. My main concerns are now resolved, to an extent. For rotation invariance the method indeed beats the weak baseline (non-rotation invariant method by design). Please follow-up on the promises to re-position the paper, i.e. claim only invariance within small angles and acknowledge that the problem is far from being solved even for small angles. The new experiments on the ShapeNet crops and nearest-neighbour baseline are steps in the right direction; I encourage you to find ways to evaluate quantitatively. The clarity will hopefully be sorted out with the help of our comments.
limitations_and_societal_impact: There is no dedicated limitation section. What should be discussed:
* applicability to real-world data;
* the paper is over-claiming that it is robust to misaligned input (e.g. line 10); in fact, the paper experiments only with a limited angle variation and the accuracy still degrades quite a bit;
* the paragraph starting with line 189 claims that the shape can be generated by sampling the sphere; however some regions of the sphere can be optional, e.g. a bicycle can have either disk brakes or V-brakes, while the generated point cloud will have both, which is not realistic.
review_title: null

Meta-review: This submission received 4 positive final ratings: 6, 6, 7, 6.
On the positive side, reviewers acknowledged importance of the problem, originality of contributions (asymmetric Chamfer distance loss) an overall meaningful and well motivated approach, well executed ablation studies and strong performance. At the same time, they expressed concerns with the method's complexity (multiple loss terms that need balancing), requested additional clarifications (on training details, rotational invariance) and suggested to test the method on a real-world dataset. These concerns were mostly addressed in the rebuttal.
The final recommendation is therefore to accept for poster presentation.
Decision: Accept (Poster)