Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work builds a capsule network, for use with point cloud data, that has units that are equivariant to SO(3) 3D rotations.  I found that the authors made the case for rotation equivariance well and I liked the analysis of the dynamic routing approach and its mapping to the Generalized Weiszfeld Iterations. 

As a caveat to the review I should point out that while I am familiar with capsule networks, they are not my main area of expertise (I do work on geometry however) so this should be taken into consideration - apologies if there is literature I have missed. 

My main concern is that group equivariance for Capsule networks has been studied before - I acknowledge that SO(3) can present additional challenges but would have expected a work that specifically targeted SO(3) to investigate the relative benefits of the different approaches to encoding the lie algebra and not just quaternions. How do we know that quaternions are the best fit for the Capsule framework? Could the authors show empirical results that indicate that other representations for the equivariance do not work as well? My experience from optimization is that the best representation can often be application specific and it would be very helpful to understand their relative merits for this sort of architecture.

The preparation of the local patches and the estimation of the rotations seems very important to these applications to me and would be worthy of more discussion and an empirical presentation of their different merits. I know the authors discussed the issue of the second axis but it would seem worth including an experiment to detect the sensitivity in a similar manner to the point cloud density.

My biggest concern about the paper is the presentation of the results. Please could the authors provide appropriate error bars for the tables - this is the least action necessary to at least start to estimate significance. Ideally something like histograms or violin plots would be more useful. The use of a single dataset makes it hard to identify the efficacy of the approach - especially when it is not clear how the training and test data are set-up. Are the same objects used in training and test but with different rotations? Or different objects from the same categories. What is the protocol for the AR test set?

I agree that table 1 confirms the equivariance so that is great but it is less clear that the network is suitable for the pose task since the baselines seem very simple for the pose estimation experiment (and the related work covers a lot of prior work in this area) and again, without error bars it is hard to judge significance.

It is nice that the authors discuss limitations and I agree with the comment on symmetries - would it be possible to include an illustrative example to confirm these suspicions?


Other notes:

I have no issue with the proofs - they all seemed to make sense to me.

I found Figure 1 (and to a lesser extent Figure 3) to be unclear and uninformative - the notation doesn't match the main part of the paper (also where does C come from in the caption?) and its not clear what the illustrations are meant to represent. Would it not be much more clear to include something like Algorithm 1 instead of Figure 1 which is much more clear and easy to replicate the precise process.

I think there is a lot of material in the appendix that should really be in the main paper - I'm happy for the proofs to be in the appendix but I think it is a bit wrong to essentially violate the page restrictions by moving important related work into the appendix. The whole of section 2 is standard text-book information about quaternions - would it not be more appropriate for that to be in the appendix and the related work to be in the main paper?

Could the authors indicate the use of the Frobenius norm in eqn (5)? There also seems to be a mixing of notations for inner products through-out the paper and it might be helpful to standardize this?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a quaternion equivariant network for 3d point clouds. It builds on the capsule networks and dynamic routing, but instead of using arbitrary 4x4 transforms, they propose to use quaternions to represent 3D rotations.

Invariances and equivariances are very important in DNN for classification/detection, so extending these properties to 3d point clouds based applications is interesting and important. The proposed approach is novel and seems to be equivariant to SO(3) rotations, translations, and set permutations.

However, there are a couple of major negatives in the paper. The first is that the details of the architecture are very unclear. A large amount of space is allocated to discussing equivariant properties but not enough is dedicated to the actual implementations of the network. In particular,  only on page 5 in section 3.2 is the QE network detailed discussed.

- How does one obtain Q_i from points X_i ?
- How are the points subsampled to create LRFs? is there a clustering first step? If so, how would that clustering be invariant to global rotation?
- Does the method require a known centroid of the object point clouds in order to compute the quaternions of the LRFs?
- Theorem 1 has shown that equation 6 of Alg. 1 is equivariant, but are the multiple interactions of DR equivariant? In particular, after k iterations of DR as presented in algorithm 1, with nonlinear sigmoid activations, are the output capsule representations equivariant?
- How do you guarantee that the LRF’s are the same under any global rotation+translations of the point cloud? Or do you assume the same LRFs are known?

The main selling point for equivarince and learning the separation of pose and object class representation is in the hope of more accurate classification. However, empirical classification results are far from the state-of-the-art. 
- As a side question, there are other methods listed on the ModelNet40 website with performances in the high 90’s, are there reasons those results are not included as a part of Table 1? 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a rotation-equivariant model for use with point-clouds.  Capsule networks with activation and rotation pose estimate in the form of a quaternion are applied to local patches of 3D points, and combined up through progressively more coarser receptive fields.  Rotation equivariance is obtained by looking for agreement between input point frames (or capsules), by iteratively clustering using the quaternion mean.  The method is evaluated on ModelNet40, obtaining much improved results in the case where rotation orientation is arbitrary.

Unfortunately, I found too many important parts of the method difficult to understand, enumerated below, and am also not very clear on the details of how they fit together.  At a high level, the approach of aggregating pose agreement with quaternion averages makes intuitive sense, the quaternion average step itself is described well, and the experimental results seem to corroborate this (results on NR/AR condition are very good, and also near identical to NR/NR condition, a major verification of the method).  However, the rest of the pieces of the method still leave me guessing too much.

*  It isn't clear what computes the activations alpha.  I don't see a description for how to compute activations alpha that are the input to Alg. 1, unless these are simply the activations of the previous layer's capsules, and constant for the initial point frames?

*  Likewise, the transform t() could use more description.  Sec 3.2 says it is R3->R4, indicating that the input to t() is a single point.  Does this mean that the mere presence of a point in a location R3 (relative to a mostly-canonicalized pose) counts as a vote for a rotation pose?  And if so, how does does this work?  Or does the regressor actually take multiple points as input?  According to the picture in Fig. 1, it looks like it takes all points in the local patch as input.

*  Algorithm 1 outer for loop for i doesn't seem quite right.  It suggests that the body is performed either independently or in sequence for each i.  I believe what may actually be intended, is that all i are used at once in parallel, so that the v_ij votes assignment is found for *all* i in order to use in finding the cluster mean for each j.  That is, "v_i,j = Q_i * t_i,j for all i", rather than having the outermost loop over i.  Is this correct?

*  Much space is dedicated to background explanations of quaternions, geodescic distance, etc.  Given my difficulty in understanding much of the system, it seems like some of could be shortened or put into the appendix, and more of the main text devoted to more detailed explanations of each component in the system (e.g. alpha and t), and how the capsule agreement step (QE DR) makes use of these.


Also, some more detailed comments/suggestions:

* Alg.1:  output alpha^hat_K, seems K should be M here?
* Table 1:  "Right hand side denotes symmetric objects" seems it actually shoudl be in the caption for Table 2.

review_title: Official Blind Review #3

Meta-review: This paper presents a capsule network to handle 3d point clouds which is equivariant to SO(3) rotations. It also provides the theoretical analysis to connect the dynamic routing approach to the Generalized Weiszfeld Iterations. The equivariant property of the method is demonstrated on classification and orientation estimation tasks of 3D shapes.
While the technical contribution of the method is sound, the main concern raised by the reviewers was the lack of details in the presentation of methodology and results. Although the authors have made substantial efforts to update the paper, some reviewers were still not convinced and thus the scores remained the same. The paper was on the very borderline, but because of the limited capacity, I regret that I have to recommend rejection. 
Invariances and equivariances are indeed important topics in representation learning, for which the capsule network is known as one of the promising approaches but still not well investigated compared to other standard architectures. I encourage authors to resubmit the paper taking in the reviewers' comments.
Decision: Reject