Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper analyzes the problem of learning orientation via self-supervised learning, especially via relative orientation data. Specifically, the paper proposes to use a new representation via Modified Rodrigues Parameters (MRP) and claim that this provides more stable learning convergence when compared to previous methods which use different rotational spaces like SO(3).

The paper then analyzes MRP experimentally on experiments including uniformly sampled rotations, the 1DSfM dataset, and an image-based neural network optimization problem. The results show that MRP improves results upon the baselines compared to previous methods.
strengths_and_weaknesses: Strengths
+ The authors are clear in the theoretical explanations of MRP and in the visuals and examples
+ The authors show clear benefit of using MRP for estimating orientation via relative rotations compared to previous methods, on the evaluated datasets
+ The authors run experiments across multiple datasets and include an end-to-end example using a deep neural network
+ The experiments show that MRP converges faster than previous methods, supporting the authors’ claims
+ The method is able to be used as a relatively straightforward, drop-in replacement for other rotation representations for the self-supervised relative rotation problem, and this ease may expand its impact

Weaknesses

- The authors show that MRP outperforms prior methods in evaluation results, but some charts like in Fig. 2. (left) show that some other methods may not have fully converged. It would be good to understand whether MRP would converge to a better performance compared to other methods, after they have all converged.
- The prior work compared against [21,22,28] use datasets/benchmarks, but none are covered in this paper, e.g. [28: modelnet40 point clouds / modelnet10 images] [21: ​​CMU Motion Capture Database] [22: pascal3D+] and instead novel benchmark sets are used
- How does MRP and the other relative rotation methods compare to using direct supervised learning? How far does MRP bridge the gap to supervised learning techniques? e.g. Should practitioners considering supervised methods here instead use MRP?
- The neural network optimization benchmark is evaluated on only one object, the YCB drill. While the method performs much better than prior work for this model, it would’ve been more convincing that this method is very compatible with neural network based optimization if an entire set of objects were tested. This may help answer some other questions, e.g.
  - Would this method work better for image-based optimization if the object has some symmetries?
  - How may a practitioner envision using this algorithm in a larger end-to-end neural network based learning problem? How practical would unseen relative rotations or learning curriculums be in this end-to-end case?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Taken as a whole, this work makes a clear case for use of MRP via theoretical explanations and in isolated experimental results for use in relative orientation self-supervised learning. However, the larger impact of the work is tied to the use and feasibility of self-supervised approaches for orientation learning compared to supervised ones, which is not examined via experiments. This limits the conclusion of the work - it is not clear whether practitioners should switch to using MRP, which requires using self-supervised methods as a whole. Noted, the authors make clear mitigations or concerns of self-supervised methods like curriculum requirements.

Limiting scope to improvement of rotation representation for self-supervised methods, there also lacks direct comparison to benchmarks used in prior work and a comprehensive picture of neural-network based performance. Thus, while improvement looks promising given the presented results, it would be hard to draw a comprehensive, apples-to-apples conclusion.


Edit: post review, the authors have sufficiently addressed my concerns and it is clear that the work is promising for usage/consideration by practitioners to improve neural network rotation prediction.
issues: Some summary of issues that could be addressed from above:

- (Fig 2 left)  Are all methods fully converged? Would running more steps improve some methods?
- A comparison on some benchmark used in prior work, e.g. [28: modelnet40 point clouds / modelnet10 images] [21: ​​CMU Motion Capture Database] [22: pascal3D+]
- How does MRP and the other relative rotation methods compare to using direct supervised learning? How far does MRP bridge the gap to supervised learning techniques? e.g. Should practitioners considering supervised methods here instead use MRP?
- The neural network optimization benchmark is evaluated on only one object, the YCB drill. While the method performs much better than prior work for this model, it would’ve been more convincing that this method is very compatible with neural network based optimization if an entire set of objects were tested. This may help answer some other questions, e.g.
  - Would this method work better for image-based optimization if the object has some symmetries?
  - How may a practitioner envision using this algorithm in a larger end-to-end neural network based learning problem? How practical would unseen relative rotations or learning curriculums be in this end-to-end case?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose the use of Modified Rodrigues Parameters (MRP), which maps Rotations in $SO(3)$ to vectors in $\mathbb{R}^3$, in an incremental manner so that it is suited for learning with gradient descent. They look at self-supervision using relative rotations, where given a set of inputs the relative rotations w.r.t. a subset of the inputs is known, such as by using ICP or camera tracking. This trick of optimizing in the space of MRP circumvents issues that come with $SO(3)$ as it is a closed manifold. The intuition behind the method is shown well in illustrative toy examples making it easy to understand. The effectiveness and efficiency of the approach are shown on two datasets, a toy dataset of uniformly sampled rotations and the 1DSfM dataset on which direct rotation parameter estimates are optimised. Good performance is also shown in an initial experiment of training neural networks on images for relative object pose estimation for synthetic images of a single object.

strengths_and_weaknesses: ### Strengths
- One of the first works exploring the use of MRP for Gradient Descent approaches rather than global optimisation.
- show faster convergence when training direct rotation parameter estimates and neural networks and also overall low error against baseline methods ( less than $5^\circ$ for pairwise relative poses) 
- provide good illustrative toy examples (with visualisations on their website) allowing the reader/viewer to easily understand the problems with optimising in $SO(3)$ and how these are circumvented in $\mathbb{R}^3$ and how their iterative approach enables faster convergence.
- The paper is nicely written and has a good progressive flow to understand the problem that is looked at and the approach presented by the authors.

(Post-Rebuttal Updates)
- The additional results shown in the supplementary material on both synthetic object poses and on real world datasets of Objects (Pascal3D+) as well outdoor scenes (1DSfM) show the efficacy of the proposed approach compared to other methods using relative supervision. The addition of visual results showing the predicted relative pose predictions helped.

### Weaknesses
- ~~Since the main focus of the work is on deriving an iterative version of MRP averaging, such that it is better suited to work with Neural Network of Gradient Descent approaches, the use of only simulated images of a single object is a very simplistic evaluation for such a claim. While the relative pose estimation with neural networks shows good initial results, its ability to generalise cannot be fully ascertained using only simulated images of a single object.~~
 - ~~The paper (and supplementary) is lacking in visual qualitative results that can better help a reader to understand the results. Visualising some results for example the intermediate results of training the neural networks or the alignments of the 1DSfM data could help one better understand the approach.~~

(Post-Rebuttal Updates)
- I did not understand the need for Section F in the appendix. This essentially shows the advantage of just MRP and not the proposed approach. I would imagine that the use of Iterative MRP as a representation for absolute orientation could also be useful for the results.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The authors propose the use of MRP for training gradient descent approaches. While the idea is simplistic, they are the first to look at this novel use of MRP in a relevant use case in robot learning. The illustration of existing issues in optimising in $SO(3)$ manifolds is well illustrated, motivating the idea of using MRP to transform the space of the loss landscape. The idea, although incremental, is novel enough to be worth considering as part of the conference. However, the simplistic nature of the evaluations, especially in the context of neural networks, prevents me from completely advocating for a strong acceptance, especially when the goal is to train neural networks well for predicting rotations. ~~The paper could benefit from a stronger set of experiments on a more diverse dataset rather than on a single object (from an already diverse dataset), especially when considering training neural networks. For example, by training existing approaches like those presented in the Related Works section using the MRP representation, I would imagine that other than just relative rotations, even learning absolute rotations could benefit by using MRP representations during training. While I would like to give more than just a weak accept, since the only next level is a strong accept without anything in between, my final recommendation is a weak accept.~~

### Post-Rebuttal Updates
The updated results show a good efficacy of the proposed approach on a range of different datasets when compared with different relative rotation estimation approaches. I did not fully understand Section F in the supplementary as it doesn't showcase the usefulness of the proposed approach but rather shows the power of using MRP. Nevertheless, the iterative extension of MRP making it suitable for use in gradient descent settings and the promising results warrant an acceptance.

issues: - While the use of optimisation for predicting relative transforms is an old one in computer vision, there have been approaches that explore the use of such optimization (like non-linear least squares) in a differentiable way within neural network frameworks trained with gradient descent (BA-Net [1], or Deep Direct VO [2] who use LM as part of the Network optimisation). I would imagine that such an approach could work in such a case of rotation averaging when considering a minibatch of samples with their neighbourhood, given that such optimisation approaches are known for handling not just $SO(3)$ manifolds but $SE(3)$ as well. 

[1] [Tang, C. and Tan, P., 2018, September. BA-Net: Dense Bundle Adjustment Networks. In International Conference on Learning Representations (ICLR).](https://arxiv.org/abs/1806.04807)

[2] [Wang, C., Buenaposada, J.M., Zhu, R. and Lucey, S., 2018. Learning depth from monocular videos using direct methods. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).](https://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Learning_Depth_From_CVPR_2018_paper.pdf)

- MRP is not a contribution of the paper but is something behind the base of the approach, maybe calling the approach iMRPAvg or something to emphasise the fact that the proposed approach is an iterative Version encapsulating MRP would be better. 
- ~~In Table 4 what is Acc $5^\circ$? Is it the same as "Avg Pairwise Angular Error < $5^\circ$ " in Table 1? This needs to be clarified in the text.~~

(Post-Rebuttal Updates)
- As mentioned above, Section F doesn't supplement the proposed approach but rather shows the advantage of MRP. It would have been better to show the absolute orientations of using the Iterative MRP approach proposed by the authors.
- The formatting of Tables S1 and S2 in the supplementary material need to be fixed.
- The XYZ axes that are overlayed on the images (Figures 3, S6-S8) can be made thicker as they are very thin to see properly. Additionally, Figures S8 and S7 seem to have an overlay on them which makes it difficult to see the images clearly.
- I would suggest the authors to export the graphs in vector graphics (SVG) or PDF formats in order to prevent pixelation of the images.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a self-supervised rotation estimation method for object or camera pose estimation. Major contributions are as follows:

(1) Author proposed to use modified Rodrigues parameters to stereographically project the closed manifold of $SO(3)$ to the open manifold of $R^3$, which can speed up the optimization and avoid the local optima.

(2) The proposed optimization could be applied to the Stochastic Gradient Descent of the training for neural networks.
strengths_and_weaknesses: **Strengths:**
* According to the experiment section, the proposed Iterative Modified Rodrigues Averaging could improve the convergence behavior of the rotation averaging problem.
**Weaknesses:**
* The novelty statement is unclear. Line 82, "the unique problems found in the rotation averaging problem" need more detatils.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The experiment section looks convincing. My major concern is the novelty and contribution compared to [24] and [25] are unclear.

Post-rebuttal:
The reply from the author partially resolves my concern, I would keep my rating as "weak accept", I think it deserves a poster presentation.
issues: Authors should clearly state their novelty and contribution against the prior works.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper uses modified Rodrigues parameters for estimating absolute rotations from relative rotations, which can be used within "self-supervised" training of neural networks. In this training regime, it is assumed that relative orientations between neighboring poses are given, for example, computed from a local alignment method like ICP, and that a neural network should be trained by using the relative poses as supervision. The paper shows that the proposed scheme can obtain better results than baseline methods for synthetic data and 1DSfM data when the rotations are optimized directly. Additionally, the paper shows a synthetic example where the rotations are not inferred directly, but a neural network is trained to infer the rotations from rendered images.
strengths_and_weaknesses: **Strengths:**

- The paper proposes an interesting method for rotation learning that is based on modified Rodrigues parameters. It nicely combines current deep learning with traditional works on rotation parametrization.
- The paper is overall well-written and the content is well-presented.


**Weaknesses**:

- The paper does not show results for an actual learning problem. The shown examples either have no learning (Sec. 6.1) or only synthetic data with overfitting, i.e. no learning as well (Sec 6.2).
- The paper does not show results on recognized benchmarks. Sec. 6.1 uses the 1DSfM dataset but does not compare with global methods, which are SOTA there. The other experiments use data generated by the authors.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper proposes a seemingly novel and interesting idea, however, an evaluation on actual learning tasks and real-world data is missing. This makes it impossible to judge the benefit of the proposed method for robot learning. If the paper would show extensive experiments on real-world learning tasks I would be willing to increase my score.
issues: The paper would be improved significantly if the authors can provide extensive evaluations on real-world learning tasks wrt. established SOTA methods on established benchmarks. However, this might be too much for a rebuttal and should be done when potentially re-submitting the paper.
review_title: null

Meta-review: metareview: The paper proposes a self-supervised learning approach to solve pose estimation. The method uses a new representation via Modified Rodrigues Parameters (MRP) to solve the problem of learning orientation via relative orientation data. The method is evaluated against uniformly sampled rotations, the 1DSfM dataset, and an image-based neural network optimization problem.

While most of the reviewers agree that the paper is well written especially with good illustrative examples, and the benefit of using MRP is clear in terms of convergence, they brought up several aspects of improvement for results:

(1) Lacks direct comparison to benchmarks used in prior work.

(2) Generalization is questionable as it only uses simulated images with single object.

(3) Evaluation on actual learning tasks and real world data will make the paper stronger.

During the reviewer discussion, most reviewers acknowledged that their major concerns are addressed in the rebuttal. Reviewers agree the additional results in the revised paper show their method's usefulness with unseen objects. For the above reasons, we recommend accepting the paper as poster.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)