Deep Projective Rotation Estimation through Relative Supervision
Authors: Brian Okorn
Chuer Pan
Martial Hebert
David Held
Venue: CoRL 2022 Poster (2022)
Keywords: rotations
deep learning
relative-supervision
self-supervision
modified rodrigues parameters
stereographic projection
pose estimation

Abstract: Orientation estimation is the core to a variety of vision and robotics tasks such as camera and object pose estimation. Deep learning has offered a way to develop image-based orientation estimators; however, such estimators often require training on a large labeled dataset, which can be time-intensive to collect. In this work, we explore whether self-supervised learning from unlabeled data can be used to alleviate this issue.  Specifically, we assume access to estimates of the relative orientation between neighboring poses, such that can be obtained via a local alignment method. While self-supervised learning has been used successfully for translational object keypoints, in this work, we show that naively applying relative supervision to the rotational group $SO(3)$ will often fail to converge due to the non-convexity of the rotational space. To tackle this challenge, we propose a new algorithm for self-supervised orientation estimation which utilizes Modified Rodrigues Parameters to stereographically project the closed manifold of $SO(3)$ to the open manifold of $\mathbb{R}^{3}$, allowing the optimization to be done in an open Euclidean space. We empirically validate the benefits of the proposed algorithm for rotational averaging problem in two settings: (1) direct optimization on rotation parameters, and (2) optimization of parameters of a convolutional neural network that predicts object orientations from images. In both settings, we demonstrate that our proposed algorithm is able to converge to a consistent relative orientation frame much faster than algorithms that purely operate in the $SO(3)$ space. Additional information can be found at https://sites.google.com/view/deep-projective-rotation.

Deep Projective Rotation Estimation through Relative Supervision
Brian Okorn; Chuer Pan," Martial Hebert, David Held Robotics Institute, School of Computer Science
Carnegie Mellon University, United States (bokorn, chuerp, mhebert, [held)@andren cmu. edu
Orientation estimation is the core to a variety of vision and robotics tasks such camera and object pose estimation. Deep learning has offered a way to develop image-base based orientation estimators; however, such estimators often re- quire training on large labeled dataset, which can be time-intensive collect. In this work, we explore whether self-supervised learning from unlabeled data can be used to alleviate this issue. Specifically, assume access estimates of the relative orientation between neighboring poses, such that can be obtained via a lo- cal alignment method. While self-supervised learning has been used successfully for translational object keypoints, in this work, we show that naively applying rel- ative supervision to the rotational group SO(3) will often fail to converge due the non-convexity of the rotational space. To tackle this challenge, we propose a new algorithm for sulf-suue orientation estimation which utilizes Modified Rodrigues Parameters to stereographically project the closed manifold of SO(3) to the open manifold of IR$, allowing the optimization to be done in an open Eu- clidean space. We empirically validate the benefits of the proposed algorithm for rotational averaging problem in two settings: (1) direct optimization on rotation parameters, and (2) optimization of parameters of convolutional neural network that predicts object orientations from images. In both settings, we demonstrate that our proposed algorithm is able to converge to consistent relative orienta- tion frame much faster than algorithms that purely operate in the SO(3) space. Additional information can be found on website.
Abstract:
Introduction
Pose estimation is critical component for wide variety of computer vision and robotic tasks. It common primitive for grasping, manipulation, and planning tasks. For motion planning and control, estimating an object's pose can help robot avoid collisions or plan how use the object for given task. The current top performing methods for pose estimation use machine learning to estimate the object's pose from image; however, training these estimators tends to rely direct supervision of the object orientation [1, 2, 3]. Obtaining such supervision can be difficult and requires either time-consuming annotations synthetic data, which might differ from the real world. In this work, we explore whether self-supervised learning can be used to alleviate this issue by training an object orientation estimator from unlabeled data. Specifically, we assume that we can estimate the relative rotation of an object between neighboring object poses in self-supervised manner. Such relative supervision can be easily obtained practice, for example through a local registration method such as Iterative Closest Point (ICP) [4] or camera pose estimation.
Relative self-supervision has been previously used for representation learning in estimating transla- tional keypoints [5, 6, 7]. These methods use only relative supervision to ensure that the keypoints consistent across views of the object, and do not directly supervise the keypoint locations. In this work, we explore whether such relative self-supervision can similarly be used in estimating object orientations. We show that naively applying such relative supervision rotations on the SO(3) manifold will often fail to converge. Unlike self-supervised learning of translational keypoints, the
Equal Contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

rotational averaging problem [8] is inherently non convex,,, with many local optima. While there exist global optimization algorithms which jointly optimize all pairs of rotations for this problem [9, 10], they are not easily integrated into the iterative, stochastic gradient descent methods used to train neural network-based pose estimators.
To address this issue, we propose a new algorithm, Iterative Modified Rodrigues Projective Averag- ing, which uses Modified Rodrigues Parameters to map from the closed manifold of SO(3) to the open space of gg' In doing so, we obtain faster convergence with a lower likelihood of falling into local optima. Our experiments show that our method converges faster and more consistently than the standard (3(3) optimization and can easily be integrated into a neural network training pipeline. Additionally, in the Appendix A, we include an intuitive theoretical example describing how, while not all local optima are removed, the dimensionality of a set of problematic configurations is greatly reduced when optimizing using our algorithm, as compared to optimizing in the space of SO(3).
The primary contributions of this work are:
We propose new algorithm, Iterative Modified Rodrigues Projective Averaging, which is an iterative method for learning rotation estimation using only relative supervision and can be applied neural network optimization.
We empirically investigate the convergence behavior of our algorithm as compared to opti- mizing on the SO(3) manifold.
We demonstrate that our algorithm can be used to train neural network-based pose esti- mator using only relative supervision.
Related Work
Averaging and Consensus Estimation: Consensus methods, sometimes referred to averaging methods, have long history of research. The goal of these methods is, given a distributed set of estimates, to produce consistent prediction
value using relative information. While there are iterative algorithms with good convergence properties in Euclidean space [11, 12, 13, 14, 15], optimizing over the closed manifold of SO(3) can be more difficult, as the region non-convex, with many local minima. Hartley et al. [8, 16] describe several methods of finding consistent set of rotations, though their convergence is similarly not guaranteed outside of radius r S x ball in SO(3). Wang and Singer [10] find an exact solution to this problem, using combination of semidefinite programming relaxation and a robust penalty function. More recently, Shonan Rota- tion Averaging [9] shows that projecting to higher dimensional spaces allows for the recovery of a globally optimal solution using semidefinite programming. Chatterjee and Govindu [17, 18) use iterative re-weighted least-squares recover global optimal solution using global error estimates. Shi and Lerman [19] extends this work, using cycle consistency and message passing. These solu- tions require global eиror estimates or semidefinate programming, which are incompatible with the stochastic gradient descent methods used to train neural networks.
2
Supervised Orientation Estimation: Past work has explored using neural network to predict an object's orientation. Traditionally, these methods rely on supervising the rotations using a known absolute orientation, whether in the form of quaternions [20, 1, 21]. axis-angle [22], or Euler an- gles [23]. More recently, 6D [24, 2], 9D [25], and 10D [26) representations have been developed for continuity and smoothness. Recently, Terzakis et al. [27] introduced Modified Rodrigues Param-
eters, a projection of the unit quaternion sphere $3 to 3 used in attitude control [28], to a range of common computer vision problems. Tezzakis al. [27] does not, however, address the unique problems found in the rotation averaging problem.
Recently, there has been research into mapping the Riemannian optimization to the Euclidean opti- mization used for network training [29, 30, 31, 32, 33). These methods focus on applying tangent space gradients from losses in 3D transformation groups. Specifically, Projective Manifold Gradient Layer [29] ensures that the gradients take into account any projection operations, such that the gra- dients point towards the nearest valid representation in the projection's 's preimage. While this does map the Riemannian optimization into a Fuclidean problem, does not solve the problems caused by the closed manifold of (O(3), as this does not alter the underlying topology of this manifold.

---

3 Problem Definition
We formally describe the problem of self-supervised orientation estimation below. We assume that we are given a set of inputs observations (Iz,..., IN], of an object where, in each input observation I, the object is viewed from an unknown orientation R, These inputs could be in the form of images, point clouds, or some other object representation. While we do not know the absolute object orientations R; in any reference frame, we assume that we do know subset of the relative rotations Rrj possibly from a local registration method like ICP, between the object images I and I,, such
that R, = =R,Rg, Our goal is to learn function f() that estimates orientation of the object in each image, f(T,) = R₂ that minimizes the pairwise error between all input pairs and their ground truth relative rotations, with respect to the geodesic distance metric d(R, R2) = || log( ( R2) Given a set of rotations R = (R....., Rx) ). the core optimization objective is thus:
Ea Id(Ri,RRR R.
min
Note that this optimization does not have unique solution, since the solution R, := SR, Vi minimizes this error for any constant rotation S. In many robotics tasks, relative rotations can be accurately estimated only when their magnitude is small as many registration algorithms, such as ICP, requires good initialization near the optimum. Following this observation, we assume that we can only accurately supervise relative rotations when they are small in magnitude. This leads to local neighborhood structure where each rotation R; is connected to Rt only in a local neighborhood around RG when d(R,, R2) e, and the set of all R₃ ' connected to R2 form the neighborhood set NG; While the algorithms described in this manuscript do not rely on this angle e, it can be scaled as needed based on the accuracy of the relative rotation estimation method (e.g. ICP, etc).
Our eventual goal is represent the function fT,) AG neural network. Thus, we restrict the methods with which we compare to iterative methods that are updated using only a sampled subset of the rotations (as opposed to methods that perform a global optimization over the entire set of rotations . Rx }). This requirement is to match the conditions required by stochastic gradient descent, the primary method of training neural networks.
4 Baselines
Preliminaries. The 3D rotational space of SO(3) A {R R3x3 R R = Tax3, det (R) 1) is a compact matrix Lie group, which topologically is a compact manifold. Due to the compactness of the SO(3) manifold, there exist configurations of pairs of points where multiple, non-unique geodesically minimal paths exist between them; for instance, there are two unique geodesically minimal paths for pair of antipodal points on circle, and there are infinitely many for a pair of antipodal points on a sphere. This is not the case in an open manifold like the 3D Euclidean space of RR, over which there exists unique geodesically minimal path between any arbitrary pair of points. The distinction in compactness between the 3D rotational space of SO(3) and 3D Euclidean space makes optimization over S3(3) more ill-conditioned than over the space of R This results in the optimization over the rotational space being non-convex. These properties of the SO(3) manifold will affect the convergence of sule-supe orientation estimation, which we discuss below.
While self-supervised learning for objects translation, specifically in the form of object keypoints [5, 6. 7], has shown great success, in this work, we show that naively applying such an iterative self- supervised formulation to the rotational group 50(3) will often fail to converge. Below we discuss two approaches self-supervised orientation estimation in SO(3).
Quaternion Averaging: A standard objective in rotation estimation is to minimize the geodesic distance between a predicted unit quaternion and its corresponding ground-truth orientation [34, 8), 0 = arccos(2 (q: )²) where q is the predicted orientation for image i and qgt is the ground-truth orientation. An objective function is often defined to directly minimize this geodesic distance [34].
In our task, defined above (Section 3), we are given the relative rotation o' between some pairs of rotations q and 4j- Using this relative supervision, we can use the geodesic distance between a sampled estimate, qi. its desired relative position with respect to a sampled neighbor and a known
3

---

relative rotation q. q: q. @ Gz. leading to the loss
2g =111- 19:1-99.00 2 43)²
(2)
where @ denotes the quaternion multiplication. Note that this loss monotonically related to the geodesic distance when using unit quaternions, while avoiding the need compute arccos.
(O(3) Averaging: To optimize the rotations with respect the non-Euclidean geometry of the rotational manifold of 50(3), one approach is described by Manton [35]. Each orien- tation iteratively updated in the tangent space using the logmap of SO(3) and projected back to 50(3) using the exponential map. Specifically, we can take the gradient of the loss
(so = log R R
(3a) Vi,Cso13) ra = log ( R
(3b)
which gives the update step Rg < R. exp(yra), where y is the learning rate and log is the logmap of SO(3). When optimizing the full set of orientations, this algorithm can fall into local optima due to the closed nature of the space which allows any orientation to be reached by more than one unique straight paths, the space wraps around itself.
Method
We propose alternative that projects the optimization distances in that space. Specifically, we use the Modified Rodriguez Projection to minimize the relative error between neighboring poses in R We provide experiments in Section 6 that show that self-supervised orientation estimation using Modified Rodriguez Projection converges much faster than self-supervised orientation estimation in SO(3), with theoretic analysis of illustrative example available in the Appendix A.
open image and optimizes the
Iterative Modified Rodrigues Projective Averaging
As mentioned previously, optimizing on closed space, such as (3(3) or S can be prob- lematic, since the relative distance between two points can eventually be minimized by mov- ing them in the exact opposite direction of the minimum path between them. To alleviate this issue, we would like to instead perform self- supervised learning in an open space, where this symmetry is broken. This can be done us- ing Modified Rodrigues Parameters (MRP) [36, 27]. MRP is the stereographic projection of the closed manifold of the quaternion sphere $ to RS, and has been widely used in attitude esti- mation and control [28]. In combining this pro- jection with the mapping between 50(3) and S3, this projection can be used to optimize ro-
19
0
(()
o(-)
Projection of relative supervision, q. shown red, between rotations q; := o ( and -Q. into the MRP space update, V/a shown in green. While could have been selected as the the goal rotation, it would have induced much larger movement in the pro- jected space.
Figure 1:
tations. We define unit quaternion g = [e v E S A ( || 1), where p E R defines the scalar component and E 3 defines the imaginary vector component of the unit quaternion.
The projection operator (b4) = & E 3 and its inverse oi (bb)
4 E 3 are given by [36, 27] 2xh
where v = & ((p v)) = I+p x and [p v) = --=
Given this projective
1+ M
|+|
orientation space, we need to map our relative rotation R² into the projective space in order to use these relative rotations for the self-supervised learning task.
This projection is required, as the relative supervision is in SO(3), and the direction and magnitude of this relative measurement are distorted differently in different regions of the projective MRP space. Given pair of estimated projected rotations J; == o(â.) and S. == o(q,), we project ei back to unit quaternion o (يو) = / E S and rotate it according to RJ qi qf 2 aj. where @ is quaternion multiplication and of is the quaternion form of RJ. The resulting unit quaternion q; is

---

then projected back into the Modified Rodrigues Parameter space, vh. A simplified visual analogy of this process is shown in Figure 1.
While this relative rotation could be applied and projected at either the sampled point vi or the neighboring location ey we select the neighboring location L².. as it does not require us to compute gradients through the forward or inverse projections <(.) and (.). respectively. This projected rotation di; represents the value the should hold, relative to the current predicted rotation vy. should be noted that p(q) j ((-g), while and -q represent the same rotation. In terms of the projective space, this means that the sign of q matters. To remove this ambiguity, we select the nearest projection to dh in the projective MRP space. It should be noted that this is different from selecting the closer antipode on S the large deformations found near the south pole² can cause the nearer antipode S to be further in MRP space. In contrast, if we were to select a consistent sign for the scalar component qt, for example ensuring the scalar component is always positive, small change in biy can cause large changes in ths when o b0)())) is near the equator of 3 While this change is required to stabilize our optimization, it does add some ambiguity to the direction of optimization. However, the directions to each of the projected locations, ((ā+) and p(-q.). can only be anti parallel (pulling in exactly opposite directions) if tbs eú intersects the origin in R
The loss with respect to given estimate, vA.. can then be written as the le distance be- tween its current value and the projected relative location, vh. relative to a given neighbor, why
Lv+ = 1 - M(ai (4a) Lu = -1103 o( x(- (4b) Lo =
(4c)
where we recall that, qi = of & qj. and فJ = x0- (}/))
Note that, while d is predicted value, we do not pass gradients through it, allowing to anchor the update to consistent orientation. The gradient update? then given by:
5 o (@:) if Lut < Lv- 4/s (- otherwise
V 3 Lo Uu
(5)
Additionally, maximum gradient step, 7). in the projective space is imposed, t/a " I if the gradient exceeds defined amount. This prevents extremely large steps from being taken, as the projective transform can distort the space.
6 Experiments
Next, we perform experiments to show that our method converges faster and more consistently than the alternative approaches. Our empirical results are grouped into two settings: (1) direct optimiza- tion of randomly generated rotations, Section 6,1, and (2) optimization of the parameters of a con- volutional neural network using synthetically rendered images, Section 6.2. In both cases, relative orientations between elements drigues Projective Averaging is able to converge faster and more often than alternative approaches. We further show in Scction 6.2 that our method can easily be used to supervise convolutional neural networks, when only relative orientation information is available.
neighborhood are provided. We show Iterative Modified Ro-
6.1 Direct Parameter Optimization
We evaluate the convergence behaviour of our Iterative Modified Rodrigues Projective Averaging method, MRP (Ours). described in Section 5, as well as the SO(3) averaging method, described in Section 4. For the (3(3) averaging method, we implement both the pure Riemannian optimization, SO(3), well as method using Projective Manifold Gradient Layer [29] to map the Riemannian gradient the (3( averaging loss, Equation 3a, to a Euclidean optimization in RDD, where we
test D = 4, 6, and 9, 4D PMG [29]. 6D PMG [24]. 9D PMG [25], respectively. Additionally, evaluate direct quaternion optimization, described in Sections 4, Quaternion.
The south pole in this case is descrihed by the quaternion -1 + 0i + 0j + 0h: We omit a constant factor for brevity, and integrate it into the learning rate, 7
5

---

Uniformly Sampled
Alamo Dataset
Mean Naratte ang Free
100
80-
60
40
40
255
7SK
123K
175K
225K
273K
SK
10c
15K
Rerations
Iterations
40 PMD
BD PMR
20 1900
40 PMG
6D PMG
9D PMG
VRP
Quaternion
MRP (Ours)
Figure 2: Relative rotation consensus with direct optimization of rotation parameters over 50 unique environ- ments with 100 random generated orientations each (left) and Alamo IDSfM [37] (right). Median average- pair-wise angular error (*) hetween each estimated rotations shown, with shaded region representing first and third quartile for each method. The max average- parrag-pair-wise angular error for each algorithm at each iteration is shown a dashed line.
Avg Pairwise Angular Error <, 5
Normalized AUC
Algorithm
Mean Steps
Max Steps
Min Steps
Mean
Max
Min
SO(3)
157.7K
Converged
85.0K
24.47
82.92
7.55
4D PMG [29]
126.
Not Converged
27.0K
15.67
52.40
3.06
6D PMG [24]
235.9K
Not Converged
80.0K
43.53
89.15
11.34
9D PMG [25]
284.5K
Not Converged
150.0K
62.94
101.77
17.77
Quaternion
160.3K
Not Converged
40.0K
23.55
84.85
3.47
MRP (Ours)
37,5K
160.0K
15.0K
15.56
2.18
Table 1: Number of iteration steps until convergence and Normalized Area Under Curve (nAUC) over 50 unique
environments of randomly generated orientations. 300K optimization steps are taken for each experiment.
Uniformly Sampled Rotations. We test the performance of each algorithm when directly optimiz-
ing the rotation parameters of set of size N 100 with known relative rotations Ri, and local neighborhood structure. Ground truth and initial estimated rotations are both randomly sampled from uniform distribution in 50(3). Each rotation, R, has neighborhood, Mr consisting of the closest [NG] = 3 rotations with respect to geodesic distance. The connectivity of this neighborhood graph is checked to ensure the graph contains only single connected component. We test all algorithms over 50 sets of unique environments. each with N 100 randomly generated orientations as described above. The estimated rotations are updated by each algorithm in batches of size 8, for 300K iterations.
As the goal of our algorithm is to improve the convergence properties of iterative averaging meth- ods, we analyze each algorithm at various stages of optimization. We are particularly interested in the average number of update steps until the algorithm has converged, which define when the average angular error between all pairs of rotations is below 5 As we can see Figure 2, the Itera- tive Modified Rodrigues Projective Averaging method, MRP (Ours), converges before the standard S0(3) averaging method. On average, our method converged to within 5 in 37K steps. The next best method, 4D PMG [29], which takes over three times many iterations converge the same
% Avg Pairwise Angular Error
5"
Final Erron()
Algorithm
30K
70K
100K
150K
300K
Mean
Median
SO(3)
0%
0%
6%
57%
94%
2.056
0.10
4D PMG [29]
32%
46%
72%
90%
1.969
0.14
6D PMG [24]
0%
0%
4%
20%
52%
20.096
3.20
9D PMG [25]
0%%
0%
0%
2%
20%
40.125
43.02
Quaternion
0%
12%
30%
56%
82%
9.72
0.04
MRP (Ours)
66%
88%
96%
98
100%
0.004
0.004
Table 2: Percentage of experiments converged and final angular errors over 50 unique environments of 100 randomly generated orientations. 300K optimization steps taken for each experiment.

---

level of accuracy. Further, Table I shows that our method is the only one to converge across all environments within 300K iterations. For each method, we also compute the mean area under the pairwise error curve, with the number of steps normalized to between zero and one (nAUC), also shown in Table We find that in the best, average, and worst case scenarios, our method has the best convergence behavior. To quantify convergence behavior, we also compute the percentage of trials that achieve average pairwise angular error below 5 at different stages of training, as shown on the left in Table 2. We find that at each stage of training, the Iterative Modified Rodrigues Projective Averaging, MRP (Ours), training has lower average pairwise error, shown in Table 2. Our method also converged far more often at each stage of training, also shown in Table 2.
Structure from Motion Dataset. To test our algorithms under natural noise conditions, we also evaluate our algorithm on the IDSIM [37] structure from motions datasets. Each environment is tested with random initializations and the estimated rotations are updated by each algorithm in batches of size 64, for 20K iterations. The results of a subset of the environments are shown in Table 3 and the remainder can be found in Appendix C. The noise characteristics of relative rotations in this dataset are similar to those found when capturing relative poses, but, unlike the Uniformly Sampled Rotations environments, the distribution of rotations does not fully cover the orientation space. As result, all methods converge relatively quickly. Our algorithm outperforms the baselines in terms of accuracy. While the Quaternion optimization converges slightly faster, it consistently finds lower accuracy configuration, resulting in a low nAUC, but higher relative and absolute accuracy. More details can be found in the Appendix C.
Mean Relative
Mean Absolute Error ())
Error (")
Mean nAUC
Algorithm
E. Island
Alamo
E. Island
Alamo
Island
Alamo
4D PGM [29]
11.94
15.00
7.34
9.94
25.60
47.20
6D PGM [24]
11.26
18.84
6.90
13.09
27.77
58.04
9D PGM [25]
10.22
16.32
6.32
11.43
29.31
60.14
Quaternion
11.58
13.40
7.23
8.93
16.01
22.57
MRP (Ours)
8.84
9.89
5.49
6,56
16.21
25.61
IRLS GM[17]
3.04
3.64
IRLS- 1 [18]
2.71
3.67
MLP[19]
2.61
3.44
Table Rotation Averaging Results on 1DSMM [37] datasct. Results before the double lines comparisons of local method after 20K iterations. Results under the double lines are obtained from global methods which are incompatible with SGD training.
6.2 Neural Network Optimization
To show that the Iterative Modified Rodrigues Projective Averaging method, MRP (Ours), can be used to learn orienta- tion using neural networks by optimizing the parameters of a simple CNN, specifi- cally ResNet18 [38]. we follow the pro- cedure as in Section 6.1 with some minor changes. Instead of operating directly on a set of rotation parameters, we learn func- tion +. = =f4 from rendered images of the YCB drill [1] model rendered at each of 100 random orientations R, We con- tinue to only supervise each method de- scribed in Section 6.1 using the relative rotations between each image.
Mean
Median
Algorithm 4D PMG [29]
Error (*)
Error (*)
5" Acc (%)
123.84
123.96
0
Quaternion
28.83
21.74
50
MRP (Ours)
3.71
3.73
100
Oracle
1.58
1.56
100
Table 4: Final results for image based rotation estima- tion. Final mean and median angular error (") and percent- age of seeds below 5 after 10K steps over unique envi- ronments of 100 random rotations.
We compare the best performing methods, and, as lower bound, we also train an oracle network, Oracle, with the ground truth rotations, R; and cosine quaternion loss. We use batch size of 32 and the Adam [39] optimizer with learning rate of 10-4 for all experiments. All methods are trained for a maximum of 10K steps, over 8 environments, each with 100 images of randomly generated rotations. We report final mean and median pairwise angular error, and the percentage
7

---

Final Test Anguiar Pairwise Error
Final Test Angudar Pairwise Error
Algurilhm
(*)
Max (")
(in())
Alguriibm
Mean (*)
(*x
(*)
4D PMG [29]
17.39 + 1.14
19,42
16.07
4D PMG [29]
34,57 = 2.31
38.13
31.90
6D PMG [29, 24]
15.20
+: 0.77
16.43
14.44
6D PMG [29, 24]
31.58 + 2.24
35,66
28.42
9D PMG (29,
0.50
15.66
14.18
PMG 25]
31.80
34.87
29.96
10D PMG 129, 261
19.28 1: 7.58
37.76
15.03
10D PMG 129, 261
32.23 + 2.10
36.9%
29.87
Quaternion
4.12
26.57
14.38
Quattemion
31.92 1:
33.61
MRP (Ours)
13,63 +: 0.78
15.08
12.62
MRP (Ours)
29.46
+ 0.66
30.74
28.62
(a)
(b)
Table 7: 3D Object Pose Estimation via Relative Supervision on Pascal3D+ test images for sefa, bicycle. Final mean test angular pairwise error on Pascal3D+ sefa (a), bicycle (b) images after 80K training iterations, over 8 random seeds.
of runs converged below 5" pairwise angular error as 5 Acc. We find that MRP (Ours) is able to converge to a rotational frame consistent with the relative rotations used for supervision relatively quickly, with a significantly lower average-paine pairwise-error than all other relative methods, shown in Table 4.
We perform experiments for generalization to unseen poses and find that curriculum is required (see Appendix D for details). For the generalization experiments, we found that MRP (Ours) achieves mean pairwise angular error 5.19", Quaternion achieves 12.41,, and 4D PMG [29] never converged, with final error of 125.09º.
6.3 3D Object Rotation Estimation via Relative Supervision from Pascal3D+ Images
Experimental Setup. Pascal3D+ [40] is a standard benchmark for categorical 6D object pose esti- mation from real images. Following [29, 25], we discard occluded truncated objects and augment with rendered images from [23]. We report 3D object pose estimation results on two object cat- egories: sofa and bicycle. We compare
method MRP with five baselines: Quaternion, 4D PMG [29], 6D PMG [24], 9D PMG [25] and 10D PMG [26], all of which use ResNet18 [38] backbone to predict the object representation. Each model is supervised using the geodesic between the relative orientation of the predicted absolute orientations, and the relative orientation between the ground truth absolute orientations, for each image pair. We use the same batch size of 20 as in [25, 29], and use Adam [39] with learning rate of 10
Result Analysis. Results for sofa are showed in Table 7()); bicycle are showed in Table 7(b). For sofa category, we find that after 80K training iterations, MRP (Ours) achieves a mean angular pairwise error of 13.63 on the test set, outperforms all other baselines. 10D PMG achieves the worst error out of all methods, with final angular pairwise error of 19.28 For bicycle category, find that after 80K training iterations, MRP (Ours) achieves 29.46 on the test set, outperforms all other baselines. In both the sofa and the bicycle category. find that MRP (Ours) has the fastest convergence speed, in addition to achieving the lowest test angular error. More details can be found in Appendix E.
angular pairwise error of
Conclusion and Limitations
In this paper, show that through the of Modified Rodrigues Parameters, we are able to open the closed manifold of SO(3), improving the convergence behavior of the rotation averaging problem. Additionally, we show that our method, Iterative Modified Rodrigues Projective Aver- aging, is able to outperform the naive application of relative-orientation supervision in both direct parameter optimization and image-based rotations estimation from neural networks. While this parameterization is valuable for learning rotations through relative supervision, it is not without limitations. One of the primary ones is the need for a curriculum to generalizing to unseen relative rotations. Without this, our experiment show that all representations fall into the local optima of predicting constant orientation. Additionally, able to achieve a final error of 5 degrees, which may not be accurate enough for many fine motor tasks. We hope our method allows more systems to convert the relative supervision of relative methods, like ICP, to consistent and accurate absolute poses.
the generalization experiments, we were only
8

---

8 Acknowledgements
We thank Prof Zachary Manchester in particulate for pointing us to Modified Rodriguez Parame- ters. Additionally, we thank Prof Frank Dellaert, Mark Gillespie, Prof Richard Hartley, Valentin Peretroukhin, Prof David Rosen, Prof Denis Zorin, and Prof Keenan Crane for taking time to meet with us and providing enormously helpful feedback and advice. This work is supported by the National Science Foundation under Grant No. I1S-1849154 and by LG Electronics.
References
[1] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. 2018.
[2] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic. Cosypose: Consistent multi niett multi object 6d pose estimation. In European Conference on Computer Vision, pages 574-591. Springer, 2020.
[3] C. Wang, D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. 2019.
[4] Q--Y. Zhou, J. Park, and V. Koltun. Fast global registration. In European conference computer vision, pages 766-782. Springer, 2016.
[5] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. International Symposium on Roboties Research (ISRR) 2019, 2019.
[6] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In ECCV, 2018.
[7] S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In NeurIPS, 2018.
[8] R. Hartley, J. Trumpf, Y. Dai, and H. Li. Rotation averaging. International of computer vision, 103(3):267-344 1006677-5,220 2013.
[9] Dellaert, D. M. Rosen, J. Wu, R. Mahony, and L. Carlone, Shonan rotation averaging: Global optimality by surfing so(p)". In European Conference Computer Vision, pages 292-308. Springer, 2020.
[10] L. Wang and A. Singer. Exact and stable recovery of rotations for robust synchronization. Information and Inference: A Journal of the IMA, 2(2)):45-193, 2013.
[11] M. H. DeGroot. Reaching a consensus. Journal of the American Statistical Association, 69 1345):118-121, 1974.
[12] S. Chatterjee and E. Seneta. Towards consensus: Some convergence theorems on repeated averaging. Journal of Applied Probability, 14(1):89 97, 1977.
[13] A. Olshevsky and J. N. Tsitsiklis. Convergence speed in distributed consensus and averaging. SIAM journal on control and optimization, 48(1):33-55, 2009.
[14] W. J. Russell, D. J. Klein, and J. P. Hespanha. Optimal estimation on the graph cycle space. IEEE Transactions on Signal Processing, 596)):284-2286 2011.
[15] A. Beck and S. Sabach. Weiszfeld's method: Old and new results. Journal of Optimization Theory and Applications, (64(1):1-44 2015.
[16] R. Hartley, K. Aftab, and J. Trumpf. LI rotation averaging using the weiszfeld algorithm. In CVPR 2011, pages 3041-3048 IEEE, 2011.
[17] A. Chatterjee and M. Govindu. Efficient and robust large-scale rotation averaging. In Proceedings of the IEEE International Conference Computer Vision, pages 521-528, 2013.

---

[18] A. Chatterjee and V. M. Govindu. Robust relative rotation averaging. IEEE transactions on pattern analysis and machine intelligence, 40(4):958- 972, 2017.
[19] Y. Shi and G. Lerman. Message passing least squares framework and its application to rotation synchronization. In International Conference Machine Learning, pages 8796-8806. PMLR, 2020.
[20] A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6- dof camera relocalization. In Proceedings of the IEEE international conference on computer vision, pages 3938-2946 2015.
[21] A. Kendall and R. Cipolla. Geometric loss functions for camera pose regression with deep learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[22] T.-T. Do, M. Cai, T. Pham, and I. Reid. Deep-6dpose: Recovering 6d object pose from single rgb image. arXiv preprint arXiv: 10367, 2018.
[23] H. Su, C. Qi. Y. Li, and L. J. Guibas. Render for cmn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE International Conference on Computer Vision, pages 2686-2694, 2015.
[24] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019.
[25] J. Levinson, C. Esteves, K. Chen, N. Snavely, A. Kanazawa, A. Rostamizadeh, and A. Maka- dia. An analysis svd for deep rotation estimation. Advances Neural Information Process- ing Systems, 13:225544-22555 2020.
[26] V. Peretroukhin, M. Giamou, D. M. Rosen, W. N. Greene, N. Roy, and J. Kelly. A smooth representation of belief over so(3) for deep rotation learning with uncertainty. In Proceedings of Robotics: Science and Systems (RSS' 2020.
[27] G. Terzakis, M. Lourakis, and D. Ait-Boudaoud. Modified rodrigues parameters: an efficient representation of orientation in 3d vision and graphics. Journal of Mathematical Imaging and Vision, 60(3):422 442222222222444 2018
[28] Crassidis and F. Markley. Attitude estimation using modified rodrigues parameters. Flight Mechanics/Estimation Theory Symposium, pages 71-86. NASA, 1996.
[29] J. Chen, Y. Yin, T. Birdal, B. Chen, L. Guibas, and H. Wang. Projective manifold gradient layer for deep rotation regression. arXiv preprint arXiv:2110.1 2021.
[30] Z. Teed and J. Deng. Tangent space backpropagation for 3d transformation groups. In Pro- ceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10338-10347, 2021.
[31] R. Brégier. Deep regression on manifolds: 3d rotation case study. In 2021 International Conference 3D Vision (3DV), pages 166-174. IEEE, 2021.
[32] R. Arora. On learning rotations. Advances in neural information processing systems, 22:55-63, 2009.
[33] S. Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 5809):22172223) 2013.
[34] S. Mahendran, H. Ali, and R. Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2174-2182, 2017.
[35] J. H. Manton. A globally convergent numerical algorithm for computing the centre of mass on compact lie groups. In ICARCV 2004 8th Control, Automation, Robotics and Vision Confer- ence, 2004., volume 3, pages 2211-2211 IEEE, 2004.
10

---

[36] T. F. Wiener. Theoretical analysis of gimballess inertial reference equipment using delta- modulated instruments. PhD thesis, Massachusetts Institute of Technology, 1962.
[37] K. Wilson and Snavely. Robust global translations with Idsfm. In Proceedings of the European Conference on Computer Vision (ECCV), 2014.
[38] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[39] D. Kingma and Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[40] Y. Xiang, R. Mottaghi, and Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In JEEE winter conference on applications of computer vision, pages 75-82. IEEE, 2014.


Reviews:
[{"review_id": "s72woUMSXHk", "reviewer": "Reviewer_WRCQ", "summary": "This paper proposed a self-supervised rotation estimation method for object or camera pose estimation. Major contributions are as follows:\n\n(1) Author proposed to use modified Rodrigues parameters to stereographically project the closed manifold of $SO(3)$ to the open manifold of $R^3$, which can speed up the optimization and avoid the local optima.\n\n(2) The proposed optimization could be applied to the Stochastic Gradient Descent of the training for neural networks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* According to the experiment section, the proposed Iterative Modified Rodrigues Averaging could improve the convergence behavior of the rotation averaging problem.\n**Weaknesses:**\n* The novelty statement is unclear. Line 82, \"the unique problems found in the rotation averaging problem\" need more detatils.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The experiment section looks convincing. My major concern is the novelty and contribution compared to [24] and [25] are unclear.\n\nPost-rebuttal:\nThe reply from the author partially resolves my concern, I would keep my rating as \"weak accept\", I think it deserves a poster presentation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Authors should clearly state their novelty and contribution against the prior works.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "RrSaxNJ8AjY", "reviewer": "Reviewer_cxjP", "summary": "The authors propose the use of Modified Rodrigues Parameters (MRP), which maps Rotations in $SO(3)$ to vectors in $\\mathbb{R}^3$, in an incremental manner so that it is suited for learning with gradient descent. They look at self-supervision using relative rotations, where given a set of inputs the relative rotations w.r.t. a subset of the inputs is known, such as by using ICP or camera tracking. This trick of optimizing in the space of MRP circumvents issues that come with $SO(3)$ as it is a closed manifold. The intuition behind the method is shown well in illustrative toy examples making it easy to understand. The effectiveness and efficiency of the approach are shown on two datasets, a toy dataset of uniformly sampled rotations and the 1DSfM dataset on which direct rotation parameter estimates are optimised. Good performance is also shown in an initial experiment of training neural networks on images for relative object pose estimation for synthetic images of a single object.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- One of the first works exploring the use of MRP for Gradient Descent approaches rather than global optimisation.\n- show faster convergence when training direct rotation parameter estimates and neural networks and also overall low error against baseline methods ( less than $5^\\circ$ for pairwise relative poses) \n- provide good illustrative toy examples (with visualisations on their website) allowing the reader/viewer to easily understand the problems with optimising in $SO(3)$ and how these are circumvented in $\\mathbb{R}^3$ and how their iterative approach enables faster convergence.\n- The paper is nicely written and has a good progressive flow to understand the problem that is looked at and the approach presented by the authors.\n\n(Post-Rebuttal Updates)\n- The additional results shown in the supplementary material on both synthetic object poses and on real world datasets of Objects (Pascal3D+) as well outdoor scenes (1DSfM) show the efficacy of the proposed approach compared to other methods using relative supervision. The addition of visual results showing the predicted relative pose predictions helped.\n\n### Weaknesses\n- ~~Since the main focus of the work is on deriving an iterative version of MRP averaging, such that it is better suited to work with Neural Network of Gradient Descent approaches, the use of only simulated images of a single object is a very simplistic evaluation for such a claim. While the relative pose estimation with neural networks shows good initial results, its ability to generalise cannot be fully ascertained using only simulated images of a single object.~~\n - ~~The paper (and supplementary) is lacking in visual qualitative results that can better help a reader to understand the results. Visualising some results for example the intermediate results of training the neural networks or the alignments of the 1DSfM data could help one better understand the approach.~~\n\n(Post-Rebuttal Updates)\n- I did not understand the need for Section F in the appendix. This essentially shows the advantage of just MRP and not the proposed approach. I would imagine that the use of Iterative MRP as a representation for absolute orientation could also be useful for the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors propose the use of MRP for training gradient descent approaches. While the idea is simplistic, they are the first to look at this novel use of MRP in a relevant use case in robot learning. The illustration of existing issues in optimising in $SO(3)$ manifolds is well illustrated, motivating the idea of using MRP to transform the space of the loss landscape. The idea, although incremental, is novel enough to be worth considering as part of the conference. However, the simplistic nature of the evaluations, especially in the context of neural networks, prevents me from completely advocating for a strong acceptance, especially when the goal is to train neural networks well for predicting rotations. ~~The paper could benefit from a stronger set of experiments on a more diverse dataset rather than on a single object (from an already diverse dataset), especially when considering training neural networks. For example, by training existing approaches like those presented in the Related Works section using the MRP representation, I would imagine that other than just relative rotations, even learning absolute rotations could benefit by using MRP representations during training. While I would like to give more than just a weak accept, since the only next level is a strong accept without anything in between, my final recommendation is a weak accept.~~\n\n### Post-Rebuttal Updates\nThe updated results show a good efficacy of the proposed approach on a range of different datasets when compared with different relative rotation estimation approaches. I did not fully understand Section F in the supplementary as it doesn't showcase the usefulness of the proposed approach but rather shows the power of using MRP. Nevertheless, the iterative extension of MRP making it suitable for use in gradient descent settings and the promising results warrant an acceptance.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- While the use of optimisation for predicting relative transforms is an old one in computer vision, there have been approaches that explore the use of such optimization (like non-linear least squares) in a differentiable way within neural network frameworks trained with gradient descent (BA-Net [1], or Deep Direct VO [2] who use LM as part of the Network optimisation). I would imagine that such an approach could work in such a case of rotation averaging when considering a minibatch of samples with their neighbourhood, given that such optimisation approaches are known for handling not just $SO(3)$ manifolds but $SE(3)$ as well. \n\n[1] [Tang, C. and Tan, P., 2018, September. BA-Net: Dense Bundle Adjustment Networks. In International Conference on Learning Representations (ICLR).](https://arxiv.org/abs/1806.04807)\n\n[2] [Wang, C., Buenaposada, J.M., Zhu, R. and Lucey, S., 2018. Learning depth from monocular videos using direct methods. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).](https://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Learning_Depth_From_CVPR_2018_paper.pdf)\n\n- MRP is not a contribution of the paper but is something behind the base of the approach, maybe calling the approach iMRPAvg or something to emphasise the fact that the proposed approach is an iterative Version encapsulating MRP would be better. \n- ~~In Table 4 what is Acc $5^\\circ$? Is it the same as \"Avg Pairwise Angular Error < $5^\\circ$ \" in Table 1? This needs to be clarified in the text.~~\n\n(Post-Rebuttal Updates)\n- As mentioned above, Section F doesn't supplement the proposed approach but rather shows the advantage of MRP. It would have been better to show the absolute orientations of using the Iterative MRP approach proposed by the authors.\n- The formatting of Tables S1 and S2 in the supplementary material need to be fixed.\n- The XYZ axes that are overlayed on the images (Figures 3, S6-S8) can be made thicker as they are very thin to see properly. Additionally, Figures S8 and S7 seem to have an overlay on them which makes it difficult to see the images clearly.\n- I would suggest the authors to export the graphs in vector graphics (SVG) or PDF formats in order to prevent pixelation of the images."}}, {"review_id": "FBmw3SwULDZ", "reviewer": "Reviewer_ziM5", "summary": "The paper analyzes the problem of learning orientation via self-supervised learning, especially via relative orientation data. Specifically, the paper proposes to use a new representation via Modified Rodrigues Parameters (MRP) and claim that this provides more stable learning convergence when compared to previous methods which use different rotational spaces like SO(3).\n\nThe paper then analyzes MRP experimentally on experiments including uniformly sampled rotations, the 1DSfM dataset, and an image-based neural network optimization problem. The results show that MRP improves results upon the baselines compared to previous methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The authors are clear in the theoretical explanations of MRP and in the visuals and examples\n+ The authors show clear benefit of using MRP for estimating orientation via relative rotations compared to previous methods, on the evaluated datasets\n+ The authors run experiments across multiple datasets and include an end-to-end example using a deep neural network\n+ The experiments show that MRP converges faster than previous methods, supporting the authors’ claims\n+ The method is able to be used as a relatively straightforward, drop-in replacement for other rotation representations for the self-supervised relative rotation problem, and this ease may expand its impact\n\nWeaknesses\n\n- The authors show that MRP outperforms prior methods in evaluation results, but some charts like in Fig. 2. (left) show that some other methods may not have fully converged. It would be good to understand whether MRP would converge to a better performance compared to other methods, after they have all converged.\n- The prior work compared against [21,22,28] use datasets/benchmarks, but none are covered in this paper, e.g. [28: modelnet40 point clouds / modelnet10 images] [21: ​​CMU Motion Capture Database] [22: pascal3D+] and instead novel benchmark sets are used\n- How does MRP and the other relative rotation methods compare to using direct supervised learning? How far does MRP bridge the gap to supervised learning techniques? e.g. Should practitioners considering supervised methods here instead use MRP?\n- The neural network optimization benchmark is evaluated on only one object, the YCB drill. While the method performs much better than prior work for this model, it would’ve been more convincing that this method is very compatible with neural network based optimization if an entire set of objects were tested. This may help answer some other questions, e.g.\n  - Would this method work better for image-based optimization if the object has some symmetries?\n  - How may a practitioner envision using this algorithm in a larger end-to-end neural network based learning problem? How practical would unseen relative rotations or learning curriculums be in this end-to-end case?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Taken as a whole, this work makes a clear case for use of MRP via theoretical explanations and in isolated experimental results for use in relative orientation self-supervised learning. However, the larger impact of the work is tied to the use and feasibility of self-supervised approaches for orientation learning compared to supervised ones, which is not examined via experiments. This limits the conclusion of the work - it is not clear whether practitioners should switch to using MRP, which requires using self-supervised methods as a whole. Noted, the authors make clear mitigations or concerns of self-supervised methods like curriculum requirements.\n\nLimiting scope to improvement of rotation representation for self-supervised methods, there also lacks direct comparison to benchmarks used in prior work and a comprehensive picture of neural-network based performance. Thus, while improvement looks promising given the presented results, it would be hard to draw a comprehensive, apples-to-apples conclusion.\n\n\nEdit: post review, the authors have sufficiently addressed my concerns and it is clear that the work is promising for usage/consideration by practitioners to improve neural network rotation prediction.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Some summary of issues that could be addressed from above:\n\n- (Fig 2 left)  Are all methods fully converged? Would running more steps improve some methods?\n- A comparison on some benchmark used in prior work, e.g. [28: modelnet40 point clouds / modelnet10 images] [21: ​​CMU Motion Capture Database] [22: pascal3D+]\n- How does MRP and the other relative rotation methods compare to using direct supervised learning? How far does MRP bridge the gap to supervised learning techniques? e.g. Should practitioners considering supervised methods here instead use MRP?\n- The neural network optimization benchmark is evaluated on only one object, the YCB drill. While the method performs much better than prior work for this model, it would’ve been more convincing that this method is very compatible with neural network based optimization if an entire set of objects were tested. This may help answer some other questions, e.g.\n  - Would this method work better for image-based optimization if the object has some symmetries?\n  - How may a practitioner envision using this algorithm in a larger end-to-end neural network based learning problem? How practical would unseen relative rotations or learning curriculums be in this end-to-end case?\n"}}, {"review_id": "0Re2i1ix7n4", "reviewer": "Reviewer_2N3Q", "summary": "The paper uses modified Rodrigues parameters for estimating absolute rotations from relative rotations, which can be used within \"self-supervised\" training of neural networks. In this training regime, it is assumed that relative orientations between neighboring poses are given, for example, computed from a local alignment method like ICP, and that a neural network should be trained by using the relative poses as supervision. The paper shows that the proposed scheme can obtain better results than baseline methods for synthetic data and 1DSfM data when the rotations are optimized directly. Additionally, the paper shows a synthetic example where the rotations are not inferred directly, but a neural network is trained to infer the rotations from rendered images.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper proposes an interesting method for rotation learning that is based on modified Rodrigues parameters. It nicely combines current deep learning with traditional works on rotation parametrization.\n- The paper is overall well-written and the content is well-presented.\n\n\n**Weaknesses**:\n\n- The paper does not show results for an actual learning problem. The shown examples either have no learning (Sec. 6.1) or only synthetic data with overfitting, i.e. no learning as well (Sec 6.2).\n- The paper does not show results on recognized benchmarks. Sec. 6.1 uses the 1DSfM dataset but does not compare with global methods, which are SOTA there. The other experiments use data generated by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a seemingly novel and interesting idea, however, an evaluation on actual learning tasks and real-world data is missing. This makes it impossible to judge the benefit of the proposed method for robot learning. If the paper would show extensive experiments on real-world learning tasks I would be willing to increase my score.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The paper would be improved significantly if the authors can provide extensive evaluations on real-world learning tasks wrt. established SOTA methods on established benchmarks. However, this might be too much for a rebuttal and should be done when potentially re-submitting the paper."}}]

Author rebuttal:

Decision: Accept (poster)