Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction
Authors: David Klee
Ondrej Biza
Robert Platt
Robin Walters
Venue: ICLR 2023 notable top 5% (2023)
Keywords: equivariance
sample efficiency
pose detection
symmetry
SO(3)

Abstract: Predicting the pose of objects from a single image is an important but difficult computer vision problem. Methods that predict a single point estimate do not predict the pose of objects with symmetries well and cannot represent uncertainty. Alternatively, some works predict a distribution over orientations in $\mathrm{SO}(3)$. However, training such models can be computation- and sample-inefficient. Instead, we propose a novel mapping of features from the image domain to the 3D rotation manifold. Our method then leverages $\mathrm{SO}(3)$ equivariant layers, which are more sample efficient, and outputs a distribution over rotations that can be sampled at arbitrary resolution. We demonstrate the effectiveness of our method at object orientation prediction, and achieve state-of-the-art performance on the popular PASCAL3D+ dataset. Moreover, we show that our method can model complex object symmetries, without any modifications to the parameters or loss function.  Code is available at \url{https://dmklee.github.io/image2sphere}.

Published as a conference paper at ICLR 2023
IMAGE TO SPHERE: LEARNING EQUIVARIANT FEATURES FOR EFFICIENT POSE PREDICTION
David M. Klee, Ondrej Biza, Robert Platt, Robin Walters
Northeastern University (klee.d, bi biza.o, r.platt, r.walteersjeeorrhhessttee....e
ABSTRACT
Predicting the pose of objects from a single image is an important but difficult computer vision problem. Methods that predict single point estimate do not predict the pose of objects with symmetries well and cannot represent uncertainty. Alternatively, some works predict a distribution over orientations in SO(3). How- ever, training such models can be computation- and we propose a novel mapping of features from the image domain the 3D rotation manifold. Our method then leverages SO(3) equivariant layers, which are more sample efficient, and outputs a distribution over rotations that can be sampled at arbitrary resolution. We demonstrate the effectiveness of our method object orientation prediction, and achieve state-of-the-art performance on the popular PASCAL3D+ dataset. Moreover, show that method can model complex object symmetries, without any modifications to the parameters or loss function. Code is available at htttt::t////// //dpss:/ddkk/ lee. github. io/ mage2sphere
Instead,
INTRODUCTION
Determining the pose of an object from an image is a challenging problem with important applications in artificial reality, robotics, and autonomous vehicles. Traditionally, pose estimation has been approached as point regression problem, minimizing the error to single ground truth 3D rotation. In this way, object symmetries are manually disambiguated using domain knowledge (Xiang et al.) 2018) and uncertainty is not accounted for. This approach to pose estimation cannot scale to the open-world setting where we wish to reason about uncertainty from sensor noise or occlusions and model novel objects with unknown symmetries.
Recent work has instead attempted to learn a distribution over poses. Single rotation labels can be modeled as random samples from the distribution over object symmetries, which removes the need for injecting domain knowledge. For instance, a table with front-back symmetry presents a challenge for single pose regression methods, but can be effectively modeled with a bimodal distribution. The drawback to learning distributions over the large space of 3D rotations is that it requires lots of data, especially when modeling hundreds of instances across multiple object categories.
This poor data efficiency can be improved by constraining the weights to encode symmetries present in the problem (Cohen Welling 2016). The pose prediction problem exhibits 3D rotational symmetry, e.g. the S3(3) abstract group. That is, if we change the canonical reference frame of an object, the predictions of our model should transform correspondingly. For certain input modalities, such as point clouds or 3D camera images, the symmetry group acts directly on the input data via 3D rotation matrices. Thus, many networks exploit the symmetry with end-to-end SO(3) equivariance to achieve sample efficient pose estimation. However, achieving 3D rotation equivariance in network trained on 2D images is less explored.
Thus, we present Image2Sphere, I2S, novel method that learns represent distributions over 3D rotations. Features extracted by convolutional network are projected from image space onto the half 2-sphere. Then, spherical convolution is performed on the features with a learned filter over the entire 2-sphere, resulting in a signal that equivariant to 3D rotations. A final SO(3) group convolution operation produces probability distribution over SO(3) parameterized in the Fourier domain. Our method can be trained to accurately predict object orientation and correctly express ambiguous orientations for objects with symmetries not specified at training time.
features to

---

Published as a conference paper at ICLR 2023
I2S achieves -fate-of-the-art performance the PASCAL3D+ pose estimation dataset, and outper- forms all baselines on the ModeINet10-SOO3 dataset. We demonstrate that our proposed architecture for learning (3) equivariant features from images empirically outperforms variety of sensible, alternative approaches. In addition, we use the diagnostic SYMSOL datasets to show that our ap- proach is more expressive than methods using parametric families of multi-modal distributions representing complex object symmetries.
Contributions:
We propose novel hybrid architecture that uses non-equivariant layers to learn SO(3)- equivariant features which are further processed by equivariant layers.
Our method uses the Fourier basis of 50(3) to more efficiently represent detailed distribu- tions over pose than other methods.
We empirically demonstrate our method able to describe ambiguities in pose due to partial observability or object symmmetry unlike point estimate methods.
I2S achieves SOTA performance on PASCAL3D+, challenging pose estimation benchmark using real-worke images.
RELATED WORK
6D Pose Estimation Predicting the 6D pose (e.g. 3D position and orientation) of objects in an image has important applications in fields like robotics (Tremblay et al., (018), autonomous vehicles (Geiger et al., 20122), and microscopy (Levy et al.! 2022). Most popular methods use deep convolutional networks, which are robust to occlusions and can handle multi-object scenes with segmentation module (He al. 2017). Convolutional networks have been trained using variety of output formulations Xiang et al. (2018) regresses the 3D bounding box of the object in pixel space, while He et al. (2020) predicts 3D keypoints on the object with which the pose can be extracted. Another line of work (Wang et al. 2019; Li et al. 2019: Zakharov et al.) 2019) outputs a dense representation of the object's 's coordinate space. Most of these methods are benchmarked on datasets with limited number of object instances (Hinterstoisser et al. 2011 I Xiang et al. 2018). In contrast, our method is evaluated on datasets that have hundreds of object instances or novel instances in the test set. Moreover, our method makes minimal assumptions about the labels, requiring only 3D rotation matrix per image regardless of underlying object symmetry.
Rotation Equivariance Symmetries present in data can be preserved using equivariant neural networks to improve performance and sample efficiency. For the symmetry group of 3D rotations, SO(3), number of equivariant models have been proposed. [Cen et al. (2021) and Fuchs al. (2020) introduce networks to process point cloud data with equivariance to the discrete icosahedral group and continuous SO(3) group, respectively. Esteves et al. (2019b) combines images from structured viewpoints and then performs discrete group convolution to classify shapes. Cohen et al. (2018a) introduces spherical convolution to process signals that live on the sphere, such as images from 3D cameras. However, these methods restricted to cases where the SO(3) group acts the input space. which prevents their use on 2D images. Falorsi et al. (2018) and Park et al. (2022) extract 3D rotational equivariant features from images to model object orientation, but were limited to simplistic datasets with a single object. Similar to our work, Esteves et al. (2019a) 9a) learns SO(3) equivariant embeddings from image input for object pose prediction; however, they loss to replicate the embeddings of spherical convolutional network pretrained on 3D images. In contrast, our method incorporates novel architecture for achieving SO(3) equivariance from image inputs that can be trained end-to-end on the challenging pose prediction tasks.
supervised
Uncertainty over SO(3) Due to object symmetry or occlusion, there may be set of equivalent rotations that result in the same object appearance, which makes pose prediction challenging. Most early works into object pose prediction have avoided this issue by either breaking the symmetry when labelling the data (Xiang al. 2014) or applying loss functions to handle to known symmetries (Xiang et al. 2018 Wang et al. 2019). However, this approach requires knowing what symmetries present the data, and does not work for objects that have ambiguous orientations due to occlusion (c.g. coffee mug when the handle is not visible). Several works have proposed models to reason about orientation uncertainty by predicting the parameters of von Mises (Prokudin et al.
2

---

Published as a conference paper at ICLR 2023
Image
Distribution over SO13)
Feature Map
S
Filter
SO(3) Signal
SO:3) Filler
RosNet
*
Figure I: Illustration of our proposed model, Image2Sphere (12S). First, output image features of a pre-trained ResNet are orthographically projected to the sphere. We convolve the features V with learned filter on S² to generate a signal on SO(3) (represented by vector field on the sphere). A final SO(3) group convolution is performed to produce detailed distribution over 3D rotations, allowing I2S to learn object symmetries and represent uncertainty during pose estimation.
2018), Fisher (Mohlin et al. 2020), and Bingham GGlittchhensii et al. 2019 Deng et al., 2022) distributions. However, mulii-mod modal distributions required for modeling complex object symmetries which can be difficult to train. Recent work by Murphy et al. (202T) leverages an implicit model to produce a non non-parametric distribution over SO(3) that can model objects with large symmetry groups. Our method also generates distribution over S3(3) that can model symmetric objects, but uses 50(3) layers to achieve higher accuracy and sample efficiency.
3 BACKGROUND
3.1 EQUIVARIANCE
Equivariance formalizes what it means for map f to be symmetry preserving. Let G be symmetry group, that is, set of transformations that preserves some structure. Assume G acts on spaces X' and y via Ty and Ty respectively. For all g E G. a map f: X' + y is equivariant to if
=
(1)
That is, if the input of f is transformed by the output will be transformed in a corresponding way. Invariance is a special case of equivariance when Tg is the identity map (i.e. applying group transformations to the input of an invariant mapping does not affect the output).
3.2 GROUP CONVOLUTION OVER HOMOGENEOUS SPACES
The group convolution operation is a linear equivariant mapping which can be equipped with trainable parameters and used to build equivariant neural networks (Cohen Welling 2016). Convolution is performed by computing the dot product with a filter as it is shifted across the input signal. In standard 2D convolution, the shifting corresponds to a translation pixel space. Group convolution (Cohen & Welling, 2016) generalizes this idea to arbitrary symmetry groups, with the filter transformed by elements of the group. Let be a group and X' be a homogeneous space, i.c. a space on which G acts transitively, for example, X' G or X G/H for subgroup H. We compute the group convolution between two functions, f. fb X + R as follows:
[ * (g)
Σ -()) (T, TEX
Note that the output of the convolution operation defined for each group element g E G, while the inputs, f and tb are defined over homogeneous space of G. By parameterizing either ff or e using trainable weights, group convolution may be used as a layer in an equivariant model.
3.3 SQQ3R-EQUVARRIACC
For reasoning about physical objects, such as pose detection or part segmentation, desirable to learn functions that are equivariant to the group S3)3) of 3D rotations. In order to build SO(3)- equivariant neural networks using group convolution, is necessary to efficiently parameterize the space of signals {f: X' R} over the homogenous spaces X' 5 or X' SO(3). Cohen et al. (2018b) give an effective and compact solution in terms of the truncated Fourier transform. Analogous to the Fourier basis over the circle, it is possible to give Fourier basis for signals defined over 2
3

---

Published as a conference paper at ICLR 2023
in terms of the spherical harmonies Y and for signals over (3)3) in terms of Wigner D D-matrix coefficients D Writing f: SO(3) R in terms of the D and then truncating to certain frequency S L we obtain an approximate representation f(g) Ei Em=o 3+ C Dmm (g).
S3(3) group convolution (2) can be efficiently computed in the Fourier domain using the convolution theorem Namely, the convolution of two functions is calculated as the element-wise product of the functions in the Fourier domain. For functions over SO(3), the Fourier domain is described by the Wigner D-matrix coefficients, corresponding to a block diagonal matrix with a (21 + 1) x (21 + 1) block for each / (Knapp) 1996). A functions over S in the Fourier domain is described by vector of coefficients of spherical harmonics and convolution is performed using an outer product (Cohen) ee al. 2018a). This generates the same block diagonal matrix output as SO(3) convolution (e.g. both 5 and (3)3) convolution generate signals that live on (3)) See Geiger et al. (2022) for efficient implementation of Fourier and inverse Fourier transforms of S and (3)3) signals.
3.4 SO(3) DISTRIBUTIONS WITH THE FOURIER BASIS
The output of the above group convolution is signal f: SO(3) + R defined as a linear combination of Wigner D-matrices. While this signal is useful for efficient group convolution, it cannot be directly normalized to produce a probability distribution. Instead, the signal can be queried at discrete points in SO(3) and then normalized using softmax. To reduce errors introduced by the discretization, points should be taken from an equivolumetric grid. Equivolumetric grids over SO (3) can be generated using an extension of the HEALPix method developed by Yershova et al. (2010). The HEALPix method (Girski et al. 2005) starts with 12 'pixels' that cover the sphere, and recursively divides each pixel into four sections. The latitude and longitude of each pixel describe two Euler angles of rotation. The extension to SO(3) specifies the final Euler angle in a way that the grid is equally-spaced in S3(3). The resolution of the final grid can be controlled by the number of recursions.
METHOD
Our method, Image2Sphere (I2S), is designed to predict object pose from images under minimal assumptions. Much previous work assumes objects with either no symmetry or known symmetry, and thus it is sufficient to output single point estimate of the pose. Our method, in contrast, outputs a distribution over poses, which allows us to represent uncertainty due to partial observability and the inherent ambiguity in pose resulting from object symmetries. Recent work using distribution learning modal distributions and high data requirements. 12S
has suffered from difficulty in training cireumvents these challenges by reasoning about uncertainty in the Fourier basis of SO()), which is simple to train over and allows us to leverage equivariant layers for better data efficiency.
The 12S network consists of encoder, a projection step, and 2-layer spherical pose predictor (see Figure D.. 12S generates SO(3) equivariant features from traditional convolutional network encoders. Our method then maps features from image space to the 2-sphere using an orthographic projection operation. Then, we perform two spherical convolution operations. Importantly, the output of the spherical convolution is signal over the Fourier basis of (3))), which can be queried in the spatial domain to create highly expressive distributions over the space of 3D rotations.
4.1 MAPPING FROM IMAGE TO SPHERE
We use orthographic projection map the output image feature map f: R.2 R. from the ResNet to a spherical signal v: 5 + R². Orthographic projection P: 52 + R detined P(r, y, 2) = (x, y) maps a hemisphere onto the unit disk. By positioning the feature map f around the unit disk before projecting, we get localized signal (zx) = f(((z)) supported over one hemisphere. The advantage of this method that it preserves the spatial information of features in the original image. Practically, to compute V, HEALPix grid is generated over a hemisphere {x,} C. 52 and mapped to positions { (*(+)) in the image feature map. The value of (ax))) interpolated from the value of f at pixels near position P(..). This process is illustrated in Figure 2
"eee Cohen et al. 2018al for explanation and visuals describing efficient SO(3) group convolution.
4

---

Published as a conference paper at ICLR 2023
We then move V the frequency domain using fast Fourier transform to express v as linear combination of spherical harmonics. Operating in the frequency domain allows us to efficiently convolve continuous signals over the sphere (S²) and the group of 3D rotations (SO(3)). We truncate
the Fourier series for V at frequency as (xx Ei 2k-o (x)) This truncation may cause sampling error if the spatial signal has high frequency. We mitigate this error by: (1) tapering the magnitude of the projected features toward the edge of the image to avoid a discontinuity on the 2-sphere, and (2) performing dropout of the HEALPix grid such that random subset of grid points is used for each projection.
S signal (N))
grid points
fenture map
Figure 2: Projection of features from image space to the 2-sphere. (a) orthographic projection links pixels in image space points on the sphere using vectors orthogonal the image plane; (b) dense feature map, visualized as an RGB image, is mapped to a signal over 5 by orthographically projecting onto all visible grid points, resulting in a signal that non-zero over half the 2-sphere.
4.2 LEARNING SO(3) EQUIVARIANT FEATURES
Once the learned features are projected to the sphere, we use operations that preserve the 3D symmetry of the representation. First, the spherical signal V/ processed with an S²~equivariant group convolution operation. Unlike traditional convolutional layers that use locally supported filters, we elect to use a filter with global support. Intuitively, we view this operation similarly to template matching where the projected hemisphere signal Vi is the template and we are looking for the rotation transformation that results in the best match with the filter. Conveniently, using a globally supported filter gives a global receptive field after one layer, allowing for shallower network. This is important since each spherical convolution operation is fairly computation and memory intensive due to the high band limit L necessary model potentially complex distributions the output.
After the Se-convolution, we perform one -eeuiiqquiaar group convolution before outputting the probability density. Using locally supported filter, this operation serves refinement step and results in slightly higher prediction accuracy (see comparison in Appendix (C2). Following Spherical CNN (Cohen et al., (0T8a), we apply non-linearities between convolution layers by mapping the signal to the spatial domain, applying a ReL.U, and then mapping back to the Fourier domain.
4.3 Loss FUNCTION
The output of the spherical convolution operation is signal f: SO(3) -> R2o that indicates the probability that the image corresponds to particular object orientation. This output is queried using an equivolumetric grid over SO(3) then normalized with a softmax operation to produce a categorical distribution, which can be optimized using a cross-entropy loss. We find that this loss is effective even when training on images with ambiguous orientation or symmetric objects. For these cases, the model learns to predict high probabilities for the set of equivalently likely orientations. Additionally, we find that the model can accurately predict likelihoods for grids of higher resolution than was trained (e.g. train with grid resolution of degrees and test with resolution of 2.5 degrees).
5

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
5.1 DATASETS
To demonstrate the strengths of our method, we evaluate it on several challenging object orientation estimation datasets. Additional details can be found in Appendix B.2
2019), is composed of rendered images of synthetic, The first dataset, ModeINet10-SOC3 (Liao et al. untextured objects from ModelNet10 (Wu et al. 2013). The dataset includes 4,899 object instances over 10 categories, with novel camera viewpoints in the test set. Each image is labelled with single 3D rotation matrix, even though some categories, such as desks and bathtubs, can have an ambiguous pose due symmetry. For this reason, the dataset presents challenge to methods that cannot reason about uncertainty over orientation.
Next, PASCAL3D+, (Xiang al. 2014), is a popular benchmark for pose estimation that includes real images of objects from 12 categories. The dataset labels symmetric object categories a consistent manner (e.g. bottles are symmetric about the z-axis, so label zero rotation about the z-axis), which simplifies the task. To improve accuracy, we follow the common practice of augmenting the training data with synthetic renderings from [Su al. (2015). Nevertheless, PASCAL3D+ still serves as a challenging benchmark due to the high variability of natural textures and presence of novel instances in the test set.
Lastly, the SYMSOL dataset was recently introduced by Murphy et al. (021)) to evaluate the expressivity of methods that model distributions over 3D rotations. It includes synthetic renderings of objects split into two groups: geometric objects like the tetrahedron or cylinder with complex symmetries (SYMSOL 1), and simple objects with single identifying feature such that the pose is ambiguous when the feature is occluded (SYMSOL II). The dataset provides the full set of equivalent rotation labels for each image so methods can be evaluated on how well they capture the distribution. Note that Murphy et al. (2021) generates results using 100k renderings per shape, which report as a baseline, but the publicly released dataset that we train on only includes 50k renderings per shape.
5.2 EVALUATION METRICS
The goal of pose prediction is to minimize the angular error between the predicted 3D rotation and the ground truth 3D rotation. Two commonly used metrics are the median rotation error (MedErr) and the accuracy within a rotation error threshold (e.g. Ace@ 15 is the fraction of predictions with 15 degrees or less rotation error). However, these metrics assume that there exists single, ground truth 3D rotation, which is not valid for symmetric objects or images with pose ambiguity. For ModeINet10-SO(3) and PASCAL3D+, only single rotation is provided so these metrics must be used. However, when the full set of equivalent rotation labels are provided, like with SYMSOL, a more informative measure is the average log likelihood, computed as the expected log likelihood that the model, pa. assigns to rotations sampled from the distribution of cquivalent rotations Per: Ex-per [log po R]])). Achieving high log likelihood requires modelling all symmetries of an object.
5.3 NETWORK AND TRAINING DETAILS
I2S uses a residual network (He et al. 2016) with weights pretrained on ImageNet (Deng et al. 2009 extract dense feature maps from 2D images. We use ResNet50 backbone for ModelNet (0-S and SYMSOL, and ResNet101 for PASCAL3D+. The orthographic projection uses a HEALPix grid with recursion level of 2, out of which 20 points are randomly selected during each forward pass. We parameterize the learned 52 filter in the Fourier domain, e.g. learn weights for each spherical harmonic. The filter the SO(3) convolutional layer locally supported over rotations up to 22.5 degrees in magnitude. We query the signal in the spatial domain using SO(3) HEALPix grids with 36k points (7.5 degree spacing) during training and 2.4M (1.875 degree spacing) during evaluation. We the same maximum frequency (L 6) all datasets. Additional details on the architecture can be found in Appendix B.I
I2S is instantiated with PoTch and the e e3nn library (Geiger al. 2022). It is trained using SGD with Nesterov momentum of 0.9 for 40 epochs using a batch size of 64. The learning rate starts at 0.001 and decays by factor of 0.1 every 15 epochs.
6

---

Published as a conference paper at ICLR 2023
5.4 BASELINES
We compare our method against competitive baselines including regression methods and distribution learning methods. [Zhou et al. (2019) and Brégier (202T) predict valid 3D rotation matrices using differentiable orthonormalization processes, Gram Schmidt and Procrustes, respectively, that pre- clude discontinuities on the rotation manifold. Liao et al. (2019) and Mahendran et al. (2018) use unique cussiffcation-regresiioo losses to predict rotation. Prokudin et al. (2018) represents rotation uncertainty with a mixture of von Mises distributions over each Euler angle, while Mohlin et al. (2020) predicts the parameters for matrix Fisher distribution. Giilitchhensii al. (2019) and Deng et al. (2022) both predict multi-modal Bingham distributions. Lastly, Murphy et al. (2021) trains an implicit model generate non-parametric distribution over 3D rotations. All baselines the same sized. pretrained ResNet encoders for each experiment. Results are from the original papers when available.
5.5 MODELNET 10-SO3
We report performance ModelNet10-SO(3) in Table 1 Our method outperforms all baselines on median error and accuracy at 15 degrees. Moreover, 12S achieves the lowest error on nine out of ten categories, and reports less than 5 degrees of median error eight out of ten categories (full breakdown in Appendix A.2). Because the dataset includes objects with symmetries, the average median error is heavily influenced by high errors categories with ambiguous pose. For instance, all methods get at least 90 degree rotation error on bathtubs, since is hard to tell the front from the back. Thus, regression methods can get stuck between ambiguous object symmetries. method generates distribution which can capture multiple symmetry modes, even if the correct one cannot be identified. We believe that 12S is more accurate than other distributional methods because its equivariant layers encode the symmetry present in the pose estimation problem.
contrast, our
Table 1:
Comparison of pose estimation performance on ModeINet10-SOC3 dataset. Data is averaged over all ten object categories. For I2S, we report mean and standard deviation over five random seeds.
Acc
301
MedErr]
Zhou et 2019
0,251
0.504
41.1
Bresser 2021
0.257
0.515
39.9
2m19
0.357
0,583
36.5
Deng al
0.562
0.694
32.6
Prokudin
Mohlin al
2r21
0.693
0.757
17.1
Murply ct
0.719
0.735
21.5
125
0.728
0.002
0.736#00002
15.7#
5.6 PASCAL3D+
PASCAL33+ is challenging benchmark for pose detection that requires robustness to real textures and generalization to novel object instances. We report the median enror of our method and competitive baselines in Table 2 Importantly, our method achieves the lowest median error averaged across all 12 categories. Note that the symmetries are broken consistently in the labels, outperforming methods that regress to a single pose, e.g. Mahendran et al. (2018), ), is particularly impressive. We hypothesize that the S0(3) equivariant layers within model provide stronger generalization capabilities. Example predictions of our method show that it captures reasonable uncertainty about object pose when making predictions (Figure
Example predictions of I2S on PASCAL3D+. The distribution visualizations show that our method captures pose uncertainty, e.g. is unclear a long boat is approaching or a shorter is headed to the right, creating ambiguity about rotation axis (middle).
Figure 3:
7

---

Published as a conference paper at ICLR 2023
org.
plane
bicycle
boat
botric
buz
car
chuit
tuble
inbile
vofa
Zhea et
2019
19.2
24.7
18.9
54.2
11.3
8.4
9.5
19.4
14.9
22.5
17.2
11.4
17.5
30.0
27.5
22.6
49.2
11.9
8.5
9.9
16.8
27.9
21.7
12.6
10.2
20.6
2019
13.0
164
29.1
688
11.6
17.1
Manhin 20220
11.5
10.1
156
24.3
7,8
533
13.5
12.5
12.9
13.8
11.7
Pratioin mi:1 AUM
12.2
9.7
15.5
45.6
5,4
2.9
4,5
13.1
12.6
11.8
9.1
431
12.0
Marx
13.6
13.8
17.7
21.3
12.
9.1
14.8
14.7
13.7
R.7
15.4
Mimpay 3020
10.3
12.9
23,4
88
5.3
10.0
7.3
13.6
9.5
125 (007S)
9.8
9.2
12.7
21.7
7.4
3.3
4.9
9.5
9.3
11.5
10.5
7.3
10.6
2:44
2004
2:07
213
20.7
10.1
20.1
20.8
#34
2:0.8
208
2005
20.6
Table 2: Median rotation error (") PASCAL3D+. First column shows average median enror over all twelve classes. For some baselines, we report the corrected results by Murphy et al. (202T) (see Appendix B.2 for details), For I2S, we report mean and standard deviation over six runs.
5.7 SYMSOL
One of the strengths of method is the ability represent distributions over 3D rotations. As shown in the previous section, this formulation is beneficial when training on symmetric objects. In this section, we quantitatively evaluate the ability model uncertainty two settings: SYMSOL I which includes simple geometric objects with complex symmetries and SYMSOL II which has objects marked with single identifier such that self-occlusion creates pose ambiguity. Because most images correspond to set of equivalent rotation labels, we measure performance using average log likelihood, reported in Table 3
The results show that method effectively represents complex distributions over SO(3). We achieve higher log likelihood on all SYMSOL shapes than methods that map to specific small family of distributions such as Mises Bingham. This highlights the advantage of parametrizing uncertainty in the Fourier basis of SO(3). which a simpler approach that avoids training multi- modal distributions. We note that Murphy et al. (202T) outperforms method when trained 100k images per object; however, our method is better when trained on only 10k images per object. This demonstrates important distinction between the approaches: method explicitly encodes the 3D rotation symmetry of the problem in the spherical convolutions, whereas Murphy et al. (2021) must learn the symmetry from data. We argue that sample efficiency is important for real world applications since it is not practical to collect 100k images of an single object to model its symmetry. We want to highlight that we use the same maximum frequency for all experiments in this work which shows that I2S can be deployed without knowing if object symmetries are present in the task.
Table Average log likelihood on SYMSOL datasets. SYMSOL includes objects with complex symmetries, while SYMSOL II includes objects whose poses can be ambiguous under self-occlusion. The highest likelihood each column bold, second-best is underlined.
num. training
SYMSOLI
SYMSOLI II
images
avg.
cube
avg.
sphx
eylO
feiX
Deng et
2022
0.16
-0.95
0.27
-4.44
-2.45
2.57
1.12
2,99
3.61
Gilitschenski elal 2019
0.43
3.84
0.88
-2.29
-2.29
-2.29
3.70
3.32
4.88
2.90
100k
Prkkndia et
-1.87
-3.34
-1.28
-1.86
-0.50
-2.39
0.48
-4.19
4.16
1.48
Murpty
4.10
4.45
4.26
5.70
4.81
1.28
7.57
7.30
6.91
8.49
125 [murs]
3.41
3.75
3.10
4.78
3.27
2.15
4.84
3.74
5.18
5.61
Murphy al.
2021
-7.94
-1.51
-2.92
6.990
10.04
-18 34
-0.73
22.51
2.02
-1.70
10k
125 (ours)
2.94
3.61
3.12
3.87
3.84
5.8
COMPARISON ALTERNATIVE IMAGE SO(3) MAPPINGS
We argue that main driver of our method's pose accuracy is -equiuuaavari processing. While many existing methods for end-to-end SO(3) -qquivariance have been explored in the literature (Fuchs) et al. 2020 Deng et al. 2021 Cohen et al.) 2018a), it is not well-understood how to combine non equivarivar and equivariant layers in a network. In this section, we consider other sensible approaches map from features in the image plane features that live SO(3) subgroup of it. The approach taken by 12S is to perform orthographic projection to link features on the sphere to the image plane (spatial projection). then convolve with filter that parametrized
discrete
8

---

Published as a conference paper at ICLR 2023
Figure 4: Distributions generated by I2S for SYMSOL objects. To visualize the probabilities over SO(3), rotations with non-nen -egliiiii probability are plotted as dots using a Mollweide projection, encoding rotation orthogonal the sphere with color (Murphy al. 2021). 12S is able capture large discrete symmetry groups of the cube and icosahedron (left column), well as continuous symmetries of the cone and cylinder (middle column). I2S predicts single pose when the dot is visible on cylinder (cylO), but uncertain when the dot is occluded (right column).
by the Fourier basis of 52 to generate signal over SO(3) (Fourier filter). Sensible alternatives can be proposed by instead linking features in the image to coefficients of the Fourier basis (Fourier projection) or parametrizing the filter with features over the sphere (spatial filter). Alternatively, one may consider the approach of directly computing the coefficients of the Fourier basis using MLP. Finally, we include modification the I2S framework using the icosahedral group convolution (a discrete group of SO(3) instead of spherical convolution. This model uses mnchor-regression framework to produce continuous rotation predictions, as outlined Chen et al. (20II.
Table 4: Comparing different approaches to learn an SO(3) signal with image inputs. Results show orientation prediction performance ModelNet1O-SOO3) with limited training views.
Acc@I 15
Acc@30
MedErr
12S (spatial projection, Fourier filter)
0.623
0.640
46.3
12S (spatial projection, spatial filter)
0.610
0.633
44.6
I2S (Fourier projection, spatial filter)
0.523
0.533
56.8
12S (Fourier projection, Fourier filter)
0.511
0.631
57.0
Fourier basis w/ MLP
0.360
0.390
69.2
Icosahedral group version
0.455
0.582
57.6
We evaluate the pose prediction performance of these alternative approaches in Table 4 on the ModelNet1O-SO(3) dataset with limited training views. I2S in all its variations, outperforms using an MLP to parametrize the Fourier basis and the discrete icosahedral group convolution version. This likely because pose estimation benefits from equivariance to the continuous rotation group. We find that using spatial projection is important for accuracy, which hypothesize is better able to preserve spatial information encoded in the dense feature map. How the filter is parametrized is less influential. We use the Fourier basis parametrize the filter, while the original spherical CNN work parametrizes the filter using a spatial grid over the sphere (Cohen et al. 2018a).
CONCLUSION
In this work, we present the first method to leverage SO(3) -equivariance for predicting distributions over 3D rotations from single images. Our method hetter suited than regression methods handling unknown object symmetries, generates more expressive distributions than methods using parametric families of multi-modal distributions while requiring fewer samples than an implicit modeling approach. We demonstrate state-of-the-art performance composed of real images. One limitation of our work is that we use a high maximum frequency, L, in the spherical convolution operations to have higher resolution predictions. Because the number of operations in a spherical convolution is quadratic in L, may be impractical for applications where more spherical convolutions required.
the challenging PASCAL3D+ dataset

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, and NASA 80NSSCI9K1474. R. Walters is supported by the Roux Institute and the Harold Alfond Foundation and NSF grants 2107256 and 2134178.
REPRODUCIBILITY STATEMENT
The code to replicate the results of our method available at httpp:://itthhbb com/ dmk ee/ mm gmaghheeerr, All datasets used are publicly available and thorough description of prepro- cessing is provided in Appendix The model architecture and training protocol are discussed in Section 5.3 and Appendix B.1 Information the baselines and their reported results is provided in Appendix B.3
REFERENCES
Romain Brégier. Deep regression on manifolds: 3d rotation case study. In 2021 International Conference on 3D Vision (3DV), pp. 166-174. IEEE, 2021.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14514-14522 2021.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999. PMLR, 2016.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cnns. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018a
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Representations, 2018b. htt.:httts///opeeee
net/ fort//frum??
HddHbbd5x2Rb
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilie, and Tolga Birdal. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation. International Journal of Computer Vision, pp. 1-28, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning, pp. 1812- 1822. PMLR, 2019a.
Carlos Esteves, Yinshuang Xu, Christine Blanchette and Kostas Daniilidis. Equivariant multi- view networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1568-1577, 2019b.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in homeomorphic variational auto-encoding, CoRR, 0tss/180.00kl 04689, 2018. URL htt.: //ap/v//a/ivvv//a org/ abs 1807.00 C4689
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se transffooom::::: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 13:1970-1981, 2020.
10

---

Published as a conference paper at ICLR 2023
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kos- tiantyn Lapchevskyi. Maurice Weiler, Michat Tyszkiewicz, Simon Batzner, Dylan Madisetti, Martin Uhrin, Jes Frellsen, Nuri Jung. Sophia Sanborn, Mingjian Wen, Josh Rackers, Marcel Rød, and Michael Bailey. Euclidean neural networks: e3nn, April 2022.
Igor Gilitschenski, Roshni Sahoo, Wilko Schwarting, Alexander Amini, Sertac Karaman, and Daniela Rus. Deep orientation uncertainty learning based on a bingham loss. In International Conference on Learning Representations, 2019.
Krzysztof M Gorski, Eric Hivon, Anthony J Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. Healpix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622(2):759, 2005,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1162-1 1144, 2020.
Stefan Hinterstoisser, Stefan Holzer, Cedric Cagniart, Slobodan Ilic, Kurt Konolige, Nassir Navab, and Vincent Lepetit. Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In 2011 I international conference on computer vision, pp. 858-865 IEEE, 2011.
Anthony W Knapp. Lie groups beyond an introduction, volume 140. Springer, 1996.
Axel Levy, Frédéric Poitevin, Julien Martel, Youssef Nashed, Ariana Peck, Nina Miolane, Daniel Ratner, Mike Dunne, and Gordon Wetzstein. Cryoai: Amortized inference of poses for ab initio reconstruction of 3d molecular volumes from real cryo-em images. In European Conference Computer Vision, pp. 540-557. Springer, 2022.
Zhigang Li, Gu Wang, and Xiangyang Ji. Cdpn: Coordinates- based disentangled pose network for real-time rgb-based 6-dof object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7678-7687, 2019.
Shuai Liao, Efstratios Gavves, and Cees GM Snoek. Spherical regression: Learning viewpoints, surface normals and 3d rotations n-spheres. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9759-9767, 2019.
Siddharth Mahendran, Haider Ali, and Rene Vidal. A mixed classification-eegrssion framework for 3d pose estimation from 2d images. arXiv preprint arXiv:1805.03222 2018.
David Mohlin, Josephine Sullivan, and Gérald Bianchi. Probabilistic orientation estimation with matrix fisher distributions. Advances in Neural Information Processing Systems, 33:4884 4893, 2020.
Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions the rotation manifold. In Proceedings of the 38th International Conference on Machine Learning, pp. 7882-7893, 2021.
Jung Yeon Park. Ondrej Biza, Linfeng Zhao, Jan- Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. CoRR, abs/2204. 11771, 2022.
Sergey Prokudin, Peter Gehler, and Sebastian Nowozin. Deep directional statistics: Pose estimation with uncertainty quantification. In Proceedings of the European conference on computer vision (ECCV), pp. 534-551, 2018.
11

---

Published as a conference paper at ICLR 2023
Hao Su, Charles R Qi, Yangyan Li, and Leonidas Guibas. Render for emn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE international conference on computer vision, pp. 2686-2694, 2015.
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birch- field. Deep object pose estimation for semantic robotic grasping of household objects. CoRR, abs/1809.10790, 2018. URL httt:httpaaaee org. abs 1809 18799
10790
Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1510-1519, 2015.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2642-2651, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1913-1920, 2015.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild, In IEEE winter conference on applications of computer vision, pp. 75-82.
IEEE, 2014.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Robotics: Science and Systems, 2018.
Anna Yershova, Swati Jain, Steven M Lavalle, and Julie C Mitchell. Generating uniform incremental grids on so (3) using the hopf fibration. The International journal of robotics research, 29(7): 801-812, 2010.
Sergey Zakharov, Ivan Shugurov, and Slobodan llie. Dpod: 6d pose object detector and refiner. In Proceedings the IEEF/CVF international conference computer vision, pp. 1941-1950, 2019.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
12

---

Published as a conference paper at ICLR 2023
A DETAILED RESULTS
A.1 MODELNET 10-SO3 LIMITED TRAINING SET
We compare the sample efficiency of pose prediction methods by training on ModeINet10-SOG) with limited training views in Table Specifically, we reduce the number of training views from 100 per instance to 20 per instance. Our method outperforms the baselines by larger margin in the low-data setting. This results highlights an important distinction between our method and the baselines: our method explicitly encodes the 3D rotation symmetry present in the pose prediction problem, whereas other methods must learn the symmetry from data.
Table 5: Comparison on ModeINet1O-SOL (03(0)00)000))))000 with limited training views. Methods are trained on five times fewer images than the experiment in Table I
Limited Training Set
Acc@ 15t
Acc@301
MedErr
Zhou et al. 2019
0.064
0.239
62.7
Bregier 2021
0.129
0.359
51.5
Murphy al. 2021
0.515
0.533
59.5
12S (ours)
0.623
0.640
46.3
A.2 MODELNET 10-SO3 PER-CH TER-CLASS BREAKDOWN
Table reports the median rotation error for each object in ModeINet10-SO(33. These results highlight the challenge pose prediction with unknown symmetries. Note that the categories that are hard to predict, e.g. bathtub, night stand and table, can have multiple correct reference frames due to symmetry.
Table 6: Per-class median rotation error ()) on ModeINet10-SO(33, Results for I2S are reported as mean and standard deviation over six random seeds.
avg.
bachrub
bed
chair
desk
dresser
monitor
stand
sofa.
table
toilet
Zhou 2019)
41.1
103.3
18.1
18.3
51.5
32.2
19.7
48.4
17.0
88.2
13.8
Breguer 20k1
39.9
98.9
17.4
18.0
50.0
31.5
18.7
46.5
17.4
86.7
14.2
Detg et
32.6
147.8
9.2
8.3
25.0
11.9
9.8
36.9
10.0
58.6
8.5
Mobili et al
17.1
89.1
4.4
13.0
6.3
13.5
4.0
25.8
Prokunin
2011
49.3
122.8
3.6
117.2
29.9
6.7
73.0
10.4
115.5
Murphy et
N021
21.5
161.0
444
5.5
7.1
5.5
5.7
7.5
4.1
9.0
16.3
124.7
3.1
4.7
3.0
429
+-28.1
+-0.0
+-0.0
+0.11
1:0.1
+0011
+0.1
4.000
+:1.0
+-0.1
13

---

Published as a conference paper at ICLR 2023
A.3 VISUALIZING PREDICTIONS
Figure 5: Example predictions of our method on ModelNer10-SO(3) with 100 training views. The dataset includes objects with two-fold (i.e bathtub) four-fold symmetry (i.e. side table), which method can model as multi modal distributions in SO(3).
Figure 6: Additional predictions our method on PASCAL33++ Ground truth label shown as circular ring. The distribution over SO(3) is visualized by converting Euler angles: the first two represented spatially via Mollweide projection and the third encoded as color. Our model captures uncertainty in the pose that result from object symmetries insufficient visual cues.
14

---

Published as a conference paper at ICLR 2023
Figure 7: 12S capturing object symmetries and pose ambiguity on SYMSOL Dataset (50k views). Each row corresponds different shape from SYMSOL: cone, cylinder, tetrahedron, cube, icosahe- dron, circleX, cylinderO,, and tetrahedronX. I2S is able to represent complex object symmetries, both discrete and continuous.
15

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
B.1 ARCHITECTURE
We use ResNet encoder with weights pretrained ImageNet. With 224x224 images as input, this generates a 7x7 featuremap with 2048 channels. The orthographic projection onto the sphere is performed using HEALPix grid of recursion level 2 restricted to half the sphere. With each forward pass, 20 of these grid points are randomly sampled and used to generate the S signal. The S signal is converted the Fourier domain with maximum frequency of 6. A spherical convolution operation is performed using a filter that is parametrized in the Fourier domain, which generates an 8-channel signal over S0(3). A non-linearity is applied by mapping the signal to the spatial domain, applying ReLU, then mapping back to Fourier domain. One final spherical convolution with a locally supported filter is performed to generate one-dimensional signal on SO(3). The signal is queried using an SO(3) HEALPix grid (recursion level 3 during training, 5 during evaluation) and then normalized using softmax. The network instantiated in PyTorch, and we use the 3nm library for the group convolution operations.
B.2 DATASET PREPARATION
ModeINet10-SO(33) is available for download at the associatee with Liao et al. 2019)) The dataset has a standardized train and test split. It provides two training sets: one with 100 views per object instance (we call this the Full Training Set), and one with 20 views per object instance (we call this the Limited Training Set). The test set has 4 views per instance. Each image is labeled with a single rotation label, and object symmetries were not broken during labeling. Based on the original work, we do not perform any data augmentation with this dataset.
SYMSOL can be downloaded from the Githut linked by Murphy et al. 2021). The dataset includes renderings synthetic objects split into two categories: SYMSOL includes tetrahedron, cube, icosahedron, cone and cylinder, while SYMSOL II includes marked tetrahedron, marked cylinder, and marked sphere. For each shape, there are 50k renderings in the training set and 5K renderings in the test set. Each image is labeled with the set of valid rotations (for continuous symmetries like the cylinder, it is provided at 1 degree increments). During training. the set of valid rotations is randomly sampled generate single rotation label to compute the loss. In Table R] we use results originally reported by MMrrhy et al. (202I). and follow the approach of training a single model on all objects from SYMSOL but different models for each object in SYMSOL II. The results reported by Murphy et al. ( were generated using 100k training renderings per shape, but the publicly released dataset only has 50k. Thus, this strongly favors the baselines.
Pascal3D+ is available for download at the link 5 provided in Xiang et al. (2014). The training data is found in the ImageNet train, ImageNet val, and PASCALVOC train folders, and the test data is in PASCALVOC val. Following Murphy et al. (021)) we discard any data that is labeled occluded, difficult or truncated. We follow the data augmentation procedure from Mohlin et al.) (2020) that randomly performs horizontal flip and slight perspective transformation during training. Additionally, we supplement the training data with synthetic images from RenderForCNN (Su et al. 2015) (this requires free ImageNet account to download), such that three quarters of data is synthetic during training. Including synthetic data is important to achieve high accuracy, and is also used by the baselines. Where possible, we use the reported results of the basclines on PASCAL3D+; however, as pointed out by MMurphy et al. (2021), some of the baselines used incorrect evaluation functions or a different evaluation set. In these cases, we report the performance from Murphy al. (202T), since they re-ran these basclines after correcting the issues.
https://e3nn.or
https:
imp licit_ccfpp
https://cvgl.
http:://gg.ccaaoooooeeee///////p html
16

---

Published as a conference paper at ICLR 2023
B.3 BASELINES
Here, include details for baselines that implemented ourselves. For information on other baselines, refer directly to their work.
Zhou et al. (2019) proposes using the Gram-Schmidt process convert 6D vector into valid, 3x3 rotation matrix. To implement this network, we used a ResNet architecture with spatial pooling on the final feature map. The resulting vector was processed with two linear layers generate the 6D vector. The model is trained with an L.2 loss function using ground truth rotation matrix. The method is trained on all classes at once, but uses a separate linear layer to predict the rotation of each class.
Brégier (021) proposes using the Procrustes method to convert a 9D vector into a valid, 3x3 rotation matrix. They provide efficient implementation of the Procrustes method. The architecture and loss function is the same as for Zhou et al. (019), except the linear layer produces a 9D vector.
The original work of Liao et al.
(2019) only showed results for PASCAL3D+, which used an incorrect evaluation function as noted by Murphy et al. (2021). Thus, for PASCAL3D+, we used the results reported by Murphy et al. (2021) with the corrected evaluation function. For ModeINet10-SO(3), we ran their code using a pretrained ResNet50 as an encoder.
B.4 CREATING VISUALS
We follow the visualization code that was publicly released by Murphy et al. (2021) to represent distributions over 3D rotations represented as a dot, with the size proportional the magnitude. The rotation is encoded by converting XYX culer angles, the first two angles correspond to latitude and longitude in Mollweide projection and the final angle is encoded as color using an HSV colormap. To make the visualizations more interpretable, we not plot any probabilities less than a given threshold. In our visualizations, generate probabilities associated with the SO(3) HEALPix grid used during model evaluation (recursion level 5; 2.4M points).
2D plot. The probability associated with each rotation is
C ADDITIONAL EXPERIMENTS
C.I EFFECT OF MAXIMUM FREQUENCY
To efficiently perform SO(3) group convolutions, we must restrict our representations to in the frequency domain. Learning with lower frequency signals can more efficient and may generalize better in some cases, while higher frequency signals may be better for encoding complex distributions like those shown in Figure In Table 7 we perform small experiment showing the effects of different maximum frequencies. L, on pose prediction accuracy. Note that use L = 6 for all other experiments in this work. Interestingly, we find that including higher frequencies in the representation, e.g. L can actually reduce accuracy, despite the additional learnable parameters.
Table 7: Varying maximum frequency L in Fourier basis. Results are generated on ModelNet1O-SOCC) with limited training views. The same is maintained through both spherical convolutions of our method.
Acc@ 15t
Acc@ 300
MedErr|
2
0.560
0.656
33.6
L 4
0.626
0.646
46.6
I. 6
0.623
0.640
46.3
8
0.605
0.621
43.7
10
0.599
0.618
46.8
17

---

Published as a conference paper at ICLR 2023
C.2 EFFECT OF ADDITIONAL SO(3) CONVOLCTIONS
Table 8: Effect of SO(3) convolutional layers on ModeINer10-SO(3) pose prediction. I2S, as pro- posed, includes one (3) convolution following the S convolution. We show performance without the (3)3) convolution, and with an additional SO(3) convolution. Additional SO(3) convolutions provide minimal benefit the expense of compute.
Full Training Set 30
Limited Training Set
Acc@ 151
MedErry
Acc@ 15t
Acc@ 301
MedErr)
I2S, S3(3) conv I2S, one SO(3) conv 12S, two (3) conv
0.729
0.737
15.6
0.616
0.634
46.4
0.729
0.737
14.4
0.623
0.640
46.3
0.729
0.737
19.0
0.625
0.643
45.5
Our method performs one S convolution followed by one SO(3) convolution. In this section, we look at the performance benefits of this final refinement layer and potential benefits of an additional convolution. We find that the performing SO(3) convolution does lead to a marginal improvement, with little be gained by performing more. This suggests that most of the SO(3) reasoning occurs within the first spherical convolution that uses a learned filter with global support.
18


Reviews:
[{"review_id": "bBM5XWImLEu", "reviewer": "Reviewer_CAH5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Image2Sphere a rotation distribution estimation method by mapping 2D image features to the 3D rotation manifold and applying SO(3) equivariant convolution on the 3D rotation manifold. This is done by first mapping the image features onto a sphere using orthographic projection, and then a learned filter is applied to transform the signal onto SO(3). Finally, SO(3) group convolution is performed to produce the distribution over the 3D rotation space. Thanks to the SO(3) equivariant group convolution operation, the pose estimation distribution can be modeled well in the results. The method is evaluated on several image-based orientation estimation results and the proposed method is reported to outperform prior methods and other baselines, especially in a low data regime. ", "summary_of_the_review": "The paper proposes a novel method for the distributional estimation of SO(3) rotation. The evaluation is solid and shows the efficacy of the method. I tend to accept this paper, yet the paper does not fully align with my background and I may not have a full understanding of the field. ", "strength_and_weaknesses": "Strength:\n1. The paper proposed a novel orientation distribution estimation method from simple image features. The idea of mapping image features to a sphere and further a 3D manifold is very interesting. \n2. SO(3)-equivariant Group convolution is devised for rotation estimation on the manifold. This brings a new idea to the field and I think it is inspiring and could be further extended to other applications. \n3. The paper is well-written and easy to follow.\n \nWeakness: \n1. The evaluation is only conducted on images with a single object, well cropped and centered in the images.  I am wondering how the method performs in other more cluttered and occluded datasets (e.g. YCB-V and LM-O). Learned or ground truth detection/segmentation may be used for this purpose. \n2. I would like to hear the authors’ comments on how well the proposed method to unseen object instances/categories. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NViDHkcfAS", "reviewer": "Reviewer_n1bx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an approach for pose estimation for objects with symmetries. The approach leverages SO(3)-equivariance to predict distributions over 3D rotations from single images. Image features are obtained from a pre-trained ResNet. These features are orthographically projected to the sphere where the features are convolved with a learned filter on S^2. As a result a signal is generated on SO(3). A group convolution on SO(3) is performed generating a detailed distribution over SO(3). This distribution can model symmetric objects. As a result object symmetries are learned and uncertainty can be represented. ", "summary_of_the_review": "This is a paper describing an approach for pose estimation of symmetric objects with prediction of uncertainty. The contribution is clear but incremental.", "strength_and_weaknesses": "The strengths include the ability to handle unknown object symmetries and the generation of distributions over poses, which in addition of allowing to take into account the pose ambiguity from object symmetries, it permits the representation of uncertainties due to partial observability. However with some datasets (SYMSOL) previous methods (Murphy et al.) have better performance. Computational cost not evaluated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "DE77DTZ_uzi", "reviewer": "Reviewer_PnmT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a novel SO(3) equivariant architecture for pose prediction of objects from 2D images. The neural network takes image features from an image encoder and maps it to a sphere, which is followed by an $S^2$ convolution and one or more $SO(3)$ convolutions. Both these layers are equivariant to $SO(3)$ transformations. An important feature is that the output of the neural network is a distribution on the $SO(3)$ pose manifold. This feature is important in the case where the objects have symmetries and the pose can be known only up to that symmetry. Experiments on multiple pose prediction datasets show improved performance over baselines as well as the ability to produce distributions that show understanding of object symmetries.", "summary_of_the_review": "Overall, I think the paper studies an important problem in computer vision, and proposes a simple and effective architecture, which takes into account symmetries in the objects for pose prediction. I do think some of the experimental results and novelty of the paper have to be explained and done better. I am inclining towards acceptance.\n\nUPDATE AFTER AUTHOR RESPONSE:\n\nI appreciate that the authors have responded too all my comments. I think the paper is in good shape to be accepted although architectural novelty is limited. It does show how to use existing spherical and SO(3) convolutional layers for an important problem and shows some improvements in performance. I am increasing my rating based on the authors' response to my comments and those of the other reviewers. ", "strength_and_weaknesses": "Strengths:\n\n1. The proposed architecture and loss function -- I2S -- are simple and intuitive. \n\n2. The paper is well-written with good illustrations and visualizations of results for important cases that show the usefulness of the I2S and that I2S works predictably for some known objects, which is important.\n\n3. Experiments on multiple datasets show that I2S indeed outperforms all the baselines considered almost always.\n\nWeaknesses:\n\n1. Novelty of the method: Given that both $SO(3)$-equivariant convolutions as well as pose prediction by predicting distributions over $SO(3)$ are existing ideas, I am not sure that the architecture in this paper is novel enough.\n\n2. Comparison with Murphy (2021): Experiments on SYMSOL indicate that Murphy (2021) outperforms I2S when the entire dataset is available. However, I2S performs better in a few-shot setting. I think the authors should try to clarify why exactly mapping to the sphere and then the SO(3) is important, this seems to be the main difference from Murphy (2021), which is not clear in the current version of the paper. \n\n3. Experimental results should have error bars and statistical significance values.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Ae9M-zRO7h", "reviewer": "Reviewer_9pzm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a deep neural network to predict poses in SO(3) from 2-D images. The proposed method first maps 2-D images onto a hemi-sphere by projection, and then convolve it with a filter on $S^2$ in the basis of spherical harmonics, and then apply rotation-equivariant layers that convolves the signals in the basis of Wigner D-matrices. The proposed method appears to be more effective than other networks on a variety of datasets in handling ambiguity that is caused by object symmetry.", "summary_of_the_review": "In summary, this is an ok submission. However, there is a large room for improvement in terms of clarity and experiments. The idea of this work is neither novel nor sufficiently motivated. ", "strength_and_weaknesses": "Strength: \n\n1. The paper aims to solve an interesting and challenging problem: estimating 3-D pose from 2-D images, which has broad applications in 3-D computer vision and 3-D imaging.\n\nWeaknesses and questions:\n\n1. Although the writing is in general ok, there are some missing details in the implementation. I encourage authors to provide a more detailed description of each layer and operations in the proposed neural network. For example, the authors did not clarify how exactly equation (2) is implemented in the Fourier space of SO(3).\n\n2. The idea of this work is not sufficiently motivated. I can see the advantages of learning the distribution of SO(3) can mitigate the issue of symmetry. However, it remains unclear how the S2 convolution and the operations shown in Figure 2 are essential to the success of the proposed method. More explainations and experiments are expected to demonstrate the effectiveness of each module of the proposed network.\n\n3. The idea of this work is not novel. It seems to me that proposed method is a combination of ideas from previous works such as group convolution,  Fourier expansion on SO(3) and etc. Learning distribution on SO(3) to break symmetry is also not a new idea, see [1].\n\n4. The proposed method is not the only work that predicts rotations from 2-D images, so there are some missing references. See [2] that uses each 2-D image to predict the pose of protein molecule for 3-D reconstruction. I also encourage the authors to have some discussions that compare this work with your method, since they are able to obtain reliable pose estimates from 2-D images using a simple neural network.\n\n[1] Synchronizing Probability Measures on Rotations via Optimal Transport.\n\n[2] CryoAI: Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%