3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction
Authors: Jongmin Lee
Minsu Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: SO(3) pose estimation
3D rotation representation
SO(3)-equivariance
3D equivariant networks
spherical harmonics
Wigner-D Matrix
spherical CNNs
Wigner-D coefficients prediction
uncertainty modeling
data sampling efficiency
Primary area: machine_vision
TL;DR: We address single-image 3D pose estimation by predicting Wigner-D coefficients in the frequency domain using SO(3)-equivariant networks, to improve pose estimation accuracy and data sampling efficiency.

Abstract: Determining the 3D orientations of an object in an image, known as single-image pose estimation, is a crucial task in 3D vision applications. Existing methods typically learn 3D rotations parametrized in the spatial domain using Euler angles or quaternions, but these representations often introduce discontinuities and singularities. SO(3)-equivariant networks enable the structured capture of pose patterns with data-efficient learning, but the  parametrizations in spatial domain are incompatible with their architecture, particularly spherical CNNs, which operate in the frequency domain to enhance computational efficiency. To overcome these issues, we propose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression, aligning with the operations of spherical CNNs. Our SO(3)-equivariant pose harmonics predictor overcomes the limitations of spatial parameterizations, ensuring consistent pose estimation under arbitrary rotations. Trained with a frequency-domain regression loss, our method achieves state-of-the-art results on benchmarks such as ModelNet10-SO(3) and PASCAL3D+, with significant improvements in accuracy, robustness, and data efficiency.

3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction
Jongmin Lee"
Minsu Cho
Pohang University of Science and Technology (POSTECH), South Korea
j1jm1121, macho)@postech. kc.kr
http://cvlab. www.o//llappps.ch...omoommmo ac at.
Abstract
Determining the 3D orientations of object in an image, known as single-image pose estimation, is crucial task in 3D vision applications. Existing methods typi- cally learn 3D rotations parametrized the spatial domain using Euler angles or quaternions, but these representations often introduce discontinuities and singulari- ties. -quivvvaraaa networks enable the structured capture of pose patterns with data-efficient learning, but the parametrizations in spatial domain incompatible with their architecture, particularly spherical CNNs, which operate the frequency domain to enhance computational efficiency. To overcome these issues, we pro- pose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression, aligning with the operations of spherical CNNs. Our SO(3)-equivariant pose harmonics predictor overcomes the limitations of spatial parameterizationss, ensuring consistent pose estimation under arbitrary rotations. Trained with rrequencydooan regression loss, our method achieves state-of- the-art results benchmarks such ModeINet10-SO(3) and PASCCLL3++ with significant improvements in accuracy, robustness, and data efficiency.
Introduction
Prodict 3D Rotations Spatial
Predict Parametric
n
D
(At 2)
Predicting the 3D pose of objects, i.e., position and orientation, in 3D space from an image is
crucial for numerous applications, including aug- mented reality [59], robotics [4, 5, 63, 69], au- tonomous vehicles [21, 50], and cryo-electron microscopy [79]. Estimating 3D orientation particularly challenging due to rotational symme- tries and the non-linear nature of rotations. In addition, unlike translations, rotations introduce unique challenges such as gimbal lock and the requirement for continuous, singularity-free rep- resentations. Existing methods often learn 3D rotations using spatial domain parameterizations like Euler angles, quaternions, or axis anile rep- resentations, illustrated in Figure However, these parameterizations suffer from issues such discontinuities and singularities [54, 58, 80), which can hinder the performance and reliability.
(eg, Euler, Quaterionn
Ganssian, vME, Binghanm
Predict Non-Parametri Disth'n
Predict Wigner-D Harmonics
80
HEALPS. Superffibonaeci
Figure Types of representations for 3D ro- tation prediction. Existing methods consider predicting 3D rotations in the spatial domain. Our method predicts Wigner-D coefficients in the fre- quency domain, to obtain accurate pose in contin- uous space using an SO(3)-equivariant network.
The current afliliation of Jongmin Lee with LG Research. Contact: jongminlee @ Igresearch.ai.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

-o(3)--quivariance enables accurate 3D pose estimation and improves generalization to unseen rotations. It ensures that oulputs consistently change with the 3D rotation of the imput, maintaining ro- tational consistency between the input and output across network layers. Despite its importance, many existing methods [3, 44, 52, 75, 80] often design networks without considering (0(3))-quivariance resulting in suboptimal performance when dealing with 3D rotations. In addition, in the context of spherical CNNs [9. 10, 12, 16-18, 36] for efficient -equi-equivariant operations, the 3D rotation parametrization in the spatial domain is inadequate because these -0(3)-qquivariant networks operate in the frequency domain.
To address these challenges, we propose an (Quuvv-aui pose harmonics regression network that directly predicts Wigner-D coefficients in the frequency domain for 3D rotation regression. Building on prior work [28, 35], our method leverages the properties of spherical CNNs [11]. which operate in the frequency domain, to guarantee -0(3)-equivariant output representation. By directly regressing Wigner-D D matrix coefficients, our approach eliminates the need to convert outputs into spatial representations during training, ensuring alignment with the operations of spherical CNNs. This design allows us to bypass the limitations inherent in traditional spatial parameterizations--ute as discontinuities and singularities [54, 58, 80]-resulting in more precise and continuous pose estimation. We further introduce 3D rotations, with the flexibility to incorporate distributional losses [52] for effectively capturing rotational symmetries in objects. Our method achieves state-of-thed afrthe-art performance on standard single object pose estimation benchmarks, including ModelNet10- (03))(0))))))) and PASCAL3D+, demonstrating high sampling efficiency and strong generalization to unseen 3D rotations.
deqquencydommane MSE loss to enable continuous training of
2 Related Work
SO(3) pose regression. The choice of rotation representation is a fundamental aspect of the current
SO(3) pose estimation methods. In the early stages of deep learning, methods for SO(3) pose regression choose the rotation representation by direct cosine matrices [29, 73). Euler angles [37, 48, 49, 60, 61]. quaternions [5, 13, 31, 32, 70, 78), and axis-angles [14, 20, 62]. However, according to [80], for any representation R in a Euclidean space of dimension d S 4, such as Euler angles and quaternions, R is discontinuous and unsuitable for deep learning. In addition, Euler angles can cause gimbal lock, which restricts certain rotations, whereas quaternions avoid this issue but their double representation of rotations in SO(3) can lead to complications such as local minima in optimization problems. As an alternative, a continuous 6D representation with Gram-Schmide orthonormalization [80] and 9D representation with singular value decomposition (SVD) [8, 41] have been proposed, and [7] proposes manifold-ane aware gradient layer to facilitate the learning of rotation regression. Denoising diffusion models are employed in the context of SO(3) pose regression [64], or for solving pose estimation by aggregating rays [76]. In contrast to existing SO(3) pose regression methods that formulate rotation representations the spatial domain, we define the Wigner-D coefficients as the output of the network in the frequency domain, using -quiiaaquivariant networks.
Pose estimation with a parametric distribution. To model rotation uncertainty, parametric distribu- tions on the rotation manifold are employed in a probabilistic manner. [57] predicts parameters of a mixture of von Mises distributions over Euler angles using Biternion networks. [5, 13, 23] utilize the Bingham distribution over unit quaternions to generate multiple hypotheses of rotations. [51, 74] leverage the matrix Fisher distribution [33] to construct a probabilistic model for SO(3) pose esti- mation. Additionally, [75] propose the Rotation Laplace distribution for rotation matrices on SO(3) to suppress outliers, and the Quaternion Laplace distribution for quaternions on S3, Nevertheless, parametric models rely on predefined priors. In contrast, our model uses non-parametric modeling during inference to capture more complex pose distributions.
Pose estimation with a non-parametric distribution. Probabilistic pose estimation can also be achieved by predicting non-parametric distributions. IPDF [52] introduces the estimation of arbitrary, non parammeti distributions on SO(3) using implicit functions with MLPs, and Hyper- PosePDF [27] uses hypernetworss to predict implicit neural representations by Fourier embedding. ExtremeRotation [6] predicts discretized distributions over N bins for relative 3D rotations trained with cross-entropy loss. RelPose [43, 77] uses an energy-based formulation to represent distributions over the discretized space of SO(3) relative rotation. Several
modeling methods construct non poramaramttric distributions by utilizing icosahedral group convolution [34], projecting image features orthographically onto sphere [35], and satisfying consistency properties of SO(3)

---

by translating them into an (oo))arruaaaranne constraint [28]. RotationNormFlow [44] uses dis- crete normalizing flows to directly generate rotation distributions on SO(3). These non-parametric methods, which are trained with loss functions in discretized distributions, such as cross-entropy and negative log-likelihood, tend to lose precision rotation prediction. In contrast, our method predicts continuous SO(3) transformations through regression, eliminating the need to approximate SO(3) poses within a discretized space and enabling our model to achieve accurate 3D rotations.
3 Preliminary
3.1 Representations of Rotations
Rotation representation in spatial domain. In 3D rotation, Euler angles are common SO(3) representation but suffer from non-uniqueness and gimbal lock. making them less suitable for neural network predictions. Quaternions offer a solution by preventing gimbal lock, but their non- unique representation (q and -q) can complicate certain optimization processes. The axis-angle representation is intuitive but can encounter singularities. The 6D and 9D representations provide newer approaches that simplify optimization in deep networks by avoiding non-linear constraints and ensuring orthogonality. However, they also introduce complexities in maintaining constraints during the learning process. Thus, choosing an appropriate rotation representation is crucial for accurate pose estimation in various computational applications. For a detailed explanation, please refer to Sec. A.1 and an overview of learning 3D rotations in [54, 58].
Rotation representation in frequency domain. In the frequency domain, 3D rotation is managed by manipulating spherical harmonies coefficients. Spherical harmonics, denoted as Y (0, ø), are functions defined on the surface of a sphere using polar (0) and azimuthal (ő) angles. These harmonies are characterized by their degree I and order m, truncated to maximum degree L for computational feasibility. The rotation of spherical harmonics is represented by the shift theorem [46], where a rotation operator Ag acts on spherical harmonics, transforming them via a matrix D (g)
Ag (x)
Σ (' (z). |
This matrix, part of the irreducible unitary representation of SO(3), expresses how each harmonic changes under rotation, summing over all orders from -/ to I, called Wigner-D matrix. The Wigner-D rotation representation is not limited to a specific case of 3D rotations but can be converted from any 3D rotation representation, such as Euler angles, quaternions, and 3D rotation matrices. Our SO(3) equivariant network predicts the Wigner-D representation in the frequency domain instead of predicting rotations in the spatial domain. For detailed explanation, please refer to Sec. A.2.
3.2 SQuii-Equivariance
Equivariance. Equivariance is useful property to have because transformations T' applied to the input produce predictable and consistent output of the features through transformations ø E &, enhancing both interpretability and data efficiency. For example, feature extractor equivariant to a transformation if applying the transformation to the input and then applying the extractor produces the same output as applying the extractor first and then the transformation:
(x), T3 (x)))
(2)
where Tg and y represent transformations acting on a group y E G of the input and output spaces, respectively. This ensures that the network's output remains consistent with transformations applied to the input. For translation groups, convolution inherently maintains this property. For rotations, additional rotaiion-equivariaan layers are integrated into the network design.
Group-equivariant convolutional networks [10] extend this concept to complex groups like rotations or other symmetries. By designing convolutions that are equivariant to these group actions, these networks can handle broader range of transformations. This can be mathematically described as:
[A <1(9)
Σ h(y) 00g MEX
where h is the input function over space X, Φ is the filter or kernel, and g E G is an element of the group. The term - represents the transformation of y by the inverse of g. This operation
3

---

Pose Estimation Network Wigner-D coefficients Y E R
Feature extractor
Spherical mapper
Fourier
S0(3)- (ide)-qvvaaaant convolution layers
transfommer
MSE Loss
(0)3)-quivariann pose harmonics estimation network
Figure 2: Overall architecture. Our network for S0(3)--quivarian) pose estimation consists of four parts: feature extraction, spherical mapper, Fourier transformer, and SO(3)-equivarian layers. First, we extract feature map using pre-trained ResNet. Next, the spherical mapper orthographically projects the extracted feature map onto spherical surface. The Fourier transformer converts this spatial information into the frequency domain. We utilize spherical convolutions to obtain the final Wigner-D harmonics coefficients V which represent SO(3) rotations of spherical harmonics, where M denotes the total number of Wigner-D matrix coefficients.
ensures that the network remains equivariant to the actions of the group G, allowing it to handle inputs transformed by any element of this symmetry group.
On the sphere in 3D, however, there is straightforward way to implement a convolution in the spatial domain due to non-uniform samplings [12]. This challenge arises because traditional convolution operations rely on uniform grid structures, which are not applicable to spherical data. To address this, specialized methods such spherical convolutions or graph-based approaches are employed to handle the unique structure and sampling patterns of spherical data, thereby ensuring effective feature extraction and equivariance on spherical surfaces.
Spherical convolutions for 50(3) -quuivaaurrr To effectively analyze complex spatial data, such for volumetric rendering and 3D pose estimation, it is necessary develop functions with cquivariance to the (3) group. Early methods for spherical convolution were defined by computing Fourier transforms and convolution on the 2-sphere [15]. However, the output of these spherical convolutions is a function on the sphere, not on SO(3). Spherical CNNs [11] extended this approach effectively convolve on the (3(3) group. Using the truncated Fourier transform, signals on S2 are modeled with spherical harmonies Y and SO(3) with Wigner-D matrix coefficients D
To efficiently compute the S and SO(3) convolution, generalized fast Fourier transforms (GFFTs) demonstrate optimized computation [1]]. The GFFTs show robustness and efficiency in spherical signal processing, where the spectral group convolutions become simpler element-wing multiplications in the Fourier domain. Specifically, for 52, the process vectors of spherical harmonic coefficients, forming a block diagonal matrix analogous to SO(3) convolution. Both convolutions on 52 and SO(3) generate output signals that reside SO(3).
Squiv-aqqi Pose Harmonics Predictor
The goal of network is accurately predict the SO(3) pose of an object in image. To achieve, we employ spherical CNNs [1]] to obtain SO(3)-equivariant representation, and our model is trained with frequency-domain supervision using Wigner-D coefficients. This approach enhances data efficiency by capturing patterns with fewer training samples and ensures precise SO(3) pose estimation by aligning the parametrization 3D rotations with the Wigner-D matrices in the frequency domain.
Figure provides an overview of our 50(3)-equivariaa) pose estimation network. In Sec. 4.1, we explain the steps for obtaining the Wigner-D representation, following the method described in [35]. In Sec. 4.2, we introduce frequency-domain regression loss, where we train the network using MSE loss between the predicted representation and the ground truth (GT) Wigner-D coefficients. Finally, in Sec. 4.3, we describe the inference process by constructing an SO(3) grid for evaluation.
4.1
In this subsection, we explain our SQuia-equivariant pose estimation network, highlighting that its key components are shared with the architecture of [35].

---

Image feature extraction. We first apply feature extractor to obtain an image feature map that encodes semantic and geometric information: F = p(T), where F R xw and denotes ResNet. We utilize a ResNet feature extractor that is pre-trained on ImageNet same to [28, 35, 44, 51, 52, 75J. We then perform dimensionality reduction on the image feature F to match the input dimension of the subsequent spherical feature using a Ixl convolution: F = Conv xx (F), where
F' E RRC X
Spherical mapper. To begin, we lift the im- age features to the 2-sphere using orthographic projection [35]. This involves mapping the 2D feature to a spherical feature v E IR xp, where p denotes the number of points the sphere. The orthographic projection links pixels in the image space to points on the sphere by or- thogonally mapping 5 coordinates to the image plane, thereby preserving the spatial information of the dense 2D feature map.
0chognatt prescies
harmori
-1.
@ D
convolution layes
Initially, we model spherical coordinates using an S HEALPix [24] grid over a hemisphere. Within this hemisphere, the set {a:} C S rep- resents the vertices of the grid. Each vertex x; is mapped to position P(x,) on the image plane. Formally, the orthographic projection P maps 3D coordinates on the hemisphere to 2D coordi- nates on the image plane as P(x, y, 2) = (x, y).
ReLU-
Non-linear
5t filter
in spatial SO(3) filter
Figure 3: Illustration of spherical mapper and spherical convolution for S0(3)-qqivvarince. This structure allows for the prediction of 3D ro- tations while preserving the SO(3)-equivariance of the input structure. Predicting the Wigner-D harmonics V enables continuous 3D rotation mod- cling, without discretizing the group actions."
Due to the fixed perspective, only one hemi- sphere of the sphere is visible, resulting in localized signal vxxx) ('(((r)) supported over this hemisphere. The value of (1x;) ) ob-
tained by interpolating F" at the pixels near P(xz)) in the image space using interpolation function 7). so vxx,;) = P(x,)) Figure 3 illustrates the processes of the spherical mapper, and the following frequency domain conversion and spherical convolution for -0(3)-equivariance.
Convert to the frequency domain. The transition of the spherical feature v into the frequency domain achieved using the fast Fourier transform (FFT) adapted for spherical topology. By employing the FFT, we efficiently convert &b to spherical signals S, represented as a sum of spherical harmonics. This transformation allows us to capture and manipulate the spatial frequencies inherent to the spherical surface. Specifically, the transition to the frequency domain enables the derivation of Wigner-D coefficients, which effectively model the 5O(3). The Fourier series of S truncated at frequency L [1]], expressed as: S(x) 2 El E'=e/ c,, Yr (z), where S E EºC xN N is the total number of spherical harmonics determined by the maximum frequency L, and Y (x) the spherical harmonics. Operating in the frequency domain facilitates the effective convolution of signals on the sphere (S²) and within the 3D rotation group (SO(3)), preserving the geometric properties of input features through spherical equivariance.
To address sampling errors from approximating the Fourier series via truncation, we apply two techniques proposed in [35]. First, to prevent discontinuities on the 2-sphere, we gradually decrease the magnitude of projected features near the image edge: v vz(x;) = w(x;) (x:)). Second, for each projection, we randomly select a subset grid points on the S HEALPix grid as a dropout.
Spherical convolution for -qiiaaaquv We aim predict 3D rotations while preserving SO(3)--quivariance using the projected features on sphere. First, the spherical signal S is processed with an -e-uqivvaria convolutional layer [11, 35]. Unlike conventional convolutions with local filters, S convolution uses globally supported filters, offering global receptive field. This allows for shallower network, which is important due to the high computational and memory demands of spherical convolutions at a high bandlimit L.
We use the visualization tools available in the source code of the image2sphere GitHub repository.
5

---

In this stage, we obtain SO(3) representations inherent to spherical CNNs [11]. The output of S convolutions lies the SO(3) domain because S convolutions replace translations with rotations. and the space of 3D rotations forms the SO(3) group. Consequently, we obtain feature results sized in XM where C" is the hidden dimension of the SO(3) features, and M is the total number of Wigner-D matrix coefficients, given by M = Ei= (21 1) (21 + 1), created by SO(3) irreps.
We apply non-linearities between convolutional layers by transforming the signal to the spatial domain. applying ReLU, and then transforming back to the frequency domain, following the approach of spherical CNNs [11]. This method can be extended to FFT-based approximate non-linearity [19] and equivariant non-linearity for tensor field networks [55, 71].
Subsequent to the -R-equívariant convolutional layer, perform an SO(3)-qquivariant group convolution [11, 35] using locally supported filter to refine the SO(3) pose space. Unlike typical spherical CNNs, we bypass the inverse fast Fourier transform (iFFT) and instead use the output harmonics of Wigner-D prediction. This approach, unlike that of [35], improves the efficiency of our method. The final output of the equivariant network is the Wigner-D matrix coefficients R.M
4.2 Frequency- -Dmaain Regression Loss
The output of the SO())-equvvarian convolutional layers is a linear combination of Wigner-D matrices, represented as a flattened vector of the Wigner-D coefficients. The output V indicates specific object orientations an image. To generate the ground-truth (GT) Wigner-D coefficients, we convert the GT 3D rotations from Euler angles using the ZY Z sequence of rotation R. expressed as R = (7) R Rg (a) to the Wigner-D matrices D'ma (a, 3, 7). where D represents an action of the rotation group SO(3). We calculate the Mean Squared Error (MSE) loss follows:
C(V, VGT
Σ Σ
wi (
1=0 m=-I
where wy are weights assigned to each harmonic frequency level I, normalizing the output Wigner- D matrices for frequency-doman specific MSE loss. This loss function enables continuous prediction of SO(3) poses using SO(3)-eqivvariant networks, whereas the previous methods [28, 35, 52] predicted outputs in discretized distribution, leading to degradation in prediction precision. With this re- parammttrizaaion in the frequency domain, we use Euclidean distance because it is simple yet effective for pose prediction. It allows straightforward calculation while considering both the direction and magnitude of the vectors. Many distance metrics defined in the spatial domain [2, 25, 30, 58] may not be directly appropriate for the frequency domain without adaptation. For example, cosine and angular distances ignore magnitude, where the amplitude of frequency components carries significant information. Chordal and geodesic distances require normalization, can be less intuitive, and often involve more complex computations.
4.3 Inference
For evaluation, the output Wigner-D representation V is converted to an SO(3) pose in the spatial domain. Figure 4 illustrates the inference process inspired by [35, 52]. Specifically, we map the predicted Wigner-D coefficients &i from the frequency domain to 3x3 rotation matrix by querying i on a predefined SO(3) grid. To achieve this mapping, we calculate the similarities between the output vector v and the SO(3) grid P( I). These similarities are then normalized using softmax function to produce a naramarae categorical distribution P(R 1). The final 3D rotation matrix î is determined either by taking the argmax of this distribution or by applying gradient ascent [52].
To generate the SO(3) equivolumetric grids, utilize the hierarchical equal area isolatitude pixelation of the sphere (HEALPix) [24, 26], consistent with methods used in [28, 35, 44, 52, 75J. To lift the S HEALPix to SO(3) HEALPix, we create equal-area grids on the 2-sphere and cover SO(3) by threading great circles through each point using the Hopf fibration from [72].
This inference scheme effectively models objects with ambiguous orientations or symmetries by em- ploying multiple hypotheses, thereby overcoming the limitations of mingle-modality predictions [47]. In addition to joint training with distributional cross-entropy loss [52], our network can model the non-paramen parammarettiic and multi-modal distribution in pose space to address pose ambiguity and aid in modeling 3D symmetry.

---

Y E RM
flattened vector Query to pre-defined
Angmax or GradA SC
R E R Output SO(3) pose
SO(3) grid
(0(33)-quivaam pose hamonnis network
P(-1)) ER
P(R R
Figure 4: Inference time. We query the output vector of Wigner-D coefficients v/ against the predefined SO(3) HEALPix grid with a resolution of Q points. We finally obtain the SO(3) probability distribution P(R| 7), where each position represents the prohability of specific SO(3) pose."
Experiment
5.1 Implementation Details
We imput 2D RGB image I E 3 A ResNet backbone, pretrained ImageNet, extracts feature maps shape F E 2048x7x7 We then perform dimension reduction using a 1x1 convolution to obtain F" 5x2xx777 In the spherical mapper, the features mapped onto an 2 grid generated by recursion level of HEALPix on half of the sphere, and then sampled at 20 points, resulting in E 52512x20 By converting & the frequency domain, the spherical signals S E RR512x49 49 obtained. The Wigner-D representation is implemented levels. For example, the matrix coefficients at a frequency level I are represented
a flattened form across different frequency
of size (2l + 1) x (21 + 1). We use maximum frequency level of L = 6, resulting in a total size of M = 455, computed as 0 + 1) x (21 + 1). These coefficients are then flattened into single vector for the Wigner-D prediction. The spherical convolution on the 5 kernel uses an 8-dimensional hidden layer with global support to obtain intermediate SO(3) features in 88 After nonlinear activation, we finally obtain the 1-dimensional output Vp E R1xx155 using an SO(3) convolution with a locally supported filter handle rotations up to 22.5º. At inference, we employ a recursive level of SO(3) HEALPix grid with 2.36 million points, achieving precision of 1.875%, as in [28, 35].
5.2 Benchmarks
ModeeNet10-SO(3) [42] common dataset for estimating 3D rotation from single image. The images are created by rendering CAD models from the ModelNet10 dataset [67]. The dataset includes 4,899 objects across 10 categories, each image labelled with a single 3D rotation. The rotations are uniformly sampled from each CAD model. From a single CAD model, the training set comprises 100 3D rotations on SO(3), while the test set includes 4 unseen 3D rotations.
PASCAL3D+ [68] a widely-used benchmark for evaluating pose estimation in images captured in real-world settings. It includes 12 categories of everyday objects, which were created by manually aligning 3D models with their corresponding 2D images. This dataset presents challenges due to the significant variation in object appearances, the high variability of natural textures, and the presence of novel object instances in the test set. To be consistent with the baselines, we conduct training data augmentation using synthetic renderings [60].
ModeINet10-SO(3) Few-shot Views is used to evaluate the data efficiency of pose estimation models. Unlike the original ModelNer1O-SO(3) [42], we have expanded this to evaluate various amounts of training data, by setting the number of training views per CAD model to 3, 5, 10, 20, 30, 40, 50, 70, 90, and 100. This benchmark verifies the sampling efficiency of our equivariant networks. We use the same test dataset as that of ModelNet10-SOG3)
Evaluation Metrics. We calculate the angular error, measured in degrees using geodesic distance, between the network-predicter SO(3) pose and the ground-truth rotation matrix: OEnor [R, Rt) [R)) and AR RRT We adopt two commonly used metrics: the median rotation
cos 2
error (MedErr) and the accuracy within specific rotation error thresholds (Ace@) and Acc@ 30%).
We use the illustration represent the S0(3) grid P(: 1) from s2ftt github repository [56].
7

---

Rot Err. (Median)
Rot Err.
Method
Acc@ 15
Acc@30 30
Method
Ace@ 30
(Median)
Zhou al. [80]
0.251
0.504
41.1"
Zhou et al. [80]
19.2
Bréiger [3]
0.257
0.515
39.9
Bréiger [3]
20.0"
Liao et al. [42]
0.357
0.583
36.5
Tulsiani & Malik [61]
13.6
Prokudin et al. [57]
0,456
0.528
49.3
Mahendran ef al. [45]
10.1
Deng et al. [13]
0.562
0.694
32.6
Liao al. [42]
0.819
13.0"
Mohlin et al. [51]
0.693
0.757
17.1
Prokudin al. [57]
0.838
12,2
Murphy et al. [52]
0.719
0.735
21.5
Mohlin al. [S1]
0.825
11.5
Yin et al. [74]
0.751
16.1
Murphy al. [52]
0.837
10.3
Yin et al. [75]
0.742
0.772
12.7
Yin al. [75]
9,4
Klee et al. [35]
0.728
0.736
15.7
Klee al. [35]
0.872
9.89
Liu ef al. (Uni) [44]
0.760
0.774
14.6"
Liu et al. (Uni) [44]
0.827
10.2
Liu et al. (Fisher) [44]
0.744
0.768
12.2
Liu et al. (Fisher) [44]
0.863
9,9º
Howell et al. [28]
17.8
Howell et al. [28]
9,2
(ResNet-50)
0.759
0.767
15.1
Ours
0.892
8.6
ours (ResNet-101)
0.773
0.780
11.9
Table 2: Results PASCAL3D+ with
Table 1: Results on ModeINet10-SO(3), The scores
ResNet-101 backbone. Scores are averaged across all twelve classes.
have been averaged across all ten object categories.
Acc@15 15
Acc@30
Rot Err. (Median)
0.8
120
2hock it (28091 Bniilio (9l1)
100
0.6
Potlapiics (RadD)
80
(aude) cur (haad3))
0.4
0.4
60
0.2
0.2
40
20
0.0
0.0
31020
100
310
50
70
100
50
70
of Training Views
Figure 5: Experiment on ModelNet10-SO(() with few-shot training views. Results with solid lines of I-PDF [52], 12S [35], and RotLaplace [75] denote ResNet-50 backbone, while dotted lines indicate ResNet-101 backbone. Our method outperforms all metrics and reduces training views. Bascline results [35, 75] were obtained using the source code provided by the authors.
5.3 Results
Table I shows the pose estimation results on the Mode(Net10-SO(3) dataset, where model outper- forms all baselines across multiple evaluation metrics. Notably, our Wigner-D harmonics prediction network surpasses methods in non- ponpabbbabilistic rotation estimation [3, 42, 80], parametric probability distribution estimation [13, 44, 57, 74, 75], and non- non-parametric distribution prediction [28, 35, 52], by leveraging SO(3) equivariance and rotation parametrization in the frequency domain.
Table 2 shows the results on the PASCAL3D+ benchmark. Our SO(3) Wigner-D harmonics predictor achieves state-of-the-an performance on these challenging benchmarks. It demonstrates robustness to changes object appearance, real textures, and generalizes well to novel object instances. We additionally report fine-scale accuracies, i.e., Acc@3", Acc @ 5" and Acc @ 10", in Tables Al and A2, and class-wise wise evaluation results Tables A3 and A4 of appendix B.
5.4 Few-shot Training Views
Figure sillustrates the pose estimation results on ModelNet10-SOO with few-shor training views. Notably, as the number of training views from a single CAD model decreases, our model consistently achieves highest accuracy and lowest error. This performance surpasses the baselines of direct rotation regression [3, 80], parametric distribution parameters regression [75], and non-parametric distribution estimation [35, 52]. This few-shot training experiment verifies that our SO(3)-equivariant model contributes superior data efficiency and generalization to unseen rotations.

---

Method
Acc@ 15
Acc@30
Rot Err.
Loss Function
Acc@15
Acc@30
Rot Err.
Wigner (ours)
0.6807
0.6956
22.27
MSE Loss
0.6807
0.6956
22.27
Euler
0.0010
0.0072
132.56
LI loss
0.6796
0.6933
22.12
Quaternion
0.0510
0.1629
75.95
Huber loss
0,6710
0,6873
19.26
Axis-Angle
0.0124
0,0815
88.66'
Cosine loss
0,4414
0,4978
64.29
Rotmat
0,3909
0,5682
37.54"
Geodesic loss
0.0009
0.0071
132.65
Table 3: Comparison of different parametriza- tions of 3D rotations. To validate our Wigner-D representation in the frequency domain, train using various output rotation representations.
Table 4: Comparison of different loss functions. To validate our choice of MSE loss, we experi- ment with various distance functions between the predicted output and the ground truth.
5.5
Ablation Studies & Design Choices
5.5.1
SO(3) Parmetrizations
Table 3 shows results validating design choices on ModelNet10-SOO3 with 20-shot training views, using a ResNet-50 backbone. First, compare the effects of different rotation parametrizations model performance by changing the prediction head and ground gound-truth rotations, to verify our proposed Wigner-D compared to other rotation representations. For all cases, retained the backbone networks and -quiivrauv layers. The only modifications were the output prediction dimension size and the ground-truth rotation representation. Our Wigner-D parametrization outperforms Euler angles (3 dim.), quaternions (4 dim.), axis-angle (4 dim.), and rotation matrices (9 dim.). This demonstrates that frequency domain rotation re-parametrization enables accurate 3D rotations when used with the =0(3)-qquivarian spherical CNNs in the frequency domain.
5.5.2 Loss Functions
Table 4 compares various loss functions trained ModeINee1O-SO(33 with 20-shot learning using ResNet-50 backbone. While Huber and L1 losses are alternatives, they do not perform as well as MSE in our context. Cosine loss measures only angle distances between vectors, ignoring magnitude, which is an essential factor in frequency-domail applications. Geodesic loss in the frequency domain ineffective because it requires separate calculations for each frequency level of the Wigner-D matrix, potentially losing the precision of the original 3D rotation, we truncate the Fourier basis at frequency level of Therefore, we choose MSE regression loss for our design choice given its simplicity and effectiveness.
5.5.3 Ablation Studies
In Table 5, we experiment with replacing SO(3)- cquivariant layers with conventional convolu- tional layers. Specifically, we use two-layer Ix1 convolutional layers with ReLU activation and final linear layer with 455 output channels. The results indicate that CNNs without equivariant layers perform poorly, especially terms of me- dian error, suggesting that using the equivariant networks generalize better to unseen samples. Additionally, the Wigner-D prediction should be paired with an SO(3) -quuvaqquv network to en- able reliable 3D rotation prediction in the fre- quency domain.
Acc@ 15
Acc@30
Rot Err.
ours
0.6807
0.6956
22.27%
w.o equivConv
0.1056
0.1308
149.25
Random SO(3)
0.6797
0.6946
22.16'
Superfihonacci
0.6785
0.6932
22.
Table Comparison of results without the S0(3)-equivariant module and with different SO(3) grids at inference. The first group shows the results using conventional convolution instead of equivariant convolution. The second group presents the results with different SO(3) grids at inference time. 'ours' denotes the proposed model architecture.
Lastly, we evaluate the impact of different SO(3) grids by switching from a HEALPix grid to a random SO(3) grid and Fibon-iibo spirals [1], which use the same number of SO(3) rotations. Our Wigner-D harmonics predictor performs consistently, regardless the SO(3) grid sampling type at inference time.

---

SYMSOL
SYMSOL II
g
cone
cyl
let
cube
icosa
avg
sphereX
cyl0
tetX
Luigiee 2.54
2.42
2.68
2.93
2.67
1.99
-8.88
4.51
-7.64
-23.52
Ldiat [35] 3.41
3.75
3.10
4.78
3.27
2.15
4.84
3,74
5.18
5.61
Luigner Liist
4.11
4.43
3.76
5.59
3.93
2.85
6.20
6.66
5.85
6.11
Table 6: Results on SYMSOL I and I1 [52]. We report the average log likelihood on both parts of
the SYMSOL datasets. Lwigner denotes the results obtained with Wigner-D regression loss. Cdist denotes the results using the distribution loss from I-PDF [52], which are the same as the results of I2S [35]. The third row presents the results of joint training using both our regression loss and the distribution loss.
5.6 Results on SYMSOL
Table 6 shows symmetric object modeling on the SYMSOL datasets [52]. Compared to the first row and second row [35], our model with only the Wigner-D regression loss derives on sharp modalities, which can be less effective than [35] for symmetric objects in SYMSOL. I.
For clearly defined pose well. However, in other SYMSOL II scenarios, the sharp distributions produced by our model can lead to low average log likelihood scores. This metric is particularly harsh on models with sharp peaks, making them vulnerable to very low scores in some failure cases.
(e.g., SphereX in SYMSOL II), Wigner-D loss alone performs
In the third row, joint training of our method with the distribution loss [35, achieves better per- formance than the baseline [35], demonstrating its ability to model symmetric objects. These re-
sults highlight the potential of our method in han- dling complex symmetries and predicting multi- ple hypotheses. Figure 6 shows the visualization of pose distribution on the SYMSOL I and II datasets.
Most real-world objects have unique, unam- biguous poses, validating single pose re- gression method (e.g., ModelNet10-SOO3 PAS- CAL3D+). If the task needs to symmetrie cases, our model can be modeled with distribu- tion loss [35, 52].
Figure 6: Visualization of pose distribution on SYMSOL. The results are obtained by joint train- ing with both our regression loss and the cross- entropy distribution loss [35, 52].
6 Conclusion
In this paper, proposed novel method for 3D rotation estimation by predicting Wigner-D coefficients directly in the frequency domain using -Q(3)-equivariant networks. Our approach effectively overcomes the limitations of existing spatial domain parameterizations of 3D rolations, such as discontinuitics and singularities, by aligning the rotation representation with the operations of spherical CNNs. By leveraging frequency-domain regression, our method ensures continuous and precise pose predictions and demonstrates state-of-the-an performance across benchmarks like ModelNet1O-SOG3 and PASCAL3D+. Additionally, it offers enhanced data efficiency and generalization to unseen rotations, validating the robustness of SO(3) -equi--quia architectures. Our method also supports the modeling 3D symmetric objects by capturing rotational ambiguities, with further accuracy improvements achievable through joint training with distribution loss. Future work can build on this foundation to explore frequency-dome representations more effective rotation representations for 3D space, and further optimize computational efficiency.
vision tasks, develop
10

---

Acknowledgement
This work was supported by IITP grants 155-2022-11220959 Few-Shot Learning of Causal Inference in Vision and Language for Decision Making (50%), 15-2022-11220290 Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense (45%), 95--001--11199006 Al Graduate School Program at POSTECH (5%)) funded by Ministry of Science and ICT, Korea.
References
[1] Marc Alexa. Super-fibonacci spirals: Fast, low-discrepancy sampling of so (3). In Proceedings of the IEFF/CVF Conference Computer Vision and Puttern Recognition, pages 8291-8300, 2022. 9
[2] Olaya Alvarez-Tunon Yury Brodskiy, and Erdal Kayacan. Loss it right: Euclidean and riemannian metries in lcarning-based visual odometry. In ISR Europe 2023; 56th International Symposium on Robotics, pages
107-111, VDE, 2023. 6
[3] Romain Brégier. Deep regression on manifolds: a rotation case study. 2021 International Conference
on 3D Vision (3DV). pages 166-174, IEEE, 2021. 2, 8, 1, 5, 6,
[4] Michel Breyer, Jen Jen Chung. Lionel Out, Roland Siegwart, and Juan Nieto. Volumetric grasping network: Real-time 6 dof grasp detection in clutter. In Conference on Robot Learning, pages 1602-1611 1. PMLR,
2021.
[5] Mai Bui, Tolga Birdal, Haowen Deng. Shadi Albarqouni, Leonidas Guibas, Slobodan Ilie, and Nassir Navab. 6d camera relocalization in ambiguous scenes via continuous multimodal inference. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings, Part XVIII 16, pages 159-157,000 Springer. 2020. 1,2
[6]
Ruojin Cai, Bharath Hariharan, Noah Snavely, and Hadar Averbuch-Flor Extreme rotation estimation using dense correlation volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Puttern Recognition, pages 19566-14577 2021. 2
[7] Jiayi Chen, Yingda Yin, Tolga Birdal, Baoquan Chen, Lconidas J Guibas, and He Wang. Projective manifold gradient layer for deep rotation regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6646-6655, 2022. 2. 1,9
[8] Kefan Chen, Noah Snavely, and Ameesh Makadia. Wide-baseline relative camera pose estimation with directional learning.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3258-3228 3268, 2021. 2.
[9] Cobb, CGR Wallis, AN Mavor-Parker, A Marignier, MA Price, M D' Avezac, and JD McFwen. Ffffiieent generalized spherical cnns. In ICLR 2021-9th International Conference on Learning Representations. volume 9. ICLR 2021 19th International Conference on Learning Representations, 2021. 2
[10] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990 2999, PMLR, 2016. 2.3, 1
[11] Taco S Coben, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In International Conference
Learning Representations, 2018 2.
[12] Michačl Defferrard, Martino Milani, Frédérick Gusset, and Nathanaël Perraudin. Deepsphere: graph- based spherical enn. In International Conference on Learning Representations, 2019. 2,
[13] Haowen Deng. Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilie, and Tolga Birdal. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation. International Journal of Computer Vision, 1327 1654, 2022. 2, 8, 5,
[14] Thanh-Toam Do, Ming Cai, Trung Pham, and Ian Reid. Deep-6dpose: Recovering 6d object pose from a single rgb image. arXiv preprint ar Xiv: 1802. 10367, 2018. 2
[15] James R Driscoll and Dennis M Hcaly. Computing fourier transforms and convolutions on the 2-sphere.
Advances in applied mathematics, 15(2):202-255 1994, 4
[16] Carlos Esteves, Christine
Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV). pages 52 -68, 2018. 2
[17] Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin-weighted spherical enns. Advances in Neural Information Processing Systems, 33:8614-8625, 2020.
[18] Carlos Esteves, Jean-Jacques Slotine, and Ameesh Makadia. Scaling spherical cans. In International Conference on Machine Learning, pages 9396-9411. PMLR, 2023. 2
[19] Daniel Franzen and Michael Wand. General nonlinearities in so (2)-equivariant enns. Advances in Neural Information Processing Systems, 14998869998,, 2021. 6
[20] Ge Gao, Mikko Lauri, Jianwei Zhang, and Simone Frintrop. Occlusion resistant object rotation regression
from point cloud segments. In Proceedings of the European Conference on Computer Vision (ECCV)
Workshops, pages 0-0, 2018. 2
[21] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti
dataset. The International Journal of Robotics Research, 32(1 32(1)): 12311-12277 2013.
[22] Mario Geiger. Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi, Maurice Weiler. Michal Tyszkiewicz, Simon Batzner, Dylan Madisetti, Martin Uhrin, Jes
11

---

Frellsen, Nuri Jung. Sophia Sanborn, Mingjian Wen, Josh Rackers, Marcel Rød, and Michael Bailey.
Euclidean neural networks: e3nn, Apr. 2022. 8
[23] Igor Gilitschenski, Roshni Sahoo, Wilko Schwarting, Alexander Amini, Sertac Karaman, and Danicla Rus. Deep orientation uncertainty learning based bingham loss. In International conference on learning representations. 2019.
[24] Krzysztof M Gorski. Eric Hivon, Anthony Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. Healpix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622(2):759, 2005. 5, 6, 8
[25] Richard Hartley, Jochen Trumpf, Yuchao Dai, and Hongdong Li. Rotation averaging. International journal
of computer vision, 103:267-305, 2013. 6
[26] HEALPix Collaboration. Healpix: Hierarchical equal area isolatitude pixelation, n.d. Accessed: 2024-10-
22. 6
[27] Timon Höfer, Benjamin Kiefer, Martin Messmer, and Andreas Zell. hygpeeeeeddlyyypeeetortsss pre- dicting the probability distribution so (3). In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2369-2379, 2023. 2
[28] Owen Howell, David Klee, Ondrej Biza, Linfeng Zhao, and Robin Walters. Equivariant single view pose prediction via induced and restriction representations. Advances in Neural Information Processing Systems,
36, 2023. 3,5, 6. 7, 8, 9
[29] Jiahui Huang. He Wang. Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas J Guibas. Multibodysyne: Multi-body segmentation and motion estimation via 3d scan synchronization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7108-7118 18,
2021. 2
[30] Du Q Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and
Vision, 35:155-164, 2009. 6
[31] Alex Kendall and Roberto Cipolla. Geometric loss functions for camera pose regression with deep learning.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5974-5983,
2017. 2
[32] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the IEEE international conference on computer vision,
pages 2938-2946 2015. 2
[33] CG Khatri and Kanti V Mardia. The von mises-fisher matrix distribution in orientation statistics, Journal of the Royal Statistical Society Series B: Statistical Methodology, 39(1):95-066 1977. 2
[34] David Klee. Ondrej Biza, Robert Platt, and Robin Walters. Image to icosahedral projection for so (3) object reasoning from single-view images. In NeurlPS Workshop on Symmetry and Geometry Neural
Representations, pages 64-80. PMLR, 2023.
[35] David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features
for efficient pose prediction. ICLR, 2023. 2, 4, 5, 6, 7, 8, 10, 3, 9, 13
[36] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018. 2 [37] Abhijit Kundu, Yin Li, and James M Rehg. 3d-renn: Instance-level 3d object reconstruction via render- and-compare. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1559-3568,10 2018. 2
[38] Jongmin Lee, Yoonwoo Jeong, and Minsu Cho. Self-supervieed learning of image scale and orientation. In 31st British Machine Vision Conference 2021, BMVC 2021, Virtual Event, UK. BMVA Press, 2021. 1 [39] Jongmin Lee, Byungjin Kim, and Minsu Cho. Self superviid equivariant learning for oriented keypoint detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4847-4857, 2022.
[40] Jongmin Lee. Byungjin Kim, Seungwook Kim, and Minsu Cho. Learning rotation-equivariant features for visual correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 21887-21897, 2023. 1
[41] Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh, and Ameesh Makadia. An analysis of svd for deep rotation estimation. Advances in Neural Information
Processing Systems, 33:22554-22565, 2020. 2, 1
[42] Shuai Liao, Efstratios Gavves, and Cees GM Snoek. Spherical regression: Learning viewpoints. surface normals and 3d rotations on n-spheres. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pottern Recognition, pages 9759-9767, 2019. 7, 8, 3, 6,
[43] Any Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++:: Recovering poses from sparse-viee observations. In 2024 International Conference on 3D Vision (3DV), 2024. 2
[44] Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang. Baoquan Chen, and He Wang. Delving into discrete normalizing flows on so (3) manifold for prohabilistic rotation modeling. In Proceedings of the IEFF/CVF
Conference on Computer Vision and Pattern Recognition, pages 21264-21273, 2023. 2, 3, 6, 8, 4,
[45] Siddharth Mahendran, Haider Ali, and Rene Vidal. A mixed classification-eeressioo framework for 3d
pose estimation from 2d images. arXiv preprint arxiv: TP05:3800.000 03225, 2018. 8, 3, 6.9
[46] Ameesh Makadia and Kostas Danillidis. Direct 3d-rotation estimation from spherical images via generalized shift theorem. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern
Recognition, 2003. Proceedings, volume 2. pages 1I-217. IEEE, 2003. 3,
12

---

[47] Osama Makansi, Eddy 1gg. Ozgun Cicek, and Thomas Brox, Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7144-7153, 2019. [48] Octave Mariotti and Hakan Bilen. Semi-supervised viewpoint estimation with geometry-awar conditional generation. In Computer Vixion-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings.
Part II 16. pages 631-647, Springer, 2020. 2
[49] Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Viewnet: Unsupervised viewpoint estimation from conditional generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10418-10423 2021. 2
[50] Rowan Thomas McAllister, Yarin Gal, Alex Kendall, Mark Van Der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller. Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning. International Joint Conferences on Artificial Intelligence, Inc., 2017.
[51] David Mohlin, Josephine Sullivan, and Gérald Bianchi. Probabilistic orientation estimation with matrix fisher distributions. Advances in Neural Information Processing Systems, 33:4884-4893, 2020. 2, 5, 8, 3,
6. 9
[52] Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit- pdf: Non-parametric representation of probability distributions on the rotation manifold. In International Conference on Machine Learning, pages 7882-7899 PMLR, 2021. 2, 5, 6, 8, 10, 3,
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, ser-orrfoormnce deep
learning library. Advances in neural information processing systems. 32, 2019. 8
[54] Alberto Pepe, Joan Lasenby, and Pablo Chacón. Learning rotations. Matkematical Methods the Applied
Sciences, (0t3::100-121113 2024. 2,
[55]
Adrien Poulenard and Leonidas Guibas. A functional approach to rotation equivariant linearitis
for tensor field networks, In Proceedings the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2174-131833 2021. 6
[56] Matthew A, Price and Jason D. McFwen. Differentiable and accelcrated spherical harmonic and wigner
transforms. Journal of Computational Physics, submitted. 2023. 7
[57] Sergey Prokudin, Poter Gehler, and Sebastian Nowozin. Deep directional statistics: Pose estimation with uncertainty quantification. In Proceedings of the European conference computer vision (ECCV), pages
534-551, 2018. 2, 8, 3, 3, 6,
[58] A René Geist, Jonas Frey, Mikel Zobro, Anna Levina, and Georg Martius. Learning with 3d rotations,
hitchhiker's s guide so (3). arXiv e-prints, pages arXiv-24044 2404, 2024. 1, 2, 3, 6
[59] Paul-Fdouard Sarlin, Mihai Dusmanu, Johannes I. Schönherger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Mare Pollefeys. Lamar: Benchmarking localization and mapping for augmented reality. In European Conference on Computer Vision, pages 686-704. Springer, 2022.
[60] Hao Su, Charles R Qi, Yangyan Li, and Lconidas Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE international conference on computer vision, pages 2686 2694, 2015. 7.
[61] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pages 1519- 2015. 2, 8, 6, 9
[62]
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy 1lg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5038-5004 5047, 2017. 2
[63] He Wang. Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642-2651,, 2019, 1
[64] Jianyuan Wang. Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In Proceedings of the IEEE/CVF International Conference on Computer
Vision. pages 9773-9783, 2023.
[65] Maurice Weiler and Gabriele Cesa. General equiavavari stecrable cnns. Advances in Neural
Information Processing Systems, 14144433-444555 2019. 1
[66] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre- training for 3d vision tasks by vress-view completion. Advances in Neural Information Processing Systems, 35:3502-3511 2022. 7
[67] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015. 7
[68] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision. pages 75 82. IEEE, 2014.
[69] Yu Xiang. Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural
network for 6d object pose estimation cluttered scenes. arXiv preprint arXiv:1711.00 000999 2017.
[70] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Poseenn: convolutional neural network for 6d object pose estimation in cluttered seenes. Robotics: Science and Systems X/V, 2018. 2
[71] Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In International Conference on
13

---

Machine Learning, pages 24596-24614, PMLR. 2022. 6
[72] Anna Yershova, Swati Jain, Steven M Lavalle, and Julie C Mitchell. Generating uniform incremental grids (3) using the hopf fibration. The International journal of robotics research, 29(7):801-88 2010. 6,
10
[73] Li Yi. Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. ACM Transactions on Graphics (TOG), 37(6):1-15. 2018.
[74] Yingda Yin, Yingeheng Cai, He Wang, and Baoquan Chen. Fishermatch: semi-supervised rotation regression via entropy-based based filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 1164-11173. 2022. 2, 8, 9
[75] Yingda Yin, Yang Wang. He Wang. and Baoquan Chen. A laplace-inspired distribution on so(3) for probabilistic rotation estimation. In International Conference on Learning Representations (ICLR), 2023.
5, 6, 8, 3, 3.4.9
[76] Jason Y Zhang. Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cam- rays: Pose estimation via ray diffusion. In International Conference Learning Representations
(ICLR), 2024. 2.7
[77] Jason Y Zhangg Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation single objects the wild. European Conference on Computer Vision, pages 592-611, 1. Springer,
2022. 2
[78] Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion equivariant capsule networks for 3d point clouds. In European conference on computer vision, pages 1-19. Springer, 2020. 2
[79] Ellen D Zhong. Tristan Bepler, Joseph II Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. In International Conference Learning Representations,
2019.
[80] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang. and Hao Li. On the continuity of rotation represen- tations neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5745-5753, 2019. 1, 2, 8, 5, 6, 9
[81] Andrea Zonca, Leo Singer, Daniel Lenz, Martin Reinecke, Cyrille Rosset, Eric Hivon, and Krzysztof Gorski. hcalpy: equal area pixclization and spherical harmonics transforms for data on the sphere in python. Journal of Open Source Software, 4(35): 1298, Mar. 2019.
14

---

Appendix / Supplementaryy Materials
A Rotation Representations
A.1 Rotation Representation in Spatial Domain
We recommend checking out detailed overview of learning 3D rotations in [54, 58]. Rotations in both 2D and 3D spaces can be represented using various mathematical frameworks, each with its own advantages and limitations, crucial for applications in fields such as computer graphics, robotics, and deep learning. In 2D space, rotation angles can be expressed as following SO(2) representations: angle (a)) Trigonometric functions (cos(a)), sin(a)). We recommend few studies dealing with rotation-equivariance in 2D: group-equivariant networks [10, 65], and its application in image matching [38-40].
In 3D space, Euler angles in R are a representative form of SO(3) representation. Euler angles have 3 DoF consisting of three angles ,3,7 [-T,, to describe 3D rotation (roll, pitch, yaw). 3D rotation matrix can be composed of a fixed sequence rotation using the angles R(a, B, 7) = R2 (a) (2) The standard Euler angles can be divided into 2 forms: Tait-Bryan angles (a.k.a. Cardan angles) which consists of permutations of three items (XYZ, XZY, YXZ, YZX, ZXY, ZYX), and proper Euler angles which starts and ends with rotations around the same axis (ZYZ, ZXZ, XYX, XZX, YZY, YXY), total 12 possible unique sequences. Because of non-uniqueness of Euler angles, existing studies [3, 54, 80] do not encourage the Euler angles output representation for 3D rotation prediction in deep neural network.
Quaternions in 8 is for 4-dimensional complex number to represent a 3D rotation. A quaternion g is composed of one real part and three imaginary parts: g = w + xr + yj + zk, where w, J. V. 2 are real numbers and i, j. & are the fundamental quaternion units. Quaternions can prevent gimbal lock, problem that occurs with Euler angles where one degree of freedom is lost during 3D rotation.
Axis-angle in R consists of an angle 0 and an axis vector u in 3D space. To rotate a point p E R about the axis by an angle 0, you can use Rodrigues' rotation formula: p' pcos (ooss00)u) + x p0 u(u (1 - (f)). The anis-angle representation can be converted into rotation matrix or quaternion. Even has advantages of intuitive form and compact to describe 3D rotation with four parameters, the axis-angle can suffer from singularities in a scenario of angle multiples of 0 2T.
6D representation [3, 80] is a relatively newer concept compared the traditional representation like Euler angles, rotation matrices and quaternions. A rotation is described by two 3D vectors that are orthogonal to each other, and the rotation matrix can be obtained by Gram-Sehmidt orthonormalization (GSO). This representation can be directly predicted and optimized by deep networks because it avoids the non-linear constraints found in quaternions and rotation matrices. Quaternions require maintaining a unit norm, which introduces complexity in ensuring the quaternion remains normalized throughout the optimization process. Therefore, [7, 8, 80] adopt this 6D representation with GSO,
9D representation [3, 41] is direct parametrization of 3 x 3 matrix, which can be projected to SO(3) using singular value decomposition (SVD). Predicting 9D representation for rotation matrices involves orthogonality constraints, meaning the rows and columns must remain orthonormal, and determinant constraints, where the determinant must equal +1. These constraints complicate the learning and optimization process, making this representation more suitable for direct prediction and optimization by deep networks. Therefore, 9D representation is less common as a direct method predict rotation, but [7, 8, 41] use this to mitigate issues associated with discontinuous parameterizations of pose.
A.2 Rotation Representation in Frequency Domain
3D rotation the frequency domain is accomplished by manipulating spherical harmonics coefficients. Spherical harmonics Y (z) y (8, Φ) are a function defined on the surface of a sphere, where o and á are the polar and azimuthal angles, respectively. Here, represents the degree of the spherical harmonics, vn is the order. To ensure computational feasibility, we truncate the degree of harmonics to finite -== 1mux- Rotations of spherical functions can represented by matrices that operate on the coefficients of their harmonics expansion. The rotation of spherical harmonics is expressed via

---

the shift theorem [46] of spherical harmonics:
(x) Σ |m|st
D' (g))
(5)
where A, (z) denotes the spherical harmonic Y (z) after rotation by and Ag is the rotation operator. Spherical harmonics Ym (I), defined by degree 1 and order m (1 2 0, |ml S l), use a to represent spherical coordinates. The matrix Um)(() (g) forms part of the irreducible unitary representation of SO(3), showing how each harmonic is transformed under rotation. The sum over all orders n from -I to I, Einggt shows that Y a linear combination of all harmonics of degree L.
This rotation of spherical harmonics can be described by the Wigner-D matrix D pm (R), which is unitary matrix that describes the effect of rotation R on the spherical harmonics basis functions. We can rewrite um (g) Dmm (ox, (3,1) Each element of the matrix represents the amplitude and phase shift that spherical harmonic Y undergoes due to the rotation R. The rotation R can be specified by Buler angles a, B, and y in the ZYZ-axes configuration. The matrix elements D (o,, B, 7) can be explicitly expressed as:
- D'n (a, B,7) = -ima d mn (3)e
(6)
where (3) the elements of the Wigner (small) d-matrix, which depend only the angle 3 and are realev.lued. These elements capture the intermediate rotation about the Y-axis, where the angle 3 represents the tilt of the axis of rotation. To rotate spherical harmonic expansion of function f. represented as:
((0
Σ
Σ
I (0)
1=0 m=-1
we need to account for the coefficients
f'n of the expansion. The rotated coefficients f are computed:
D D (R)
(8)
where D' (R) encodes the effect of the rotation on the original coefficients. This transformation preserves the orthonormality and completeness of the spherical harmonics basis, ensuring that the rotated function fO Φ) remains valid representation of the original function f(0, Φ) under the rotation R. The expansion coefficients fr can be calculated using the original spatial coordinates (0, Φ) of the sphere surface according the spherical harmonic expansion:
fmm -
J
f(0, Φ) 4) (0, 6) sin d0, do,
(9)
where y? (8, denotes the complex conjugate of the spherical harmonic function Y YL, Φ))
B Additional Results
B.1
Results of Finer Threshold Accuracy
B.1.1 ModeINet10-SO(3)
Table A1 presents a comparison of existing methods with different backbones on the ModelNer10- SO(3) dataset', highlighting performance across multiple accuracy thresholds and median error. For the ResNet-50 backbone, our method achieves the highest accuracy at 3 (0.422) and 5" (0.640) with median error of 15.1% outperforming the existing methods [13, 28, 35, 52]. In particular, our model demonstrates significantly better performance than the strong baselines [28, 35] that use equivariant networks estimate non-parametrie SO(3) healpix distribution the finer thresholds. Compared
'We obtained the ModeINet10-SOG3 dataset from the following link: https://githin com/ http/eeeshin////// md and the PASCAL3D+
Spherical Snterrcca__egrrsseee Regressiicccoooooee.oo.. on/ http///lbb//////////////////s//t dateeeeerr/attteeeeeeeeeeeeeeeee KKooeeeeeeee dataset from the following link sttpss///vvl.. http://wll.l........o//eo stanford. edu/ edueero ptwpeetssppssaaal.ee. html.

---

Method
Backbone
3
Acc@5
Acc@ 10
Acc@ 15
Ace@ 30
Med. (9)
Deng et al. [13]
ResNet-34
0.138
0.301
0,502
0.562
0,694
31.6
Murphy et al. [52]
ResNct-50
0.294
0,534
0.680
0.719
0.735
21.5
Klee et al. [35]
ResNeet-50
0.310
0.561
0.705
0.728
0.736
15.7
Howell al. [28]
ResNet-50
17.8
ours
ResNet-50
0.422
0.640
0.744
0.759
0.767
15.1
Liao et al. [42]
ResNet-101
0.496
0.658
28.7
Mohlin al. [51]
ResNet-101
0.164
0.389
0.615
0.693
0.757
17.1
Yin el al. [75]
ResNet-101
0.447
0.611
0.715
0.742
0.772
Liu et al. [44]
ResNet-101
0.511
0.637
0.719
0.744
0.768
12.2
ours
ResNet-101
0.513
0.688
0.763
0.773
0.780
11.9
Table Al: Comparison with finer thresholds on ModeNNet10-S033) We compare additional
thresholds, including Acc@3' Acc@5' and Acc@ 10 The tables are organized by the size of the
backbone. The scores are averaged across all ten object categories.
Method
Acc@3
Acc@5"
Acc@ 10"
Acc@
Acc@
Med. (*)
Tulsiani Malik [61]
0.808
13.6
Prokudin et al. [57]
0.838
12.2
Mahendran et al. [45]
0.859
10.1
Liao al. [42]
0.819
13.0
Mohlin al. [51]
0.089
0.215
0.484
0.650
0.827
11.5
Murphy et [52]
0.102
0,242
0.524
0,672
0.838
10.2
Yin et al. [75]
0.134
0.292
0.574
0.714
0.874
9.3
Klee et al. [35]
0.134
0.270
0.580
0.716
0.867
9.6
Liu el al. [44]
0.117
0.264
0.552
0.706
0.863
10.0
Howell al. [28]
9.2
ours
0.153
0.310
0.595
0.754
0.892
8.6
Table A2: Comparison with finer thresholds on PASCAL3D+. We compare additional metries, including @c Acc@5' and Ace@ 10", adding on the Table 2. Most baselines [35, 42, 44, 45, 51, 52, 75], including ours, use ResNet-101 backbone networks in this experiment.
to [35], our model achieves 1 11.2%p. 7.9%p, and 3.9%p higher at the 3", 5%, and 10 thresholds, respectively. For the ResNet- backbone, our method also demonstrates superior performance with the highest accuracy at 3 (0.513) and 5 (0.688), and the lowest median eтror of 11.9, compared [42, 44, 51, 75]. The table includes accuracies 10,, 15a, and 30 thresholds, where our method consistently shows top performance across these metrics. The scores are averaged across all ten object categories.
B.1.2 PASCAL3D+
Table A2 presents a comparison of various methods on the PASCAL3D+ dataset, focusing on finer accuracy thresholds (Ace@3", @cc@5 and Acc@ 109) and median error. The table includes methods that primarily use the ResNet-101 backbone, comparing our approach against several existing methods. Our method achieves the highest accuracies with 0.153 at 3". 0.310 at 5% and 0.595 at 10 with the lowest median error of 8.6". Compared to the previous state-of-the-ar Yin el al. [75], our method shows performance improvements of 1.9%p at 3", 1.8% at 5%, and 2.1 %pp at 10" while also reducing the median error by 0.7"
B.1.3 ModeINet10-SOG3 Few-shot Views
Figure A1 shows the results with finer thresholds, Acc@3", Acc@5 and Ace@ 10 on the ModeINet10-SO(33 few-shot training views, which are additional results Figure 5. The graphs illustrate that our method outperforms all other methods across all metrics (Ace@3 A and Ace@ 10") and requires fewer training views achieve high accuracy, even at finer thresholds. This shows that our model is capable of more precise pose estimation with less number of training data, proving the data efficiency our -Quiiaaqui harmonies pose estimator. Baseline results [35, 75] were obtained using the source code provided by the authors.

---

Acc@3
Acc@5
Acc@
0.8
0.5
0.6
0.6
0.3
0.4
0.4
0.2
0.2
0.2
65 650tddtt
RelappcccRoo0 (Rodd
eet )osio0)
0.0
0.0
0.0
31020
50
70
100
310
100
31020
100
# of Training Views
Figure Al: Results with finer thresholds on M0)0)Net10-SO(3) few-shot training views. Results with solid lines denote ResNet-50 backbone, while dotted lines indicate ResNet-101 backbone. Our method outperforms all metrics and reduces training views even at finer thresholds. For comparison, the I2S (ResNet-50) model [35] is shown with blue line, and the RotLaplace (ResNet-50 and ResNet-101) models [75] are depicted with purple solid and dashed lines, respectively.
B.2 Per-Category Results
B.2.1 ModelNet-SO(3) Categorical Results
Table A3 provides a comprehensive comparison across 10 object categories on the ModeINet10-SO(3) dataset, including Bathtub, Bed, Chair, Desk, Dresser, TV Monitor, Night Stand. Sofa, Table, and Toilet. Each image is labeled with a single 3D rotation matrix, even though some categories, such desks and bathtubs, may have ambiguous poses due symmetry. This poses challenge for methods that cannot handle uncertainty over orientation. However, in terms of accuracy at 15,, our model consistently achieves the best performance in the desk and bathtub categories, demonstrating robustness against pose ambiguity and symmetry. The table is divided into sections for ResNet-50 and ResNet-101, indicating different network architectures used. For ResNet-50, our method achieves the lowest average median error of 15.1%. with particularly strong performance in 9 categories: bed (2.7%), chair (3.89), desk (4.2%), dresser (2.7%), monitor (2.7)), night stand (3.4%), sofa (7.2%), and toilet (3.0)) For ResNet- 101, our model demonstrates the lowest average median error of 11.99 Although Liu et al. (Uni.) [44] generally obtain better results in terms of median error with the ResNet- backbone, our model outperforms Liu et al. (Uni.) [44] on Acc @ 15" in most cases. This indicates that our model estimates poses correctly at a finer level of detail.
In terms of accuracy at 15", our method achieves the highest average accuracy of 0.759 for ResNet-50, with best performance in all categories. For ResNet-101 at 15,, our model leads with an average accuracy of 0.773, achieving state-of-the-ar performance in out of 10 categories. In terms of accuracy 30%, our method achieves the highest average accuracy of 0.773 for ResNet-50, with the best results in 9 out of 10 categories. For ResNet-101 at 30", our model maintains the highest average accuracy of 0.905, with strong results categories such as Chair, Desk, TV Monitor, and Sofa. Overall, our equivariant harmonics pose estimator demonstrates superior performance across both ResNet-50 and ResNet-101 architectures in terms of both median error and accuracy at different angles. This highlights its effectiveness and robustness across various object categories in the ModeINet10-Sonn 10-SO(3) dataset, consistently outperforming other recent methods in most categories and metrics.
B.2.2 PASCAL3D+ Categorical Results
Table A4 presents comprehensive comparisons on the PASCAL3D+ dataset across 12 categories (Aeroplane, Bicycle, Boat, Bottle, Bus, Car, Chair, Dining Table, Motorbike, Sofa, Train, TV Monitor) using median eтror and Acc@30 30 metrics. This benchmark is challenging due to significant variations object appearances, high variability of natural textures, and the presence of novel object instances in the test set. Our method demonstrates superior performance with the lowest average median emor of 8.6 and the highest average accuracy of 0.892, excelling in categories such as "aero," "bike," " "car," "chair," "table," and "mbike" in median error, and achieving best performance in 8 out of 12 categories

---

ModeINet10-5OGG) categories
Methods
Avg.
Bath
Bed
Chair
Desk
Dress
Tv
Stand
Sofa
Table
Toilet
Prokudin et al. (2018) (57)
49.3
122.8
3.6
9.6
17.2
29.9
6.7
73.0
10.4
115.5
4.1
Zhou et al. (2019) [80]
41.1
103.3
18.1
18.3
51.5
32.2
19.7
48.4
17.0
88.2
13.8
ResNet-S0
Deng et (2022) [13]
32.6
147.8
9.2
8.3
25.0
11.9
36.9
10.0
58.6
8.5
Brégier (2021) [3]
39,9
98.9
17.4
18.0
50.0
31.5
18.7
46.5
17.4
86.7
14.2
Median error (*)
Murphy al. (2021) [52]
21.5
16L.0
4.4
5.5
7.1
5.5
5.7
Klee et al. (2023) [35]
16.3
124.7
3.1
4.4
4.7
3.4
4.4
4.1
3.0
7.7
3.6
Howell al. (2023) [28]
17.8
124.7
5.5
6.9
Ours
15.1
117.4
2.7
3.8
4.2
2.7
3.8
3.4
2.5
7.2
3.0
Liao al. (2019) [42]
36.5
113.3
13.3
13.7
39.2
26.9
16.4
44.2
12.0
74.8
10.9
ResNet-0 ReNeet100
Moblin al. (2020) [51]
17.1
89.1
4.4
5.2
13.0
6.3
13.5
25.8
4.0
Yin et al. (2023) [75]
12.2
85.1
2.3
3.4
5.4
2.7
3.7
4.8
2.1
9.6
2.5
Liu (2023) (Umi.) [44]
14.6
124.8
Liu et al. (2023) (Fisher.) [44]
12.2
91.6
1.8
3.0
5.5
2.0
3.2
4.3
1.6
6.7
2.1
Ours
11.9
91.9
2.3
3.3
3.2
2.1
2.6
Prokudin al. (2018) [57]
0,456
0.114
0.822
0.662
0.023
0,406
0.704
0.187
0.590
0,108
0.946
Deng et al. (2022) [13]
0,562
0.140
0.788
0.800
0,345
0.563
0.708
0.279
0.733
0.440
0.832
Murphy al. (2021) [52]
0.719
0.392
0.877
0.874
0.615
0.687
0.799
0.523
0.945
Acc@15
Klee et al.* (2023) [35]
0.736
0.414
0.845
0.888
0.641
0.672
0.793
0.654
0,900
0.548
0.957
Ours
0.759
0.425
0.891
0.918
0.704
0.743
0.833
0.646
0.935
ResNet- 101
Mohlin al. (2020)
0.693
0.322
0.882
0.881
0.536
0.682
0.790
0.516
0.919
0,446
0.957
Yin et al. (2023)
0,741
0.390
0.902
0.909
0.644
0.722
0.815
0.590
0,934
0.521
0.977
Liu (2023) (Uni.) [44]
0.760
0.402
0.896
0.927
0.704
0.753
0.843
0.561
0.975
Liu et al. (2023) (Fisher.) [44]
0,744
0.439
0.890
0.909
0.638
0.715
0.810
0.585
0,938
0.535
0.978
Ours
0.773
0.453
0.903
0.932
0.726
0.763
0.842
0.654
0.540
0.980
ResNec-50
Prokudin al. (2018) (57)
0.528
0.175
0.847
0.777
0.061
0,500
0.788
0.306
0.673
0,183
0.972
Deng et al. (2022) [13]
0.694
0.325
0,880
0.908
0.556
0.649
0.807
0.466
0.902
0.485
0.958
Murphy al. (2021) [52]
0.735
0,410
0.883
0.917
0.629
0.688
0.832
0.570
0.531
0.967
Acce 30
Klee et al." (2023) [35]
0,736
0.427
0.848
0.915
0.642
0.672
0.819
0.565
0,902
0.555
0.964
Ours
0.767
0.434
0.894
0.939
0.708
0.743
0.940
0.529
0.979
ResNec-101 101
Mohlin al. (2020) [51]
0.757
0.403
0.908
0.935
0.674
0.739
0.863
0.614
0,944
0,511
0,981
Yin et al. (2023) [75]
0,770
0.430
0.911
0.940
0.698
0.751
0.869
0.625
0.946
0.541
0.986
Liu et (2023) (Uni.) [44]
0.774
0.419
0.904
0.946
0.722
0.766
0.868
0.617
0.948
0.567
0.982
Liu et al. (2023) (Fisher.) [44]
0,768
0.460
0.898
0.934
0.694
0.738
0.859
0.615
0.948
0.544
0.987
0.780
0.459
0.905
0.950
0.728
0.763
0.871
0.654
0,943
0,544
0.983
Table A3: Evaluation on MedelNet10-SO(3) by method across different categories. denotes
reproduced results from the source code provided by authors.
in accuracy at 30", The results indicate that different methods have varying strengths across different categories, with our -Quui-equivariant pose harmonics estimation method consistently outperforming others in both accuracy and error metrics, demonstrating its robustness and efficacy.
B.3 Impact of SO(3) Discretization Sizes and Continuity of Rotations
Table A5 reports the effect of varying the grid size (Q) on performance for the ModelNet10 benchmark. We observe comparable results in common evaluation metrics, such as Accuracy at 15 degrees (Ace@ 15) and 30 degrees (Ace@30), even with lower grid resolution (Q 4.6K). A higher resolution grid (18.87M) improves performance under stricter evaluation thresholds. With our chosen grid size of Q 2.36M, the model achieves strong performance sufficiently, particularly for low-threshold metrics like Acc@3. Table Al provides additional comparisons to baseline methods.
Therefore, we carefully claim that our learning method focuses on continuous rotations. Our model directly learn the Wigner-D coefficients, which are derived from 3D rotations (Euler angles), without any discretization during the training phase. During inference the use of the SO(3) HEALPix grid serves two purposes: 1) To convert SO(3) rotations from the frequency domain to the spatial domain, and 2) To address pose ambiguity by providing multiple solutions. As a result, we obtain distribution with very sharp modality. By taking the argmax of this distribution, we achieve sufficient precision in 3D orientation estimation, specifically around 1.5a
Maintaining continuity in rotations allows our method to deliver more accurate and precise pose predictions, giving us a clear advantage over the methods that experience precision loss from discretization during training. As a result, we achieve consistently high accuracy across different levels of discretization.
5

---

PASCAL3D+ categories
Method
avg.
bike
boat
bottle
bus
chair
table
mbike
sofa
train
Zhou et al. (2019) [80]
19.2
24.7
18.9
54.2
11.3
8.4
9.5
19.4
14.9
22.5
17.2
11.4
17.5
Brégier (2021)
20.0
27.5
22.6
49.2
11.9
8.5
9.9
16.8
27.9
21.7
12.6
10.2
20.6
Liao al. (2019) [42]
13.0
13.0
16.4
29.1
10.3
4.8
6.8
11.6
12.0
17.1
12.3
8.6
14.3
Mohlin et al. (2020) [51]
11.5
10.1
15.6
24.3
7.8
3.3
5.3
13.5
12.5
12.9
13.8
7,4
11.7
Median ervor (*)
Prokudin et (2018)
12.2
9.7
15.5
45.6
2.9
13.1
9.1
4.3
12.0
Tulsiani & Malik (2015) [61]
13.6
13.8
17.7
21.3
12.9
5.8
9.1
14.8
15.2
14.7
13.7
8.7
15.4
Mahendran al. (2018) [45]
8.5
14.8
20,5
9.3
11.7
Murphy et (2021) [53]
10.3
10.8
12.9
23.4
8.8
3.4
5.3
10.0
7.3
13.6
9.5
6.4
12.3
Yin al. (2023) [75]
8.6
11.7
21.8
6.9
2.8
7.9
9.1
12.2
8.1
6.9
11.6
Liu et (Uni.) (2023) [44]
10.2
8.9
15.2
24.9
6.9
2.9
4.3
8.7
10.7
12.8
9.3
6.3
11.3
Liu et (Fisher) (2023) [44]
9.9
9.6
12.4
22.7
7.5
3.1
4.8
9.2
8.6
13.5
8.6
6.7
11.6
Klee (2023) [35]
9.2
12.7
21.7
3.3
9.5
9.3
11.5
10.5
7.2
10.6
Howell et al. (2023) [28]
9.2
9.3
12.6
17.0
8.D
3.0
9.4
6.7
11.9
12.1
6.9
9.9
8,3
12.1
17.2
2.9
5.5
10.4
9.3
7.1
10.7
Liso et (2019) [42]
0.819
0.82
0.77
0.55
0.93
0.94
0.94
0.85
0.61
0.80
0,95
0,83
0.82
Moblin et al. (2020) [51]
0.825
0.80
0.75
0.53
0.95
0.96
0,96
0.78
0.62
0.87
0.93
0.77
0.84
Prokudin al. (2018) [57]
0.838
0.89
0.83
0.46
0.96
0.93
0,90
0.80
0.76
0.90
0,90
0.82
0.91
Tulsiani Malik (2015) [6]]
0.808
0.81
0.77
0.59
0.96
0.98
0.89
0.80
0.62
0.88
0.92
0.80
0.90
Acc
Mahendran al. (2018)
0.87
0.81
0.97
0.92
0.67
0.85
0.97
0.82
0.88
Murphy al. (2021) (52]
0.837
0.81
0.85
0.56
0.93
0.95
0.94
0.87
0.78
0.85
0.88
0.78
0.86
Yin (2023) [75]
0.90
0,97
Liu et al. (Uni.) (2023) [44]
0,827
0.83
0.78
0.56
0.95
0.96
0.93
0.87
0.62
0.85
0,90
0.81
0.86
Liu et (Fisher) (2023) [44]
0.863
0.89
0.89
0.55
0.98
0.95
0.94
0.67
0.91
0.95
0.82
0.85
Klee al." (2023) [35]
0,851
0.89
0.84
0.60
0.90
0.98
0.95
0.86
0.71
0.90
0,93
0,83
0,82
ours
0.892
0.92
0.88
0.65
0.92
1.00
0.99
0.94
0.86
0.93
0.93
0.84
0.85
Table A4: Evaluation on PASCAL3D+ by method across different categories.
denotes repro-
duced results from the source code provided by authors.
# of points
width of a bin
Rot
Acc3
Acc50
Acc10
Acc15
Acc30
Err.
72
60%
0.000
0.002
0.016
0.055
0.415
45.9
576
30"
0.002
0.014
0.122
0.396
0.765
27.1
4.6K
15.
0.026
0.116
0.615
0.750
0.767
19.7
36.9K
7.5
0.150
0.464
0.734
0.757
0.766
17.1
294.9K
3.75"
0.343
0.611
0.742
0.758
0.766
15.8
2.36M
1.875
0.424
0.641
0.746
0.760
0.767
14.7
18.87M
0.938
0.443
0.646
0.746
0.759
0.768
15.0
Table A5: Evaluation by changing the size of the SO(3) grid inference. analyze the sensitivity
of discretization on precision (Q of Fig. 4), we vary the recursion levels of the SO(3) HEALPix from to We use ResNet-50 backbone on ModeINet10-SO(3)
inference
Acc@1 15c
Acc@ Rcc@:300
Rot Err.
Backbone
Acc@ 15c
Acc 30"
Rot Err.
w/ argmax
0.7576
0.7651
12.79
ResNet-50
0.6807
0.6956
22.27
w/ grad ascent
0.7591
0.7660
12.43
ViT
0.6384
0.6525
40.66
Table A6: Comparison of inference methods on pose distribution. We compare argmax and gradient ascent in the predicted distribution.
Table A7: Results of using transformer instead of convolution. We train our models by replacing the backbone with Vision Transformer (ViT).
B.4 Discretised distribution on SO(3)
Table A6 shows the evaluation results using gradient ascent the predicted SO(3) pose distribution in ModelNet10-SOG3) to fully exploit the distribution prediction in Sec. 4.3 during inference time. While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.
B.5 Transformer instead of convolution
Table presents the results when transformers are used as the backbone network instead of the convolutional feature extractor, trained ModeINet1O-SO(3) with 20-shot learning. We trained the

---

model using a Vision Transformer (ViT) backbone pre-trained by the geometric task of cross-view masked image modeling [66]. Although the ViT is heavier and requires longer training time (1.4x), its performance actually declines. This suggests that convolutional image feature extractor may still be more effective for this 3D orientation estimation task.
B.6 Searching Frequency Level L
Table A8 presents the impact of varying the maximum frequency level by truncation for efficient SO(3) group convolutions pose prediction aceuracy and median error. The results show that as L increases from to 5, there is consistent improvement in accuracy metrics, The optimal performance observed at L 5.
Beyond this point, additional frequency levels do not con- tribute to improved accuracy and can even degrade per- formance. When L > 5, the accuracy does not improve
L
Acc@15
Acc@30 30
Rot Err.
1
0.3637
0,5598
38.76
significantly and starts fluctuate, with rotation error
2
0.5850
0.6839
29.00'
0,6302
0.6972
26.34
remaining relatively low up to L = 10. However, at
0.6670
0.6998
23.639
L = 11, accuracy starts to decline more noticeably. For 2 12, there is a sharp decline in performance, with
0.6816
0.7014
22.23
6
0.6807
0.6956
22.27
Acc @ 15 dropping 0.5815 and continuing decrease,
7
0.6731
0.6870
21.69º
Acc@ 30 33" following a similar trend, and rotation error increasing significantly over 55%. We infer that high fre- quencies do not improve performance despite the increase learnable parameters because they lead to overfitting to frequeryy noise. This overfitting occurs when the eigh-frequeccy model captures irrelevant noise and pat- terns in the training data, reducing its generalizability to new, unseen data.
8
0.6724
0.6848
21.46'
0.6761
0.6884
25.34
0.6701
0.6817
21.89
0.6625
0.6736
21.51
12
0.5815
0.5956
55.23'
13
0.5390
0.5586
55.53
14
0.5228
0.5427
58.38
Table A8: Results of various num-
ber of maximum frequency in
In conclusion, including higher frequencies (L > 6) ap- pears introduce more noise overfitting, leading decreased accuracy and increased rotation error. However, we choose a maximum frequency level L for a fair comparison to [35], and to balance efficiency and accuracy.
ModelNet10-SO(3) 20-shot training views.
B.7 Justification of the Spherical Mapper
The spherical mapper in Sec. 4.1 maintains the geometric structure of the image when project- ing onto the 52 sphere, detailed in [35]. This method involves lifting the 2D image onto the sphere and converting spherical points using spherical harmonics. Table A9 shows that the spherical mapper outperforms simple Fourier transforms on 2D feature maps.
ModelNetlO-SOOG
projection mode spherical mapper
Acc@15
Acc@:30
Rot Err.
0.7590
0,7800
15.11
MI.P mapper
0.7396
0.7457
14.98
PASCAL33D
Spherical mapper MLP mapper
0.7535
0.8918
8.64
0.7283
0.8745
9.11
ModeINe 20-Shots
Spherical mapper MLP mapper
0.6807
0.6956
22,27
Using depth information from methods like DepthAnythingy2 for 3D lifting is a good idea and can enhance geometric accuracy. Addition- ally, centroid ray regression has been explored in research such as [76]. However, incorporating external depth modules increases computational costs and broadens research scope, so consider this for future work.
0.6446
0.6567
44.52"
Table A9: Validating the design choice of the spherical mapper. The 'MIP mapper' denotes the Fourier projection, which directly maps im- age features to harmonics using an MLP, and the 'spherical mapper' denotes our choice of ortho- graphic projection [35].
B.8 OOD Evaluation
Evaluating out-of-distribution (OOD) performance is generally not the primary focus in the context of this 3D orientation estimation task. However, we have conducted OOD generalization experiments using proposed method by training the model on different datascts, between ModelNet10 and
7

---

Training Dataset
Evaluation Dataset
Acc@ 15
Acc@
Rot. Ert.
ModelNettSOO33)
ModeINet-SOG33
0.7590
0.7668
15.08
ModeINet-SO(()
PASCAL33D+
0.0004
0.0019
112,98
PASCAL3D+
ModelNet-SO(3)
0.0015
0.0086
130.44
PASCAL3D+
PASCAL3D+
0.7495
0.8965
8.92"
Table A10: Cross-dataset evaluation for validating out-of-distributioo generalization on ModeINet10-SO(3) and PASCAL3D+ datasets.
Klee et al. [35]
Yin et al. [75]
Liu et al. [44]
ours
Time (sec. / frame)
0.0286
0.0171
3.9960
0.0109
GPU memory (GB)
1.156
0.912
1.130
5.172
Table A11: Comparison of computational cost. We compare the inference time one image and
GPU memory consumption ModelNet1O-SC0 test split. To measure the inference time, average the results of total 18,160 samples of ModelNet10-SO(3) test split.
PASCAL3D+.
The results are presented in Table A10. the results indicate, the model does not perform well when evaluated on an -iit--i-iittittioon dataset. Nevertheless, we recognize this as an important area for future research.
B.9 Computational Cost Analysis
Table All presents detailed comparison of computational cost, focusing on both inference time and GPU memory consumption. The comparison includes the recent baselines: Image2Sphere (2023) [35], RotationLaplace (2023) [75], and RotationNNrmFFow (2023) [44]. The evaluation is based on the ModelNet10-SO(3) test split, averaging the results from a total of 18,160 samples. We use machine equipped with an Intel i7-8700 CPU and an NVIDIA GeForce RTX 3090 GPU, utilizing batch size of 1. The key metrics presented the table are the inference time per frame (in seconds) and the GPU memory consumption (in gigabytes).
Our model demonstrates the best inference time of 0.0109 seconds per frame, significantly outper- forming other models in terms of speed. This efficient inference time translates to approximately 92.5 frames per second (FPS) for an image size of 224x224, making our model suitable for real-time applications. However, this performance comes at the cost of higher GPU memory consumption, which is recorded 5.172 GB. Since our model performs all operations on the GPU, we achieve temporal advantage in inference time, despite having many overlapping modules with Klee et al. [35], whose model performs some computations on the CPU. In summary, our model achieves the best inference time, facilitating real-time application potential, by trading off increased GPU memory consumption.
B.10 Experiment of Statistical Significance
Table A12 presents the results of 5-trial experiment to evaluate the training sensitivity of our models, with ResNet-50 and ResNet-101, on the ModeINet1O-SO33) 20-shot training views. The table lists individual trial results, along with the average (μ) and standard deviation (a) for each metric. The standard deviation values (α) for both backbones are relatively small across all metrics, suggesting that the models yield consistent results over multiple trials. For instance, the standard deviation of is 0.0052, which are both quite low. This
Acc @ for ResNet-50 0.0047, and for ResNet-1011. low variance indicates that the training results are stable and reproducible. These findings highlight the robustness and reliability of the training process and the effectiveness ResNet- 101 for the given
task.
C Training Details
We utilize the e3nn library [22] for S2 and SO(3) convolutions for efficient handling both Fourier and inverse Fourier transforms, healpy [24, 81] for HEALPix grid generation, and PyTorch [53]

---

Trial
Acc@3
Acc@5
Acc@ 10
Acc@ 1.5
Acc@ 30
Med Err.
1
0.2299
0,4632
0.6490
0.6807
0.6956
22.27
2
0.2336
0,4687
0,6503
0,6793
0.6944
21.97
ResNer-S 50
3
0.2292
0,4691
0,6522
0.6827
0.6963
21.00*
0.2375 4
0.2252
0,4593
0.6465
0.6802
0.6949
28.06
0.4699
0.6535
0.6847
0.6978
18,93*
0.2311
0,4660
0.6503
0.6815
0.6958
22,49
0.0047
0.0046
0.0027
0.0022
0.0013
3.92
I
0.3108
0,5418
0,6877
0.7099
0.7214
20.91
2
0.3115
0.5414
0.6848
0.7076
0.7184
20.77
ResNer- 101
0.3054
0.5413
0.6843
0.7067
0.7188
21.61
0.3158
0.5451
0.6875
0.7101
0.7216
20,87
0.3027
0,5366
0.6842
0.7088
0.7204
16,83
0.3092
0,5412
0.6857
0.7086
0.7201
20.20'
0.0052
0.0030
0.0018
0.0015
0.0015
1.91
Table A12: Experiment of 5-trials training of our model for statistical significance on
ModeINe(10-SO(3) 20-shot training views. / denotes the average, and denotes the standard deviation.
for model implementation. We use a machine with an Intel i7-8700 CPU and an NVIDIA GeForce RTX 3090 GPU. With a batch size of 64, our network trained for 50 epochs on ModelNet1O taking 25 hours, and for 80 epochs on PASCAL3D+ taking 28 hours. We start with an initial learning rate of 0.1, which decays by a factor of 0. every 30 epochs. We use the SGD optimizer with Nesterov momentum set at 0.9. Unlike baselines that encode object class information via embedding layer during training [35] and both training and testing [44, 75], our model does not use class embeddings, maintaining class-agnostic framework during both training and testing. Additionally, train single model for categories in each dataset, unlike [7]. which trains separate models for each class.
D Baselines of Single- View Pose Estimation
We compare our method against competitive single-view SO(3) pose estimation baselines including regression methods and distribution learning methods. Zhou al. [80] predict 6D representations using Gram- Schmidt orthonormalization processes for 3D rotations, analyzing the discontinuities in rotation representations. Brégier [3] extends deep 3D rotation regression with differentiable Pro- crustes orthonormalization. which maps arbitrary inputs from Euclidean space onto a non-Euclidean manifold. Tulsiani and Malik [61] train CNN using logistic loss to predict Euler angles. Mahendran et al. [45] predict three Euler angles using classification-rerree reegressinnnnn loss to estimate fine-post pose while modeling multi-modal pose distributions. Liao er al. [42] also predict Euler angles using Dassification-regeesss regression output.
loss by introducing a spherical exponential mapping on n-spheres at the
On the other hand, the other baselines are generating probability distributions for estimating SO(3) pose. Prokudin et al. [57] represents rotation uncertainty with a mixture of von Mises distributions over each Euler angle, while Mohlin et al. [51] predicts the parameters for a matrix Fisher distribution. Deng al. [13] predict multi-modal Bingham distributions. Murphy al. [52] trains an implicit model to generate a non- non-parametric distribution over 3D rotations. Yin et al. [74, 75] predict the parameter of SO(3) parametric distribution using -Fishrr distribution and rotation Laplace distribution, respectively. Klee er al. [35] predicts non parameerii distribution with equivariant feature prediction by orthographic projection, and Howell el al. [28] extends to construct neural architectures to satisfy SO(3) equivariance using induced and restricted representations. Liu el al. [44] use discrete normalizing flows for rotations to learn various kinds of distributions on SO(3). Results are from the original papers when available.

---

E
Qualitative Results
Figures A2 and A3 show qualitative results randomly selected from ModelNet10-SOG3) and PAS- CAL3D+, respectively. For visualization, we display distributions over SO(3) as proposed in I-PDF [52]. To illustrate the SO(3) distribution, we use the Hopf fibration to visualize the entire space of 3D rotations [72]. This approach maps each point on great circle in SO(3) to point on the discretized 2-sphere and uses a color wheel to indicate the location on the great circle. Essentially, each point on the 2-sphere represents the direction of canonical z-axis, and the color represents the tilt angle around that axis. To depict probability score, we adjust the size of the points on the plot. Lastly, we present the 2-sphere's 's surface using the Mollweide projection.
Comparison of pose visualization. Figures A4 and A5 show a comparison of pose visualizations on ModeINet10-SOC3) and PASCAL33++, respectively. This visualization method is the same to those used in Figures A2 and A3. We compare our model to the 12S [35] baseline. The numbers next to "Err" above the input images represent the error in degrees between the model's predicted pose and the ground truth (GT) pose. These results demonstrate that our model provides more accurate and precise pose estimations, even in cases where the I2S baseline fails. Additionally, benchmark, which includes objects captured in real-world scenarios, our model consistently shows correct pose estimations, particularly in challenging scenarios where the I2S baseline struggles.
the PASCAL3D+
F
Limitation
Our proposed method significantly advances 3D rotation estimation accuracy; however, a notable challenge in pose estimation the issue of pose ambiguity. partieularly for objects with symmetrical features or those viewed from certain angles, e.g., bathtub category in ModelNet10-SOCC) high accuracy, our method can suffer from significant errors due to the loss of spatial information when projecting 3D data onto spherical harmonics. Future work could integrate additional contextual or spatial information to mitigate these ambiguities, improve reliability, and enhance the model's robustness in diverse scenarios. Additionally, while the mathematical rigor of using spherical harmonics and Wigner-D coefficients supports the model's success and improves interpretability through equivariant networks, further exploration is needed to make the model more interpretable. Finally, the computational cost associated with Wigner-D coefficients and SOu))-equivarian networks should be improved to enhance practicality for real-time applications and deployment on devices with limited processing power.
Despite
G Broader Impacts
The method proposed this paper has several potential positive societal impacts. First, can enhance robotics and automation. Accurate 3D pose estimation is crucial for these fields, and improved accuracy can lead to more efficient and safer robotic systems in manufacturing, healthcare, and service industries. Second, it can significantly advance augmented reality (AR) applications by providing more precise alignment of virtual objects with the real world, which can be beneficial in education, gaming, and industrial design. Third, the method can improve autonomous vehicles, which rely on precise 3D pose estimation to understand their environment, contributing to safer and more reliable autonomous driving systems. Finally, in medical imaging, accurate pose estimation can improve the analysis and interpretation of complex 3D data, aiding in diagnosis and treatment planning.
However, the paper also suggests potential negative societal impacts. Improved pose estimation techniques could be used in surveillance systems, leading to privacy concerns if deployed without proper regulations and oversight. Enhanced 3D pose estimation could be exploited to create more realistic deepfakes, contributing to the spread of disinformation and manipulation. Deployment of these technologies could inadvertently reinforce existing biases if the training data is not representative of diverse populations, leading unfair treatment of specific groups in applications like security and hiring. Additionally, the misuse of accurate pose estimation in security-sensitivn areas, such as military applications or unauthorized monitoring, could pose significant risks.
To address these potential negative impacts, several mitigation strategies could be implemented. Controlled release of models and methods to ensure ethical and responsible use is one approach.
10

---

300
Em: 8.17
Em: 5.92
Em: 3.22
Er: 1.94
3.08
Em: 1.17
Em 4.67
Figure A2: Randomly selected qualitative results of pose estimation ModelNet10-SO(() using our SO(3) equivariant harmonics pose estimator. The error value, indicating the difference between the estimated and ground truth orientations in degrees, is labeled above each plot. Most images with clearly posed objects in the input image show an eтror 10 less, demonstrating high accuracy of the pose estimation algorithm. The example in the first row, second column, shows significant error of 179.70". This high error is attributed to the ambiguity in pose information, as the projection of the 3D object causes loss of spatial information, resulting in larger discrepancies between the ground truth and estimated poses. Other examples with low errors, such as the top-left corner (Err: 1.36%) and second row, second column (Err: 1.94%), indicate successful pose estimations.
Regular audits to ensure the training data and algorithms do not propagate biases are also crucial. Implementing robust privacy protection measures can safeguard individual privacy applications involving surveillance. Developing complementary technologies to detect and mitigate the effects of deepfakes and other forms disinformation another necessary step.
11

---

Frr: 4,97
Ert: 6.42
Fi 11.76
Er 6.01
6,52
Figure A3: Randomly selected qualitative results on PASCAL3D+ using our SO(3) equivariant pose harmonics estimator. The error value, indicating the difference between the estimated and ground truth orientations in degrees, is labeled above each plot. Most images with clearly posed objects in input image show an error of 10 or less, demonstrating high accuracy the pose estimation algorithm. For example, the airplane in the first row, second column, shows low error of 2.540, indicating precise pose estimation. However, some objects, like the monitor (Err: 38.50"), airplane (Err: 28.82%) in the fifth row, cxhibit larger errors, possibly due to posc ambiguity in the input image by symmetry. The variability in errors across different objects highlights the our model's 's performance variability depending the object's shape and the clarity of pose information in the input image.
12

---

Figure A4: Comparison of pose visualization with 12S [35] baseline on Ner0SNer
I2S
Emn mm-22.00
Em: 1.41
Lr:: 2:00
m
OWIS
tim 0.04
Ein 0.00
Lam 0.22
0.24
Figure A5: Comparison of pose visualization with I2S [35] baselie on PASCAL3D4.
I2S
3.07
STr. 20
EIn 79
OULES
Ei 0.16
EH TE
Er 05
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We give an overview and scope of the SO(3) equivariant pose harmonics predictor in section 1. At the introduction section, summarize contribution, the prediction of wigner-D coefficients via 3D rotation reparametrization to the frequency domain, is a first attempt.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes we discuss our limitation in section F. We also discuss our experiments on training sensitivity (section B. 10) and limitations in the frequency domain (section 6).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we discuss the necessary prior knowledge in the preliminary part of the text (section 3) and develop it through formulas in the method part (section 4). More detailed assumptions and proofs are discussed in sections ??, A of the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we have described the implementation details section 5.1, including the detailed training method and specific configuration to reproduce the experimental results. We have also described the benchmark settings and dataset split. (section 5.2)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes. provide sources and links the publicly available data, the ModelNet CoreleeerO--OO10 (0SO(33 dataset and the PASCAL3D+ dataset in section B. We will also make our code publicly available after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
htlpiii/guides//ooeeettteee wtt.SSmbbiiiiooooicocooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
httppsccccpppcccceeeeeeeeeecsece htt///uiissee/ee http:/uuddeeeoeduuiiieeeeeee /CodeSoomiisseeeeeee ssisssolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, we have described the implementation details such as data splits, hyper- parameters, type of optimizer in section 5.1. We also reported experiments to validate our design choice section 5.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
16

---

Yes, we describe our experiments for statistical significance in section B.10. Multiple experiments show that our experiments are not wrong or accidental. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We have written information about compute resources such as CPU and GPU specification, training time for each dataset section. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics bttps://neuus
hccppbbicc/thiccsuuut ines?
Answer: [Yes]
Justification: Yes, we have reviewed the Code of Ethics on the NeurIPS homepage and confirm that we are in compliance with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
17

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, discuss the possible positive and negative impacts of this study in section G: broader impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do use the ImageNet pretrained model, but we do not discuss it in the paper we do not believe it has high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we are attributing any code, libraries, or illustrations footnotes, or in the text at the section 1. and following the licenses.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We are not introducing any new assets in this paper, but we will be releasing the code publicly to aid in reproduction after acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "zF1RgIDxkf", "reviewer": "Reviewer_iMp8", "summary": "In this work authors predicts SO(3) poses for objects by predicting Wigner-D coefficients in frequency space. Similar to other work [1], it first lifts 2D features to 2-sphere using pre-defined grid and orthographic projection to sphere using this grid, convert them to frequency domain, applies SO(3)-equivariant spherical convolution on top to predict a vector of Wigner-D coefficients. At inference time, these Wigner-D coefficients are mapped back to spatial rotation R.  \n\nAuthors show experiments on multiple datasets showing better accuracy compared to other work. Authors also show few-shot views training accuracy and ablation study for different design components of the network.\n\n\nReferences\n\n[1] Klee, David M., et al. \"Image to sphere: Learning equivariant features for efficient pose prediction.\" arXiv preprint arXiv:2302.13926 (2023).", "questions": "How would this method work for in-the-wild datasets?\n\nWould this method be able to handle occlusions?\n\nIf we have to use \"transformers\" instead of convolution, what would change?", "limitations": "Authors discuss limitations of their approach in the last section. Societal impact is discussed in appendix in detail.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, the whole paper is well organized with sufficient figures to aid in understanding. Most of the math needed is included in the paper or cited the right resources. Experiments show that method is outperforming other pose-estimation methods that works in spatial / frequency domain. Ablation studies confirm authors claim of different network components. \n\nAuthors also provide sufficient detail on the model architecture, spherical convolution design, and training hyper-paramters, suggesting reproducibility.\n\nAppendix has lots of nice extra experiments along with inference time.", "weaknesses": "Some pose visualization on objects would be nicer and should be compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer iMp8 for constructive comments and suggestions.\n\n**[W1. Comparison of pose visualization]**\n\nWe present a comparison of pose visualizations in Figure R2. This visualization method is the same to those used in Figures A2 and A3 in the appendix.\n\nWe compare to the I2S [34] baseline. The left side of Figure R2 shows results from ModelNet10-SO(3), while the right side presents on PASCAL3D+. The numbers next to \"Err\" above the input images represent the error in degrees between the model's predicted pose and the ground truth (GT) pose.\n\nThese results demonstrate that our model provides more accurate and precise pose estimations, even in cases where the I2S baseline fails. Additionally, on the PASCAL3D+ benchmark, which includes objects captured in real-world scenarios, our model consistently shows correct pose estimations, particularly in challenging situations where the I2S baseline struggles.\n\n**[Q1. Evaluation on in-the-wild datasets]**\n\nThe PASCAL3D+ dataset is an in-the-wild dataset captured in typical real-world environments, which we evaluated in Table 2 and Figure R2. As far as we know, PASCAL3D+ is the primary in-the-wild dataset used for 3D orientation estimation. We have not found any other in-the-wild datasets that are commonly used for this task.\n\n\n\n**[Q2. Handling occlusion]**\n\nYes, our method is capable of handling occlusions.\n\nWhile our SO(3) equivariant estimator can offer some robustness to occlusions due to its rotational invariance and localized feature extraction, handling occlusions effectively often requires additional techniques and strategies. Our network can extract local features in a manner that is robust to 3D rotations, allowing it to identify visible parts of objects even when other parts are occluded. Additionally, the network's ability to recognize objects regardless of their orientation helps in identifying partially visible objects.\n\nThe results in Table R1 for the SYMSOL II dataset, which includes self-occlusion scenarios in symmetric solids, demonstrate that \nour model jointly trained with  $\\mathcal{L}_{\\text{dist}}$ [48, 34] handle occlusions better than the I2S [34] baseline. Our model improves accuracy by directly predicting a single clear pose, unlike the baseline model.\n\nFigure R1 shows the SYMSOL II visualization results, specifically in examples 4, 5, and 6. When provided with occluded marks of symmetric objects, our model effectively uses these cues to estimate a single pose.\n\n**[Q3. Transformer instead of convolution]**\n\nWe choose a convolution-based design, because using transformers instead of convolutions results in a slight performance drop. \n\nTable R3 presents the results when transformers are used as the backbone network instead of convolutions. We trained the model using a Vision Transformer (ViT) pre-trained backbone pre-trained by the geometric task of cross-view masked image modeling [A].\n\nAlthough the ViT is heavier and requires longer training time (1.4x), its performance actually declines. This suggests that convolutional backbones may still be more effective for 3D orientation estimation tasks.\n\n[A] CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion (Weinzaepfel et al., NeurIPS 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aqpmbTmBM7", "reviewer": "Reviewer_HnJh", "summary": "The paper proposes a method for regressing the rotation of an object from an image, where several (~10s) of training views of the object with known rotations are available. In particular, the approach proposes a rotation-equivariant network that predicts continuous Wigner-D matrix coefficients in the frequency domain, which can be converted into a discretised heatmap in SO(3) to extract the predicted rotation(s). The contributions are (1) a method for predicting in the SO(3) frequency domain directly and (2) an approach for decoding this to a rotation at inference time. The approach is evaluated on two datasets with respect to rotation error and demonstrates good performance. Ablations and sensitivity analyses in the paper and appendix validate many of the design choices.", "questions": "1. Is 4.1-4 pre-existing work? If so, please make this clear in the paper.\n2. Recommendation to rework the list of contributions to focus on the new aspects of the proposed approach and their significance.\n3. Recommendation to validate the claim that the method does not lose precision due to discretisation.\n4. Recommendation to expand on the design decisions W2.1-4.\n5. Recommendation to test objects with complex symmetries via the SYMSOL dataset; and test the ability to predict multiple hypotheses as claimed.", "limitations": "Yes, the authors adequately address the limitations and broader impact in the appendix, with an appropriate level of self-reflection and consideration.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. Originality. The idea of directly predicting Wigner-D matrix coefficients is original and is a sound design decision. It avoids some of the discretisation limitations of prior work and is intuitive.\n\nS2. Quality. On the whole, the paper is well-presented, with useful figures and a clear structure. The writing is also quite decent, and the results are clearly displayed. In particular, the related work is well-written, clear, concise and sufficiently complete to position the work in its research context. In addition, the experiments validated the overall efficacy of the approach well, except for a couple of points below, especially the graphs of performance w.r.t. training data cardinality.\n\nS3. Clarity. The paper is clear overall and does a good job of directing the reader's attention appropriately. One caveat is that it is not self-contained (with respect to the main paper and also the main+appendix), with some components left out of the main treatment that reduced the overall clarity of the treatment. Nonetheless, the introduction motivated the problem and approach very well, and each section was itself well-motivated, making the paper easy to follow.\n\nS4. Significance. The paper's contributions of predicting the frequency-domain coefficients and decoding them to a heatmap are likely to be of positive but mild significance to the NeurIPS community. The task itself is of high significance, since rotation estimation from an image (for known objects) is a challenging task that is important for many applications, including factory robotics, augmented reality, automotive, etc.", "weaknesses": "W1. Contributions/claims.\nW1.1 The method in sections 4.1-4 is presented as part of the contribution, but appears to be the same as previous work, including [34]. If this is not the case, the design differences and the rationale should be better explained. If it is the case, this limits the contribution to sections 4.5-6, which is (a) directly predict and optimise the Wigner-D matrix coefficients, and (b) convert to a discretised heatmap and take the argmax. While (a) is likely key to the success of the method, it would appear to be a small variation on an existing approach; (b) on the other hand is very minor and indeed not claimed as a contribution. What this reviewer would like to see here is an expansion of these sections with some interrogation (and testing, in section 5) of the design choices. For example, is MSE a theoretically-grounded choice here, what are the weights w, why argmax instead of clustering for multi-hypothesis outputs?\n\nW1.2 Relatedly, the list of contributions has 3 items, but 2-3 do not seem to be contributions of this paper. That is, the contribution of leveraging SO(3)-equivariant representations to guarantee equivariance has clearly been done before (e.g., [27,34,40,33], among others); and achieving SOTA performance is not a contribution.\n\nW1.3 The claim that the proposed method is continuous and therefore better than the discretised methods (e.g., line 97) is dubious, when the predicted rotations are indeed discretised (section 4.6). The claim of loss of precision due to discretisation (for other methods) is also not validated in the experiment section; and one would assume that this method would also lose precision for the same reason (as Q decreases).\n\nW1.4 The claim that parametric models are insufficiently expressive for this task (L82) is not validated. It seems likely, but would depend on the model and would ideally be directly tested to back up the claim.\n\nW2. Design choices. (Partially overlaps with above)\nW2.1 Spherical mapper. While this is taken from existing work, it is not explained why it's a reasonable design decision to warp a 2D planar signal to the sphere, or given any motivation. It's clear that it's necessary for the method, to allow S2- and SO(3)-equivariant operations, but geometrically it seems quite dubious. Could the feature warping be made more geometrically meaningful by first running the image through DepthAnythingv2 and projecting out from a point on the crop's centroid ray?\n\nW2.2 The text mentions other approaches for non-linearities that avoid the extra (i)FFT steps (L233) but does not test or validate this design choice.\n\nW2.3 MSE (5). This does not seem to be a theoretically- or empirically-justified choice but is key to the contribution. Lines 254-258 loosely refer to this, but this section would seem to be a good opportunity to provide a justification for the design.\n\nW2.4 Discretised distribution on SO(3). The design choices around this part are not interrogated or justified - it reads as rushed or incomplete. For example, it states that the inference scheme \"models objects with ambiguous orientations or symmetries by employing multiple hypotheses\", but this is never done, as far as I can see. Instead, the argmax or gradient ascent is used, returning a single mode. Nowhere does the paper demonstrate how to use the approach to extract multiple hypotheses or test efficacy on extracting equivalent rotations for symmetric objects (see experiment section below). This is a missed opportunity and leads to the paper arguing against itself.\n\nW3. Clarity.\nW3.1 As alluded to earlier, the treatment is not self-contained. In particular, the Wigner-D matrix representation is central to the method and, for completeness and ease of understanding, should be included in the main paper. The reader would want to see the formulation that directly connects the task to the representation, instead of leaving that jump a bit vague (L121) and referring to the appendix. The function f in (4) and (12) are presumably not the same, but are not specifically defined for this task. Essentially, it reads as being incomplete, missing some of the material that connects the theory to the specific instantiation here.\nW3.2 L248. Please provide/derive the weights w for completeness.\n\nW4. Experiments.\nW4.1 The comparisons (especially in Table 1) would be more meaningfully grouped by the backbone extractor. For example, we should actually be comparing the similar method [34] with \"ours (ResNet-50)\", where a 0.6deg improvement is achieved. Similarly, it would be useful to compare the ResNet101 version on [34] in this table (like the version in Tab 2) since it is the closest existing method.\nW4.2 Missing dataset. SYMSOL, used in the closest related work [34,40], is not included in this paper, despite being a good opportunity to evaluate the performance with complex symmetries. Leaving it out gives the impression that the method cannot handle such cases, even though it seems that it ought to be able to.\nW4.3 Multi-hypotheses untested (despite L271). This claim ought to be experimentally tested, since it is such a relevant and interesting potential feature of the approach.\nW4.3 Ablations/sensitivity analyses. A sensitivity analysis on the influence of discretisation on precision (Q) is missing (despite claim in L96); the choice of MSE is not tested.\n\n\nW5. Minor.\nW5.1 The paper needs a proofread - there are many typos, syntactical and grammatical errors.\nW5.2 Confusing formatting error with footnote 2 appearing the page before it appears in the caption (L262.5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer HnJh for constructive comments and suggestions.\n\n**[W1.1, W1.2, Q1, Q2 clarifying the contributions]**  \n\nPlease see our general response above.\n\n**[W1.3, W4.4, Q3. Continuity of rotations, Sensitivity analysis of the SO(3) HEALPix discretization]**\n\nWe carefully claim that our learning method focuses on continuous rotations. We directly learn the Wigner-D coefficients, which are derived from 3D rotations (Euler angles), without any discretization during the training phase. The use of the SO(3) HEALPix grid during inference serves two purposes:\n\n1. To convert SO(3) rotations from the frequency domain to the spatial domain, and\n2. To address pose ambiguity by providing multiple solutions.\n\nAs a result, we obtain a distribution with very sharp modality. By taking the argmax of this distribution, we achieve sufficient precision in 3D orientation estimation, specifically around 1.5 degrees. \n\nTable R2 shows the effect of changing $Q$ on performance is. For the ModelNet10 benchmark, we achieve similar results in the common target metrics such as Acc@15 and Acc@30 even with a lower size of SO(3) discretization grid ($Q$=4.6K).\nWith our model's choice of $Q$=2.36M, we obtain high scores even on low-threshold evaluation metrics like Acc@3. Comparative experiments at lower thresholds are provided in appendix Table A1.\n\nTherefore, the continuity of rotations in our method ensures that we can predict an accurate, more precise pose. This is a significant advantage over other methods that suffer from precision loss due to discretization, as our approach maintains high accuracy across varying levels of discretization.\n\n\n**[W1.4 The claims of parametric model]**\n\nOur statement about the insufficient expressivity of parametric models (L82) refers to the potential lack of flexibility due to the dependency on predefined prior models, compared to non-parametric models. We acknowledge that this claim was not directly validated in our work. To provide a more accurate representation, we will revise our claim to reflect that.\n\n**[W2.1. Justification of the spherical mapper]**\n\nThe spherical mapper maintains the geometric structure of the image when projecting onto the S2 sphere, as detailed in [34]. This method involves lifting the 2D image onto the sphere and converting spherical points using spherical harmonics. Table R6 shows that the spherical mapper outperforms simple Fourier transforms on 2D feature maps.  \n\nUsing depth information from methods like DepthAnythingv2 for 3D lifting is a good idea and can enhance geometric accuracy. Additionally, centroid ray regression has been explored in research such as [70]. However, incorporating external depth modules increases computational costs and broadens our research scope, so we consider this for future work.\n\n\n**[W2.2 Non-linearites for equivariant layers]**\n\nThe FFT-based approximate non-linearity [19] and equivariant non-linearity for tensor field networks [51,65] mentioned in the text provide non-linearity based on Fourier kernels, avoiding the need for FFT and iFFT steps. However, these methods are not included in our quantitative evaluation as their code is not publicly available. Testing these approaches could potentially improve performance or efficiency when combined with our Fourier-based SO(3) equivariant network.\n\n**[W2.3, W.4.3, Q4 Design choice of MSE]** \n\nPlease see general response above.\n\n**[W.2.4, W4.2, W4.3, Q5. Evaluation on SYMSOL]**  \n\nPlease see general response above.\n\n**[W2.4 Discretised distribution on SO(3). (argmax vs. clustering)]**\n\nTable R5 shows the evaluation results using gradient ascent on the pose distribution.  While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.\n\n**[W3.1 Clarification of Wigner-D representation, ]**\n\nWe will clarify the Wigner-D matrix representation in the main paper. While detailed equations are provided in Sections A.2, A.2.2, and B.2 of the appendix, we will include a concise explanation in the main text.\n\nSpecifically, the Wigner-D representation is implemented in a flattened form at different frequency levels. For example, the matrix coefficients in a frequency level $l$ is represented as a flattened vector of size $(2l+1)*(2l+1)$. We use a maximum frequency level of 6, resulting in a vector of size 455 (i.e., $1∗1+3∗3+5∗5+...+13∗13=455$) for the Wigner-D coefficients. This will be clearly explained to directly connect the task to the representation in our final manuscript.\n\n**[W3.1 Notation of $f$ in eq (4) and (12)]**\n\nWe will clarify the notation to differentiate the two $f$ functions in Equations (4) and (12). These functions are indeed different.\nEquation (4) represents the general form of a group equivariant network.\nEquation (12) describes the rotation of spherical harmonics using the rotated coefficients of the Wigner-D matrix.\nWe will update the notation to clearly distinguish between these two functions and ensure the connection between the theory and its specific instantiation is clearly defined.\n\n**[W3-2. Derive the weights $w$]**  \n\nPlease see our response above.\n\n**[W4.1. Fair comparison of  the backbones in Table 1]**\n\nThank you for the suggestion. We include a comparison in the appendix (Table A1) that aligns the sizes of the backbone networks. Some networks, like Inception-v3, are not directly comparable, so we group representative methods for fair comparison based on similar backbones. These results show that our method outperforms existing methods, even on finer metrics like Acc@3 and Acc@5, demonstrating its effectiveness.\n\n\n**[W5. Typos and formatting errors]**\n\nWe will conduct a thorough proofreading to correct typos, syntactical, and grammatical errors. Additionally, we will adjust the positioning of footnote 2 to ensure it appears at the correct location where it is cited in the text. Thank you for your helpful suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGHl26fe7i", "reviewer": "Reviewer_HCET", "summary": "The paper tackles 3D pose estimation from a single image. It builds on Image2Sphere which maps CNN features to the sphere which are then processed by spherical CNNs to produce a distribution over SO(3). The submission proposes using an MSE loss function in the spectral domain instead of cross-entropy in the spatial domain. It outperforms the baselines on pose estimation on ModelNet10-SO(3) and Pascal3D+.", "questions": "In L212, is looks like S is a function on the sphere defined in the spatial domain, but it is said to be C'xN where N is the number of spherical harmonics. Should N be the number of points on the sphere instead?\n\nTypos:\n\nL44: leverages -> leverage\n\nL70: faciltate -> facilitate\n\nL87: Foruier -> Fourier", "limitations": "Possible limitation on symmetric objects should be discussed (see W3).", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "S1) There could be value in studying losses and other operations in the spectral domain instead of spatial.\n\nS2) The method seems to outperform the baselines in the attempted benchmarks.", "weaknesses": "W1) The paper seems to be a lightly modified version of Image2Sphere [1], but this is not properly acknowledged in the writing. Following Figure 2 and Section 4, the feature extraction, spherical mapper, Fourier transformers and pose predictor seems to be the same, the only difference being that in the submission, the last IFT is skipped and the loss is computed in the spectral domain during training (at test time it still needs the IFT so it is the same as Image2Sphere as far as I understand). Although Image2Sphere [1] is cited, the method section mostly describes the same things which implies they are new contributions. Please rewrite it focusing on the new contributions and being explicitly about what comes from prior work. \n\nW2) The main difference with respect to Image2Sphere is the loss being computed between Wigner-D coefficients instead of spatially. However how this is actually done is poorly explained. \n\na) How does one L245 \"convert the GT 3D rotations from Euler angles (...) to the Wigner-D matrices\"? The text seems to imply that the Wigner-Ds are evaluated at the ground truth rotation, but then the loss in Eq (5) does not seem to make sense since the output of the model are the coefficients associated to each D matrix. I think what should be done is inverting a function on SO(3) that is an impulse on the ground truth pose but there is no mention of it in the paper.  \n\nb) What is the \"similarity between Wigner-D coefficients and an SO(3) grid\" is L263? It seems that this somehow outputs a distribution on SO(3) that is queried spatially, so it seems like it should be an inverse SO(3) Fourier transform?\n\nW3) Experiments on SYMSOL are missing. The dataset is designed to assess performance on symmetric objects, so methods that predict distributions over SO(3) are necessary; the baseline Image2Sphere handles it well. I believe the proposed method might perform poorly on it because of the way the ground truth is computed or the MSE loss, which are the differences with respect to Image2Sphere. I believe a comparison and discussion of this possible limitation is warranted. \n\nW4) The paragraph L113 \"Rotation representation in frequency domain.\" is confusing. It talks generally of 3D rotations but actually describes the specific case of rotations of spherical functions. I think the U in eq (1) should be the same as the D in eq (2) but they are described as different things.\n\n## References \n[1] Klee et al, \"Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction\", ICLR'23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer HCET for constructive comments and suggestions.\n\n**[W1. Clarifying our contributions compared to I2S [34]]** \n\nPlease see our general response above.\n\n**[W2-a. Conversion from Euler angles to Wigner-D during training (L245)]**\n\nThe conversion from Euler angles to Wigner-D matrices involves a mathematical transformation using the $ZYZ$ sequence of rotations and the corresponding Wigner-D matrix elements. The model outputs are indeed the coefficients associated with the Wigner-D matrices. The loss function in Eq. (5) compares these predicted coefficients directly with the ground-truth coefficients, facilitating the direct regression of 3D rotations in the frequency domain.\n\nAdditionally, unlike Image2Sphere [34], we do not perform the inverse Fourier transformation (iFT) of the output of the networks during training. Instead, we train on the output Wigner-D coefficients in the frequency domain, and we do not apply the iFT even at test time.\n\nWe hope this explanation clarifies the process and resolves any confusion. Thank you again for your valuable feedback.\n\n\n**[W2-b. Computing the similarity with the SO(3) grid during inference (L263)]**\n\nThe similarity calculation in L263 is different from an inverse SO(3) Fourier transform. It is a process to find candidate Euler angles from the predicted Wigner-D coefficients. The predefined SO(3) HEALPix grid includes a set of predefined Wigner-D coefficients corresponding 1:1 to Euler angles. We calculate the similarity between this grid and the predicted Wigner-D coefficients to obtain a discretized distribution. This probability distribution provides multiple hypotheses for the pose (i.e., Euler angles) predictions of the input image. We then use argmax or gradient ascent to determine the final predicted Euler angles at inference, as shown in the results of Table R5. This method is similar to the approaches used in [48, 34].\n\nWhile an inverse SO(3) Fourier transform could convert the predicted Wigner-D coefficients to Euler angles, it has high time complexity, as it requires separate calculations for each frequency level. The SO(3) iFT method produces a single value, making distribution modeling difficult. \n\nIn contrast, our design choice of querying the predefined HEALPix SO(3) grid enables distribution modeling by calculating simple vector-matrix multiplication. This allows us effective symmetric object modeling by combining the existing cross-entropy loss [48, 34], as demonstrated in Table A1 and Figure A1.\n\n**[W3, L1. Evaluation on SYMSOL]** \n\nPlease see our general response above.\n\n**[W4. Clarification on paragraph L113 \"Rotation representation in frequency domain\"]**\n\nThe paragraph \"Rotation representation in frequency domain\" describes how 3D rotations are represented in the frequency domain using spherical harmonics and the Wigner-D matrix.\n\nThe Wigner-D rotation representation in L113-125 is not limited to a specific case of 3D rotations but can be converted from any 3D rotation representation, such as Euler angles, quaternions, and 3D rotation matrices. Our SO(3) equivariant network predicts the Wigner-D representation in the frequency domain instead of predicting rotations in the spatial domain (Euler angles, quaternions, etc.).\n\nTo address the confusion, we clarify that the $U$ in Equation (1) and the $D$ in Equation (2) are indeed the same. We will update the manuscript to use consistent notation in Equations (1) and (2). We apologize for any confusion caused by this inconsistency.\n\n\n\n**[Q1. Clarification of the spherical notation in L212]**\n\nIn Section 4.3, $\\mathcal{S}$ is a spherical harmonics function defined in the frequency domain.\n$N$ denotes the total number of spherical harmonics in the frequency domain.\n$p$ denotes the number of points on the sphere in the spatial domain.\nWe will update the manuscript to clearly indicate that  $\\mathcal{S}$ is defined in the frequency domain and ensure the distinction between $N$ (number of spherical harmonics) and $p$ (number of points on the sphere) is clear.\n\n\n**[Q2. Typos]**\n\nWe will revise our manuscript according to your findings of typos. Thank you for your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xYj5AtgIog", "reviewer": "Reviewer_f5v8", "summary": "This paper proposes a method for object orientation estimation from images.  The method predicts object orientation in the frequency domain of SO(3), by using SO(3) equivariant layers that operate on coefficients of Wigner-D matrices.  The network is trained with a MSE loss between the predicted Wigner-D coefficients and the Wigner-D coefficients associated with a ground truth rotation label.  The network is evaluated on two datasets, PASCAL3D+ and ModelNet10-SO(3), and outperforms all baselines in terms of rotation precision and accuracy.  Additionally, an ablation study is performed which demonstrates that the proposed output formulation is better than traditional rotation formulations, like quaternions, euler angles and rotation matrices, and that the method is robust to the sampling strategy used at inference time.", "questions": "- Did you explore evaluating the model with a higher resolution grid or with gradient ascent?  Given that the regression loss can be precise, the model may be able to achieve even lower rotation error with more grid points.\n- The regression loss is very effective at achieving high precision, but seems to lose the ability to model object symmetries or uncertainty (according to qualitative results in Figure A2).  Is there a way to combine the benefits of high precision and distribution modeling?  Other pose works have used a classification-regression framework [1], and it would be interesting to see if this idea could be applied here.  For instance, apply the classification loss over a HEALPix grid first, then apply the proposed regression loss if the prediction is close.\n- In Appendix C.6, it seems like the frequency level is set for both the S2 and SO(3) convolutions in the network.  Did you try only modifying the frequency for the output of the network?  The issue with higher frequencies reducing performance may be due to the non-linearity (ReLU in spatial domain), which could be a bit unstable with high frequency signals.  Some other works have used gating functions for non-linearities on Wigner D coefficients [2].\n- From the introduction, \"in the context of spherical CNNs, ..., the 3D rotation parametrization in the spatial domain is inadequate because these SO(3) equivariant networks operate in the frequency domain.\"  What do the authors mean by inadequate?  Clearly, conversion from frequency to spatial domain is possible (the proposed method does so in its non-linearity layers).  \n\n[1] Liao, Shuai, Efstratios Gavves, and Cees GM Snoek. \"Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[2] Liao, Yi-Lun, and Tess Smidt. \"Equiformer: Equivariant graph attention transformer for 3d atomistic graphs.\" arXiv preprint arXiv:2206.11990 (2022).", "limitations": "The limitations are discussed in Appendix F.   \n\nWhat is meant by, \"our method can suffer from significant errors due to the loss of spatial information when projecting 3D data onto spherical harmonics\"?  What 3D data?  Where are these significant errors described in the experimental results or discussion?\n\nThe authors mention that, \"computational cost associated with Wigner-D coefficients and SO(3) equivariant networks should be improved...\";\nthis is a bit vague and contradicts Table A6, which shows the proposed method has the fastest frame rate during evaluation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces a new way to train Wigner-D coefficients for accurate object pose prediction.  They apply a weighted MSE loss between the predicted coefficients and  ground truth coefficients.\n- The proposed approach achieves SOTA performance on the PASCAL3D+ and ModelNet10-SO(3) datasets. \n- The experimental section is thorough.  Competitive baselines are included and statistical data is reported for the proposed method.  The authors include a new variation on the ModelNet10-SO(3) dataset, which evaluates the sample efficiency of the method and baselines.", "weaknesses": "- The contribution is interesting, but minimal compared to existing work.  The neural network model and evaluation scheme used in this paper was introduced in [1].  However, this is not entirely clear the way the paper is written.  For instance, Sections 4.2, 4.3, 4.4, and 4.6 are all describing specific techniques from [1].  It would be easier to understand the contributions specific to this paper's network if these sections were moved to Background.  \n- The second contribution of the paper, as stated, is confusing.  What is meant by the \"intrinsic properties of SO(3) equivariant representations\"?  Also, the claim \"structurally guarantee 3D rotational equivariance\" is not correct.  It is not possible to guarantee 3D rotational equivariance of the network when the input is an image.\n- The core contribution of the paper is the use of a regression loss applied to learned Wigner-D coefficients.  However, this aspect of the method is not examined closely.  There is no information of how the values of $w_l$ are defined in Eqn. 5, nor any experiment justifying why MSE is preferred.  The contribution would be strengthened by a comparison of other sensible loss functions, an experiment that showed the sensitivity of the method to the values of $w_l$.  Additionally, it is not clear how the proposed loss is affected by object symmetries.  Could the authors explain why regression on Wigner-D coefficients does not suffer from inaccurate predictions when the orientation is ambiguous (e.g. tables have 180 degree symmetry in ModelNet10-SO(3))?\n- The ablation experiment in Section 5.5 needs to be described in more detail.  How exactly was the method modified to produce euler, quaternion, etc. outputs?  Were SO(3) equivariant layers still used?\n-  Some of the writing could be improved for clarity.  In Section 1, \"Accurate determination of 3D orientation presents a more complex problem than translation due to the intricacies of rotation symmetries and the high dimensionality of pose space\" is confusing: how is 3D orientation higher dimensional than 3D translation?  Also, \"SO(3)-equivariance is crucial for accurate pose estimation ...\" is not entirely true, as evidenced by the competitiveness of non-equivariant baselines in Tables 1&2.\n- Figure 1, some of the images are taken from other sources without attribution (e.g. the healpix grid is from https://healpix.sourceforge.io\n\n\n\n[1] Klee, David M., et al. \"Image to sphere: Learning equivariant features for efficient pose prediction.\" arXiv preprint arXiv:2302.13926 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer f5v8 for constructive comments and suggestions.\n\n**[W1. Clarifying contributions]** \n\nPlease see our general response above\n\n**[W2. Claims of structural guarantee of 3D rotational equivariance]**\n\nThe statement \"structurally guarantee 3D rotational equivariance\" is indeed misleading. While the network can use components like spherical convolutions and Wigner-D matrix predictions to maintain equivariance, perfect 3D rotational equivariance cannot be guaranteed with 2D image inputs, as they inherently lack full 3D structural information. The network approximates 3D equivariance using the spherical mapper to handle 3D rotations effectively, but absolute equivariance is not achievable with 2D data. We will clarify that the network aims to approximate 3D rotational equivariance as closely as possible given these constraints.\n\n**[W3. symmetry, other losses, $w_l$]** \n\nPlease see our response above\n\n**[W4. Detailed configurations of Table 3 ablation studies]**\n\nTable 3 in Section 5.5 demonstrates the effectiveness of our proposed Wigner-D matrix compared to other rotation representations in the spatial domain (Euler angles, Quaternion, axis-angle, rotation matrix). For all cases, we retained the backbone networks and SO(3) equivariant layers. The only modifications were the output prediction dimension size and the ground-truth rotation representation.\n\n**[W5-1. Clarification on the complexity of 3D orientation and translation]**\n\nWe acknowledge that the original statement might be confusing.  To clarify, we can represent the 3D translation and orientation:\n\n* 3D translation: movement along the x, y, and z axes. (3 dims)\n* 3D orientation: involves rotations around these axes, represented by Euler angles (3 dims), quaternions (4 dims), or rotation matrices (9 dims).\n\nWe will update our manuscript to improve clarity: 3D orientation is more complex due to rotational symmetries and the non-linear nature of rotations. Unlike translation, rotations present challenges such as gimbal lock and the need for continuous representation without singularities.\n \n\n**[W5-2. Overstatement of SO(3) equivariance]**\n\nThe claim \"SO(3)-equivariance is crucial for accurate pose estimation\" might be overstated. While it can enhance performance and robustness, it is not the only way to achieve competitive results. As shown by the non-equivariant baselines in Tables 1 and 2, alternative approaches also deliver strong performance. We will revise the manuscript to reflect this nuanced perspective.\n\n\n**[W6. Attribution of images in figure 1]**\n\nWe apologize for the oversight in attributing images. We will revise the figure to clearly credit all original sources.\n\n**[Q1. Evaluation of model with higher resolution grid and gradient ascent]**\n\nTable R2 shows the impact of changing the grid resolution $Q$ on performance. The higher resolution (18.87M) of the SO(3) grid improves the performance at a lower threshold at inference time.  \n\nTable R5 shows the evaluation results using gradient ascent on the pose distribution.  While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.\n\n**[Q2. Combining high precision and distribution modeling]**\n\nOur method currently struggles to model object symmetries and uncertainty. However, Table R1 demonstrates that combining our approach with a distribution loss based on cross-entropy [48, 34] is effective for symmetric object modeling, and a classification-regression framework [38] can be an alternative solution for distribution modeling.\n\n\n**[Q3. Adjusting frequency levels and mitigating high-frequency instability]**\n\nIncreasing the frequency level at the network's output is challenging due to the need to define the maximum frequency level of spherical harmonics at the model initialization.  Adjusting the final output frequency can help mitigate performance drops at high frequencies caused by non-linearity. Using gating functions on Wigner-D coefficients in Equiformer (Liao and Smidt, ICLR 2023) could address instability issues. \n\n\n**[Q4. Clarification on spatial domain parametrization in spherical CNNs]**\n\n\nThe term \"inadequate\" refers to the practical challenges and inefficiencies of using spatial domain parametrizations directly within spherical CNNs. These challenges arise because spherical CNNs naturally operate in the frequency domain, where rotations are represented and manipulated more efficiently using spherical harmonics and Wigner-D matrices. \n\nAlthough conversion between the frequency and spatial domains is possible (by inverse Fourier transform), maintaining consistent parametrization in the frequency domain is more straightforward for the SO(3)-equivariance with spherical CNNs and proves to be more effective, as demonstrated in Table 3 of the main paper.\n\n\n\n\n**[L1. Clarifying the loss of spatial Information in spherical harmonics projection]**\n\nIn Section F, the statement about significant errors refers to the loss of spatial information due to truncating the frequency level of spherical harmonics for efficiency. \"3D data\" refers to data points on the 2-sphere. These errors, not explicitly detailed in the experimental results, arise from truncating higher frequency components, leading to a loss of spatial detail. We will update the manuscript to clarify this point and provide a more precise explanation.\n\n**[L2. Clarifying computational costs in the limitation section]**\n\nIn Section F, we note that while our method achieves the best inference time (Table A6), there is room for improvement in space complexity. The computational cost, due to the complexity of Wigner-D coefficients and convolutions in SO(3)-equivariant networks, remains high. These operations require handling high-dimensional representations and specialized mathematical processes. Therefore, reducing memory usage and computational load is a target for future enhancement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their constructive comments and recognition of the strengths of our paper:
* Reviewer iMp8: We appreciate your positive feedback on the paper's structure, figures, equations, detailed method section, and ablation studies.
* Reviewer HnJh: Thank you for highlighting the strengths of our Wigner-D prediction in overcoming discretization method limitations. We value your compliments on the clarity and structure of our manuscript.
* Reviewer HCET: We appreciate your recognition of our use of a loss function in the spectral domain and the superior performance on benchmarks.
* Reviewer f5v8: Thank you for acknowledging our novel approach to object pose estimation and the thoroughness of our experimental section.

We address common questions in below:

##  1. Contributions

**R: HnJh [W1.1, W1.2, Q1, Q2. Clarifying the contributions, List of contributions]**

**R: HCET [W1. Clarifying our contributions compared to I2S [34]]**

**R: f5v8 [W1. Clarifying contributions]**


We would like to clarify that while our method shares the same backbone structure as [34], the key innovation lies in how we optimize using the Wigner-D matrix directly. This is a significant departure from [34], which does not utilize the Wigner-D matrix in this manner. Specifically, we propose a continuous rotation learning method through direct Wigner-D prediction.

We clarify that Sections 4.1-4 describe foundational concepts and methodologies based on pre-existing work, which may give the impression that these are new contributions. To address this, we will revise the paper to make it clear that these sections are based on prior work and move them to the background section. This will better highlight our novel contributions in Sections 4.5-6.

To address the reviewer's concerns, we clarify and emphasize the novel aspects of our contributions:
1. *Frequency-Domain Prediction*: Our approach uniquely predicts Wigner-D coefficients directly in the frequency domain, avoiding issues like discontinuities and singularities in traditional spatial domain methods, ensuring precise and continuous pose estimation.
2. *Tailored MSE Loss*: We introduce a frequency-domain specific Mean Squared Error (MSE) loss. This tailored loss function supports continuous training for SO(3) pose estimation and has the potential to integrate cross-entropy loss for distribution modeling, effectively addressing object symmetry challenges.
3. *Superior Performance and Data Efficiency*: Our SO(3)-equivariant network consistently outperforms existing methods on standard pose estimation benchmarks, demonstrating data sampling efficiency in data-limited scenarios.

These contributions provide substantial improvements in accuracy and robustness over the baseline methods, as demonstrated in our experimental results. We believe that our approach represents a meaningful advancement in the field of 3D orientation estimation.


## 2. Results on SYMSOL


**R: HnJh [W.2.4, W4.2, W4.3, Q5. Evaluation on SYMSOL, Test of the distribution on SO(3).]**

**R: HCET [W3, L1. Evaluation on SYMSOL]**

**R: f5v8 [W3-3. Effects on Wigner-D regression loss in symmetric objects]**

Table R1 shows symmetric object modeling on the SYMSOL datasets. Compared to the first row and second row [34], our model with only the Wigner-D regression loss derives on sharp modalities, which can be less effective than [34] for symmetric objects in SYMSOL I.

For clearly defined pose cases (e.g., SphereX in SYMSOL II), our Wigner-D loss alone performs well. However, in other SYMSOL II scenarios, the sharp distributions produced by our model can lead to low average log likelihood scores. This metric is particularly harsh on models with sharp peaks, making them vulnerable to very low scores in some failure cases.

In the third row, joint training of our method with the distribution loss [48, 34] achieves better performance than the baseline [34], demonstrating its ability to model symmetric objects. These results highlight our method's potential in handling complex symmetries and predicting multiple hypotheses. Figure R1 shows the visualization of pose distribution.

Most real-world objects have unique, unambiguous poses, validating our single pose regression method (e.g., ModelNet10-SO3, PASCAL3D+). If the task needs to cover symmetric cases, our model can be modeled with distribution loss [48, 34].


## 3. Design choice of MSE


**R: HnJh [W1.1, W2.3, W4.3, Q4. Design choice of MSE]**

**R: f5v8 [W3-2. Comparison with other loss functions]**

The choice of MSE for regression loss is for its simplicity and effectiveness. Cosine and angular distances measure only angles between vectors, ignoring magnitude, which is crucial in frequency domain applications. Chordal and geodesic distances often require normalization to the unit sphere, making them less intuitive and more computationally intensive, which can be a drawback in practical terms.

Table R4 compares various loss functions. While Huber and L1 losses are alternatives, they do not perform as well as MSE in our context. Geodesic loss in the frequency domain is ineffective because it requires separate calculations for each frequency level of the Wigner-D matrix, potentially losing the precision of the original 3D rotation, as we truncate the Fourier basis at a frequency level of 6.

**R: HnJh [W3-2. Derive the weights $w$ (L248)]**

**R: f5v8 [W3-1. The value of $w_l$]**

Figure R3 shows visualization of the weights $w$. The figure shows a consistent learning pattern, even as the # of training views changes. Notably, the weights increase with higher frequency levels, indicating that more complex rotations are effectively modeled with greater emphasis on higher frequencies. This demonstrates the effectiveness of weighting more complex rotations more heavily during training.

We have expanded these sections and included the relevant experiments in Section 5 to provide a comprehensive understanding of our design choices.

Meta-review: Initially there was some confusion about the similarity to [34] and the contribution. The authors promised to update the paper according to the rebuttal. As reviewer HnJh states "While the contribution is relatively small in my view, it is sufficient and the material in the rebuttal has strengthened it". Three reviewers (HnJh, f5v8, iMp8) agree on this statement. 

The fourth reviewer (HCET) has reservations because many things were not clear initially and need to be udpated in the manuscript.

Overall, the AC recommends trusting the authors that they will update the manuscript according to the comprehensive discussion in the rebuttal and recommends to accept the paper.
Decision: Accept (poster)