3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction

Authors: jongmin  lee
minsu  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:29:13

Abstract: Determining the 3D orientations of an object in an image, known as single-image pose estimation, is a crucial task in 3D vision applications. Existing methods typically learn 3D rotations parametrized in the spatial domain using Euler angles or quaternions, but these representations often introduce discontinuities and singularities. SO(3)-equivariant networks enable the structured capture of pose patterns with data-efficient learning, but the  parametrizations in spatial domain are incompatible with their architecture, particularly spherical CNNs, which operate in the frequency domain to enhance computational efficiency. To overcome these issues, we propose a frequency-domain approach that directly predicts Wigner-D coefficients for 3D rotation regression, aligning with the operations of spherical CNNs. Our SO(3)-equivariant pose harmonics predictor overcomes the limitations of spatial parameterizations, ensuring consistent pose estimation under arbitrary rotations. Trained with a frequency-domain regression loss, our method achieves state-of-the-art results on benchmarks such as ModelNet10-SO(3) and PASCAL3D+, with significant improvements in accuracy, robustness, and data efficiency.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We give an overview and scope of the SO(3) equivariant pose harmonics predictor in section 1. At the introduction section, we summarize our contribution, the prediction of wigner-D coefficients via 3D rotation reparametrization to the frequency domain, is a first attempt.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Yes we discuss our limitation in section F. We also discuss our experiments on training sensitivity (section B.10) and limitations in the frequency domain (section 6).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
representations of rotations: 
 Rotation representation in spatial domain. In 3D rotation, Euler angles are a common SO(3) representation but suffer from non-uniqueness and gimbal lock, making them less suitable for neural network predictions. Quaternions offer a solution by preventing gimbal lock, but their nonunique representation (q and -q) can complicate certain optimization processes. The axis-angle representation is intuitive but can encounter singularities. The 6D and 9D representations provide newer approaches that simplify optimization in deep networks by avoiding non-linear constraints and ensuring orthogonality. However, they also introduce complexities in maintaining constraints during the learning process. Thus, choosing an appropriate rotation representation is crucial for accurate pose estimation in various computational applications. For a detailed explanation, please refer to Sec. A.1 and an overview of learning 3D rotations in [51, 55] .Rotation representation in frequency domain. In the frequency domain, 3D rotation is managed by manipulating spherical harmonics coefficients. Spherical harmonics, denoted as Y l m (θ, ϕ), are functions defined on the surface of a sphere using polar (θ) and azimuthal (ϕ) angles. These harmonics are characterized by their degree l and order m, truncated to a maximum degree L for computational feasibility. The rotation of spherical harmonics is represented by the shift theorem [43] , where a rotation operator Λ g acts on spherical harmonics, transforming them via a matrix D l mn (g):EQUATIONThis matrix, part of the irreducible unitary representation of SO(3), expresses how each harmonic changes under rotation, summing over all orders n from -l to l, called Wigner-D matrix. The Wigner-D rotation representation is not limited to a specific case of 3D rotations but can be converted from any 3D rotation representation, such as Euler angles, quaternions, and 3D rotation matrices.Our SO(3) equivariant network predicts the Wigner-D representation in the frequency domain instead of predicting rotations in the spatial domain. For a detailed explanation, please refer to Sec. A.2.Equivariance. Equivariance is a useful property to have because transformations T applied to the input produce predictable and consistent output of the features through transformations ϕ ∈ Φ, enhancing both interpretability and data efficiency. For example, a feature extractor Φ is equivariant to a transformation if applying the transformation to the input and then applying the extractor produces the same output as applying the extractor first and then the transformation:EQUATIONwhere T g and T ′ g represent transformations acting on a group g ∈ G of the input and output spaces, respectively. This ensures that the network's output remains consistent with transformations applied to the input. For translation groups, convolution inherently maintains this property. For rotations, additional rotation-equivariant layers are integrated into the network design.Group-equivariant convolutional networks [10] extend this concept to complex groups like rotations or other symmetries. By designing convolutions that are equivariant to these group actions, these networks can handle a broader range of transformations. This can be mathematically described as:EQUATION)where h is the input function over space X, ϕ is the filter or kernel, and g ∈ G is an element of the group. The term g -1 y represents the transformation of y by the inverse of g. This operation ensures that the network remains equivariant to the actions of the group G, allowing it to handle inputs transformed by any element of this symmetry group.On the sphere in 3D, however, there is no straightforward way to implement a convolution in the spatial domain due to non-uniform samplings [12] . This challenge arises because traditional convolution operations rely on uniform grid structures, which are not applicable to spherical data.To address this, specialized methods such as spherical convolutions or graph-based approaches are employed to handle the unique structure and sampling patterns of spherical data, thereby ensuring effective feature extraction and equivariance on spherical surfaces.Spherical convolutions for SO(3)-equivariance. To effectively analyze complex spatial data, such as for volumetric rendering and 3D pose estimation, it is necessary to develop functions with equivariance to the SO(3) group. Early methods for spherical convolution were defined by computing Fourier transforms and convolution on the 2-sphere [15] . However, the output of these spherical convolutions is a function on the sphere, not on SO(3). Spherical CNNs [11] extended this approach to effectively convolve on the SO(3) group. Using the truncated Fourier transform, signals on S 2 are modeled with spherical harmonics Y l n , and on SO(3) with Wigner-D matrix coefficients D l mn . To efficiently compute the S 2 and SO(3) convolution, generalized fast Fourier transforms (GFFTs) demonstrate optimized computation [11] . The GFFTs show robustness and efficiency in spherical signal processing, where the spectral group convolutions become simpler element-wise multiplications in the Fourier domain. Specifically, for S 2 , the process uses vectors of spherical harmonic coefficients, forming a block diagonal matrix analogous to SO(3) convolution. Both convolutions on S 2 and SO(3) generate output signals that reside on SO(3).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: Yes, we discuss the necessary prior knowledge in the preliminary part of the text (section 3) and develop it through formulas in the method part (section 4). More detailed assumptions and proofs are discussed in sections ??, A of the appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
so(3)-equivariant pose harmonics predictor: 
 The goal of our network is to accurately predict the SO(3) pose of an object in an image. To achieve, we employ spherical CNNs [11] to obtain SO(3)-equivariant representation, and our model is trained with frequency-domain supervision using Wigner-D coefficients. This approach enhances data efficiency by capturing patterns with fewer training samples and ensures precise SO(3) pose estimation by aligning the parametrization of 3D rotations with the Wigner-D matrices in the frequency domain.Figure 2 provides an overview of our SO(3)-equivariant pose estimation network. In Sec. 4.1, we explain the steps for obtaining the Wigner-D representation, following the method described in [35] . In Sec. 4.2, we introduce a frequency-domain regression loss, where we train the network using MSE loss between the predicted representation and the ground truth (GT) Wigner-D coefficients. Finally, in Sec. 4.3, we describe the inference process by constructing an SO(3) grid for evaluation.In this subsection, we explain our SO(3)-equivariant pose estimation network, highlighting that its key components are shared with the architecture of [35] .Image feature extraction. We first apply a feature extractor to obtain an image feature map that encodes semantic and geometric information: F = ρ(I), where F ∈ R C×H ′ ×W ′ and ρ denotes ResNet. We utilize a ResNet feature extractor that is pre-trained on ImageNet same to [28, 35, 41, 48, 49, 71] . We then perform dimensionality reduction on the image feature F to match the input dimension of the subsequent spherical feature using a 1x1 convolution: Spherical mapper. To begin, we lift the image features to the 2-sphere using orthographic projection [35] . This involves mapping the 2D feature F ′ to a spherical feature ψ ∈ R C ′ ×p , where p denotes the number of points on the sphere. The orthographic projection links pixels in the image space to points on the sphere by orthogonally mapping S 2 coordinates to the image plane, thereby preserving the spatial information of the dense 2D feature map.EQUATIONInitially, we model spherical coordinates using an S 2 HEALPix [24] grid over a hemisphere. Within this hemisphere, the set {x i } ⊂ S 2 represents the vertices of the grid. Each vertex x i is mapped to a position P (x i ) on the image plane. Formally, the orthographic projection P maps 3D coordinates on the hemisphere to 2D coordinates on the image plane as P (x, y, z) = (x, y).Due to the fixed perspective, only one hemisphere of the sphere is visible, resulting in a localized signal ψ(x) = F ′ (P (x)) supported over this hemisphere. The value of ψ(x i ) is obtained by interpolating F ′ at the pixels near P (x i ) in the image space using an interpolation function η, so ψ(x i ) = η(F ′ , P (x i )). Figure 3 illustrates the processes of the spherical mapper, and the following frequency domain conversion and spherical convolution for SO(3)-equivariance.Convert to the frequency domain. The transition of the spherical feature ψ into the frequency domain is achieved using the fast Fourier transform (FFT) adapted for spherical topology. By employing the FFT, we efficiently convert ψ to spherical signals S, represented as a sum of spherical harmonics. This transformation allows us to capture and manipulate the spatial frequencies inherent to the spherical surface. Specifically, the transition to the frequency domain enables the derivation of Wigner-D coefficients, which effectively model the SO(3). The Fourier series of S is truncated at frequency L [11] , expressed as:S(x) ≈ L l=0 l m=-l c l m Y l m (x), where S ∈ R C ′ ×N , N is the total number of spherical harmonics determined by the maximum frequency L, and Y l m (x) are the spherical harmonics. Operating in the frequency domain facilitates the effective convolution of signals on the sphere (S 2 ) and within the 3D rotation group (SO(3)), preserving the geometric properties of input features through spherical equivariance.To address sampling errors from approximating the Fourier series via truncation, we apply two techniques proposed in [35] . First, to prevent discontinuities on the 2-sphere, we gradually decrease the magnitude of projected features near the image edge: ψ ′ (x i ) = w(x i ) • ψ(x i ). Second, for each projection, we randomly select a subset of grid points on the S 2 HEALPix grid as a dropout.Spherical convolution for SO(3)-equivariance. We aim to predict 3D rotations while preserving SO(3)-equivariance using the projected features on sphere. First, the spherical signal S is processed with an S 2 -equivariant convolutional layer [11, 35] . Unlike conventional convolutions with local filters, S 2 convolution uses globally supported filters, offering a global receptive field. This allows for a shallower network, which is important due to the high computational and memory demands of spherical convolutions at a high bandlimit L.In this stage, we obtain SO(3) representations inherent to spherical CNNs [11] . The output of S 2 convolutions lies in the SO(3) domain because S 2 convolutions replace translations with rotations, and the space of 3D rotations forms the SO(3) group. Consequently, we obtain feature results sized in R C ′′ ×M , where C ′′ is the hidden dimension of the SO(3) features, and M is the total number of Wigner-D matrix coefficients, given by M = L l=0 (2l + 1) × (2l + 1), created by SO(3) irreps. We apply non-linearities between convolutional layers by transforming the signal to the spatial domain, applying a ReLU, and then transforming back to the frequency domain, following the approach of spherical CNNs [11] . This method can be extended to FFT-based approximate non-linearity [19] and equivariant non-linearity for tensor field networks [52, 67] .Subsequent to the S 2 -equivariant convolutional layer, we perform an SO(3)-equivariant group convolution [11, 35] using a locally supported filter to refine the SO(3) pose space. Unlike typical spherical CNNs, we bypass the inverse fast Fourier transform (iFFT) and instead use the output harmonics of Wigner-D prediction. This approach, unlike that of [35] , improves the efficiency of our method. The final output of the equivariant network is the Wigner-D matrix coefficients Ψ ∈ R M .The output of the SO(3)-equivariant convolutional layers is a linear combination of Wigner-D matrices, represented as a flattened vector of the Wigner-D coefficients. The output Ψ indicates specific object orientations in an image. To generate the ground-truth (GT) Wigner-D coefficients, we convert the GT 3D rotations from Euler angles using the ZY Z sequence of rotation R, expressed as R = R z (γ)R y (β)R z (α) to the Wigner-D matrices D l mn (α, β, γ), where D represents an action of the rotation group SO(3). We calculate the Mean Squared Error (MSE) loss as follows:EQUATIONwhere w l are weights assigned to each harmonic frequency level l, normalizing the output Wigner-D matrices for a frequency-domain specific MSE loss. This loss function enables continuous prediction of SO(3) poses using SO(3)-equivariant networks, whereas the previous methods [28, 35, 49] predicted outputs in a discretized distribution, leading to degradation in prediction precision. With this re-parametrization in the frequency domain, we use Euclidean distance because it is simple yet effective for pose prediction. It allows straightforward calculation while considering both the direction and magnitude of the vectors. Many distance metrics defined in the spatial domain [2, 25, 30, 55] may not be directly appropriate for the frequency domain without adaptation. For example, cosine and angular distances ignore magnitude, where the amplitude of frequency components carries significant information. Chordal and geodesic distances require normalization, can be less intuitive, and often involve more complex computations.For evaluation, the output Wigner-D representation Ψ is converted to an SO(3) pose in the spatial domain. Figure 4 illustrates the inference process inspired by [35, 49] . Specifically, we map the predicted Wigner-D coefficients Ψ from the frequency domain to a 3x3 rotation matrix R by querying Ψ on a predefined SO(3) grid. To achieve this mapping, we calculate the similarities between the output vector Ψ and the SO(3) grid P (• | I). These similarities are then normalized using a softmax function to produce a non-parametric categorical distribution P (R | I). The final 3D rotation matrix R is determined either by taking the argmax of this distribution or by applying gradient ascent [49] .To generate the SO(3) equivolumetric grids, we utilize the hierarchical equal area isolatitude pixelation of the sphere (HEALPix) [24, 26] , consistent with methods used in [28, 35, 41, 49, 71] . To lift the S 2 HEALPix to SO(3) HEALPix, we create equal-area grids on the 2-sphere and cover SO(3) by threading great circles through each point using the Hopf fibration from [68] .This inference scheme effectively models objects with ambiguous orientations or symmetries by employing multiple hypotheses, thereby overcoming the limitations of single-modality predictions [44] .In addition to joint training with distributional cross-entropy loss [49] , our network can model the non-parametric and multi-modal distribution in pose space to address pose ambiguity and aid in modeling 3D symmetry. We input a 2D RGB image I ∈ R 3×224×224 . A ResNet backbone, pretrained on ImageNet, extracts feature maps of shape F ∈ R 2048×7×7 . We then perform dimension reduction using a 1x1 convolution to obtain F ′ ∈ R 512×7×7 . In the spherical mapper, the features are mapped onto an S2 grid generated by recursion level 2 of HEALPix on half of the sphere, and then sampled at 20 points, resulting in ψ ∈ R 512×20 . By converting ψ to the frequency domain, the spherical signals S ∈ R 512×49 are obtained. The Wigner-D representation is implemented in a flattened form across different frequency levels. For example, the matrix coefficients at a frequency level l are represented as a flattened vector of size (2l + 1) × (2l + 1). We use a maximum frequency level of L = 6, resulting in a total size of M = 455, computed as 6 l=0 (2l + 1) × (2l + 1). These coefficients are then flattened into a single vector for the Wigner-D prediction. The spherical convolution on the S 2 kernel uses an 8-dimensional hidden layer with global support to obtain intermediate SO (3) features in R 8×455 . After nonlinear activation, we finally obtain the 1-dimensional output Ψ ∈ R 1×455 using an SO(3) convolution with a locally supported filter to handle rotations up to 22.5 • . At inference, we employ a recursive level 5 of SO(3) HEALPix grid with 2.36 million points, achieving a precision of 1.875 • , as in [28, 35] .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes, we have described the implementation details in section 5.1, including the detailed training method and specific configuration to reproduce the experimental results. We have also described the benchmark settings and dataset split. (section 5.2) Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
benchmarks: 
 ModelNet10-SO(3) [39] is a common dataset for estimating a 3D rotation from a single image. The images are created by rendering CAD models from the ModelNet10 dataset [63] . The dataset includes 4,899 objects across 10 categories, each image is labelled with a single 3D rotation. The rotations are uniformly sampled from each CAD model. From a single CAD model, the training set comprises 100 3D rotations on SO(3), while the test set includes 4 unseen 3D rotations.PASCAL3D+ [64] is a widely-used benchmark for evaluating pose estimation in images captured in real-world settings. It includes 12 categories of everyday objects, which were created by manually aligning 3D models with their corresponding 2D images. This dataset presents challenges due to the significant variation in object appearances, the high variability of natural textures, and the presence of novel object instances in the test set. To be consistent with the baselines, we conduct training data augmentation using synthetic renderings [57] .ModelNet10-SO(3) Few-shot Views is used to evaluate the data efficiency of pose estimation models. Unlike the original ModelNet10-SO(3) [39] , we have expanded this to evaluate various amounts of training data, by setting the number of training views per CAD model to 3, 5, 10, 20, 30, 40, 50, 70, 90, and 100 . This benchmark verifies the sampling efficiency of our equivariant networks. We use the same test dataset as that of ModelNet10-SO(3).Evaluation Metrics. We calculate the angular error, measured in degrees using geodesic distance, between the network-predicted SO(3) pose and the ground-truth rotation matrix: θ Error (R, R) = cos -1 trace(∆R)-1, and ∆R = R RT . We adopt two commonly used metrics: the median rotation error (MedErr) and the accuracy within specific rotation error thresholds (Acc@15 • and Acc@30 • ). Results with solid lines of I-PDF [49] , I2S [35] , and RotLaplace [71] denote to a ResNet-50 backbone, while dotted lines indicate a ResNet-101 backbone. Our method outperforms all metrics and reduces training views. Baseline results [35, 71] were obtained using the source code provided by the authors.Table 1 shows the pose estimation results on the ModelNet10-SO(3) dataset, where our model outperforms all baselines across multiple evaluation metrics. Notably, our Wigner-D harmonics prediction network surpasses methods in non-probabilistic rotation estimation [3, 39, 76] , parametric probability distribution estimation [13, 41, 48, 54, 70, 71] , and non-parametric distribution prediction [28, 35, 49] , by leveraging SO(3) equivariance and rotation parametrization in the frequency domain.Table 2 shows the results on the PASCAL3D+ benchmark. Our SO(3) Wigner-D harmonics predictor achieves state-of-the-art performance on these challenging benchmarks. It demonstrates robustness to changes in object appearance, real textures, and generalizes well to novel object instances. We additionally report fine-scale accuracies, i.e., Acc@3 • , Acc@5 • and Acc@10 • , in Tables A1 and A2 , and class-wise evaluation of the results in Tables A3 and A4 of appendix B. Notably, as the number of training views from a single CAD model decreases, our model consistently achieves the highest accuracy and lowest error. This performance surpasses the baselines of direct rotation regression [3, 76] , parametric distribution parameters regression [71] , and non-parametric distribution estimation [35, 49] . This few-shot training experiment verifies that our SO(3)-equivariant model contributes to superior data efficiency and generalization to unseen rotations. To validate our choice of MSE loss, we experiment with various distance functions between the predicted output and the ground truth.Table 3 shows results validating design choices on ModelNet10-SO(3) with 20-shot training views, using a ResNet-50 backbone. First, we compare the effects of different rotation parametrizations on model performance by changing the prediction head and ground-truth rotations, to verify our proposed Wigner-D compared to other rotation representations. For all cases, we retained the backbone networks and SO(3)-equivariant layers. The only modifications were the output prediction dimension size and the ground-truth rotation representation. Our Wigner-D parametrization outperforms Euler angles (3 dim.), quaternions (4 dim.), axis-angle (4 dim.), and rotation matrices (9 dim.). This demonstrates that frequency domain rotation re-parametrization enables accurate 3D rotations when used with the SO(3)-equivariant spherical CNNs in the frequency domain.Table 4 compares various loss functions trained on ModelNet10-SO(3) with 20-shot learning using a ResNet-50 backbone. While Huber and L1 losses are alternatives, they do not perform as well as MSE in our context. Cosine loss measures only angle distances between vectors, ignoring magnitude, which is an essential factor in frequency-domain applications. Geodesic loss in the frequency domain is ineffective because it requires separate calculations for each frequency level of the Wigner-D matrix, potentially losing the precision of the original 3D rotation, as we truncate the Fourier basis at a frequency level of 6. Therefore, we choose MSE regression loss for our design choice given its simplicity and effectiveness.Acc@15°Acc@30°Rot Err. In Table 5 , we experiment with replacing SO(3)equivariant layers with conventional convolutional layers. Specifically, we use two-layer 1x1 convolutional layers with ReLU activation and a final linear layer with 455 output channels. The results indicate that CNNs without equivariant layers perform poorly, especially in terms of median error, suggesting that using the equivariant networks generalize better to unseen samples. Additionally, the Wigner-D prediction should be paired with an SO(3)-equivariant network to enable reliable 3D rotation prediction in the frequency domain.Lastly, we evaluate the impact of different SO(3) grids by switching from a HEALPix grid to a random SO(3) grid and super-Fibonacci spirals [1] , which use the same number of SO(3) rotations. Our Wigner-D harmonics predictor performs consistently, regardless of the SO(3) grid sampling type at inference time. Table 6 : Results on SYMSOL I and II [49] . We report the average log likelihood on both parts of the SYMSOL datasets. L wigner denotes the results obtained with our Wigner-D regression loss. L dist denotes the results using the distribution loss from I-PDF [49] , which are the same as the results of I2S [35] . The third row presents the results of joint training using both our regression loss and the distribution loss.Table 6 shows symmetric object modeling on the SYMSOL datasets [49] . Compared to the first row and second row [35] , our model with only the Wigner-D regression loss derives on sharp modalities, which can be less effective than [35] for symmetric objects in SYMSOL I.For clearly defined pose cases (e.g., SphereX in SYMSOL II), our Wigner-D loss alone performs well. However, in other SYMSOL II scenarios, the sharp distributions produced by our model can lead to low average log likelihood scores. This metric is particularly harsh on models with sharp peaks, making them vulnerable to very low scores in some failure cases.Figure 6 : Visualization of pose distribution on SYMSOL. The results are obtained by joint training with both our regression loss and the crossentropy distribution loss [35, 49] .In the third row, joint training of our method with the distribution loss [35, 49] achieves better performance than the baseline [35] , demonstrating its ability to model symmetric objects. These results highlight the potential of our method in handling complex symmetries and predicting multiple hypotheses. Figure 6 shows the visualization of pose distribution on the SYMSOL I and II datasets.Most real-world objects have unique, unambiguous poses, validating our single pose regression method (e.g., ModelNet10-SO(3), PAS-CAL3D+). If the task needs to cover symmetric cases, our model can be modeled with distribution loss [35, 49] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Yes, we provide sources and links to the publicly available data, the ModelNet10-SO(3) dataset and the PASCAL3D+ dataset in section B. We will also make our code publicly available after acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we proposed a novel method for 3D rotation estimation by predicting Wigner-D coefficients directly in the frequency domain using SO(3)-equivariant networks. Our approach effectively overcomes the limitations of existing spatial domain parameterizations of 3D rotations, such as discontinuities and singularities, by aligning the rotation representation with the operations of spherical CNNs. By leveraging frequency-domain regression, our method ensures continuous and precise pose predictions and demonstrates state-of-the-art performance across benchmarks like ModelNet10-SO(3) and PASCAL3D+. Additionally, it offers enhanced data efficiency and generalization to unseen rotations, validating the robustness of SO(3)-equivariant architectures. Our method also supports the modeling of 3D symmetric objects by capturing rotational ambiguities, with further accuracy improvements achievable through joint training with distribution loss. Future work can build on this foundation to explore frequency-domain representations in 3D vision tasks, develop more effective rotation representations for 3D space, and further optimize computational efficiency.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Yes, we have described the implementation details such as data splits, hyperparameters, type of optimizer in section 5.1. We also reported experiments to validate our design choice in section 5.5.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Yes, we describe our experiments for statistical significance in section B. 10 . Multiple experiments show that our experiments are not wrong or accidental. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Yes, we discuss the possible positive and negative impacts of this study in section G: broader impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do use the ImageNet pretrained model, but we do not discuss it in the paper as we do not believe it has a high risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, we are attributing any code, libraries, or illustrations in footnotes, or in the text at the section 5.1, and following the licenses. 
: 
 (e.g., HEALPix, SuperFibonacci ..) (e.g., Gaussian, vMF, Bingham ..)Figure 1 : Types of representations for 3D rotation prediction. Existing methods consider predicting 3D rotations in the spatial domain. Our method predicts Wigner-D coefficients in the frequency domain, to obtain accurate pose in continuous space using an SO(3)-equivariant network.Predicting the 3D pose of objects, i.e., position and orientation, in 3D space from an image is crucial for numerous applications, including augmented reality [56] , robotics [4, 5, 60, 65] , autonomous vehicles [21, 47] , and cryo-electron microscopy [75] . Estimating 3D orientation is particularly challenging due to rotational symmetries and the non-linear nature of rotations. In addition, unlike translations, rotations introduce unique challenges such as gimbal lock and the requirement for continuous, singularity-free representations. Existing methods often learn 3D rotations using spatial domain parameterizations like Euler angles, quaternions, or axis-angle representations, as illustrated in Figure 1 . However, these parameterizations suffer from issues such as discontinuities and singularities [51, 55, 76] , which can hinder the performance and reliability.SO(3)-equivariance enables accurate 3D pose estimation and improves generalization to unseen rotations. It ensures that outputs consistently change with the 3D rotation of the input, maintaining rotational consistency between the input and output across network layers. Despite its importance, many existing methods [3, 41, 49, 71, 76] often design networks without considering SO(3)-equivariance, resulting in suboptimal performance when dealing with 3D rotations. In addition, in the context of spherical CNNs [9, 10, 12, [16] [17] [18] 36] for efficient SO(3)-equivariant operations, the 3D rotation parametrization in the spatial domain is inadequate because these SO(3)-equivariant networks operate in the frequency domain.To address these challenges, we propose an SO(3)-equivariant pose harmonics regression network that directly predicts Wigner-D coefficients in the frequency domain for 3D rotation regression. Building on prior work [28, 35] , our method leverages the properties of spherical CNNs [11] , which operate in the frequency domain, to guarantee SO(3)-equivariant output representation. By directly regressing Wigner-D matrix coefficients, our approach eliminates the need to convert outputs into spatial representations during training, ensuring alignment with the operations of spherical CNNs. This design allows us to bypass the limitations inherent in traditional spatial parameterizations-such as discontinuities and singularities [51, 55, 76] -resulting in more precise and continuous pose estimation. We further introduce a frequency-domain MSE loss to enable continuous training of 3D rotations, with the flexibility to incorporate distributional losses [49] for effectively capturing rotational symmetries in objects. Our method achieves state-of-the-art performance on standard single object pose estimation benchmarks, including ModelNet10-SO(3) and PASCAL3D+, demonstrating high sampling efficiency and strong generalization to unseen 3D rotations.2 Related Work SO(3) pose regression. The choice of rotation representation is a fundamental aspect of the current SO(3) pose estimation methods. In the early stages of deep learning, methods for SO(3) pose regression choose the rotation representation by direct cosine matrices [29, 69] , Euler angles [37, 45, 46, 57, 58] , quaternions [5, 13, 31, 32, 66, 74] , and axis-angles [14, 20, 59 ]. However, according to [76] , for any representation R in a Euclidean space of dimension d ≤ 4, such as Euler angles and quaternions, R is discontinuous and unsuitable for deep learning. In addition, Euler angles can cause gimbal lock, which restricts certain rotations, whereas quaternions avoid this issue but their double representation of rotations in SO(3) can lead to complications such as local minima in optimization problems. As an alternative, a continuous 6D representation with Gram-Schmidt orthonormalization [76] and 9D representation with singular value decomposition (SVD) [8, 38] have been proposed, and [7] proposes manifold-aware gradient layer to facilitate the learning of rotation regression. Denoising diffusion models are employed in the context of SO(3) pose regression [61] , or for solving pose estimation by aggregating rays [72] . In contrast to existing SO(3) pose regression methods that formulate rotation representations in the spatial domain, we define the Wigner-D coefficients as the output of the network in the frequency domain, using SO(3)-equivariant networks.Pose estimation with a parametric distribution. To model rotation uncertainty, parametric distributions on the rotation manifold are employed in a probabilistic manner. [54] predicts parameters of a mixture of von Mises distributions over Euler angles using Biternion networks. [5, 13, 23] utilize the Bingham distribution over unit quaternions to generate multiple hypotheses of rotations. [48, 70] leverage the matrix Fisher distribution [33] to construct a probabilistic model for SO(3) pose estimation. Additionally, [71] propose the Rotation Laplace distribution for rotation matrices on SO (3) to suppress outliers, and the Quaternion Laplace distribution for quaternions on S 3 . Nevertheless, parametric models rely on predefined priors. In contrast, our model uses non-parametric modeling during inference to capture more complex pose distributions.Pose estimation with a non-parametric distribution. Probabilistic pose estimation can also be achieved by predicting non-parametric distributions. IPDF [49] introduces the estimation of arbitrary, non-parametric distributions on SO(3) using implicit functions with MLPs, and Hyper-PosePDF [27] uses hypernetworks to predict implicit neural representations by Fourier embedding. ExtremeRotation [6] predicts discretized distributions over N bins for relative 3D rotations trained with cross-entropy loss. RelPose [40, 73] uses an energy-based formulation to represent distributions over the discretized space of SO(3) relative rotation. Several SO(3)-equivariant modeling methods construct non-parametric distributions by utilizing icosahedral group convolution [34] , projecting image features orthographically onto a sphere [35] , and satisfying consistency properties of SO (3) by translating them into an SO(2)-equivariance constraint [28] . RotationNormFlow [41] uses discrete normalizing flows to directly generate rotation distributions on SO (3) . These non-parametric methods, which are trained with loss functions in discretized distributions, such as cross-entropy and negative log-likelihood, tend to lose precision in rotation prediction. In contrast, our method predicts continuous SO(3) transformations through regression, eliminating the need to approximate SO (3) poses within a discretized space and enabling our model to achieve accurate 3D rotations.3 Preliminary 
 
 Appendix: We recommend checking out a detailed overview of learning 3D rotations in [51, 55] . Rotations in both 2D and 3D spaces can be represented using various mathematical frameworks, each with its own advantages and limitations, crucial for applications in fields such as computer graphics, robotics, and deep learning. In 2D space, rotation angles can be expressed as following SO(2) representations: angle (α) or Trigonometric functions (cos(α), sin(α)).In 3D space, Euler angles in R 3 are a representative form of SO(3) representation. Euler angles have 3 DoF consisting of three angles α, β, γ ∈ [-π, π) to describe a 3D rotation (roll, pitch, yaw). 3D rotation matrix can be composed of a fixed sequence rotation using the angles Quaternions in S 3 is for 4-dimensional complex number to represent a 3D rotation. A quaternion q is composed of one real part and three imaginary parts: q = w + xi + yj + zk, where w, x, y, z are real numbers and i, j, k are the fundamental quaternion units. Quaternions can prevent gimbal lock, a problem that occurs with Euler angles where one degree of freedom is lost during 3D rotation.R(α, β, γ) = R z (α)R y (β)R z (γ).Axis-angle in R 4 consists of an angle θ and an axis vector u in 3D space. To rotate a point p ∈ R 3 about the axis u by an angle θ, you can use Rodrigues' rotation formula:p ′ = pcos(θ)+u×psin(θ)+ u(u • p)(1 -cos(θ)).The axis-angle representation can be converted into a rotation matrix or a quaternion. Even it has advantages of intuitive form and compact to describe 3D rotation with four parameters, the axis-angle can suffer from singularities in a scenario of angle multiples of θ = 2π.6D representation [3, 76] is a relatively newer concept compared to the traditional representation like Euler angles, rotation matrices and quaternions. A rotation is described by two 3D vectors that are orthogonal to each other, and the rotation matrix can be obtained by Gram-Schmidt orthonormalization (GSO). This representation can be directly predicted and optimized by deep networks because it avoids the non-linear constraints found in quaternions and rotation matrices. Quaternions require maintaining a unit norm, which introduces complexity in ensuring the quaternion remains normalized throughout the optimization process. Therefore, [7, 8, 76] adopt this 6D representation with GSO.9D representation [3, 38] is direct parametrization of 3 × 3 matrix, which can be projected to SO(3) using singular value decomposition (SVD). Predicting 9D representation for rotation matrices involves orthogonality constraints, meaning the rows and columns must remain orthonormal, and determinant constraints, where the determinant must equal +1. These constraints complicate the learning and optimization process, making this representation more suitable for direct prediction and optimization by deep networks. Therefore, 9D representation is less common as a direct method to predict rotation, but [7, 8, 38] use this to mitigate issues associated with discontinuous parameterizations of pose.3D rotation in the frequency domain is accomplished by manipulating spherical harmonics coefficients. Spherical harmonics Y l m (x) = Y l m (θ, ϕ) are a function defined on the surface of a sphere, where θ and ϕ are the polar and azimuthal angles, respectively. Here, l represents the degree of the spherical harmonics, m is the order. To ensure computational feasibility, we truncate the degree of harmonics to a finite L = l max . Rotations of spherical functions can be represented by matrices that operate on the coefficients of their harmonics expansion. The rotation of spherical harmonics is expressed via the shift theorem [43] of spherical harmonics:EQUATIONwhere Λ g Y l m (x) denotes the spherical harmonic Y l m (x) after rotation by g, and Λ g is the rotation operator. Spherical harmonics Y l m (x), defined by degree l and order m (l ≥ 0, |m| ≤ l), use x to represent spherical coordinates. The matrix U l mn (g) forms part of the irreducible unitary representation of SO(3), showing how each harmonic is transformed under rotation. The sum over all orders n from -l to l, |n|≤l , shows that Y l m is a linear combination of all harmonics of degree l.This rotation of spherical harmonics can be described by the Wigner-D matrix D l mn (R), which is a unitary matrix that describes the effect of a rotation R on the spherical harmonics basis functions. We can rewrite U l mn (g) = D l mn (α, β, γ). Each element of the matrix represents the amplitude and phase shift that a spherical harmonic Y l m undergoes due to the rotation R. The rotation R can be specified by Euler angles α, β, and γ in the ZYZ-axes configuration. The matrix elements D l mn (α, β, γ) can be explicitly expressed as:EQUATIONwhere d l mn (β) are the elements of the Wigner (small) d-matrix, which depend only on the angle β and are real-valued. These elements capture the intermediate rotation about the Y-axis, where the angle β represents the tilt of the axis of rotation. To rotate a spherical harmonic expansion of a function f , represented as:EQUATIONwe need to account for the coefficients f l m of the expansion. The rotated coefficients f ′l m are computed:EQUATIONwhere D l mn (R) encodes the effect of the rotation on the original coefficients. This transformation preserves the orthonormality and completeness of the spherical harmonics basis, ensuring that the rotated function f ′ (θ, ϕ) remains a valid representation of the original function f (θ, ϕ) under the rotation R. The expansion coefficients f l m can be calculated using the original spatial coordinates (θ, ϕ) of the sphere surface according to the spherical harmonic expansion:EQUATIONwhere Y l m (θ, ϕ) denotes the complex conjugate of the spherical harmonic function Y l m (θ, ϕ).B.1 Results of Finer Threshold AccuracyB.1.1 ModelNet10-SO(3)Table A1 presents a comparison of existing methods with different backbones on the ModelNet10-SO(3) dataset 3 , highlighting performance across multiple accuracy thresholds and median error. For the ResNet-50 backbone, our method achieves the highest accuracy at 3°(0.422) and 5°(0.640) with a median error of 15.1°, outperforming the existing methods [13, 28, 35, 49] . In particular, our model demonstrates significantly better performance than the strong baselines [28, 35] that use equivariant networks to estimate non-parametric SO(3) healpix distribution at the finer thresholds. Compared to [35] , our model achieves 11.2%p, 7.9%p, and 3.9%p higher at the 3°, 5°, and 10°thresholds, respectively. For the ResNet-101 backbone, our method also demonstrates superior performance with the highest accuracy at 3°(0.513) and 5°(0.688), and the lowest median error of 11.9°, compared to [39, 41, 48, 71] . The table includes accuracies at 10°, 15°, and 30°thresholds, where our method consistently shows top performance across these metrics. The scores are averaged across all ten object categories.Method Backbone Acc@3°Acc@5°Acc@10°Acc@15°Acc@30°Med. (°) Table A1 : Comparison with finer thresholds on ModelNet10-SO(3). We compare additional thresholds, including Acc@3°, Acc@5°, and Acc@10°. The tables are organized by the size of the backbone. The scores are averaged across all ten object categories.Method Acc@3°Acc@5°Acc@10°Acc@15°Acc@30°Med. (°) Table A2 : Comparison with finer thresholds on PASCAL3D+. We compare additional metrics, including Acc@3°, Acc@5°, and Acc@10°, adding on the Table 2 . Most baselines [35, 39, 41, 42, 48, 49, 71] , including ours, use ResNet-101 backbone networks in this experiment.Table A2 presents a comparison of various methods on the PASCAL3D+ dataset, focusing on finer accuracy thresholds (Acc@3°, Acc@5°, and Acc@10°) and median error. The table includes methods that primarily use the ResNet-101 backbone, comparing our approach against several existing methods.Our method achieves the highest accuracies with 0.153 at 3°, 0.310 at 5°, and 0.595 at 10°, with the lowest median error of 8.6°. Compared to the previous state-of-the-art Yin et al. [71] , our method shows performance improvements of 1.9%p at 3°, 1.8%p at 5°, and 2.1%p at 10°, while also reducing the median error by 0.7°.Figure A1 shows the results with finer thresholds, Acc@3°, Acc@5°, and Acc@10°, on the ModelNet10-SO(3) few-shot training views, which are additional results to Figure 5 . The graphs illustrate that our method outperforms all other methods across all metrics (Acc@3°, Acc@5°, and Acc@10°) and requires fewer training views to achieve high accuracy, even at finer thresholds. This shows that our model is capable of more precise pose estimation with less number of training data, proving the data efficiency of our SO(3)-equivariant harmonics pose estimator. Baseline results [35, 71] were obtained using the source code provided by the authors.Table A3 Our method outperforms all metrics and reduces training views even at finer thresholds. For comparison, the I2S (ResNet-50) model [35] is shown with a blue line, and the RotLaplace (ResNet-50 and ResNet-101) models [71] are depicted with purple solid and dashed lines, respectively. as desks and bathtubs, may have ambiguous poses due to symmetry. This poses a challenge for methods that cannot handle uncertainty over orientation. However, in terms of accuracy at 15°, our model consistently achieves the best performance in the desk and bathtub categories, demonstrating robustness against pose ambiguity and symmetry. The table is divided into sections for ResNet-50 and ResNet-101, indicating different network architectures used. For ResNet-50, our method achieves the lowest average median error of 15.1°, with particularly strong performance in 9 categories: bed (2.7°), chair (3.8°), desk (4.2°), dresser (2.7°), tv monitor (2.7°), night stand (3.4°), sofa (7.2°), and toilet (3.0°). For ResNet-101, our model demonstrates the lowest average median error of 11.9°. Although Liu et al. (Uni.) [41] generally obtain better results in terms of median error with the ResNet-101 backbone, our model outperforms Liu et al. (Uni.) [41] on Acc@15°in most cases. This indicates that our model estimates poses correctly at a finer level of detail.In terms of accuracy at 15°, our method achieves the highest average accuracy of 0.759 for ResNet-50, with best performance in all categories. For ResNet-101 at 15°, our model leads with an average accuracy of 0.773, achieving state-of-the-art performance in 8 out of 10 categories. In terms of accuracy at 30°, our method achieves the highest average accuracy of 0.773 for ResNet-50, with the best results in 9 out of 10 categories. For ResNet-101 at 30°, our model maintains the highest average accuracy of 0.905, with strong results in categories such as Chair, Desk, TV Monitor, and Sofa. Overall, our equivariant harmonics pose estimator demonstrates superior performance across both ResNet-50 and ResNet-101 architectures in terms of both median error and accuracy at different angles. This highlights its effectiveness and robustness across various object categories in the ModelNet10-SO(3) dataset, consistently outperforming other recent methods in most categories and metrics.Table A4 presents comprehensive comparisons on the PASCAL3D+ dataset across 12 categories (Aeroplane, Bicycle, Boat, Bottle, Bus, Car, Chair, Dining Table, Motorbike, Sofa, Train, TV Monitor) using median error and Acc@30°metrics. This benchmark is challenging due to significant variations in object appearances, high variability of natural textures, and the presence of novel object instances in the test set. Our method demonstrates superior performance with the lowest average median error of 8.6°and the highest average accuracy of 0.892, excelling in categories such as "aero," "bike," "car," "chair," "table," and "mbike" in median error, and achieving best performance in 8 out of 12 categories in accuracy at 30°. The results indicate that different methods have varying strengths across different categories, with our SO(3)-equivariant pose harmonics estimation method consistently outperforming others in both accuracy and error metrics, demonstrating its robustness and efficacy. 3) grid at inference. To analyze the sensitivity of discretization on precision (Q of Fig. 4 ), we vary the recursion levels of the SO(3) HEALPix from 0 to 6. We use a ResNet-50 backbone on ModelNet10-SO(3). able A7: Results of using transformer instead of convolution. We train our models by replacing the backbone with Vision Transformer (ViT).Table A5 reports the effect of varying the grid size (Q) on performance for the ModelNet10 benchmark. We observe comparable results in common evaluation metrics, such as Accuracy at 15 degrees (Acc@15) and 30 degrees (Acc@30), even with a lower grid resolution (Q = 4.6K). A higher resolution grid (18.87M) improves performance under stricter evaluation thresholds. With our chosen grid size of Q = 2.36M, the model achieves strong performance sufficiently, particularly for low-threshold metrics like Acc@3. Table A1 provides additional comparisons to baseline methods.Therefore, we carefully claim that our learning method focuses on continuous rotations. Our model directly learn the Wigner-D coefficients, which are derived from 3D rotations (Euler angles), without any discretization during the training phase. During inference the use of the SO(3) HEALPix grid serves two purposes: 1) To convert SO(3) rotations from the frequency domain to the spatial domain, and 2) To address pose ambiguity by providing multiple solutions. As a result, we obtain a distribution with very sharp modality. By taking the argmax of this distribution, we achieve sufficient precision in 3D orientation estimation, specifically around 1.5 • .Maintaining continuity in rotations allows our method to deliver more accurate and precise pose predictions, giving us a clear advantage over the methods that experience precision loss from discretization during training. As a result, we achieve consistently high accuracy across different levels of discretization.Table A6 shows the evaluation results using gradient ascent on the predicted SO(3) pose distribution in ModelNet10-SO(3), to fully exploit the distribution prediction in Sec. 4.3 during inference time. While gradient ascent does provide some performance improvement, the increase in inference time outweighs these gains, so argmax is our preferred method for simplicity and fast evaluation.Table A7 presents the results when transformers are used as the backbone network instead of the convolutional feature extractor, trained on ModelNet10-SO(3) with 20-shot learning. We trained the model using a Vision Transformer (ViT) backbone pre-trained by the geometric task of cross-view masked image modeling [62] . Although the ViT is heavier and requires longer training time (1.4x), its performance actually declines. This suggests that convolutional image feature extractor may still be more effective for this 3D orientation estimation task.Table A8 presents the impact of varying the maximum frequency level L by truncation for efficient SO(3) group convolutions on pose prediction accuracy and median error. The results show that as L increases from 1 to 5, there is a consistent improvement in accuracy metrics. The optimal performance is observed at L = 5.Acc@15 • Acc@30 Beyond this point, additional frequency levels do not contribute to improved accuracy and can even degrade performance. When L > 5, the accuracy does not improve significantly and starts to fluctuate, with rotation error remaining relatively low up to L = 10. However, at L = 11, accuracy starts to decline more noticeably. For L ≥ 12, there is a sharp decline in performance, with Acc@15°dropping to 0.5815 and continuing to decrease, Acc@30°following a similar trend, and rotation error increasing significantly to over 55°. We infer that high frequencies do not improve performance despite the increase in learnable parameters because they lead to overfitting to high-frequency noise. This overfitting occurs when the high-frequency model captures irrelevant noise and patterns in the training data, reducing its generalizability to new, unseen data.In conclusion, including higher frequencies (L > 6) appears to introduce more noise or overfitting, leading to decreased accuracy and increased rotation error. However, we choose a maximum frequency level L = 6 for a fair comparison to [35] , and to balance efficiency and accuracy.Acc@15°Acc@30°Rot Err. able A9: Validating the design choice of the spherical mapper. The 'MLP mapper' denotes the Fourier projection, which directly maps image features to harmonics using an MLP, and the 'spherical mapper' denotes our choice of orthographic projection [35] .The spherical mapper in Sec. 4.1 maintains the geometric structure of the image when projecting onto the S 2 sphere, as detailed in [35] . This method involves lifting the 2D image onto the sphere and converting spherical points using spherical harmonics. Table A9 shows that the spherical mapper outperforms simple Fourier transforms on 2D feature maps.Using depth information from methods like DepthAnythingv2 for 3D lifting is a good idea and can enhance geometric accuracy. Additionally, centroid ray regression has been explored in research such as [72] . However, incorporating external depth modules increases computational costs and broadens our research scope, so we consider this for future work.Evaluating out-of-distribution (OOD) performance is generally not the primary focus in the context of this 3D orientation estimation task. However, we have conducted OOD generalization experiments using our proposed method by training the model on different datasets, between ModelNet10 and PASCAL3D+. The results are presented in Table A10 . As the results indicate, the model does not perform well when evaluated on an out-of-distribution dataset. Nevertheless, we recognize this as an important area for future research.Training Dataset Evaluation Dataset Acc@15 Acc@30 Rot. Err.ModelNet A11 : Comparison of computational cost. We compare the inference time of one image and GPU memory consumption on ModelNet10-SO(3) test split. To measure the inference time, we average the results of total 18,160 samples of ModelNet10-SO(3) test split.Table A11 presents a detailed comparison of computational cost, focusing on both inference time and GPU memory consumption. The comparison includes the recent baselines; Image2Sphere (2023) [35] , RotationLaplace (2023) [71] , and RotationNormFlow (2023) [41] . The evaluation is based on the ModelNet10-SO(3) test split, averaging the results from a total of 18,160 samples. We use a machine equipped with an Intel i7-8700 CPU and an NVIDIA GeForce RTX 3090 GPU, utilizing a batch size of 1. The key metrics presented in the table are the inference time per frame (in seconds) and the GPU memory consumption (in gigabytes).Our model demonstrates the best inference time of 0.0109 seconds per frame, significantly outperforming other models in terms of speed. This efficient inference time translates to approximately 92.5 frames per second (FPS) for an image size of 224x224, making our model suitable for real-time applications. However, this performance comes at the cost of higher GPU memory consumption, which is recorded at 5.172 GB. Since our model performs all operations on the GPU, we achieve a temporal advantage in inference time, despite having many overlapping modules with Klee et al. [35] , whose model performs some computations on the CPU. In summary, our model achieves the best inference time, facilitating real-time application potential, by trading off increased GPU memory consumption.Table A12 presents the results of a 5-trial experiment to evaluate the training sensitivity of our models, with ResNet-50 and ResNet-101, on the ModelNet10-SO(3) 20-shot training views. The table lists individual trial results, along with the average (µ) and standard deviation (σ) for each metric. The standard deviation values (σ) for both backbones are relatively small across all metrics, suggesting that the models yield consistent results over multiple trials. For instance, the standard deviation of Acc@3°for ResNet-50 is 0.0047, and for ResNet-101, it is 0.0052, which are both quite low. This low variance indicates that the training results are stable and reproducible. These findings highlight the robustness and reliability of the training process and the effectiveness of ResNet-101 for the given task.We utilize the e3nn library [22] for S 2 and SO(3) convolutions for efficient handling of both Fourier and inverse Fourier transforms, healpy [24, 77] for HEALPix grid generation, and PyTorch [50] for model implementation. We use a machine with an Intel i7-8700 CPU and an NVIDIA GeForce RTX 3090 GPU. With a batch size of 64, our network is trained for 50 epochs on ModelNet10-SO(3) taking 25 hours, and for 80 epochs on PASCAL3D+ taking 28 hours. We start with an initial learning rate of 0.1, which decays by a factor of 0.1 every 30 epochs. We use the SGD optimizer with Nesterov Trial Acc@3°Acc@5°Acc@10°Acc@15°Acc@30°Med Err. momentum set at 0.9. Unlike baselines that encode object class information via an embedding layer during training [35] and both training and testing [41, 71] , our model does not use class embeddings, maintaining a class-agnostic framework during both training and testing. Additionally, we train a single model for all categories in each dataset, unlike [7] , which trains separate models for each class.We compare our method against competitive single-view SO(3) pose estimation baselines including regression methods and distribution learning methods. Zhou et al. [76] predict 6D representations using Gram-Schmidt orthonormalization processes for 3D rotations, analyzing the discontinuities in rotation representations. Brégier [3] extends deep 3D rotation regression with a differentiable Procrustes orthonormalization, which maps arbitrary inputs from Euclidean space onto a non-Euclidean manifold. Tulsiani and Malik [58] train a CNN using logistic loss to predict Euler angles. Mahendran et al. [42] predict three Euler angles using a classification-regression loss to estimate fine-pose while modeling multi-modal pose distributions. Liao et al. [39] also predict Euler angles using a classification-regression loss by introducing a spherical exponential mapping on n-spheres at the regression output.On the other hand, the other baselines are generating probability distributions for estimating SO(3) pose. Prokudin et al. [54] represents rotation uncertainty with a mixture of von Mises distributions over each Euler angle, while Mohlin et al. [48] predicts the parameters for a matrix Fisher distribution. Deng et al. [13] predict multi-modal Bingham distributions. Murphy et al. [49] trains an implicit model to generate a non-parametric distribution over 3D rotations. Yin et al. [70, 71] predict the parameter of SO(3) parametric distribution using matrix-Fisher distribution and rotation Laplace distribution, respectively. Klee et al. [35] predicts non-parametric distribution with equivariant feature prediction by orthographic projection, and Howell et al. [28] extends to construct neural architectures to satisfy SO(3) equivariance using induced and restricted representations. Liu et al. [41] use discrete normalizing flows for rotations to learn various kinds of distributions on SO (3) . Results are from the original papers when available.Figures A2 and A3 show qualitative results randomly selected from ModelNet10-SO(3) and PAS-CAL3D+, respectively. For visualization, we display distributions over SO(3) as proposed in I-PDF [49] . To illustrate the SO(3) distribution, we use the Hopf fibration to visualize the entire space of 3D rotations [68] . This approach maps each point on a great circle in SO(3) to a point on the discretized 2-sphere and uses a color wheel to indicate the location on the great circle. Essentially, each point on the 2-sphere represents the direction of a canonical z-axis, and the color represents the tilt angle around that axis. To depict probability score, we adjust the size of the points on the plot. Lastly, we present the 2-sphere's surface using the Mollweide projection.Comparison of pose visualization. Figures A4 and A5 show a comparison of pose visualizations on ModelNet10-SO(3) and PASCAL3D+, respectively. This visualization method is the same to those used in Figures A2 and A3 . We compare our model to the I2S [35] baseline. The numbers next to "Err" above the input images represent the error in degrees between the model's predicted pose and the ground truth (GT) pose. These results demonstrate that our model provides more accurate and precise pose estimations, even in cases where the I2S baseline fails. Additionally, on the PASCAL3D+ benchmark, which includes objects captured in real-world scenarios, our model consistently shows correct pose estimations, particularly in challenging scenarios where the I2S baseline struggles.Our proposed method significantly advances 3D rotation estimation accuracy; however, a notable challenge in pose estimation is the issue of pose ambiguity, particularly for objects with symmetrical features or those viewed from certain angles, e.g., bathtub category in ModelNet10-SO (3) . Despite high accuracy, our method can suffer from significant errors due to the loss of spatial information when projecting 3D data onto spherical harmonics. Future work could integrate additional contextual or spatial information to mitigate these ambiguities, improve reliability, and enhance the model's robustness in diverse scenarios. Additionally, while the mathematical rigor of using spherical harmonics and Wigner-D coefficients supports the model's success and improves interpretability through equivariant networks, further exploration is needed to make the model more interpretable. Finally, the computational cost associated with Wigner-D coefficients and SO(3)-equivariant networks should be improved to enhance practicality for real-time applications and deployment on devices with limited processing power.The method proposed in this paper has several potential positive societal impacts. First, it can enhance robotics and automation. Accurate 3D pose estimation is crucial for these fields, and improved accuracy can lead to more efficient and safer robotic systems in manufacturing, healthcare, and service industries. Second, it can significantly advance augmented reality (AR) applications by providing more precise alignment of virtual objects with the real world, which can be beneficial in education, gaming, and industrial design. Third, the method can improve autonomous vehicles, which rely on precise 3D pose estimation to understand their environment, contributing to safer and more reliable autonomous driving systems. Finally, in medical imaging, accurate pose estimation can improve the analysis and interpretation of complex 3D data, aiding in diagnosis and treatment planning.However, the paper also suggests potential negative societal impacts. Improved pose estimation techniques could be used in surveillance systems, leading to privacy concerns if deployed without proper regulations and oversight. Enhanced 3D pose estimation could be exploited to create more realistic deepfakes, contributing to the spread of disinformation and manipulation. Deployment of these technologies could inadvertently reinforce existing biases if the training data is not representative of diverse populations, leading to unfair treatment of specific groups in applications like security and hiring. Additionally, the misuse of accurate pose estimation in security-sensitive areas, such as military applications or unauthorized monitoring, could pose significant risks.To address these potential negative impacts, several mitigation strategies could be implemented. Controlled release of models and methods to ensure ethical and responsible use is one approach. Regular audits to ensure the training data and algorithms do not propagate biases are also crucial.Implementing robust privacy protection measures can safeguard individual privacy in applications involving surveillance. Developing complementary technologies to detect and mitigate the effects of deepfakes and other forms of disinformation is another necessary step. NeurIPS Paper ChecklistWe use the visualization tools available in the source code of the image2sphere GitHub repository.We use the illustration to represent the SO(3) grid P (• | I) from s2fft github repository[53].We obtained the ModelNet10-SO(3) dataset from the following link: https://github.com/leoshine/ Spherical_Regression/blob/master/dataset/ModelNet10-SO3/Readme.md , and the PASCAL3D+ dataset from the following link https://cvgl.stanford.edu/projects/pascal3d.html. 


Decision: Initially there was some confusion about the similarity to [34] and the contribution. The authors promised to update the paper according to the rebuttal. As reviewer HnJh states "While the contribution is relatively small in my view, it is sufficient and the material in the rebuttal has strengthened it". Three reviewers (HnJh, f5v8, iMp8) agree on this statement. 

The fourth reviewer (HCET) has reservations because many things were not clear initially and need to be udpated in the manuscript.

Overall, the AC recommends trusting the authors that they will update the manuscript according to the comprehensive discussion in the rebuttal and recommends to accept the paper.

Reviews: review: paper_summary: paper_summary: In this work authors predicts SO(3) poses for objects by predicting Wigner-D coefficients in frequency space. Similar to other work [1], it first lifts 2D features to 2-sphere using pre-defined grid and orthographic projection to sphere using this grid, convert them to frequency domain, applies SO(3)-equivariant spherical convolution on top to predict a vector of Wigner-D coefficients. At inference time, these Wigner-D coefficients are mapped back to spatial rotation R.  

Authors show experiments on multiple datasets showing better accuracy compared to other work. Authors also show few-shot views training accuracy and ablation study for different design components of the network.


References

[1] Klee, David M., et al. "Image to sphere: Learning equivariant features for efficient pose prediction." arXiv preprint arXiv:2302.13926 (2023).
strength_weakness: strength_weakness: Some pose visualization on objects would be nicer and should be compared to other methods.
questions: questions: How would this method work for in-the-wild datasets?

Would this method be able to handle occlusions?

If we have to use "transformers" instead of convolution, what would change?
limitations: limitations: Authors discuss limitations of their approach in the last section. Societal impact is discussed in appendix in detail.
paper_summary: paper_summary: The paper proposes a method for regressing the rotation of an object from an image, where several (~10s) of training views of the object with known rotations are available. In particular, the approach proposes a rotation-equivariant network that predicts continuous Wigner-D matrix coefficients in the frequency domain, which can be converted into a discretised heatmap in SO(3) to extract the predicted rotation(s). The contributions are (1) a method for predicting in the SO(3) frequency domain directly and (2) an approach for decoding this to a rotation at inference time. The approach is evaluated on two datasets with respect to rotation error and demonstrates good performance. Ablations and sensitivity analyses in the paper and appendix validate many of the design choices.
strength_weakness: strength_weakness: W1. Contributions/claims.
W1.1 The method in sections 4.1-4 is presented as part of the contribution, but appears to be the same as previous work, including [34]. If this is not the case, the design differences and the rationale should be better explained. If it is the case, this limits the contribution to sections 4.5-6, which is (a) directly predict and optimise the Wigner-D matrix coefficients, and (b) convert to a discretised heatmap and take the argmax. While (a) is likely key to the success of the method, it would appear to be a small variation on an existing approach; (b) on the other hand is very minor and indeed not claimed as a contribution. What this reviewer would like to see here is an expansion of these sections with some interrogation (and testing, in section 5) of the design choices. For example, is MSE a theoretically-grounded choice here, what are the weights w, why argmax instead of clustering for multi-hypothesis outputs?

W1.2 Relatedly, the list of contributions has 3 items, but 2-3 do not seem to be contributions of this paper. That is, the contribution of leveraging SO(3)-equivariant representations to guarantee equivariance has clearly been done before (e.g., [27,34,40,33], among others); and achieving SOTA performance is not a contribution.

W1.3 The claim that the proposed method is continuous and therefore better than the discretised methods (e.g., line 97) is dubious, when the predicted rotations are indeed discretised (section 4.6). The claim of loss of precision due to discretisation (for other methods) is also not validated in the experiment section; and one would assume that this method would also lose precision for the same reason (as Q decreases).

W1.4 The claim that parametric models are insufficiently expressive for this task (L82) is not validated. It seems likely, but would depend on the model and would ideally be directly tested to back up the claim.

W2. Design choices. (Partially overlaps with above)
W2.1 Spherical mapper. While this is taken from existing work, it is not explained why it's a reasonable design decision to warp a 2D planar signal to the sphere, or given any motivation. It's clear that it's necessary for the method, to allow S2- and SO(3)-equivariant operations, but geometrically it seems quite dubious. Could the feature warping be made more geometrically meaningful by first running the image through DepthAnythingv2 and projecting out from a point on the crop's centroid ray?

W2.2 The text mentions other approaches for non-linearities that avoid the extra (i)FFT steps (L233) but does not test or validate this design choice.

W2.3 MSE (5). This does not seem to be a theoretically- or empirically-justified choice but is key to the contribution. Lines 254-258 loosely refer to this, but this section would seem to be a good opportunity to provide a justification for the design.

W2.4 Discretised distribution on SO(3). The design choices around this part are not interrogated or justified - it reads as rushed or incomplete. For example, it states that the inference scheme "models objects with ambiguous orientations or symmetries by employing multiple hypotheses", but this is never done, as far as I can see. Instead, the argmax or gradient ascent is used, returning a single mode. Nowhere does the paper demonstrate how to use the approach to extract multiple hypotheses or test efficacy on extracting equivalent rotations for symmetric objects (see experiment section below). This is a missed opportunity and leads to the paper arguing against itself.

W3. Clarity.
W3.1 As alluded to earlier, the treatment is not self-contained. In particular, the Wigner-D matrix representation is central to the method and, for completeness and ease of understanding, should be included in the main paper. The reader would want to see the formulation that directly connects the task to the representation, instead of leaving that jump a bit vague (L121) and referring to the appendix. The function f in (4) and (12) are presumably not the same, but are not specifically defined for this task. Essentially, it reads as being incomplete, missing some of the material that connects the theory to the specific instantiation here.
W3.2 L248. Please provide/derive the weights w for completeness.

W4. Experiments.
W4.1 The comparisons (especially in Table 1) would be more meaningfully grouped by the backbone extractor. For example, we should actually be comparing the similar method [34] with "ours (ResNet-50)", where a 0.6deg improvement is achieved. Similarly, it would be useful to compare the ResNet101 version on [34] in this table (like the version in Tab 2) since it is the closest existing method.
W4.2 Missing dataset. SYMSOL, used in the closest related work [34,40], is not included in this paper, despite being a good opportunity to evaluate the performance with complex symmetries. Leaving it out gives the impression that the method cannot handle such cases, even though it seems that it ought to be able to.
W4.3 Multi-hypotheses untested (despite L271). This claim ought to be experimentally tested, since it is such a relevant and interesting potential feature of the approach.
W4.3 Ablations/sensitivity analyses. A sensitivity analysis on the influence of discretisation on precision (Q) is missing (despite claim in L96); the choice of MSE is not tested.


W5. Minor.
W5.1 The paper needs a proofread - there are many typos, syntactical and grammatical errors.
W5.2 Confusing formatting error with footnote 2 appearing the page before it appears in the caption (L262.5).
questions: questions: 1. Is 4.1-4 pre-existing work? If so, please make this clear in the paper.
2. Recommendation to rework the list of contributions to focus on the new aspects of the proposed approach and their significance.
3. Recommendation to validate the claim that the method does not lose precision due to discretisation.
4. Recommendation to expand on the design decisions W2.1-4.
5. Recommendation to test objects with complex symmetries via the SYMSOL dataset; and test the ability to predict multiple hypotheses as claimed.
limitations: limitations: Yes, the authors adequately address the limitations and broader impact in the appendix, with an appropriate level of self-reflection and consideration.
paper_summary: paper_summary: The paper tackles 3D pose estimation from a single image. It builds on Image2Sphere which maps CNN features to the sphere which are then processed by spherical CNNs to produce a distribution over SO(3). The submission proposes using an MSE loss function in the spectral domain instead of cross-entropy in the spatial domain. It outperforms the baselines on pose estimation on ModelNet10-SO(3) and Pascal3D+.
strength_weakness: strength_weakness: W1) The paper seems to be a lightly modified version of Image2Sphere [1], but this is not properly acknowledged in the writing. Following Figure 2 and Section 4, the feature extraction, spherical mapper, Fourier transformers and pose predictor seems to be the same, the only difference being that in the submission, the last IFT is skipped and the loss is computed in the spectral domain during training (at test time it still needs the IFT so it is the same as Image2Sphere as far as I understand). Although Image2Sphere [1] is cited, the method section mostly describes the same things which implies they are new contributions. Please rewrite it focusing on the new contributions and being explicitly about what comes from prior work. 

W2) The main difference with respect to Image2Sphere is the loss being computed between Wigner-D coefficients instead of spatially. However how this is actually done is poorly explained. 

a) How does one L245 "convert the GT 3D rotations from Euler angles (...) to the Wigner-D matrices"? The text seems to imply that the Wigner-Ds are evaluated at the ground truth rotation, but then the loss in Eq (5) does not seem to make sense since the output of the model are the coefficients associated to each D matrix. I think what should be done is inverting a function on SO(3) that is an impulse on the ground truth pose but there is no mention of it in the paper.  

b) What is the "similarity between Wigner-D coefficients and an SO(3) grid" is L263? It seems that this somehow outputs a distribution on SO(3) that is queried spatially, so it seems like it should be an inverse SO(3) Fourier transform?

W3) Experiments on SYMSOL are missing. The dataset is designed to assess performance on symmetric objects, so methods that predict distributions over SO(3) are necessary; the baseline Image2Sphere handles it well. I believe the proposed method might perform poorly on it because of the way the ground truth is computed or the MSE loss, which are the differences with respect to Image2Sphere. I believe a comparison and discussion of this possible limitation is warranted. 

W4) The paragraph L113 "Rotation representation in frequency domain." is confusing. It talks generally of 3D rotations but actually describes the specific case of rotations of spherical functions. I think the U in eq (1) should be the same as the D in eq (2) but they are described as different things.

## References 
[1] Klee et al, "Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction", ICLR'23.
questions: questions: In L212, is looks like S is a function on the sphere defined in the spatial domain, but it is said to be C'xN where N is the number of spherical harmonics. Should N be the number of points on the sphere instead?

Typos:

L44: leverages -> leverage

L70: faciltate -> facilitate

L87: Foruier -> Fourier
limitations: limitations: Possible limitation on symmetric objects should be discussed (see W3).
paper_summary: paper_summary: This paper proposes a method for object orientation estimation from images.  The method predicts object orientation in the frequency domain of SO(3), by using SO(3) equivariant layers that operate on coefficients of Wigner-D matrices.  The network is trained with a MSE loss between the predicted Wigner-D coefficients and the Wigner-D coefficients associated with a ground truth rotation label.  The network is evaluated on two datasets, PASCAL3D+ and ModelNet10-SO(3), and outperforms all baselines in terms of rotation precision and accuracy.  Additionally, an ablation study is performed which demonstrates that the proposed output formulation is better than traditional rotation formulations, like quaternions, euler angles and rotation matrices, and that the method is robust to the sampling strategy used at inference time.
strength_weakness: strength_weakness: - The contribution is interesting, but minimal compared to existing work.  The neural network model and evaluation scheme used in this paper was introduced in [1].  However, this is not entirely clear the way the paper is written.  For instance, Sections 4.2, 4.3, 4.4, and 4.6 are all describing specific techniques from [1].  It would be easier to understand the contributions specific to this paper's network if these sections were moved to Background.  
- The second contribution of the paper, as stated, is confusing.  What is meant by the "intrinsic properties of SO(3) equivariant representations"?  Also, the claim "structurally guarantee 3D rotational equivariance" is not correct.  It is not possible to guarantee 3D rotational equivariance of the network when the input is an image.
- The core contribution of the paper is the use of a regression loss applied to learned Wigner-D coefficients.  However, this aspect of the method is not examined closely.  There is no information of how the values of $w_l$ are defined in Eqn. 5, nor any experiment justifying why MSE is preferred.  The contribution would be strengthened by a comparison of other sensible loss functions, an experiment that showed the sensitivity of the method to the values of $w_l$.  Additionally, it is not clear how the proposed loss is affected by object symmetries.  Could the authors explain why regression on Wigner-D coefficients does not suffer from inaccurate predictions when the orientation is ambiguous (e.g. tables have 180 degree symmetry in ModelNet10-SO(3))?
- The ablation experiment in Section 5.5 needs to be described in more detail.  How exactly was the method modified to produce euler, quaternion, etc. outputs?  Were SO(3) equivariant layers still used?
-  Some of the writing could be improved for clarity.  In Section 1, "Accurate determination of 3D orientation presents a more complex problem than translation due to the intricacies of rotation symmetries and the high dimensionality of pose space" is confusing: how is 3D orientation higher dimensional than 3D translation?  Also, "SO(3)-equivariance is crucial for accurate pose estimation ..." is not entirely true, as evidenced by the competitiveness of non-equivariant baselines in Tables 1&2.
- Figure 1, some of the images are taken from other sources without attribution (e.g. the healpix grid is from https://healpix.sourceforge.io



[1] Klee, David M., et al. "Image to sphere: Learning equivariant features for efficient pose prediction." arXiv preprint arXiv:2302.13926 (2023).
questions: questions: - Did you explore evaluating the model with a higher resolution grid or with gradient ascent?  Given that the regression loss can be precise, the model may be able to achieve even lower rotation error with more grid points.
- The regression loss is very effective at achieving high precision, but seems to lose the ability to model object symmetries or uncertainty (according to qualitative results in Figure A2).  Is there a way to combine the benefits of high precision and distribution modeling?  Other pose works have used a classification-regression framework [1], and it would be interesting to see if this idea could be applied here.  For instance, apply the classification loss over a HEALPix grid first, then apply the proposed regression loss if the prediction is close.
- In Appendix C.6, it seems like the frequency level is set for both the S2 and SO(3) convolutions in the network.  Did you try only modifying the frequency for the output of the network?  The issue with higher frequencies reducing performance may be due to the non-linearity (ReLU in spatial domain), which could be a bit unstable with high frequency signals.  Some other works have used gating functions for non-linearities on Wigner D coefficients [2].
- From the introduction, "in the context of spherical CNNs, ..., the 3D rotation parametrization in the spatial domain is inadequate because these SO(3) equivariant networks operate in the frequency domain."  What do the authors mean by inadequate?  Clearly, conversion from frequency to spatial domain is possible (the proposed method does so in its non-linearity layers).  

[1] Liao, Shuai, Efstratios Gavves, and Cees GM Snoek. "Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[2] Liao, Yi-Lun, and Tess Smidt. "Equiformer: Equivariant graph attention transformer for 3d atomistic graphs." arXiv preprint arXiv:2206.11990 (2022).
limitations: limitations: The limitations are discussed in Appendix F.   

What is meant by, "our method can suffer from significant errors due to the loss of spatial information when projecting 3D data onto spherical harmonics"?  What 3D data?  Where are these significant errors described in the experimental results or discussion?

The authors mention that, "computational cost associated with Wigner-D coefficients and SO(3) equivariant networks should be improved...";
this is a bit vague and contradicts Table A6, which shows the proposed method has the fastest frame rate during evaluation.

Comments: title: Response of the Official Comment by Reviewer HCET
Response of the Official Comment by Reviewer HnJh
Response of the Official Comment by Reviewer iMp8
Response of the Official Comment by Reviewer f5v8
Thanks for the rebuttal.
comment: ## Mapping from SO(3) to Wigner-D Coefficients:
Unfortunately it is still not clear to me how the map from Euler angles to Wigner-D coefficients happen. The appendices A and B mostly repeat textbook information that is not helpful:

- definition of spherical harmonics (6)
- relation of wigner D and d (7)
- rotating spherical harmonics with Wigner-D (8)
- rotation in 3D from Euler angles (A.2.1)
- expression for Wigner-D (9)
- different ways to represent rotation (B.1)
- rotation of spherical harmonics again (10) -- same as (8).
- relation of wigner D and d again (11) -- same as (7).
- expansion into spherical harmonics (12).
- rotation of spherical harmonics coefficients (13).
- decomposition into spherical harmonics (14).

Another suggestion is to clean up the repetitive writing. For example equations (1), (8) and (10) are the same and show the rotation of the spherical harmonics using Wigner-Ds which doesn't even seem relevant to the approach. Equations (2), (7) and (11) are also the same. 

Another guess at what might be happening is that the Wigner-Ds might be *evaluated* at the ground truth Euler angles to produce the supervision signal so the model is trained to predict a set of *values* of Wigner-D functions and not the Wigner-D coefficients that come out of a SO(3) FT. This would look a lot like "positional encoding" that maps coordinates to their evaluation in Fourier basis (sin/cos for the Euclidean spaces). But then I don't see how the output Wigner-D coefficients of the spherical CNN are mapped to the evaluation of the same coefficients at the ground truth. This would also contradict most of the text that refers to "Wigner-D coefficients" and not Wigner-D evaluations.

## Clarification on the Inference Procedure and Similarity Calculation:
I think computing the similarity against $O(B^{3})$ vectors of $O(B^{3})$ coefficients each is also $O(B^{6})$ even if all vectors are precomputed so doing a fast SOFT would still be faster?

## Conclusion
I think significant rewrite is needed, both to clarify technical details and also the contributions wrt Image2Sphere [see W1] so I do not recommend acceptance at this time.
Thank you for your thorough comment and insightful feedback.

**[Reply to W1]**

We will ensure that the main text is updated to reflect the clarifications provided in the rebuttal, as multiple reviewers have raised concerns about the clarity of the method section.


* * *
**[Reply to W2-a]**

We apologize for any lack of clarity in our original submission regarding the "mathematical transformation”. Specifically, we map the Euler angles, which represent an element of $ SO(3) $ , to a set of coefficients corresponding to the Wigner-D matrix elements. These coefficients are critical in capturing the rotational properties in the frequency domain.

**Mapping from SO(3) to Wigner-D Coefficients:**
As you correctly pointed out, each Wigner-D matrix element provides a fixed map from $ SO(3) $ to $ \mathbb{C} $. Our method involves predicting these Wigner-D coefficients directly, bypassing the complexities and potential pitfalls of spatial domain parametrizations.

To achieve this, we utilize an equivariant network to predict the Wigner-D matrix coefficients in the frequency domain directly from the input image features. The prediction process does not explicitly involve an inverse $ SO(3) $ Fourier transform of an impulse function centered on the ground truth rotation. Instead, our network learns to map the input features to the Wigner-D coefficients that represent the corresponding $ SO(3) $ rotation, leveraging the equivariant properties of the network to ensure rotational consistency.

The output of our network is a vector of Wigner-D coefficients that directly encode the rotation in the frequency domain. These coefficients are then converted back to a rotation matrix or another suitable representation (e.g., Euler angles) as needed for evaluation or further processing.

**Clarification of the GT Transformation:**
We would like to gently remind you that Appendix A (and B.2) contains the detailed conversion equations between the Euler angles and the Wigner-D matrix, including the small Wigner-d matrix. We will further revise the transformation process in our final manuscript, elaborating on how the conversion between Euler angles and Wigner-D matrices is handled in our approach.
 
* * *
**[Reply to W2-b]**

Thank you for your detailed feedback on the computational efficiency of our SO(3) grid generation and similarity computation for SO(3) distributions. We would like to clarify and justify our approach.

**Clarification on the Inference Procedure and Similarity Calculation**:
The similarity calculation described in L263 is designed to identify candidate Euler angles from the predicted Wigner-D coefficients by comparing them against a predefined SO(3) HEALPix grid at inference.

You are correct that our approach involves precomputing the inverse SO(3) Fourier transforms to generate the set of predefined Wigner-D coefficients. This approach indeed requires significant space complexity, with a high time complexity of $ O(B^6) $ naively to initialize the predefined SO(3) grid. Consequently, our inference method is actually slower and uses more  significantly more memory than just computing the IFT of the predicted coefficients.

However, after the predefined SO(3) grid is initialized, both the training and testing phases benefit from this precomputation, leading to reduced computation times during these phases. Additionally, for repeated runs, the grid can be stored and reloaded as needed, further optimizing execution time. 

We apologize for the inaccurate expression: "The SO(3) iFT method produces a single value, making distribution modeling difficult”. What we intended to convey was that the inverse SO(3) Fourier transform results in a specific mapping, not a single value.

We hope this explanation addresses your concerns and clarifies our design choices. Our approach aims to balance computational feasibility with the ability to accurately model complex pose distributions in SO(3) space.

* * *
**[Reply to W3]**

Thank you for acknowledging the addition of the SYMSOL experiments. As you noted, the proposed MSE loss and the previously used likelihood loss work complementarily. However, our model does not surpass the performance of I-PDF [48] on SYMSOL, and we agree that future research should focus on further improving symmetry modeling to address these challenges.
Thank you for increasing the score to WA and acknowledging that our rebuttal successfully addressed concerns W1, W2, and W4.

We will remove the claim that the proposed method is continuous and therefore better than discretized methods, as in L97, by following your suggestion. We recognize that while the method is continuous during training, it does undergo discretization during inference. Consequently, we will carefully rephrase this point to emphasize the continuity of our method during training.
I thank authors for the clarifications. I acknowledge the response.
Thank you for your detailed response, and in particular the results included in the PDF. My primary concerns---regarding (W1) the contribution overlap with [34], (W2) the design choice of the MSE, and (W4) the lack of evaluation on SYMSOL and its symmetric objects---have largely been addressed in the rebuttal.

Specifically, the authors have committed to moving the material from existing work into a background section, making it substantially clearer what contributions have been made. While the contribution is relatively small in my view, it is sufficient and the material in the rebuttal has strengthened it to an extent (Table R1 especially). The authors have also interrogated the choice of MSE quite convincingly, addressing its strengths and shortcomings, and ablating it in Table R4. Finally, and most importantly for my decision, the authors have evaluated on the SYMSOL dataset, showing that the method performs poorly for highly-symmetric objects, but interestingly can be combined with the loss from [34] to achieve greater performance than either, on this dataset.

A small note in relation to the rebuttal: I maintain that the claim that the proposed method is continuous and therefore better than the discretised methods (e.g., line 97) should be removed. I understand that it is continuous in the training loop, but it most certainly suffers from discretisation at inference time. This claim should be more carefully worded, especially now that the authors have good evidence regarding this effect (in Table R2).

Having carefully read the rebuttal and the responses from and to the other reviewers, I am inclined to increase my rating to a WA.
W1: Thank you, I really hope this is updated in the main text since multiple reviewers pointed out that the method section might be seen as claiming contributions from prior work. 

W2-a: I am interested precisely in the "mathematical transformation" mentioned in the rebuttal which does not seem to be described anywhere. Specifically, the Euler angles represent an element of SO(3), while each Wigner-D matrix element is a fixed map from SO(3) to $\mathbb{C}$. So I think the map is from an element of SO(3) to a set of coefficients corresponding to the Wigner-D matrix elements; one way to obtain this map is by the inverse SO(3) Fourier transform of the impulse function on SO(3) centered on the ground truth rotation, but this procedure is not described anywhere so I am not sure if that's what is being done or I am missing something.

W2-b: "The predefined SO(3) HEALPix grid includes a set of predefined Wigner-D coefficients corresponding 1:1 to Euler angles" -> sounds like this is exactly what the inverse SO(3) Fourier transform of function that is one at the cell corresponding to some Euler angles and zero elsewhere would give. So my understanding is that instead of computing the IFT of the predicted coefficients, the IFTs are precomputed for each discrete delta function of the grid, and the similarity between predicted coefficients and the precomputed is used as a distribution on SO(3). So I believe that for functions of bandwidth $B$ the approach stores $O(B^{6})$ coefficients (the grid has $O(B^{3})$ points and each delta function is represented by $O(B^{3})$ coefficients). The procedure to compute the distribution would also be $O(B^{6})$, which is the same as the naive algorithms for the SO(3) Fourier transform (SOFT). Since faster algorithms can reduce that to $O(B^{4})$ and even $O(B^{3}\log^{2}(B))$, I think the proposed method is actually slower and uses significantly more memory than just computing the IFT of the predicted coefficients. Please clarify. 

"The SO(3) iFT method produces a single value, making distribution modeling difficult." -> This seems incorrect; Fourier transforms and inverses are maps from function to function, not single values.

W3: Thank you for adding SYMSOL experiments. They show that the proposed MSE and the previously used likelihood losses are complementary and combining both may help -- I think this is a much more convincing result than the current ones in the submission. It should be noted however, that it still doesn't surpass IPDF results on SYMSOL and I believe there are follow-ups that outperform it.
First, thank you for taking the time to read and consider our rebuttal.

* * *
**Reply to R1 regarding pose visualization comparison:**

We would like to clarify that the visualization of pose distribution using the surface of the 2-sphere with the Mollweide projection [A] conveys the same information as visualizations that use bounding boxes or the $x$, $y$, $z$ axes. For example, the pose distributions shown on the spheres in Figures R2, A2, and A3 can be converted into single pose values that can be directly plotted onto the object, as demonstrated in Figure 6 of [B].

[A] Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold (Murphy et al., ICML 2021)  
[B] A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation (Yin et al., ICLR 2023)

However, we believe that the visualization method on the 2-sphere using the Mollweide projection [A], as used in Figures R2, A2, and A3, offers the additional benefit of enabling multi-hypothesis plotting in 3D pose space. This allows for deeper insights into symmetry and pose ambiguity modeling.

* * *
**Reply to Q1 regarding in-the-wild datasets (OOD scenarios):**

Evaluating out-of-distribution (OOD) performance is generally not the primary focus in the context of this 3D orientation estimation task. However, we have conducted OOD generalization experiments using our proposed method by training the model on different datasets, between ModelNet10 and PASCAL3D+. The results are presented below:

| Training Dataset | Evaluation Dataset | Acc@15  | Acc@30  | Rot. Err. |
|------------------|--------------------|---------|---------|-----------|
| ModelNet-SO(3)   | ModelNet-SO(3)      | 0.7590   | 0.7668   | 15.08°     |
| ModelNet-SO(3)   | PASCAL3D+           | 0.0004  | 0.0019  | 112.98°   |
| PASCAL3D+        | ModelNet-SO(3)      | 0.0015  | 0.0086  | 130.44°   |
| PASCAL3D+        | PASCAL3D+           | 0.7495  | 0.8965  | 8.92°     |

**Table:** Cross-dataset evaluation for validating out-of-distribution generalization on ModelNet10-SO(3) and PASCAL3D+ datasets.

As the results indicate, the model does not perform well when evaluated on out-of-distribution dataset. Nevertheless, we recognize this as an important area for future research, and we appreciate your suggestion to explore this further.
Thank you for updating your rating by reviewing our rebuttal and revision proposal of the background/related work sections, and thank you for your thoughtful feedback.

* * * 
**Takeaway from the plot in Table R3**: The plot demonstrates that the model consistently assigns greater weight to higher frequencies ($y$-axis and value), which correspond to more complex rotational components in the Wigner-D matrix. This suggests that the model effectively captures and prioritizes these complex aspects, regardless of the number of training views ($x$-axis).

Additionally, the plot further indicates that the weighted MSE loss plays a key role in enabling the model to focus on more complex rotations early stage in the training process, ensuring that these components are adequately emphasized even when data is limited (very low data regime).

* * * 
**Understanding the pitfalls of the MSE Loss in Table R4**: Yes, exploring whether the MSE loss could potentially push the model away from the ground truth rotation is indeed a valuable direction for future research. However, at this point, we have not observed or verified instances where the model moves away from the ground truth rotation due to the MSE loss.

* * * 
**The “sweet spot” in terms of the model precision in Tables R2 and R5**:
The "sweet spot" observed in Tables R2 and R5 is indeed an interesting phenomenon. However, we do not attribute this is due to the limited precision of the truncated Fourier series. Instead, it seems to be more related to the trade-off between computational cost and performance.

In Table R2, as the SO(3) grid precision increases, we observe that model accuracy improves and then saturates. This suggests that beyond a certain point, further increasing the precision (e.g. the number of points, $Q$) yields diminishing returns in accuracy relative to the additional computational cost. For example, we identified a good trade-off at $Q$=2.36M points (6th row), where further increasing to $Q$=18.87M points (7th row) improves accuracy at finer error thresholds (Acc@3° ) but also significantly increases space complexity.

Similarly, in Table R5, employing gradient ascent does enhance model accuracy and reduces error, particularly for more precise pose estimations. However, this also results in a significant increase in inference time. Thus, the observed "sweet spot" in precision is more likely due to the inherent trade-offs in computational resources and the model's ability to effectively utilize them, rather than limitations imposed by the truncated Fourier series.
Reply W1: I was rather asking to fit bounding boxes onto the object to compare with Ground-truth. Errors on sphere doesn't show exactly, how well the predicted pose is fitting on the object. 

Reply Q1: I meant the performance on dataset outside the training distribution. Does the model generalize to different dataset that the one it is trained on?
Thank you for the response.  I appreciate all the new results presented in the PDF, and the decision to revise the background/related work sections.

It is exciting to see the proposed MSE loss can be combined with the Cross-Entropy loss from I2S to achieve both high precision and accurate distributions.

Regarding R3, what exactly is the takeaway from this plot?  That the model puts more weight on higher-frequencies early on in training?  Is this something that could be addressed with a weighted MSE loss to boost performance in the very low data regime?

Regarding R4, it would be nice to understand the pitfalls of the MSE loss over Wigner-D coefficients.  For instance, could the gradient of the MSE loss push the model away from the ground truth rotation in some cases?  This doesn't need to be addressed here but it would be interesting to study in the future.

Regarding R5 and R2, it is interesting that there is a sweet spot in terms of how precise the model can be.  Do you think this is due to the limited precision of the truncated Fourier series?

Given the proposed revisions and the new results, I updated my rating.