Symmetry-Robust 3D Orientation Estimation
Authors: Christopher Scarvelis
David Benhaim
Paul Zhang
Venue: ICML 2025 poster (2025)
Keywords: 3d orientation
shape analysis
3d deep learning
geometric deep learning
Primary area: applications->computer_vision
TL;DR: We introduce a state-of-the-art method for 3D orientation estimation.

Abstract: Orientation estimation is a fundamental task in 3D shape analysis which consists of estimating a shape's orientation axes: its side-, up-, and front-axes. Using this data, one can rotate a shape into canonical orientation, where its orientation axes are aligned with the coordinate axes. Developing an orientation algorithm that reliably estimates complete orientations of general shapes remains an open problem. We introduce a two-stage orientation pipeline that achieves state of the art performance on up-axis estimation and further demonstrate its efficacy on full-orientation estimation, where one seeks all three orientation axes. Unlike previous work, we train and evaluate our method on all of Shapenet rather than a subset of classes. We motivate our engineering contributions by theory describing fundamental obstacles to orientation estimation for rotationally-symmetric shapes, and show how our method avoids these obstacles.

Commetry-Robusse 3D Orientation Estimation
Christopher Scarvelis'
David Benhaim Paul Zhang?
Abstract
A shape's orientation axes are not intrinsic geometric quan- tities: They are determined by humans based on physical and functional considerations. It is therefore difficult to construct simple geometric algorithms for orientation es- timation. However, given a large dataset of canonically oriented shapes, one may pose orientation estimation as a supervised learning problem. This task is challenging. and developing an orientation pipeline that reliably estimates complete orientations of general shapes remains open problem.
Orientation estimation is fundamental task in 3D shape analysis which consists of estimating a shape's orientation axes: its side-, up- and front- axes. Using this data, one can rotate a shape into canonical orientation, where its orientation axes are aligned with the coordinate axes. Developing orientation algorithm that reliably estimates complete orientations of general shapes remains an open problem. We introduce a two-stage orien- tation pipeline that achieves state of the art perfor- mance on up-axis estimation and further demon- strate its efficacy on full-orientation estimation, where one seeks all three orientation axes. Unlike previous work, we train and evaluate our method on all of Shapenet rather than subset of classes. We motivate our engineering contributions by the- ory describing fundamental obstacles to orienta- shapes,
The naive deep learning approach is to train a model with an L2 loss to directly predict shape's orientation from a point cloud surface samples. However, this strategy fails for shapes with rotational symmetries, where the optimal solution to the L2 regression problem is the Euclidean mean (Moakher, 2002) of a shape's 's orientations over all of its symmetries. In contrast, works such as Poursaeed et al. (2020) discretize the unit sphere into a set of fixed rotations and train a classifier to predict a probability distribution over these rotations, but find that this approach fails for any sufficiently dense discretization of the unit sphere.
tion estimation for and show how our method avoids these obstacles.
1. Introduction
Our key insight is to divide orientation estimation into two tractable sub- trtb-problms.. In the first stage (the quotient orien- ter), we solve continuous regression problem to recover a shape's 's orientation up to octahedral symmetries. In the second stage (the flipper. we solve a discrete classification problem to predict one of 24 octahedral flips that returns the first-stage output to canonical orientation. Octahedral symmetries form small set covering substantial pro- portion of the symmetries occurring in real-world shapes. Consequently, quotienting our first-stage regression prob- lem by octahedral symmetries prevents its predictions from collapsing to averages, while also keeping the subsequent classification problem tractable.
Orientation estimation is fundamental task in 3D shape analysis which consists of estimating shape's orientation axes: its side-, up- and front-axes. Using this data, one can rotate a shape into canonical orientation, in which the shape's orientation axes are aligned with the coordinate axes. This task is especially important as a pre-processing step 3D deep learning, where deep networks are typically trained on datasets of canonically oriented shapes but ap- plied to arbitrarily-orienn data augmentation or equivariant and invariant architectures may improve a model's robustness to input rotations, these techniques come at the cost of data efficiency and model expressivity (Kuchnik & Smith, 2019; Kim et al., 2023). In contrast, orientation estimation allows one to pre-process shapes at inference time so that their orientation matches model's training data.
shapes inference time. While
Using this strategy, our method accurately estimates a shape's orientation up one of its symmetries, which suf- fices for returning the shape to an upright and front-facing pose. As we are unaware of prior work on this problem, we also benchmark our pipeline's 's performance on the well- studied task of up-axis prediction, in which one seeks to return a shape to an upright (but not necessarily front-facing) pose, and find that our work achieves sfate-of-the-art per- formance on this task. Unlike previous work, we train and
MIIT CSAIL, Cambridge, MA BBackflip AI, San Francisco, CA. Correspondence to: Christopher Scarvelis <searv mit.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Symmetry-Robun 3D Orientation Estimation
Another set of elassical methods observe that many man- made objects are designed to stand on flat surfaces, their up axis is normal to supporting base. Motivated by this observation, these methods attempt to identify a shape's 's sup- porting base rather than directly infer their up axis. Fu et al. (2008) generate a set of candidate bases, extract geometric features, and combine a random forest and SVM to predict a natural base from the candidates. Lin & Tai (2012) simplify shape's convex hull, cluster the resulting facets obtain set of candidate bases, and compute hand-designed score to select the best base. Both of these methods rely heavily on feature engineering and fail on shapes that do not have natural supporting bases.
(a) Rotated shape
(b) Shape in canonical ori- entation
Figure I: Orientation estimation allows users rotate ar- bitrary shapes (a) into canonical orientation (b), in which the shape's s orientation axes are aligned with the coordinate axes.
evaluate our model on the entire Shapenet dataset rather than a subset of classes. We further demonstrate its generaliza- tion capabilities on Objaverse, a large dataset of real-world 3D models of varying quality.
Deep learning- based methods. Motivated by the limita- tions of classical approaches, several works use deep learn- ing for orientation estimation. Liu et al. (2016) train two neural networks on voxel representations of 3D shapes. A first-stage network assigns each shape to one of C classes. Based this prediction, the shape is routed to one of C second-stage networks that are independently trained to pre- dict the up axis from voxel representations of shapes in their respective classes, This method is unable to handle shapes that lie outside the C' classes on which the networks were trained.
shape's ground truth orientation may be ambiguous. This challenge is especially salient for nearly-symmetric shapes, where multiple orientations may yield nearly indistinguish- able shapes. To resolve this issue, we use conformal pre- diction to enable our flipper to output adaptive prediction sets (Romano et al., 2020) whose size varies with the flipper model's 's uncertainty. For applications with human in the loop, this enables the end user to choose from a small set of plausible candidate orientations, dramatically simplifying the orientation estimation task while preserving user control over the outputs.
Pang et al. (2022) draw inspiration from classical methods and train a segmentation network to predict points that be- long to a shape's supporting base. They fit plane to the predicted base points and output normal vector to this plane as the predicted up axis. This method represents the current state of the art for orientation estimation, but strug- gles to handle shapes without well-defined natural bases and and only predicts a shape's up axis. In contrast, our method succeeds on general shapes and predicts full rotation ma- trix that returns shape to canonical orientation.
Our contributions include the following: (1) we identify fun- damental obstacles to orientation estimation and study the conditions under which naive regression- based approach to orientation estimation fails; (2) we propose a two-stage orientation estimation pipeline that sidesteps these obsta- cles; (3) we train and test our model on Shapenet and show that it achieves SOTA performance for up-axis prediction and orientation estimation; (4) we use conformal prediction to enable end users to resolve ambiguities in a shape's ' orien- tation; (5) we release our code and model weights to share our work with the ML community.
Chen et al. (2021) use reinforcement learning to train a model to gradually rotate a shape into upright orientation. While this algorithm performs well, it is evaluated on few classes and is costly to train. Kim et al. (2020) adopt a similar perspective to Fu et al. (2008), but use ConvNets to extract features for random forest classifier that pre- dicts natural base. Poursaeed et al. (2020) use orientation estimation as pretext task to learn features for shape clas- sification and keypoint prediction. They also investigate a pure classification- based approach to orientation estimation that discretizes the 3D rotation group into K rotations and predicts distribution over these rotations for an arbitrarily- rotated input shape. They find that its performance decays rapidly as K increases, reaching an accuracy as low as 1.6% for K = 100 rotations.
2. Related Work
Classical methods. A simple method for orientation es- timation is to compute a rotation that aligns a shape's 's prin- cipal axes with the coordinate axes; Kaye & Ivrissimtzis (2015) propose robust variant of this method for mesh alignment. However, Kazhdan et al. (2003) find that PCA- based orientation estimation is not robust to asymmetries. Jin et al. (2012); Wang et al. (2014) propose unsupervised methods that leverage low-rank priors on axis-aligned 2D projections and third-order tensors, respectively, constructed from input shapes. These priors are restrictive, and the re- sulting orientation pipelines also fail on asymmetric shapes.
We also highlight a related literature on canonical align- ment. This literature includes works such as Kim et al. (2023); Sajnani al. (2022); Spezialetti et al. (2020); Zhou
2

---

Symmetry-Robust 3D Orientation Estimation
al. (2022), which seek to map arbitrarily-rotated shapes to class-consistent pose, as well Katzir et al. (2022); Sun et al. (2021), which seek to learn pose-invariant represen- tations of 3D shapes. These works only attempt to learn a consistent orientation within each class, but this orientation is not consistent across classes and is not generally aligned with the coordinate axes. In contrast, we tackle the more challenging task of inferring a canonical orientation that is consistent across all objects.
orientation, then its side-, up- and front-axes (which we will jointly refer to as its orientation axes) are aligned with the {z, y; =} coordinate axes, respectively. We depict canonically oriented shape S along with its orientation Ωg in Figure 2.
More broadly, geometric deep learning studies general meth- ods for exploiting symmetries in data by designing machine learning models that are invariant or equivariant to certain transformations. For instance, Kaba et al. (2023) proposes using learned canonicalization functions to obtain equivari- ant machine learning methods, and Puny al. (2022) pro- poses frame averaging to induce invariance or equivariance in deep learning architectures. Our work may be viewed as an efficient canonicalization method for the specific case of 3D shapes with rotational symmetries.
Figure 2: A shape's orientation Qg a rotation matrix whose columns are the shape's side-, up-, and front-axes (plotted in yellow, magenta, eyan, resp).
A shape's ' orientation axes are not intrinsic geometric quanti- ties. Instead, they are determined by humans based on phys- ical and functional considerations. For instance, a shape's ' up-axis may be determined by how a shape naturally lies under gravity, whereas its front-axis may be determined by the direction from which a human typically interacts with it. It is therefore challenging to devise simple geometric algorithms for orientation estimation.
3. Method
In this section, we motivate and describe our orientation pipeline. We first identify fundamental obstacles to orienta- tion estimation and show that learning shape's 's orientation with the Le loss fails when the shape is rotationally sym- metric. Motivated by these observations, we introduce our two-stage orientation pipeline consisting of a quotient ori- enter followed by a flpper. Our quotient orienter model solves a regression problem to recover shape's 's orientation up to octahedral symmetries, which commonly occur in real-world shapes. The flipper then predicts one of 24 oc- tahedral flips that returns the first-stage output to canonical orientation. We finally use conformal prediction to enable our flipper to output prediction sets whose size varies with the model's uncertainty. This allows end users to resolve ambiguities in shape's orientation by choosing from a small set of plausible candidate orientations.
However, there exist large datasets of shapes such as Shapenet (Chang et al., 2015) that have been assigned a canonical orientation by human annotators. Given train- ing set DD of shapes S E S paired with their ground truth orientations Pgg (where Ug = / for datasets of canonically oriented shapes), a natural strategy for orientation estima- tion is to parametrize the orienter as a neural network fe and solve the following supervised learning problem:
min E fo R-U(SO(3) (S.D))ED
[l]) - -R9s 11:]
(I)
where U(SO(3)) is the uniform distribution over SO(3).
Many real-world shapes possess at least one non-trivial rotational symmetry R. This rotation leaves the shape un- changed, RS = S. However, since shape's 's orientation is rotation-eqquivariant, Uns RΩg * Ωs, and the same shape S will necessarily be associated with two distinct orientations Us and RΩg. It follows that the ground truth orienter map S + Ωgg of a rotationally-symmed is one-to-many and therefore not a function. We formally state and prove this result Proposition 1, and depict an instance of this phenomenon in Figure 3.
3.1. Orientation estimation under rotational symmetries
In this section, we introduce the orientation estimation problem and motivate our approach. Throughout these preliminaries, we consider 3D shapes S E S lying in some space of arbitrary shape representations S. Orien- tation estimation consists of learning an orienter function f: + S3(3) that maps a shape S E S to a predicted orientation 2g E SO(3), where 50(3) denotes the 3D rotation group. An orientation is rotation matrix Qg as- sociated with shape 5 that is rotation-equivariint:: If one rotates S by R SO(3) to obtain R.S then Qss Rlls. We interpret the columns of Ug (w3, wi, wg)) as the side- up- and front-axes of S, respectively, and say that S is in canonical orientation if Ωg = I. If S is in canonical
shape
On the other hand, fo is function, which implies that the solution to Equation 1 cannot be the true orienter map. The following proposition shows that for single shapes with rotational symmetries, the solution to Equation 1 is instead Euclidean mean of the rotated orientations RQΩ2 across all rotations Q in the symmetry group Reg (Moakher, 2002).
3

---

Symmetry-Robust 3D Orientation Estimation
are common in practice. We present a partial solution this problem. Suppose know finite group R. 2 R. that contains shape S's rotational symmetries. We can then quotient the L2 loss by R to obtain the following problem:
(a) Shape in canonical orienta- tion
(b) Shape after 180 rotation about the y-axis
min E J:S (:S-80(3) R-U(S0(3) (S,0g)ED
min Ilf(RS) RQQs (3)
LQER
Figure 3: Rotating a shape by one of its symmetries changes its orientation while leaving the shape unchanged. Here, the front axis (in eyan) and side axis (in yellow) are flipped when the shape is rotated 180% about the y-axis.
This loss is small if f(RS) is close the orientation RQΩs of the rotated shape RQS for any E R; Mehr et al. (2018) use similar techniques learn latent shape representations that are invariant under group of geometric transforma- tions. Intuitively, whereas Equation 2 attempts to make f f(S) close to all QΩt, a minimizer of Equation 3 merely needs to make f(S) close to any Q9gg Formally:
Proposition 3.1. Let S E S be fixed shape which is symmetric under non-trivial group of rotations Rs S SO(3). Let Sg be the shape's orientation, and suppose solves the following regression problem:
Proposition 3.2. Let S S S be a fixed shape which is sym- metric under group of rotations Rg S CR SO(3). Let Ug be the shape's is orientation, and consider the following quotient regression problem:
min f-S (-8-) (-U(()(())
E
[I((RS)) R
Then (RS) = proj s0(3)
1 |Rss EQERS RQUis +
RΩg, where projso(3) denotes the orthogonal projection onto SO(3).
min f:8-80(3) R~U(50(3)
min []((S) - RQQS (4) LQER
We prove this proposition in Appendix A.2. This problem may be highly degenerate, even for shapes with single non- trivial symmetry. For example, consider the bench shape S depicted in Figures 1, 2, 3. As shown in Figure 3, this shape has two rotational symmetries: The identity rotation, and a 180% rotation about the y-axis. is straightforward to show that the solution f* to Equation 2 non-unique and may be any rotation about the y-axis, which we illustrate Figure 4. (See Appendix A.3 for further details.) This demonstrates that even a single non-trivial rotational symmetry leads to entire submanifold of solutions f (S) to Problem 2, which substantiates the observations of Liu et al. (2016); Poursaeed et al. (2020) that orientation estimation via L2 regression typically performs poorly practice.
Then for any SO(3), there exists a solution f S SO(3) of the form f (RS) RQ Us for some Q" R.
We prove this proposition in Appendix A.4. In contrast to naïve L2 regression, quotient regression learns a function that correctly orients anationnlly-symmmett shapes up to rotation in the group R. While this is only a partial solution to the orientation estimation problem, the remainder reduces to a discrete classification problem: Predicting the rotation Q' E R such that f (*(S) RQ*Qs. In the following section, will show how solution this problem allows one map RS to the canonically oriented shape S.
3.3. Recovering orientation via classification.
By solving the quotient regression problem in Equation 3, one can recover arbitrarily shape R.S's orientation to rotation Q" E R. In this section, we propose training a classifier to predict this rotation Q" given the solution f (RS) = RQ Ωg to the quotient regression problem. We now further assume that the shape S's ground truth orientation G2g the canonical orientation f2g 1. We show that even if S is symmetric under some group of symmetries Rs C R, the optimal classifier's predictions enable one to map RSS to the canonically oriented shape S.
K
Figure 4: The solution (S) to Problem 2 evaluated at the bench shape S may be any rotation about the y-axis.
3.2. A partial solution.
Predicting a rotation Q" E R from the output f (RRS) = RQ" the quotient regression model is an IR -class classi- fication problem. While one may hope that composing the
The previous section shows that orientation estimation via La regression fails for onationally-symmetri shapes, which

---

Symmetry-Robust 3D Orientation Estimation
3.4. Implementation
Informed by our insights from Sections 3.2 and 3.3, we now present state-of-the-ard method for orientation es- timation. Our pipeline consists of two components. Our first component, which we call the quotient orienter, is neural network trained to solve Problem 3. We quotient the Le objective by R == O S SO(3). the octahedral group containing the 24 rotational symmetries of a cube. This is among the largest finite subgroups of (O(3) (only the cyclic group C. for contain more subgroups), and it includes many rotational symmetries that commonly occur in real-world shapes.
(a) Quotient regression output
(b) Classifier outputs
Figure 5: The quotient regression problem 4 correctly ori- ents an arbitrarily rotated shape RS up to a rotation in R. The classification problem 5 then recovers the orientation of RS up to one of its rotational symmetries, which suffices for mapping RSS to the canonically oriented shape S.
48 and dihedral group D, for
can
Our second component, which we call the flipper, is clas- sifier trained to predict the rotation Q' E R from the output I (RS) = RQ* of the quotient regression model. We lustrate the output of each stage of this pipeline in Figure 5. As many shapes possess multiple plausible orientations, we use conformal prediction to enable our flipper to output adaptive prediction sets whose size varies with the flipper model's uncertainty. We provide further implementation details below.
quotient regression model and the optimal classifier yields function that outputs correct orientations siz R regardless of its inputs' symmetries, such function cannot exist for symmetric shapes. However, the following result shows that this pipeline recovers the orientation of a shape RS up to one of its rotational symmetries, which suffices for mapping RS to S.
Proposition 3.3. Let S E S be fixed shape which is symmetric under group of rotations Rg S R C SO(3), and suppose S is canonically oriented, so Ωg = T. Let f : S + SO(3) be a solution to Equation 3, so that (RS) = RQ" for some Q' E R. Finally, suppose that solves the following problem:
Quotient orienter. We parametrize our quotient orienter by DGCNN (Wang et al., 2019) operating on point clouds. To ensure that our predicted orientations lie in SO(3), we follow Brégier (2021) and map model outputs from 3x3x3 to SO(3) by solving the special orthogonal Procrustes prob- lem. We train the quotient orienter on point clouds sampled from the surfaces of meshes in Shapenet (Chang al., 2015). As these meshes are pre-aligned to lie in canonical orienta- tion, we fix 22g = / for all training shapes S. We provide full architecture and training details in Appendix B.
p² S + A (|
min -- (R)
CCE (p(QS))
(5)
In our experiments, we observe that our quotient orienter yields accurate predictions for most input rotations R but fails for small subset of rotations. To handle this, fol- low Liu al. (2016) and employ test-time augmentation to improve our model's predictions. This consists of (1) randomly rotating the inputs RS by K random rotations Rg ~ U(SO(3)), h: = 1,..., K, (2) obtaining the quotient orienter's predictions (R, RS) for each shape, (3) return- ing these predictions the original input's orientation by computing R fo R2 RS), and (4) outputting the prediction RT- fo (Ra- RRS) with the smallest average quotient distance (o to the remaining predictions.
where (R) denotes the uniform distribution on R, CE(-) denotes the cross-entropy loss, and &Q E A 1 is a one- hot vector centered at the index Q E R. Then for any R E SO(3), p² (RS) RS) is the uniform distribution over F F Rg). For any (Q*)T F in the support of this distribution, FT)
[RS) RS S,
second-stage prediction first-stage prediction
so using f and p", one may recover S from the arbitrarily- rotated shape RS.
We prove this proposition in Appendix A.5. The results in this section show that unlike naive L2 regression, a two- stage pipeline consisting of quotient regression followed by discrete classification can successfully recover shape's 's orientation up to its symmetries, provided one quotients the L2 objective by a sufficiently large subgroup of 50(3). We combine these results in the following section to implement a state-of-the-ar arr method for orientation estimation.
We also parametrize our flipper by DGCNN op- erating on point clouds. We train the flipper point clouds sampled from the surface of Shapenet meshes by solving the RRI-class classification problem described in Section 3.3. We draw rotations Q ~ UOO) during training, and simulate inaccuracies in our quotient orienter's predictions by further rotating the training shapes about randomly drawn axis by
Flipper.
5

---

Symmetry-Robust 3D Orientation Estimation
an angle uniformly drawn from [0, 10] degrees. We provide full architecture and training details in Appendix B.
Angutar errer up-asls prediction, Shopanet 100kl
Aspular errer of sp-cc prodictions Modelhee00
We also employ test-time augmentation (TTA) to improve our flipper model's 's predictions. Similarly to the case with the quotient orienter, we (1) randomly flip the inputs by K
Oen with
Oal
TTA
with
10 b0uleres
10 deganes
random rotations Rg ~ R = O, (2) obtain the flipper's predictions for each shape, (3) return these predictions to
(a) Angular error histograms for Shapenet
(b) Angular error histograms for ModelNet40
the original input's orientation, and (4) output the plurality prediction.
Figure 6: Comparison of angular errors between the esti- mated and ground truth up-axis on the Shapenet validation set (left) and on ModelNer40 (right). We plot the empirical CDF of the angular errors of each model's outputs. The dashed lines indicate the 10 error threshold beyond which prediction is treated as incorrect. With test-time augmen- tation (TTA), our algorithm's 's error rate is 64.6% lower than the prior state of the art.
Adaptive prediction sets. Many real-world shapes have several plausible canonical orientations, even when they lack rotational symmetries. Furthermore, the flipper model may map nearly-symmetric shapes with unique canonical orien- tations to a uniform distribution over their near-symmnt mymmtresssss due to factors such as insufficiently dense point clouds or the smoothness of the flipper function.
To mitigate this issue in pipelines with a human in the loop, we enable our flipper model to output adaptive prediction sets whose size varies with the flipper's ' uncertainty (Ro- mano et al., 2020). This method uses a small conformal cal- ibration set drawn from the validation data to learn a thresh- old parameter T > 0 associated with a coverage probability (0, 1) that controls the size of the prediction sets. Given the flipper model's 's output probabilities (5) E A 1 for some shape S, one sorts (S) in descending order and adds elements of R to the prediction set until their total mass in (S)) reaches r. Intuitively, these sets will be small when the flipper is confident in its prediction and assigns large classes. Conversely, the
4.1. Upright orientation estimation
We construct a random 90-10 train-test split of Shapenet, draw 10k point samples from the surface of each mesh, and train our quotient orienter and flipper on all classes in the training split. We train our quotient orienter for 1919 epochs and our flipper for 3719 epochs, sampling 2k points per point cloud at each iteration and fixing learning rate of 10-4, We also train Upright-Net with 2048 points per cloud on the same data for 969 epochs at the same learning rate, at which point the validation accuracy has plateaued. We follow the annotation procedure in Pang et al. (2022) to obtain ground truth segmentations of each point cloud into supporting base points and non-base points.
mass to the gighest-probabile sets will be large when the flipper is uncertain and assigns similar mass to most classes.
Table I: Up-axis estimation accuracy for our pipeline trained on Shapenet
4. Experiments
We now evaluate our method's performance on orientation estimation. We first follow the evaluation procedure in Pang et al. (2022) and benchmark against their "Upright-Net," which represents the current state of the art for orientation estimation. Upright-Net can only map shapes into upright orientation, where a shape's up-axis is aligned with the y-axis; in contrast, our method recovers a full orientation Os for each shape. We therefore follow this benchmark with an evaluation of our method on the more challenging task of full-orientation estimation. We incorporate adaptive prediction sets this stage and demonstrate that our method reliably provides a plausible set of candidate orientations for diverse shapes unseen during training. We train and evaluate all models on Shapenet (Chang et al., 2015), as this is the largest and most diverse dataset we are aware of containing canonically oriented shapes. However, we report qualitative results for our method's ' out-off-dstrrbuu performance on Objaverse in Appendix C.
Method
Accuracy (t)
Shapenet
ModelNet40
Ours (with TTA)
89.2 %
77.7 %
Ours (without TTA)
85.3 %
72.3 %
Upright-Net (Pang et al., 2022)
69.5 %
62.3 %
We then follow the evaluation procedure in Pang et al. (2022) to benchmark our method against their SOTA method for upright orientation estimation. We randomly rotate shapes S in the validation set, use our two-stage pipeline and Upright-Net to estimate the up-axis w/s of each ran- domly rotated shape R.S, and then measure the angular er- ror arccos (aooss((")sss)s0ss hs whs)) between the estimated and ground truth up-axis. Our method's estimated up-axis is the second column of our estimated orientation matrix Ωss.
In contrast, Upright-Net predicts a set of base points for R.S. fits plane to these points, and returns this plane's normal
6

---

Symmetry-Robust 3D Orientation Estimation
proves over the state of the art in up-axis estimation. In the following section, show that our method also success- fully recovers the full orientation liR of rotated shape up to its symmetries, more challenging task than upright orientation estimation. Using our estimated orientations, we return a wide variety of shapes into canonical orientation.
4.2. Full-orientation estimation
We now evaluate our method's performance on full- orientation estimation, in which we use our model's full predicted orientation matrix l2Rs to transform an arbitrarily- rotated shape RS to the canonically oriented shape S. We draw shapes S from the Shapenet validation set and ran- domly rotate them to obtain inputs RS to our pipeline. Be- cause several orientation matrices may be associated with a single canonically oriented shape when this shape has rotational symmetries, we report the symmetric chamfer dis- lance between our pipeline's s output shape and the ground truth shape. Because this is a shape-to-shapp metrie, is insensitive rotational symmetries: If RS S, then d T) d(RS, T), where d is the chamfer distance and T is reference shape.
(a) Ours
(b) Upright-Net
Figure 7: Comparison of oriented shapes recovered from randomly rotated inputs using our algorithm (left) and Upright-Net (right). Failures are rendered in red. Our algo- rithm recovers correct upright and front-facing orientations for most shapes, whereas Upright-Net cannot recover front- facing orientations and fails over 2.8x as often at up-axis prediction.
vector pointing towards the shape's center of mass. This method relies on a restrictive prior on the geometry of the input shapes that does not hold for shapes which do not naturally lie on a supporting base. We follow Pang et al. (2022) and define our methods' respective accuracies to be the proportion of validation meshes whose angular error is less than 10
To our knowledge, our algorithm is the first to solve this task for generic shapes without requiring class information at training time or at inference time. We therefore construct two baselines using Upright-Net. As this algorithm only predicts shape's up-axis (the second column of its orien- tation matrix), we must augment Upright-Net's predictions with front-axis and a side-axis. We generate a baseline called "Upright-Net-Orace predicted up-axis with the shape's ground truth frost--xiis, and a second bascline called "Upright-Ner Neee--eee--Rnoon by aug- menting the predicted up-axis with a random vector that we orthogonalize with respect the up-axis. In both cases, we obtain the side-axis by taking the cross product of the respective up- and front-axes. "Upright-Net-Orace Oraclee upper- bounds the performance of a hypothetical orienter built on Upright-Net, whereas BUpright-Nee-Randooe provides a lower bound on such an orienter's performance.
We depict the results of this benchmark in Table 1. Our method improves on Upright-Net's up-axis estimation accuracy by nearly 20 percentage points, corresponding to a 64.6% reduction in the error rate relative to the previous state of the art. To provide a more comprehensive picture of our respective models' performance, we also report the empirical CDF of angular errors for our model and Upright- Net in the left panel of Figure 6. Our model primarily fails by outputting orientations that are 902 or 180 away from the correct orientation, which correspond to failures of the flipper. In contrast, Ueri failures are more evenly distributed across angular errors. Finally, we depict a grid of non-cherry -ibeerr-picked outputs of our model and Upright-Net in Figure 7 and highlight failure cases in red.
by augmenting Upright-Net's
We benchmark our pipeline's performance against each base- line in Table 2, where report the mean chamfer distance between each model's predicted shapes and the correspond- ing ground truth shapes, which are in canonical orientation. Our method achieves an 84% reduction in mean chamfer distance between predicted and ground truth shapes rela- tive to the oracle baseline, which augments Upright-Net's predicted up-axis with the shape's 's ground truth front-axis. In practice, one does not have access this front-axis at inference time, so this represents a loose upper bound on the potential performance of shape orientation algorithm built upon Upright-Net's predicted up-axes.
We quantitatively evaluate our model's generalization by performing the same experiment on ModelNet40 (Wu et al., 2015). Both models' performances deteriorate in this set- ting, but our algorithm continues to substantially outperform Uer. Furthermore, the right panel of Figure 6 shows that our model's failures on ModelNet40 are more heavily weighted towards flipper failures (where the angular error is close to 90" and 180%). In the following section, we will show how a human in the loop can resolve these failures by choosing from a small set of candidate flips, which substan- tially improves our pipeline's 's quantitative performance.
These results demonstrate that our method significantly im-
This empirically validates our Proposition 3.2 and demon-
7

---

Symmetry-Robust 3D Orientation Estimation
strates that our pipeline successfully recovers canonically oriented shapes, even when symmetries preclude the orienter map from being represented by function. We further illus- trate our pipeline's 's performance in Figure 10 in Appendix C, where we depict a large grid of shapes that have been oriented by our pipeline. Even when our model's outputs disagree with the ground truth pose in Shapenet (we render the corresponding meshes in red), the recovered shape is often acceptable in practice.
biguities by having our orientation pipeline output adaptive prediction sets (APSs), whose size varies with the flipper's s uncertainty. We now incorporate this technique into our evaluation. We learn a threshold parameter T corresponding to a coverage probability of a = 0.3, and at inference time oulput a set of flips in descending order of probability under the flipper's s output distribution until their total probability mass reaches т. Intuitively, this set will be small when the flipper is confident in its prediction and large otherwise.
Finally, in Figure 12 in Appendix C, we depict transformed shapes obtained by applying our method to randomly- rotated shapes from the Objaverse dataset (Deitke et al., 2023). This dataset contains highly diverse meshes of vary- ing quality and therefore serves as a useful test case for our method's s performance on out-of-distribution shapes. (As these shapes are not canonically oriented, we cannot train on them or report meaningful error metrics.) Using our orientation pipeline, one reliably recovers shapes that are canonically oriented up to an octahedral flip. Our flipper has greater difficulty handling out-of oft-of--istribute distriiiibtt meshes, but predicts an acceptable flip in many cases. We expect that training our flipper on larger dataset of oriented shapes will further improve its generalization performance.
By applying each flip to the first-stage orienter's output, we obtain a shape orientation pipeline that outputs sets of candidate shapes S, given a single arbitrarily- input RS. To evaluate its performance, we compute the chamfer distance between each candidate shape and the ground truth shape, and take the minimum over these chamfer distances. As one may trivially reduce the minimum chamfer distance over an APS by outputting arbitrarily large sets, we report histogram of APS sizes in Figure 8 to demonstrate that these sets are typically small.
We report our pipeline's 's performance with adaptive predic- tion sets in the last row of Table 2. Allowing our pipeline to output an APS reduces the mean chamfer distance be- tween the closest shape in this set to the ground truth by factor of 4 relative to Orient Anything without this feature. Figure 8 shows that with our chosen coverage probability, most APSs are small; the median APS size is 2, and 90% of APSs have at most 8 shapes. This demonstrates that our flipper typically assigns most of its probability mass to a few octahedral symmetries, which often contain a rotation that returns the shape to its canonical pose in Shapenet. In Figure 11 in Appendix C, we also include a grid of meshes in the prediction sets associated with 20 randomly- validation shapes. These prediction sets typically consist of a small collection of plausible shape orientations; by man- ually inspecting them, human in the loop may select the orientation that is most suitable for their target application.
Table 2:
Full-orientation estimation performance on the Shapenet validation set.
Method
Mean chamfer distance (1)
Upright-Net-Raute Random
0. = 0.13824
Upright-Net-Oracle Ora-le
0.05481 = 0.13016
Ours (TTA)
0.00856 + 0.0396
Ours (w/o TTA)
0.01107 = 0.04342
Ours (with APS)
0.00208= = 0.01407
Histogram of prediction set sizes
2000
Median 90th percentile
1750
5. Conclusion
1500
Frequency
1250
This work introduces state-of-the-arr method for 3D ori- entation estimation. Whereas previous approaches can only infer upright orientations for limited classes of shapes, our method successfully recovers entire orientations for general shapes. We show that naive regression based approaches for orientation estimation degenerate on caatiionally-yymmettte shapes, which are common in practice, and develop two- stage orientation pipeline that avoids these obstacles. Our pipeline first orients an arbitrarily rotated input shape up to an octahedral symmetry, and then predicts the octahedral symmetry that maps the first-stage output to the canoni- cally oriented shape. We anticipate that this factorization of geometric learning problems will be broadly applicable throughout 3D deep learning for tackling problems that are ill posed due to the presence of symmetries.
1000
750
250
12
16
20
24
Prediction
Figure 8: Histogram of APS sizes with coverage probability a = 0.3. The median APS contains 2 shapes, and 90% of APSS contain at most shapes.
Adaptive prediction sets. Many real-world shapes have several plausible canonical orientations, even if they lack rotational symmetries. In particular, while most real-worid shapes have a well-defined upright orientation, their front- facing orientation is often ambiguous. In Section 3.4, we proposed enabling human in the loop to resolve these am-
8

---

Symmetry-Robust 3D Orientation Estimation
Acknowledgments
Jin, Y., Wu, Q. and Liu, L. Unsupervised upright orientation of man-made models. Graphical Models, 74(4):99-108, 2012.
We thank the authors of Pang et al. (2022) for their gracious assistance with their source code and Justin Solomon for valuable discussions. Chris thanks Backflip AI for hosting him and generously supporting him with mentorship and compute throughout his internship, and gratefully acknowl- edges the support of a 2024 Exponent fellowship.
Kaba, S.-O., Mondal, A. K., Zhang, Y., Bengio, Y., and Ravanbakhsh, S. Equivariance with learned canonicaliza- tion functions. In International Conference on Machine Learning, pp. 15546-15566. PMLR, 2023.
Impact statement
Katzir, O., Lischinski, D., and Cohen-Or, D. Shape- pose disentanglement using se(3)-equivariant vector neu- rons. In Computer Vision ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III, pp. 468 484, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 978- 378-3-031-20061-1 doi: 10. 10.10077998.33001--0002--- 27. URL https: //o.///oiii/////////////..
This paper presents novel algorithm for 3D shape ori- entation. We do not believe that there are notable ethical considerations or societal impacts of our work meriting specific consideration.
References
Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009. /41993,,9.1199,, 2020.
Kaye, D. and Ivrissimtzis, I. Mesh alignment using grid based pca. In International Conference on Computer Graphics Theory and Applications, volume 2, pp. 174- 181. SCITEPRESS, 2015.
Brégier, R. Deep regression on manifolds: a 3d rotation case study. 2021 International Conference on 3D Vision (3DV), pp. 166-174. IEEE, 2021.
Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. Rota- tion invariant spherical harmonic representation of 3 shape descriptors. In Symposium on geometry processing. volume 6, pp. 156 164, 2003.
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang. Q. Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J. Yi, L., and Yu, F. ShapeNet: An Information-Ree Rich 3D Model Repository. Technical Re- port 112.00112 [cs. GR.. Stanford University Princeton University- - Toyota Technological Institute at Chicago, 2015.
Kim, S., Park, C., Jeong, Y., Park, J., and Cho, M. Stable and consistent prediction of 3d characteristic orientation via invariant residual learning. In Proceedings of the 40th In- ternational Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Chen, I.., Xu, J., Wang, C., Huang, H., Huang, H., and Hu, R. Uprightrl: upright orientation estimation of 3d shapes via reinforcement learning. In Computer Graphics Forum, volume 40. pp. 265-275. Wiley Online Library, 2021.
Kim, S. h.. Hwang, Y., and Kweon, I. S. Category-specific upright orientation estimation for 3d model classification and retrieval. Image and Vision Computing, 96: 103900, 2020.
Deitke, M., Schwenk, D., Salvador, J.. Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13142- 13153,200 2023.
Kuchnik, M. and Smith, V. Efficient augmentation via data subsampling. In International Conference on Learning Representations, 2019. URL https: /pwpneevie.eeeeeee/e net
Lin, C.-K. and Tai, W.-K. Automatic upright orientation and good view recognition for 3d man man-made models. Pattern recognition, 45(4):1524-1))) 1524-1300, 2012.
Fu. H., Cohen-Or, D., Dror, G., and Sheffer, A. Up- right orientation of man-made objects. In ACM SIG- GRAPH 2008 Papers, SIGGRAPH '08, New York, NY, USA, 2008. Association for Computing Machin- cry. ISBN 9781450301121, doi: 10.1145/1399504, 1360641. URL https: wtd.ii.oog//...100 0195504.1360444
Liu, Z., Zhang, J., and Liu, L. Upright orientation of 3d shapes with convolutional networks. Graphical Models, 85:22-29, 2016.
Mchr, E., Lieutier, A., Bermudez, F. S., Guitteny, V., Thome, N., and Cord, M. Manifold learning in quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9165-917777 2018.
Gower, J. C. and Dijksterhuis, G. B. Procrustes problems, volume 30. OUP Oxford, 2004.

---

Symmetry-Robust 3D Orientation Estimation
Moakher, M. Means and averaging in the group of rotations. SIAM journal matrix analysis and applications, 24(1): 1-16, 2002.
Zhou, K., Bhatnagar, B. L.,, Schiele, B., and Pons-Moll, G. Adjoint rigid transform network: Task-conditioned alignment of 3d shapes. In 2022 international conference on 3D vision (3DV), pp. 1-11. IEEE, 2022.
Pang, X., Li, F., Ding, N., and Zhong, X. Upright- Learn- ing upright orientation for 3d point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1491 1-14919, 2022.
Poursaeed, O., Jiang, T., Qiao, H., Xu, N., and Kim, V. G. Self-supervised learning of point clouds via orientation estimation. In 2020 International Conference on 3D Vision (3DV), pp. 1018-1028. IEEE, 2020.
Puny, O., Atzmon, M., Smith, E. J., Misra, I., Grover, A., Ben-Hamu, H., and Lipman, Y. Frame averag- ing for invariant and equivariant network design. In International Conference Learning Representations, 2022. URL http::::///ppeeeeeiiiiee net/ Eorum? id=zIUyj55nXR,
Romano, Y., Sesia, M., and Candes, E. Classification with valid and adaptive coverage. Advances in Neural Infor- mation Processing Systems, $3:3581-3573 2020.
Sajnani, R., Poulenard, A., Jain, J., Dua, R., Guibas, L. J., and Sridhar, S. Condor: Self-supervieed canonicalization of 3d pose for partial shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16969-16979, 2022.
Spezialetti, R., Stella, F., Marcon, M., Silva, L.,, Salti, S., and Di Stefano, L. Learning to orient surfaces by self- supervised spherical cnns. Advances in Neural informa- 2020.
tion processing systems,
Sun, W. Tagliasacchi, A., Deng, B., Sabour, S., Yazdani, S., Hinton, G. E., and Yi, K. M. Canonical capsules: supe-supe capsules in canonical pose. Advances in Neural information processing systems, 34:24993-25005, 2021.
Wang, W., Liu, X., and Liu, L. Upright orientation of shapes via tensor rank minimization. Journal of Mechan- ical Science and Technology, 28:2469-282 2477, 2014.
Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., and Solomon, M. Dynamic graph for learning point clouds. ACM Transactions Graphics (tog), 38 (5):1-12, 2019.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, I.., Tang, X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912- 1920, 2015.
10

---

Symmetry-Robust 3D Orientation Estimation
A. Additional results and proofs
A.1. The orienter map for symmetric shapes is not a function
Proposition A.1. Let S E S a fixed shape which is symmetric under non-trivial group of rotations Rş S SO(3), and let fgg be its orientation. Then there is no function f such that f(RS) RSls for all R SO(3).
Proof: Suppose S(RS) - Rag for all R E SO(3), and fix some non-identity rotation E Rg under which S symmetric. Then RS SS but S(RS) Rag + f2 = f(S), so f(RS) + [(5) even though RS S. Hence must be one-to-many map and is therefore not function.
A.2. Proposition 3.1
The key insight is that if is a function, then (RS) = (R'S) for R' SO(3) such that RS R'S. Equation 2 will then drive the optimal (RSS) the Euclidean mean (Moakher, 2002) of the rotation matrices R' such that RS R.S. We begin by showing that these are precisely the matrices RQ for Q Rs.
As R.s is the group of symmetries of S, QS S for all Q E R.s. Given some rotation R E (O(3), left-multiplying by π then yields RQS = RS for all Q E Rs. This relationship also holds in reverse: If RS = R'S for R. R' E SO(3), then R' = RQ for some Q E Rgg. To see this, note that if RS = R'S, then S = R. R'S and hence R R' E Rg. Consequently, R' = R(R Rt' = RQ for Q = R R' Rg. It follows that:
(R' E 80(3): RS R'S} {RQ : e Rs).
We can therefore write solution to Equation 2 evaluated at RS as follows:
(RS) argmin ('
E [| - R'ss [1:]]
argmin TESSOCYRQQQRRR
E [|| - RQ
RQ:QERs
argmin
E
|||0" RQQ
QEU(O())EU(RRa)))
argmin R' ES03))
||R RQNs|E.
Rg
QERs
This is the Euclidean mean of the matrices RQNs as defined in Moakher (2002). Proposition 3.3 in the same reference states that the solution this problem is found by computing the arithmetic mean Rsl EQER: RQNs and then orthogonally projecting this onto (0(3). In particular,
1
(*(S) = =projso(3)
RQΩs
# RΩg.
R
QERs
Hence L2 regression fails to learn the orientation Ps of shape S e S that possesses non-trivial set of rotational symmetries Rss.
A.3. Non-uniqueness of solution to Problem 2
Consider the bench shape S depicted in Figures 1, 2, As shown in Figure 3, this shape has two rotational symmetries: The identity rotation, and 180" rotation about the y-axis. One may represent these rotations by the matrices / and Q = (-ea, Ey, -ez), respectively, where Ez, ey: are the standard basis vectors.
Proposition 3.1 states that one solves the L regression problem 2 for the bench shape by computing the arithmetic mean of and Q and then orthogonally projecting this matrix onto SO(3). The arithmetic mean of T, Q is the matrix M == (0, cg. 0),

---

Symmetry-Robust 3D Orientation Estimation
and computes its orthogonal projection onto SO(3) by solving special Procrustes problem (Gower & Dijksterhuis, 2004):
min RESO(3)
| - M
(6)
The solution to this problem is non-unique for M == (0, Ey, 0), and the minimum is attained by any rotation about the y-axis (i.e. any rotation matrix whose second solumn is ey). This shows that even a single non-trivial rotational symmetry leads to an entire submanifold of solutions (S) to Problem 2.
A.4. Proposition 3.2
We begin by defining an equivalence relation over 50(3). Given two rotations R,, R2 E SO(3), we call R,, R₂ equivalent and write R1 ~ R2 if there exists some Q R such that R2 = R1Q. We verify that this is an equivalence relation:
Reflexivity: E R since R a group and R1 = R2I, so R1 ~ R1.
Symmetry: Suppose R1 R2. Then R₂ R.Q for some Q R. As R is a group, R R-1 E R as well, and R₂Q -R1 so R2 Rg.
Transitivity: Suppose Rı R2 and R₂ R3. Then there are Q, R such that R2 R1Q and R3 R2Q'. Hence R3 R2= QQ'Q'' and as R is group, QQ' R. We conclude that Ri R3.
This confirms that ~ is a valid equivalence relation. Using this equivalence relation, we partition SO(3) into equivalence classes, choose a unique representative for each class, and use [R] E SO(3)/ ~ to denote the unique representative for the equivalence class containing R E SO(3). We then use this map to define candidate solution Equation over the space of rotated shapes { (RS: f* is not one-to-many), and then show that attains a loss value in Equation 4.
SO(3) as (RS)) == [ss We will first verify that this defines valid function (i.e. that
We first show that f defines valid function. To do so, must show that if R&S = R2S, then (R) (R2 To this end, suppose that S,S = R2S. Then S = R R2S, so Q == RT R2 E RRs C R. It follows that R2 Rí R Rs Ri Q for some Q E R, so R1 ~ Rg. Since R₁ ~ R2, [R,] [R and so (R) = [R,]s = [Ra][2s = f (R2S). This shows that f defines a valid function.
We now show that f attains a loss value of in Equation For any R E SO(3), we have:
min [RS) RQQ
min QER
[|][
RQNsIIE.
QER
But clearly R [R), so there exists some Q* R such that [R] RQ".
Hence
min QER
[R]As RR0--RQQs- 11 0,
and as this reasoning holds for any (O()), follows that
E
min IIf(RS) RQQ.11? 0.
R~U((0(31)
LQER
We conclude that f is a minimizer Equation 4. Furthermore, f*( = [[]2s = Q2 for some Q R, which completes the proof of the proposition.
A.5. Proposition 3.3
F E Rs, then FS = S, so QFS = QS for any other rotation Q e SO(3) and (QFS F E Rs} contains the symmetries of the rotated shape QS. The optimal solution p" to Equation maps a rotated shape QS (where Q E R) the
12

---

Symmetry-Robust 3D Orientation Estimation
empirical distribution of the targets Q E R conditional on shape QS. But if QFS = QS for all F E Rg, then this is the uniform distribution over the set {QF: F Rs).
Since (*(S)) = RQ" for some Q E R. (*SR)) RS (Q") and applying the general result from above, we conclude that (RS) RS) = p" ((Q")" S) is the uniform distribution over the set {(Q") FF: F Rs).
For any (Q*) F", one then computes ((Q")TF) F T RS = FTS. But as Rg is a group, F E Ry whenever F is, so F S and we conclude that ((Q*) (RS) RS = S.
B. Implementation details
B.1. Quotient orienter
We parametrize our quotient orienter by DGCNN and use the author's Pytorch implementation (Wang et al., 2019) with 1024- -24eesinnn embeddings, k = 20 neighbors for the EdgeConv layers, and a dropout probability of 0.5. Our DGCNN outputs unstructured 3 x 3 matrices, which we then project onto SO(3) by solving a special orthogonal Procrustes problem; we use the roma package (Brégier, 2021) to efficiently compute this projection.
We train our quotient orienter on point clouds consisting of 10k surface samples from Shapenet meshes. We subsample 2k points per training iteration and pass batches of 48 point clouds per iteration. We train the quotient orienter for 1919 epochs at learning rate of 10
Test-time augmentation for the quotient orienter. Our use of TTA for the quotient orienter is motivated by the observation that the quotient orienter succeeds on most rotations and fails on small subsets of rotations. Given an arbitrarily ceptrarily-orrented oriented input shape RS, we would like to mitigate the possibility that the shape is in an orientation for which the quotient orienter fails. To do so, we apply K random rotations R3 to the input shape to obtain randomly re-rotated shapes Rgg R.S. Because the quotient orienter only fails on small subsets of rotations, we expect it to succeed on most of the R2 RS and recover the orientations of Rg. RS up to an octahedral symmetry.
Because we are interested in the orientation of RS rather than R, we then need to apply the inverse of R3 to each predicted orientation Fo( RSS) to obtain set of K predicted orientations R2 (RS). These will be correct orientations of R.S up to octahedral symmetries for each l where the quotient orienter succeeds.
Because we expect this to be the case for most k, we employ a voting scheme to select one of the candidate orientations R fo RS). In this step, we compute each candidate's 's average quotient L2 loss with respect to every other candidate (i.c. the loss in Problem 3), and choose the candidate which minimizes this measure. Because we expect the quotient orienter to have succeeded for most of the RSS, this average quotient loss will be small for most candidates and large for the few outlier candidates on which the orienter failed. Choosing the candidate with the minimum average quotient loss with respect to the other candidates filters out these outliers and makes it likelier that we output the correct orientation of RS up to an octahedral symmetry.
B.2. Flipper
We parametrize our flipper by a DGCNN and use the author's Pytorch implementation (Wang et al., 2019) with 1024- dimensional embeddings. k= = 20 neighbors for the EdgeConv layers, and dropout probability of 0.5. Our flipper outputs 24-dimensional logits, as we quotient our first-stage regression problem by the octahedral group, which contains the 24 rotational symmetries of cube.
We train our flipper on point clouds consisting of 10k surface samples from Shapenet meshes. We subsample 2k points per training iteration and pass batches of 48 point clouds per iteration. We train the quotient orienter for 3719 epochs at a learning rate of 14- We draw rotations Q E (O)) during training, and simulate inaccuracies in quotient orienter's predictions by further rotating the training shapes about a randomly drawn axis by an angle uniformly drawn from [0, 10] degrees.
Test-time augmentation for the flipper. The TTA procedure for the flipper is similar to the quotient orienter's TTA procedure. We are given an input shape FS, which is correctly oriented up to an octahedral symmetry (a "flip") F if the quotient orienter succeeded. We apply K random flips Fù to the input shape to obtain randomly re- re-rotated shapes F&FS and apply the flipper to each of these shapes. Similarly to the quotient orienter, if the flipper succeeds on some FFFS, it
13

---

Symmetry-Robust 3D Orientation Estimation
predicts F rather than the flip F are actually interested in. We consequently left-multiply each prediction g F& FS) by FKT which maps each successful prediction g FS) Fi to the true flip F. Because expect the flipper have succeeded on most inputs F&FS and failed on minority of inputs, we again use voting scheme to pick out the plurality prediction. In this case, we simply return the most common flip among the set of Fi (FF.F))
B.3. Adaptive prediction sets
We implement adaptive prediction sets following the method in Angelopoulos al. (2020) with their regularization parameter A set to 0. To calibrate our conformal flipper, we first draw a subset of the validation set (the calibration set), apply random octahedral flip Q ~ U to each calibration shape, and then pass each flipped shape QS through the trained flipper to obtain class probabilities (Q() E A33. The calibration score for a shape is the sum of the model's s class probabilities p(QS); ranked in descending order, up to and including the true class i* corresponding to the ground truth flip Q. We fix a confidence level a and return the (1 a--th quantile 7 of the calibration scores for each shape in the calibration set. In general, smaller values of 1 lead to smaller values of r, which ultimately results in smaller prediction sets at inference time, whereas large values of 1 lead to larger prediction sets at inference time but with stronger guarantees that these sets include the true flip.
At inference time, we first obtain the flipper model's output probabilities (S()) E A | for some shape S, then sort P*(S) in descending order and add elements of R. to the prediction set until their total mass in (5()) reaches т. Intuitively, these sets will be small when the flipper is confident in its prediction and assigns large mass the bigbest-probability classes. Conversely, the sets will be large when the flipper is uncertain and assigns similar mass to most classes.
B.4. Motivation for test-time augmentation.
In practice, our trained orienter exhibits discontinuities at certain rotations. To demonstrate this, we perform an experiment on bench shape, which is symmetric under 180 degree rotation about the y-axis. We rotate the bench around this axis in increments of degree and pass the resulting shape into our orienter for each rotation. We track two metries throughout this process:
The quotient loss of our orienter's prediction (i.e. the loss function from Problem 3, where we quotient the Le loss by the octahedral group). This measures the accuracy of our orienter's predictions up to an octahedral symmetry.
The chamfer distance between the shapes obtained by applying the orienter's output to the rotated bench at subsequent angles of rotation. This detects discontinuities in the orienter's output.
We plot both of these metries across the rotation angle in Figure 9. Our orienter exhibits several sharp discontinuities, manifested spikes in the chamfer distance plot. These discontinuities are associated with spikes in the quotient loss, which are occasionally large. However, these spikes in the quotient loss are highly localized, and our orienter performs well for the vast majority of rotations. These localized spikes motivate our use of TTA to improve our pipeline's performance. We provide implementation details for TTA in Appendices B. and B.2.
Figure 9: Our orienter exhibits several sharp discontinuities when rotating an input shape about a fixed axis (red curve).
These discontinuities are associated with localized spikes the quotient loss (blue curve), which motivate our use of TTA to improve our pipeline's performance.
14

---

Symmetry-Robur 3D Orientation Estimation
C. Additional figures
W
Figure 10: A grid of our model's outputs given randomly rotated, non-cherrypicked meshes from Shapenet. We render each mesh whose chamfer distance with respect the ground truth is below predetermined success threshold in blue, and render the remainder in red. Our success threshold is the average chamfer distance between shape randomly rotated by 100 and the original shape; we compute this average across 100 Shapenet validation meshes. Even when predicted shapes do not agree with the ground truth orientation determined by the creators Shapenet, their orientation is often acceptable in practice.
15

---

Symmetry-Robur 3D Orientation Estimation
Figure 11: A grid of 20 adaptive prediction sets generated using our pipeline given randomly-rotated, non-cherrypicked meshes from Shapenet. We render each mesh whose chamfer distance with respect to the ground truth below a predetermined success threshold in blue, and we render the remainder in red. Our success threshold is the average chamfer distance between shape randomly rotated by 10" and the original shape; we compute this average across 100 Shapenet validation meshes. Our adaptive prediction sets are typically small and include reasonable orientations, which an end user may choose between needed for their target application.
16

---

Symmetry-Robust 3D Orientation Estimation
Figure 12: A grid of model's outputs given randomly rotated, non-cherrypicked meshes from Objaverse. Our quotient regressor consistently succeeds out of distribution meshes, most of pipeline's 's outputs correctly oriented up cube flip. Our flipper has greater difficulty generalizing, but predicts acceptable flip in many cases. We expect that training on a larger and more diverse dataset of oriented shapes will improve our flipper's generalization performance.
17


Reviews:
[{"review_id": "VsioURUhBi", "reviewer": "Reviewer_3giS", "summary": "The paper introduces a two-stage method for estimating the pose of an object (3D point cloud). In the first stage, the pose is regressed modulo octahedral symmetries, which prevents prediction collapse for symmetric objects. In the second stage, the remaining octahedral ambiguity is resolved through classification. Experiments on ShapeNet/ModelNet show that the approach works well.\n\n## update after rebuttal\nThe authors adequately responded to my comments. A refined title, refined propositions, and including the shown discontinuity plot in the paper/appendix improve the work. So, I have raised my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and your helpful comments. We address your questions below. \n\n**Proposition 3.2:**\n\n*If I'm not mistaken, the proof shows that there exists an $f^\\*$ of the specified form, not that all minimizers are of the specified form (as claimed in the prop.).*\n\nThis is correct; thank you for pointing this out. We will gladly amend Prop. 3.2 in the camera-ready to state that there exists a solution of the specified form.\n\n*Let $S$ be a cube. Let $\\hat{R}$ be the octahedral group. Consider rotating the cube around one of its symmetry axes continuously from 0 to 90 degrees. [...] Is there a discontinuity somewhere in between? What happens if one inputs a cube into the trained network and varies its rotation?*\n\nIn practice, our trained orienter does exhibit discontinuities at certain rotations. To demonstrate this, we perform an experiment on a bench shape, which is symmetric under a 180 degree rotation about the y-axis. We rotate the bench around this axis in increments of 1 degree and pass the resulting shape into our orienter for each rotation. We track two metrics throughout this process:\n\n- The quotient loss of our orienter’s prediction (i.e. the loss function from problem (3), where we quotient the $L_2$ loss by the octahedral group). This measures the accuracy of our orienter’s predictions up to an octahedral symmetry.\n- The chamfer distance between the shapes obtained by applying the orienter’s output to the rotated bench at subsequent angles of rotation. This detects discontinuities in the orienter’s output.\n\nWe have plotted both of these metrics across the rotation angle [here](https://imgur.com/a/hjvfOM1). Our orienter exhibits several sharp discontinuities, manifested as spikes in the chamfer distance plot. These discontinuities are associated with spikes in the quotient loss, which are occasionally large. However, these spikes in the quotient loss are highly localized, and our orienter performs well for the vast majority of rotations. These localized spikes motivate our use of TTA to improve our pipeline’s performance. We provide further details on TTA in our response to Reviewer gQb9.\n\n*What happens when $S$ has more symmetries than present in $\\hat{R}$? For instance, in the experiments, there are several objects that have SO(2) symmetry (e.g. vases). Does the regression objective degenerate for these, similarly to in Prop 3.1?*\n\nIf S has more symmetries than $\\hat{R}$, then we would expect the regression objective to degenerate. As you note, this typically occurs for shapes with continuous symmetries such as vases, because the octahedral group covers many of the symmetries of shapes with finitely-many rotational symmetries. However, this degeneracy is in fact benign for such shapes. For example, the solution to (2) for a vase which is symmetric under any rotation about its up-axis may be any rotation about the up-axis (the shape’s continuous axis of symmetry). But unlike the bench shape that we discuss in lines 192-205, any such rotation is a valid orientation of the vase up to one of its symmetries, so our pipeline will output a rotation that returns the vase to its correct pose (up to a symmetry).\n\n**Experiments:**\n\n*Why is the network trained for \"3719 epochs\"? Is it early stopping with a larger number of epochs?*\n\nWe were primarily bottlenecked by computational constraints, and the validation accuracy was continuing to increase (albeit very slowly) when we decided to end our final training run. We believe that our pipeline’s accuracy could be slightly improved with further training, particularly for the flipper.\n\n*What is the runtime of the proposed method and Upright-Net respectively?*\n\nOur method’s inference time with TTA is 0.4517 seconds/shape (std=0.0252). Its inference time without TTA is 0.0126 seconds/shape (std=0.0466). Upright-Net’s inference time is 0.0672 seconds/shape (std=0.0252). We will add these details to the camera-ready.\n\nOur method’s inference time without TTA is faster than Upright-Net’s, but employing TTA makes our method’s inference time notably longer than Upright-Net’s. However, the new metrics that we have computed for our pipeline without TTA (links below) show that our method continues to strongly outperform Upright-Net if we omit TTA.\n\n*The proposed method uses TTA, does the baseline Upright-Net?*\n\nUpright-Net does not use TTA. For the sake of comparison, we have re-run the experiments in our paper without TTA. The angular error plots in Figure 6, now including our method without TTA, are [here (6a, Shapenet)](https://imgur.com/a/YpsbHrw) and here [(6b, ModelNet40)](https://imgur.com/a/FwyEydw). A screenshot of a new version of [Table 1 is here](https://imgur.com/a/y5KYtOR), and [Table 2 is here](https://imgur.com/a/4jBy2ko).\n\nOmitting TTA causes our method’s up-axis estimation accuracy to fall by roughly 4-5% depending on the dataset, but our method continues to strongly outperform Upright-Net without TTA.", "claims_and_evidence": "Broadly, the claims are well supported and the paper is well written.\n\n1. The conclusion states \"*Whereas previous approaches can only\ninfer upright orientations for limited classes of shapes, our\nmethod successfully recovers entire orientations for general\nshapes.*\" But, as mentioned in the introduction \"*Our work may be viewed as\nan efficient canonicalization method for the specific case of\n3D shapes with rotational symmetries.*\" In particular, earlier energy-based canonicalization approaches would not have an issue with symmetric objects.\n\n2. This is slightly subjective, but the title could be more specific than \"Orient Anything\". In this paper, 3D point clouds of man-made objects are oriented.", "methods_and_evaluation_criteria": "The evaluation makes sense.", "theoretical_claims": "I skimmed most of the proofs. I checked Prop 3.2 in more detail.\n\nIn general, there is an issue where the minimization problems specified minimize over functions $f$, but it is not specified what function class $f$ belongs to. This can lead to problems, e.g., if the proof requires $f$ to be discontinuous, not aligning well with neural network approximators.\n\n### Proposition 3.2\n1. If I'm not mistaken, the proof shows that there exists an $f^*$ of the specified form, not that all minimizers are of the specified form (as claimed in the prop.).\n2. Here is an example, which I think should be explained.\nLet $S$ be a cube. Let $\\hat{R}$ be the octahedral group. Consider rotating the cube around one of its symmetry axes continuously from 0 to 90 degrees. At 0 and 90 degrees, the output must be the same since the input is the same. Is there a discontinuity somewhere in between? What happens if one inputs a cube into the trained network and varies its rotation?\n3. What happens when $S$ has more symmetries than present in $\\hat{R}$? For instance, in the experiments, there are several objects that have SO(2) symmetry (e.g. vases). Does the regression objective degenerate for these, similarly to in Prop 3.1?", "experimental_designs_or_analyses": "1. Why is the network trained for \"3719 epochs\"? Is it early stopping with a larger number of epochs?\n2. The inference time should be reported.\n3. The proposed method uses TTA, does the baseline Upright-Net?", "relation_to_broader_scientific_literature": "It is a promising idea to factor out symmetries in canonicalization methods. It requires the existence of a subgroup that covers the symmetries of standard inputs.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "- The authors state that the octahedral group is among the largest subgroups of SO(3), without mentioning the icosahedral group. It would be interesting to see if using the icosahedral group leads to worse performance due to not being aligned with the symmetries of common man-made objects.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Let me copy my questions to this section for the convenience of the authors.\n\n### Proposition 3.2\n1. If I'm not mistaken, the proof shows that there exists an $f^*$ of the specified form, not that all minimizers are of the specified form (as claimed in the prop.).\n2. Here is an example, which I think should be explained.\nLet $S$ be a cube. Let $\\hat{R}$ be the octahedral group. Consider rotating the cube around one of its symmetry axes continuously from 0 to 90 degrees. At 0 and 90 degrees, the output must be the same since the input is the same. Is there a discontinuity somewhere in between? What happens if one inputs a cube into the trained network and varies its rotation?\n3. What happens when $S$ has more symmetries than present in $\\hat{R}$? For instance, in the experiments, there are several objects that have SO(2) symmetry (e.g. vases). Does the regression objective degenerate for these, similarly to in Prop 3.1?\n\n### Experiments\n1. Why is the network trained for \"3719 epochs\"? Is it early stopping with a larger number of epochs?\n2. What is the runtime of the proposed method and Upright-Net respectively?\n3. The proposed method uses TTA, does the baseline Upright-Net?\n\nI will raise my score to accept if these are reasonably addressed and no other important issues are discovered during the reviewing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oiJbzjNcuc", "reviewer": "Reviewer_gQb9", "summary": "The paper presents a two-stage deep learning method to orient shapes, dubbed Orient Anything. It proves why naive L2 regression of the orientation matrix fails for shapes with symmetries, and presents a theoretical framework to overcome the problem. The method consists in selecting a finite group $\\hat{R}$ that contains the rotational symmetries of the object. In practice, the authors choose the octahedral group that contains the 24 rotational symmetries of a cube and captures most of the rotational symmetries of real-world shapes. Then, the problem of orientation estimation is divided into two stages, corresponding to two neural networks: the first regresses the orientation up to a rotation in $\\hat{R}$ (so called \"quotient regressor\"), the second one selects one of the rotations in $\\hat{R}$ (so called \"flipper\"). The composition of the two transformations gives the estimated orientation. Finally, since the orientation of some shapes is inherently ambiguous, the authors propose to use conformal prediction when a human is in the loop, to let the network output adaptive prediction sets whose size varies with the flipper's uncertainty. The method is trained on ShapeNet and quantitatively tested on ShapeNet and ModelNet40, both standard and challenging benchmarks for up-axis estimation, i.e. estimating only the vertical axis of the object, and estimation of the full orientation. It outperforms an existing baseline, UpRightNet, for up-axis estimation and it outperforms some made-up baselines, based again on UpRightNet, for the full orientation case. Some qualitative results of generalization capabilities to ObjaVerse are reported in the appendix.\n\n## Update after rebuttal\nI carefully read all reviews and responses. The authors clarified how TTA is implemented and performed an ablation study on its contribution, which was found to be not critical for the overall performance of the method and its superior performance with respect to the baseline. The explanation of the first step of Prop 3.1 is also clear now. I'd suggest to add this explanation in the appendix. The other reviews do not uncover critical weaknesses and the responses to them seems convincing. For all these reasons, I confirm my overall recommendation. However, I agree that the title should be toned down, if possible. In particular, I'm in favor of the proposal by the authors \"3D Orientation Estimation for Symmetric Shapes\" (dropping \"Robust\" for the reasons discussed by reviewer ffTb).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and your helpful comments. We will gladly fix the typos you have pointed out and make the changes to the figures that you have requested in the camera-ready. **Please see our response to Reviewer 3giS for links to new versions of Tables 1 and 2 and Figure 6, where we have now included our method’s performance without TTA.** Below, we provide further details on how we implement TTA, and we clarify the first step of the proof of Prop. 3.1.\n\n**TTA implementation details.**\n\n**Quotient orienter.**\n\nOur use of TTA for the quotient orienter is motivated by the observation that the quotient orienter succeeds on most rotations and fails on small subsets of rotations; see our response to Reviewer 3giS for a discussion of how this may originate from discontinuities in its outputs. Given an arbitrarily-oriented input shape $RS$, we would like to mitigate the possibility that the shape is in an orientation for which the quotient orienter fails. To do so, we apply $K$ random rotations $R_k$ to the input shape to obtain randomly re-rotated shapes $R_kRS$. Because the quotient orienter only fails on small subsets of rotations, we expect it to succeed on most of the $R_kRS$ and recover the orientations of $R_kRS$ up to an octahedral symmetry.\n\nBecause we are interested in the orientation of $RS$ rather than $R_kRS$, we then need to apply the inverse of $R_k$ to each predicted orientation $f_\\theta(R_kRS)$ to obtain a set of $K$ predicted orientations $R_k^\\top f_\\theta(R_kRS)$. These will be correct orientations of $RS$ up to octahedral symmetries for each $k$ where the quotient orienter succeeds. \n\nBecause we expect this to be the case for most $k$, we use a voting scheme to choose one of the candidate orientations $R_k^\\top f_\\theta(R_kRS)$. (This is step (4) in B.1, which you have inquired about.) In this step, we compute each candidate’s average quotient $L_2$ loss w.r.t. every other candidate (i.e. the loss in Problem (3)), and choose the candidate which minimizes this measure. Because we expect the quotient orienter to have succeeded for most of the $R_kRS$, this average quotient loss will be small for most candidates and large for the few outlier candidates on which the orienter failed. Choosing the candidate with the minimum average quotient loss wrt the other candidates filters out these outliers and makes it likelier that we output the correct orientation of $RS$ up to an octahedral symmetry.\n\n**Flipper.**\n\nThe TTA procedure is similar here. We are given an input shape $FS$, which is correctly-oriented up to an octahedral symmetry (a “flip”) $F$ if the quotient orienter succeeded. We apply $K$ random flips $F_k$ to the input shape to obtain randomly re-rotated shapes $F_kFS$ and apply the flipper to each of these shapes. Similarly to the previous case, if the flipper succeeds on some $F_kFS$, it predicts $F_kF$ rather than the flip $F$ we are actually interested in. We consequently left-multiply each prediction $g(F_kFS)$ by $F_k^\\top$, which maps each successful prediction $g(F_kFS) = F_kF$ to the true flip $F$. Because we expect the flipper to have succeeded on most inputs $F_kFS$ and failed on a minority of inputs, we again use a voting scheme to pick out the plurality prediction. In this case, we simply return the most common flip among the set of $F_k^\\top g(F_kFS)$.\n\nWe do not use TTA when outputting adaptive prediction sets. We would be pleased to add these details to the relevant appendices in the camera-ready.\n\n**Proof of Prop. 3.1.**\n\nWe now clarify the first step of the proof of Prop. 3.1. Because we do not place any restrictions on the functions $f : \\mathcal{S} \\rightarrow SO(3)$ other than being functions (i.e. not one-to-many), Problem (2) decouples over the inputs to $f$. That is, we can independently solve for the optimal $f^*(RS)$ for any input shape of the form $RS$ with $R \\in SO(3)$. If $S$ has rotational symmetries, there will be several $R’ \\in SO(3)$ for which $RS = R’S$. Because $f^*$ cannot be one-to-many, we must have $f^*(RS) = f^*(R’S)$ for all such $R’$. Consequently, the terms in Problem (2) that are relevant to determining $f^*(RS)$ are precisely those involving the $R’$ s.t. $RS = R’S$.\n\nThis means that the $f^*$ which solves Problem (2) is defined at any input shape $RS$ as the rotation $f^*(RS) := R^* \\in SO(3)$ which solves the problem on line 576.", "claims_and_evidence": "The claims are fully supported by clear and convincing theoretical results and experimental evidence.", "methods_and_evaluation_criteria": "Evaluation criteria make sense and, for the up-axis estimation, are the one used in previous work. The main weakness is the lack of proper baselines for the full orientation case. The baselines created by the authors, being based on UpRightNet, were clearly at a disadvantage, so, in some sense, the second experiment does not provide new evidence in terms of comparisons, Yet, the authors tried their best and the experiment provides the absolute performance of the method on the harder problem of full orientation estimation. Hence, I think it is valuable and should stay in the final version.", "theoretical_claims": "I  believe the theoretical claims are correct. My only issue is with the first step of the proof of proposition 3.1 in A.2. It is not clear why the equation at line 576 is equivalent to Equation 2. In equation 2, the Expected value is on $R \\in U(SO(3))$. Not sure why here it becomes the expected values on $R'$ such that $RS = R'S$, even after reading the introduction to the section. I'd suggest to clarify this step.", "experimental_designs_or_analyses": "Experimental designs and analyses are in general sound and valid.\n\nOne concern I have is on the effect of test-time augmentation: the authors did not ablate its contribution, so I think OrientAnything without TTA should be added to Table 1 and 2, to assess its importance. Moreover, for the flipper, it is not clear how a single output is selected when not using conformal prediction. The paper just reads \"(4) output the plurality prediction\", but there are multiple predictions from the multiple augmentations. Moreover, it is not clear how this works when using adaptive sets: in which order are logits from the different shapes ranked? A full rank across different input orientations or is some criterion deployed to choose one of them? I'd suggest to provide more details on TTA.", "relation_to_broader_scientific_literature": "The paper cites the relevant literature and provides a clear original improvement with respect to it, both at the theoretical and at the performance level.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "S1. The paper is clearly written and presented.\n\nS2. The idea of dividing orientation estimation into two steps based on the quotient regression is original, sound, non-trivial and effective.\n\nS3. The method is shown to outperform the considered baseline.\n\nW1. Test-time augmentation is not explained clearly, its hyperparameters are not reported and its importance is not ablated.", "other_comments_or_suggestions": "Fonts in Figure 2 are too small.\n\nFigure 4 does not really illustrate that Equation 2 has many solutions. This became clear to me only after reading A.3. I'd suggest to remove Figure 4 and move A.3 in the main text or to create a more informative figure.\n\nline 353 \"validation\" -> is it \"test\"?", "questions_for_authors": "Q1. Please clarify how TTA is realized and its impact on the model performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AmYg7RYVDA", "reviewer": "Reviewer_yNUC", "summary": "This paper explores the challenges faced by current baselines that aim at predicting 3D-shape orientation. It shows that trying to minimize an L2 distance cannot recover the ground-truth orientation in the presence of intrinsic symmetries as the solution to the L2 distance will not be unique. To address this challenge, the paper proposes a two stage pipeline that(1) predicts a set of approxiamations to the ground truth orientation (which recovers a shape’s orientation up to octahedral symmetries), classify this approximations to return the best orientation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and your helpful comments. In this rebuttal, we will answer the two questions you have posed to us in your review.\n\n*We are wondering in lines 296-306 of the experimental setup. It seems the proposed method is trained on 10K points per shape and 2K points are sampled out of the 10k points per shape for each epoch, while UprightNet is only trained on 2k points in total per shape?*\n\nThis is correct. We followed the original implementation of Upright-Net as closely as possible, which calls for drawing a fixed reservoir of 2048 points to use in each iteration. Because one can draw arbitrarily large point clouds from the surface of a mesh, we saw no reason to restrict ourselves to a fixed reservoir of 2048 samples when training our pipeline.\n\n*We would also like to ask the authors how the proposed pipeline performs when trained on a similar dataset to that proposed in Upright-Net?*\n\nBecause our models are relatively costly to train, we have not had the opportunity to run this experiment during the rebuttal period. However, we believe that training both methods on all of Shapenet provides a more complete and realistic picture of how each method will perform in practice than training both methods on a small subset of classes from ModelNet40.", "claims_and_evidence": "Look okay. The paper proposes to separately predict approximations to the grounth-truth approximations which they do using their proposed \"quotient orienter\", and then classify these proposed approximations to return the best approximation using their proposed \"flipper\". The main idea being that directly solving the regression problem to predict the orientation in the presence of intrinsic symmetries is not possible which they show using their propositions 1-3.", "methods_and_evaluation_criteria": "Looks okay. But we have two questions:\n- We are wondering in lines 296-306 of the experimental setup. It seems the proposed method is trained on 10K points per shape and 2K points are sampled out of the 10k points per shape for each epoch, while UprightNet is only trained on 2k points in total per shape?  \n- We would also like to ask the authors how the proposed pipeline performs when trained on a similar dataset to that proposed in Upright-Net?", "theoretical_claims": "Propositions 3.1-3.3 all look okay in the paper and appendix.", "experimental_designs_or_analyses": "Looks okay. However, please refer to the \"Methods And Evaluation Criteria\" Section.", "relation_to_broader_scientific_literature": "Looks okay.", "essential_references_not_discussed": "No recommendations.", "other_strengths_and_weaknesses": "No recommendations.", "other_comments_or_suggestions": "No recommendations.", "questions_for_authors": "Please refer to the \"Methods And Evaluation Criteria\" Section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1tKKX36xfA", "reviewer": "Reviewer_ffTb", "summary": "This work presents a full orientation estimation method for generic shapes. Concretely, a two-stage framework is proposed for this task. The method first uses a quotient orienter to recover the shape's orientation up to octahedral symmetries by continuous regression. Then a flipper is employed to predict one of 24 octahedral flips that returns the first-stage output to canonical orientation via standard classification. Additionally, a conformal prediction stage is used to enable the flipper to output adaptive prediction sets, resolving ambiguities in the results through human-in-the-loop interaction. Experimental results show that the proposed method achieves SOTA performance in up-axis prediction and full-orientation recovery.\n\n## update after rebuttal\nThe reviewer would be happy to accept this paper if the title is changed to \"3D Orientation Estimation for Symmetric Shapes\" as the reviewer `gQb9` suggested.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and your helpful comments. In this rebuttal, we will address many of the questions you have asked in your review.\n\n*In the experiments, the ShapeNet dataset is divided into a 90% training split and a 10% testing split. Then, where does the validation set come from?*\n\nBecause our two models are relatively costly to train (see below), we did not tune our DGCNN’s hyperparameters and instead used the defaults from the original DGCNN implementation. We chose to subsample 2k points per point cloud in each training iteration to roughly match Upright-Net’s 2048 points, and chose the largest batch sizes possible given our GPU’s memory constraints. We did monitor validation metrics during training, but our decision to end our training runs were driven primarily by computational constraints, and our validation metrics were continuing to improve (albeit very slowly) when we ended our final training runs.\n\nWe consequently do not have a separate test set. We have referred to our metrics as “validation” metrics rather than “test” metrics elsewhere in the manuscript, and would be happy to refer to the 90-10 split as a training-validation split. Our results on ModelNet40 and on Objaverse provide a comprehensive picture of our method’s performance on fully unseen data, and our ModelNet40 results show that our method also strongly outperforms Upright-Net on fully unseen data.\n\n*How robust is the method to noise? In practice, reconstructed shapes frequently include noise points. Could the authors evaluate the method’s resilience to noisy shapes, especially nearly symmetric shapes?*\n\nThank you for this suggestion. We have repeated our experiments testing our pipeline’s up-axis estimation accuracy on Shapenet, where we now add Gaussian noise with a standard deviation of 0.05 and 0.1 to the point clouds (which have been normalized to lie in the unit ball) and normals (we re-normalize the normals after adding noise). We depict the result of this experiment [here](https://imgur.com/a/K1rTayu). Our method is fairly robust to small amounts of noise, with a noise std of 0.05 resulting in our pipeline’s accuracy (\\% of shapes with angular error of up-axis prediction $<10^\\circ$) dropping from 89.2% to 86.2%. Increasing the noise std results in a larger accuracy penalty, with our pipeline’s accuracy dropping further to 77.6%.\n\nWe also highlight that Figure 11 in the appendices depicts our pipeline’s outputs on meshes from Objaverse. These meshes are both highly ood for our pipeline, which was trained on Shapenet, and generally lower-quality than those found in Shapenet.\n\n*Could the authors provide runtime analyses for both the training and inference stages of the method?*\n\nPlease see our response to Reviewer 3giS for our method’s inference times. Our orienter takes 420 seconds per epoch to train on a single V100 GPU – so 1719 epochs of training is 9.33 days for the orienter. Our flipper takes 410 seconds per epoch on the same V100 GPU – so 3719 epochs is 17.6 days for the flipper. We found that both models’ validation metrics continued to improve after many epochs of training. Improving our problem’s conditioning so that both models require fewer epochs to converge would be a valuable direction for future work.\n\n*How were the standard deviations in Table 2 calculated?*\n\nThe mean and standard deviation of the chamfer distances were computed over the Shapenet validation set.\n\n*The term \"anything\" should be used very carefully. To show that the method can achieve the \"anything\" capability, more experimental results might be needed. For instance, the method can be tested on human/face/hand/animal shapes and protein molecule shapes.*\n\nWe would be pleased to change our paper's title if the program chairs allow it, especially since we have become aware of another recent paper titled \"Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models.\" For example, we could call our paper \"Robust 3D Orientation Estimation for Symmetric Shapes\".", "claims_and_evidence": "Mostly yes. But the claim of \"anything\" is somewhat problematic.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, but I am not able to check the correctness of all the proofs.", "experimental_designs_or_analyses": "Yes. There could be some more experimental analyses to support the claim of \"anything\".", "relation_to_broader_scientific_literature": "This paper is related to orientation estimation, shape canonicalization and geometric deep learning. The main contribution is to make the full orientation estimation possible for a wide range of generic shapes.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n- As noted by the authors, this is the first attempt to solve the task of full-orientation estimation for generic shapes without class information.\n- The paper analyzed why a naive regression approach would fail for full orientation estimation for generic shapes.\n- The code is available.\n\nWeaknesses\n- In the experiments, the ShapeNet dataset is divided into a 90% training split and a 10% testing split. Then, where does the validation set come from?\n- How robust is the method to noise? In practice, reconstructed shapes frequently include noise points. Could the authors evaluate the method’s resilience to noisy shapes, especially nearly symmetric shapes?\n- The term \"anything\" should be used very carefully. To show that the method can achieve the \"anything\" capability, more experimental results might be needed. For instance, the method can be tested on human/face/hand/animal shapes and protein molecule shapes.\n- Could the authors provide runtime analyses for both the training and inference stages of the method?", "other_comments_or_suggestions": "- The initial two sentences of both the Abstract and the Introduction are essentially identical, rendering them somewhat redundant.\n- The titles of certain subsections should not end with full stops.\n- How were the standard deviations in Table 2 calculated?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Orient Anything, a two-stage framework for estimating the full 3D orientation of objects, particularly addressing challenges posed by symmetrical shapes. The first stage employs a quotient orienter that predicts the object's orientation up to the 24 rotational symmetries of the octahedral group, effectively handling ambiguities inherent in symmetric objects. The second stage utilizes a flipper to classify and resolve these symmetries, refining the orientation prediction to a unique solution. To manage residual uncertainties, especially in cases of inherent ambiguity, the method incorporates conformal prediction, providing adaptive prediction sets.

This paper was appreciated by all reviewers. Reviewer 3giS and gQb9 gave accept scores, and reviewer yNUC and ffTb gave weak accept scores. The authors provided extensive comments to clarify reviewers' concerns about technical details and experiment settings. AC recommends applying the reviewers' comments and suggestions in the revised version. Especially since the paper title is ambitious, the reviewer ffTb's final comment regarding testing with reconstructed 3D shapes by other approaches is reasonable.
Decision: Accept (poster)