Orient Anything

Authors: Christopher Scarvelis, David Benhaim, Paul Zhang
Primary area: learning on graphs and other geometries & topologies
Keywords: 3d orientation, shape analysis, 3d deep learning, geometric deep learning

Abstract:
Orientation estimation is a fundamental task in 3D shape analysis which consists of estimating a shape's orientation axes: its side-, up-, and front-axes. Using this data, one can rotate a shape into canonical orientation, where its orientation axes are aligned with the coordinate axes. Developing an orientation algorithm that reliably estimates complete orientations of general shapes remains an open problem. We introduce a two-stage orientation pipeline that achieves state of the art performance on up-axis estimation and further demonstrate its efficacy on full-orientation estimation, where one seeks all three orientation axes. Unlike previous work, we train and evaluate our method on all of Shapenet rather than a subset of classes. We motivate our engineering contributions by theory describing fundamental obstacles to orientation estimation for rotationally-symmetric shapes, and show how our method avoids these obstacles.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730216737346

thread_type:
official_review

<thread_object>
creator: reviewer_acsd

<content>
summary:
The paper addresses the problem of estimating the orientation of rigid 3D shapes. The paper starts from the fact that, theoretically, it is not possible to have a function that, for any arbitrarily oriented shape with non-trivial rotational symmetries, predicts the right orientation (output is not unique, so such a map cannot be a function). The paper also shows that L2 objectives are not suitable for recovering the solution by optimization.

Hence, the paper proposes to decompose the problem in two parts:
1) The Quotient Orienter: a neural network (DGCNN) that solves for the rotation with an L2 objective quotient by the octahedral group.
2) The Flipper: a neural network (DGCNN) trained to solve a classification problem to recover one of the 24 octahedral flips, returning the canonical orientation.

The approach is compared against one baseline, which shows significantly better accuracy on ShapeNet and ModelNet.

strengths:
- The paper addresses a valuable problem that could potentially impact many different research areas. Symmetries have been widely studied in computer graphics and computer vision and have mainly focused on the recent Geometric Deep Learning theory. 
- While previous approaches focus on learning orientation for a single object class, the proposed approach aims to provide a unified model that works on all the object classes at once.
- The theoretical discussion of the paper seems reasonable, and the code is provided in supplementary. This could be an instrumental tool for several downstream tasks, and results are probably easy to replicate.

weaknesses:
1) My main concern lies in the experimental evaluation of the approach. The method compares against Upright-net, Shapenet, and ModelNet. Although I consider favorably the use of the entire ShapeNet dataset, I generally think more comparisons would be informative. For example, ART ("Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes", Zhou et al., 3DV 2022) seems relevant but not discussed in the paper and could serve as a baseline for the category of "canonical alignment" methods. Notice that ART also showcases results on humans. This latter is also an interesting domain, and I would be interested to know the authors' opinions on whether their method would suit this case. Similarly, I evaluate positively that the method has been tested on Objaverse, but the few qualitative examples do not help to understand the actual flexibility of the method.
2) Although I enjoyed reading the method, I think there is room to improve the presentation. The most severe problem is probably the choice of qualitative visualization, which is difficult to read and compare. I would suggest visualizing fewer examples but bigger ones with clearer axe orientations, as well as pairing them with the GT for comparison. The method is also quite extensive. In the end, the conclusion is relatively straightforward: although the problem of recovering an orientation cannot be formulated as a function, it is actually enough to restrict to predict one valid solution (and actually, the only doable things, since the solution in the same equivalence class are indistinguishable). As a consequence, this also makes the problem tractable with an L2 optimization. This is reasonable and could be expressed in a much more direct way. Finally, Figure 6 seems a bit deceptive due to the logarithmic scale. The number of valid shapes (bars on the left of the dashed lines) seems similar between the two methods, but the quantitative metrics indicate differently. The larger difference between the blue and red bars is often in the scale of 10¹, while some small differences in the bins of 90 and 180 degrees are actually in the 10³. I suggest removing the logarithmic scale or providing a cumulative curve visualization (e.g., on the y-axis, the percentage of shapes below or equal to the x-axis angular error).
3) The papers do not offer any ablation nor analysis of the method. This would be important to validate intuition and guide future work. For example:\
(a) how would the Flipper perform without the quotient oriented or with an oriental that does not quotient the space? \
(b) How does the model scale with the number of different data and classes? Is it beneficial to consider all the classes together, or would it be better to train specialized models for every class? \
(c) What is the computational time in terms of inference and training? \
(d) Figure 6 seems to suggest that the error of the proposed method focuses on some sharp bins (e.g., 90 and 180 degrees, but also 45 and 135 for ModelNet). Is it due to the octahedral nature of the Flipper classifier? \
(e) Why use octahedral and not icosahedral symmetries? 

I think these are all aspects that would be worth further investigation.

questions:
Please, refer to the questions listed in points (1) and (3) of weaknesses.

Minor:
1) Line 335:"We observe that our quotient orienter [...] fails for a small subset of rotations". Could you elaborate more on this? Have you noticed a recurrent pattern in the failure mode?
2) "Our method improves on Upright-Net’s up-axis estimation accuracy by nearly 20 percentage points, corresponding to a 64.6% reduction in the error rate relative to the previous state of the art.". Could you elaborate a bit more about what are the terms of this "reduction"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644373462

thread_type:
official_review

<thread_object>
creator: reviewer_xs2k

<content>
summary:
The paper presents an approach to predict the canonical 3D pose of an arbitrarily posed 3D point cloud of an object. The proposed two-stage pipeline first predicts an upright pose, using a DGCNN backbone to regress a rotation matrix from a point cloud. Then it performs a 24-way classification to resolve the remaining degree of freedom. Compared to Upright-Net (2022), it is able to find the upward orientation better on ShapeNet and ModelNet40. Top-1 and top-4 accuracy is reported for full orientation prediction.

strengths:
Clarity: easy to read and the problem definition is clear.

weaknesses:
In my view, the technical rigor does not meet the conference's standards. I appreciated the careful explanation and presentation, but spending multiple pages on why a direct L2 regression of rotational symmetry doesn't work seems unnecessary. Considering the presentation and focus of other papers published in this venue, I think this could be seen as common sense. I did not find enough significance, although it may be that no one's done this exact proof before.

Experimental validation: Upright-Net is the only other method compared against. While the numbers are better, I think the experiments are simple enough that a more thorough validation should have been done; using the same backbone network, etc. And I was not sure what the take-home message was; so is it better to predict the upright pose as a matrix and orthogonalize as presented? What was the reason for the improvement? And for the full-orientation prediction, N-way classification is a reasonable approach, but it is not compared against any baseline, and the final pose is dependent on the upright direction prediction which I did not find convincing enough.

questions:
Experiment suggestion: The paper says "our method reliably provides a plausible set of candidate orientations for diverse shapes unseen during training". I think it might be good to do a novel category evaluation experiment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707068182

thread_type:
official_review

<thread_object>
creator: reviewer_zyej

<content>
summary:
This paper proposes a supervised method for predicting the orientation of a 3D object. The main observation is that direct regression will fail due to the abundance of symmetry in 3D shapes. Therefore, a two-stage approach is approached. Using the DGCNN backbone, the method first regresses the orientation up to octahedral symmetries (chosen as it is the single group that contains the most common symmetries in SO3). Then, the method classifies the transfomation that "flips" the shape, oriented by the first step, into the correct orientation. The method is intuitively sound and avoids issues with previous regression based works, by having better overall performance, work on more general settings (full orientation, single model over entire collection of shapes, instead of per catregory), and have error characteristics (centered around few modes) that make further human-in-the-loop correction easier.

strengths:
- Practically useful approach that clearly contributes to an important problem in 3D shape analysis. Clearly outperforms relevant works, achieving better quantitative performance and is more general.
- The idea/intuition behind this approach is not only useful for the particular setting (orientation estimation), but generally applicable to a much wider range of 3D tasks where certain steps involve predictions that can be complicated by symmetries (just naming a few that comes to my mind: part-based analyis/generation, assembly, articulation, shape retrieval, etc.)
- Solid mathematical analysis to back up the method - though I do not have the background to and did not check thoroughly for its correctness. The proofs does sound reasonable.

weaknesses:
- One could argue that the method itself, though wrapped in solid mathematics, is not *that* novel, since hybrid classification+regression strategies are very common overall, and the key observations about symmetries complicating L2 regression is not new. It might help to further discuss/clarify the novelty.
- The octahedral group, while covering a good amount of 3D symmetries, also leave out a lot of important ones e.g. the very common 5-way symmetry for chair bases, many mechnical objects e.g. bolts/gears. While I think there can be extensions to this method to cover an even wider range of shapes, the present version probably would not work well on these. And unlike a single step approach, I do have the feeling that this method will struggle a lot more for these shapes. Some analysis here would be ideal - either on how to sidestep these cases or to show that the proposed method works no worse than single step approaches.
- Arguably, ShapeNet is no longer the go-to dataset now given that there exist much larger shape datasets now. Would be interesting to see more analysis there. Figure 11 does cover objaverse but might be good to have more in the main body.

questions:
I am overall leaning towards acceptance. I think this paper contains a solid, although somewhat straightforward, idea, that can be useful in many applications. I am currently not willing to champion for the paper, as I am personally not too sure about the magnitude of impact this paper can make, and given that there are clearly cases where the proposed method cannot solve (but perhaps can be adapted to solve). More clarification on novely/contribution and more analysis on these potential limitations can make me more positive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732737489406

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Joint response to reviewers

comment:
We are grateful for the reviewers’ valuable comments and their continued engagement throughout the discussion period. We have uploaded a final revision of the manuscript, where we have fixed the caption for Figure 9. (We thank Reviewer acsD for pointing out this issue.) Below, we outline our final case for this work’s publication.

The reviewer guidelines state that “different objectives [...] require different considerations as to potential value and impact.” Our objective in “Orient Anything” is to build a shape orientation pipeline that surpasses the previous state of the art by a wide margin on well-established benchmarks in this domain. **All reviewers agree that we have delivered on our objective**, with Reviewer Zyej stating that Orient Anything “clearly outperforms relevant works,” Reviewer XS2K stating that our results are “strong,” and Reviewer acsD stating that we achieve “significantly better accuracy on ShapeNet and ModelNet.” We emphasize that our performance improvements are far from marginal: We exceed the previous SOTA (Upright-Net) on up-axis estimation by nearly 20 percentage points, corresponding to a 64.6% reduction in the error rate. As noted by Reviewer Zyej, our method is also “more general” than competing methods, which we demonstrate by training and testing our pipeline on all of Shapenet, rather than a subset of classes as in previous work.

Reviewers Zyej and acsD agree on the importance of the shape orientation problem, with the former characterizing it as an “important problem in 3D shape analysis,” and the latter calling it a “valuable problem that could potentially impact many different research areas.”  Reviewer XS2K asks “how other researchers can benefit from [our] findings.” Zyej believes that our two-stage strategy is “generally applicable to a much wider range of 3D tasks where certain steps involve predictions that can be complicated by symmetries,” and cites tasks such as part-based analysis and generation, assembly, articulation, and shape retrieval as examples. In light of its strong performance and reproducibility, acsD also anticipates that Orient Anything “could be an instrumental tool for several downstream tasks.” We believe this indicates that the shape analysis and 3D deep learning communities will benefit greatly from our contributions.

Some reviewers shared concerns about our method’s simplicity, with Zyej stating that “hybrid classification+regression strategies are very common overall” and XS2K stating that our method “feels more like an approach that would typically serve as a baseline.” Our method’s simplicity is among its *key advantages*, making theory tractable, which enables us to back up our method’s SOTA performance by rigorous results that help explain its empirical success. To our knowledge, there is no prior work in shape orientation employing our two-stage problem decomposition; as such, Orient Anything is a clearly novel contribution to this literature.

The reviewers’ comments have shown that Orient Anything is a **novel** algorithm for solving an **important** problem in 3D shape analysis, **achieving SOTA by a wide margin** while being **more general** than previous approaches. We believe this suggests that our work will have a substantial impact on shape analysis and 3D deep learning, serve as a useful tool to the community, and form a valuable part of this year’s ICLR program.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper proposes to predict the canonical orientation of a 3D object represented as a point cloud.  A two stage pipeline is proposed: 1) an orienter (DGCNN) regressor network that identifes the orientation (up to octahedral symmetries)  2) a flipper (classification network) that selects among the symmetries (24 for octahedral group).  Experiments on up prediction, and full-orientation prediction are conducted to show the effectiveness of the proposed approach.

The main contributions of the work are: 1) revisiting the task of canonical pose estimation using modern techniques 2) proposal of a two stage approach that takes into account that many shapes contains rotational symmetries, 3) mathematical analysis of the soundness of the proposed method.

**Strengths**

1. The paper addresses a practical problem of orienting 3D objects. [acsD]   
2. Proposed approach is shown to perform prior work [Zyej]
3. Solid mathematical analysis backing up the method [Zyej,acsD]
4. Idea is interesting and can potentially be useful for other 3D tasks involving dealing with shape symmetries [Zyej,acsD]
5. Code is provided [acsD]

**Weaknesses**

1. Lack of discussion on what constitutes "canonical orientation" across different object categories. As noted by reviewer acsD, there are ambiguities on what would be considered the "canonical orientation".  
   - For instance, for a "book", is the "canonical orientation" the book with the front cover of the book as the front (as typically placed on desks) or the spine of the book as the front (as typically found on bookcases).  For a L-shaped couch, which side would be considered the front?  
2. Weak experiments and lack of ablation and analysis [XS2K,acsD]
   - No comparisons / baselines for full-orientation estimation.  For instance, a class-specific method can be used as a comparison point. 
3. Presentation needs improvement [XS2K,acsD]
   - Overcomplicated long explanation in the main paper of why L2 loss is not ideal [XS2K]
   - Qualitative visualizations are poorly quality and hard to read [acsD]
4. Training cost and how well would the model scale [acsD]
5. Limited discussion on various design decisions (octahedral group) [acsD]

**Recommendation**

The AC believe predicting the canonical pose of a 3D object can be useful in many settings.  The paper presents an interesting approach that can potentially be applied to another shape-analysis problems that involves dealing with shape symmetries.  

Overall, while the work has strong mathematical foundations, reviewers felt there was limited discussion on the problem itself (e.g. whether it makes sense for a canonical orientation across object categories) and limited experiments. Reviewers also expressed the opinion that the presentation can be improved so that the paper is easier to read.  Finally, the AC note that while the problem statement is important for different applications, the topic may be more appropriate for a graphics or vision venue.

Based on the above and reviewer ratings, the AC feels that the work is not ready for acceptance at ICLR.  The AC encourages the authors to improve the work and consider resubmitting to a future venue.  

**Additional notes**

A bit of discussion to address weakness 1, and some improved experiments (including perhaps metrics that take into account that some orientations are equally valid), together with improved presentation will make this work much more compelling. 
For instance, the problem statement and proposed solution can be stately in a more concise and simple manner, with some details moved to the appendix.  It would also be good to discuss limitations of the work.  

In addition to the above, the AC notes the following (minor points):
- The paper refers to solutions to different problems (e.g. "Problem 2").  The AC believe these refer to the numbered equations in the paper and recommends referring to these as "Equation 2", etc.
- The notation for L2 regression uses "L^2", where it is more common to use "L_2"