Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Authors: Zehan Wang
Ziang Zhang
Tianyu Pang
Chao Du
Hengshuang Zhao
Zhou Zhao
Venue: ICML 2025 poster (2025)
Keywords: Object pose
Orientation
Primary area: applications->computer_vision
TL;DR: Practical Solution for Estimating Object Orientation in Images

Abstract: Orientation is a fundamental attribute of objects, essential for understanding their spatial pose and arrangement. However, practical solutions for estimating the orientation of open-world objects in monocular images remain underexplored. In this work, we introduce Orient Anything, the first foundation model for zero-shot object orientation estimation. A key challenge in this task is the scarcity of orientation annotations for open-world objects. To address this, we propose leveraging the vast resources of 3D models. By developing a pipeline to annotate the front face of 3D objects and render them from random viewpoints, we curate 2 million images with precise orientation annotations across a wide variety of object categories. To fully leverage the dataset, we design a robust training objective that models the 3D orientation as probability distributions over three angles and predicts the object orientation by fitting these distributions. Besides, we propose several strategies to further enhance the synthetic-to-real transfer. Our model achieves state-of-the-art orientation estimation accuracy on both rendered and real images, demonstrating impressive zero-shot capabilities across various scenarios. Furthermore, it shows great potential in enhancing high-level applications, such as understanding complex spatial concepts in images and adjusting 3D object pose.

Orient Anything:
Learning Robust Object Orientation Estimation from Rendering 3D Models
Zehan Wang If Ziang Zhang !t Tianyu Pang
Chao Du
Hengshuang Zhao Zhou Zhao
https: / /trp/otreenttt wwwrriinntnnyttiinnne anything hww.thiig.gighu..... github.io/
Figure introduce novel method estimating the object orientation in images, represented by red axis, while the blue and green axes indicate the upward and left sides of the object. More examples provided in Appendix. Best viewed screen with zoom.
Abstract
gories. To fully leverage the dataset, we design a robust training objective that models the 3D ori- entation as probability distributions over three an- gles and predicts the object orientation by fitting these distributions. Besides, we propose several strategies to further enhance the synthetic-to-real transfer. Our model achieves state-of-the-ars ori- entation estimation accuracy on both rendered and real images, demonstrating impressive zero- shot capabilities across various scenarios (Fig. 1). Furthermore, shows great potential in enhane- ing high-level applications, such understanding complex spatial concepts in images and adjusting 3D object pose.
Orientation is fundamental attribute of objects, essential for understanding their spatial pose and arrangement. However, practical solutions for es- timating the orientation of open- world objects in monocular images remain underexplored. In this work, introduce Orient Anything, the first foundation model for zero-shot object orientation estimation. A key challenge in this task is the scarcity of orientation annotations for open-world objects. To address this, we propose leveraging the vast resources 3D models. By developing a pipeline annotate the front face of 3D objects and render them from random viewpoints, curate million images with precise orientation annotations across wide variety of object cate-
1. Introduction
Equal contribution Project leader *Corresponding author, 'Zhejiang University 'Sea Al Lab 'The University Hong Kong. Correspondence to: Zhou Zhao <zhaozhou @ zju.
Perceiving object properties in single image is the core problem in computer vision. Current visual foundation models and large lasiunel models (VLMs) excel in tasks like object recognization (Zhang et al., 2024; Liu et al., 2024). localization (Liu et al., 2023; Li et al., 2022), tracking (Wang et al., 2023; Rajič al., 2023), and segmen-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
tation (Kirillov al., 2023; Ravi al., 2024). However, estimating orientation of open-world objects, which is crit- ical for understanding object pose and arrangement, has been underexplored due to the lack of annotated data, Ob- jectNer3D (Xiang et al., 2016) and Omni3l) (Brazil et al., 2023) enable 3D orientation prediction with manually an- notated objects, but its scope is still restricted to specific domains and limited object categories, making it difficult to generalize to diverse real-world scenarios.
From Falcon's 's (left man in image) perspoctive, is Captain America fright man in image) on his left or right?
On your Left!
Furthermore, even the most advanced general visual under- standing systems, such as GPT-4 (Hurst et al., 2024) and Gemini (Team et al., 2023; 2024), struggle to comprehend basic object orientation. The perception of this fundamental object attribute has not emerged in current vision- vision-language models. As result, they perform poorly on tasks involving orientation, such predicting object movement trends or understanding spatial relationships, as shown in Fig. 2.
For Fulcon, Captuin America is on his right.
For Fulcon, Captain America is on his right.
Figure 2. Understanding object orientation is essential for spatial reasoning. However, even advanced VLMs like GPT-40 and Gem- ini-1.5-po pro are not yet able to resolve the basic orientation issue.
In this paper, we first propose to learn how various objects look under different orientations by rendering 3D models. By annotating the front face of massive 3D objects (Deitke et al., 2023), we can easily and cheaply obtain precise orien- tation labels for rendered views. In particular, we leverage advanced VLM (Team et al., 2024) to identify the front side of 3D objects from orthographic views, complemented by canonical pose detection and symmetry analysis to simplify the task and improve accuracy. Then, we render images from random perspectives, using azimuth and polar angles relative to the 3D orientation vector, combined with the camera rotation angle, to represent the 3D orientation. This idea provides scalable, diverse, and easy-to-acquired data, enabling the development of accurate and generalizable ori- entation estimation models.
fitting task as the learning objective to stabilize the training process and improve generalization.
We investigate various model initialization and data augmentation strategies to improve synthetic-to-real transfer.
Our model exhibits much stronger orientation estima- tion ability compared to both the expertise model (Cube RCNN) and leading VLMs (GPT-40 and Gemini).
2. Related Work
2.1. Object Orientation Estimation in Images
Although scalable orientation data is available now, training a reliable orientation prediction model remains non-trivial. Direct regression of the three angles struggles to converge, resulting in poor performance. To overcome this challenge, we reformulate the single angle values as probability distri- butions to better capture the correlation between adjacent angles. By driving the model to fit these angle probability distributions, we simplify the learning process and signifi- cantly enhance model robustness. Furthermore, considering the domain gap between the rendered and real images, we in- vestigate various model initializations to better incorporate real-world prior knowledge, alongside data augmentation strategies to improve synthetic-to-rea. transfer.
Some tasks attempt recognize object orientations in im- ages under certain conditions or with extra information.
Object viewpoint estimation (Su et al., 2015; Tulsiani & Ma- lik, 2015) aims to predict the object viewpoints for limited categories. 6DoF object pose estimation (Guan et al., 2024) focuses on detecting the 3D position and orientation of ob- jects in images. However, these methods typically require the guidance of CAD model or other reference views of the target object (Sundermeyer et al., 2018; Peng et al., 2019; Goodwin et al., 2022; Nguyen et al., 2024; Fan et al., 2024), and are often limited to small set of categories.
Rotated object detection (Xie et al., 2021; Han et al., 2021: Brazil et al., 2023) concerned with generating ro- tatable 2D or 3D bounding boxes for objects in images. Omni3D (Brazil et al., 2023) unifies multiple 3D object detection datasets and develops Cube R-CNN. While Cube R-CNN demonstrates certain abilities in detecting object orientation in 3D space, it is primarily trained on indoor scenes and street environments, limiting its zero-shot gener- alization to in-the-wild scenarios.
Our contribution can be summarized as:
We develop an automatic and reliable 3D object ori- entation annotation pipeline, and highlight the values of rendering 3D objects for cost-effective, diverse, and scalable images with precise orientation labels.
We introduce the orientation probability distribution
2

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Unlike the aforementioned tasks, our work focuses on 3D orientation estimation of open-world objects and only re- quires monocular images.
Object Direction
Spatial Purt
Spatial Relation
Overall
Random
12,93
22.12
17.54
16.75
GPT-4o
49.32
15.38
27.27
32.50
Gemini- 1.5-ro
58.90
15.38
18.18
33.00
2.2. Orientation-based Understanding
Orient Anything+I.IMM
67.12
46.15
40.91
51.50
Object orientation provides context about how objects are positioned relative one another and to the viewer (the cam- era), which is fundamental for object pose and relationship understanding. Accurate orientation understanding plays a key role in many advanced applications.
Table J. Quantitative results on the proposed Ori-Bench.
VLMs to understand object orientation and tackle related questions. We manually curate 200 images in total, with 100 from COCO (Lin et al., 2014) and 100 generated by DALL-E (Betker al., 2023). To substantívely evalu- ate the understanding of object orientation, each image is horizontally flipped to produce a paired mirrored version, with answers adapted accordingly. A sample will be marked as solved only if the model correctly answers the question on both versions. There are three kinds of tasks: (1) Ob- ject Direction Recognition (73+73 samples): identifying the orientation of object within images; (2) Spatial Part Reasoning (39+39 samples): distinguish parts of an object with specific spatial meanings, like left vs. right hand of human; and (3) Spatial Relation Reasoning (88+88 sam- ples): imagining the relative position of one object from the perspective of another.
In 3D scene understanding, many studies (Chen et al., 2020; Achlioptas et al., 2020; Azuma et al., 2022) have highlighted the importance of spatial relationships informed by object orientation. SQA3D (Ma et al., 2022) first describes the position and orientation of an agent in a 3D scene, then tasks the model with answering questions based on the given spatial context. EmboidedScan (Wang et al., 2024) manually annotates orientations for 3D objects and utilizes the pose information describe spatial relationships among objects in 3D space.
In the domain of 2D images, understanding object orienta- tion is also fundamental for accurately interpreting (Góral et al., 2024; Wu et al., 2024) or generating (Shi et al., 2023; Huang et al., 2024; Wei et al., 2023) spatial rela- tionships and properties. Góral al. (2024) propose the visual perspective-taking task assess 2D VLM's ability to understand the orientation and viewpoint of person in images and highlight various applications based on this ability. Furthermore, the object orientation relative to the camera determines its pose in the image, which is essen- tial for distinguishing spatial properties such as the front wheels of cars and the left shoulder of a person, along with complex spatial relationships. Moreover, generating objects with given pose conditions is vital for controllable image generation (Huang al., 2024; Wei et al., 2023).
In Tab. 1, we show the accuracy of GPT-4o and Gemini- Pro. In the basic direction recognition task, the advanced VLMs are only able to correctly solve around 60% of the samples. This limitation further impacts their performance in spatial reasoning and relation questions, where the power- ful GPT-4o and Gemini- Cemmini-1.5-Pe 1.5-Pro perform similarly to random guessing. The pilot study highlights the need for fundamen- tal tools precisely estimate object orientation in images. Our simple baseline, Orient Anything+ (see Sec. 7.1 for details), outperforms the powerful GPT-4o and Gemini- 1.5-Pro in these orientation-related tasks.
Although object orientation is related to numerous applica- tions, practical solutions for estimating the orientation of arbitrary objects in images are still undercxplored. Our work fills this gap by proposing the first foundation model for ob- ject orientation estimation, which exhibits strong zero-shot performance in real-world scenarios.
4. Orientation Data Collection
The scarcity of annotations is the main obstacle to learn- ing general orientation estimation. Existing annotations for images, typically captions (Schuhmann al., 2021; 2022), bounding boxes (Lin et al., 2014), or segmentation masks (Kirillov et al., 2023; Zhou et al., 2017), seldom include object orientation information, and manually an- notating object orientation in images is extremely time- consuming and costly. To overcome this limitation, we propose to utilize 3D assets. Annotating the front face of 3D objects and rendering images from random perspectives pro- vides an efficient and effective way to generate large-scale images with precise orientation annotations.
3. Orientation Understanding in 2D VLMs
Before proposing our method for object orientation estima- tion, we need to address an important question: "Do current 2D VLMs, trained on web-scale image datasets with bil- lions of parameters, inherently learn to distinguish the orientation of various objects?"
To this end, we first develop an automatic 3D asset annota- tion and rendering pipeline, as shown in Fig. 3. Each step of the pipeline is detailed below.
To this end, we introduce Ori-Bench, the first VQA bench- mark specifically designed to assess the capacity of 2D
3

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Canonical 3D Models Filtering
b. Orientation Annotating
Free-view Rendering
Casanical
Top view
A view
view
C View
D view
(cotacion)
Asymmetric
Symmetrical
(polar)
Titel
Five orthographic views of the same
object are provided, Select the front view of the object from three options: 1.A view. view, 3. No front view
2D VLM
(azimuth)
A.A view is the front view.
Figure 3. The orientation data collection pipeline is composed of three steps: 1) Canonical 3D Model Filtering: This step removes any 3D objects tilted poses. 2) Orientation Annotating: An advanced 2D VLM used to identify the front face from multiple orthogonal perspectives, with view symmetry employed to narrow the potential choices. 3) Free-view Rendering: Rendering images from random and free viewpoints, and the object orientation is represented by the polar 0, azimuthal yp and rotation angle ò of the camera.
Step1: Canonical 3D Models Filtering We use Obja- verse (Deitke et al., 2023), large-scale dataset containing 800K object assets, as our database. Although most objects in this dataset are modeled in canonical poses', some are tilted along orthogonal axes, as shown in Fig. 3.a. To sim- plify the annotation process and enhance reliability, we first exclude all the tilted assets, focusing solely on 3D objects in canonical poses. This idea reduces the 3D object orientation annotation classification task. Rather than identifying the specific orientation vector, we only need to determine the front face from images rendered alone x, -2, and -y axes, or to conclude that the object has no front face.
information. Since the front and back faces of objects are typically asymmetrical, we leverage this prior knowledge to further narrow down the possible choices. Specifically, we use combination of SIFT (Lowe, 2004), structural similar- ity, and pixel color similarity to assess the similarity between opposing views. Two views are considered symmetrical if their similarity exceeds the threshold. Gemini-1.55 Pro is tasked with identifying the front face of objects from asym- metrical opposing views. If the object is symmetrical along both the z - and y vs. -y, it is regarded as having no meaningful front face and orientation.
To filter out tilted objects, we analyze the tilt along the x, y. and axes for each object. Specifically, we extract the object edges for each view and use Principal Component Analysis (PCA) to identify the principal directions of these edges. If the principal edge direction is aligned with any coordinate axis (with tolerance of 2 for robustness) across all renderings, the object is considered to be in the canonical pose. Otherwise, it is deemed tilted.
Step3: Free- View Rendering Once the 3D asset's front face is annotated in 3D space, we can obtain its 3D orien- tation in images from any viewpoints. For simplicity and clarity, we use the spherical coordinate system to define object orientation. As depicted in Fig. 3.c, we calculate the relative polar angle and azimuth angle yp between the cam- era position and the object orientation axis, as well as the camera rotation angle 5, to represent the object orientation from the specific viewpoint.
Starting with the initial pool of 800K objects in the Ob- javerse dataset, we first curate 80K 3D models with high- quality texture. Applying tilt- filt-filtrring criteria, we select 55K objects in canonical poses for subsequent processing.
Before rendering, all 3D objects are scaled to a unit cube, with their centers aligned to the origin of the coordinate sys- tem. For each object, 40 images are rendered from random perspectives, with the camera aimed at the origin and each image rendered at 512x512 resolution. In total, we collect 2M rendered images with precise orientation annotations.
Step2: Orientation Annotating Using the selected 3D objects in canonical poses, we render five orthogonal views from the x, -x, y. -y and z axes. Although our pilot study in Sec. 3 indicates that current 2D VLMs struggle to accurately predict orientation from a single view, we find that they perform well in identifying which view is facing the camera when multiple orthogonal views are presented for comparison and reference, as shown in Fig. 3.b.
Statisties To assess the diversity of our dataset, we evaluate its categorical coverage across WordNet enti- ties (Fellbaum, 2010), following the methodology of Obja- verse (Deitke et al., 2023). The 55K objects we used span 7,204 distinct entities. In contrast, the previous 3D dataset with orientation annotations, ObjectNet3D (Xiang et al., 2016), includes only 100 categories.
Additionally, mitigate VLM hallucinations and improve annotation accuracy, we incorporate symmetry auxiliary
In our definition, standing upright and facing one of four orthogonal directions along the x. -x, y, and - y axes
4

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
be formulated as follows:
pal
Garssion Diumivation
exp
(1-9)² 203
Rendering
P pol (o)0,
Civrlar Disrivution
2n exp
180
[n 0)2
20g
Encoder
byot
Cirotir Diaribation
where: i = ,....., 180% and dg is the variance hyperparame- ter for polar distribution. For the ground truth azimuth angle o and rotation angle % due to its periodicity (e.g.. 359", 360", and 1 are adjacent), we employ the circular Gaussian distribution to form their target distribution P Pazi P Φp)) and Prot (i) 6, σT). For brevity, we illustrate this process us- ing azimuth as an example:
Figure 4. Orient Anything consists of simple visual encoder and multiple prediction heads. is trained to judge if the object in the imput image has meaningful front face and fits the probability distribution of 3D orientation.
cos( o
exp
P azi (1)p,0p))
2
5. Orient Anything
where i = ,..... 360 ox is the variance for polar distri- bution, and (o( is the zero-order modified Bessel func- tion of the first kind, which can be represented as:
Based on the massive images of objects with annotated 3D orientation 0, ip and 8, we train Orient Anything for general object orientation estimation in images.
5.1. Orientation Probability Distribution Fitting
Lo
(n!)2
2
Despite the availability of precise 3D orientation annota- tions, formulating an effective learning objective for robust orientation estimation remains non-trivial challenge. The initial approach, which involved directly predicting con- tinuous angle values with MSE loss as supervision, faced convergence issues and yielded suboptimal performance.
As shown in Fig. 4, the circular Gaussian distribution effec- tively models the periodicity of azimuth and rotation angles, ensuring the stability of the optimization process.
Training and Inference Given the input image I, we use a visual encoder to extract its latent feature, followed by prediction heads (simple linear layers) to output the distri- butions polar, azimuth and rotation angles: Pol E IR 180 P Pusi E R360 and P not E 360 respectively. representing the object orientation in 3D space. Additionally, the model out- puts an orientation confidence E IR,, to indicate whether the object has a meaningful front face and to aid in identify- ing centrally symmetric objects, such as balls and stools.
To address this, we first simplify the challenging continuous regression task into discrete classification problem, which is easier to optimize. Specifically, we divide the 360 range interval.
into 360 individual classes, each representing While lowering the task difficulty improves performance over continuous regression, fails to capture correlations between adjacent angles that produce nearly identical out- comes practice (e.g., rendering at polar 29", 30%, and 31%) Treating these close angles as independent classes neglects their inherent relationships, which may confuse the model. Therefore, we further reformulate the classification task as a discrete probability distribution fitting problem, which is also easy to converge and can fully capture the potential relationship between different orientations.
The target distributions P Pii 08), Paz(ilip o.) and P (io d.)) are defined above, with the orientation label being 1 if the object has front face, and 0 otherwise. We use cross-entropy (CE) loss to supervise the predicted ori- entation distributions, and the corresponding loss terms are denoted as: T Jazzi and Lyot- For è, binary cross-entropy (BCE) loss is employed, yielding Le- The final training loss is linear combination of the above four terms, and for objects without meaningful orientation, the Lpols Lazi List will be disabled:
Target Probability Distribution We first transform the ground-trutt -ount angles into target probability distributions, rep- resented as Gaussian distributions centered on the ground- truth angle, with manually set variances. These distributions are subsequently discretized into a grid-based format at 1 intervals. For given ground- -ruth polar angle (in degrees), the probability distribution of polar angle P (i I,, 0o) can
Abe: Lpal + Ligi + Liot + XLe c=1
0
I.
(4)
where is the loss coefficient for orientation judgment.
5

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
During the inference process, objects whose orientation confidence is lower than 0.5 would be thought to have meaningful front face and orientation. Otherwise, the angles with the highest probability in each distribution: P Pazi. P nor are taken as the predicted polar, azimuth, and rotation angle: 0. p. 8.
6. Experiments
6.1. Implementation Details
We train models at three scales for different purposes: VIT-S, ViT-B, and ViT-L, all initialized with DINOv2. The loss coefficient À in Eq. 4 is set to 1. The variance hyperparame- ters 0o 0e. and os configured 2.0", 20.0", and 1.0%. For optimization, we use the AdamW (Loshchilov, 2017) optimizer, with learning rate of le-5 for the pre-trained visual encoder and 1e-3 for the newly introduced prediction heads. The models are trained for 50,000 steps with a batch size of 64 on the curated 2M object orientation dataset. All trainings are conducted on A100 (40GB) GPUs.
5.2. Sythetic- Transferring
Although the rendered images of 3D objects provide exten- sive data with orientation annotations, there is distribution shift between synthetic rendered images and real images. We try to prompt effective synthetic-to-real transfer from two aspects: integrating real-world pre-training knowledge and narrowing the training- inference domain gap.
iraining-infrrnce
6.2. Zero-shot Real-world Orientation Recognition
The primary goal of this work is to estimate orientations of open-world objects in real images. To assess the model performance in real-world scenarios, construct two kinds of evaluation benchmarks.
Inheriting Real-world Knowledge by Initialization As demonstrated in (Yang et al., 2024; Ke et al., 2024), initial- izing the model with strong visual encoders pre-trained on real images can significantly improve its synthetic-to-red transfer ability. To evaluate this in our orientation estimation task, we train models initialized from 3 widely-used image pre-trained encoders: MAE (He et al., 2022). CLIP (Radford et al., 2021), and DINOv2 (Oquab et al., 2023). After trials and failures, DINOv2 yields satisfactory results, attributed to its task-agnostic pre-training, fine-grained perception, and strong generalization capabilities. Consequently, we develop our model using DINOv2 initialization.
1) For objects in domains previously explored, we com- pare Orient Anything with Cube RCNN(a fully-supervised model) (Brazil et al., 2023) across four indoor and street datasets: SUN RGB-D (Song et al., 2015), KITTI (Geiger et al., 2012), nuScenes (Caesar et al., 2020), and Objec- tron (Ahmadyan et al., 2021). For each dataset, 1,000 ob- jects with 3D orientation annotations were randomly se- lected and cropped from real images. We separately com- pute the Absolute Error (Abs) for azimuth, polar, and ro- tation angles for comparison. Additionally, we benchmark our model against the zero-shot and fine-tuned baselines on ImageNet3D (Ma al., 2024), currently the largest dataset for object orientation annotations. We also compare Orient Anything with FSDet few-shot model) (Xiao et al., 2022) on Pix3D (Sun et al., 2018) and Pascal3D+ (Xiang et al., 2014). Acc@ and Abs on the overall 3D spherical space are reported, following FSDetView
Narrowing Domain Gap by Data Augmentation There are two main differences between rendered and real images. We employ corresponding data augmentation strategies to reduce the domain gap and enhance transfer performance.
First, objects in rendered images are typically fully visible, whereas real-world images often contain partially visible or occluded objects. To bridge this gap, we incorporate random cropping as a training data augmentation strategy. This technique simulates the occlusion situation in real-world images, thereby improving the model's ability to generalize to real-world scenarios.
2) For objects in the wild, we collect samples from COCO (Lin et al., 2014) and manually annotate their ori- entations. Due the challenges of accurately annotating 3D orientations, we simplify the task by labeling object ori- entations on the horizontal plane in eight directions: front, back, left, right, front-left, front-right, back-left, and back- right. From the 80 categories in the COCO validation set, we select 20 images per category, resulting in a comprehen- sive benchmark of 1,600 samples. Two tasks are used for evaluation: (1) Orientation Judgment: Determine whether the object has a meaningful orientation; (2) Horizontal Ori- entation Recognition: Identify the direction the object is facing on the horizontal plane, selecting from one of the eight pre-defined directions.
Second, to avoid ambiguity, the rendered image contains only a single object, whereas real-world images often feature multiple objects. Given the remarkable success of current object detection and segmentation models, our approach only focuses on unambiguous orientation estimation for single object. To extend our model to multi-instance scenarios, we will isolate each object using segmentation model and estimate objects' orientation individually. This strategy not only enhances the model's 's applicability to real- world images but also mirrors the style of rendered images.
As shown in Tabs. 2, and 4, despite never being exposed to real-world images during training, each version of Orient
6

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
SUN RGB-D
KITTI
nuScenes
Objectron
Pascal3D+
Pix3D
Models
Azimuth Polar Rotation Azimuth Polar Rotation Azimuth Polar Rotation Azimuth Polar Rotation
Acc@30%
Absl
Acc@30
Absl,
Abs}
Abst
Abst
Abst
A
Absl,
Absl,
Abst
Absl,
Abst,
Abs),
FSDetView
46.00
38.3
49.00
Cube RCNN;
93.58
39.73
140.10
98.61
39.73
121.21
89.63
15.64
132.57
122.99
60.01
113.31
Ours (ViT-S) 58.20
11.63
3.59
65.85
5.00
1.08
72.68
5.58
2.16
39.45
23.47
18.26
46.00
37.50
68.00
Ours (ViT-B)
56.34
9.15
3.75
54.02
5.86
0.21
66.56
5.72
1.28
36.49
22.13
18.34
48.00
35.17
69.00
Ours (ViT-L)
42.98
8.38
3.66
44.22
3.57
0.89
55.17
4.08
1.78
30.09
22.19
18.54
55.00
22.90
66.00
Table 2. Zero-shot orientation estimation on six unseen real-image henchmarks. i indicates that the model is used in a few-shot manner, while & denotes fully-supervised models. Orient Anything, on the other hand, is used in zero-shot manner. These "unequal" comparisons further emphasize the effectiveness of our models. The best results are highlighted in bold.
Setting
Models
Avg.
Electronics
Furniture
Household
Music
Sports
Vehicles
Work
SmaagNee3DDRRsNNee Ours (ViT-B)
37.1
30.1
35.6
28.1
11.8
51.7
36.7
40.9
7ero-shot
48.5
61.0
66.8
37.9
27.3
25.6
70.8
33,4
ImageNec3D-ResNeae
53.6
49.2
52.4
45.8
26.0
65.2
56.5
58.5
Fine-tuning
ImageNer3D-DINON
64.0
75.3
47.9
32.9
23.5
74.7
38.1
64.0
Ours (ViT-B)
71.3
77.6
89.7
64.4
54.4
47.6
87.4
61.2
Table 3. Orientation estimation on Large-scale ImageNet3D, which covers 200 object categories. The Acc@30 30 results are reported.
Rendered Image
Real Image
Single View
Canonical Views
Canonical&
Models
Judgment Aoct
Azonurt Acc 22 5 Accl85't
Rotation A
Judgmont Recopuition
Models
Symmetrical
Acct
Acct
Random
$0.00
12,50
5:55
16.67
50.00
12:50
Gemini- 1.p-po
44.00
74.00
86.00
Cube RCNN
12.44
10.37
2.50
20.25
Gemini-1 1. ppr
57.29
19:06
16.31
85.12
66.96
31.95
GPT-4o
31.00
87.00
92.00
GPT-4o
61.85
19.94
17.56
81.00
69.29
45.78
Ours (WT-S)
73.88
63.18
71.62
97.06
78.54
63.44
Table 5. Ablation study for Orientation Annotation.
Ours (VIT-B)
74.88
71.94
81.37
99.56
81.25
70.19
Ours (VTT-L)
76.00
73.94
86.75
98.31
80.30
72.44
Table 4. Orientation estimation on both in-domain rendcred images and oft-oo-domain real images. The best results bold.
We evaluate methods from two aspects: 1) Orientation Judg- menf: Determine if the object has a meaningful front face. 2) Separated Orientation Estimation: Predict the accurate azimuth, polar, and camera rotation angles for objects, using Acc X (accuracy within tolerances of +X²) as metrics. Rotated object detection model: Cube RCNN (Brazil et al., 2023) and VLMs: GPT-4o (Hurst et al., 2024) and Gemini- 1.5-pro (Team et al., 2024), are used as baselines.
Anything clearly outperforms existing methods in recogniz- ing object orientations in real images. Orient Anything con- sistently exceeds the performance of previous approaches by significant margin across most categories, achieving over 70% accuracy in the majority of COCO categories. Detailed results on each benchmark provided in Appendix.
The results presented in Tab. 4 demonstrate the superior performance of our model in predicting 3D orientation for in-domain rendered images. In practical azimuth estimation, our method achieves more than three times the accuracy of alternatives. Notably, the performance of Cube RCNN and advanced VLMs is only marginally better than ran- dom guessing, with success rate of 19.94%, compared to 12.50% In contrast, the Orient Anything ViT-L. achieves 73.94% accuracy, highlighting its practical value in reliably distinguishing object orientation on the horizontal plane.
We visualize our model predictions in Fig. 1 and 7, 6, 8- 12 in Appendix. These qualitative results highlight Orient Anything's 's remarkable zero-shot capability across images captured or created by real cameras, human artists, or gen- crative models, as well as a variety of scenarios, including continuous video frames, multi-view images, and complex scenes containing multiple objects.
6.3. Rendered-Image Orientation Estimation
We first quantitatively validate models by estimating the numerical 3D orientation of the in-domain rendered images. We manually select and annotate 300 objects from Obja- verse, including 1 150 with orientation annotations and 150 without a meaningful orientation. For each object, we ren- der 16 images from random viewpoints, resulting in a total of 4,800 images for evaluation.
Due to similar definitions and the same random guess results, "Acc@22.5" for rendered image azimuth estimation and "Acc" for real image horizontal direction recognition are comparable. Our models achieve similar results on both metrics, highlighting the excellent synthetic-to- to-real transfer performance. For VLMs, recognizing horizontal directions in words is more accurate than predicting precise azimuth
7

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
(Rostan ReRist
Rendering Image
Real Image
71.5
Design
Variants
Azimuth
Polar
Recognition
21.0
0.5
2'
Acc@22..5
Acc@5
Acc
80.5
70.0
1000
19.5
lecrring Objective
Classification
Regression
68.75
12.00
79.00
20.50
21.48
66.93
Fitting
71.88
80.56
69.85
10
67.19
78.19
63.67
Figure 5. Ablation study for hyper-parameter do, o and 07g-
20
67.94
78.88
65.47
Number of Views
30
70.06
78.13
68.62
values in numbers, which reveals the shortcomings of VLMS in predicting precise values for 3D orientation. On the other hand, Cube RCNN performs significantly worse on rendered images due to its limited generalization capability.
40
71.88
80.56
69.85
69.12
80.69
66.48
CLII
58.44
71.88
49.27
Training Initialization
MAE
58,44
64.63
57.26
DINOv2
71.88
80.56
69.85
6.4. A Ablation Study
Training
None
71.88
80.56
69.85
To verify the effectiveness of our key designs, we conduct ablation experiments using the ViT-B encoder.
Augmentation
Cropping
71.94
81.37
70.19
Inference
Box
71.88
80.56
67.49
Augmentation
Mask
71.88
80.56
69.85
Designs in Orientation Annotating We evaluate our ori- entation annotating methods using the 300 manually anno- tated 3D objects introduced in Sec. 6.3. The results in Tab. 5 indicate that while VLMs achieve only 44% and 31% accu- racy when identifying object orientation from the top view alone, providing orthogonal perspectives substantially en- hances performance. Furthermore, incorporating symmetry as an extra condition further raises accuracy to nearly 90%, underscoring the effectiveness of our orientation annotating strategy and proving the reliability of the rendering data.
Table 6. Ablation study for Learning Objective, Number of Views, Training Initialization and Data Augmentation.
Effect of Model Initialization We compare several pow- erful pre-trained visual encoders as initialization for our orientation estimation task in Tab. 6. We empirically find that DIONv2 exhibits much better performance in both in- domain convergence and out-of-domain transfer compared to others, which may be attributed to its large-scale task- agnostic pre-training and superior fine- fine-grained perception.
Effect of Fg, Oi and 0g Fig. 5 shows the effect of vari- ance hyper-parameter for three kinds of angle probability distribution. In general, our method is insensitive to the variance selection, while most configurations yield superior results compared to the one-shot label.
Efffec of Data Augmentation Tab. 6 present the effect of data augmentation for improving sythetic-to-real trans- fer. During training, random cropping enables rendered images to mimic the object occlusions, which significantly enhances the performance in real-world scenarios. For in- ference, using segmentation masks to isolate objects aligns more closely with the style of rendered images compared to bounding boxes, thereby narrowing the domain gap and improving overall performance.
Effect of Probability Prediction In Tab. 6, we ablate the three learning objectives discussed Sec. 5.1: continuous value regression, discrete angle classification, and proba- bility distribution fitting. Direct regression yields poor per- formance, while angle classification performs significantly better but remains suboptimal. The final proposed proba- bility distribution fitting method surpasses the alternatives, achieving markedly superior performance.
7. Applications
7.1. Spatial Understanding
Number of Rendering Views We explore the effect of the number of images rendered pre-3D object in Tab. For a fair comparison, we train models converge for each set- ting. The results indicate that too few views fail to provide sufficient information about objects from different perspec- tives, while overly dense sampling results in redundant im- ages within the dataset, potentially hindering convergence. Empirically, rendering 40 views for each object achieves the best balance and yields the optimal results.
Orientation is a key attribution for accurately understanding
the spatial relations, as we highlighted in Sec. and Fig. 2. We find that using Grounded-Sae -SMde (Ren et al., 2024) and our Orient Anything to identify object position and orien- tation in images, and then conveying these spatial details in pure text to an LLM (Hurst et al., 2024), effectively ad- dresses orientation- based questions that confuse GPT-4o and Gemini- 1.5-pro, as shown in Tab. and Fig. 13-15. These results underscore the importance of our model in
8

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
enhancing spatial understanding.
Acknowledgements
This work was supported in part by National Natural Science Foundation of China under Grant No. No.624B2128, No.62222211 and No.U24A20326
7.2. Spatial Generation Scoring
As shown in Fig. 7, we empirically find that even leading image generation models, like DALL-E 3 (Betker et al., 2023) and FLUX (Labs, 2024), struggle to generate content that conforms to given object orientation spatial relation- ship conditions. Our model can help distinguish whether the generated image follows the given spatial condition, demonstrating its potential as a reward model to guide gen- erative models in adhering to the desired orientation- and perspective- spatial concepts.
References
Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., and Guibas, L. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer -ision-ECCV202D) 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 422- 440. Springer, 2020.
7.3. 3D Models Orientation Voting
Ahmadyan, A., Zhang, L., Ablavatski, A., Wei, J., and Grundmann, M. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7822-7831, 2021.
Many existing 3D data exhibit varied orientations, with some even tilted relative to the coordinate axes. As shown in Fig. I, our method achieves consistent orientation predic- tions across multi-view images, enabling robust voting for 3D assets' orientation. Accurately estimating the orienta- tion of 3D assets is valuable for further scaling up rendering images with orientation annotations or adjusting the poses of 3D assets to a desired direction.
Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. Scanqa: 3d question answering for spatial scene under- standing. In proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 19129- 19139, 2022.
8. Conclusion
Baruch, G., Chen, Z., Dehghan, A., Dimry, T., Feigin, Y., Fu, P., Gebauer, T., Joffe, B., Kurz, D., Schwartz, A., et al. Arkitscenes: A diverse real-world dataset for 3d in- door scene understanding using mobile rgb-d data, arXiv preprint arXiv:21 1.08897,.1.1111.111 2021.
In this paper, we present Orient Anything, practical ap- proach for estimating the orientation of open-worid objects from single images. We design an automated and reliable 3D object annotation and rendering pipeline, enabling the collection of large-scale images with accurate orientation annotations. To fully leverage the value of the new dataset, we design orientation probability distribution fitting task to learn robust orientation estimation. Additionally, we improve synthetic-to-real transfer performance by incorpo- rating real-world knowledge and minimizing the domain gap. As a result, Orient Anything achieves impressive zero- shot object orientation estimation in real-world images and demonstrates great potential as a foundational tool for en- abling applications such as complex spatial understanding and generation scoring.
Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Sci- ence. https://edn. openai. com/papers/dallee., pdf. 2(3): 8, 2023.
Brazil, G., Kumar, A., Straub, J., Ravi, N., Johnson, J., and Gkioxari, G. Omni3d: A large benchmark and model for 3d object detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13154-131 13164, 2023.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Impact Statement
Orient Anything provides a fundamental tool for estimat- ing the orientation of open-worle objects in images. This highly beneficial for high-dimensional applications such as controllable 3D/2D content generation, 3DD scene under- standing, and autonomous driving. In different application scenarios, it is essential to follow the corresponding usage guidelines to ensure its proper and ethical application, mini- mizing any potential risks.
Chen, D. Z., Chang, A. X., and Nießner, M. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E.,, Schmidt, L.., Ehsani, K., Kembhavi,

---

Orient Anything:
Learning Robust Object Orientation Estimation from Rendering 3D Models
A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13142- 13153, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T.,, Whitehead, S., Berg, A. A., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4025-4426, 2023.
Fan, Z., Pan, P., Wang, P., Jiang. Y., Xu, D., and Wang, Z. Pope: 6-dof promptable pose estimation of any object in any scene with one reference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7771 1-781, 2024.
Labs, B. Flux, 2024, URL black forest -ensst-aaae waassaaalaiiiaa
Li, L., H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L.,, Zhang, L., Hwang. J.-N., et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
Fellbaum, C. Wordnet. In Theory and applications of ontology: computer applications, pp. 231-243. Springer, 2010.
Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Goodwin, W., Vaze, S., Havoutis, I., and Posner, I. Zero- shot category-level object pose estimation. In European Conference on Computer Vision, pp. 516-532. Springer, 2022.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Góral, G., Ziarko, A., Nauman, M., and Wolezyk, M. Sceing through their eyes: Evaluating visual perspec- tive taking in vision language models. arXiv preprint arXiv:2409. 12969, 2024.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang. J., Jiang, Q., Li, C., Yang, J., Su, H., al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 054499, 2023.
Guan, J.. Hao, Y., Wu, Q. Li, S., and Fang, Y. A sur- vey of 6dof object pose estimation methods for different application scenarios. Sensors, 24(4):1076, 2024.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang. J., Jiang, Q., Li, C., Yang, J., Su, H., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision pp. 38-55. Springer, 2025.
Han, J., Ding, J., Li, J., and Xia, G.-S. Align deep fea- tures for oriented object detection. IEEE transactions on geoscience and remote sensing, 60:1-11, 2021.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Lowe, D. G. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60: 91 91-110, 2004.
Huang, Y., Li, Z., Chen, Z., Ren, Z., Lin, G., Morstatter, F., and Xu, Y. Orientdream: Streamlining text-to-3d gen- eration with explicit orientation control. arXiv preprint arXiv:2406. 10000, 2024.
Ma, w., Zhang, G., Liu, Q., Zeng, G., Kortylewski, A., Liu, Y., and Yuille, A. Imagenet3d: Towards general-purposs object-level 3d understanding. Advances in Neural Infor- mation Processing Systems, 37:96127-966 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-40 system card. arXiv preprint arXiv: 200:2240.21266 2024.
Ma, X., Yong, S., Zheng, Z., Li, Q. Liang, Y., Zhu, S.-C., and Huang, S. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07447,, 2022.
Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R. C.,, and Schindler, K. Repurposing diffusion-based image generators for monocular depth estimation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9492-9502, 2024.
Nguyen, N., Groueix, T., Ponimatkin, G., Hu, Y., Marlet, R., Salzmann, M., and Lepetit, V. Nope: Novel object pose estimation from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17923-17932, 2024.
10

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T. Tenenbaum, J. B., and Freeman, W. T. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2974-2983, 2018.
Peng. S., Liu, Y., Huang, Q.. Zhou, X., and Bao, H. Pynet: Pixel-wise voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4561-4570, 2019.
Sundermeyer, M., Marton, Z.-C.,, Durner, M., Brucker, M., and Triebel, Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the european conference computer vision (ECCV), pp. 699-715, 2018
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /2. 1805, 2023.
Rajič, F., Ke, L., Tai, Y.-W., Tang, C.-K., Danelljan, M., and Yu, F. Segment anything meets point tracking. arXiv preprint arXiv:2307.01199 2023.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1,5: Unlocking multimodal understand- ing across millions of tokens of context, arXiv preprint arXiv:2403.05530 2024.
Ravi, N., Gabeur, V., Hu, Y.TT., Hu, R., Ryali, C.,, Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.,, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000 00714, 2024.
Tulsiani, S. and Malik, J. Viewpoints and keypoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1510-1519, 2015.
Ren, T. Liu, S.. Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang. X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., and Zhang, I.. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Wang, Q. Chang, Y.-Y,, Cai, R., Li, Z., Hariharan, B., Holynski, A., and Snavely, N. Tracking everything ev- erywhere all once. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19795- 19806, 2023.
Schuhmann, C., Veneu, R., Beaumont, R., Kaczmarczyk,
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111 11.0114, 2021.
Wang. T., Mao, X., Zhu, C., Xu, R., Lyu, R., Li, P., Chen, X., Zhang, W., Chen, K., Xue, T., et al. Embodiedscan: A holistic multi-modaa 3d perception suite towards em- bodied ai. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19757- 19767, 2024.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 -55244, 2022.
Wei, Q. A., Ding, S., Park, J., Sajnani, R., Poulenard, A., Sridhar, S., and Guibas, L. Lego-net: Learning regular rearrangements of objects in rooms. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19037-19047, 2023.
Shi, Y., Wang, P., Ye, J., Long, M., Li, K., and Yang. X. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint ar0iv:2308. 165/2, 2023.
Wu, Z., Li, J., and Liu, C. K. Human-object inter- action from human-level instructions. arXiv preprint 2rXiv:2406.1788 2024.
Song, S., Lichtenberg, S. P., and Xiao, J. Sun rgb-d: rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Xiang, Y., Mottaghi, R., and Savarese, S. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision, pp. 75-82. IEEE, 2014.
Su, H., Qi, C. R., Li, Y., and Guibas, L. J. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE international conference on computer vision, pp. 2686-2694, 2015.
Xiang, Y., Kim, W., Chen, W., Ji, J., Choy, C., Su, H., Mot- taghi, R., Guibas, L.,, and Savarese, S. Objectnet3d:

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
large scale database for 3d object recognition. In Com- puter Vision-ECCV 20/6: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part VIIII 14, pp. 160-176. Springer, 2016.
Xiao, Y., Lepetit, V., and Marlet, R. Few-shot object de- tection and viewpoint estimation for objects in the wild. IEEE transactions pattern analysis and machine intel- ligence, 45(3):3090-31 3106, 2022.
Xie, X., Cheng, G., Wang. J., Yao, X., and Han, J. Oriented cnn for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3520- 3529, 2021.
Yang. L., Kang, B., Huang, Z. Zhao, Z., Xu, X., Feng, J., and Zhao, H. Depth anything v2. arXiv preprint 2rXiv:2406.094/ 2024.
Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T., Li, Y., Liu, S., et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1724-1732, 2024.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 633-641, 2017.
Figure 6. More visualization of images in the wild.
A. Detailed Results on COCO Benchmark
In Tab. 7, we provide the detailed horizontal direction recog- nition accuracy for each object category in COCO that annotated with front face and orientation.
Our model achieves excellent performance across most ob- ject categories with clear orientations, attaining an accuracy exceeding 80%. However, it performs relatively poorly in categories where the distinction between front and back is ambiguous or the objects are too small. Compared to previ- ous alternatives, Orient Anything achieves significantly bet- accuracy in most categories than the best results achieved by previous models.
B. More Visualizations of Images in The Wild
In Fig. 6, we present more visualizations of images from var- ious domains containing different objects. In these images, our model shows consistently accurate orientation predic- tion results, further highlighting the impressive zero-shot capability of our Orient Anything.
C. Visualization of Real-image Benchmarks
In Fig. 8, 9, 10, 11 and 12, we present the qualitative results on objects of SUN RGB-D (Song et al., 2015),
12

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
a). "A car is driving away at 30 degrees to the left.
b). "Charmander holding ice cream in its left hand."
20
Case (30" to the right)
Casc (30' the left)
Case (in the right hand)
Case (in the left hand)
Figure 7. Generated images with given textual prompt (left two from DALL-E (Betker al., 2023), right two from FLUX (Labs, 2024)).
Accurate orientation estimation helpful to confirm whether generated contents follow the given orientation perspective condition.
KITTI (Geiger al., 2012), nuScenes (Caesar al., 2020), Objectron (Ahmadyan et al., 2021) and ARK- itScenes (Baruch et al., 2021), respectively. Our model can robustly and accurately predict the object orientation in images of various sources and resolutions.
amples are provided in Fig. 13, 14, and 15.
Although this method has obvious disadvantages (innrrig depth and 3D) object relationships), it still performs much better than the Gemini-1.5 1.5-rro and GPT-4o.
The template description of the object relationship is:
D. Visualization of Ori-Bench
For OBJ1 located in [x1. with predicted azimuth angle و and OBJ2 located in [x2, y2),
All Ori-Bench samples, along with the responses from GPT- 4o, Gemini-1.5-pro, and Orient Anything+LLM, are in- cluded in the attached file. We visualize the three kinds of subtasks in Ori- Bench in Fig. 13, 14 and 15, respectively.
if < x2:
"From the perspective of viewer <OBJI> is the left and <OBJ2> is on the right of the view."
292.2.5 < & < 360 or 0 < & < 67.5%:
Our observations reveal that these questions, which are in- tuitive for humans, often confuse the state-of-the-art VLM models like GPT-4o and Gemini-1.5-pro This highlights the inherent limitations of existing approaches to understanding orientation. By utilizing the simple template to describe object orientations estimated by Orient Anything to LLM, we outperform alternative methods by a substantial margin.
"<OBJ2> is on the left of <OBJ1>."
if 112.5%:
"<OBJ2> is behind <OBJI>."
ff < 247.5":
"<OB]2>> is on the right of <OBJ1>."
i/247.5 < < 292.5
"<OBJ2>> is in front of <BJ1>."
ifxl > x2:
E. Orient Anything for Orientation Understanding
"From the perspective of viewer <OBJ2> is the left and <OBJI> is on the right of the view."
292. < & < 360' or 0 < & < 67.5%
In Section 7.1 of the main text, we briefly introduce the use of Orient Anything for solving orientation understanding problems, Here, we provide a detailed implementation.
"<OBJ2> is on the right of <OBJ1>."
if67.5" < E < 1/2.5%:
"<OBJ2> is in front of <OBJ1>"
ff < & < 247.5
For the open domain orientation understanding problem, we first use LLM to extract the object nouns in the question, then use Grounding-SAAe SAM (Ren et al., 2024; Liu et al., 2025) to determine the coordinates of each object, and use Orient Anything to predict the horizontal orientation of each ob- ject. We convert the detected spatial information into text descriptions with simple templates. For multiple objects, we use their coordinates to express their left-right relation- ship in the image. For each object, we only consider the azimuth angle and convert it into the horizontal 8-direction description. Finally, we provide these templated spatial descriptions, questions, and options in LLM. Practical ex-
"<OB,]>> is on the left of <OBJ1>"
2/247.5 < & < 292.5
" "<OBJ2> is behind <OBJ1>."
The template description of object direction is:
13

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
For OBJ with predicted azimuth angle ;p if 292.5" < & < 360 ()" < Φ < 22.5": "The <OBJ> is facing the viewer."
Accurate Orientation Angles Estimation: will ask you question about the content of the picture. Here is the question: <image> Align the front of the object towards the viewer. Rotate the object x degrees to right (i.e., clockwise from a top view), using a 360 per full circle unit system. Adjust the height of the viewer to form pitch Angle y with the object (same unit of degrees; y is positive if the viewer is looking down at the object, and is negative if the viewer is looking up at the object). Finally, the viewer rotated clockwise by an Angle (same unit of degrees) with the line connecting the viewer and the object the axis, and a negative z indicates counterclockwise rotation. Now, please directly predict the values of x, y. and z in float format.
22. < 6 < 67.5%:
"The <OBJ> facing the viewer and the left of the viewer."
6f 3 < 112.5%:
"The <OBJ> is facing to the left of the viewer."
112.5 < < 157.5%:
"The <OBJ> is facing away from the viewer and the left of the viewer."
157.5" < & 202.5%:
"The <OBJ> is facing away from the viewer." 202.5 < $ < 247.5%:
"The <OBJ> is facing away from the viewer and to the right of the viewer."
24 < ; 292.5%:
"The < <OBJ> > is facing to the right of the viewer."
292.55 <, S < 337.5%:
"The <OBJ> is facing the viewer and to the right
of the viewer." "
F. Prompts for VLMs
Question Answering for Ori-Bench and Orien- tation Recognition: I will ask you single-choice question about the content of the picture. Here question: <image> <question>> <options>.
Orientation Annotating for Orthogonal Render- ing views: I'm going to show four images of the same object from four viewpoints turn and label them 'A.' 'B.' 'C.' 'D.' Four options, Option 'E.' "No front face or More than One front Face". De- cide whether has a front and if yes, which one is the front of the object after the presentation. Note that: If the object is gun, bow and arrow, etc., please use the muzzle of the gun as the front. Stick tools and weapons such as swords, axes, knives, and wrenches are considered have front. If you cannot decide there is more than one front, you should choose 'E'' A. <image viewA> B. B.<image viewB> C. C.<image viewC> D.<image viewD> E. front face.
14

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Cube RCNN
Orient Anything
Category
Cuhe RCNN
Gemini
GPT-4o
Orient Anything (VIT-L)
hed
75%
15%
40%
100%((25%))
monitor
35%
50%
50%
(00%(()50%)
oven
50%
10%
65%
(+35())))%)
teddy
(005())55))
motorbike
5%
20%
40%
95%(+5533)
parking meter
40%
55%
65%
95%(++30%)
laptop
65%
45%
50%
95%((+)0%%
sheep
15%
45%
45%
90%(++459)
elephant
5%
30%
55%
sofa
5%
25%
50%
90%((+409)
20%
50%
(055))
cell phone
35%
75%
80%
(0%(+-109)
microwave
35%
25%
50%
90%(++009)
clock
20%
45%
60%
bus
10%
20%
40%
85 (5455)))))
traffic light
0%
35%
50%
85 (55
stop sign
0%
70%
75%
85 (+108))
20%
20%
(6555))
beur
5%
30%
40%
zehra
5%
30%
50%
85 (+55)))
sink
0%
0%
30%
85 (55
cat
20%
43%
60%
(0%9)+)
dog
10%
35%
60%
80% (+206))
horse
50%
35%
(+30
chair
10%
20%
30%
(0%())
book
45%
80%
45%
(0%6(+0%)
10%
40%
45%
(500)))))
truck
15%
35%
60%
(5
cow
20%
40%
40%
(+35))))
HEL
HEL
person
5%
35%
40%
70% (+30%))
acroplanc
15%
20%
60%
(+
refrigerator
20%
25%
55%
70%(++199)
bird
10%
25%
60%
(5%))
giraffe
15%
30%
55%
65 (50%)+
train
30%
15%
60%
(95)))
fire hydrant
20%
20%
30%
55 (555))))
boat
10%
25%
45%
(+56))
backpack
50%
50%(-159)
15%
0%
0%
50%(++351)
kite
5%
40%
60%
45%K 1593)
hair drier
0%
20%
55%
45%%--10%)
bicycle
5%
30%
30%
40%(++00%)
tonster
50%
25%
30%
40%(-109%)
remote
10%
5%
10%
596(-59)
keyboard
0%%
(69(-105)
Table Detailed horizontal direction recognition accuracy for each object category in COCO that is annotated with front face and orientation. The differences between Orient Anything and the best results achieved by other alternative methods are also provided.
Figure 8. Qualitative results COCO
15

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Ground Truth
Cube RCNN
Orient Anything
Figure 9, Qualitative results on SUN RGB-D.
Ground Truth
Cube RCNN
Orient Anything
Ground Truth
Cube RCNN
Orient Anything
Figure 10. Qualitative results on KITTI and nuScenes.
Ground Truth
Cube RCNN
Orient Anything
Figure 12. Qualitative results on ARKitScenes.
Figure 11. Qualitative results on Objectron.
16

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Which side is animal facing the picture?
Which side the animal facing the picture?
A. A.Faa Facing the viewer
A. Facing the viewer
Facing mway
viewer
Facing away from the viewer
C. Facing left
C. Facing left
Facing right
D. Facing right
GP:--to: The bird
right the
its
blid's head
Gemini-1 Sepmo: hind's head pointing lowords left side the imgge.
Genini-I.S-pr hichs bak sre pointing right side of image. tail feathers point left.
Anything-LA viewer implies the viower'
facing
Ang
facing
right viewer, mosting towards right.
Which orientation the horse flacin Facing lhe viewer while slightly to the right
facing A. Facing the viewer while slightly to the right
Which orientation
Facing the viewer while slighll the left
B. Facing the viewer while slightly the left
Facing may from viewer while slightly to right
C. Facing away from the viewer while slighly to the right
Facing away from the viewer while slightly to the left
D). Facing away
while siihtly the
diresion sway
GFT-4:: hoese's samed
body
GPT-4a: U. The homee's head is pointed the the image.
Gemisi-L.S-pre hunch
lead
Gemini-
head tumedd orienteal slightly
slighlly
facing away
Spotial Contest: house is facing away froms the viover and theriglte
the stowe.
Special Contest: hore fixinn paaay fhondh viewer and to loftoftte venwer
Orleat Anything - LLLE C: The question stres that the homs facing awy from the viewer and to the right of the veewer. This
Octent Anything- LLME D. The hosse is described as ftzling away from the viewer and the left of the viewer, This aligns with oprion D.
dinetly matchas the douription uption
Which side is the girl facing in the picture?
Which side is the girl facing in the picture?
A. Facing the Facing away camera
Facing the
Facing mway from the camera
Facing lef)
Facing left
Facing right
Facing right
GPI-4e: D. The gir is rumning and her body anglcal trwords unc vile.
GPT-4o: D. The girl oppears moving towards the right side of the inag::
1.nnint-1---prr:: pirts
indicring
rimning
body
Gemini-LL5 rr: right
image.
body
Sparial Content: and facine vs the loft the viensrr Orient Anything M. C: Since the girl facing to the left of the viewer she tumed her left from the pesspective of the camaniiioow..
Spurtalt Contents parla fieing therishraf' the virwer. Orient Anything U.I. MMM D) The gir is facing the right of the viewer, mean ing diom the viewer's pesspective, shee tumed to her own left site. option D.
which
Figure 13, Visualization and qualitative comparison the Object Direction Recognition task of bri-Bench.
17

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
Which paw is the dog raising?
Which pow is the dog mising?? its left front pow
A its left front paw
Buits right front paw
right front gans
dog's pimpartio intiaiin righ
viiibl pan aspeas the sile
GPT-40: The paw shown the
inage.
Gentni-J, J.re A. The dog's right pow would be or the left side of the inage from viower's propective The dog miking the yow that pas.
Gemini 1.5 pre: A. The dogss head shee sightle to the rishs, ssdicsting Buit die pas bekua nasedd is left aide Ala, consideria the cumorikk perpertivv, the pone viusdly chour vicker would the left paw.
appeas right
the image,
Spetial Contests dog facing riowss 2 Freme persaaiine
vienen, deg aus the lelt pnd fio gfatof viow 3. Frose the
Log
sinaee pare anf dogg the of vila
perppcctioa criding
Cannewt: deg ln builiggfe cinwer 2. thome the curperctionn
lookin gav
imghes
perepostiz, perpective, right
Anything!
facing
would
Which the man mising?
Which hand the man raising?
A his left hand
A his left hand
Bihis right hand
B.his right
right
indiowing
micng appears right
hend
image, comeeporuk
S.ger
makng
siald
oft
This hand.
Sperial Canticsts
viewer Forntlic perpest perpoctiveaf
Spetial Contente Lusas laving viower Eorperpen
perpective man
of raar.
mrmaaa handlion thelahhl
Orken LLM It From the man's perpective, the hand the riats las right hand. Since vizver seas thz hand on the
Orfeat Asythigg-E.IMM M.I.M.g-...M... A, Fiem the perpoctive of man, the hand oz kell being masiooed. Sance the man Saita the viewer and his left hond on ttz viewers right miaes he mexing los left hand.
let, the mun would mecing his right loest.
ECaide
$
ELEPHANT
log elephant
log theelephant closes camero??
A Its front
its left front
Butt right front
right from
right rear
D. Dight rear
GPT--4r The skptuer's left from log sppois prominmdty Serogmaand
GPT-kk: The ilsemiiion shows kst free by ae mod yourinuss
Gemini-1 Segre it. The diephants right froer lig cioems to cunnera. We can tell becuse the persective the inaae aukes appex loges and more prouinect dhnn the let frout lyg
Comini-1. The clephanss let thore log portiely abecard by Is tunk and head, making the right froor lyp dlarly the dossit to the cureaa
Sparial Conteot I chepheetis linn the diose and the iin ont cower. Orlent Aaything LLM. A. The dephars a faiins dhe viewes, and dt loft sade is dlio deses to the viewe. Theetoe the los aloest to the canaaa is is let Bout lg. A
Sparial Conmecti aheyhen fuding two novr and thet oo voum Orken Anythina LLM. b. Since the elephurr & facng the viower and to the cain the view es is rall side is dose the caner. Auoos the giren options io riglt. Bost log the dkoess.
Figure 14, Visualization and qualitative comparison on the Spatial Part Reasoning task of Ori- Bench.
18

---

Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models
From the dog's perspective, is the cat on its left or right?
From the dog's perspective, is the cat on left or right? Lefe B.Right
Ae Right
GPT-4o: The os is pookioed sloser the window and fixing the left side
GPT-4s; B. The puclioned en the right side of the viewed from dog's parspective
dog.
Gemini-I, S-pre: B. The day is looking at the and the lorking
Gamini-1 Sugen: B. The dog is faeing the sliding doce. The cal is on the other sode of the glass. From the dog's perspettie. the cat its right.
down. To the dog's perspective, the cat on right.
Spottal Context: deg
viewer and letbof viewes
Spacial Contests
the viewer. 2. facinu uway thepempeeion
facing way dogitin lehufdog From
vion.
Orlent Anything LLM, the
the viewer, anple. cat, viewers right would then logically be on the dog's
Orfent Anything-Lin LLM. perspctive, right
foxing companed stewerss persective, on the right from the dog's perspectiee
from the
looking perspeetive, magino niming
dog
passectivee Therefore,
the girl run into the lamp post if walks forward" Yes
Will the girl run into the lemp post if she walks forward?
A. Yes
B.
GPT-4e: The gir is stenling close the heep and ficing walks directy forward, she litely
GPT-4s: B. The is standingg bevide the net cinatly of
Gemini-I. p.rpp: B. The girl is wonding neat ts the lump posi, facing the camera If walks forward, siee will walk away from the tamp post, down the street
Gemini-1 S.pen: B. The girl starding nost the hamp pasi, facing the steet If walks forward, she will walking away from the lamp post, down the strett
Spatial Content: 1 gil foiin the left ohe vieves 2. Froon the penorctive ricoer gill
Spetial Contesti 1. girl facing to the of viewer Fromite perppccive pirppeecivv
lame therishtef
noo
pos therightofdh
lomp pos
Any LLLL) post behind Zer. So, she walks Rorward, she will not inoo
perspestive
kamp
Oclent Anything- LLM Since lamp post perspective, walking forward meats moving away trom the lamp post.
If someone walks out the door, on which side would the tree be relarive to them?
If someone walks out the door, on which side would the tree be rolative to them?
A side
B. riightt side
B.right side
The
when louse
Gemini-1 1-ppp: B. The tree is located the right side the house, observed from frunt door.
1.Siaa--.Spprr A. The on the left sitl the door foon the someons exiing the homse
Sparial Contect: 1. house foiing vewer the right of virwor perspectivee viewer. house left
Spotial Contect house friiigg the slower From the perputtive the vicw.
rigas
viewer tre
house
perpuction LLM:
thenighoot
Anything-LMM
person
out
facing viewes, tree is on right from tte house's perspective, then someone walking out the door will luve the tre on faii rialn siáe. B
have the perspctive of house. Since the tree on the left
luuse from this penpactive, the tree bill on the left sade.
Figure 15, Visualization and qualitative comparison on the Spatial Relation Reasoning task Ori-Bench.
19


Reviews:
[{"review_id": "zvQgtDvhNb", "reviewer": "Reviewer_ev5k", "summary": "This paper proposes Orient Anything, a method that obtains orientation through 3D assets and distilled VLM annotation. Although this paper is somewhat overclaimed, it is pioneering.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review for recognizing the significance of our paper and invaluable suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n\n---\n\n***W1&W2: Scaling up Orient Anything.***\n\nThank you for your suggestion. The SoFar dataset is really helpful, and we will incorporate it to further scale up.\n \n\nIn fact, we are also actively working on expanding our training data to the scale of 10+ million to further improve Orient Anything. Recent breakthroughs in open-source 3D asset generation models (e.g., Hunyuan-3D [1], TRELLIS [2]) have demonstrated impressive results, with outputs now reaching sufficiently high quality. On the other hand, as discussed in Section 7.3 of our manuscript, the initial Orient Anything model can serve as an annotator to robustly label the orientation of 3D assets through multi-view voting. This voting mechanism helps reduce prediction errors, achieving robustness beyond the model's original capabilities.\n\n\n\nThe above two observations show the potential to freely increase annotated 3D assets to any scale. In <https://anonymous.4open.science/r/visualization-B728/Synthesized_Assets_and_Voting_Annotation/>, we showcase examples of synthesized 3D assets and the voting-based orientation annotations using Orient Anything. We hope this further observation can address your concerns regarding scaling up.\n\n[1] Zhao Z, Lai Z, Lin Q, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation[J]. arXiv preprint arXiv:2501.12202, 2025.\n\n[2] Xiang J, Lv Z, Xu S, et al. Structured 3d latents for scalable and versatile 3d generation[J]. arXiv preprint arXiv:2412.01506, 2024.\n\n---\n\n \n\n***Q1: Generalization to object position and size.***\n\nIn <https://anonymous.4open.science/r/visualization-B728/Corner_TooBig_TooSmall_Case/>, we provide visualizations of objects that are too large, too small, or on the corner. Overall, our model demonstrates generalization to these scenarios.\n\n---\n\n***Q2: Items do not have a clear “front side”.***\n\nDuring annotating, 3D assets lacking a clear \"front side\" are explicitly labeled as having no front side, as shown in <https://anonymous.4open.science/r/visualization-B728/Training_Samples/>. We identify these cases using two methods: symmetry detection and VLM-based semantic understanding (as illustrated in Figure 3b, \"Orientation Annotating\").\n\nDuring training, Orient Anything learns to predict a confidence score indicating the likelihood of an object having a clear front side.\n\nDuring inference, Orient Anything can predict low confidence scores for objects lacking a clear \"front side\" to reflect their ambiguous orientation. Actually, the \"Judgment\" column in Table 3 is the accuracy of judging whether an object has a distinguishable front face.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The paper is meaningful, most previous academic research has focused on location or spatial relationships, especially robotic tasks.\n\nBut I think this paper is over claims; the author only renders images on 80K 3D objects to generate orientations, which is far fewer than the sample size used in other xxx-Anything works such as Segment Anything, Depth Anything, etc., and even less than the COCO dataset. I am skeptical of its generalizability. In addition, the zero-shot performance in Table 3 appears to be not ideal.", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The concept of orientation comes from object pose. This article directly obtains the orientation in the camera coordinate system from 2D images and does not depend on any template.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "This paper has many experiments and visualizations, and the combination of the model and the LLM has impressed me with its ability to enhance orientation understanding.", "other_comments_or_suggestions": "Recently, SoFar used a similar approach for orientation learning and understanding, and has been filtered and trained on a larger dataset (the full set of Objaverse1.0). I suggest that the authors use similar methods and data to scale up Orient Anything.\n\n\nSoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation", "questions_for_authors": "1. Does Orientation-Anything generalize the position (on the corner) and size (too large or too small) of the object in the image?\n2. Many items do not have a clear \"front side\", such as apples, footballs, light bulbs, etc., how should the orientation of such objects be defined?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jS72hxqW53", "reviewer": "Reviewer_MxHL", "summary": "The paper introduces Orient Anything, a foundation model for zero-shot object orientation estimation. The key contributions include: 1) Leveraging 3D models and VLMs to annotate front faces, generating 2M synthetic images with orientation labels; 2) Modeling orientation as Gaussian distributions over angles (azimuth, polar, rotation) to improve training stability; 3) Using DINOv2 initialization and data augmentation (random cropping) to bridge domain gaps.; 4) A VQA benchmark revealing VLMs’ limitations in orientation understanding.\nResults show state-of-the-art zero-shot performance on real-world datasets (e.g., SUN RGB-D, KITTI) and significant improvements over VLMs (GPT-4o, Gemini) on orientation-related tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review for recognizing the significance of our paper and invaluable suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n\n---\n\n***W1&Q2&Q3: Limited ablation studies for key components: 1. distribution fitting, 2. augmentation, 3. random cropping, 4. sensitivity for variances (σ_θ, σ_φ, σ_δ).***\n\n \n\nIn Table 5 of the current manuscript, we compare **different learning objectives** (continuous regression vs. discrete classification vs. distribution fitting), inference augmentations (box vs. mask), and **training augmentations** (with vs. without random cropping).\n\n \n\nIn Figure 5, we analyze the impact of **different selections for (σ_θ, σ_φ, σ_δ)**. Generally, our method is not sensitive to the hyper-parameter.\n\n \n\nWe respectfully inquire whether you are referring to other kinds of the ablation study. If so, could you kindly provide more specific settings, and we will make the necessary additions?\n\n---\n\n***W2: Clear real-world evaluation protocol.***\n\nCurrently, the 6 benchmarks in Table 2 are evaluated on real-world datasets and reported with quantitative 3D orientation error.\n\n \n\nMoreover, we have also tested on the latest and largest 3D orientation benchmark, ImageNet3D [1], which covers 200 object categories. The Acc@30° results are as follows:\n\n|Setting | Model                             | Avg. | Electronics | Furniture | Household | Music | Sports | Vehicles | Work |\n|---| --------------------------------- | ---- | ----------- | --------- | --------- | ----- | ------ | -------- | ---- |\n|Zero-shot| ImageNet3D-ResNet50   | 37.1 | 30.1        | 35.6      | 28.1      | 11.8  | **51.7** | 36.7     | **40.9** |\n|| Orient Anything-B     | **48.5** | **61.0** | **66.8** | **37.9** | **27.3** | 25.6 | **70.8** | 33.4 |\n|Fine-tuning| ImageNet3D-ResNet50 | 53.6 | 49.2        | 52.4      | 45.8      | 26.0  | 65.2   | 56.5     | 58.5 |\n|| ImageNet3D-DINOv2-B | 64.0 | 75.3        | 47.9      | 32.9      | 23.5  | **74.7** | 38.1     | **64** |\n|| Orient Anything-B     | **71.3** | **77.6**    | **89.7**  | **64.4**  | **54.4** | 47.6   | **87.4** | 61.2 |\n\nNote that we couldn't find detailed definitions for the major categories like *Electronics*, *Furniture*, and *Household* in ImageNet3D, so we used GPT to map its 200 categories into the 7 general categories. As a result, we used GPT to map its 200 categories into 7 broader ones. Therefore, the comparison results for each general category may vary, and the average score provides a more meaningful comparison.\n\n[1] Ma W, Zhang G, Liu Q, et al. ImageNet3d: Towards general-purpose object-level 3d understanding[J].  NIPS 2024\n\n---\n\n \n\n**W3: Missing discussion.**\n\nWe will include more related works on synthetic-data approaches and self-supervised orientation methods. We also respectfully inquire if you could provide more specific related works.\n\n---\n\n***Q1: Mapping from predicted angles to directions in COCO evaluation.***\n\nAs discussed in Section 6.2, the COCO direction focuses only on the horizontal plane (e.g., azimuth angle). Specifically, we simply map the predicted azimuth angle (0-360°) to the 8 directions with 45-degree intervals. For example, 0±22.5° corresponds to the front, 45±22.5° to front-left, 90±22.5° to left, and so on.\n\n---\n\n***Q4: Extend to video clip.***\n\nSimply performing per-frame predictions on video data, followed by cross-frame smoothing through simple averaging, can yield relatively consistent and accurate orientation estimation. Some examples are provided in <https://anonymous.4open.science/r/visualization-B728/Video_Cases/>.", "claims_and_evidence": "Basically yes. 2M images from 55K 3D models across 7,204 categories (vs. 100 in ObjectNet3D). But there's no ablation studies for key components (distribution fitting, augmentation). COCO evaluation’s 8-direction simplification lacks clarity on mapping from 3D angles.", "methods_and_evaluation_criteria": "Synthesizing data via 3D rendering and distribution-based training is sensible. Probability distributions effectively handle angle periodicity. Real-world benchmarks (SUN RGB-D, KITTI) are appropriate but compared unfairly to supervised models.", "theoretical_claims": "No theoretical proofs; methods are empirically validated.", "experimental_designs_or_analyses": "​COCO Evaluation uses a simplified 8-direction task, which may not fully reflect 3D orientation. Mapping from predicted angles to directions needs clarification. Manual setting of variances (σ_θ, σ_φ, σ_δ) lacks sensitivity analysis. Impact of random cropping is asserted but not quantified.", "relation_to_broader_scientific_literature": "Well-situated against 6DoF pose estimation and viewpoint detection. Connects to VLMs’ limitations via Ori-Bench. Missing discussion of recent synthetic-data approaches or self-supervised orientation methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Novel use of synthetic data, Ori-Bench benchmark, strong empirical results.\n\nCons: Sometimes the results are not good in my experiments of running the provided code and checkpoint. Limited ablation studies, unclear real-world evaluation protocol.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How are 3D angles (θ, φ, δ) mapped to 8 horizontal directions in COCO evaluation? Could this simplification misrepresent orientation?\n- What is the quantitative effect of data augmentation on synthetic-to-real transfer?\n- How were σ_θ, σ_φ, σ_δ chosen? Was sensitivity analysis performed?\n- How to extend the proposed method to a video clip, where the object's orientation needs to be estimated in a temporal manner?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mW5WVK1q60", "reviewer": "Reviewer_dtES", "summary": "The paper introduces Orient Anything, a foundational model designed for zero-shot estimation of object orientation from monocular images. Due to the scarcity of orientation annotations for open-world objects, the authors develop an automated 3D object orientation annotation pipeline that effectively utilizes the extensive resources of 3D models to create a diverse dataset of 2 million images with precise orientation labels. To enhance training stability and improve generalization, the authors introduce a robust training objective that models the 3D orientation as a probability distribution. Additionally, they propose several strategies to improve synthetic-to-real transfer, achieving state-of-the-art accuracy in orientation estimation for both rendered and real images. Experimental results demonstrate the superiority of the proposed method and highlight the significant potential of Orient Anything for high-level applications, such as enhancing spatial understanding and scoring spatial generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review for recognizing the significance of our paper and invaluable suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n\n---\n\n***Q1: Improvement of \"Orient Anything+LLM\".***\n\n\"Orient Anything + LLM\" is designed to demonstrate the accuracy of Orient Anything in real-world applications and highlight the importance of orientation perception for understanding spatial relationships. Since we use a text-based LLM that cannot perceive visual content, it relies on Orient Anything and the object detection model to understand spatial relationships in images. The improvements indeed come from the accuracy of these models.\n\n---\n\n***Q2: Example shown in Figure2.***\n\nFirst, we conducted the test in November 2024, and subsequent version updates of GPT-4o may result in different results.\n\nSecond, we believe the phrase \"In Falcon's view (or perspective)\" provides a coordinate reference, making both our question and your Q2 unambiguous. Adding a prefix question (e.g., \"Does Falcon face me?\") may explicitly prompt the model to transform the perspective first, which should be a natural reasoning step in a single Q&A. Breaking down a question into sub-questions natively lowers its difficulty, so comparing direct answers with manually split versions may not be entirely fair.\n\n---\n\n***W1: Increasing the complexity of the question.***\n\nWhen constructing Ori-Bench, we created different subsets for distinct purposes. The basic \"object direction\" subset directly tasks the model with recognizing the orientation of individual objects, which aligns with the capabilities of Orient Anything. The more advanced \"spatial part\" subset requires understanding the pose of individual objects. Finally, the \"spatial relation\" subset involves open-domain questions that require further reasoning to solve.\n\n\n\nWe believe these three progressive subsets offer a more comprehensive evaluation of a model's ability to address orientation-related questions. The most challenging \"spatial relation\" subset contains the most test samples, and improvements in this subset highlight the importance of understanding orientation when handling high-level spatial reasoning tasks.\n\n \n\nThank you very much for your suggestion. In future updates, we will include more challenging questions and try to distill Orient Anything's knowledge into a VLM, thereby inherently integrating object orientation understanding capabilities.\n\n---\n\n***W2: Discussion about “Synthetic-to-Real Transferring”.***\n\nSeveral works have discussed the relationship between pre-trained models and synthetic-to-real transfer. Marigold[1] transforms Stable Diffusion into a depth diffusion model using only synthetic data. Depth Anything v2[2] provides a more systematic analysis of how different pre-trained models impact \"Synthetic-to-Real Transfer.\"\n\n\n\nAdditionally, while the crop operation is a standard processing technique, in our case, it is specifically aligned with the goal of transferring from (complete) synthetic training objects to (often occluded) real objects.\n\n[1] Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation. CVPR 2024\n\n[2] Yang L, Kang B, Huang Z, et al. Depth anything v2. NIPS 2024\n\n\n\n---\n\n***W3: Speed and cost of Orient Anything.***\n\nWe only add four very lightweight MLP prediction heads (about 1M parameter) to the standard DINOv2 encoder. The parameter number remains nearly identical to the original DINOv2.\n\n|              | DINOv2 Encoder | Project Heads |\n| ------------ | -------- | ------------ |\n| Orient-Anything-S | 22.06M   | 1.17M        |\n| Orient-Anything2-ViT-B | 86.58M   | 1.51M        |\n| Orient-Anything-ViT-L | 304.37M  | 1.74M        |\n\n---\n\n***W4: Difference and Advantage over 6D pose estimation method (e.g., FoundationPose).***\n\n**Difference:** Traditional 6D pose estimation methods focus on relative orientation to a reference frame or template 3D model, while Orient Anything focuses on semantic orientation (e.g., the semantic “front face” of an object) without any reference.\n\n**Advantage:** Orient Anything does not require multi-view reference images or a ground-truth 3D model during inference. The predicted orientation is inherently linked to the object's semantic front, enabling broader applications, such as enhancing spatial understanding in VLMs and image generation.\n\n ---\n\n**Other: Extend to video.**\n\nPlease refer to the response to Reviewer MxHL’s Q4.", "claims_and_evidence": "The claims made in the submission are well supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method holds significant value for a variety of applications, including understanding spatial information from images, verifying the spatial accuracy of generated images, and more.\n\nMy main concern is the weak novelty of the method itself. Most of the designs are common in computer vision.\n\nNevertheless, I think the contribution of this article in terms of dataset and foundation model is still meaningful.", "theoretical_claims": "I have check the correctness of equations(1-4) in section 5.1 and have no issues about them.", "experimental_designs_or_analyses": "The experimental design and analysis are generally sound.\nTable 2&3 show the superiority of the proposed method on both in-domain and out-of-domain datasets.\nTable 4&5 fully verify the key designs of the proposed method.\n\nBesides, I still have some questions:\n1. In figures 13 and 14, it appears that the input spatial context provided for 'Orient Anything+LLM' already contains the answer to the question. This raises a question about whether it is valid to assess the LLM's response in this scenario. Does the improvement of 'Orient Anything+LLM' shown in Table 1 primarily rely on the accuracy of Orient Anything itself?\n2. The example shown in figure2 is inappropriate. The author does not provide a clear coordinate reference for LLM, which makes the answer ambiguous. After trying the following two questions, GPT-4o got the correct answer:\nQ1: Does Falcon face to me? & Q2: So, in Falcon's view, where is Captain America.", "relation_to_broader_scientific_literature": "The model proposed in this paper can evaluate the spatial correctness of the results of the generative model. As discussed in \"Towards Foundation Models for 3D Vision: How Close Are We?\", state-of-the-art Vision-Language Models still perform poorly for spatial understanding, which is consistent with the findings of this paper.", "essential_references_not_discussed": "Essential References have been well discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The ablation experiments in this paper are very sufficient. They fully verify the main design and contribution of this paper.\n2. This paper is well-written and easy to follow.\n3. The created large dataset is of great significance to the development of the community.\n\nMain weaknesses:\n1. The experiments and analysis on LLMs are biased as discussed in 'Experimental Designs Or Analyses'. The authors should provide a more comprehensive comparison and analysis. For example, increasing the complexity of the question to weaken its connection with the given spatial context.\n2. The use of 'Synthetic-to-Real Transferring' in the paper lacks rigor. Firstly, employing a better pre-trained model can enhance the model's performance on its own, which is not directly related to synthetic-to-real transfer. Secondly, the crop operation is a standard procedure in deep learning.\n3. What is the speed and cost of the method?\n4. Compared with the 6D pose estimation methods (e.g., FoundationPose), does the proposed method have more advantages in orientation estimation?", "other_comments_or_suggestions": "Can the method proposed in this paper be applied to video data? Additionally, is it capable of achieving stable orientation estimation results from video content?", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dMAqTfS9fN", "reviewer": "Reviewer_83jQ", "summary": "This paper proposes orient anything, a foundation model for predicting object orientation from a single image in an zero-shot manner. The key of the paper is curating a large-scale dataset for the orientation estimation task, which is rendered from Objaverse including 2M images. The authors propose to deal with the ambiguity of low-dimensional poses by using probabilistic distribution over object azimuth, elevation/polar angle, and camera rotation. The proposed method is evaluated on both synthetic images (on orientation error) and real images (on recognition tasks).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and suggestions. Below we respond to the comments in **Weaknesses (W)** and **Questions (Q)**.\n\n---\n\n***W1.1 & Q1.1: Quantitative orientation error on real images.***\n\nCurrently, the 6 benchmarks in Table 2 are evaluated on real-world data with quantitative 3D orientation error.\n\n \n\nMoreover, we have also tested on the latest and largest object 3D orientation benchmark, ImageNet3D [1], which covers 200 object categories. The Acc@30° results are as follows:\n|Setting | Model                             | Avg. | Electronics | Furniture | Household | Music | Sports | Vehicles | Work |\n|---| --------------------------------- | ---- | ----------- | --------- | --------- | ----- | ------ | -------- | ---- |\n|Zero-shot| ImageNet3D-ResNet50   | 37.1 | 30.1        | 35.6      | 28.1      | 11.8  | **51.7** | 36.7     | **40.9** |\n|| Orient Anything-B     | **48.5** | **61.0**    | **66.8**  | **37.9**  | **27.3** | 25.6     | **70.8** | 33.4     |\n|Fine-tuning| ImageNet3D-ResNet50 | 53.6 | 49.2        | 52.4      | 45.8      | 26.0  | 65.2   | 56.5     | 58.5 |\n|| ImageNet3D-DINOv2-B | 64.0 | 75.3        | 47.9      | 32.9      | 23.5  | **74.7** | 38.1     | **64** |\n|| Orient Anything-B     | **71.3** | **77.6**    | **89.7**  | **64.4**  | **54.4** | 47.6   | **87.4** | 61.2 |\n\nNote that we couldn't find detailed definitions for the major categories like *Electronics*, *Furniture*, and *Household* in ImageNet3D, so we used GPT to map its 200 categories into the 7 general categories. As a result, we used GPT to map its 200 categories into 7 broader ones. Therefore, the comparison results for each general category may vary, and the average score provides a more meaningful comparison.\n\n[1] Ma W, Zhang G, Liu Q, et al. Imagenet3d: Towards general-purpose object-level 3d understanding. NeurIPS 2024.\n\n---\n\n***W1.2: Generalization to real and even in-the-wild images.***\n\nWe provide numerous visualizations on real and in-the-wild images in both our main text and supplementary materials. Furthermore, in response to your W1.1 & Q1.1, we present the results on the current largest 3D orientation estimation benchmark, ImageNet3D, which also demonstrates strong generalization to these real-world scenarios.\n\n---\n\n**W2.1 & Q4.1: Study on the dataset quality.**\n\nIn <https://anonymous.4open.science/r/visualization-B728/Training_Samples/>, we provide lots of visualization cases from our curated training dataset, showcasing both high-quality images and accurate orientation annotations.\n\n---\n\n***W2.2 & Q4.2: Influence of training data.***\n\nWe train ViT-B version models using different ratios of data. Below are the results:\n\n| Data Ratio | COCO(Acc) | ImageNet3D(Acc@30°) |\n| ---------- | --------- | ------------------- |\n| 25%        | 63.55     | 45.47               |\n| 50%        | 65.47     | 44.08               |\n| 75%        | 66.82     | 47.65               |\n| 100%       | **69.85** | **48.52**           |\n\n ---\n\n***Q1.2 Difference and advantage to object 6D pose estimation.***\n\n**Difference:** Traditional 6D pose estimation methods focus on relative orientation to a reference frame or template 3D model, while Orient Anything focuses on semantic orientation (e.g., the semantic “front face” of an object) without any reference. Therefore, the previous benchmark for object pose estimation is not suitable for our task.\n\n \n\n**Importance of no reference mesh/image:** Monocular images are the most accessible and widely used form of visual input, and in many scenarios, references for the desired object are often unavailable. By not relying on reference meshes or images, Orient Anything enables broader applications, such as solving spatial reasoning questions and evaluating whether the generated image adheres to the desired spatial relationships, as discussed in Section 7.\n\n\n\n---\n\n***Q2 Generalization to unseen categories.***\n\nIn response to your W1.1 & Q1.1, we further provide evaluation results on the current largest single-view orientation estimation benchmark, ImageNet3D, which significantly surpasses existing methods and demonstrates strong generalization to real images and various categories.\n\nAdditionally, in response to Reviewer ev5K’s W1 & Q1, we discussed how to further scale the annotated data and expand the covered categories through synthetic 3D assets and voting-based annotation strategy.\n\n---\n\n***Q3 Why trained on synthetic data can generalized to real data.***\n\nThe synthetic-to-real generalization ability is mainly obtained through the task-agnostic pre-trained model that is trained on massive real images. The similar idea has been discussed and validated in Marigold [2] and Depth Anything V2 [3]. For further discussions, please refer to the response to Reviewer dtES’s W3.\n\n[2] Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation. CVPR 2024\n\n[3] Yang L, Kang B, Huang Z, et al. Depth anything v2. NIPS 2024", "claims_and_evidence": "The authors claimed this paper to be the first foundation model for object orientation estimation. While it seems interesting, I am worried about the evaluation protocol on real data, where the orientation error is not studied.", "methods_and_evaluation_criteria": "The proposed methods include two parts: 1) data curation and 2) training an orientation prediction model. The data curation method includes filtering non-axis-aligned objects, identifying ambiguous symmetry objects, and rendering synthetic data. The orientation prediction model is based on a ViT encoder with several decoders for predicting the probabilistic distribution of rotation angles.\n\nThe trained model is evaluated on both real and synthetic data. On synthetic data, it is evaluated mainly with orientation error; while on real data, due to the lack of pose annotations, it is evaluated with higher-level recognition tasks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I am not fully convinced with the evaluations, especially the lack of results on orientation error on real images. Showing the recognition task performance is not enough for convincing me that this is an foundation model. And I am not very sure how the method trained on synthetic data with white background can generalize to real and even in-the-wild images.\n\nBesides, one of the biggest contribution of this paper is the curated training data. However, there is no study on the dataset quality, even no visualization on how the dataset look like. Besides, I think it would be better if the authors perform an experiment on using different ratio of the curated dataset for training and explore whether there can be any scaling effect.", "relation_to_broader_scientific_literature": "The posed task can be useful for object 6D pose estimation and may have potential to benefit robotic applications.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Please see comments above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Why not including quantitative results on real images? The authors claim that there are no data for performing evaluation. However, I don't think this is true. There are bunch of object pose estimation benchmarks that can be used for evaluation. Besides, the object 6D pose estimation works should also be discussed in details -- how the orientation estimation task is different, and why using no reference mesh/image is important.\n\n2. Object orientation is not a well-defined task. The definition of orientation is closely related to the semantic meaning of the specifc object categories, i.e., the canonical pose space, where the objects are aligned as forward-facing. Thus, on novel categories, the orientation of objects are not well-defined. Thus, it is necessary to study how the model generalize to categories that are unseen during training, as the categories in objaverse is not guaranteed to include every category in real life.\n\n3. The authors should also discuss why the model trained on synthetic data can generalize to real data.\n\n3. Besides, the authors should study the quality of the curated dataset, and study the influence of training data scale.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors presented a foundation model for object orientation estimation. The main contributions are in scaling the size of the dataset through the use of 3D models and VLMs, and a loss function. While the reviewers noted that the modeling innovations are weak, e.g., use of DINO pretrained weights, use of small head for output, supervised learning, etc., they agreed that the proposed method produces strong empirical results. The value brought comes largely from laying the ground work in curating the large-scale dataset and training a foundational model for the task.
Decision: Accept (poster)