SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our reconstruction module requires camera poses for the  $4 \times n$  source view images. Note that we adopt Zero123 for image synthesis, which parameterizes cameras in a canonical spherical coordinate frame,  $(\theta, \phi, r)$ , where  $\theta, \phi$  and  $r$  represent the elevation, azimuth, and radius. While we can arbitrarily adjust the azimuth angle  $\phi$  and the radius  $r$  of all source view images simultaneously, resulting in the rotation and scaling of the reconstructed object accordingly, this parameterization requires knowing the absolute elevation angle  $\theta$  of one camera to determine the relative poses of all cameras in a standard XYZ frame. More specifically, the relative poses between camera  $(\theta_0, \phi_0, r_0)$  and camera  $(\theta_0 + \Delta \theta, \phi_0 + \Delta \phi, r_0)$  vary for different  $\theta_0$  even when  $\Delta \theta$  and  $\Delta \phi$  are the same. Because of this, changing the elevation angles of all source images together (e.g., by 30 degrees up or 30 degrees down) will lead to the distortion of the reconstructed shape (see Figure 10 for examples).
Therefore, we propose an elevation estimation module to infer the elevation angle of the input image. First, we use Zero123 to predict four nearby views of the input image. Then we enumerate all possible elevation angles in a coarse-to-fine manner. For each elevation candidate angle, we compute the corresponding camera poses for the four images and calculate a reprojection error for this set of camera poses to measure the consistency between the images and the camera poses. The elevation angle with the smallest reprojection error is used to generate the camera poses for all  $4 \times n$  source views by combining the pose of the input view and the relative poses. Please refer to the appendix for details on how we calculate the reprojection error for a set of posed images.
For each input image, we generate  $n = 8$  images by choosing camera poses uniformly placed on the sphere surface and then generate 4 local images ( $10^{\circ}$  apart) for each of the 8 views, resulting in 32 source-view images for reconstruction. During training, we freeze the Zero123 [41] model and train our reconstruction module on the Objverse-LVIS [12] dataset, which contains 46K 3D models in 1,156 categories. We use BlenderProc [14] to render ground-truth RGB images. For images with background, we utilize an off-the-shelf segmentation network SAM [33] with bounding-box prompts for background removal. Please refer to the appendix for more details.
4.2 Single Image to 3D Mesh
We present qualitative examples of our method in Figures 1 and 5, illustrating its effectiveness in handling both synthetic images and real images. We also compare One-2-3-45 with existing zero-shot single image 3D reconstruction approaches, including Point-E [57], Shap-E [30], Zero123 (Stable
Figure 6: We compare One-2-3-45 with Point-E [57], Shap-E [30], Zero123 (Stable Dreamfusion version) [41], 3DFuse [72], and RealFusion [48]. In each example, we present both the textured and textureless meshes. As 3DFuse [72] and RealFusion [48] do not natively support the export of textured meshes, we showcase the results of volume rendering instead.
Table 1: Quantitative Comparison on GSO [15] and Objaverse [12] datasets.
Prior Source	F-Score	CLIP Similarity	Time
GSO	Obj.	avg.	GSO	Obj.	avg.
Point-E [57]	internal	81.0	81.0	81.0	74.3	78.5	76.4	78s
Shap-E [30]	3D data	83.4	81.2	82.3	79.6	82.1	80.9	27s
Zero123+SD [41]	2D diffusion models	75.1	69.9	72.5	71.0	72.7	71.9	~15min
RealFusion [48]	66.7	59.3	63.0	69.3	69.5	69.4	~90min
3DFuse [72]	60.7	60.2	60.4	71.4	74.0	72.7	~30min
Ours	84.0	83.1	83.5	76.4	79.7	78.1	45s
Figure 7: Error distribution of predicted elevations. The median and average are 5.4 and 9.7 degrees.
Dreamfusion version) [41], 3DFuse [72], and RealFusion [48]. Among them, Point-E and Shap-E are two 3D native diffusion models released by OpenAI, which are trained on several million internal 3D data, while others are optimization-based approaches leveraging priors from Stable Diffusion [68].
Figure 6 presents the qualitative comparison. While most methods can generate plausible 3D meshes from a single image, notable differences exist among them in terms of geometry quality, adherence to the input, and overall 3D consistency. In terms of geometry quality, approaches like RealFusion [48] and 3DFuse [72], which optimize a neural radiance field, face challenges in extracting high-quality meshes. Likewise, Point-E [57] produces a sparse point cloud as its output, resulting in numerous holes on the reconstructed meshes. In contrast, our approach utilizes an SDF presentation and favors better geometry. Regarding adherence to the input, we observe that most baseline methods struggle to preserve the similarity to the input image. Although Shap-E performs slightly better, it still produces lots of failure cases (see the backpack without shoulder straps, distorted shoe, and stool with three legs). In contrast, our approach leverages a powerful 2D diffusion model to directly produce high-quality multi-view images, rather than relying on 3D space hallucination. This strategy provides
Figure 8: Ablations on training strategies of the reconstruction module and the number of views.
better adherence to the input views, alleviates the burden of the 3D reconstruction module, and yields results that are more finely attuned to the input. Furthermore, many approaches encounter challenges in achieving consistent 3D results (also known as the Janus problem [48; 60]), as highlighted in the right figure (two-handle mug, multi-face Mario, and two-face backpack). One of the contributing factors to this issue is that several methods optimize each view independently, striving to make each view resemble the input. In contrast, our method capitalizes on the view-conditioned 2D diffusion model, inherently enhancing 3D consistency.
We also quantitatively compare the approaches on Objaverse [12] and GoogleScannedObjects (GSO) [15] datasets. For each dataset, we randomly choose 20 shapes and render a single image per shape for evaluation. To align the predictions with the ground-truth mesh, we linearly search the scaling factor and the rotation angle, apply Iterative Closest Point (ICP) for sampled point clouds, and select the one with the most number of inliers. We follow RealFusion [48] to report F-score (with a threshold of 0.05) and CLIP similarity, and the runtime on an A100 GPU. As shown in Table 1, our method outperforms all baseline approaches in terms of F-Score. As for CLIP similarity, we surpass all methods except a concurrent work Shap-E [30]. We find that CLIP similarity is very sensitive to the color distribution and less discriminative in local geometry variations (i.e., the number of legs of a stool, the number of handles of a mug). Regarding running time, our method demonstrates a notable advantage over optimization-based approaches and performs on par with 3D native diffusion models, such as Point-E [57] and Shap-E [30]. Specifically, our 3D reconstruction module reconstructs a 3D mesh in approximately 5 seconds, with the remaining time primarily spent on Zero123 predictions, which takes roughly 1 second per image on an A100 GPU.
Training strategies. We ablate our training strategies in Figure 8. We found that without our 2-stage source view selection strategy, a network trained to consume 31 uniformly posed Zero123 predictions (fourth column) suffers from severe inconsistency among source views, causing the reconstruction module to fail completely. If we feed only 7 source views (sixth column) without the four nearby views, the reconstruction fails to capture local correspondence and cannot reconstruct fine-grained geometry. During training, we first render  $n$  ground-truth renderings and then use Zero123 to predict four nearby views for each of them. If we train directly on  $8 \times 4$  ground-truth renderings without Zero123 prediction during training (second column), it fails to generalize well to Zero123 predictions
Figure 9:  $360^{\circ}$  reconstruction vs. multiview fusion. Meshes from different views are in different colors.
Figure 10: Incorrect elevations lead to distorted reconstruction. Our elevation estimation module can predict an accurate elevation of the input view.
Figure 11: Text to 3D. First row: "a bear in cowboy suit." Second row: "a kungfu cat." We utilize DALL-E 2 [63] to generate an image conditioned on the text and then lift it to 3D. We compare our method with Stable Dreamfusion [60] and 3DFuse [72]. For baselines, volume renderings are shown.
during inference, with many missing regions. Instead, if we replace the  $n$  ground-truth renderings with  $n$  Zero123 predictions during training (first column), the network fail to generate sharp details (see the strips of the backpack).
Elevation estimation. Our reconstruction module relies on accurate elevation angles of the input view. In Figure 10, we demonstrate the impact of providing incorrect elevation angles (e.g., altering the elevation angles of source views by  $\pm 30^{\circ}$ ), which results in distorted reconstruction results. Instead, utilizing our predicted elevation angles can perfectly match results with ground truth elevations. We also quantitatively test our elevation estimation module by rendering 1,700 images from random camera poses. As shown in Figure 7, our elevation estimation module predicts accurate elevations.
Number of source views. In Figure 8, we also investigate the impact of varying the number of source views on 3D reconstruction. We observe that our method is not very sensitive to the number of views as long as the reconstruction module is retrained with the corresponding setting.
$360^{\circ}$  reconstruction vs. multi-view fusion. While our method reconstructs a  $360^{\circ}$  mesh in a single pass, most existing generalizable neural reconstruction approaches [45; 29; 6] primarily focus on frontal view reconstruction. An alternative approach is to independently infer the geometry for each view and subsequently fuse them together. However, we have observed that this strategy often struggles with multi-view fusion due to inconsistent Zero123 predictions, as illustrated in Figure 9.
4.4 Text to 3D Mesh
As shown in Figure 11, by integrating with off-the-shelf text-to-image 2D diffusion models [68; 63], our method can be naturally extended to support text-to-image-3D tasks and generate high-quality textured meshes in a short time. See supplementary for more examples.
In this paper, we present a novel method for reconstructing a high-quality  $360^{\circ}$  mesh of any object from a single image of it. In comparison to existing zero-shot approaches, our results exhibit superior geometry, enhanced 3D consistency, and a remarkable adherence to the input image. Notably, our approach reconstructs meshes in a single forward pass without the need for time-consuming optimization, resulting in significantly reduced processing time. Furthermore, our method can be effortlessly extended to support the text-to-3D task.
This work is supported in part by gifts from Qualcomm. We would like to thank Ruoxi Shi, Xinyue Wei, Hansheng Chen, Jiayuan Gu, Fanbo Xiang, Xiaoshuai Zhang, and Yulin Liu for their helpful discussions and manuscript proofreading.
We would like to thank the following sketchfab users for the models used for the demo images in this paper: dimaponomar2019 (backpack), danielpeng (bag), pmlzbt233 (wooden barrel), felixyadomi (cactus), avianinda (burger), shedmon (robocat), ie-niels (stool), phucn (armchair), techCIR (mug), sabriny (fox). All models are CC-By licensed.
arXiv:2210.06978, 2022.
[97] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[98] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449-5458, 2022.
[99] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In CVPR, 2023.
[100] Silvia Zuffi, Angjoo Kanazawa, and Michael J Black. Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3955-3963, 2018.
[101] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6365-6373, 2017.
We first show more qualitative comparison in Section A, which is followed by a demonstration of additional examples on real-world images and the text-to-3D task in Sections B and C respectively. Furthermore, we present the details of our elevation estimation module in Section D, training and evaluation details in Section E. We finally show the failure cases and discuss the limitations in Section F.
A More Qualitative Comparison
Figure 12: We compare One-2-3-45 with Point-E [57], Shap-E [30], Zero123 (Stable Dreamfusion version) [41], 3DFuse [72], and RealFusion [48]. In each example, we present both the textured and textureless meshes. As 3DFuse [72] and RealFusion [48] do not natively support the export of textured meshes, we showcase the results of volume rendering instead.
In Figure 12, we demonstrate more qualitative comparison on Objaverse [12] and GoogleScannedObjects (GSO) [15] datasets. Note that all test shapes are not seen during the training of our 3D reconstruction module.
B More Examples on Real-World Images
In Figure 13, we showcase more examples on real-world images and compare our method with the concurrent method Shap-E [30]. The input images are from unsplash.com or captured by ourselves. Note that our results exhibit a closer adherence to the input image.
C More Examples on Text-to-3D
In Figure 14, we present additional examples for the text-to-3D task. It is evident that existing approaches struggle to capture fine-grained details, such as a tree hollow, or achieve compositionality, as seen in examples like an orange stool with green legs, a pineapple-shaped Havana hat, or a rocking horse chair. In contrast, our method produces superior results that adhere more closely to the input text. We hypothesize that controlling such fine-grained attributes in the 3D space using existing optimization strategies is inherently challenging. However, by leveraging established 2D
Figure 13: We compare One-2-3-45 with Shap-E [30] on real-world images. In each example, we present the input image, generated textured and textureless meshes.
text-to-image diffusion models, our method becomes more effective in lifting a single 2D image to a corresponding 3D textured mesh.
D Details of Elevation Estimation
To estimate the elevation angle  $\theta$  of the input image, we first utilize Zero123 [41] to predict four nearby views (10 degrees apart) of the input view. With these predicted views, we proceed to enumerate all possible elevation angles and compute the re-projection error for each candidate angle.
Figure 14: Text-to-3D: We compare our method against two native text-to-3D approaches Stable DreamFusion [60] and 3DFuse [72]. To enable text-to-3D, our method first uses a pretrained text-to-image model DALL-E 2 [63] to generate an image from input text (prompted with "3d model, long shot"), and then uplifts the image to a 3D textured mesh.
The re-projection error assesses the consistency between camera poses and image observations, akin to the bundle adjustment module employed in the Structure-from-Motion (SfM) pipeline.
Specifically, we enumerate all candidate elevation angles in a coarse-to-fine manner. In the coarse stage, we enumerate elevation angles with a 10-degree interval. Once we have determined the elevation angle  $e^{*}$  associated with the smallest re-projection error, we proceed to the fine stage. In this stage, we enumerate elevation angle candidates ranging from  $e^{*} - 10^{\circ}$  to  $e^{*} + 10^{\circ}$  with a 1-degree
interval. This coarse-to-fine design facilitates rapid estimation, completing the elevation estimation module in under 1 second for each shape.
Given a set of four predicted nearby views, we perform feature matching to identify corresponding keypoints across each pair of images (a total of six pairs) using an off-the-shelf module LoFTR [74]. For each elevation angle candidate, we calculate the camera pose for the input image by employing the spherical coordinate system with a radius of 1.2 and an azimuth angle of 0. Note that the azimuth angle  $\phi$  and the radius  $r$  can be arbitrarily adjusted, resulting in the rotation and scaling of the reconstructed object accordingly. Subsequently, we obtain the camera poses for the four predicted views by incorporating the specified delta poses.
Once we have the four posed images, we compute the re-projection error by enumerating triplet images. For each triplet of images  $(a,b,c)$  sharing a set of keypoints  $P$ , we consider each point  $p\in P$ . Utilizing images  $a$  and  $b$ , we perform triangulation to determine the 3D location of  $p$ . We then project the 3D point onto the third image  $c$  and calculate the reprojection error, which is defined as the  $l1$  distance between the projected 2D pixel and the estimated keypoint in image  $c$ . By enumerating all image triplets and their corresponding shared keypoints, we obtain the mean projection error for each elevation angle candidate.
E Details of Training and Evaluation
Training We train the reconstruction module using the following loss function:
$$
\mathcal {L} = \mathcal {L} _ {r g b} + \lambda_ {1} \mathcal {L} _ {\text {e i k o n a l}} + \lambda_ {2} \mathcal {L} _ {\text {s p a r s i t y}} \tag {1}
$$
where  $\mathcal{L}_{rgb}$  represents the  $l1$  loss between the rendered and ground truth color, weighted by the sum of accumulated weights;  $\mathcal{L}_{eikonal}$  and  $\mathcal{L}_{sparsity}$  are the Eikonal and sparsity terms, respectively, following SparseNeuS [45]. We empirically set the weights as  $\lambda_0 = 1$ ,  $\lambda_1 = 0.1$ , and  $\lambda_2 = 0.02$ . For  $\lambda_2$ , we adopt a linear warm-up strategy following SparseNeuS [45]. To train our reconstruction module, we utilize the LVIS subset of the Objaverse [12] dataset, which consists of 46k 3D models across 1,156 categories. The reconstruction module is trained for 300k iterations using two A10 GPUs, with the training process lasting approximately 6 days. It is important to note that our reconstruction module does not heavily rely on large-scale training data, as it primarily leverages local correspondence to infer the geometry, which is relatively easier to learn and generalize.
Evaluation We evaluate all baseline approaches using their official codebase. Since the approaches take only a single image as input, the predicted mesh may not have the same scale and transformation as the ground-truth mesh. To ensure a fair comparison, we employ the following process to align the predicted mesh with the ground-truth mesh. First, we align the up direction for the results generated by each approach. Next, for each generated mesh, we perform a linear search over scales and rotation angles along the up direction. After applying each pair of scale and z-rotation, we utilize the Iterative Closest Point (ICP) algorithm to align the transformed mesh to the ground-truth mesh. Finally, we select the mesh with the largest number of inliers as the final alignment. This alignment process helps us establish a consistent reference frame for evaluating the predicted meshes across different approaches. To calculate CLIP similarity, we render both ground-truth and generated meshes, capturing 24 views around the 3D shape from fixed viewpoints - 12 views at  $30^{\circ}$  elevation and 12 views at  $0^{\circ}$  elevation.
F Failure Cases and Limitations
Our method relies on Zero123 for generating multi-view images, which introduces challenges due to its occasional production of inconsistent results. In Figure 15, we present two typical cases that exemplify such inconsistencies. The first case involves an input view that lacks sufficient information, such as the back view of a fox. In this scenario, Zero123 struggles to generate consistent predictions for the invisible regions, such as the face of the fox. As a consequence, our method may encounter difficulties in accurately inferring the geometry for those regions. The second case involves an input view with ambiguous or complex structures, such as the pulp and peel of a banana. In such situations, Zero123's ability to accurately infer the underlying geometry becomes limited. As a result, our method may be affected by the inconsistent predictions generated by Zero123. It is important to acknowledge that these limitations arise from the occasional scenarios, and they can impact the
our recon.
Figure 15: Failure cases. Our method relies on Zero123 to generate multi-view images, and we encounter challenges when Zero123 generates inconsistent results. (a) The input view lacks sufficient information. (b) The input view contains ambiguous or complicated structures.
our recon.
performance of our method in certain cases. Addressing these challenges and refining the reliability of Zero123's predictions remain areas for further investigation and improvement.
We have also noticed slight artifacts on the back side of our generated results. As one of the first works in combining view-conditioned 2D diffusion models with generalizable multi-view reconstruction, we believe that there is still ample room for exploring more advanced reconstruction techniques and incorporating additional regularizations. By doing so, we expect to significantly mitigate the minor artifacts and further enhance results in the future.