SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minghua Liu $^{1*}$
Chao  $\mathbf{X}\mathbf{u}^{2*}$
Haian Jin $^{3,4*}$
Linghao Chen $^{1,4*}$
Mukund Varma  $\mathbf{T}^1$
Zexiang Xu
Hao Su
<sup>1</sup> UC San Diego <sup>2</sup> UCLA <sup>3</sup> Cornell University <sup>4</sup> Zhejiang University <sup>5</sup> Adobe Research
Project Website: http://one-2-3-45.com
Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.
Single image 3D reconstruction, the task of reconstructing a 3D model of an object from a single 2D image, is a long-standing problem in the computer vision community and is crucial for a wide range of applications, such as robotic object manipulation and navigation, 3D content creation, as well as AR/VR [47; 9; 92]. The problem is challenging as it requires not only the reconstruction of visible parts but also the hallucination of invisible regions. Consequently, this problem is often ill-posed and corresponds to multiple plausible solutions because of insufficient evidence from a single image. On the other hand, humans can adeptly infer unseen 3D content based on our extensive knowledge of the 3D world. To endow intelligent agents with this ability, many existing methods [31; 19; 25; 11; 87; 91; 16; 83; 39; 10; 37] exploit class-specific priors by training 3D generative networks on 3D shape datasets [4]. However, these methods often fail to generalize to unseen categories, and their reconstruction quality is constrained by the limited size of public 3D datasets.
Figure 1: One-2-3-45 reconstructs a full  $360^{\circ}$  mesh of any object in 45 seconds given a single image of it. In each example, we showcase the input image in the left column, alongside the generated textured and textureless meshes from three different views.
In this work, we pursue a generic solution to turn an image of any object, regardless of its category, into a high-quality 3D textured mesh. To achieve this, we propose a novel approach that can effectively utilize the strong priors learned by 2D diffusion models for 3D reconstruction. Compared to 3D data, 2D images are more readily available and scalable. Recent 2D generative models (e.g., DALLE [64; 63], Imagen [69], and Stable Diffusion [68]) and visual-language models (e.g., CLIP [61]) have made significant strides by pre-training on Internet-scale image datasets. Since they learn a wide range of visual concepts and possess strong priors about our 3D world, it is natural to marry 3D tasks with them. Consequently, an emerging body of research [26; 23; 52; 60; 36], as exemplified by DreamField [26], DreamFusion [60], and Magic3D [36], employs 2D diffusion models or vision language models to assist 3D generative tasks. The common paradigm of them is to perform per-shape optimization with differentiable rendering and the guidance of the CLIP model or 2D diffusion models. While many other 3D representations have been explored, neural fields are the most commonly used representation during optimization.
Although these optimization-based methods have achieved impressive results on both text-to-3D [60; 26; 36] and image-to-3D tasks [48; 72], they face some common dilemmas: (a) time-consuming. Pershape optimization typically involves tens of thousands of iterations of full-image volume rendering and prior model inferences, resulting in typically tens of minutes per shape. (b) memory intensive. Since the full image is required for the 2D prior model, the volume rendering can be memory-intensive when the image resolution goes up. (c) 3D inconsistent. Since the 2D prior model only sees a single view at each iteration and tries to make every view look like the input, they often generate 3D inconsistent shapes (e.g., with two faces, or the Janus problem [48; 60]). (d) poor geometry. Many methods utilize the density field as the representation in volume rendering. It is common that they produce good RGB renderings but extracting high-quality mesh tends to be difficult.
In this paper, instead of following the common optimization-based paradigm, we propose a novel approach to utilize 2D prior models for 3D modeling. At the heart of our approach is the combination of a 2D diffusion model with a cost-volume-based 3D reconstruction technique, enabling
the reconstruction of a high-quality  $360^{\circ}$  textured mesh from a single image in a feed-forward pass without per-scene optimization. Specifically, we leverage a recent 2D diffusion model, Zero123 [41], which is fine-tuned on Stable Diffusion [68] to predict novel views of the input image given the camera transformation. We utilize it to generate multi-view predictions of the input single image so that we can leverage multi-view 3D reconstruction techniques to obtain a 3D mesh. There are two challenges associated with reconstruction from synthesized multi-view predictions: (a) the inherent lack of perfect consistency within the multi-view predictions, which can lead to severe failures in optimization-based methods such as NeRF methods [53; 5]. (b) the camera pose of the input image is required but unknown. To tackle them, we build our reconstruction module upon a cost volume-based neural surface reconstruction approach, SparseNeuS [45], which is a variant of MVSNeRF [6]. Additionally, we introduce a series of essential training strategies that enable the reconstruction of 360-degree meshes from inherently inconsistent multi-view predictions. We also propose an elevation estimation module that estimates the elevation of the input shape in Zero123's canonical coordinate system, which is used to compute the camera poses required by the reconstruction module.
By integrating the three modules of multi-view synthesis, elevation estimation, and 3D reconstruction, our method can reconstruct 3D meshes of any object from a single image in a feed-forward manner. Without costly optimizations, our method reconstructs 3D shapes in significantly less time, e.g., in just 45 seconds. Our method favors better geometry due to the use of SDF representations, and generates more consistent 3D meshes, thanks to the camera-conditioned multi-view predictions. Moreover, our reconstruction adheres more closely to the input image compared to existing methods. See Figure 1 for some of our example results. We evaluate our method on both synthetic data and real images and demonstrate that our method outperforms existing methods in terms of both quality and efficiency.
2.1 3D Generation Guided by 2D Prior Models
Recently, 2D generative models (e.g., DALL-E [64; 63], Imagen [69], and Stable Diffusion [68]) and vision-language models (e.g., CLIP [61]) have learned a wide range of visual concepts by pre-training on Internet-scale image datasets. They possess powerful priors about our 3D world and have inspired a growing body of research to employ 2D prior models for assisting 3D understanding [38; 40] and generative tasks. Exemplified by DreamField [26], DreamFusion [60], and Magic3D [36], a line of works follows the paradigm of per-shape optimization. They typically optimize a 3D representation (i.e., NeRF, mesh, SMPL human model) and utilize differentiable rendering to generate 2D images from various views. The images are then fed to the CLIP model [23; 26; 52; 35; 3; 32; 2; 28; 89; 43] or 2D diffusion model [60; 36; 72; 48; 13; 78; 88; 51; 99; 62; 75] for calculating the loss functions, which are used to guide the 3D shape optimization. In addition to optimization-based 3D shape generation, some works train a 3D generative model but leverage the embedding space of CLIP [8; 44; 71], and some works focus on generating textures or materials for input meshes using 2D models' prior [52; 82; 7; 51; 67].
2.2 Single Image to 3D
Before the emergence of CLIP and large-scale 2D diffusion models, people often learn 3D priors from 3D synthetic data [4] or real scans [65]. Unlike 2D images, 3D data can be represented in various formats and numerous representation-specific 3D generative models have been proposed. By combining 2D image encoder and 3D generators, they generate 3D data in various representations, including 3D voxels [19; 85; 11; 87; 86; 91], point clouds [16; 94; 20; 1; 49; 96], polygon meshes [31; 79; 83; 56], and parametric models [59; 100; 101]. Recently, there has been an increasing number of work on learning to generate a 3D implicit field from a single image [90; 50; 70; 25; 58; 18; 21; 27; 55; 84; 54].
As previously mentioned, several recent works leverage 2D diffusion models to perform per-shape optimization, allowing for the text-to-3D task [60; 36; 26] given that diffusion models are typically conditioned on text. To enable the generation of 3D models from a single image, some works [48; 13; 51] utilize textual inversion [17], to find the best-matching text embedding for the input image, which is then fed into a diffusion model. NeuralLift-360 [24] adds a CLIP loss to enforce similarity between the rendered image and the input image. 3DFuse [72] finetunes the Stable Diffusion model with LoRA layers [24] and a sparse depth injector to ensure greater 3D consistency. A recent work
Figure 2: Our method consists of three primary components: (a) Multi-view synthesis: we use a view-conditioned 2D diffusion model, Zero123 [41], to generate multi-view images in a two-stage manner. The input of Zero123 includes a single image and a relative camera transformation, which is parameterized by the relative spherical coordinates  $(\Delta \theta, \Delta \phi, \Delta r)$ . (b) Pose estimation: we estimate the elevation angle  $\theta$  of the input image based on four nearby views generated by Zero123. We then obtain the poses of the multi-view images by combining the specified relative poses with the estimated pose of the input view. (c) 3D reconstruction: We feed the multi-view posed images to an SDF-based generalizable neural surface reconstruction module for  $360^{\circ}$  mesh reconstruction.
Zero123 [41; 73] finetunes the Stable Diffusion model [69] to generate a novel view of the input image based on relative camera pose. In addition to these methods, OpenAI trains a 3D native diffusion model Point-E [57], which uses several million internal 3D models to generate point clouds. Very recently, they published another model Shap-E [30] which is trained to generate parameters of implicit functions that can be used for producing textured meshes or neural radiance fields.
2.3 Generalizable Neural Reconstruction
Traditional NeRF-like methods [53; 80] use a neural network to represent a single scene and require per-scene optimization. However, some approaches aim to learn priors across scenes and generalize to novel scenes. These methods typically take a few source views as input and leverage 2D networks for extracting 2D features. The pixel features are then unprojected into 3D space, and a NeRF-based rendering pipeline is applied on top of them. In this way, they can generate a 3D implicit field given a few source views in a single feed-forward pass. Among the methods, some [81; 65; 22; 95; 93; 42; 34; 76; 77] directly aggregate 2D features with MLPs or transformers, while others explicitly construct the 3D feature/cost volume [6; 29; 98; 45], and utilize the voxel feature for decoding density and color. In addition to the density field representation, some methods such as SparseNeuS [45] and VolRecon [66] utilize SDF representations for geometry reconstruction.
Our overall pipeline is illustrated in Figure 2. In Section 3.1, we introduce a view-conditioned 2D diffusion model, Zero123 [41], which is used to generate multi-view images. In Section 3.2, we show that traditional NeRF-based and SDF-based methods fail to reconstruct high-quality meshes from inconsistent multi-view predictions even given ground truth camera poses. Therefore, in Section 3.3, we propose a cost volume-based neural surface reconstruction module that can be trained to handle inconsistent multi-view predictions and reconstruct a 3D mesh in a single feed-forward pass. Specifically, we build upon the SparseNeuS [45] and introduce several critical training strategies to support  $360^{\circ}$  mesh reconstruction. Additionally, in Section 3.4, we demonstrate the necessity of estimating the pose of the input view in Zero123's canonical space for 3D reconstruction. While the azimuth and radius can be arbitrarily specified, we propose a novel module that utilizes four nearby views generated by Zero123 to estimate the elevation of the input view.
3.1 Zero123: View-Conditioned 2D Diffusion
Recent 2D diffusion models [64; 69; 68] have demonstrated the ability to learn a wide range of visual concepts and strong priors by training on internet-scale data. While the original diffusion
TensoRF
NeuS
NeuS
Figure 3: NeRF-based method [53] and SDF-based method [80] fail to reconstruct high-quality meshes given multi-view images predicted by Zero123. See Figure 1 for our reconstruction results.
models mainly focused on the task of text-to-image, recent work [97; 24] has shown that fine-tuning pretrained models allows us to add various conditional controls to the diffusion models and generate images based on specific conditions. Several conditions, such as canny edges, user scribbles, depth, and normal maps, have already proven effective [97].
The recent work Zero123 [41] shares a similar spirit and aims to add viewpoint condition control for the Stable Diffusion model [68]. Specifically, given a single RGB image of an object and a relative camera transformation, Zero123 aims to control the diffusion model to synthesize a new image under this transformed camera view. To achieve this, Zero123 fine-tunes the Stable Diffusion on paired images with their relative camera transformations, synthesized from a large-scale 3D dataset [12]. During the creation of the fine-tuning dataset, Zero123 assumes that the object is centered at the origin of the coordinate system and uses a spherical camera, i.e., the camera is placed on the sphere's surface and always looks at the origin. For two camera poses  $(\theta_{1},\phi_{1},r_{1})$  and  $(\theta_{2},\phi_{2},r_{2})$ , where  $\theta_{i}$ ,  $\phi_{i}$ , and  $r_i$  denote the polar angle, azimuth angle, and radius, their relative camera transformation is parameterized as  $(\theta_{2} - \theta_{1},\phi_{2} - \phi_{1},r_{2} - r_{1})$ . They aim to learn a model  $f$ , such that  $f(x_{1},\theta_{2} - \theta_{1},\phi_{2} - \phi_{1},r_{2} - r_{1})$  is perceptually similar to  $x_{2}$ , where  $x_{1}$  and  $x_{2}$  are two images of an object captured from different views. Zero123 finds that such fine-tuning enables the Stable Diffusion model to learn a generic mechanism for controlling the camera viewpoints, which extrapolates outside of the objects seen in the fine-tuning dataset.
3.2 Can NeRF Optimization Lift Multi-View Predictions to 3D?
Given a single image of an object, we can utilize Zero123 [41] to generate multi-view images, but can we use traditional NeRF-based or SDF-based methods [5; 80] to reconstruct high-quality 3D meshes from these predictions? We conduct a small experiment to test this hypothesis. Given a single image, we first generate 32 multi-view images using Zero123, with camera poses uniformly sampled from the sphere surface. We then feed the predictions to a NeRF-based method (TensoRF [53]) and an SDF-based method (NeuS [80]), which optimize density and SDF fields, respectively. However, as shown in Figure 3, both methods fail to produce satisfactory results, generating numerous distortions and floaters. This is primarily due to the inconsistency of Zero123's predictions. In Figure 4, we compare Zero123's predictions with ground-truth renderings. We can see that the overall PSNR is not very high, particularly when the input relative pose is large or the target pose is at unusual locations (e.g., from the bottom or the top). However, the mask IoU (most regions are greater than 0.95) and CLIP similarity are relatively good. This suggests that Zero123 tends to generate predictions that are perceptually similar to the ground truth and have similar contours or boundaries, but the pixel-level appearance may not be exactly the same. Nevertheless, such inconsistencies between the source views are already fatal to traditional optimization-based methods. Although the original Zero123 paper proposes another method for lifting its multi-view predictions, we will demonstrate in experiments that it also fails to yield perfect results and entails time-consuming optimization.
3.3 Neural Surface Reconstruction from Imperfect Multi-View Predictions
Instead of using optimization-based approaches, we base our reconstruction module on a generalizable SDF reconstruction method SparseNeuS [45], which is essentially a variant of the MVSNeRF [6] pipeline that combines multi-view stereo, neural scene representation, and volume rendering. As illustrated in Figure 2, our reconstruction module takes multiple source images with corresponding camera poses as input and generates a textured mesh in a single feed-forward pass. In this section, we will first briefly describe the network pipeline of the module and then explain how we train the module, select the source images, and generate textured meshes. Additionally, in Section 3.4, we will discuss how we generate the camera poses for the source images.
Figure 4: We analyze the prediction quality of Zero123 by comparing its predictions to ground truth renderings across various view transformations. For each view transformation, we report the average PSNR, mask IoU, and CLIP similarity of 100 shapes from the Objaverse [12] dataset. The prediction mask is calculated by considering foreground objects (i.e., non-white regions). Zero123 provides more accurate predictions when the view transformation is small.
As shown in Figure 2, our reconstruction module takes  $m$  posed source images as input. The module begins by extracting  $m$  2D feature maps using a 2D feature network. Next, the module builds a 3D cost volume whose contents are computed by first projecting each 3D voxel to  $m$  2D feature planes and then fetching the variance of the features across the  $m$  projected 2D locations. The cost volume is then processed using a sparse 3D CNN to obtain a geometry volume that encodes the underlying geometry of the input shape. To predict the SDF at an arbitrary 3D point, an MLP network takes the 3D coordinate and its corresponding interpolated features from the geometry encoding volume as input. To predict the color of a 3D point, another MLP network takes as input the 2D features at the projected locations, interpolated features from the geometry volume, and the viewing direction of the query ray relative to the viewing direction of the source images. The network predicts the blending weights for each source view, and the color of the 3D point is predicted as the weighted sum of its projected colors. Finally, an SDF-based rendering technique is applied on top of the two MLP networks for RGB and mask rendering [80]. In each iteration, we randomly choose one view to build the cost volume and another view for rendering supervision.
2-Stage Source View Selection and Groundtruth-Prediction Mixed Training. Although the original SparseNeuS [45] paper only demonstrated frontal view reconstruction, we have extended it to reconstruct 360-degree meshes in a single feed-forward pass by selecting source views in a particular way. Specifically, our reconstruction model is trained on a 3D object dataset while freezing Zero123. We follow Zero123 to normalize the training shapes and use a spherical camera model. For each shape, we first render  $n$  ground-truth RGB images from  $n$  camera poses uniformly placed on the sphere. For each of the  $n$  views, we use Zero123 to predict four nearby views. During training, we feed all  $4 \times n$  predictions with ground-truth poses into the reconstruction module and randomly choose one of the  $n$  ground-truth RGB images views as the target view. We call this view selection strategy as 2-stage source view selection. We supervise the training with both the ground-truth RGB and mask values. In this way, the module can learn to handle the inconsistent predictions from Zero123 and reconstruct a consistent  $360^{\circ}$  mesh. We argue that our two-stage source view selection strategy is critical since uniformly choosing  $n \times 4$  source views from the sphere surface would result in larger distances between the camera poses. However, cost volume-based methods [45; 29; 6] typically rely on very close source views to find local correspondences. Furthermore, as shown in Figure 4, when the relative pose is small (e.g., 10 degrees apart), Zero123 can provide very accurate and consistent predictions and thus can be used to find local correspondences and infer the geometry.
During training, we utilize  $n$  ground-truth renderings in the initial stage. We find that employing  $n$  predicted images at this stage would suffer from notable inconsistencies across different views, complicating the network's ability to learn sharp details (see examples in ablation study). However, during inference, we can replace the  $n$  ground-truth renderings with Zero123 predictions, as shown in Figure 2, the network can automatically generalize to some extent. We will show in the experiments that this groundtruth-prediction mixed training strategy is also important. To export the textured mesh, we use marching cubes [46] to extract the mesh from the predicted SDF field and query the color of the mesh vertices as described in [80]. Although our reconstruction module is trained on a 3D dataset, we find that it mainly relies on local correspondences and can generalize to unseen shapes very well.
Figure 5: Qualitative examples of One-2-3-45 for both synthetic and real images. Each triplet showcases an input image, a textured mesh, and a textureless mesh.
3.4 Camera Pose Estimation