One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization
Authors: Minghua Liu
Chao Xu
Haian Jin
Linghao Chen
Mukund Varma T
Zexiang Xu
Hao Su
Venue: NeurIPS 2023 poster (2023)
Keywords: single image reconstruction
3d generation
mesh reconstruction
diffusion models

Abstract: Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing a neural radiance field under the guidance of 2D diffusion models but suffer from lengthy optimization time, 3D inconsistency results, and poor geometry. In this work, we propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given a single image, we first use a view-conditioned 2D diffusion model, Zero123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon an SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360-degree meshes. Without costly optimizations, our method reconstructs 3D shapes in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely to the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off-the-shelf text-to-image diffusion models.

One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization
Minghua Liu!*
Chao Xu
Haian J
Linghao Chen
Mukund Varma T
Zexiang Xu
Hao Su
UC San Diego
UCLA
Cornell University
Zhejiang University Adobe Research
Project Website: 1ttr:///oee233---0 com
Abstract
Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. Many existing methods solve this problem by optimizing neural radiance field under the guidance of 21) diffusion models but suffer from lengthy optimization time, inconsistency results, and poor geometry. In this work, we propose novel method that takes a single image any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass. Given single image, first use view-conditioned 2D diffusion model, 123, to generate multi-view images for the input view, and then aim to lift them up to 3D space. Since traditional reconstruction methods struggle with inconsistent multi-view predictions, we build our 3D reconstruction module upon SDF-based generalizable neural surface reconstruction method and propose several critical training strategies to enable the reconstruction of 360- degree meshes. Without costly optimizations, in significantly less time than existing methods. Moreover, our method favors better geometry, generates more 3D consistent results, and adheres more closely the input image. We evaluate our approach on both synthetic data and in-the-wild images and demonstrate its superiority in terms of both mesh quality and runtime. In addition, our approach can seamlessly support the text-to-3D task by integrating with off- the-shelf text-to-image diffusion models.
method reconstructs 3D shapes
1 Introduction
Single image 3D reconstruction, the task of reconstructing a 3D model of object from single 2D image, is a long-standing problem in the computer vision community and crucial for wide range of applications, such as robotic object manipulation and navigation, 3D content creation, as well as AR/VR 47: 92). The problem is challenging as it requires not only the reconstruction of visible parts but also the hallucination of invisible regions. Consequently. this problem is often ill-posed and corresponds to multiple plausible solutions because of insufficient evidence from single image. On the other hand, humans can adeptly infer unseen 3D content based on our extensive knowledge of the 3D world. To endow intelligent agents with this ability, many existing methods BI
exploit class- priors by training 3D generative networks on 3D shape datasets 4 However, these methods often fail to generalize to unseen categories, and their reconstruction quality constrained by the limited size of public 3D datasets.
19 25
87 16
39
10
37
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

One-2-3-45 reconstructs full 360 mesh of any object in 45 seconds given a single image of it. In each example, we showcase the input image in the left column, alongside the generated textured and textureless meshes from three different views.
Figure I:
In this work, we pursue generic solution turn an image of any object, regardless of its category, into a high-quality 3D textured mesh. To achieve this, we propose novel approach that can effectively utilize the strong priors learned by 2D diffusion models for 3D reconstruction. Compared to 3D data, 2D images are more readily available and scalable. Recent 2D generative models (e.g., DALL- (68) and visual-language models (e.g., CLIP (1]) have made significant strides by pre-training on Internet-scale image datasets. Since they learn wide range visual concepts and possess strong priors about our 3D world, natural to marry tasks with them. Consequently,
664 633. Imagen [69), and Stable Diffusion
emerging body of research [26] 23 52 60: 36), exemplified
by DreamField [26], DreamFusion [60]. and Magic3D (36), employs 2D diffusion models or vision language models assist 3D generative tasks. The common paradigm of them is to perform per-shape optimization with differentiable rendering and the guidance of the CLIP model or 2D diffusion models. While many other 3D representations have been explored, neural fields the most commonly used representation during optimization.
Although these optimization- based methods have achieved impressive results on both text-to-3D 660
26 36 and image-to-3D tasks [48 72], they face some common dilemmas: (a) time-consuming. Per- shape optimization typically involves tens of thousands of iterations of full-image volume rendering and prior model inferences, resulting in typically tens of minutes per shape. (b) memory intensive. Since the full image is required for the 2D prior model, the volume rendering can be memory-intene when the image resolution goes up. (c) 3D inconsistent. Since the 2D prior model only sees a single view at each iteration and tries to make every view look like the input, they often generate 3D inconsistent shapes (e.g., with two faces, or the Janus problem 488 60)). methods utilize the density field the representation in volume rendering. It is common that they produce good RGB renderings but extracting high-qu quah--yulityy mesh tends to be difficult.
(d) poor geometry. Many
In this paper, instead of following the common oprimization-baase paradigm, we propose novel approach to utilize 2D prior models for modeling. At the heart of our approach the combi- nation of a 2D diffusion model with nost-volume-bsee 3D reconstruction technique, enabling

---

the reconstruction of a high-quality 360 textured mesh from a single image in ferd-forward pass without per-scene optimization. Specifically, we leverage a recent 2D diffusion model, Zerol23 [4]] which is fine-tuned on Stable Diffusion [8] to predict novel views of the input image given the camera transformation. We utilize it to generate multi-view predictions of the input single image so that we can leverage multi-view 3D reconstruction techniques to obtain a 3D mesh. There are two challenges associated with reconstruction from synthesized multi-view predictions: (a) the inherent lack of perfect consistency within the multi-view predictions, which can lead to severe failures in optimization-ban bbased methods such as NeRF methods [53] [5. (b) the camera pose of the input image is required but unknown. To tackle them, we build our reconstruction module upon cost polume-based neural surface reconstruction approach, SparseNeuS [45]. which is a variant of MVSNeRF [6]. Additionally, we introduce series of essential training strategies that enable the reconstruction of 360-degree meshes from inherently inconsistent multi-view predictions. We also propose an elevation estimation module that estimates the elevation of the input shape in Zero canonical coordinate system, which is used to compute the camera poses required by the reconstruction module.
By integrating the three modules of multi-view synthesis, elevation estimation, and 3D reconstruction, our method can reconstruct 3D meshes of any object from a single image in a feed-forward manner. Without costly optimizations, our method reconstructs 3D shapes in significantly less time, e.g., in just 45 seconds. Our method favors better geometry due to the use of SDF representations, and generates more consistent 3D meshes, thanks to the camera-conditionn reconstruction adheres more closely to the input image compared to existing methods. See Figure [ for some of our example results. We evaluate our method on both synthetic data and real images and demonstrate that our method outperforms existing methods in terms of both quality and efficiency.
multi-view predictions. Moreover,
Related Work
2.1 3D Generation Guided by 2D Prior Models
Recently, 2D generative models (e.g.. DALL-E 64 63], Imagen 699. and Stable Diffusion 68 and
vision-language models (e.g.. CLIP [1] have learned a wide range of visual concepts by pre-training on Internet-scale image datasets. They possess powerful priors about our 3D world and have inspired
growing body of research to employ 2D prior models for assisting 3D understanding [38 40 and
generative tasks. Exemplified by DreamField [26). DreamFusion [60], and Magic3D [36]. line of works follows the paradigm of per-shape optimization. They typically optimize a 3D representation (i.e., NeRF, mesh, SMPL human model) and utilize differentiable rendering to generate 2D images from various views. The images are then fed to the CLIP model (23 26 52 35
3 32 28
89 43 or
2D diffusion model [60 36 72
48 13 78 88 51 99 (5) for calculating the loss functions, which used to guide the 3D shape optimization. In addition to optimization-be based 3D shape generation, some works train a 3D generative model but leverage the embedding space of CLIP [8) 44 /T]], and some works focus on generating textures or materials for input meshes using 2D models'
prior 152
7551
2.2 Single Image to 3D
Before the emergence of CLIP and large-scale 2D diffusion models, people often learn 3D priors from 3D synthetic data 4 or real scans (65]. Unlike 2D images, 3D data can be represented in various formats and numerous representation-specifee 3D generative models have been proposed. By
combining 2D image encoder and 3D generators, they generate 3D data in various representations,
including 3D voxels [19 85 II 87, 86 91], point clouds [16] 94| 20 I 49; 96), polygon meshes [31
79: 83 (6), and parametric models (59 100 [01]. Recently, there has been an increasing number of
work on learning to generate a 3D implicit field from single image [90] 50 70 25 58
18
21
27
55 84 [4].
As previously mentioned, several recent works leverage 2D diffusion models to perform per-shape
optimization, allowing for the text-to-3D task 600 36 26 given that diffusion models are typically conditioned on text. To enable the generation of 3D models from a single image, some works [48]
13 51
utilize textual inversion [17], to find the best-matching text embedding for the input image, which is then fed into a diffusion model. NeuralLift-366 [24] adds CLIP loss to enforce similarity between the rendered image and the input image. 3DFuse 72 finetunes the Stable Diffusion model with LoRA layers 24 and sparse depth injector ensure greater 3D consistency. A recent work
3

---

Lrgh
mearty posss
Zero120
Elevation Extimation
of input view
elevation
camerra
poasa
rgb
Rendering
Vaume
rendened mask
rencered rgb
Cenasky
21. 1DP
meee
360-deg t
neartty
5000 U.P.P
Marching Bddes
Zoro
MLP
Input Image
30 Conv
8: frozen module ( point donsity
OcSt volume
geomatty volume
SDF prediction
toxtured mosh
pose (aP.
Figure 2: Our method consists of three primary components: (a) Multi-view synthesis: we use a view-conditioned 2D diffusion model, Zerol23 [4]] to generate multi view images in two-stage manner. The imput of Zero 123 includes single image and a relative camera transformation, which is parameterized by the relative spherical coordinates (A0, Ad, Ar). (b) Pose estimation: we estimate the elevation angle then obtain the poses of the multi-view images by combining the specified relative poses with the estimated pose of the input view. (c) 3D reconstruction: We feed the multi-view posed images to an SDF-based generalizable neural surface reconstruction module for 360 mesh reconstruction.
the input image based on four nearby views generated by Zero123. We
Zero M1 731 finetunes the Stable Diffusion model 69 to generate a novel view of the input image based on relative camera pose. In addition to these methods, OpenAI trains 3D native diffusion model Point-E (57), which uses several million internal 3D models to generate point clouds. Very recently, they published another model Shap-E [30] which is trained to generate parameters of implicit functions that can used for producing textured meshes or neural radiance fields.
2.3 Generalizable Neural Reconstruction
Traditional NeRF-like methods [53
a neural network to represent a single scene and require per-scene optimization. However, some approaches aim to learn priors across scenes and generalize novel scenes. These methods typically take few source views as imput and leverage 2D networks for extracting 2D features. The pixel features are then unprojected into 3D space, and a NeRF-basce rendering pipeline applied on top of them. In this way, they can generate a 31) implicit field given few source views in single feed-forward pass. Among the methods, some [8] 65 22
95 93
42 34 76: 77 directly aggregate 2D features with MLPs or transformers, while others explicitly construct the 3D feature/cost volume 26 98 45), and utilize the voxel feature for decoding density and color. In addition to the density field representation, some methods such as SparseNeuS [45] and VolRecon [66] utilize SDF representations for geometry reconstruction.
Method
we introduce a view-conditioned 2D diffusion model, Zero123 [41]. which is used to generate multi-view images. In Section [322 we show that traditional NeRF-based and SDF-based methods fail reconstruct high-quality meshes from inconsistent multi- -iel predictions even given ground truth camera poses. Therefore, in Section 3.3 we propose a cost volume-basse neural surface reconstruction module that can be trained to handle inconsistent multi-view predictions and reconstruct 3D mesh in a single feed-forward pass. Specifically, we build upon the SparseNeuS (45) and introduce several critical training strategies to support 360" mesh reconstruction. Additionally, in Section 3.4, we demonstrate the necessity of estimating the pose the input view in Zero 12'ss canonical space for 3D reconstruction. While the azimuth and radius can be arbitrarily specified, propose novel module that utilizes four nearby views generated by Zero123 to estimate the elevation of the input view.
Our overall pipeline illustrated in Figure 2 In Section 3.1
3.1 Zero123: View-Conditioned 2D Diffusion
have demonstrated the ability to learn a wide range of visual concepts and strong priors by training internet-ccaee data. While the original diffusion
Recent 2D diffusion models 64
69
68

---

input
TensoRF
NeuS
input
TensoRF
NeuS
Figure 3: NeRF-based method
55 and SDF-based method
80
fail reconstruct high-quality
meshes given multi-view images predicted by Zero1 12. See Figure for our reconstruction results.
models mainly focused on the task of text-to-image, recent work 97 24 has shown that fine-tuning pretrained models allows to add various conditional controls to the diffusion models and generate images based on specific conditions. Several conditions, such as canny edges, user scribbles, depth, and normal maps, have already proven effective (97]
The recent work Zero [41 shares a similar spirit and aims to add viewpoint condition control for the Stable Diffusion model [68]. Specifically, given single RGB image of object and relative camera transformation, Zero123 aims to control the diffusion model to synthesize a new image under this transformed camera view. To achieve this, 123 fine-tunes the Stable Diffusion on paired images with their relative camera transformations, synthesized from a large-scale 3D dataset [12]. During the creation of the fine-tuning dataset, Zero assumes that the object is centered at the origin of the coordinate system and uses a spherical camera, i.e,, the camera is placed on the sphere's is surface and always looks the origin. For two camera poses (01, 01,71) and (02, 22, r2), where 0g, Φ., and T3 denote the polar angle, azimuth angle, and radius, their relative camera transformation is parameterized
(02 01, -1, 72 rI). They aim learn model f. such that f( 02 - 0,, 22 Φ1, 12 TI) is perceptually similar to 22, where I and 22 are two images of object captured from different views. Zero123 finds that such fine-tuning enables the Stable Diffusion model to learn generic mechanism for controlling the camera viewpoints, which extrapolates outside of the objects seen in the fine-tuning dataset.
3.2 Can NeRF Optimization Lift Multi-View Predictions to 3D?
Given single image of an object, we utilize Zero123 44 to generate multi-view images, but can we use traditional NeRF-based or SDF-based methods |5] 80 to reconstruct high-quality 3D meshes from these predictions? We conduct small experiment test this hypothesis. Given single image, we first generate 32 multi-view images using Zero123, with camera poses uniformly sampled from the sphere surface. We then feed the predictions a NeRF-based method (TensoRF (53 and an SDF-based method (NeuS [80]. which optimize density and SDF fields, respectively. However, as shown in Figure 3 both methods fail to produce satisfactory results, generating numerous distortions and floaters. This is primarily due to the inconsistency of Zero Aero123"* 123's predictions. In Figure we compare Zero123's predictions with ground-trute renderings. We can see that the overall PSNR not very high, particularly when the input relative pose is large or the target pose is at unusual locations (e.g., from the bottom or the top). However, the mask IoU (most regions are greater than 0.95) and CLIP similarity are relatively good. This suggests that 7ero 123 tends to generate predictions that are perceptually similar to the ground truth and have similar contours or boundaries, but the pixel-level appearance may not be exactly the same. Nevertheless, such inconsistencies between the source views are already fatal to traditional optimization-base
methods. Although the original Zero 123 paper proposes another method for lifting its multi-view predictions, we will demonstrate in experiments that it also fails to yield perfect results and entails time-consuming optimization.
3.3 Neural Surface Reconstruction from Imperfect Multi-Viee Predictions
Instead of using optimization-bann based approaches, base our reconstruction module on a generalizable SDF reconstruction method SparseNeuS [45). which is essentially a variant of the MVSNeRF [6] pipeline that combines multi-view stereo, neural scene representation. and volume rendering. As
illustrated in Figure our reconstruction module takes multiple source images with corresponding camera poses as input and generates a textured mesh in a single beed-fowward pass. In this section, we will first briefly describe the network pipeline of the module and then explain how we train the
module, select the source images, and generate textured meshes. Additionally, in Section 3.4 discuss how we generate the camera poses for the source images.
we will
5

---

-90
PSNR
28
-80
mask loU
0.975
-00
CLIP Similarity
slevaton
60
5
80
0.950
etevatton
60
0.86
-30
22
-30
0.925
0.84
30
20
30
88
18
90
60
0.875
-180
110
120
150
.180
150
-120
.00
.80
720
150
0.850
190
-100
200
800
170
150
0.80
azimuth
azimuth
A azimuth
Figure 4: We analyze the prediction quality of Zero123 by comparing its predictions to ground truth renderings across various view transformations. For each view transformation, we report the average PSNR, mask IoU, and CLIP similarity of 100 shapes from the Objaverse [12] dataset. The prediction mask is calculated by considering foreground objects (i.e., non-white regions). Zero123 provides more accurate predictions when the view transformation is small.
As shown in Figure our reconstruction module takes m posed source images as input. The module begins by extracting m 2D feature maps using 2D feature network. Next, the module builds a 3D cost volume whose contents are computed by first projecting each 3D voxel to m 2D feature planes and then fetching the variance of the features across the m projected 2D locations. The cost volume is then processed using a sparse 3D CNN to obtain a geometry volume that encodes the underlying geometry of the input shape. To predict the SDF at arbitrary 3D point, an MLP network takes the 3D coordinate and its corresponding interpolated features from the geometry encoding volume as input. To predict the color of 3D point, another MLP network takes as input the 2D features the projected locations, interpolated features from the geometry volume, and the viewing direction of the query ray relative the viewing direction of the source images. The network predicts the blending weights for each source view, and the color of the 3D point is predicted as the weighted sum of its projected colors. Finally, an SDF-based rendering technique is applied on top of the MLP networks for RGB and mask rendering (80). In each iteration, we randomly choose one view to build the cost volume and another view for rendering supervision.
2-Stage Source View Selection and original SparseNeuS [45] paper only demonstrated frontal view reconstruction, we have extended it to reconstruct 360-degree meshes in a single feed-forward pass by selecting source views in a particular way. Specifically,
Seanndtruut-Predictinn Mixed Training. Although the
reconstruction model is trained on a 3D object dataset while freezing Zero123. We follow Zero123 to normalize the training shapes and use a spherical camera model. For each shape, we first render n ground-truth RGB images from л camera poses uniformly placed on the sphere. For each of the n views, we use Zero 13 to predict four nearby views. During training, we feed all 4 x n predictions with ground truth poses into the reconstruction module and randomly choose one of the n ground-trudh RGB images views as the target view. We call this view selection strategy as 2-stage source view selection. We supervise the training with both the ground-truth RGB and mask values. In this way, the module can learn to handle the inconsistent predictions from Zero123 and reconstruct consistent 360 mesh. We argue that our two-stage source view selection strategy is critical since uniformly choosing X 4 source views from the sphere surface would result in larger distances between the camera poses. However, cost volume-based methods (45] 29
typically rely on very close source views to find local correspondences.. Furthermore, as shown in Figure 4 when the relative pose is small (e.g.. 10 degrees apart), Zerol can provide very accurate and consistent predictions and thus can be used to find local correspondences and infer the geometry.
During training, we utilize ground-trute -tuth renderings in the initial stage. We find that employing predicted images at this stage would suffer from notable inconsistencies across different views, complicating the network's 's ability learn sharp details (see examples in ablation study). However, during inference, we can replace the n ground-trune -tutth renderings with Zero123 predictions, as shown in Figure 2 the network can automatically generalize to some extent. We will show in the experiments that this groundtruth-predie prediction mixed training strategy is also important. To export the textured mesh we use marching cubes [46] to extract the mesh from the predicted SDF field and query the color of the mesh vertices as described in [80]. Although our reconstruction module is trained on a 3D dataset, we find that it mainly relies on local correspondences and can generalize to unseen shapes very well.

---

Figure Qualitative examples of One-2-3-45 for both synthetic and real images. Each triplet showcases an input image, a textured mesh, and textureless mesh.
3.4 Camera Pose Estimation
Our reconstruction module requires camera poses for the 4 source view images. Note that we adopt Zero 123 for image synthesis, which parameterizes cameras in a canonical spherical coordinate frame, (0, o, r). where o and r represent the elevation, azimuth, and radius. While we can arbitrarily adjust the azimuth angle & and the radius r of all source view images simultaneously, resulting in the rotation and scaling of the reconstructed object accordingly, this parameterization requires knowing the absolute elevation angle 0 of one camera to determine the relative poses of all cameras in standard XYZ frame. More specifically, the relative poses between camera (0g. Φo. and camera (0o + 26, Φ + ΔΦ, ro) vary for different tn even when A and AΦ are the same. Because of this, changing the elevation angles of all source images together (e.g.. by 30 degrees up or 30 degrees down) will lead to the distortion of the reconstructed shape (see Figure 10 for examples).
Therefore, we propose an elevation estimation module infer the elevation angle of the input image. First, we use Zero 123 to predict four nearby views of the input image. Then we enumerate all possible elevation angles in a coarse-to-fine manner. For each elevation candidate angle. we compute the corresponding camera poses for the four images and calculate reprojection eгтor for this set of camera poses to measure the consistency between the images and the camera poses. The elevation angle with the smallest reprojection error is used to generate the camera poses for all 4 x n source views by combining the pose of the input view and the relative poses. Please refer to the appendix for details on how we calculate the reprojection error for set of posed images.
4 Experiments
4.1
Implementation Details
For each input image, we generate n 8 images by choosing camera poses uniformly placed on the sphere surface and then generate 4 local images (10" apart) for each of the 8 views, resulting in 32 source-view images for reconstruction. During training, we freeze the Zero123 41 model and train our reconstruction module on the Objaverse-LVIs [12] dataset, which contains 46K 3D models in 1,156 categories. We use BlenderProc 14 to render ground-trun -ruth RGB images. For images with background, we utilize an aff-the-shelf segmentation network SAM (33 with bounding-box prompts for background removal. Please refer to the appendix for more details.
4.2 Single Image to 3D Mesh
We present qualitative examples of our method in Figures 1 and 5 illustrating its effectiveness in handling both synthetic images and real images. We also compare One-2-3-45 with existing zero-shot single image 3D reconstruction approaches, including Point-E (5), Shap-E [30], 123 (Stable
7

---

input
Point-E
Shap-E
3DFuse Zero1 (rr013+-SD
Ours RealFusion
Figure 6: We compare One-2-3-45 with Point-E (57 Shap-E 33
Zero 123 (Stable Dreamfusion version) [4]) 3DFuse [72]. and RealFusion [88] In each example, we present both the textured and textureless meshes. As 3DFuse 772 and RealFusion (48) do not natively support the export of textured meshes, we showcase the results of volume rendering instead.
Table Quantitative Comparison on GSO and Objaverse 12] datasets.
90%
(degroe)
80% 13.6
F-Score Obj.
CLIP Similarity
Source
GSO
avg.
GSO
Obj.
avg.
Time
Point-E
57
internal
81.0
81.0
81.0
74.3
78.5
76.4
78s
Shap
3D data
83.4
81.2
82.3
79.6
82.1
80.9
27s
Zero 123+SD 1
75.1
69.9
72.5
71.0
72.7
71.9
115mmn
RealFusion
diffusion
2D
66.7
63.0
69.5
~90mi
Figure Error distribution of predicted elevations. The median and average are 5.4 and 9.7 degrees.
3DFuse
12
models
60.7
60.2
60.4
71.4
74.0
72.7
330mi
Ours
84.0
83.1
83.5
76.4
79.7
78.1
45%
Dreamfusion version) 41 3DFuse (72,, and RealFusion [48]. Among them, Point-E and Shap-E are 3D native diffusion models released by OpenAI, which trained on several million internal 3D
data, while others are optimization-base approaches leveraging priors from Stable Diffusion [68]
Figure 6 presents the qualitative comparison. While most methods can generate plausible 3D meshes from a single image, notable differences exist among them in terms geometry quality, adherence to the input, and overall 3D consistency. In terms of geometry quality. approaches like RealFusion [48] and 3DFuse [72]. which optimize a neural radiance field, face challenges in extracting high-quality meshes. Likewise, Point-E [57] produces a sparse point cloud its output, resulting in numerous holes on the reconstructed meshes. In contrast, our approach utilizes an SDF presentation and favors better geometry. Regarding adherence to the input, observe that most baseline methods struggle to preserve the similarity to the input image. Although Shap-E performs slightly better, it still produces lots of failure three legs). In contrast, our approach leverages a powerful 2D diffusion model to directly produce high-quality multi-vies view images, rather than relying on 3D space hallucination. This strategy provides
(see the backpack without shoulder straps, distorted shoe. and stool with

---

ours (Bx4)
all pred
all gt
no mask
31x1
4x4
7x1
6x4
8x3
Figure 8: Ablations on training strategies of the reconstruction module and the number of views.
better adherence to the input views, alleviates the burden of the 3D reconstruction module, and yields results that are more finely attuned to the input. Furthermore, many approaches encounter challenges in achieving consistent 3D results (also known as the Janus prob-
lem [48] 600), as highlighted in the right figure (two-handle mug, multi-face Mario, and two-face backpack). One of the contributing factors to this issue is that several methods optimize each view inde- pendently, striving to make each view resemble the input. In contrast, our method capitalizes on the view-conditioned 2D diffusion model, inherently enhancing 3D consistency.
Shap-E
Reee Fisson
ReeFFusion
Durs
Our
Oura
We also quantitatively compare the approaches on Objaverse 2 and GoogleScannedObjectt (GSO) [5] datasets. For each dataset, we randomly choose 20 shapes and render single im- age per shape for evaluation. To align the predictions with the ground-trute mesh, we linearly search the scaling factor and the rotation angle, apply Iterative Closest Point (ICP) for sampled point clouds, and select the one with the most number of inliers. We follow RealFusion (48) to report F-score (with threshold of 0.05) and CLIP similarity, and the runtime on 1100 GPU. As shown in Table [ our method outperforms all baseline approaches in terms of F-Score. As for CLIP similarity, we surpass all methods except concurrent work Shap-E (30. We find that CLIP similarity is very sensitive to the color distribution and less discriminative in local geometry variations (i.e., the number of legs of a stool, the number of handles of mug). Regarding running time, our method demonstrates notable advantage over optimization-bn based approaches and performs on par with 3D native diffusion models, such Point-E [57] and Shap-E [30]. Specifically, our 3D reconstruction module reconstructs 3D mesh in approximately 5 seconds, with the remaining time primarily spent on Zero 123 predictions, which takes roughly 1 second per image A100 GPU,
4.3 Ablation Study
Training strategies. We ablate our training strategies Figure We found that without our 2-stage source view selection strategy, network trained to consume 31 uniformly posed Zero123 predictions (fourth column) suffers from severe inconsistency among source views, causing the reconstruction module to fail completely. If we feed only 7 source views (sixth column) without the four nearby views, the reconstruction fails to capture local correspondence and cannot reconstruct fine-grained geometry. During training, we first render ground-truth renderings and then use Zero 23 to predict four nearby views for each of them. If we train directly on 8 x ground-truth renderings without Zero 123 prediction during training (second column), it fails to generalize well to Zero 123 predictions

---

SperseNouS
Ge
Ours (360 recon.)
Input
w/ pred. elew.
w/ gi elev.
w/ elev.
gt elev. -00 degree
(muttiviow fusion) jmuttivva fuskon)
+30 dogree
Vinni
WHIG
WARTEL
WARD
Figure 9: 360 reconstruction multi- view fusion. Meshes from different views in different colors.
Figure 10: Incorrect elevations lead to distorted re- construction. Our elevation estimation module can predict an accurate elevation of the input view.
Stable Dreamfusion
3D Fuse
Ours DALL-E2 2
Figure 11: Text to 3D. First row: "a bear in cowboy suit." Second row: "a kungfu cat." We utilize DALL-E [63] to generate an image conditioned on the text and then lift it to 3D. We compare our method with Stable Dreamfusion [60] and 3DFuse [72]. For baselines, volume renderings are shown.
during inference, with many missing regions. Instead, if we replace the n ground- ground-truth truth renderings with n Zero123 predictions during training (first column), the network fail to generate sharp details (see the strips of the backpack).
Elevation estimation. Our reconstruction module relies on accurate elevation angles of the input view. In Figure T0 we demonstrate the impact of providing incorrect elevation angles (e.g., altering the elevation angles of source views by +330), which results in distorted reconstruction results. Instead, utilizing our predicted elevation angles can perfectly match results with ground truth elevations. We also quantitatively test our elevation estimation module by rendering 1,700 images from random camera poses, As shown in Figure 7 our elevation estimation module predicts accurate elevations.
Number of source views. In Figure we also investigate the impact of varying the number of source views on 3D reconstruction. We observe that our method is not very sensitive to the number of views as long as the reconstruction module is retrained with the corresponding setting.
360 reconstruction vs. multi-view fusion. While our method reconstructs a 360 mesh in a single pass, most existing generalizable neural reconstruction approaches [45] 29) 6 primarily focus on frontal view reconstruction. An alternative approach is to independently infer the geometry for each view and subsequently fuse them together. However, we have observed that this strategy often struggles with multi-view fusion due to inconsistent Zero 123 predictions, as illustrated in Figure 9
4.4 Text to 3D Mesh
As shown in Figure 11 by integrating with off-the-shelf text-to-image 2D diffusion models 68 [$ method can be naturally extended to support text-to-ine exxt-t-immage-D tasks and generate high-quality textured meshes in a short time. See supplementary for more examples.
5 Conclusion
In this paper, we present novel method for reconstructing high-quality 360 mesh of any object from a single image of it. In comparison to existing zero-shot approaches, our results exhibit superior geometry, enhanced 3D consistency, and remarkable adherence to the input image. Notably, our approach reconstructs meshes a single forward pass without the need for time-consuming optimization, resulting in significantly reduced processing time. Furthermore, our method can be effortlessly extended to support the text-to-3ID task.
10

---

Acknowledgments
This work is supported in part by gifts from Qualcomm. We would like to thank Ruoxi Shi, Xinyue Wei, Hansheng Chen, Jiayuan Gu, Fanbo Xiang, Xiaoshuai Zhang, and Yulin Liu for their helpful discussions and manuseript proofreading.
We would like to thank the following sketchfab users for the models used for the demo images in this paper: dimaponomar2019 (backpack), danielpeng (bag), pmlzbt233 (wooden barrel), felixyadomi (cactus), avianinda (burger), shedmon (robocat), ie-niels (stool), phucn (armchair), techCIR (mug), sabriny (fox). All models are CC-By licensed.
References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018.
[2] Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Clipface: Text-guided editing of textured 3d morphable models. arXiv preprint arXiv:22/2.014p00 2022.
[3] Zehranaz Canfes, M Furkan Atasoy, Alara Dirik, and Pinar Yanardag. Text and image guided
3d avatar generation and manipulation. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 4421-4431, 2023.
[4] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich
3d model repository. arXiv preprint 15/2.030 2015.
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf:
Tensorial radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October
23-27, 2022, Proceedings, Puri XXXII, pages 333-350. Springer, 2022.
[6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-
14133, 2021.
[7] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprini arXiv:2303.11300
2023.
[8] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liangyan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. arXiv preprint arXiv:2212. 0449244444.222 2022.
[9] Han-Pang Chiu, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Automatic class-specific 3d reconstruction from a single image. CSA/I,, pages 1-9, 2009.
[10] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. 2023.
[11] Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2:
A unified approach for single and multi-view 3d object reconstruction. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628 644. Springer, 2016.
[12] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe annotated 3d objects. arXiv preprint arXiv:2212 08051, 2022.
[13] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. arXiv preprint urXiv:2212.0322
2022.
[14] Maximilian Denninger, Dominík Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias Humt, and Rudolph Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software, 8(82):4901, 2023.
[15] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high- quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation ( (ICRA), pages 2553-2560. IEEF, 2022.
[16] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605 005-613,400 2017.

---

[17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.
2022.
[18] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-311
2022.
[19] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part
VI 14, pages 484-499. Springer, 2016,
pages
[20] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-mâché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and paltern recogmition, pages 216-224, 2018.
[21] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation, arXiv preprint wrXiv:230300537
2023.
[22] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 4709, 2021.
[23] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205.00 08535,, 2022.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
AxNiv:2/0600966: 2021.
[25] Zixuan Huang. Stefan Stojanov, Anh Thai, Varun Jampani, and James M Rehg. Planes vs. chairs: Category-guided 3d shape learning without any 3d cues. Computer ViC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I, pages 727-744. Springer. 2022.
[26] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
[27] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12949-12958 2021.
[28] Nikolay Jetchev Clipmatrix: Text-controlled creation of 3d textured meshes. arXiv preprint
urXiv:2109.129)) 2021.
[29] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18365-18375, 2022.
[30] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305. 02.63,24433222 2023.
[31] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collectionssirdhar200 6learning........... In Proceedings of the European Conference on Computer Vision (ECCV). pages 371-386, 2018
[32] Nasir Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Text mesh without 3d supervision using limit subdivision. arXiv preprint arXiv:2203. 13333, 2022.
[33] Alexander Kirillov, Eric Mintun. Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,
Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint 1.ti:::2300.02444. 2023.
[34] Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviı, Israel, October 23-27, 2022, Proceedings, Part XV, pages 198-216. Springerr 2022.
[35] Han-Hung Lee and Angel X Chang. Understanding pure clip guidance for voxel grid nerf models. arXiv preprint arXiv:2209. /5/72, 2022.
[36] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440, 2022.
12

---

[37]
Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAl conference on artificial intelligence, volume 34, pages 11596-1 1603, 2020.
[38] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. arXiv preprint arXiv:23055 10764, 2023.
[39] Minghua Liu, Minhyuk Sung. Radomir Mech, and Hao Su. Deepmetahanddes: Learning deformation meta-handles of 3d meshes with biharmonic coordinates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12-21, 2021.
[40] Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained amage-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21736-21746, 2023.
[41] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. arXiv preprint 2rxiv22333.11322 2023.
[42] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang. Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image- maage-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
7824-7833, 2022.
[43] Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu, Iss: Image as stetting stone for text-guided 3d shape generation. arXiv preprint urXiv:2209.04/ 045, 2022.
[44] Zhengzhe Liu, Peng Dai. Ruihui Li, Xiaojuan Qi, and Chi-Wing Fu. Iss++: Image stepping stone for text-guided 3d shape generation. arXiv preprint arXiv:2303. 15181, 2023.
[45] Xiaoxiao Long, Cheng Lin, Peng Wang. Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI1, pages 210-227. Springer, 2022.
[46] William E. Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 2134):163-169, 1987. [47] Oier Mees, Maxim Tatarchenko, Thomas Brox, and Wolfram Burgard. Self-supervised
3d shape and viewpoint estimation from single images for robotics. In 20/9 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 6083-6088 IEEE, 2019.
[48] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360
{ {'deg) reconstruction of any object from a single image. arXiv preprint 2rX2::2302.10663,
2023.
[49] Luke Melas Karrazi,,,, Christian Rupprecht, and Andrea Vedaldi. pc2:
Projection conditioned point cloud diffusion for single- -mage 3d reconstruction. arXiv preprint 2:Xiv:2302. 2023,
[50] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pages 4460-4470,
2019.
[51] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint (tti::2//1007600 2022.
[52] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13492-13502, 2022.
[53] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[54] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022.
[55] Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Buld, Matthias Nießner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3971-3980, 2022.
[56] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning, pages 7220-
13

---

7229. PMLR, 2020.
[57] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
[58] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174,
2019.
[59] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19975-10983 2019.
[60] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. arXiv preprint arXiv:2209. 12..29999480 /4988, 2022.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[62] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprini urXiv:2303. 12.30...300 13508, 2023.
[63] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061
2022. [64] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- text-to-image generation. In International Conference on Muchine Learning, pages 8821-8831. PMLR, 2021.
[65] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life
3d category reconstruction. In International Conference on Computer Vision, 2021.
[66] Yufan Ren, Fangjinhua Wang, Tong Zhang, Mare Pollefeys, and Sabine Süsstrunk. Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction.
arXiv preprint arXiv:22 2 2.08667, 2022.
[67] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.0172 2023.
[68] Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages (0684-10695 2022. [69] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image imaee diffusion models with deep language understanding. arXiv preprint arXiv:2205. 14477 2022.
[70] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao
Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304-2314 2019.
[71] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin- Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 18603-18613, 2022.
[72] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin- Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency
for robust text-to-3d generation. arXiv preprint 1rxiv:22003.0999
2023.
[73] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 133++: single image consistent multi-view diffusion base model. arXiv preprint arXiv:2310. 10/223105///11 15/10, 2023.
[74] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8931, 2021.
[75] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang. Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3dt High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023.
14

---

[76] Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15122-55192, 2021.
[77] Mukund Varma, Peihao Wang. Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations, 2022.
[78] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint
arXiv:2212.00 2,0774, 2022.
[79] Nanyang Wang. Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52-67, 2018.
[80] Peng Wang, Lingji Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for milti-view reconstruction. arXiv preprint arXiv:2 10689, 2021.
[81] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo MaatinnBruullle Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[82] Jiacheng Wei, Hao Wang, Jiashi Feng, Guosheng Lin, and Kim-Hui Yap. Taps3d: Text-guided
3d textured shape generation from pseudo supervision, 2023.
[83] Chao Wen, Yinda Zhang, Zhuwen Li, and
Fu. Pixel2mesh++: Multi-view 3d mesh generation via deformation. In Proceedings of the IEEE/CVF international conference computer vision, pages 1042-1051, 2019.
[84] Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction. arXiv preprini arXiv:2301 100847, 2023.
[85] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. Marmet: 3d shape reconstruction via 2.5 d sketches. Advances in neural information processing systems, 30, 2017.
[86] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangehen Zhou, and Shengping Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the JEEE/CVF international conference on computer vision, pages 2690-2698, 2019.
[87] Haozhe Xie, Hongxun Yao, Shengping Zhang, Shangchen Zhou, and Wenxiu Sun. Pix2vox++: Multi-scale context-aware 3d object reconstruction from single and multiple images. Interna- tional Journal of Computer Vision, 128(12):2919 (129)2919-2999 2935,, 2020.
[88] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift- 360: Lifting an in-the-wild 2d photo to a 3d object with 360 {'dee] views. arXiv preprint vrXiv:2211.1644
2022.
[89] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot rext-to-3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:2212. 14704, 2022.
[90] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Uirich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in neural information processing systems, 32, 2019.
[91] Farid Yagubbayli, Yida Wang, Alessio Tonioni, and Federico Tombari. Legoformer: Trans- formers for block-by-bloc multi-view 3d reconstruction. arXiv preprint arXiv:2106.12100
2021.
[92] Daniel Yang, Tarik Tosun, Benjamin Eisner, Volkan Isler, and Daniel Lee. Robotic grasp- ing through combined image- basee grasp proposal and 3d reconstruction. In 2021 IEEE International Conference on Roboties and Automation (ICRA). pages 3350-6356. IREE, 2021.
[93] Hao Yang. Lanqing Hong, Aoxue Li, Tianyang Hu, Zhenguo Li, Gim Hee Lee, and Liwei Wang. Contranerf: Generalizable neural radiance fields for synthetic-to-real novel view synthesis via contrastive learning. arXiv preprint arXiv:2303.J 1052, 2023.
[94] Yaoqing Yang. Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 206-215, 2018
[95] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4577-4587.1 2021.
[96] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint
15

---

UrXiv:2210.06978, 2022.
[97] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[98] Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radi- ance fields for large-scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449-5455 2022.
[99] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling niw-conditioned diffusion for
3d reconstruction. In CVPR, 2023.
[100] Silvia Zuffi, Angjoo Kanazawa, and Michael Black. Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3955-3963, 2018.
[101] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose animals. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1365-63733 2017.
16

---

Appendix
We first show more qualitative comparison Section which is followed by demonstration of additional examples on real-world images and the text-to-3D task in Sections B and C Cespectivelyy Furthermore, we present the details of our elevation estimation module in Section D training and evaluation details in Section Section
We finally show the failure cases and discuss the limitations in
A More Qualitative Comparison
PornEE
Zaro123-80
RealFunton 3DFuse
Zero 123 (Stable Dreamfusion version) [4]], 3DFuse (72), and RealFusion 48 In each example, present both the textured and textureless meshes. As 3DFuse 772 and RealFusion [48] do not natively support the export of textured meshes, we showcase the results of volume rendering instead.
Figure 12: We compare One-2-3-45 with Point-E (57
Shap-E 30
In Figure 12 demonstrate more qualitative comparison Objaverse [12 and GoogleScannedOb- jects (GSO) [5] datasets. Note that all test shapes are not seen during the training of our 3D reconstruction module.
B More Examples on
Images
In Figure 13 we showcase more examples on real-world images and compare our method with the concurrent method Shap-E (30]. The input images are from unsplash. com captured by ourselves. Note that our results exhibit a closer adherence to the input image.
C More Examples on Text-to-3D
In Figure 14 we present additional examples for the text-to-3D task. It is evident that existing approaches struggle to capture fine- tneeggained details, such as tree hollow, or achieve compositionality. as seen examples like an orange stool with green legs, a pineapple-shaped Havana hat, or rocking horse chair. In contrast, our method produces superior results that adhere more closely to the input text. We hypothesize that controlling such fine-grained attributes in the 3D space using existing optimization strategies is inherently challenging. However, by leveraging established 2D
17

---

Ours
Shap-E
Ours
Shap-E
Figure 13:
We compare One-2-3-45 with Shap-E (30) on real-world images. In each example, present the input image, generated textured and textureless meshes.
text-to-image diffusion models, our method becomes more effective in lifting single 2D image to corresponding 3D textured mesh.
D Details of Elevation Estimation
To estimate the elevation angle 0 of the input image, we first utilize Zero123 41 to predict four nearby views (10 degrees apart) of the input view. With these predicted views, we proceed to enumerate all possible elevation angles and compute the re-projection error for each candidate angle.
18

---

DALL-E
Ours
Stable DreamFusion
3D Fuse
woodan apple.
A blueberry ice cream wattle
free with hallow.
An orange stool with
pineapple-shaped Havana hat.
A wooden mushroom.
A rocking horse chair.
blue spherical mug
monkay dressed white robe playing a wizard.
bear cowboy suit.
A kungfu cat.
Figure 14: Text-to-3D: We compare our method against two native text-to-3D approaches Stable DreamFusion (60) and 3DFuse (72.. To enable text-to-3D, our method first uses pretrained text-to- image model DALL-E 2 (63 to generate an image from input text (prompted with "3d model, long shot"), and then uplifts the image 3D textured mesh.
The re-projection error assesses the consistency between camera poses and image observations, akin to the bundle adjustment module employed in the Piructue-from--Me (SfM) pipeline.
Specifically, we enumerate all candidate elevation angles in a coarse-to-fine manner. In the coarse stage, we enumerate elevation angles with 10-degree interval. Once we have determined the elevation angle associated with the smallest re-projection error, we proceed to the fine stage. this stage, enumerate clevation angle candidates ranging from e* 10" to e* + 10 with I 1-degree
19

---

interval. This coarse-to- torsie design facilitates rapid estimation, completing the elevation estimation module in under 1 second for each shape.
Given a set of four predicted nearby views, we perform feature matching to identify corresponding keypoints across each pair of images (a total of six pairs) using an off-the-shelf module LoFTR (74). For each elevation angle candidate, we calculate the camera pose for the input image by employing the spherical coordinate system with a radius of 1.2 and an azimuth angle of O. Note that the azimuth angle & and the radius can be arbitrarily adjusted, resulting in the rotation and scaling of the reconstructed object accordingly. Subsequently, we obtain the camera poses for the four predicted views by incorporating the specified delta poses.
Once have the four posed images, compute the re- re-projection error by enumerating triplet images. For each triplet of images (a, b, e) sharing a set of keypoints P. we consider each point p E P. Utilizing images and b, perform triangulation to determine the 3D location of p. We then project the 3D point onto the third image e and calculate the reprojection error, which is defined as the l1 distance between the reprojected 2D pixel and the estimated keypoint in image c. By enumerating all image triplets and their corresponding shared keypoints, we obtain the mean projection error for each elevation angle candidate.
E
Details of Training and Evaluation
Training We train the reconstruction module using the following loss function:
L Lrgb + X1 Leskonal + Xeeosprriity
(1)
where Ligh represents the loss between the rendered and ground truth color, weighted by the sum of accumulated weights; Leikonal and Lxparsity are the Eikonal and sparsity terms, respectively, following SparseNeuS [45]. We empirically set the weights as A 1, A1 = 0.1, and A = 0.02.
For λ2, adopt a linear warm-up strategy following SparseNeuS 45 To train our reconstruction module, we utilize the LVIS subset of the Objaverse [12] dataset, which consists of 46k 3D models across 1,156 categories. The reconstruction module is trained for 300k iterations using two A10 GPUs, with the training process lasting approximately 6 days. It is important to note that our reconstruction module does not heavily rely on large- scale training data, as it primarily leverages local correspondence to infer the geometry, which is relatively easier to learn and generalize.
Evaluation We evaluate all baseline approaches using their official codebase. Since the approaches take only a single image as input, the predicted mesh may not have the same scale and transformation as the ground-truth mesh. To ensure a fair comparison, we employ the following process to align the predicted mesh with the ground-truth mesh First, we align the up direction for the results generated by each approach. Next, for each generated mesh, we perform linear search over scales and rotation angles along the up direction. After applying each pair of scale and z-rotation, we utilize the Iterative Closest Point (ICP) algorithm to align the transformed mesh the ground-truth mesh. Finally, we select the mesh with the largest number of inliers as the final alignment. This alignment process helps us establish a consistent reference frame for evaluating the predicted meshes across different approaches. To calculate CLIP similarity, we render both ground-truth and generated meshes, capturing 24 views around the 3D shape from fixed viewpoints 12 views 30 clevation and 12 views at 0 elevation.
F Failure Cases and Limitations
Our method relies on Zero123 for generating multi-view images, which introduces challenges due to its occasional production of inconsistent results. In Figure [5] we present two typical cases that exemplify such inconsistencies. The first case involves an input view that lacks sufficient information, such as the back view of a fox. In this scenario, Zero123 23 struggles to generate consistent predictions for the invisible regions, such as the face of the fox. As a consequence, our method may encounter difficulties in accurately inferring the geometry for those regions. The second case involves an input view with ambiguous or complex structures, such as the pulp and peel of a banana. In such situations, Zero1 123's s ability to accurately infer the underlying geometry becomes limited. As a result, our method may be affected by the inconsistent predictions generated by Zero123. It is important to acknowledge that these limitations arise from the occasional scenarios, and they can impact the
20

---

input
zero 23 prediction
our recon.
imput
zero 128 prediction
our recon.
(a)
(b)
Figure 15: Failure cases. Our method relies on Zero123 to generate multi-view images, and we encounter challenges when Zero123 generates inconsistent results. (a) The input view lacks sufficient information. (b) The input view contains ambiguous or complicated structures.
performance of our method in certain cases. Addressing these challenges and refining the reliability of Zerol23's predictions remain areas for further investigation and improvement.
We have also noticed slight artifacts the back side of our generated results. As one of the first works in combining view-conditioned 2D diffusion models with generalizable multi-view reconstruction, we believe that there is still ample room for exploring more advanced reconstruction techniques and incorporating additional regularizations. By doing so, we expect to significantly mitigate the minor artifacts and further enhance results in the future.
21


Reviews:
[{"review_id": "gLxbzwjJVb", "reviewer": "Reviewer_Ph7H", "summary": "This paper introduces an efficient method for single-image 3D reconstruction that significantly improves upon previous techniques by generating textured mesh in one feed-forward pass. The authors employ a view-conditioned 2D diffusion model, Zero123, to create multi-view images from a single input view and then lift these images into 3D space using SparseNeuS. The proposed technique is much faster than existing methods, while reserving 3D consistency.", "questions": "See the Weaknesses above.", "limitations": "Despite the concerns raised in the weaknesses section, a positive review score is given in recognition of the effort to formulate a single-pass pipeline and for the considerable credit on the superior generation efficiency.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method enables the generation of 3D meshes in one feed-forward pass after training. Due to its training-based, one-pass approach, the generation speed significantly outperforms many concurrent optimization-based methods.\n\nThe quality of writing in the introduction and related work sections is commendable. The related work section covers the most recent work in the field and is well-organized.", "weaknesses": "The authors propose a multi-stage method primarily built upon the combination of multiple existing works. The work seems more engineering-oriented than technically innovative. Many aspects of the methodology, such as elevation estimation, 2-Stage Source View Selection, and Groundtruth-Prediction Mixed training, focus on bridging various components borrowed from existing baseline models.\n\nAdditionally, as the method relies on Zero123 and SparseNeuS, will it inevitably inherit their limitations? Can this method circumvent the failure cases of Zero123 or SparseNeuS? The method 12345 is trained on 1,156 categories from the Objaverse-LVIS dataset. How does 12345 perform when the images fall outside these 1,156 categories? Many concurrent image-to-3D works can generalize to uncommon objects due to their large diffusion models. Could 12345's training in aligning Zero123 and SparseNeuS diminish the zero-shot capability?\n\nThe authors provide limited visualization cases in their ablation study. Including quantitative results to support the ablation study would be beneficial, as 3D generation is typically unstable and hard to reproduce.\n\nThe organization and presentation of the methodology section could be improved. The method involves multiple stages, but the authors do not strictly adhere to the method structure order and insert discussions throughout. This makes sections 3.1 and 3.3 difficult to follow.\n\nFrom the visualization, the mesh generation quality is not so impressive. \n\nCould the authors explain their motivation for learning SDF and RGB from two separate MLPs? Many NeRF papers typically predict these from the same network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Perform outside 1,156 categories?**\n\nYes, our method can generate meshes for images  beyond the scope of the 1,156 training categories, as shown in Figure 1 of the attached PDF (see main rebuttal). Our 3D reconstruction module inherently leverages local correspondences for reconstruction and is thus designed to have a robust generalization ability to unseen data. This characteristic ensures that our method retains its zero-shot capability.\n\n**Circumvent the failure cases of Zero123 or SparseNeuS?**\n\nOur method demonstrates a certain level of tolerance towards the inconsistencies present in Zero123's multi-view predictions. In contrast, traditional optimization-based NeRF methods fail completely, as shown in Figure 3.\n\nWe agree that our approach may also fail for the severe failure cases of Zero123. However, our method is not tightly coupled with Zero123. We believe that more progress will be made in 2D diffusion models for multi-view prediction,  which will enhance our approach.\n\n**Bridging various components**\n\nIt’s nontrivial to combine 2D diffusion models and 3D reconstruction, and there exist many challenges. As highlighted by Reviewer L2hY, we demonstrate originality through our well-motivated and smart design choices and the underlying reasoning.\n \nAlso, we would like to emphasize our pivotal contribution in proposing a new paradigm for generalizable single-image 3D generation, which effectively addresses the limitations inherent in existing paradigms, such as efficiency and 3D consistency issues for optimization-based methods and poor generalization for 3D native generative models.\n\n**Limited examples in ablation studies**\n\nThanks for the suggestion. We will add more qualitative examples and quantitative results in our revised version of supplementary materials.\n\n**Why two separate MLPs?**\n\nMany NeRF papers use two separate MLPs [1,2,3] because the SDF/density branch only takes XYZ coordinates as input, while the RGB branch may include both XYZ coordinates and view directions. It is common for them to share some feature networks (same as ours), but the final decoders are usually different.\n\n\n\n**Generation quality is not so impressive?**\n\nWe improved our reconstruction module and fixed some notable artifacts on the backside. We will update our results in the revision.\n\n\n\n\n[1] Chen, Anpei, et al. \"Tensorf: Tensorial radiance fields.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Mildenhall, Ben, et al. \"Nerf: Representing scenes as neural radiance fields for view synthesis.\" Communications of the ACM 65.1 (2021): 99-106.\n\n[3] Wang, Peng, et al. \"Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.\" arXiv preprint arXiv:2106.10689 (2021)."}, {"review_id": "gI8ru3QLNY", "reviewer": "Reviewer_L2hY", "summary": "This paper tackles the task of single image 3D reconstruction, where given a single input image of an object, the proposed method aims to generates a 360 degree 3D textured mesh. \nCompared to prior works tackling a similar problem setting, this paper primarily aims to improve the quality of the reconstruction and reduce the time costed during inference time. \n\nThe proposed method contain 3 modules: (1) an off-the-shelf view-conditioned 2D diffusion model to generate multi-view images from a single input image, (2) a camera pose estimator, (3) an existing SDF-based multi-view reconstruction method to lift these images into 3D space. \nThe method is evaluated qualitatively and quantitatively on synthetic and real data, and outperforms most baselines. \n", "questions": "In general I find this paper above the acceptance bar. The questions below aim to elicit additional clarification and insights of the paper: \n\n- If allowing an additional test-time optimization during inference time, will it further improve the quality of the outputs? (Or will it get harmed by the inconsistent multi-view images?)  \n\n- The authors mentioned the proposed method suffers less from the multi-face issue (the Janus problem). Is it a benefit of the prior work Zero123 or from original design of the proposed method?   \n\n- Minor:   \nL.98 generates -> generate  \nDescription in L.150 “uses a spherical camera” and L.203 “a spherical camera model” is ambiguous. ", "limitations": "The paper discusses failure cases in supp, including inconsistent multi-view images from Zero123, and artifacts on the back side. I encourage the authors to also explicitly mention in the main paper. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\nOriginality: \n\n- This paper is well positioned in the literature. It properly summarizes and tackles existing limitations of related works with a similar task setting. \n- This proposed method is a smart combination of existing methods, and the proposed design choices are well motivated. \n\nClarity:\n- The writing is clear and easy to follow. \n\nQuality: \n\n- The qualitative and quantitative results outperform baselines and achieves SOTA. It still has a significant room for improvements. But it’s acceptable considering the challenges of the task. \n\n\nSignificance: \n\n- The proposed method presents a novel idea for the literature -- incorporate a feed-forward model for diffusion-based methods. ", "weaknesses": "- The novelty of the proposed method is relatively limited, as it heavily relies on prior works. Specifically, two out of the three modules of the proposed method (Zero123 and SparseNeuS) are built upon existing approaches. However, the authors do demonstrate originality through their design choices and the underlying reasoning for the combination of existing ideas. \n\n\n- The proposed method contains several off-the-shelf or disconnected modules. It could have been beneficial to provide more discussion and analysis on how errors of one module affect the final performance. \nFor example, pose estimation is a task that is not completely solved, how will the pose estimation error affect the pipeline? For inference of real world images, how will the inaccuracy of segmentation masks affect the final quality? \n\n\n- Although the quality of the final results achieved by the proposed method can be considered state-of-the-art, there is still significant room for improvement. From the qualitative results, most reconstructed 3D model is far from the quality of the input image. It can be helpful to expand the discussion of limitation to analyze why these artifacts can happen (e.g. blurriness, distorted shape). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**How do errors in one module affect the final performance?**\n\nCorrect segmentation mask and pose estimation are important for the final generation quality, as ablated in Figure 10. \n\nProvided that the multi-view prediction is accurate, our pose estimation module is inherently capable of estimating precise elevation angles. For example, if we are given four nearby ground truth views, the module should be able to estimate the pose accurately in most cases. Although the median error of the current estimation module is 5 degrees, we attribute this error primarily to the inconsistency of the multi-view prediction, which can be regarded as the internal error of Zero123 and can be improved by a stronger multi-view prediction module.\n\nThe current segmentation models (e.g., SAM) are very powerful. They can generate very accurate segmentation masks with a small amount of human input. As a result, we don’t feel the segmentation module can be the bottleneck.\n\nInstead, the main bottleneck remains with Zero123 and the reconstruction module. While our method is somewhat tolerant of inconsistencies of multi-view predictions, our approach may still fail for the severe failure cases of Zero123. However, our method is not tightly coupled with Zero123. Also, Zero123 is still the first few attempts in the direction of multi-view prediction. We believe that more progress will be made on 2D diffusion models for multi-view prediction, which will also enhance our method.\n\n**Reasons for the artifacts**\n\nWe have observed some systematic artifacts in the original results, such as waffle-like textures on the backside. We find that these can be fixed by replacing the depth loss with the mask loss during the training of the reconstruction module. We will update our results in the revision. For other types of artifacts (e.g., distortion and blurriness), we feel that the inconsistency of multi-view predictions mainly causes them since the results are much better when feeding with ground-truth multi-view renderings. We also improved the mesh rendering script used in the original submission so that the rendering can better reflect the colors of the generated mesh instead of dimming it out.\n\n**Additional test-time optimization**\n\nThere are two ways to add test-time post-optimization. The first one is fine-tuning with multi-view predicted images as done in the original SparseNeus. However, we find that it typically does not improve the results due to underlying inconsistencies among multi-view predictions, as shown in Figure 2 of the PDF (see main rebuttal).\n\nAnother strategy is to leverage priors from 2D diffusion models (e.g., StableDiffusion) to further fine-tune the generated mesh as in DreamFusion. Our generated results can serve as a good initialization and accelerate the optimization. While NeRF representation and volume rendering is used in the original DreamFusion to better handle topological changes, in our fine-tuning, mesh representation and surface rendering can also be used to reduce the rendering time since there are no significant topological changes.\n\n**Reasons for less multi-face issue**\n\nYes, less multi-face issues mainly benefit from view-conditioned 2D diffusion (e.g., Zero123). Our reconstruction module also tolerates and fixes some small inconsistencies in Zero123's predictions.\n\n**Limitations**\n\nThanks for your suggestion. We will move the limitation from the supplementary to the main text.\n"}, {"review_id": "Q3VxklQKPR", "reviewer": "Reviewer_AK29", "summary": "The proposed method overcomes the challenges of lengthy optimization time, 3D inconsistency, and poor geometry that are common in existing methods. This method uses a view-conditioned 2D diffusion model, Zero123, to generate multi-view images from a single input image, and then lifts these images to 3D space. The 3D reconstruction is based on an SDF-based neural surface reconstruction method, with several training strategies proposed to enable the creation of 360-degree meshes. This method is faster, produces better geometry, and generates more 3D consistent results than existing methods. It has been evaluated on synthetic data and real-world images, demonstrating superior mesh quality and runtime. Additionally, it can be integrated with text-to-image diffusion models to support the text-to-3D task.", "questions": "Going forward, which is a better path for 3D generation, novel view synthesis + mvs (like in this paper) or 3D data-driven (like in Shape-E)?", "limitations": "The issue of multi-view inconsistency is not addressed but bypassed. It would be nicer to have a method that is more fundamental to addressing this issue.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main idea of this paper is to combine the view synthesis method Zero123 with the multi-view stereo (MVS) method SparseNeuS for 3D generation. This simple approach provides interesting insights and results, as it enables fast 3D generation that visually outperforms sophisticated 2D diffusion+NeRF methods like Dreamfusion, and also appears to be superior to other data-driven 3D generation methods such as Point-E and Shape-E.\n\nThe contribution of this paper is valid considering the advantages of runtime and better visual results.", "weaknesses": "- While simple and effective, the idea of the paper is not that eye-opening in the sense that it seems to be a natural extension or improvement of Zero-123.\n\n- The performance of the proposed method is upper-bounded by the Zero-123. If Zeros-123 fails, it seems that there is no way for this method to generate reasonable output as well. In some sense, the problem of multi-view inconsistency is not solved but bypassed via the use of SparseNeuS.\n\n- The area is advancing fast. Follow-ups of DreamFusion such as Magic3D and Fantasia3D have already achieved big improvements. It's not sure how the proposed method compares with more recent 2DDiffusion+NeRF methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Natural extension or improvement of Zero123?**\n\nWe would like to emphasize our contribution in proposing a new paradigm for generalizable single-image 3D generation, which overcomes the limitations of existing paradigms, such as efficiency and 3D consistency for optimization-based methods and poor generalization for 3D native generative models.\n\nIt’s nontrivial to extend Zero123 for 3D reconstruction in a feed-forward manner. Even in Zero123’s paper, they utilize optimization-based methods for 3D reconstruction. While it seems natural to combine Zero123 and generalizable NeRF methods, there still exists multiple challenges when combining them: (a) original SparseNeuS only focuses on frontal-view reconstruction while we need 360-degree full mesh reconstruction; (b) original SparseNeuS only considers  consistent multi-view images as input instead of inconsistent multi-view predictions; (c) we need the camera poses of the input view for 3D reconstruction.\n\nTo overcome these challenges, we propose several critical training strategies and a pose estimation module. As highlighted by Reviewer L2hY, we demonstrate originality through our well-motivated and smart design choices and the underlying reasoning.\n \n**Multi-view inconsistency is not solved but bypassed.**\n\nWe agree that multi-view inconsistency is still the main bottleneck and has not been fully addressed. However, our method is not tightly coupled with Zero123. Also, Zero123 is still the first few attempts in the direction of multi-view prediction. We believe that more progress will be made on 2D diffusion models for multi-view prediction, and any updates will also improve our method. Except when perfectly consistent multi-view prediction is possible, our proposed method is still effective and can be used to deal with small inconsistencies.\n\n**Comparison with more recent methods**\n\nWe would like to point out that many 2D Diffusion+NeRF methods (including Magic3D and Fantasia3D) mainly support text conditions and cannot support the single image to 3D task natively. In the paper, we showed the result of Zero123+Stable DreamFusion, which differs from the original DreamFusion. Moreover, Magic3D has not released its code and uses an internal 2D diffusion model. \n\nWe also would like to mention that we have compared the proposed method with most of the recent single image-to-text works: Point-E (Dec 22), RealFusion (March 23), 3DFuse (March 23), Zero123 (March 23), and even Shap-E (May 5).\n\n**Which is a better path for 3D generation**\n\nThis is a great open question. It’s hard to say which path will dominate the task, as different applications may pose different requirements. We feel that multi-view prediction + 3D reconstruction (like in our paper) is a promising direction for open-world and time-sensitive applications. On the one hand, 3D native generative models (e.g., Shap-E) suffer from limited 3D data, while our method can benefit from richer 2D priors (e.g., millions of 3D shapes vs billions of 2D images), which makes open-world applications more possible. On the other hand, our method is much more efficient and better reserves 3D consistency than optimization-based methods (e.g., DreamFusion), which is important in time-sensitive applications (e.g., AR/VR, robotics, social media apps).\n\nIt would be beneficial to combine various paths. For example, by using the generated results of our method as initialization, we can leverage optimization-based methods to fine-tune the results further and add more fine-grained details. With good initialization, optimization time should be significantly reduced, and 3D inconsistency issues should no longer exist. Furthermore, suppose we care about the internal structure of 3D shapes, it is better to combine our method with 3D native generative models since our method and optimization-based methods currently focus on the shape surface and cannot reconstruct the internal structure of a shape.\n"}, {"review_id": "UCrhxlegGD", "reviewer": "Reviewer_yirP", "summary": "The authors tackle the problem of 3D reconstruction from a single image, which is challenging due to the lack of 3D information. They propose a novel method that combines a 2D diffusion model, Zero123, with a cost-volume-based 3D reconstruction technique, SparseNeuS, to generate a 360-degree 3D textured mesh in a feed-forward pass. They also estimate the elevation of the input shape and introduce several training strategies to improve the consistency and quality of the 3D mesh. Their contributions are:\n\n* A novel method that leverages 2D prior models for 3D modeling without per-scene optimization.\n* An elevation estimation module that computes the camera poses required by the reconstruction module.\n* A series of essential training strategies that enable the reconstruction of 360-degree meshes from inherently inconsistent multi-view predictions.", "questions": "* Would it be possible to do a human evaluation with hundreds of 3D assets comparing your results with that of Shap-E? This would enhance the quality of the current evaluation.\n\n* Would it be possible to report numerical scores for hundreds of generated 3D assets rather than 20? To the same extend, can you report what the assets used were and whether they exist in Objaverse-LVIS?\n\n* Objaverse created a licensing issue when it came out with many artists requesting to retract their 3D assets from the dataset. Did the authors respect this by checking the no-AI flag in Sketchfab for both training and evaluation?\n\n+ Comments in Weaknesses ", "limitations": "Yes to some extend", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The work tackles 3D reconstruction from a single image, which is challenging and useful. The work can handle potentially any object category and generate a full 3D mesh from a single image.\n* The proposed method achieves high-quality 3D reconstruction in a feed-forward manner without optimization, which is faster and more efficient. The work does not require per-scene optimization but relies on a single feed-forward pass to generate the 3D mesh and as such it is much faster than existing methods.\n* The work leverages 2D prior models for 3D modeling, which is somewhat novel. It uses a 2D diffusion model to synthesize multi-view images of the input, and then uses multi-view 3D reconstruction techniques to obtain a 3D mesh.", "weaknesses": "* The evaluation section is noticeably light. The authors used 20 shapes from GSO and Objaverse to report F-Score and CLIP similarity. The number of shapes (20) is too low for any meaningful comparison considering that the method was trained with 46k 3D assets.  \n\n* The paper is framed that the work solves the image to 3D problem, which is true to some extend. In reality the work solve the problem of 3D lifting a single object, which is a much more constrained problem. I would advise the authors to change their wording to a more accurate description of the task.\n\n* The work mentions that it used 46k Objaverse assets for training, i.e. the whole Objaverse-LVIS dataset . All figures have Objaverse-LVIS assets and they keep repeating in the figures (backpack, super mario, horse, minion etc). A quick search show nearly all of them, for example this is the backpack: https://skfb.ly/6XCoS . All other can be found with search on the website https://objaverse.allenai.org/explore . On the contrary, there a minimal results on assets not from objaverse which rises the question whether the reconstructions we are seeing in the paper are assets from the training set. \n\n* The work has showcases a lot of licensed assets like pokemon, Super-mario and others. These assets require a licensing agreement with the respect companies that hold the rights. It might be sensible to replace them. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Number of shapes for evaluation**\n\nWe first would like to clarify that we used 40 shapes (20 from GSO and 20 from Objaverse, excluding the LVIS subset) instead of 20 shapes for evaluation. The limited number of evaluation shapes is mainly because many baseline algorithms are very time-consuming. For example, RealFusion takes ~90 minutes to generate a single shape on an A100 GPU. It's not practical to evaluate them on hundreds of shapes (e.g., it will take 2 A100 weeks for RealFusion to generate 200 shapes). Faced with these challenges, recent 3D AIGC papers have yet to quantitatively evaluate the method on large-scale datasets. For example, RealFusion evaluates the methods on only 21 shapes (7 categories) with one baseline, while we have 40 shapes and five baselines.\n\n**\"Only reconstruct single objects, not single image to 3D?\"**\n\nWe agree that our current method mainly focuses on the generation of a single object. We have explicitly stated that in the abstract (line 6) and introduction (line 32). We will follow your suggestion to make the wording more rigorous.\n\n**”Results shown in the paper are from the training set?”**\n\nWe would like to clarify that all results shown in the paper and supplementary are not seen during training of One-2-3-45, as stated in the supplementary. We split the Objaverse-LVIS dataset into training and validation sets and used only 43k of the 46k shapes for training. The backpack image is not from the LVIS subset. The images of super mario, the horse, and the minion are from somewhere other than the Objaverse dataset.\n\n**Report numerical scores for hundreds of generated 3D assets?**\n\nSince it's computationally expensive to evaluate optimization-based methods on hundreds of shapes, we add a comparison between the proposed method, Point-E, and Shap-E. We evaluate methods on 200 shapes from the GSO dataset, which doesn't contain Objaverse-LVIS shapes. The F-score for our method, Point-E, and Shap-E are 93.9, 91.5 and 93.3, respectively. The CLIP score for our method, Point-E, and Shap-E are 67.6, 65.1, and 68.7, respectively.\n\n**Objaverse license**\n\nThanks for pointing this out. We utilize shapes with CC-BY 4.0 license for training and will add attribution in our paper.\n"}, {"review_id": "2OE8dftWZ9", "reviewer": "Reviewer_eL63", "summary": "This paper introduces a novel approach for converting a single image into a 3D textured mesh without the necessity for per-shape optimization. The method employs Zero-123 for generating novel views of the input image, which is then utilized by a generalizable MVSNeRF to reconstruct the 3D shape. To tackle issues like inconsistent multi-view predictions from Zero-123 and inaccurate camera poses, the authors have developed an elevation estimation module and specialized training strategies, including 2-Stage Source View Selection and Groundtruth-Prediction Mixed Training.", "questions": "see the strengths and weaknesses.", "limitations": "The paper does not discuss limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a method that reconstructs 3D shapes from a single image, eliminating the need for per-shape optimization. It does this through the effective application of feed-forward SparseNeuS.\n2. The authors discuss and address several challenges associated with lifting Zero-123 predictions to 3D, such as inconsistent multi-views and inaccurate camera poses. The introduction of 2-Stage Source View Selection, Groundtruth-Prediction Mixed Training, and an Elevation Estimation Module appears to be effective solutions.\n3. The results demonstrate that One-2345 enhances both the quality and efficiency of the single-image to 3D conversion. Additionally, the authors adeptly extend this to text-3D conversion using a pre-trained text-image model. The ablation studies provide strong evidence for the efficacy of the introduced modules and training strategies.", "weaknesses": "1. The paper mentions the use of SparseNeuS for reconstructing 3D shapes from multi-view predictions by predicting blending weights. The colors of the 3D points are computed as the weighted sum of projected colors. However, since SparseNeuS usually takes real images as input and this work uses potentially inaccurate multi-view predictions, I am concerned that directly aggregating colors from these predictions could lead to blurry textures and artifacts. I suggest the authors investigate the effects of aggregating or refining colors more thoroughly.\n2. There is a noticeable discrepancy between training and inference. During training, the 2-Stage Source View Selection module generates multiple nearby views from ground truth anchor views for reconstruction. However, during inference, there are no ground truth anchor views available, and Zero-123 must rely on its own predicted anchor views, which may not always be accurate. The paper does not address these challenges or offer solutions. Additionally, the “2-Stage Source View Selection and Groundtruth-Prediction Mixed Training” section could benefit from clearer explanations.\n3. SparseNeuS employs post-optimization after the feed-forward pass. It would be helpful to know if One-2345 adopts a similar strategy, and, if so, how much improvement can be attributed to such a lightweight post-optimization.\n4. The overall concept of integrating a feed-forward MVSNeRF with Zero-123 is not groundbreaking. Moreover, the Groundtruth-Prediction Mixed Training, though practical, lacks novelty. However, the motivation for employing a feed-forward approach to elevate Zero-123 to 3D shape reconstruction is sound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**Weighted sum of projected colors**\n\nThanks for pointing this out. We agree that computing the point colors as a weighted sum of projected colors may not be the optimal solution in our case, considering the underlying inconsistencies. We noted that some generalizable NeRF works (e.g., MVSNeRF) use an MLP to aggregate projected colors and directly predict point colors instead of their linear weights. We have launched an experiment to explore this strategy, and training is ongoing. We will update the results after the training is complete.\n\n**Discrepancy between training and inference**\n\nWe also noticed this discrepancy. The high-level motivation is that each anchor image (one of eight) and its local images (four views, second stage) are expected to control a local region of the shape, and these eight regions should be relatively independent of each other. Since the local predictions for each anchor image are relatively accurate and the second-stage local predictions are also used in training, each local region should be reconstructed accurately. As a result, even if we utilize ground-truth anchor images during training, it should be possible to generalize to inconsistent predicted anchor images.\n\nIn fact, we have experiments trying to utilize predicted anchor images during training. However, we found it challenging to supervise the training. For example, if we utilize ground-truth renderings to supervise predicted anchor images, it actually leads to worse results, as shown in Figure 8 of the main paper.\n\nWe will clarify “2-Stage Source View Selection and Groundtruth-Prediction Mixed Training” in our revision, given more space.\n\n**Additional post-optimization**\n\nThere are two ways to add post-optimization. The first one is fine-tuning with multi-view predicted images as done in the original SparseNeuS. This may help if the multi-view predictions are very consistent. However, we find that it typically does not improve our results due to underlying inconsistencies among multi-view predictions, as shown in Figure 2 of the PDF (see main rebuttal).\n\nAnother strategy is to leverage priors from 2D diffusion models (e.g., StableDiffusion) to further fine-tune the generated mesh as in DreamFusion. Our generated results can serve as a good initialization and accelerate the optimization. While NeRF representation and volume rendering is used in the original DreamFusion to better handle topological changes, in our fine-tuning, mesh representation, and surface rendering can also be used to reduce the rendering time since there are no significant topological changes.\n\n**Integrating a feed-forward MVSNeRF with Zero-123**\n\nWe would like to emphasize our contribution in proposing a new paradigm for generalizable single-image 3D generation, which overcomes the limitations of existing paradigms, such as efficiency and 3D consistency for optimization-based methods and poor generalization for 3D native generative models.\n\nIt’s nontrivial to integrate a feed-forward MVSNeRF with Zero123. There still exists multiple challenges when combining them: (a) original SparseNeuS only focuses on frontal-view reconstruction while we need 360-degree full mesh reconstruction; (b) original SparseNeuS only considers consistent multi-view images as input instead of inconsistent multi-view predictions; (c) we need the camera poses of the input view for 3D reconstruction.\n\nWe propose several critical training strategies and a pose estimation module to overcome these challenges. As highlighted by Reviewer L2hY, we demonstrate originality through our well-motivated and smart design choices and the underlying reasoning.\n\n**”Does not discuss limitations?”**\n\nWe would like to clarify that we have discussed limitations in supplementary and will move them to the main text in our revision."}]

Author rebuttal:
Dear Reviewers,

Thank you for dedicating your time to review our paper and offering insightful feedback. We sincerely appreciate your efforts to help enhance the quality of our research. We are also pleased to note that all reviewers were supportive of our work:

(a) Recognize our contribution in proposing a novel single-pass paradigm for 3D generation (Ph7H, L2hY, yirP, eL63), which completely differs from existing 3D AIGC paradigms of optimization-based methods and 3D native generative models.

(b) Praise our paper is well-positioned in the literature and properly summarizes and tackles existing limitations of related works (Ph7H, L2hY, AK29), including lengthy optimization time, 3D inconsistency, poor geometry, and poor input adherence.

(c) Acknowledge our superior generation efficiency (Ph7H, AK29, yirP, eL63) and outperforming previous techniques (Ph7H, L2hY, AK29, eL63).

(d) Acknowledge our efforts in addressing several challenges associated with lifting Zero123 predictions to 3D, and find our solutions and design choices to be effective (eL63), smart, and well-motivated (L2hY).

(e) Praise our insightful ablation study, discussion, and underlying reasoning (L2hY, AK29, eL63).

(f) Find our paper well-written and easy to follow (L2hY, Ph7H for introduction and related work).


Meta-review: The paper received positive feedbacks. Despite the novelty is a major concern as it is basically a combination of prior works, the quality and speed are superior to prior works. Most of the concerns are addressed in the rebuttal. Please revise the final manuscript accordingly as promised.
Decision: Accept (poster)