Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
Authors: Guocheng Qian
Jinjie Mai
Abdullah Hamdi
Jian Ren
Aliaksandr Siarohin
Bing Li
Hsin-Ying Lee
Ivan Skorokhodov
Peter Wonka
Sergey Tulyakov
Bernard Ghanem
Venue: ICLR 2024 poster (2024)
Keywords: Neural Radiance Fields
Shape from Image
Generative 3D models
Primary area: generative models
TL;DR: We introduce Magic123, a novel coarse-to-fine pipeline for high-quality image-to-3D generation that uses a joint 2D and 3D prior to guide the novel views

Abstract: We present ``Magic123'', a two-stage coarse-to-fine approach for high-quality, textured 3D mesh generation from a single image in the wild using *both 2D and 3D priors*. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference-view supervision and novel-view guidance by a joint 2D and 3D diffusion prior. We introduce a trade-off parameter between the 2D and 3D priors to control the details and 3D consistencies of the generation. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on diverse synthetic and real-world images.

Published as a conference paper at ICLR 2024
MAGIC123: ONE IMAGE TO HIGH-QUALITY 3D OB- JECT GENERATION USING BOTH 2D AND 3D DIFFU- SION PRIORS
Guocheng Jinjie Mai', Abdullah Haadii,,Jan Ren", Aliaksandr Siarohin", Bing Li', Hsin- Lee?, Ivan Skorokhodov", Peter Wonka', Sergey Tulyakov?, Bernard Ghanem' 'King Abdullah University of Science and Technology (KAUST), SSaa Inc.
Viiuull Geometry Group, University of Oxford (guocheng qian, bernard. ghanem @kaust edu.sa
Input
Magic123
Input
Magic123 Novel View
Reference View
Novel View
Normals
Reference View
Normals
Figure 1: Magic123 reconstruct high-fidelity 3D content with detailed geometry and high-resolution renderings (1024 x 1024) from single image in the wild. Visit https:/ wracheenggian.githiee http://pooecctmaagicccoo for immersive visualizations and code.
ABSTRACT
We present "Magic/23", a two-stage coarse-to-fine approach for high-quality, textured 3D mesh generation from single image in the wild using both 21) and 3D priors. In the first stage, we optimize neural radiance field to produce a coarse geometry. In the second stage, adopt memory -ffficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference-viee view supervision and novel-view guidance by joint 2D and 3D diffusion prior. We introduce a trade-off parameter between the 2D and 3D priors control the details and 3D consistencies of the generation. Magic122 23 demonstrates a significant improvement over previous image-to-31 techniques, as validated through extensive experiments on diverse synthetic and real-world images.
INTRODUCTION
3D reconstruction from single image (image-to-3D) is challenging because it is an undetermined problem. A typical image-to-3D system optimizes 3D representation such as neural radiance field (NeRF) (Mildenhall et al., 2020), where the reference view and random novel views are differentially rendered during training. While the reference view be optimized to match the input, there is no available supervision for the novel views. Due to this poo-po nature, the primary focus of image-to-3D is how to leverage priors guide the novel view reconstruction.

---

Published as a conference paper at ICLR 2024
Common Objects
Uncommon Objects
Input Single Image
3D Prior Only High Consistency but Low Generalizability
Joint 2D&3D Prior
2D Prior Only High Generalizability but Low Consistency
Figure 2: The effects of the joint 2D and 3D priors. We compare image-to-3D in three cases: a teddy bear (common object), two stacked donuts (less common object), and dragon statue (uncommon object). Magic 123 with sole 3D prior (on the left) yields consistent yet potentially simplified 3D with reduced shape and texture details due its low generalizability. Magic 123 with a sole prior (on the right) shows strong generalizability in producing content with high details while potentially lacking 3D consistency. Magic12) 123 proposes to use a joint 2D and 3D prior that consistently offers 3D with fine-grained geometry and visually appealing texture.
Current mainstream image-to-3D systems such as NeuralLift (Xu et al., 2023) and RealFusion MMelas-Kyriazi et al., 2023) employ 2D priors, e.g. text-to-image diffusion models (Rombach et al., 2022; Saharia et al., 2022), for 3D reconstruction. The novel views are guided by the 2D priors using text prompts associated with the input image by captioning (Li et al., 2022: 2023) or textual inversion (Gal et al., 2023). Without any 3D data, 2D prior-based solutions can distill 2D knowledge 3D generation
zero-shot fashion through score distillation sampling (SDS) (Poole al.,
2022). Thanks to the billion-scale training dataset (Schuhmann et al., 2021), 2D priors have been showing strong generalizability 3D generation (Poole et al., 2022; Lin et al., 2023): successfully yielding detailed 3D content respecting various prompts. However, methods relying on 2D priors alone inevitably compromise on 3D consistency due to their restricted 3D knowledge. This leads to low-fidelity 3D generation, such as yielding multiple faces (Janus problems), mismatched sizes, and inconsistent texture. Fig.2 (right column) shows failure cases of using only 2D) priors: the multiple faces of the teddy bear (top row) and the donuts merged into one at the back (middle row).
Another approach to image-to-3) 3D is to employ 3D-aware priors'. Earlier attempts at 3D reconstruction leveraged geometric priors like topology constraints (Wang al., 2018 8) and coarse 3D shapes (Michel et al., 2022) to assist in 3D generation. However, these manually crafted 3D priors fall short of generating high-quality 3D content for various prompts. Recently, approaches like 3Dim (Watson et al., 2023) and Zero-1-to-3 (Liu et al., 2023) trained/finetuned view-dependent diffusion models and utilized them as 3D priors for image-to-31 generation. Since trained in 3D data, these 3D priors are more effective in generating content with high 3D consistency. Unfortunately. (1) the scale of 3D datasets is small: the largest public dataset Objaverse-XL (Deitke et al., 2023a) only contains around 10M instances; (2) 3D datasets contain mostly limited-quality instances with simple shapes. Consequently, 3D priors are limited in generalizability and tend to generate simple geometry and texture. As illustrated in Fig.2, while 3D prior-based solution effectively processes common objects (for instance, the teddy bear example in the top row), it struggles with less common ones, yielding oversimplified, sometimes even flat 3D geometry (e.g., dragon statue at bottom left).
We abbreviate 3D-aware prior 3D prior throughout the paper.
2

---

Published as a conference paper at ICLR 2024
In this paper, rather than solely relying 2D or 3D prior, we advocate for the simultaneous use of both priors to guide novel views. By modulating the simple yet effective tradeoff parameter between the 2D and 3D priors, we can manage balance between generalizability and 3D consistency in the generated 3DD content. In many cases where both 2D prior and 3D prior fail due to low 3D consistency and low generalizability, the proposed joint 21) and 3D prior propose can produce 31) content with high fidelity. Refer to Fig.2 for comparisons. Contributions of this work is summarized as follows:
We introduce Magic novel coarse-to-fine pipeline for image-to-3D generation that uses joint 2D and 3D prior to guide the novel views.
Using the exact same set of parameters for all examples without any additional reconfiguration, Magic123 achieves state-of-the the--rr Image-to-3DD results in both real-world and synthetic scenarios.
2 RELATED WORK
Multi-view 3D reconstruction. The development Neural Radiance Fields (NeRF) (Mildenhall et al., 2020; Lombardi et al., 2019) has prompted shift towards reconstructing 3D as volume radi- ance (Tagliasacchi & Mildenhall, 2022), enabling the synthesis of photo-realistic novel views (Barron et al., 2022). NeRF requires as many as 100 images to reconstruct scene. Subsequent works have explored the optimization of NeRF in few-shot (e.g. (Jain et al., 2021; Kim al., 2022; Du et al., 2023)) and one-shot (e.g. (Yu et al., 2021; Chan et al., 2022)) settings. However, these methods fail to generate 360 3D content due to the lack of strong priors for the missing novel-view information.
In-domain single-view 3D reconstruction. 3D reconstruction from single view requires strong priors on the object geometry. Direct supervision the form of 3D shape priors is a robust way to impose such constraints for particular domain, like human heads (Blanz & Vetter, 2003; Booth et al., 2016), hands (Pavlakos et al., 2019) or full bodies (Loper et al., 2015; Martinez et al., 2017). Such supervision requires expensive 3D) annotations and manual 3D prior creation. Thus several works explore unsupervised learning of 3D geometry from object-centric datasets (e.g. (Kanazawa al., 2018; Duggal & Pathak, 2022:
2013: Siarohin et al., 2023)). These methods are typically structured as auto-encoders (Wu et al., 2020; Kar et al., 2015; Cheng et al., 2023) or generators (Cai et al., 2022; Sun et al., 2022) with explicit 3D decomposition under the hood. Due to the lack of large-scale 3D data, in-domain 3D reconstruction is limited to simple shapes (e.g. chairs and cars) and cannot generalize to more complex or uncommon objects (e.g. dragons and statues).
Zero-shot single-view 3D reconstruction. Foundational multi-modal networks (Radford et al., 2021; Rombach et al., 2022) have enabled various zero-shot 3D synthesis tasks. Earlier works employed CLIP (Radford et al., 2021) guidance for 3D generation (Jain et al., 2022; Hong et al., 2022; Mohammad Khalid et al., 2022; Xu et al., 2022) and manipulation (Michel et al., 2022; Patashnik et al., 2021) from text prompts. Modern zero-shot text-to-image generators (Ramesh et al., 2021; Rombach et al., 2022; Saharia et al., 2022: Zhang et al., 2023) improve these results by providing stronger synthesis priors (Poole et al., 2022; Wang et al., 2023a; Metzer et al., 2022:
Mikaeili al., 2023). DreamFusion (Poole al., 2022) is seminal work that proposed to distill an off-the-shelf diffusion model into a NeRF for given text query. It sparked numerous follow-up approaches to improve the quality (Lin et al.. 2023; Chen et al., 2023b; Wang al., 2023b: Chen et al., 2023a). image-to-3D reconstruction (Melas- Kyriazi et al.. 2023; Tang et al.. 2023b; Höllein et al., 2023; Richardson et al., 2023). Inspired by the success of text-to-DD, 2D diffusion priors were also applied to image-to-3D with additional reference view reconstruction loss (Melas-Kyrizii et al., 2023; Xu et al., 2023; Seo et al., 2023; Lin et al., 2023; Raj et al., 2023). Recently, (Watson et al., 2023; Liu et al., 2023) trained pose-dependent diffusion models that are 3D-aware and used them to improve the 3D consistency. However, they suffered from low generalizability and tended to generate oversimplified geometry due to the limited quality and the small scale of 3D datasets. Our work instead explores a joint 2D and 3D prior to balance the generalizability and 3D consistency.
METHODOLOGY
We propose Magic123, coarse-to pipeline for high-quality 3D object generation from single reference image. Magic123 123 is supervised by the reference view reconstruction and guided by a joint 2D and 3D prior, shown in Fig.
3

---

Published as a conference paper at ICLR 2024
weights Sourrce view
Noval View
Noval Viaw
High quality
Novel view
Nuise
Noive
Tentiall Inversioa PSL imageet sp-
@ Dultasion Prters
è 2D + Dilisskn Priers
NeitF
D Mesh
High quality renderings
Foongroand Sequoostation
NeRF
DMfiet
tossg
View
loss
Rec. View
bosssrec
Iu LosSyec
Fine
Imput Imaaee
Stage
Stage
Figure 3: Magic123 23 pipeline. Magic12; 123 is a two-stage coarse-to-fine framework for high-quality 3D generation from a single reference image. Magic 123 is supervised by the reference image reconstruction and guided by joint 2D and 3D diffusion prior. At the coarse stage, we optimize an Instant-NGP NeRF for a coarse geometry. At the fine stage, we initialize a DMTet differentiable mesh from the NeRF output and optimize it with high-resolution rendering (1024 x 1024). Textural inversion is used in both stages to generate object-preserving geometry and view-consistent textures.
3.1 MAGIC 1 PIPELINE
Image preprocessing. Magic 123 is aimed at object-level image-to-3D generation. We leverage an off-the-shelf segmentation model, Dense Prediction Transformer (Ranftl et al., 2021), to segment the object. We denote the extracted binary segmentation mask M. To prevent flat geometry, we further extract the depth map by pretrained depth estimator (Ranftl et al., 2020). The foreground image is used as the input, while the mask and the depth map are used in the optimization as regularization priors.
Coarse-to-fine pipeline. Inspired by the text-to-3D work Magic3D (Lin et al., 2023), Magic 123 adopts coarse-to-fine pipeline for image-to-30 optimization. The coarse stage of Magic is targeted at learning underlying geometry that respects the reference image. Due to its strong ability to handle complex topological changes in a smooth and continuous fashion, adopt Instant-NGP (Müller et al., 2022). It only offers low-resolution renderings (128 128) during training because of memory memory-expensive volumetric rendering and possibly yields 3D shapes with noise due its tendency to create high-frequency artifacts. Therefore, we introduce the fine stage that uses DM'eet (Shen et al., 2021) to refine the coarse 3D model by the NeRF and to produce a high-uesolutton and disentangled geometry and texture. We use 1024 x 1024 rendering resolution in the fine stage, which is found to have a similar memory consumption to the coarse stage. To reconstruct 3D faithfully from a single image, we optimize the pipeline through (i) novel view guidance; (ii) reference view reconstruction supervision; (iii) two standard regularizations: depth regularization and normal smoothness.
Novel view guidance Lg is necessary to dream up the missing information. As significant difference from previous works, we do not rely solely on a 2D prior or a 3D prior, but we leverage a joint 2D and 3D prior to optimize the novel views. See $3.2 for details.
Reference view reconstruction loss Cree is to ensure the reference image I' can be reconstructed from the reference viewpoint (v"). Mean squared error is adopted on both I' and its mask follows:
Cree = Argb M @ (I' ()(("))) + Amask M ((G(vv")))
(I)
where 0 are the NeRF parameters to be optimized, C is the Hadamard product, Ge(v") is a NeRF rendered RGB image from the reference viewpoint, M () is the foreground mask acquired by integrat- ing the volume density along the ray of each pixel. Since the foreground object is extracted input, we do not model any background and simply use pure white for the background rendering for all experiments. Angs and Amask are the weights for the foreground RGB and the mask, respectively.
Depth regularization La is a standard tool to avoid overly-flar flat or caved-in 3D content (Xu et al., 2023; Tang et al., 2023b). We would like the depth from the reference viewpoint to be similar to
4

---

Published as a conference paper at ICLR 2024
high-resolution DSLR image of <
Text Prompt
2D SDS loss
Stable Diffusion
L0
A20
Lg
[R,t]
Z Zero-1- 1-to-3 SDS loss
Lg Ago
Nolsy Rendered View
Camera Pose
Figure 4: 2D 3D Diffusion priors. Magic123 uses Stable Diffusion (Rombach et al., 2022) the 2D prior and viewpoint-condition diffusion model Zero-1-to-3 (Liu et al., 2023) as the 3D prior.
the depth d' estimated by pretrained depth estimator (Ranftl al., 2020). Due to the mismatched values of and d", we regularize them linearly through normalized negative Pearson correlation:
Ld 1 cov(M @ d' M @
(M & (M d)
where cov(-) denotes covariance and (.) measures standard deviation.
Normal smoothness L., is common idea to reduce high-frequency artifacts. Finite differences of depth are used to estimate the normal map. Gaussian smoothness with 9 x 9 kernel is applied:
C n (gg())]l
where 7(-) denotes the stopgradient operation and g(-) is a Gaussian blur. Overall, both the coarse and fine stages optimized by combination of losses:
he Lg free + AaCa An
We note here that we empirically find the depth and normal regularization only have marginal affects the final performance. We keep here standard practice.
3.2 NOVEL VIEW GUIDANCE: A JOINT 2D AND 3D PRIOR
2D priors. Using single reference image is insufficient to optimize 3D. DreamFusion (Poole et al., 2022) proposes to use 2D text-to-image diffusion model as the prior to guide the novel views via the proposed score distillation sampling (SDS) loss. SDS encodes the rendered view latent, adds noise to it, and guesses the clean novel view conditioned on the input text prompt. Roughly speaking, SDS translates the rendered view into image that respects both the content from the rendered view and the text. The SDS loss is illustrated in the upper part of Fig. and is formulated as:
Oz 01
C2D & Er,e (())ee( -) 0I 30
(5)
where I is a rendered view, and z, is the noisy latent by adding a random Gaussian noise of a time step to the latent of I. e, ed. Φ, 0 are the added noise, predicted noise, parameters of the diffusion prior, and the parameters of the 3D model. / can be MLPs of NeRF for the coarse stage, or SDF, triangular deformations, and color field for the fine stage. DreamFusion points out that the Jacobian öz term of the image encoder 2T in Eq. equation can be further climinated, making the SDS loss much more efficient terms of both speed and memory.
Textural inversion. Note the prompt we use for each reference image is not pure text chosen from tedious prompt engineering. Using pure for image-to-3D generation sometimes results in inconsistent texture due to the limited expressiveness of the human language. For example, using "A high-resolution DSLR image
colorful teapot" will generate different colors that do not respect Melas-Kyrrazi textual inversion (Gal et al., 2023) technique to acquire a special token <e> to represent the object in the reference image. We use the same prompt for all examples: "A high-resolution DSLR image of <e>". We find that Stable Diffusion can generate an object with a more similar texture and style to the reference image with the textural inversion technique compared to the results without it.
the reference image. We thus follow RealFusion (Melas-Ke Kyriazi al., 2023) to leverage the same
3D prior. Using only the 2D prior is not sufficient to capture consistent 3D geometry due to its lack of 3D knowledge. Zero-1-to-3 (Liu al., 2023) thus proposes 3D(-aware) prior solution.
5

---

Published as a conference paper at ICLR 2024
1-to-1 finetunes Stable Diffusion into view-dependent version Objaverse (Deitke et al., 2023b). fero-1-to-2 takes a reference image and a viewpoint as input and can generate a novel view from the given viewpoint. Zero- 1-to-3 thereby can be used strong 3D prior for 3D reconstruction. The usage of (ero--lto-3 in an image-to-31) generation pipeline using SDS is formulated as:
01
Van - Er,e (()( 26; ,4,,, - -) 30
where R, T are the camera poses passed to Zero-I-to-3. The difference between using the 3D prior and the 2D prior is illustrated in Fig. 4, where show that the 2D prior uses text embedding as a condition while the 3D prior uses the reference view I' with the novel view camera poses as conditions. The 3D prior utilizes camera poses to encourage 3D consistency and enable the usage of more 3D information compared to the 2D prior counterpart.
A joint 2D and 3D prior. We find that the 2D and 3D priors are complementary each other. The 2D prior favors high imagination thanks to its strong generalizability stemming from the large-scale training dataset of diffusion models, but might lead inconsistent geometry due to the lack of 3D knowledge, On the other hand, the 3D prior tends to generate consistent geometry but with simple shapes and less generalizability due to the small scale and the simple geometry of the 3D dataset. In the case of uncommon objects, the 3D prior might result in over-simplified geometry and texture. Instead of relying solely on a 2D or a 3D prior, we propose to use a joint 2D) and 3D prior:
al 30
VLg Baiita wo(() (AaD ( 11,1) - -e1) + AaD (EeSD (Z1z , 12, R. T) - c2)
where AD and ÀςD determine the strength of 2D and 3D prior, respectively. Increasing AD leads to better generalizability, higher imagination, and more details, but less 3D consistencies. Increasing ASD results in more 3D consistencies, but worse generalizability and fewer details. However, tuning two parameters at the same time is not user-friendly. Interestingly, through both qualitative and quantitative experiments, we find that Zero-1-to-3, the 3D prior we use, is much more tolerant to Aan than Stable Diffusion X2D (see SC.1 for details). When only the 3D prior is used, i.e. A2D Zero-I-to-3 generates consistent results for ADD ranging from 10 to 60. On the contrary, Stable Diffusion is rather sensitive to \2D- When setting λAD to 0 and using the 2D prior only, the generated geometry varies a lot when ADD is changed from to 2. This observation leads us to fix AsD = 40 and to rely on tuning the X22 to trade off the generalizability and 3D consistencies. We set AnD = 1.0 for all experiments, but this value can be tuned according to the user's 's preference. More details and discussions on the choice of 2D and 3D priors weights are available in Sec.4.3.
4 EXPERIMENTS
4.1 SETUPS
NeRF4. We use NeRF4 dataset that we collect from 4 scenes, chair, drums, ficus, and microphone- from the synthetic NeRF dataset (Mildenhall et al., 2020). These four scenes cover complex objects (drums and ficus), a hard case (the back view of the chair), and a simple case (the microphone).
RealFusion15. We further use the 15 natural images released by RealFusion AMelas-Kyrian et al., 2023) that consists of both synthetic and real images in a broad range for evalution.
Optimization details. We use exactly the same set of hyperparameters for all experiments and do not perform any pbr-otject hyperparameter optimization. Most training details and camera settings are set to the same as RealFusion (Melas-Kyriaz et al., 2023). See $A and $B for details, respectively.
Evaluation metrics. Note that accurately evaluating 3D generation remains an open problem in the field. In this work, we eschew the use of singular 3D ground truth due to the inherent ambiguity in deriving 3D structures from a single image. Instead, we adhere to the metrics employed in the most prior studies (Xu et al., 2023; Melas Kyriazi et al., 2023), namely PSNR, LPIPS (Zhang et al., 2019 8), and CLIP similaray (Radford et al., 2021). PSNR and LPIPS are gauged in the reference view to measure reconstruction quality and perceptual similarity. CLIP CLIP-similarity calculates average CLIP distance between the 100 rendered image and the reference image to measure 3D consistency through appearance similarity across novel views and the reference view.

---

Published as a conference paper at ICLR 2024
Input
Point-E
Shap-E
3DFuse
Neural-Lift
RealFusion
Zero-1-to-
Magic123
Reference
Novel
Reference
Novel
Reference Novel Normals
Input
Point-E
Shap-E
3DFuse
Neural-Lift
Resiiussion
Zero-I-to-3
Magie123 (Ours)
Reforence
Novel
Reference
Reference
Novel Normals
Figure 5: Qualitative comparisons image-to-3D generation. We compare Magic123 recent methods (Point-E (Nichol et al., 2022), ShapeE (Jun & Nichol, 2023), 3DFuse (Seo et al., 2023), RealFusion (Melas-Kyriazi al., 2023), and Zero-I-to-3 (Liu al., 2023)) for generating 3D objects from a single unposed image (the leftmost column). We show results on the RealFusion dataset at the top, while the NeRF4 dataset comparisons are shown the bottom.
4.2 RESULTS
Quantitative and qualitative comparisons. We compare Magic123 against the state-of-the- PointE (Nichol et al., 2022), Shap-E (Jun & Nichol, 2023), 3DFuse (Seo et al., 2023), NeuralLift (Xu et al., 2023), RealFusion (Melas-Kyriazi et al., 2023) and Zero-I-to-3 (Liu et al., 2023) in
7

---

Published as a conference paper at ICLR 2024
Table I: Magic123 results. We show quantitative results terms of Similarity PSNR
LPIPS The results are shown on the NeRF4 and Realfusion15 datasets, while bold reflects the best.
Dataset
Metroes
Point-E
Shap-E
3DFuse
NeuralLift
RealFusion
Zero-I-to-3
Magic123 (Ours)
CLIP- Similariy
0.48
0.60
0.60
0.52
0.38
0.62
0.80
NeRF4
PSNR
0.70
0.99
11.64
12.55
15.37
23.96
24.62
IPIPSS
0.76
0.40
0.03
CLIP- 7
0.53
0.59
0.67
0.65
0.75
0.82
RealFusion1
PSNRT
0.98
1.23
10.32
11.08
18.87
19,49
19.50
LPIPS]
0.78
0.74
0.38
0.39
0.14
0.11
0.10
both NeRF4 and RealFusion datasets. For Zero-1-to-3, we adopt the implementation from (Tang, 2022), which yields better performance than the original implementation. For other works, their officially released code. All baselines and Magic12; 123 are run with their default settings. As shown Table Magic achieves Top-1 performance across all the metrics in both datasets when compared to previous approaches. It is worth noting that the PSNR and LPIPS results demonstrate significant improvements over the baselines, highlighting the exceptional reconstruction performance of Magic125 23. The improvement of CLIP-Similarity reflects the great 3D coherence regards to the reference view. Qualitative comparisons are available in Fig. 5. Magic 123 achieves the best results in terms of both geometry and texture. Note how Magic 123 greatly outperforms the 3D-based zero- I-to- 3 (Liu et al., 2023) especially in complex objects like the dragon statue and the colorful teapot in the first two rows, while at the same time greatly outperforming 2D-based RealFusion (Melas- -yriazzi et al., 2023) in all examples. This performance demonstrates the superiority of Magic 123 over the state-of-the- and its ability to generate high-quality 3D content.
4.3 ABLATION AND ANALYSIS
Magic 123 introduces coarse-to-fine pipeline for single image reconstruction and joint 2D and 3D prior for novel view guidance. We provide analysis and ablation studies to show their effectiveness.
The effect of the coarse-to-fine pipeline is shown Fig. A consistent improvement quantitative performance is observed throughout different setups when the fine stage is used. The use of textured mesh DMTet representation enables higher quality 3D content that fits the objective and produces more compelling and higher resolution 3D visuals. Qualitative ablation for the coarse-to porse-to-fing fine pipeline available in SC.2.
Combining both 2D and 3D priors and the trade-off factor Azn- Fig. 6 demonstrates the effectiveness the joint 2D and prior quantitatively. In Fig. 7, we further ablate the joint prior qualitatively and analyze the effectiveness of the trade-off hyperparameter ADp in Eqn 7. We start from = to use only the 3D prior and gradually increase AD to 0.1, 0.5, 1.0, 2, 5, and finally to use only the 2D prior with > and >ap=0 (we also denote this case λạn=oo for coherence). The key observations include: (1) Relying on sole 3D prior results consistent geometry (e.g. teddy bear) but falters in generating complex and uncommon objects, often rendering oversimplified geometry with minimal details (e.g. dragon statue); (2) Relying on a sole 2D prior significantly
0.540
0.830
0.600
Fine Stage
0.829
Fine Stage
0.760
0.720
0.790
0,590
0.789
0.640
0.779
0.500
0.520
0.749
only
3D only NeRF4 Dutmet
Mugic123
2D only
3D only
Magic133
Reali'aston Dateset
Figure 6: Ablation study (guantitative). We guantitatively compare using the coarse and fine stages in Magic 123. both setups, we ablate utilizing only 2D prior (A=D=1,)g==0), utilizing only 3D prior =0 and utilizing joint 2D and 3D prior (A20=1.A:D=40),
8

---

Published as a conference paper at ICLR 2024
ApD
0.5
1.0
8
Input
(Balanced Point)
App 1. AD
Figure 7: Setting A2p- We study the effects of AD Magic123, Increasing A leads to 3D geometry with higher imagination and more details but less 3D consistencies and vice versa. Azp=1 provides a good balance and thus is used as default throughout all experiments.
improves performance in conjuring complex scenes like the dragon statue but simultaneously triggers 3D inconsistencies such the Janus problem in the bear; (3) As X escalates, the imaginative prowess of Magic123 is enhanced and more details become evident, but there is a tendency to compromise 3D consistency. We assign λ2n=1 the default value for all examples. Aςn could also be fine-tuned for even better results on certain inputs.
5 CONCLUSION AND DISCUSSION
This work presents Magic123, 23,, coarse-to-ine solution for generating high-quality, textured 3D meshes from a single image. By leveraging a joint 2D and 3D prior, Magic 123 achieves a performance that not reachable in sole 2D 3D prior-based solution and the new state of the art in image-to-31 generation, A trade-off parameter between the 2D and 3D priors allows for control over the generalizability and the 3D consistency. Magic 123 outperforms previous techniques terms of both realism and level of detail, demonstrated through extensive experiments on real-world images and synthetic benchmarks. Our findings contribute to narrowing the gap between human abilities in 3D reasoning and those of machines, and pave the way for future advancements in single-image 3D reconstruction. The availability of our code, models, and generated 3D assets will further facilitate research and applications in this field.
Limitation. One limitation is that M2 might suffer from inconsistent texture and inaccurate geometry due incomplete information from single image. Specially, clear inconsitency appers in the boundary mostly because of blended foreground and background along the boundary segmentation errors. A texture consistency loss might be helpful. Similar previous work, Magic also tends to generate over-saturated textures due to the usage of the SDS loss. The over-saturation issue becomes more severe for the second stage because of the higher resolution. See examples in Fig. for these failure cases: the incomplete foot of the bear, the inconsistent texture between the front (pink color) and back views (less pink) of the donuts, the round shape of the drums, and the oversaturated color of the hoarse and the chair. Similar to other per- ser-prompt optimization methods, Magic 123 also takes around hour to get 3D model and with limited diversity. This time can be reduced through (1) replacing to Gaussian Splatting in stage as shown in (Tang et al., 2023a), and (2) sampling from small range of time steps in stage 2, following the suggestions of ICLR reviewers. The diversity issue might be possible to alleviate through VDS (Wang et al., 2023b) training with prior guidance plus diverse 3D data.
Acknowledgement. The authors would like to thank Xiaoyu Xiang for the insightful discussion and Dai-Jie Wu for sharing Point-E and Shap-E results. This work was supported by the KAUST Office of Sponsored Research through the Visual Computing Center funding, well as, the SDAIA-KAUST Center Excellence in Data Science and Artificial Intelligence AEDAIA-KAUSS AD). Part of the support is also coming from KAUST Ibn Rushd Postdoc Fellowship program.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo BaariinBruuala,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
pp.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5470-5479, 2022.
Volker Blanz and Thomas Vetter. Face recognition based on fitting 3d morphable model. IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 156771063--107, 2003.
James Booth, Anastasios Roussos, Stefanos Zafeiriou, Allan Ponniah, and David Dunaway. A 3d morphable model learnt from 10,000 faces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5543-5552, 2016.
Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3981-3990, June 2022.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Samch Khamis, et al. Efficient geometry-awan -aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16123-16133, 2022.
Dave Zhenyu Chen, Yawar Siddiqui, Tisin-Ying Lee, Sergey Tulyakov, and Matthias NieBner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 1396,
2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 12223333.37.. 2023b.
Yen-Chi Cheng, Hsin-Ying Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan
Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.05663 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13142-13151 2023b.
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning render novel views from wide-baseline stereo pairs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Shivam Duggal and Deepak Pathak. Topologically-aware deformation fields for single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1536-1546, 2022.
Rinon Gal. Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing rext-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023.
Lukas Höllein, Ang Cao. Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303.11900 19989, 2023.
10

---

Published as a conference paper at ICLR 2024
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation 3d avatars. arXiv preprint arXiv:2205.085. 2022.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5885-58854 2021.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 867-876, 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386, 2018.
Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-specific object reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 1966-1974, 2015.
Ira haaeelmaache--Shzzemmaae Internet based morphable model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3256-3263, 2013.
Mijeong Kim, Seonguk Sco, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12912-12921, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference Machine Learning, pp. 22888-12900 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML., volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yt Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1tto-- Zero-shot one image to 3d object. arXiv preprint 23t.: 2023.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751, 2019.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics (TOG), 34(6):1-16, 2015.
Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1640-2649, 2017.
Luke Melas Marias,Krriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 (deg) reconstruction of any object from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint w:Xiv:2211.07600 2022.
11

---

Published as a conference paper at ICLR 2024
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Aryan Mikaeili, Or Perel, Daniel Cohen- Or, and Ali Mahdavi-Amin Sked: Sketch-guided text-based 3d editing. arXiv preprint arXiv:2303.10735, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405-421. Springer, 2020.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-meshi Gener- ating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-8, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. In ACM Transactions Graphics (SIGGRAPII), 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.087511 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 2095-2094, 2021.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975-10985, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. International Conference on Learning Representations (ICLR), 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML). pp. 8748-8763. PMLR, 2021.
Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXhv:2303.13500, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 8821-8831. PMLR, 2021.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladien Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 44(3):1623-1677, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. Proceedings of the IEEESCVF International Conference on Computer Vision (ICCV), pp. 12159- 12168, 2021.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing 3d shapes. arXiv preprint arXiv:2302.01721, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479-36494, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 I 1.02/ 14, 2021.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jachoon Ko, Hyeonsu Kim, Junho Kim, Sin-wwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yo Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 6087-6101, 2021.
Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Hsin- Ying Lee, Jian Ren, Menglei Chai, and Sergey Tulyakov. Unsupervised volumetric animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning (ICML), pp. 2256-2265. PMLR, 2015.
Jingxiang Sun, Xuan Wang. Yichun Shi, Lizhen Wang. Jue Wang, and Yebin Liu. Ide-3d: Interac- tive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Transactions on Graphics (TOG), -160 2022. doi: 10. (0.11445755045443555555
Andrea Tagliasacchi and Ben Mildenhall. arXiv:2209.024177 2022.
Volume rendering digest (for nerf). arXiv preprint
Jiaxiang Tang.
Stable-dreamfussion: http://hwwbb.oommmaaaaaa//eaeaeb dreamfusion..
Text-to-3d with stable-diffusion, 2022.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1665: 2023a.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023b.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023a.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Grn Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pp. 52-67, 2018.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High- figeliiiyytty and diverse text-to-3d generation with variational score distillation. arXiv preprint 2305 10..3555..11.. 16213, 2023b.
Daniel Watson, William Chan, Ricardo Martin-Brualle Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In International Conference Learning Representations (ICLR), 2023.
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmet- ric deformable 3d objects from images in the wild. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
13

---

Published as a conference paper at ICLR 2024
Dejia Xu, Yifan Jiang. Peihao Wang, Zhiwen Fan, Yi Wang. and Zhangyang Wang. Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360 deg) views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Jiale Xu, Xintao Wang. Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:2212. 14704, 2022.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4578-4587, 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effecctivnness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
We use exactly the same set of hyperparameters for all experiments and do not perform any per-object hyperparameter optimization. Both coarse and fine stages are optimized using Adam with 0.001 learning rate and no weight decay for 5, 000 iterations. Argh, Amask are set to 5, 0.5 for both stages, A2D and ADD are set to 1 and 40 for the first stage and are lowered to 0.001 and 0.01 in the second stage for refinement to alleviate oversaturated textures. We adopt the Stable Diffusion (Sohl-Dickstere et al., 2015) model of V1.5 as the 2D prior. The guidance scale of the 2D prior is set 100 following (Poole et al., 2022). For the 3D prior, Zero-1-to-3 (Liu et al., 2023) (105, 000 iterations finetuned version) is leveraged. The guidance scale of Zero-1 --t-33 is set to 5 following (L.iu al., 2023). The NeRF backbone is implemented by three layers of multi-layer perceptrons with 64 hidden dims. Regarding lighting and shading, we keep nearly the same as (Poole et al., 2022). The difference is we set the first 1, 000 iterations in the first stage to normals' shading to focus on learning geometry as inspired by (Chen et al., 2023b). For other iterations as well as the fine stage, we use diffuse shading with probability 0.75 and textureless shading with a probability 0.25. The rendering resolutions are set to 128 x 128 and 1024 x 1024 for the coarse and the fine stage, respectively. For both stages, we use depth regularization and normal smoothness regularization with X 0.001 and A.. = 0.5. Following the standard practice (Barron et al., 2021; Poole et al., 2022; Melas-Kyriaze et al., 2023). we additionally add the 0.001 entropy regularization and 0.01 orientation regularization in the NeRF stage. Our implementation is based on the Stable DreamFusion repo (Tang, 2022). The training of Magic takes roughly hour on a 32G V100 GPU, while the coarse stage and the fine stage take 40 and 20 minutes, respectively.
B CAMERA SETTINGS
Frontal view setting, Since the reference image is unposed, our model assumes frontal reference view (90% elevation and 0' azimuth) for simplicity. However, in real-world applications, these angles can be adjusted according to the input, either through intuitive estimation or camera pose detection. simple tuning of the elevation angle can improve the benchmark performance of Magic 123. For instance, in the chair example shown in Fig. II, altering the clevation angle from 90ºh to 60 addresses squeezed reconstruction of the chair. For research purposes, we propose to exclude this camera estimation it does not impact the comparison between different methods and only requires engineering efforts in application.
Rendering camera setting. We set the camera parameters for the rendering as follows. The camera is placed 1.8 meters from the coordinate origin, iee. the radial distance is 1.8. The field of view (FOV) of the camera is 40 We highlight that the 3D reconstruction performance is not sensitive to camera parameters, as long as they are reasonable, e.g. FOV between 20 and 60, and radial distance between 1 to meters. In Fig. II, we validate that using the same camera parameters RealFusion Mielas-Kyrrazz et al., 2023) that is different from ours, i.e. camera radius [1.0, 1.5] and FOV [40, 70]. achieves results without obvious differences as ours.
C MORE ANALYSIS AND ABLATION STUDIES
C. ABLATION AND ANALYSIS ON THE USAGE OF 2D AND 3D PRIORS
3D priors only. We first turn off the guidance of 2D prior by setting AnD = 0, such that we only use the 3D-aware diffusion prior Zero-I-to-3 Liu et al. (2023).
Note that 7ero-1-to- in our paper (results in Tab. 1, Fig. 5) denotes our improved reimplemented Teo-I-to-3 using the same training configurations as Magic123. i.e. 7ero-1-to-3 in our paper refers to the first stage results of 12 3D prior only (Fig. 2, 6, 7), where both of them use second stage DMTet fine-tuning for fair comparison.
Furthermore, we study the effects of Xap by performing a grid search and evaluate the image-to-3D reconstruction performance, where AAD = 5, 10, 20, 40, 60, 80. Interestingly, we find that Zero- 3 is very robust to the change of Asp- Tab. I demonstrates that different ASD leads to consistent quantitative result. We thus simply set Aap = 40 throughout the experiments since it achieves a slightly better CLIP simi-aiyyy score than other values.
15

---

Published as a conference paper at ICLR 2024
Table I: Effects of ARD and AD in Magic using only 2D or 3D prior on NeRF4 dataset.
varying AzD when Azn=0
varying Aan when Azn=
10
20
40
60
80
0.1
2
CLIP-similariyy
0.58
0.61
0.62
0.61
0.58
0.54
0.60
0.72
PSNRT LPIPS
23.96
24.05
23.96
23.75
23.34
23.62
24.11
22.42
0.04
0.04
0.05
0.06
0.08
0.04
0.04
0.07
Taput
M2gic123 (Course)
2a 3D
Magie133 (Cunrse)
Magic123
Magie123 (Cuarse)
Magie
M2gie123 wio textual inversion fextual inversion (Course)
Magic
Figure I: Ablation study (qualitative), We qualitatively compare the novel view renderings from the coarse and fine stages in Magic123. We ablate utilizing only 2D prior (>2==IA)===)), only 3D prior (\===0,Aa==40), and a joint 2D and 3D prior (Aan=1,As===0), We also ablate the effects of textual inversion at the right.
2D priors only. We then turn off the 3D prior and study the effect of X2D. As shown in Tab. I, the image-to-3D system is sensitive the weights of the 2D prior. With the increase of X sharp increase in CLIP similarity and a drop in PSNR are observed. This is because a larger 2D prior weight leads to more imagination, which unfortunately might result in 3D inconsistency. Due to the observation that the 3D prior is more robust than the 2D prior to the weight, we use Xnn as the tradeoff parameter to control the imagination and 3D consistency.
C.2 ABLATION ON THE COARSE-TO-FINE PIPELINE
In $4.3 we ablate the effect of the coarse-to coarse-to-fine fine pipeline quantitatively. Here provide the visual comparisons in Fig. I. The fine stage consistently augments the sharpness of the rendering and the geometry and texture details. See the edge of the wings and claws of the dragon and the toppings of the donuts for examples.
C.3 ABLATE THE REGULARIZATION
M2 is optimized additionally by depth regularization, normal smoothness regularization, entropy regularization, and orientation regularization, with weights of 0.01, 0.5, 0.001, and 0.01, respectively. In Fig. II, we show that the depth. the entropy, and the orientation regularizations have minimal impact on the image-to-3D reconstruction performance. However, we keep them in our implementation as they are common practices in the NeRF family. The normal smoothness is more important in alleviating the high-frequency noise.
C.4 ABLATE THE TEXTUAL INVERSION
We use textual inversion in both stages for consistent geometry and texture with the input reference image. Fig. I we additionally ablate the effects of textural inversion by removing and using pure texts in the guidance. In the two examples, we change the prompts from "A high-resolution DSLR image of <e>" to "A high-resolution DSLR image of a metal dragon statue", and "A high-resolution DSLR image of two donuts", respectively. As observed, textual inversion has marginal effects on the image-to-30 to-3D reconstruction performance. However, it helps with keeping the consistency between the input image and the generated 3D content. Without textual inversion, the dragon with a different style of horns and golden textures appears that is not consistent with the input image. The donuts without textual inversion have distinet toppings from the input image.
16

---

Published as a conference paper at ICLR 2024
Ingar
wee deepi regularization
nermall smoothoess regularization
entrapyarrtenattonn regalarization
elevation
Magle113
of
Figure II: Qualitative ablation study for the effects of depth regularization, normal smoothness, entropy and orientation regularization. camera parameters, and ghe front-view assumption (elevation angle). Normal smoothness reduces high-frequency noise. Other factors like depth, entropy, and orientation regularization exert minimal influence on image-to-3D reconstruction results but are maintained in Magic123, adhering to common practice. Using different camera parameters, including camera radius (1.8 meters v.s. [1.0, 1.5] in RealFusion) and field of view (40 v.s. [40, 70J in RealFusion), have a marginal impact performance. Magic123 opts for simple configuration, setting the camera radius to 1.8 meters and the FOV to 40. Setting the elevation angle from 90º to reasonable value also improves reconstruction quality (see the chair example).
D MORE COMPARASIONS
Here, we additionally compare Magic123 with most recent methods RealFusion (Melas-Kyrian et al., 2023), Zero-I-to-3 (Liu et al., 2023), and Make-Ir-3D (Tang et al., 2023b). Magic123 outperforms all of them in terms of both 3D geometry and texture quality by a large margin.
17

---

Published as a conference paper at ICLR 2024
Input
uuion
Zero-I-to-3
Make-It-3D
Magic123 (Ours)
Reference
Novel
Reference
Novel
Reference
Novel
Reference
Novel Normals
Figure III: Qualitative comparisons. We compare Magic123 to the most recent methods RealFusion AMelas-yriia et al.. 2023), Zero-l-to-3 (Liu et al., 2023). and Make-It-3D (Tang et al., 2023b).
18


Reviews:
[{"review_id": "kHjW1YVoUp", "reviewer": "Reviewer_Ea4Z", "summary": "This proposes a diffusion-based method for single-view 3D reconstruction. The key novelty is to combine 2D diffusion prior, which can achieve higher resolution more realistic appearance and 3D diffusion prior, which can achieve better 3D consistency. It designs a 2 stage optimization pipeline where the first stage uses hash grid volume representation to get the coarse reconstruction and the second stage uses DMTet representation to refine high resolution texture. Experiments on widely used datasets, such as NeRF dataset, shows significant improvements compared to prior state-of-the-arts.", "questions": "Overall I am very positive towards accepting this paper. I cannot see if any questions that will change my opinions so far. I will be glad to learn from authors rebuttal and other reviewers' comments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. High-quality single-view 3D reconstruction. \nThe proposed method shows high-quality 3D reconstruction that is significantly better than previous state-of-the-arts. \n\n2. Comprehensive ablation studies and convincing experiments.\nAuthors report various benchmarks to show improvements compared to previous works, including LPIPS, PSNR, SSIM and CLIP similarities. They did ablation studies that clearly show the benefit of combining 2D and 3D diffusion priors, both qualitatively and quantitatively (Figure 6 and 7).\n\n3. Clear novelties and technical contributions. \nThe idea of combining 2D and 3D diffusion prior is novel and is proven to be useful in many applications. \n\n4. Well-written with sufficient implementation details.\nThis paper is well-written and easy to follow. Authors provide enough details to re-implement this paper.", "weaknesses": "This is a solid paper which significantly improves the baseline of single-view 3D reconstruction. The idea of using a diffusion model fine-tuned on a large-scale 3D dataset for text-to-3D or sparse-view 3D has been adopted by many recent works and the reconstruction quality has been significantly improved since then. I do not find any specific weaknesses of this paper and followings are either limitations of current works or some design choices that can improve the results.\n\n1. Relightability.\nWhile this work uses a diffuse shading model, the reconstructed material texture is not really relightable and will have lighting baked in. To reconstruct relightable 3D contents, we need to understand the environment lighting of the input images and use that for optimization. However, how to do that from sparse view input is still an open problem.\n\n2. Reconstruction speed.\nThe reconstruction speed of the stage one may be improved by progressively increase the level of hashgrid and the resolution of the rendered image. The reconstruction speed of the second stage may be improved by sampling from a small range of time steps. \n\n3. Backside of the object.\nIn some examples, the backside of the object still has a strong color shift compared to the reference views. This may be improved by using VSD loss instead of SDS loss or simply using a strong 3D diffusion prior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIBroR2tlO", "reviewer": "Reviewer_UQdQ", "summary": "This paper proposes to combine the SDS loss from both a 2D pretrained diffusion model (StableDiffusion) and a fine-tuned 3D-aware 2D diffusion model (Zero1-to-3). They show that this combination allows the generated 3D assets to have more realistic texture than using the Zero1-to-3 only, and have more plausible shapes than using the StableDiffusion only.", "questions": "This paper is written in a clear way, and I have no further questions after reading the manuscript carefully.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Paper is well-written and easy to follow.\n2. The proposed method seems easy to reproduce, and code is attached.\n3. On 4 nerf objects and 15 real objects, the proposed system outperforms baseline single-image-to-3D methods both qualitatively and quantitatively.", "weaknesses": "1. This seems to be a A+B style paper with majority of the components have appeared in prior works, e.g., coarse-to-fine optimization (first stage uses NeRF, second stage uses DMTet) from Magic3D, text-conditioned (using textual inversion) SDS loss from RealFusion, image-conditioned SDS loss from Zero1-to-3.\n\n2. The novel part of this work seems the combination of the text-conditioned SDS loss and image-conditioned SDS loss. But this part might be a bit straightforward in the sense that the 3D awareness of Zero1-to-3 can help learn better geometry (as shown in Zero1-to-3), and the text-conditioned StableDiffusion can generate better appearance (as it’s a model trained on billions of real images and not tuned on the limited amount of Objaverse renderings)\n\n3. Given single image, there're multiple plausible 3D reconstructions. This kind of diversity is missing in the evaluation part of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9q1AzJgwp", "reviewer": "Reviewer_25qf", "summary": "The authors propose a novel method for generating high-quality, textured 3D meshes from a single unposed image in the wild using both 2D and 3D priors. For 2D priors, the framework uses Stable Diffusion and for 3D Zero-1-to-3 is used. The approach uses two-stage coarse-to-fine pipeline that first optimizes a NeRF to produce a coarse geometry and texture, and then refines it using a memory-efficient differentiable mesh representation (DMTet) for high-resolution renderings. State of the art performance is demonstrated compared to recent image-to-3D methods.", "questions": "1. What is effect of applying this approach to scenes without segmentation instead of objects?\n2. Is TI better than DreamBooth for 2D SDS loss?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novelty**: The proposed approach introduces a key insight of used 2D and 3D priors together to aid in better 3D geometry generation as opposed to 2D priors only as was common. Additionally, also presents the distinct advantage over 3D prior alone, since 2D diffusion models are trained on so much more data than that available to train for 3D. \n2. **Paper quality**: The paper is well written with great attention to detail, all the components are described in detail and adequately motivated. \n3. **Reproducibility**: The framework is implemented using available open source code. Additionally, all network and training details have been provided to aid in reproducibility of the approach. Furthermore, code has been provided in the supplementary as well to match the results shown in the paper.\n4. **Result quality**: The generated assets have impressive quality for a single view lifting approach. Results are demonstrated on both synthetic objects and objects in the wild to highlight the efficiency of the approach. \n5. **Related work**: An adequate treatment of the related work in the space of text-to-3D and single image-to-3D have been provided to place the given approach in the context of relevant literature.\n6. **Ablations**: Key ablations have been provided, particularly, the effect of 2D only, 3D only and combined priors and demonstrating the effect of change of 2D and 3D prior weights. \n6. **Supplementary materials**: The provided materials are very helpful, as it shows turntable videos of generated assets, along with several key ablations which provides a lot of insight into the different components of the approach.", "weaknesses": "1. **Segmentation**: The quality of the final asset is limited by the performance of the segmentation model of choice(as acknowledged by the authors). Several questions arise in this setting: \n  > a. Does the diffusion priors correct for some of inaccuracies of the segmentation model?  \n  > b. Is it hard to reconstruct objects that are in a cluttered environment?  \n  > c. Is the DMTet algorithm affected by inaccurate segmentations?  \n  > d. What happens in instances where the segmentation algorithm outputs two or more disjoint segments, does the inductive bias of the NeRF overcome this issue? (this is potentially addressed to some extent under the limitation mentioned by the authors about handling discontinuities).   \n  > e. Is a segmentation network required even if the model is on a plain background? (this is potentially the case, as the mask appears in equation (1)\n\n2. **Depth and Smoothness**: Similar to the above concern, the performance is capped by the effectiveness of the pre-trained depth and normal estimation network. This poses inherent limitations to the kind of input images that it can be applied to. Providing some insights on where these kind of estimation networks fail would be helpful. Furthermore, depth and normals potentially don't make much sense, till the NeRF has converged to a reasonably estimate of geometry.Is there a schedule associated with these losses to have an effect after initial shape has been estimated by the NeRF?\n\n3. **Centered objects**: Owing to the nature of Zero-1-to-3, it appears that this kind of lifting works only on centered objects. What is the effect on the recovered geometry if the object is off-center but can still be segmented out accurately ?\n\n4. **Textual inversion vs Dreambooth**: The authors mention that textual inversion is used for the 2D-SDS loss to capture some of the image specific texture information. Providing some insight on how this compares to performing Dreambooth on the model to achieve the same would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oTbURNruy6", "reviewer": "Reviewer_jWBN", "summary": "Magic123 proposes a technique to create textured 3D meshes from a single image by a combination of both 2D and 3D priors. The method follows a two-stage strategy, transitioning from coarse to fine geometry. They also introduced trade-off parameter that attempts to balance the influences of the 2D and 3D priors, aiming to optimize quality and consistency. They showed comparisons with existing works on 3D datasets.", "questions": "1. The paper does not convincingly demonstrate that the combination of 2D and 3D priors leads to markedly better outcomes than using either alone. The real challenge should be to extract the unique advantages of both priors for a superior result, not just to find a balance between the two. I'd suggest the authors to add more comparison with Zero1-2-3 or include results to prove clear improvements without poor sacrifice of multi-view consistency in figure 5. \n2. The paper should provide evidence of improved geometric detail and not just enhancements in the output rendering quality. Merely augmenting the contrast or enhancing the visual appeal with a 2D prior is insufficient. Could this method result in sharper geometric details as well?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Magic123 = Magic3D and Zero123? An interesting and practical idea of introducing a balance between the 2D and 3D priors could offer a way to adjust generation outcomes.\n2. The two-stage approach might offer a systematic way to enhance the granularity and accuracy of the 3D representations.", "weaknesses": "1. Comparison with Zero1-2-3: Although the authors claim \"significant improvements over existing method\", the qualitative comparison in figure 5 does not significantly support this claim. The improvements in terms of quality and sharpness are noticeable, but there seems to be a potential trade-off in consistency. Moreover, the supplementary videos are missing a direct comparison with Zero1-2-3, and the quantitative comparison with Zero1-2-3 is comparable, which further weakens the claim (in limited 3D results and examples). \n2. The paper does not provide details on how the weights for the 2D and 3D priors are chosen. If these weights are determined and tuned during training and are inflexible during inference, this approach might not be the most effective way to integrate the priors.\n3. The model's inherent assumption of a frontal reference limits its versatility. This is evident from the removal of four examples from the training dataset that do not meet the front view assumption. That could potentially limit the scale of 3D datasets and this could lead to even lower generalization and a drop in result quality.\n4. The coarse-to-fine process, while effective, isn't novel as it has been employed in other works like Magic3D.  The Tetrahedral \nrepresentation seems to be hard to scale and appears to be optimally suited for objects. This could limit the method's applicability to a broader range of scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)