IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts
Authors: Bohan Zeng
Shanglin Li
Yutang Feng
Ling Yang
Juan Zhang
Hong Li
Jiaming Liu
Conghui He
Wentao Zhang
Jianzhuang Liu
Baochang Zhang
Shuicheng YAN
Venue: ICLR 2025 Poster (2025)
Keywords: 3D generation
Diffusion model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advances in 3D generation have been remarkable, with methods such as DreamFusion leveraging large-scale text-to-image diffusion-based models to guide 3D object generation. These methods enable the synthesis of detailed and photorealistic textured objects. However, the appearance of 3D objects produced by such text-to-3D models is often unpredictable, and it is hard for single-image-to-3D methods to deal with images lacking a clear subject, complicating the generation of appearance-controllable 3D objects from complex images. To address these challenges, we present IPDreamer, a novel method that captures intricate appearance features from complex **I**mage **P**rompts and aligns the synthesized 3D object with these extracted features, enabling high-fidelity, appearance-controllable 3D object generation. Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts, highlighting its promising capability in appearance-controlled, complex 3D object generation.

Published as a conference paper at ICLR 2025
IPDREAMER: AAPEARANCE-CONTROLLABBA 3D ОВJЕCТ GENERATION WITH COMPLEX IMAGE PROMPTS
Bohan Zeng"; Shanglin Li Yutang Feng"; Ling Yang"! Juan Zhang Hong Li', Jiaming Liu², Conghui He?, Wentao Zhang', Jianzhuang Liu? Baochang Zhang"; Shuicheng Yan
!Peking University Shenzhen Institute of Advanced Technology Tiamat AI
*Beihang University Skywork AI National University of Singapore
"Shanghai AI Laboratory
ABSTRACT
Recent advances in 3D generation have been remarkable, with methods such DreamFusion leveraging large-scale text-to-image diffusion-based models guide 3D object generation. These methods enable the synthesis of detailed and photorealistic textured objects. However, the appearance of 3D objects produced by such text-to-3D models is often unpredictable, and it is hard for single-image- to-3D methods to deal with images lacking a clear subject, complicating the gen- eration of appearance- controllllle 3D objects from complex images. To address these challenges, we present IPDreamer, a novel method that captures intricate ap- pearance features from complex Image Prompts and aligns the synthesized 3D ob- ject with these extracted features, enabling high-fidelity, aopearance-controllabb 3D object generation. Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts, highlighting its promising capability in appearance--controlled.. complex 3D object generation. https::/// https::/github com wtt.h//...c.oeeeccccn21111111 IPDreamer
INTRODUCTION
The rapid evolution of 3D technology has revolutionized the way create and interact with virtual worlds. 3D technology is now essential in a wide range of fields, including architecture, gam- ing, mechanical manufacturing, and AR/VR. However, creating high-quality 3D content remains a challenging and time-consuming task, even for experts. To address this challenge, researchers have developed text-to-3D methodologies that automate the process of generating 3D assets from textual descriptions. Built on the 3D scene representation capabilities of Neural Radiance Fields (NeRFs) (Mildenhall et al. 2021 Müller al. 2022) and the rich visual prior knowledge of pre- trained diffusion models (Rombach et al. 2022; Saharia et al. (202aa), recent research (Jain et al.)
2022 Mohammad Khalid al. 2022 Poole et al. 2022 Lin et al.) 2023) Chen al. 2023b Wang et al. 2023 Shi et al.) 2023b) has made significant progress, simplifying the text-to-3D pipeline and making it more accessible, which causes a significant shift these fields.
Recent advances in diffusion models have significantly enhanced the capabilities of text-to-imant to-image generation. State-of-the-ae (SOTA) systems, leveraging cutting-edge diffusion-based techniques (Nichol et al. 2021 Rombach et al. 2022 Brooks et al.) 20233 /Zhang & Agrawala, 2023 Hu et al.) 2021), can now generate and modify images directly from textual descriptions with vastly improved quality. Inspired by the rapid development in text-to-image generation, recent works (Poole et al.) 2022 Lin et al.) 2023 Chen et al., 2023b Wang et al., (023) have extended these models to 3D by utilizing pretrained text-to-image to-ie diffusion models in conjunction with the Score Distillation Sam- pling (SDS) algorithm and its variants to optimize 3D representations. These methods are capable of
'These authors contributed equally.
I Project Leader.
*oorrsponniigg Author: hezhang buaa. maa..edu.nn, zhang juan www..bdu.cccm

---

Published as a conference paper at ICLR 2025
guided Reference Image #1
Generated Object #1
Cuarse NeRF Model
Reference Imagge #2
Generated 3D Object #2
(b)
Guided by
Input Image
Guided LGM
Guided by zero123 Guided by
Leaves flying in the wind
Figure 1: IPDreamer can generate controllable, high-quality 3D objects based on both textual and image prompts. (a) illustrates two high-quality 3D objects with rich details. initialized by the same NeRF model and guided by different complex reference image prompts. (b) demonstrates the 3D synthesis under challenging textual conditions, where our method outperforms existing text-to-3D
method (Wang et al. 2023). image-to-3D methods LGM (Tang et al., 2024) and cero123++ (Shi
et al. 2023a). Besides, the input image in (b) is generated by SD Rombach et al. (2022).
generating high-quality 3D objects and scenes. However, due the lack of explicit appearance in- formation in textual prompts, the appearance of the generated results remains largely uncontrollable, limiting the precision of the visual output.
Unlike the unpredictability in text-to-3D generation, single-image-t-3DD generation allows for strict control over the appearance of the generated 3D results. However, existing single-image-t--DD methods (Liu et al. 2023b c Shi et al.) 2023b) are limited to simple images with clear subjects, often struggling with complex images that feature rich content and intricate compositions. For example, aa) contains complex images that lack a clear subject, making it difficult to seg- ment a simple, singular image for single-image-to-31 generation. Furthermore, as demonstrated in Fig[Tb), when text prompts are ambiguous or lack a clear main subject- succh as "Leaves fly- ing in the wind"-neither current text-to-3D nor image-to-3D methods can achieve reasonable 3D synthesis.
To tackle these challenges, we introduce IPDreamer, novel method for complex image-to-3D gen- eration. Specifically, extend SDS to Image Prompt Score Distillation Sampling (IPSDS), which leverages detailed features extracted from complex image prompts and corresponding normal maps guide the optimization of both 3D mesh texture and geometry. With IPSDS, IPDreamer enables high-quality 3D object generation with controllable appearances based on complex image inputs. To ensure stable 3D object generation across various challenging scenarios, we propose mask- guided compositional alignment strategy for IPSDS, enabling 3D object generation from multiple complex image prompts. In particular, we leverage Multimodal Large Language Model (MLLM) to localize the features from multiple image prompts onto the generated 3D objects. This allows IPDreamer handle diverse situations, including cases where multiple highly divergent images are used to guide the synthesis of single 3D object, and scenarios where the guiding complex image exhibits significant structural differences from the initial coarse object. As shown in Fig Ta), IPDreamer effecttiely transfers the appearances of reference images to NeRF models, generating high-quality 3D objects even in cases with ambiguities or unclear primary subjects. For the more challenging scenarios illustrated in Fig (b), IPDreamer successfully produces the desired 3D object where existing text-to-3D and single-image-tor to-3 methods fall short.
In summary, the main contributions of this paper as follows:
We present IPDreamer, a novel 3D object synthesis framework that allows users to con- sistently create controllable, high-quality 3D objects. Compared with previous methods,
2

---

Published as a conference paper at ICLR 2025
IPDreamer excels in synthesizing high-quality 3D objects that closely align with complex image prompts.
We introduce Image Prompt Score Distillation Sampling (IPSDS), which utilizes a sub- stantial image prompt feature to guide 3D mesh optimization.
We propose a Mask-guided Compositional Alignment strategy for IPSDS, enabling high- quality 3D objects synthesis based on multiple complex image prompts, in cases where initial NeRF models deviate significantly from the provided image prompts, or when mul- tiple diverse image prompts are needed to guide the synthesis of a single 3D object.
Comprehensive experiments show that IPDreamer achieves high-quality 3D generation and excellent rendering results, outperforming existing SOTA methods.
2
RELATED WORK
2.1 DIFFUSION MODELS
Diffusion models (DMs) were initially introduced as generative model for gradually denoising images corrupted by Gaussian noise generate samples (Sohl-Dickstein al. (2015). Recent ad-
vancements in DMs (Ho et al. 2020 Song et al. 2020 Dhariwal & Nichol 2021] Vahdat et al.,
2021 Rombach al. 2022] Peebles Xie 2022) have shown their exceptional performance in image synthesis. DMs have also achieved state-of-the thee-----hh-arr results in various synthesis tasks, includ-
ing text-to-image generation (Saharia et al., 2022a Nichol et al. 2202]] Ramesh et al. 2022: Podell
et al. 2024: Yang et al., 2024). inpainting (Avrahami et al. 2 Lugmayr et al.] 2022 Ye et al.,
2023). 3D object synthesis (Li et al. 2022b Luo & Hu 2021). video synthesis (Ho et al. 2022b a
speech synthesis (Kong et al. 2020 Liu et al. 2021), super-resolution (Li et al. 2022a Saharia
jet al. al.
2022b; Gao et al. 2023), face animation (Qi et al. 2023). ext-to-motion generation (Tevet
2022). and brain signal visualization (Takaii & Nishimoto 2022 2023). Some DMs (Kulikov
ct al.
2022 Wang et al. 2022) can produce diverse results by learning the internal patch distribution
from single image. (Mokady al., 2023) Tumanyan al. 2023 Wu et al., 2023 Geyer al.,
2023)
enhance image/video editing with pre-trained DMs in zero-shot or one-shot manner: These
advancements highlight the versatility and potential of DMs across a wide range of syntheses.
2.2
CONTROLLABLE GENERATION AND EDITING
Controllable generation and editing of 2D images and 3D objects are goals of generative tasks. With the emergence of large language models (L.LMs) such as GPT-3 and Llama (Brown et al. 2020 Touvron al.) 2023a b). instruction-based user-friendly generative control has gained much attention. InstructPix2PM5 (Brooks et al.) 2023) and MagicBrush (Zhang et al. (023a) build datasets based on LLMs and large texl-to-image models to achieve effective instruction control on 2D im- ages. InstrRt (Haque et al. (023) combines this method with NeRF scene reconstruc- tion (Mildenhall et al. (021) introduce instruction control into 3D generation. Meanwhile, series of adapter methods such as ControlNet and IP-Adapter (Zhang & Agrawala 2023 Hu et al.)
2021 Mou et al. 2023 Zhao et al. 2023b Ye et al., 20033 [Huang et al.) 2023) provide reliable approaches for fine-tuning large pre-trained DMs (e.g., Stable Diffusion (Rombach et al. (022) and Imagen (Saharia et al. 2022a)) for conditional controllable generation (c.g., using sketch, canny, pose, etc. to control image structure). Among them, image prompt adaption methods (Ye et al. 2023 Zhang et al.) 2023b) introduce a decoupled cross-attention mechanism to achieve effective appearance generation control using image prompts.
2.3
3D GENERATION
In recent years, 3D generative modeling has attracted large number of researchers. Inspired by the recent neural volume rendering, many 3D-aware image rendering methods (Chan et al.) 2022 2 Gu et al. 2021 Hao al. 2 Niemeyer & Geiger 2021 are proposed generate high- quality rendered 2D images for 3D visualization. Meanwhile, with the development of text-to-image synthesis, researchers have shown growing interest in text-to-3D generation. Early methods such as DreamField (Jain et al., 2022) and CLIPmesh (Mohammad Khalid et al.; (002) achieve text-to- 3D generation by utilizing pretrained image-text aligned model CL.P (Radford et al. 2021). They
3

---

Published as a conference paper at ICLR 2025
Provide Complex Image Prompt Conditions
Stage 2: Mesh Training Guided by Complex Image Features
Prompt
Geometry Optimization
Extraction
DPT
Prat
Nornal Map
Mysh SALS
EiP
Argo
Stage 1: Coarse 3D object Obtaining
Texture Optimization
Siggle Ceeppier
Emaage
Stage
Training
Mesh
Eip
Geometry Optimization
Yuna
Texture Optimization Guided Image
Optimization Guided by Multiple Images
Complex imsgee
Image features
Locallistion testual grompee provided
Trutef
emoode
Cranoo Dwow-AAiealioeii
Frozen weights
Subtraction
Mésh
30x
Rendeting (RGB image)
mage
lnd byliriaaa
Rendering (Normal image)
Hendered nurmal map default vieoppoitt
Rendered normal map Rendered RGB image Meate
Peatures of difierent Denssiing modal with Image Pronngt Cross arseniie layees
in random viewpoint
in random viewpeint Marau Yusee image prompts
Figure 2:
IPDreamer designed to generate high-quality, appearance-coontollabbe 3D meshes that align with single/multiple complex image prompts.
optimize the underlying 3D representations (NeRFs and meshes) to ensure that all 2D renderings have high text-image alignment scores. Recently, (Poole al.) 2022 Lin et al., 2023 Chen et al. 2023b. Wang et al. 2023 Chen et al. 2023a) have achieved high-quality 3D synthesis (NeRFs and meshes) by leveraging robust pretrained text-to-image DM as strong prior to guiding the training of the 3D model. Other works (Shi et al. 2023b] Zhao et al.] 2023a Liu et al. (023b) introduce multi-view DMs to enhance 3D consistency and provide strong structured semantic priors for 3D synthesis. IT3D (Chen et al. 2023c) combines SDS and GAN to refine the 3D model and obtain high-quality 3D synthesis. (Tang al. 2023a Liang et al. 2023) combine 3D Gaussians (Kerbll et al., 2023) with SDS-based optimization to improve 3D synthesis and reduce generation time.
Additionally, (Melas-Krrizzi et al., 2023; Tang al. 2023b Liu et al. 2023a Qian al. 2023)
are capable to generate 3D representations based on single images, and (Liu et al. 2023b Shi al. (2033)] Yang et al., 2023) achieve 2D images in multiple viewpoints, which enable consistant 3D object generation. In this work, we introduce IPDreamer, method that leverages complex image prompts to provide comprehensive appearance information, effectively guiding the synthesis of high-quality 3D objects.
3 METHOD
In this section, present the details of IPDreamer. We start with brief definition of 3D mesh, followed by the problem statement for text-to-3D and single-image-to-3D generation, and a re- view of SDS preliminaries. Next, present the design and analysis of our proposed IPSDS and Mask-guided Compositional Alignment. Note that 3D meshes are the most common form of 3D representation in industry, so our approach focuses on optimizing 3D meshes.
3.1 PRELIMINARIES AND MOTIVATION
3D Mesh. The 3D mesh can be represented as a deformable tetrahedral grid (V,T), where each vertex V; has a signed distance field value s; E S and deformation Av, E AV from its canonical position. During optimization, the surface mesh is rendered into high-resolution images using a differentiable rasterizer (Munkberg al. 2022).
Score Distillation Sampling. Given a textual prompt or an image I, text-to- Dit-tto-DDsinnle--mmae to-3D generation aims synthesize novel views and optimize the parameters of 3D object/scene
4

---

Published as a conference paper at ICLR 2025
corresponding to the given y I. DreamFusion (Poole al. 2022 utilizes a pretrained text-to- image DM Epretrain to optimize an MLP parameterized as 8 representing a 3D volume, where a differentiable generator g renders 9 to create 2D images g(O, e) given sampled camera pose c, based on the gradient of the Score Distillation Sampling (SDS) loss:
dr
VgLSDS (0) Er,e w(()) (Epretrain ( y, - c)
(1)
00!
where w(t) is weighting function, Epretrain (zt; y, predicts the noise noisy image Ty, text prompt features y and timestep t.
10) given the
Motivation. Although (Lin et al. 2023 Chen et al. 2023b Wang et al. 2023 show excellent text-to-3D generation, the appearances of their 3D synthesis results are uncontrollable. A feasible solution to realize controllable 3D object generation is to use 2D image as prior. However, existing
methods find obtaining high-quality 3D object synthesis from com- plex images difficult. To overcome these obstacles, we extract feature details from complex image prompts to provide comprehensive appearance information for 3D object synthesis, and we propose the IPSDS and a Mask-guided Compositional Alignment strategy to enable stable, high-fidelity 3D object generation.
3.2 IMAGE PROMPT SCORE DISTILLATION SAMPLING (IPSDS)
In this section, leverage the image features extracted by the image encoder Emage from (Ye et al. 2023) to optimize the geometry and texture of 3D objects. The 3D mesh is initialized using either user-provided 3D mesh, or coarse NeRF model generated by existing text-to-3D methods or IPSDS. As shown in Fig. b)),, we first explain how IPSDS optimizes AV, S, and 6. Subsequently, we analyze how IPSDS effectively utilizes complex image prompts Iggb and corresponding normal image prompts I, to guide the synthesis of high-quality 3D objects.
Optimizing 3D Mesh with IPSDS. Existing methods directly use a text-conditioned DM to guide geometry optimization. However, it can be challenging because the DM's pre-training dataset lacks normal map images. To address this, we adopt an additional normal image prompt feature yn = Emagg (In) to provide richer and more robust geometric information with normal map optimization, instead of solely using the textual prompt y (Chen et al. 2023b Wang et al. 2023).
The geometry optimization process computes the gradients of the IPSDS geometry loss as:
O2n
Vav GepsDs-Geo (VV,S) E.,[(w(t)) (Cip(2n Un. y, t) dAv
(2)
8zm
V CIPDSSS- (AV, (AV = (w() (Zn, Mer y, t) E)
as
where n, denotes the noisy latent of rendered normal map in random view In,ran at timestep t.
After optimizing the estimated normal map, the geometry of the 3D mesh becomes more reasonable. Then we further optimize the texture through IPSDS. We first extract the image prompt features basic guidance for the texture optimization. Then we devise geometry prompt difference 6yco for yrgb to compensate for the morphological disparity between Tryb and Lrgh. Let def. and ran be the rendered normal map of the 3D mesh from the default viewpoint and randomly sampled viewpoint, respectively. We extract their image prompt featuress, fn,def = Exmage (amege((nn,d)) (#n,de/)) and Un.ren Esmage (Tn,ran). The difference between Ya.ran and /n,def called the geometry prompt difference ōgeo:
Uryb (Trg))
8gc0
Vn,ran Do,def-
(4)
The texture optimization process computes the gradients of the IPSDS texture loss as:
OErgb
VoLIrDD--Tee AV, S) Er,[(w(()) (Cip(Ergh, Vrgb 5yoo y, t) c)
},
(5)
00
d Dergb
V AIIIDSSS-TE AV, = (w(t) (Etp (Zrgb,i Brgb + 8geas y, <) dAV
(6)
j2ryb
V AV. = [e,((w(() (Eip(Ergh Wrah 8gens y. 1) -) 0s
(7)
5

---

Published as a conference paper at ICLR 2025
"Drw""
guided by
goidedd by Image Prompte
Image Pronpt
"Hier"
Coarse Nett model
object Maskoguided
Course NeRE model
3D abject uy Mask-guuded Composilional Aligument
Compositional Alignment
adorable doll.
Figure Illustration of the effectiveness of Mask-guided Compositional Alignment.
where Zrgb,t denotes the noisy latent of Zggb (random viewpoint) in timestep t. The geometry prompt difference 6geo can effectively represent the Morphological distance between Fn,ron and de f in the image prompt feature space. Thus it is used to compensate Yrgb (default viewpoint) such that Mrgb + dge represents the RGB image Trgh-
Incorporating Image Prompt into 3D Generation. Here we explain how our method can effec- tively use a complex, high-quality image to guide 3D object synthesis, by introducing cross-attention for the image prompt. Given the query features Z which are derived from the latent representations of the 2D rendering results of the 3D object from various viewpoints, and the image prompt embed- ding Yrgb. the rross-attention for the image prompt is formulated as follows:
7' = Softmax QK
V
(8)
where Q ZW = yrgh W V Yrghh W represent the queries, keys, and values within the cross-attention module, respectively, Z' denotes the output features of the module, and the W q. W and Wir are the projection matrices used for linear transformations. The reasons why IPSDS can utilize complex images to guide the generation of 3D objects while existing single-image-to-31 methods cannot are two-fold: First, the encoder of the image prompt adaption method effectively extracts the image features Grgb from the reference high-resolution image prompt. Secondly, as the attention map can accurately align the features Yrgb with specific positions of rendered images (Hertz et al. 2022) from the 3D object, the features from the original complex image are precisely positioned on the most relevant parts of the 3D object.
3.3 MASK-GUIDED COMPOSITIONAL ALIGNMENT FOR MULTIPLE IMAGE PROMPTS
Motivation of Mask-guided Compositional Alignment. When the rendered 2D images from the NeRF model significantly differ from the image prompt or when the input includes multiple diverse image prompts, relying solely on gross-attention for the image prompt fails to effectively align the features Vrgb with specific positions on the 3D object. As illustrated in Fig. BT.) it is evident that IPDreamer encounters difficulties with this challenging sample. To address this issue, we have designed a Mask-guided Compositional Alignment strategy. Specifically, we collect multiple images 1790 from the input complex image prompts. Then, we employ a large multimodel model (GPT- 4v) to provide localization words yfaa for corresponding 175t. We adopt the cross attention in LDM (Rombach et al. 2022) to obtain localization masks:
QKT Ent. )), Q ZW,,, K y Wist = 1,2, Ript
m4 13l(Softmax(
Vd
where Z represents the same query features as in Equation g BI denotes binarization operator and nip is the number of the input multiple images. Subsequently, the mask m, obtained from the textual description st is used to adjust the computation of the cross attention corresponding to the
feature y: rgb of the image prompt 1r06
Thpp
T QK
z'
my Softmax
ip.
V sp.is
(10)
Tipp
Vd
where Q = ZW q. Kip.i 1/1 rgg W'P V Vip.i = y W With the help of our strategy, we modify the cross-attention calculation during the IPSDS supervision process, enabling the features of multiple images to be localized onto the 3D object, as shown in Fig. Bib. Next, we provide more details of the IPSDS training process with the Mask-guided Compositional Alignment.

---

Published as a conference paper at ICLR 2025
NeRF model
"face"
"clothes"
"hair"
Figure 4: Visualization of localization masks.
Training Process of IPSDS with Mask-guided Compositional Alignment. We briefly describe how IPSDS is designed with the Mask-guided Compositional Alignment. First, we utilize GPT- 4v to generate localization textual prompts, yf ..... Vhug to map features of the complex images onto 3D objects. Specifically, we imput the multiple complex reference images and rendered im- ages of coarse NeRF model into GPT-4v, which analyzes and identifies the regions that need to be segmented from the input complex images and generates the corresponding localization textual prompts. Based on the analysis, we employ SAM (Kirillov et al. 2023) to segment multiple partial images, 11 ...., 1rgb from the complex images. Additionally, both the localization textual prompts, Hspp
yiah . ...., y and the segmented partial images, 17 ..... Dipp can be adjusted by users.
Given potential semantic differences between Ligh and the coarse NeRF model (e.g.. "magnif- icent magic castle" vs. "adorable cottage") and the possibility that the multiple partial images 17th .I.h may lack detail or resolution, optimization. We adopt super-resolution model (Zhang & Agrawala 2023 in conjunction with 17 ..... Irgh and y!th ..... y to generate new 17 ..... 1rgh. This preprocessing step improves the quality of both the guided images and the resulting 3D object.
crucial to enhance them before initiating texture
Subsequently, extract image prompt features y1th ..... gg from corresponding partial image prompts 1796 "".."""""""""" Irgb. Then, we modify the cross-attention calculation based on Equation 9 and Equation duriigg the IPSDS supervision process, to localize the image features onto the 3D object according to yl fatt ..... yki. Fig. shows an example of the effect of localization masks calculated in guided Compositonal Alignment. The IPSDS supervision in this part can be
the process written as:
1st VeLIPSDS Tex AV, (Br,, (w(() (Zig(Eng vi .... Viep sot .... Vrig ÖZrgb 00
),
(11)
Vav (0, SV (wr(t (Zigh yo on Voip ... Woip ()
rgh
txt
txt
0zrgb ØAv
(12)
Ö2rgb
V AV. E.,, (w)( (cep (Zngh, o0 .... Weap V1 Wrep : 1) c)
(13)
as
After initially localizing the partial image prompts onto the 3D object, it is then necessary to further optimize the texture of the 3D object globally. We input all features of the partial images and the provided complex images into the IPSDS loss optimize the 3D object simultancously:
fglobal concat(y
rgb Frezy Bryb + 8geo),
(14)
D2rgb
VoLIPSD-Tex (0, AV, = Ee, [u. Fgtobal, t) -
(15)
00
Ozrgh
gloobat, 1) S VAV Tea AV, V Re, (w) (Ergh,
(16)
DAV
02rgb as
VSLIPSD-TEE AV.
Ex,e (t)p:(t)))(s(<rbbrr foloeat
fylobal, (t t)
(17)
4 EXPERIMENTS
4.1 3D GENERATION WITH SINGLE COMPLEX IMAGE
As depicted Fig. we show generated 3D objects that use diverse image prompts to guide synthesis. This demonstrates IPDreamer's ability produce high-quality 3D objects that align with

---

Published as a conference paper at ICLR 2025
Coarse NeRF models
(8)
(b)
3D objects generated by IPDreamer
Sample #1
Sample
Sample
Sample #2
Sample #3
Sample
Sample #3
Sample #4
female game character."
"An iron breastplate."
Figure 5: generated 3D objects with different image prompts. (a) Image prompts used for Coarse NeRF model generation. (b) Rendering of Coarse NeRF models. We show four samples for each textual prompt. each sample, the top left is selected complex image prompt, and the bottom left and the right illustrate the high-quality 3D object optimized by IPDreamer based on the coarse NeRF model.
Reffeence
3D Objects
Refeernce
3D Objects
ug Madi-giited Aligsment
Mati-guided Alignment
(a)
Mack-puided
M/ Mankogaided Compositiona
Compactional
Aliguneet
appearance phoenix."
"An adorable cottage.
Charse NcRF
Model
Reference Imsgee
3D Object
Coarse NettF Medel
Reference Inages
3D Object
basc"
"Calle"
(b)
CCatle boav"
"Castl""
Figure 6: Effectiveness of Mask-guided Compositional Alignment.
the styles the provided images. Remarkably, IPDreamer can appropriately transfer the appearance of the image prompts to the synthesized 3D objects, regardless of the structure difference between the image prompts and the coarse NeRF models. To our knowledge, this high-quality appearance transfer task is not achievable by existing text-to-3D or sagll-imaae-tor for the textual prompt "An iron breastplate", although both the textual and image prompt features are closely,
methods. In Sample
provided for 3D object synthesis, the generated result resembles leather breastplate which aligns with the image prompt rather than the "iron" mentioned in the textual prompt. This illustrates that the image prompt exerts stronger influence on the synthesis the 3D object than the textual prompt. Consequently, such powerful ability to edit 3D object textures greatly facilitates applications in the gaming and video industries.
4.2 3D GENERATION WITH MULTIPLE COMPLEX IMAGES
To demonstrate the stability of our IPDreamer in generating 3D models guided by multiple com- plex images or when the initial coarse 3D object significantly differs from these guiding images, produced more 3D objects under these conditions. As shown in Fig. a)), when provided im-
8

---

Published as a conference paper at ICLR 2025
Table Quantitive comparison of text-to-3D
Table 2: Percentage of the preference in the user study of text-to-3D generation.
generation.
Method
FID
CLIP-Score
Method
Prefer baseline
Prefer ours
DreamFusion
320.16
0.2413
DreamFusion
6.45
93.55
Magic3D
320.56
0.2582
Magic3D
10.89
89.11
Fantasia3D
294.79
0.2557
Fantasia3D
25.82
72.18
PreaificDreamer
277.35
0.2603
ProlificDreamer
41.65
58.35
LRM
304.81
0.2522
LRM
29.73
70.27
LGM
296.62
0.2605
LGM
34.52
65.48
2+++
269.58
0.2561
7ero123++
33.75
66.25
SV3D
266.21
0.2681
SV3D
43.75
56.25
IPDreamer(Ours)
253.32
0.2716
IPDreamer
ProaifieDreamer
IPDreamer
Fantasia3D
chimpanzee dressed like Henry VIII king of England.
A 3D model of an adorable cottage with thatched roof.
Michelangelo style statue dog reading cellphone.
vintage record player.
IPDreamer
Magic3D
IPDreamer
DreamFusion
squirrel-octopus octopus hybrid.
raccoon astronout holding his helmet.
DreamFusion
Magic3D
Fantasia3D
ProlifieDreamer
IPDreamer
Zero123++
SV3D
The shining sun.
Qualitative Comparison of Text-to-3D Generation. It is worth noting that for the "Shining Sun" sample, our IPDreamer can generate luminous 3D sphere with natural light rays emitting, which is difficult for other baseline methods achieve.
Figure 7:
age prompts vastly differ from the coarse NeRF models, the 3D objects guided by IPSDS with Mask-guides Compositional Alignment retain the semantic essence the original coarse NeRF models and achieve the intended outcomes. And in Fig. btb), provide samples with the same coarse NeRF model and multiple diverse complex image prompts but different mask- guidedd textual prompts; the generated results of these two samples quite different and well follow the

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study of
ó dgeo.
4/o Lipene-Gre
Sample
w/ Livorie-Gre
Sample
w/ 4ipgn--Ges
Methods
CLIP score
Figure 8: Visualization of the initial normal map of the 3D object at the beginning of geometry optimization, along with the image prompt and the refined normal map after geome- try optimization for each sample.
w/o ggeo w/ Ogeo
0.8228
0.8389
textual requirements, showing that IPDreamer effectively enhances the diversity of the generated 3D objects, offering new perspectives for the advancement in the 3D research domain.
4.3 COMPARISON ON TEXT-TO-3D
To validate the quality of the results generated by our method, we conducted a comparative analysis
with text-to-3D methods (Poole et al., [022] Lin et al. [2033] Chen et al.) 2023b; Wang et al.) (023)
and single-image-to-3D methods (Hong et al. [024]] Tang et al. 2024 Shi et al. 2023a| Voleti al,,
2025) in the text-to-3D generation task. As illustrated in Fig. 7 IPDreamer surpasses these base-
line methods by producing highly controllable and realistic 3D objects that align closely with the provided textual prompts. Additionally, we compare IPDreamer with all baseline methods under example "The shining sun", where existing text-to-3D and erate clear and coherent subjects. In contrast, method successfully generates results that meet the requirements, further demonstrating its effectiveness.
methods fail to gen-
For quantitative evaluation, randomly select 30 textual prompts and compare the performance of IPDreamer against state-of-the-art (SOTA) methods, as shown in Table [ IPDreamer achieves superior performance, evidenced by lower FID score, indicating higher quality 3D object genera- tion, and a higher CLIP score, reflecting better aligmment with the input textual prompts. To provide more comprehensive assessment of the generated results, we also conduct user study, the results are demonstrated in Table. 2 The details of the CLIP score, FID, and user study are introduced in Appendix A.2 Besides, we provide more generated 3D results in Appendix A.pendix/A.1.3
4.4 ABLATION STUDY
We conduct ablation study evaluate the impact of LIrSDS -ee Geo and Egeo optimizing 3D objects. Their effectiveness illustrated in Fig. X and Table In Fig. 8 we showcase the optimized normal maps of two samples. After geometry optimization, Sample and Sample 2 learn the high- frequency details from their corresponding image prompts. The difference in the optimized normal maps between Sample and Sample 2 is readily discernible in Fig. [8] illustrating the efficacy of LIPSDS-Geo in learning geometry representations from image prompts. In Table 3 we compare the CLIP score of 3D objects optimized with and without 89ea- We conduct the quantitive comparison using the samples mentioned in Section 4.3 and employ CL.P score to compare the alignment of rendered images of 3D objects generated with and without &geo in different viewpoints with the reference image prompt. The experimental results show that with 6geo. the rendered images of the 3D object in different viewpoints are more consistent with the reference image prompt.
CONCLUSION
In this work, we propose IPDreamer, novel framework that enables the generation of high-quality, controace-controllable 3D objects from complex image prompts. By introducing Image Prompt Score Distillation Sampling (IPSDS), our method effectively captures rich and intricate appearance features from complex images to guide the optimization of both texture and geometry in 3D mesh generation. Our approach supports multiple complex images various contexts to guide 3D object generation, enabling the stable production of high-quality 3D results. IPDreamer addresses the limi- tations of existing text-to-3D and pongle-image-to--- methods by producing 3D objects that are con- sistent with textual descriptions and the detailed appearances of complex image prompts. Compre- hensive experiments demonstrate that IPDreamer outperforms statc-of-the -he methods, highlighting its promising capability in advancing ogpearance-conttollaht complex 3DD object generation.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
The work was supported by the National Key Research and Development Program of China (Grant No. 2023YFC3306401). This research was also supported by Zhejiang Provincial Natural Sci- ence Foundation of China under Grant No. LD24F020007. Beijing Natural Science Foundation L223024, L244043 and Z241 100001324017, "One Thousand Plan" projects in Jiangxi Province Jxsq202310220 102268, National Natural Science Foundation of China under Grant No. NSFC12201024, and in part by NUS Start-up Grant A-0010106-00-00.
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, et al. Language models are few-shot learners. In NeurIPS, 2020.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
Eric R Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, et Efficient geometry- aware 3d generative adversarial networks. In CVPR, 2022.
Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 11396, 2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13877 2023b.
Yiwen Chen, Chi Zhang, Xiaofeng Yang, Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. It3d: Improved text-to-3d generation with explicit view synthesis. arXiv preprint
arXiv:2308. 1 11433, 2023c.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nadas Clip-guided domain adaptation of image generators. ToG, 2022.
Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xi- antong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. arXiv preprint arXiv:2303. 2et.223333..11, 2023.
Michal Geyer, Omer Bar- Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In ICLR, 2023.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2/10.0880 2021.
Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Yu Liu. Gancraft: Unsupervised 3d neural
rendering of minecraft worlds. CVPR, 2021.
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2neff:: Editing 3d scenes with instructions. arXiv preprint arXiv:2303. 12789, 2023.

---

Published as a conference paper at ICLR 2025
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Soompttto-promp image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02300 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In NeurlPS, 2022b.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint urXIv:2302.09778 2023.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ToG, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.026433 2023.
Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv:2009.09761, 2020.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: single image denoising diffusion model. In PMLR, 2022.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022a.
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffu- sion. arXiv:2212.03299: 2022b.
Yixun Liang, Xin Yang. Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fideltyy text-to-3d generation via interval score matching. arXiv preprint vrXiv:2311.11284 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Diffusion acoustic model for singing voice synthesis. arXiv:2105.02444 2021.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306. 16928, 2023a.
12

---

Published as a conference paper at ICLR 2025
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: seoo one image to 3d object. arXiv preprint arXiv:2303.11325 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consiser images from a single-view image. arXiv preprint arXiv:2309.03453, 2023c.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabílistic models. In CVPR, 2022.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
Luke KeriszKyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from single image. In CVPR, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. COMMUN ACM, 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generat- ing textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qic T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453. 2023.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ToG, 2022.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rX11:2112.10741, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv:2212.097443 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv: 14988, 2022.
Zipeng Qi. Xulong Zhang, Ning Cheng. Jing Xiao, and Jianzong Wang. Difftalker: Co-driven audio- image diffusion for talking faces via intermediate landmarks. arXiv preprint arXiv:2309.07509, 2023.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov. Peter Wonka, Sergey Tulyakov, and Ghanem Bernard. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning trans- ferable visual models from natural language supervision. In ICMI., 2021.
13

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Ho Jonathan, et al. Photore- alistic texi-to-image diffusion models with deep language understanding, In NeurIPS, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI, 2022b.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++H a single image to consistent multi-view diffusion base model. arXiv preprint arXhv:2310.151100 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 2023b.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICML., 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. biorxiv. In CVPR, 2022.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In CVPR, 2023.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. /6653, 2023a.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.050054 2024.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023b.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit Bermano. Human motion diffusion model. arXiv:2209. 14916, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 13977, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In NewrlPS, 2021.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. In ECCV, 2025.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning a diffusion model from a single natural image. arXiv:2211.12441 2022.
14

---

Published as a conference paper at ICLR 2025
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: high-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint 2rXiv:2305.16211, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. ICCV, 2023.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.0660 06677, 2022.
Jiayu Yang. Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consis- tency for multi-view images diffusion. arXiv preprint arXiv:2310. 10343, 2023.
Ling Yang. Zhaochen Yu, Chenlin Meng. Minkai Xu, Stefano Ermon, and Bin Cui. Mastering texi-to-image diffusion: Recaptioning, planning, and generating with multimodal Ilms. ICML,
2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 06721, 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guides image editing, arXiv preprint arXiv:2306.10012, 2023a.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models.
arXiv preprint arXiv:2302.05545 2023.
Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. arXiv preprint urXiv:2312.16277 2023b.
Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu. Efficientdreamer: High- fighliiielity and robust 3d creation via orthogonal-view diffusion prior. arXiv preprint arXiv:2308. /3223, 2023a.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. arXiv preprint arXiv:2305. 163.2 2023b.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
In Appendix A.1 we present additional synthesized 3D objects generated by IPDreamer. Detailed
implementation information provided in Appendix A.2 Furthermore, Appendix A.5 analyzes the social impact of IPDreamer.
A.1
MORE EXAMPLES OF 3D OBJECTS GENERATED BY IPDREAMER
(n)
(b)
(d)
cool sword."
"A beautiful ring with diamonds." "
AA adorable doll.*
"A sacred altar."
"A high-quality shoe." "
Figure 9: generated 3D objects with different image prompts. (a) Scribble object outlines and corresponding image prompts for Coarse 3D object generation. (b) Renderings of coarse NeRF models. (c) (d) Two samples demonstrated for each textual prompt. In each sample, the top left is reference complex image prompt, and the bottom left and the right illustrate the 3D object optimized by IPDreamer based on the coarse NeRF model.
A.1.1 MORE EXAMPLES OF 3D OBJECTS GUIDED BY IPSDS
To further demonstrate IPDreamer's s remarkable ability to manipulate appearance, we conduct more 3D synthesii experiments. These experiments diverse textual prompts, each accompanied
16

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(a)
(b)
(c)
"A mini car."
"A pistol."
Figure 10: More samples of 3D object editing.
(a) Coarse NeRF models.
(b) Provided image
prompts. (c) 3D objects generated by IPDreamer.
NeRF Model
Image Prompt
3D object w/o Mask-guided Compositional Alignment
3D object w/ Mask Mlask-guided
Compositional Alignment
"A knapsack made of leather."
"A mechanical bear paw." "
"An iron sword."
"A ghost in a red wedding dress."
Figure 11: Comparison of 3D objects synthesis with and without Mask-guided Compositional Alignment.
by two distinct image prompts. evident in Fig. P IPDreamer consistently produces impressive 3D object synthesis, regardless of the geometric shape of the acquired NeRF or the image prompts used for texture editing. The generated results highlight IPDreamer's s powerful texture editing capabilities for 3D objects, suggesting its potential to effectively in the 3D gaming and video industries.
Besides, to further demonstrate the appearance guidance capability IPSDS in generating 3D ob- jects,
samples whose reference image prompts particularly complex and somewhat different from the initial coarse NeRF models, as shown in Fig. 10 Even in such challenging cases, IPDreamer still achieve high-quality 3D objects, such the cyborg-style mini generated in
17

---

Published as a conference paper at ICLR 2025
NeRF Model
Complex Image Prompt
3D Object
crystal armor wrapped vines."
"A sofa made of leather."
"A cyborg style mini car."
"A phoenix."
NeRF Model
Complex Image Prompt
3D
Object
'An ice-creamer made crystal."
"The barbecue stacked in the shape of hamburger."
heart flowed
aircraft carrier."
magma."
Figure 12: 3D objects synthesis with Mask- guided Compositional Alignment.
the first example, and the futuristic toy pistol in the second example. By utilizing IPDreamer's style editing ability for 3D objects, the generated results can be more diverse.
A.1.2 MORE EXAMPLES OF 3D OBJECTS GUIDED WITH MASK-GUIDED COMPOSITIONAL ALIGNMENT
While IPDreamer can achieve remarkable object synthesis in numerous challenging cases even without Mask-guided Compositional Alignment, difficulties emerge when the appearance of the supplied image prompts substantially diverge from the initial coarse NeRF model. To emphasize the potent 3D object optimization capability of Mask-guided Compositional Alignment within IP- Dreamer, we offer a comparison of the generated 3D objects with and without Mask-guided Com- positional Alignment in Fig. II The outcomes validate the exceptional high-fidelity capability of Mask-guided Compositional Alignment. further clucidate the superiority of Mask-guided Com- positional Alignment, present additional generation results in Fig. 12
A.1.3 MORE TEXT-TO-3D GENERATION RESULTS
To further demonstrate the capability of our IPDreamer in generating the desired 3D objects, provide additional qualitative comparison results in Fig. 13 The baseline methods, including Pro- lifieDreamer (Wang al. 2023) well single-image-tor
methods (Hong al. 2024 Tang jet al. 2024 Shi et al.) 2023a] Voleti et al.) (025). are also included. is evident that for the am- biguous testing samples or lack clear main subject, existing text-to-3I) and single-image-to-31)
18

---

Published as a conference paper at ICLR 2025
IPDreamer
LGM
LRM
1
SV3D
PreaifiDrreamer
Splashing waves.
A broken bridge.
A broken egg.
Figure 13: More Qualitative comparison of text-to-3D synthesis.
A rotary telephone carved out of wood.
A car made out of sushi.
classic Packard car.
A cauldron full of gold coins.
A plush dragon toy.
A praying mantis wearing roller.
stuffed grey rabbit holding pretend carrot.
A DSLR photo of an imperial state crown of England.
A blue tulip.
Figure 14: More generated 3D objects of IPDreamer.
methods struggle to produce the desired 3D results. In contrast, IPDreamer succeeds in gen- erating accurate 3D results. To further showease the diversity of our model, we provide additional text-to-3D generation results in Fig. 14
19

---

Published as a conference paper at ICLR 2025
A praying mantis wearing roller. Michelangelo-stylee statue of dog reading news on a cellphone.
A matte painting of castle made of cheesecake surrounded by a moat made of ice cream.
A chimpanzee dressed like Henry VIII king of England. 3D model of an adorable cottage with a thatched roof.
A plate piled high with chocolate chip cookies.
A vintage record player.
A car made out of cheese.
A beautifully carved wooden knight chess piece. car made out sushi.
A squirrel-octopus hybrid.
A small saguaro cactus is planted in clay pot.
A DSLR photo of an imperial state crown of England.
A rotary telephone carved out of wood.
A raccoon astronaut holding his helmet.
A classic Packard car.
A cauldron full of gold coins. blue tulip.
A stuffed grey rabbit holding pretend carrot.
A plush dragon toy.
broken egg.
A popped balloon.
Leaves flying in the wind.
A robot assembles itself.
Lightning.
The shining sun.
A melting ice cube. Ripples on water. broken bridge. Splashing waves.
Table 4: Textual prompts used in the quantitative comparison.
A.2 IMPLEMENTAION DETAILS
A.2.1 OPTIMIZATION
In this work, we conduct all of our experiments on one A100-SXM4-40GB GPU. Stage we optimize 5k steps with Adam optimizer Xie et al. (2022) to obtain a NeRF model. In Stage 2, we optimize 10k: steps for geometry optimization and 15k steps for texture optimization. During each optimization progress in Stage 2, we initially sample the timesteps ~ 4(0002,, 0.98) for the first 5k steps, and then sample from i (0.022, 0.5) for the rest steps. Each optimization process in Stage 2 requires approximately 9GB GPU memory with batch size 1 and a rendering resolution of 512.
A.2.2 TEXTUAL PROMPTS USED FOR COMPARISON
We provide the 30 randomly selected textual prompts for quantitative comparison and user study in Table. 4 To fully compare the generation capabilities of different methods and demonstrate the effectiveness of our method, the testing textual prompts include 20 textual prompts that frequently used in previous text-to-3D methods as well as 10 relatively challenging textual prompts that do not have clear main subject.
20

---

Published as a conference paper at ICLR 2025
w/o Mask-guided Compositionl ligmment
NeRF' model
Reference image
w/o global optimization
Full guidance
Figure 15: Ablation study of global optimization.
A.2.3
METRICS
perform quantitative comparisons to evaluate IPDreamer's performance, with the following met- rics:
CLIP score Gal et al. (2022) We employ CLIP score in Section 4.2 of the main paper. By assessing the alignment between the textual descriptions and the rendered images of 3D objects from various viewpoints, we can judge whether text-to-3D methods successfully generate 3D objects that match the input textual prompts.
Fréchet Inception Distance (FID) (FID) Heusel et al. 2017): To evaluate the quality of the gener- ated results, we utilize FID to compare the similarity between the rendering images of 3D objects and the images generated by the text-to-image model, Stable Diffusion.
A.2.4 USER STUDY
To further verify the quality of our generated results, we follow previous works Lin al. (2023)
Chen et al. (2023b); Wang et al. (2023) and conduct a user study by comparing IPDreamer with the
six SOTA methods Poole al. (2022) Lin et al. (2023); Chen al. (2023b); Wang et al. (0233
Hong et al. (2024): TTagg et al. (202)) under 16 prompts randomly selected from Table Each
of the 80 volunteers provided with 16 pairs of results corresponding the 16 prompts. In each
pair, one from IPDreamer and one from a randomly selected baseline. Thus, there are a total of 1280 pairwise comparisons. The volunteers are then asked to choose the better result in terms of faithfulness, quality, and fidelity.
To enhance the reliability of our user study, we provide information on the demographic distribution of the participants. The 80 volunteers included 30 university students, 20 employees from internet companies, and 30 individuals without computer science background. This diverse composition, encompassing both participants with relevant academic experience and those without, suggests that the results of our user study are reliable and generalizable.
A.2.5 DETAILS OF GPT-4V ANALYSIS
In this section, we provide detailed description of how GPT-4v is utilized generate localization prompts. Specifically, the prompt given to GPT-4v is as follows:
"You will act as an image analysis agent. Based on the input complex image condition <image">, you need to analyze which parts of the image can be used to guide 3D object synthesis. The multi- view renderings of the initialized 31) model will be provided to you the form of a video <video>.. Based on the input image condition, you are required to generate textual prompts that describe the segmented partial images, which will be used to guide the segmentation of partial image features. Each segmented partial image must also have corresponding localization prompt, mapping these partial image features onto the 3D object. Please respond in the following format: Partial image textual prompts: <textl>, <text2>, ... Corresponding localization textual prompts: <yl>, <y2>,, Note that the numbering of the partial image textual prompts must correspond one-to-one with the localization textual prompts."
21

---

Published as a conference paper at ICLR 2025
w/o Image Prompt Enhancement
w/ Image Prompt Enhancement
Figure 16: Ablation study of partial images enhancement.
Human-selected Owen-VI-SAM GPT-&--AA
Localization Prompts
Generated Objects
Initial NeRF Model
Refervncce Image
Partial Images
Selected Partial Selected Partial
Images
Images
"head"
"wing"
"tail"
"roof"
"door"
"iindow"
"yessel"
"atrium"
Figure 17: Ablation study of the accuracy of the GPT-4v and SAM.
A.3 ADDITIONAL ANALYSIS
A.3.1 EFFECTIVENESS OF GLOBAL OPTIMIZATION
To demonstrate the effectiveness of global optimization, We conduct an ablation study. As shown in Fig. 15 we present results optimized using only Mask-guided Compositional Alignment, only global optimization, and full guidance. When Mask-guided Compositional Alignment is omitted, the features partial images fail to align properly with the corresponding regions on the 3D object. Conversely, without global optimization, the generated 3D object appears distorted and unnatural. By combining the Mask Mask-guided Compositional Alignment strategy with global optimization, we can produce coherent, high-quality 3D objects. This highlights the role of Mask Alignment in accurately localizing features and the importance of global optimization in enhancing the overall quality of the generated 3D object.
Mask-guided guided Compositional
22

---

Published as a conference paper at ICLR 2025
A.3.2 EFFECTIVENESS PARTIAL IMAGES ENHANCEMENT
We utilize the super-resolution model to enhance the partial images in the main paper, especially when the resolution of partial images is low. In this part, We compare the generated 3D results with and without partial image enhancement. As shown Fig. [6] the generated 3D models may exhibit noticeable noise and artifacts when the low-resolution partial images are not enhanced. In contrast, applying partial image enhancement leads significantly improved 3D results, demonstrating the effectiveness of partial image enhancement.
A.3.3 EFFECTIVENESS OF GPT-4v ANALYSIS
To validate the effectiveness of GPT-4v in analyzing and extracting parts, as well as to make the multi-image-guideer guidded 3D object generation process easier to understand, we provide additional visu- alizations in Fig. 17 These visualizations include manually extracted partial images, partial images obtained using -4T and SAM, partial images obtained by Qwen- VL et al. (2023) and SAM, and the 3D objects generated from the partial images obtained by GPT-4v and SAM. The results show that analyzing the input conditions and extracting partial images not a particularly difficult task. The partial images extracted using MLLM (including GPT-4v and Qwen-Vil and SAM resem- ble those manually extracted. By leveraging analytical capabilities of GPT-4v and the Mask- lask-guided Compositional Alignment strategy, method can autonomously generate reasonable and high- quality 3D objects. This further demonstrates the effectiveness of GPT-4v and SAM in accurately obtaining partial images.
A.4 FUTURE WORK
Our IPDreamer can leverage complex images to guide high-quality 3D object synthesis and editing. Compared to existing text-to-3D and single-image-to33D methods, it enables more flexible and controllable synthesis of the desired 3D results. In future work, will focus improving the alignment between the generated 3D objects and the provided complex image conditions, ensuring that the generated 3D results better reflect the finer details in the complex image.
Although our IPDreamer has successfully guided high-quality 3D object synthesis in most cases using complex images, there may still be cases of failure. Future work will focus identifying and addressing such failure cases, while enhancing the generalization ability of IPDreamer and improving the quality of the generated 3D objects.
A.S SOCIAL IMPACT
Our IPDreamer does not have direct negative impact on society. However, it is important to recognize the potential of high-quality 3D objects and ensure they are not adopted for malicious purposes.
23


Reviews:
[{"review_id": "gs5T21FEgZ", "reviewer": "Reviewer_PRnJ", "summary": "The paper introduces IPDreamer which, by leveraging the complex image prompts for the first time, can generate detailed 3D objects. To achieve this task, IPDreamer first proposes an Image Prompt Score Distillation Sampling (IPSDS) that leverages both RGB features and normal features to help guide the denoising process. The authors further introduce a mask-guided compositional alignment strategy that allows for extracting corresponding features from different images of the same objects, further improving the details of the 3D generation. Extensive qualitative and quantitative experiments have been provided in the paper.", "questions": "Based on my comments on the strengths and weaknesses, I currently still lean a little bit toward the positive rating. I would like to hear from the other reviewers and the authors during the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper is the first time to consider generating 3D objects from complex images. It's quite interesting considering the current progress of the current 2D generative models.\n\n+ The paper is well-written and easy to follow.", "weaknesses": "- Fig.1 is not clear. It's not able to showcase that existing methods struggle with complex images.\n\n- The results showcased are not quite aligned with the input image.\n\n- The masks in Fig.4 are not quite aligned with the corresponding parts.\n\n- It's hard to see the effectiveness of mask-guided compositional alignment. \n\n- The results provided in Fig. 5 are not very good.\n\n- What if we apply the best text-to-2D diffusion model to the DreamFusion or other text-to-3D pipeline and carefully design the text prompts? For example, the text-to-2D diffusion model that's capable of generating complex and high-resolution images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pr2l8FoWs2", "reviewer": "Reviewer_zssc", "summary": "The paper introduces a text/image-to-3D approach for controlling the appearance of generated 3D objects given complex input images where the subject is not clearly identified.\nThe proposed approach encompasses multiple components. \nFirst, IPAdapter image encoder is used to extract image features that are used as texture guidance within the Score Distillation Sampling (SDS).\nTo be able to handle complex images with multiple components, a mask-guided compositional alignment strategy exploits a Multi-Modal Language Model (MLLM) to provide localization part labels given the image and the provided coarse Nerf model.\nThen, cross-attention maps are used to localize those parts by computing attention between the image feature and the textual labels produced by the MLLM.\nFinally, the localized parts are optimized jointly to produce a globally consistent 3D object.\nExperiments show that the proposed approach produces high-quality results that abide by the guidance image.", "questions": "- I do not understand Figure 1b. What is being generated, a 3D shape or an image? both the leaves and the water ripples look like images!\n\n- What is the difference between equations (11-13) and (14-17)? Are both used during optimization?\n\n- What is the impact of employing the super-resolution model, ControlNet tiling, on the final generated quality?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The idea of splitting complex objects into parts that are optimized jointly is interesting and can be potentially employed for more complicated 3D scenes.\n\n- The method section is comprehensive and provides an overview of SDS, making it self-contained.\n\n- The visual quality of the provided results is compelling.", "weaknesses": "- The paper primarily focuses on controlling the generation of 3D objects from complex input images. As noted in line 537, \"IPDreamer addresses the limitations of existing text-to-3D and **single-image-to-3D** methods.\" However, the paper does not include comparisons with relevant single-image-to-3D methods, such as [1] and [2]. Could the authors clarify why these comparisons were omitted?\n\n- In Figure 7, the qualitative comparison presents different samples for each method. Conventionally, all methods are evaluated on the same samples to ensure consistency in comparisons. Could the authors provide insight into this choice?\n\n- The proposed method incorporates several additional components beyond the standard SDS pipeline, including ChatGPT, SAM, ControlNet, and IPAdapter. Could the authors provide details on the runtime overhead introduced by each component, as well as the overall runtime?\n\n- The method illustration in Figure 2 appears challenging to interpret. It does not effectively aid in understanding the proposed pipeline, and I found it difficult to correlate it with the text. A more intuitive figure might improve readability and clarity.\n\n[1] Shi, Ruoxi, et al. \"Zero123++: a single image to consistent multi-view diffusion base model.\" arXiv preprint arXiv:2310.15110 (2023).\n\n[2] Voleti, Vikram, et al. \"Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion.\" European Conference on Computer Vision. Springer, Cham, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XbtYcZJ7Q1", "reviewer": "Reviewer_sW6c", "summary": "The paper introduced a controllable 3D object generation approach using image prompts (similar to style transfer). The proposed IPDreamer approach is a novel method that could capture the intricacies of appearance features from image prompts, and could generate high fidelity and controllable 3D objects. The approach is tested on some public benchmarks with user studies available as well, and was proven to be effective.", "questions": "See the weakness section", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is tackling an important and very challenging 3D genAI problem. Comparing to existing approaches, the IPDreamer could edit the objects using more complex image prompts\n- The introduced prompt score distillation sampling approach is a reasonable formulation that builds on existing SDS approaches, and the masked-guided alignment strategy seems to be highly effective \n- Experimental results suggest that the approach is better comparing to other counterparts. User studies is also provided.", "weaknesses": "I think this is a nice paper and a good extension to many of the existing approaches. The final output of the algorithm seems to be good enough. I do have a few clarification questions that I hope the authors could address in future revisions of the papers:\n- The paper leverages GPT-4v as MLLM inputs. How accurate should the MLLM be, in case people don't have access to this advanced MLLM algorithm? Would the output become much worse? \n- It's very nice to conduct user studies for genAI works in general. Could authors provide more demographics information in the appendix section? (age, gender, background, etc)\n- I don't fully understand Fig 1b, especially the right images -- what is the contents in the input and what is the actual real-world application of this particular input/output pair?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZ1f57JCoO", "reviewer": "Reviewer_4vLr", "summary": "This paper present a novel method to capture intricate appearance features from the Image prompts, which is further used to enhance the alignment of the image prompts with the generated objects. Experiments demonstrate the proposed method generate objects which is well-aligned with image prompts, show better ability in complex generation.", "questions": "See Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper proposes a novel framework for 3D generation by breaking an image prompt into several parts and adopting a multi-guidance optimization process. Experiments demonstrate the effectiveness of the proposed framework.\n* The idea of the paper that breaks the complex images into several parts is interesting and good. Breaking a complex thing into parts makes a hard problem much easier.", "weaknesses": "* The written of the paper is not so clear, some details are lack:\n  * The description on how to adopt GPT-4v to generate localization prompts is lack in the paper.\n  * In Figure 1 (b), the author gives comparison between VSD and IPSDS on text-based generation. But is the proposed method IPSDS need an image prompt? How to compare IPSDS with VSD on text-based generation? Moreover, for the cases in Fig (a), could the author provide the images parts extracted from the reference image of the castles. It’s hard to understand how could we break such things into parts.\n  * For eq.9 and eq.10,the author highlights that “they localize the features of the multiple images onto 3D object” in many places such as Line 321-322, 349-350, which makes me very confused. I think the author is adopting eq.9 and eq.10 to fuse information from different image parts to do SDS loss. Therefore, this description is inaccurate and leads to misunderstanding. 、\n  * Some annotation in the equations are missing, like $Z$ in eq.9.\n* In line 360, the author declares that a global optimization is further needed, which is achieved by simply concatenating all the features from the multiple images instead of adopting a mask based strategy. Why we need such a global optimization? What if we directly adopt global optimization without the mask-guided one? I think the author should provide such evaluation.\n* Finally, I think the evaluation of the paper is not enough. The accuracy of adopting SAM and GPT-4v to break into parts is not evaluated. Moreover, I think the author should provide more visualization examples on the extracted image parts together with the generation results, which will make overall process easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)