Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts
Authors: Xinhua Cheng
Tianyu Yang
Jianan Wang
Yu Li
Lei Zhang
Jian Zhang
Li Yuan
Venue: ICLR 2024 poster (2024)
Keywords: Text-to-3D Creation
3D Content Editing
Attribute Mismatching
Primary area: generative models

Abstract: Recent text-to-3D generation methods achieve impressive 3D content creation capacity thanks to the advances in image diffusion models and optimizing strategies. However, current methods struggle to generate correct 3D content for a complex prompt in semantics, i.e., a prompt describing multiple interacted objects binding with different attributes. In this work, we propose a general framework named Progressive3D, which decomposes the entire generation into a series of locally progressive editing steps to create precise 3D content for complex prompts, and we constrain the content change to only occur in regions determined by user-defined region prompts in each editing step. Furthermore, we propose an overlapped semantic component suppression technique to encourage the optimization process to focus more on the semantic differences between prompts. Extensive experiments demonstrate that the proposed Progressive3D framework generates precise 3D content for prompts with complex semantics through progressive editing steps and is general for various text-to-3D methods driven by different 3D representations.

Published as a conference paper at ICLR 2024
PROGRESSIVE3D: PROGRESSIVELY LOCAL EDITING FOR TEXT-TO-3D CONTENT CREATION WITH COM- PLEX SEMANTIC PROMPTS
Xinhua Cheng" Tianyu Yang!?, Jianan Wang", Yu Li", Lei Zhang", Jian Zhang', Li Yuan" 1,3
!Peking University
Intternational Digital Economy Academy (IDEA)
"Peng Cheng Laboratory
ABSTRACT
Recent text-to-3D generation methods achieve impressive 3D content creation ca- pacity thanks to the advances in image diffusion models and optimizing strate- gies. However, current methods struggle to generate correct 3D content for a complex prompt in semantics, i.e., prompt describing multiple interacted ob- jects binding with different attributes. this work, we propose a general frame- work named Progressive3D, which decomposes the entire generation into se- ries of locally progressive editing steps to create precise 3D content for complex prompts, and constrain the content change only occur in regions determined by user-defined region prompts in each editing step. Furthermore, we propose overlapped semantic component suppression technique encourage the op- timization process to focus more on the semantic differences between prompts. Experiments demonstrate that the proposed Progressive3D framework is effec- tive in local editing and is general for different 3D representations, leading to precise 3D content production for prompts with complex semantics for various text-to-3D methods. Our project page is https: pestps://cxhoie /cxh0519 1, gi thub. io/ Sefjects//rroressivvee
INTRODUCTION
High-quality 3D digital content that conforms to the requirements of users is desired due to its var- ious applications in the entertainment industry, mixed reality, and robotic simulation. Compared to the traditional 3D generating process which requests manual design in professional modeling software, automatically creating 3D content with given text prompts is more friendly for both begin- ners and experienced artists. Driven by the recent progress of neural 3D representations (Mildenhall et al,, 2020; Wang et al., 2021; Yariv et al., 2021; Shen et al., 2021) and text-to-image (T2I) diffusion models (Nichol al., 2021; Rombach et al., 2022; Mou et al., 2023; Zhang et al., 2023), Dreamfu- sion (Poole et al., 2022) demonstrates impressive 3D content creation capacity conditioned on given prompts by distilling the prior knowledge from T2I diffusion models into Neural Radiance Field (NeRF), which attracts board interests and emerging attempts in text-to-3D creation.
Although text-to-3D methods have tried use various 3D neural representations (Lin et al., 2023; Chen et al., 2023; Tsalicoglou et al., 2023) and optimization strategies (Wang et al., 2023a; Huang al., 2023b; Wang al., 2023b) for improving the quality of created 3D content and achieving remark accomplishments, they rarely pay attention to enhancing the semantic consistency between generated 3D content and given prompts. As a result, most text-to-3D methods struggle to produce correct results when the text prompt describes a complex scene involving multiple objects binding with different attributes. As shown in Fig. I(a), existing text-to-3D methods suffer from challenges with complex prompts, leading significant object missing, attribute mismatching, and quality reduction. While recent investigations (Feng et al., 2022; Huang et al., 2023a; Lu et al., 2023) have demonstrated that current T2I diffusion models tend to generate inaccurate results when facing
Work done during the internship at IDEA.
"ooresponndig Authors.

---

Published as a conference paper at ICLR 2024
Dream Time
astronout holding rifle and riding green origami torcycle and wearing cyon chef's
astronaut holding red rifle and riding green origami motoncycle
"An astrenaut hoiding red rifle and riding green origami motorcycle and wearing cyan chef's
"An astronaut."
"An astronaut holding red rifle."
TextMesh
"A yellow tullp and a blue peony and rose golden vase.
"A blue peonyy and red golden vose."
"A yellow tulip and a blue peorry red rose golden vase."
golden
"A red rose golden wase."
Fantasia3D
"The Zronman with Gundom's and Hulk's and Spidermon's body.""
"The Zronman with Sundom's
"The Ironman with Gundam's and
"The Ironmon with Gundom's and Hulk's orm and Spiderman's body."
"The Ironman."
MVDream
"A medieval soldier with metal armor wearing white astronaut helmet and holding a golden oxe riding tervacotta woif."
"A medieval soldier with metal anner."
medieval soldier with metal armor
medieval soldier with metal ormor haiding a golden riding tervocotte wolf."
soldier metal armor wearing on white astronaut heimet and holding golden and riding terracotta wolf."
holding a golden oxe.
(a) Generating Directly
(b) Generating with Progressive3D
Conception. Current text-to-3D methods suffer from challenges when given prompts de- scribing multiple objects binding with different attributes. Compared to (a) generating with existing methods, (b) generating with Progressive3D produces 3D content consistent with given prompts.
Figure I:
prompts with complex semantics and existing text-to-3D methods inherit the same issues from T2I diffusion models, works evaluating improving the performance of text-to-3D methods in com- plex semantic scenarios are still limited. Therefore, how to generate correct 3D content consistent with complex prompts is critical for many real applications of text-to-3D methods.
To address the challenges of generation precise 3D content from complex prompts, we propose general framework named Progressive3D. which decomposes the difficult creation of complex prompts into series of local editing steps, and progressively generates the 3D content as is shown Fig. 1(b). For a specific editing step. our framework edits the pre-trained source representation in the 3D space determined by the user-defined region prompt according to the semantic differ- ence between the source prompt and the target prompt. Concretely, we propose two content-related constraints, including a consistency constraint and initialized constraint for keeping content be- yond selected regions unchanged and promoting the separate target geometry generated from empty space. Furthermore, technique dubbed Overlapped Semantic Component Suppression (OSCS) is carefully designed to automatically explore the semantic difference between the source prompt and the target for guiding the optimization process of the target representations.
To evaluate Progressive3D, we construct a complex semantic prompt set dubbed CSP-100 consisting of 100 various prompts. Prompts in CSP-100 are divided into four categories including color, shape,
2

---

Published as a conference paper at ICLR 2024
material and composition according appeared attributes. Experiments conducted on existing text- to-3D methods driven by different 3D representations including NeRF- DreamTime (Huang et al., 2023b) and MVDream(Shi et al., 2023), SDF-based TextMesh (Tsalicoglou et al., 2023), and DMTet-based Fantasia3D (Chen et al., 2023) demonstrate that our framework produces precise 31) models through multi-step local editing achieve better alignment with text prompts both in metrics and user studies than current text-to-3D creation methods when prompts are complex in semantics.
Our contribution can be summarized as follows: (1) We propose framework named Progressive3D for creating precise 3D content prompted with complex semantics by decomposing a difficult gener- ation process into a series of local editing steps. (2) We propose the Overlapped Semantic Compo- nent Suppression to sufficiently explore the semantic difference between source and target prompts for overcoming the issues caused by complex prompts. (3) Experiments demonstrate that Progres- sive3I is effective in local editing and is able to generate precise 3D content consistent with complex prompts with various text-to-3D methods driven by different 3D neural representations.
2 RELATED WORKS
Text-to-3D Content Creation. Creating high-fidelity 3D content from only text prompts has at- tracted broad interest in recent years and there are many earlier attempts (Jain et al., 2022; Michel et al., 2022; Mohammad Khalid ct al., 2022). Driven by the emerging -ix--to-mmae diffusion mod- els. Dreamfusion (Poole et al., 2022) firstly introduces the large-scale prior from diffusion models for 3D content creation by proposing the score distillation sampling and achieves impressive re- sults. The following works can be roughly classified into two categories, many attempts such as SJC (Wang et al., 2023a), Lete (Metzer et al., 2022), Score Debiasing (Hong et al., 2023) Dream'Time (Huang al., 2023b), ProlificDreamer (Wang al., 2023b) and MVDream(Shi al., 2023) modify optimizing strategies to create higher quality content, and other methods including Magic3D (Lin et al., 2023), Fantasia3D (Chen et al., 2023), and TextMesh (Tsalicoglou al., 2023) employ different 3D representations for better content rendering and mesh extraction. However, most existing text-to-3DD methods focus on promoting the quality of generated 3D content, thus their methods struggle to generate correct content for complex prompts since no specific techniques are designed for complex semantics. Therefore, we propose a general framework named Progressive3D for various neural 3D representations to tackle prompts with complex semantics by decomposing the difficult generation into a series of local editing processes, and our framework successfully produces precise 3D content consistent with the complex descriptions.
Text-Guided Editing on 3D Content. Compared to the rapid development of text-to-3D creation methods, the explorations of editing the generated 3D content by text prompts are still limited. Although Dreamfusion (Poole et al., 2022) and Magic3D (Lin et al., 2023) demonstrate that content editing can be achieved by fine-tuning existing 3D content with new prompts, such editing is unable to maintain 3D content beyond editable regions untouched since the fine-tuning is global to the entire space. Similar global editing methods also include Instruct NeRF2NeRF (Haque et al., 2023) and Instruct 3D-to-3D (Kamata et al., 2023), which extend a powerful 2D editing diffusion model named Instruct Pix2Pix (Brooks et al., 2023) into 3D content. Furthermore, several local editing methods including Vox-E (Sella et al., 2023) and DreamEditor (Zhuang et al., 2023) are proposed to edit the content in regions specified by the attention mechanism, and FocalDreamer (Li et al., 2023) only generates the incremental content in editable regions with new prompts to make sure the input content is unchanged. However, their works seldom consider the significant issues in 3D creations including object missing, attribute mismatching, and quality reduction caused by the prompts with complex semanties. Differing from their attempts, our Progressive3D emphasizes the semantic difference between source and target prompts, leading to more precise 3D content.
3 METHOD
Our Progressive3D framework is proposed for current text-to-3D methods to tackle prompts with complex semantics. Concretely, Progressive3D decomposes the 3D content creation process into a series of progressively local editing steps. For each local editing step. assuming we already have a source 3D representation &. supervised by the source prompt y., we aim to obtain a target 3D representation Φe which is initialized by d. satisfy the description of the target prompt y, and
3

---

Published as a conference paper at ICLR 2024
the 3D region constraint of user-defined region prompts yb- We first convert user-defined region prompts to 2D masks for each view separately to constrain the undesired contents in Φi untouched (Sec. 3.1), which is critical for local editing. Furthermore, we propose the Overlapped Semantic Component Suppression (OSCS) technique to optimize the target 31) representation Φ, with the guidance of the semantic difference between the source prompt ys and the target prompt yı (Sec. 3.2) for emphasizing the editing object and corresponding attributes. The overview illustration of our framework is shown in Fig
3.1 EDITABLE REGION DEFINITION AND RELATED CONSTRAINTS
In this section, we give the details of the editable region definition with region prompt y/ and designed region-related constraints. Instead of directly imposing constraints on neural 3D represen- tations to maintain 3D content beyond selected regions unchanged, we adopt 2D masks rendered from 3D definitions as the bridge to connect various neural 3D representations (e.g.. NeRF, SDF, and DMTet) and region definition forms (e.g., 3D bounding boxes, custom meshes, and 2D/3D seg- mentation results (Liu et al., 2023; Cheng et al., 2023; Cen et al., 2023)), which enhances the gener- alization of our Progressive3D. We here adopt NeRF the neural 3D representation and define the editable region with 3D bounding box prompts for brevity.
Given a 3D bounding box prompt y/, = [Cas Cys Cgi S2. Sy: which user-defined for specifying Cy: Ca] is the coordinate position of the box center, and [sz, Sy. sz] is the box size on the {a: y, =]-axis respectively. We aim to obtain the corresponding 2D mask M, converted from the prompt and pre-trained source representation Φ. that describes the editable region for specific view v. Concretely, we first calculate the projected opacity map Ô and the projected depth map D of d. similar to the Eq. 10. Then we render the given bounding box to obtain its depth D2 = render v, R), where a is the current view and R is the rotate matrix of the bounding box. Before calculating the 2D editable mask M, at specific v, modify the projected depth map D according Ô to ignore the floating artifacts mistakenly generated in Φ.:
the editable region in 3D space, where [Ca:Cy:
D(r) { OG, if ((r) < Toi
(r)), otherwise;
where r E R is the ray set of sampled pixels in the image rendered at view v, and To is the filter threshold. Therefore, the 2D mask M, of the editable region, well the 2D opacity mask M.. can be calculated for the following region-related constraints:
M((r) [1, if Do(r) D(r); Mc(r) = -(1 1, if Ô(r) To: (o, otherwise.
0, otherwise.
Content Consistency Constraint. We emphasize that maintaining 3D content beyond user-defined editable regions unchanged during the training of the target representation d: is critical for 3D editing. We thus propose content consistency constraint to impose the content between the target representation Φi and the source representation d, to be consistent in undesired regions, which conditioned by obtained 2D mask M, which represents the editable regions:
Lrossist
Σ PER
M.(r)).(r)
- (((()))-((r)
(3)
where M, = Mt the inverse editable mask, M, = 1 M, the inverse opacity mask, and Če Ce are projected colors of Φb. Φi respectively.
Instead of constraining the entire unchanged regions by color similarity, we divide such regions into a content region and an empty region according to the modified opacity mask M,, and an additional term is proposed to impose the empty region remains blank during training. We separately constrain content and empty regions to avoid locking the backgrounds during the training, since trainable backgrounds are proved (Guo et al., 2023) beneficial for the quality of foreground generation.
Content Initialization Constraint. our progressive editing steps, a usual situation is the corre- sponding 3D space defined by region prompts is empty. However, creating the target object from
4

---

Published as a conference paper at ICLR 2024
Overlapped Semontic Component Suppression
User-Defined Region Prompt Cay 8g. R2)
DG
(2(rr)
Kelih V.n!
3G
Pre-troning
Source Representation
Source Prompt
Inihoiize
aatronout"
M,(r)
23 Ditfus Model
Lans
Tanget Prompt
ostronaut hoiding
Coonater
Target Representation
C,(r)
Cindtal
Figure 2: Overview of local editing step of our proposed Progressive3D. Given source repre- sentation Φ, supervised by source prompt ys. our framework aims to generate a target representation Φr conforming to the input target prompt y/: in 3d space defined by the region prompt yo- Condi- tioned on the 2D mask (r(r), we constrain the 3D content with Lomsist and Linital- We further propose an Overlapped Semantic Component Suppression technique impose the optimization fo- cusing more on the semantic difference for precise progressive creation.
(d) Source Content and Region Prompt
(b) 2D Mask M,
(c)w/o Cennsse
(d) w/o Cunttaal (e) w/o Overlapped Semantic Component Suppression
(f) Ours
Figure Qualitative ablations. The source prompt y,="A medieval soldier with metal armor holding golden axe." and the target prompt yA="A medieval soldier with metal armor holding a golden axe and riding a terracotta wolf." " where green denotes the overlapped prompt and red denotes the different prompt.
scratch often leads rapid geometry variation and causes difficulty in generation. We thus provide a content initialization constraint to encourage the user-defined 3D space filled with content, which is implemented by promoting Ô increase in editable regions during the early training phase:
k K)
Linstal = <sk))) Σ M.(r) - =11 m(k) = A(1
if o
(4)
PER
otherwise.
where x(k) is weighting function of the current training iteration k. is the scale factor of the maximum strength, and is the maximum iterations that apply this constraint to avoid impacting the detail generation in the later phase.
3.2 OVERLAPPED SEMANTIC COMPONENT SUPPRESSION
Although we ensure the content edits only occur in user-defined regions through region-related constraints, obtaining desired representation d, which matches the description in the target prompt Y: is still challenging. An intuitive approach to create Φ, is fine-tuning the source representation Φ, with the target prompt y, directly (Poole et al., 2022; Lin et al., 2023). However, we point out that merely leveraging the target prompt y, for fine- fine-grained editing will cause attribute mismatching issues, especially when yt describes multiple objects binding with different attributes.
For instance in Fig, 3, have obtained source representation &. matching the source prompt ys and a target prompt ys for the following local editing step. If we adjust d, guided by ye directly, shown in Fig. 3(e), the additional content "wolf" could be both impacted by additional attribute "terracotta" and overlapped attribute "metal, golden" " during the generation even the overlapped attribute has been considered in Φ., which leads undesired result with attribute confusing.

---

Published as a conference paper at ICLR 2024
Dream Time Dream
Prognessives)
grean on shar-dhsped Enay."
red aronge and and red chefie hot."
"A blue pisk
tny on Aereyona tolle."
cream and riding instacyclle
culee in yellow Eney."
(o) Object Missing
Attribute Mismatching
(c) Quality Reduction
Figure 4: Current text-to-3D methods often fail to produce precise results when the given prompt describes multiple interacted objects binding with different attributes, leading to significant issues including object missing, attribute mismatching, and quality reduction.
Furthermore, the repeated attention overlapped prompts causes the editing process less consider the objects described in additional prompts, leading to entire or partial object ignoring (e.g., "wolf" mistakenly created without its head and integrated with the soldier). Hence, guiding the optimiza- tion in local editing steps to focus more on the semantic difference between ys and instead of yı itself is critical for alleviating attribute mismatching and obtaining desired 3D content.
Therefore, we proposed technique named Overlapped Semantic Component Suppression (OSCS) inspired by (Armandpour et al., 2023) to automatically discover the overlapped semantic component between ys and y/: with vector projection, and OSCS then suppresses the overlapped component to enhance the influence of the different semantic during the training of Φi for precise content creation. Concretely, both prompts ys and ye firstly produce separate denoising components with the unconditional prediction co (zz, t):
A = cœ Ye.t) t) - c0( Dej = c yt. 6) - =E(
(5)
As shown in Fig. 2, we then decompose Ae' into the projection component Deg and the perpen- dicular component Aegrey by projecting Aeg on Aeg:
Ach = (Aeg. Acg)
(Ac
A Aeg
AC
| ACt =
AeG proj
+ Acgrep
(6)
Projection Compoment
Perpendicular Component
where (.,-) denotes the inner product. We define Ac as the overlapped semantic component since it is the most correlated component from Αeğ AEă and regard Ae as the different semantic component which represents the most significant difference in semantic direction. Fur- thermore, we suppress the overlapped semantic component Aepros during training for reducing the influence appeared attributes, and the noise sampler with OSCS formulated as:
èo (xt, yos yr. t) = co (at,
Acg proj + wAek W 1,
W
where w the original guidance scale in CFG described in Eq. 14, and W the weight to control the suppression strength for the overlapped semantics. We highlight that W 1 is important for the suppression, since éø (x,, Ye Yr, t) is degenerated to éø Ye, t) when W 1. Therefore, the modified Score Distillation Sampling (SDS) with OSCS is formulated follows:
Ox
V CoLS (0, x) E,,e (t))(E Uo Yc.
c)
(8)
dΦ
Compared to Fig. 3(e), leveraging OSCS effectively reduces the distraction of appeared attributes and assists Progressive3D in producing desired 3D content, as is shown in Fig. 3(f).
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We only provide important experimental settings including dataset, metrics, and baselines here due the page limitation, more detailed experimental settings be found at Appendix B.

---

Published as a conference paper at ICLR 2024
Dream Time
"A toole."
goldin table"
and golden table"
"A round cobleet
M Aexeyona oup on roudd
"A yeilaa pianopple Resegonal cup on round
Dream Time *Progressive33
Text
"A toy robot.
tay robet nearing o goiden shirt"
"A tay robut seervig geider shirt one
of buildeg
model buildig with square reot."
of buildhy with
TextMesh +Progr eesivresseeeD
haragead
Fontasia3D
ettnndigg Neok
"A standing black Shibe Try aroving golden aventer."
"A stonetnn binck Shiba Iru wearing & galder sancter and
head tervecotto westivggs red aungiass.
head tecrocotto only waaring red nnnglose and groy hot
Dontosio3D -Progressive3D
termoctto
MVDream
inttll
wid rose it
yellow tulp
golddn opione."
"s ceroni lin driving golsón piplase""
cenamic lon asarieg brotze chef and driving o golden byptone.
+Progressie essiieeeDDDD
MVDream
Figure 5: Progressive editing processes driven by various text-to-3I methods equipped with Progressive3D. Compared to original methods, Progressive3D assists current methods in tackling prompts with complex semantics well. 3D Cyan boxes denote the user-defined region prompts.
Dataset Construction. We construct a Complex Semantic Prompt set named CSP-100 which in- volves 100 complex prompts to verify that current text-to-3I methods suffer issues when prompts are complex semantics and proposed Progressive3D efficiently alleviates these issues. CSP-100 introduces four sub-categories of prompts including color, shape, material, and composition accord- ing to the appeared attribute and more details are in Appendix B.
Evaluation Metries. Existing text-to-3D methods (Poole et al., 2022; Tsalicoglou et al., 2023; Li et al., 2023) leverage CLIP-based metrics to evaluate the semantic consistency between generated 3D creations and corresponding text prompts. However, CLiP-based metrics verified (Huang et al., 2023a; Lu et al., 2023) that fail to measure the fine- geaiieained correspondences between de- scribed objects and binding attributes. We thus adopt recently proposed metrics fine-grained
7

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison on met- rics and user studies over CSP-100.
Dream Time
Dream Time +CEBM
Dream Time +AME
Dream Time -Progressive3D
Metrics
Method
B-VQA t mGPT-CoT Preference
Dream Time
0.227
0.522
16.8%
CEBM
0.186
0.491
"A ceramic tea pot and lego car on a golden table."
A&&
0.243
0.528
Progressive3D
0.474
0.609
83.2%
"As actenaad kadrga prldo estiooal
Ours
'A triangie sandwich on round cabinet."
mentunng
"A yellow tulip and a white tulip pink vase."
Figure Qualitative ablations between fine-tuning with target prompts and editing with Progressive3D on MVDream.
Figure 6: Visual comparison with DreamTime-Be
compositional generation baselines.
including BLIP-VQU VQA and mGPT-CoT (Huang et al., 2023a), evaluate the generation capacity of current methods and our Progressive3D when handling prompts with complex semantics.
Progressive3D with 4 text-to-3D methods driven by different 3D representations: (1) DreamTime (Huang et al., 2023b) is a NeRF-based method which enhances DreamFusion (Poole al., 2022) in time sampling strategy and produce better results. We adopt DreamTime as the main baseline for quantitative comparisons and ablations due to its stability and training efficiency. (2) TextMesh (Tsalicoglou et al., 2023) leverages SDF the 3D representation to improve the 3D mesh extraction capacity. (3) Fantasia3D (Tsalicoglou et al., 2023) is driven by DMTet which produces impressive 3D content with disentangled modeling process.
Baselines. We incorporate
(4) MV- Dream (Shi et al., 2023) is NeRF-based method which leverages pre- pre-trained multi-view consis- tent -ox--to- model for text-to-3D generation and achieves high-quality 3D content generation performance. To further demonstrate the effectiveness of Prgressive3D., we re-implement two com- posing text-to-image methods including Composing Energy-Based Model (CEBM) (Liu al., 2022) and Attend-and-Exiin (A&E) (Chefer et al., 2023) Dream for quantitative comparison.
4.2 PROGRESSIVE3D FOR TEXT-TO-3D CREATION AND EDITING
Comparison with current methods. We demonstrate the superior performance of our Progres- sive3D compared current text-to-3D methods in both qualitative and quantitative aspects this section. We first present visualization results in Fig. 4 to verify that DreamTime faces significant challenges including (a) object missing, (b) attribute mismatching, and (c) quality reduction when given prompts describe multiple interacted objects binding with different attributes. Thanks to our careful designs, Progressive3D effectively promotes the creation performance of DreamTime when dealing with complex prompts. In addition, more progressive editing processes based on various text-to-3D methods driven by different neural 3D representations are shown in Fig. 5, which further demonstrate that Progressive3D stably increases the generation capacity of based methods when given prompts are complex, and our framework is general for various current text-to-3D methods.
We also provide quantitative comparisons on fine-grained semantic consistency metrics including BLIP-VQA and mGPT-CoT, and the results are shown Tab. 1, which verify that our Progres- sive3D achieves remarkable improvements for 3D content creation with complex semantics com- pared to DreamTime based baselines. As shown in Fig. 6, baselines that combine 2D composing T21 methods including CEBM (Liu et al., 2022) and A&E (Chefer al., 2023) with Dream'Time
8

---

Published as a conference paper at ICLR 2024
Quantitative ablation studies for proposed constraints and the OSCS tech- nique based on Dream'Time over CSP-100.
Table 2:
TextMett DreanTime
Components
Metrics
"An astronal helding med ieslle pu
Index Comssis Cintiaa OSCS B-VQA mGPPTCCTT
0.255
0.567
Source Content and Region Prompt
Target Prompt
W
IV m
W -
0,370
01577
0.347
0,581
0.474
0.609
Figure Qualitative ablations for suppression weight W in proposed OSCS.
still achieve limited performance for complex 3D content generation, leading to significant issues including object missing, attribute mismatching, and quality reduction. Furthermore, we collected 20 feedbacks from humans to investigate the performance of our framework. The human prefer- ence shows that users prefer our Progressive3D in most scenarios (16.8% vs. 83.2%), demonstrating that framework effectively promotes the precise creation capacity of Dream when facing complex prompts.
4.3 ABLATION STUDIES
In this section, we conduct ablation studies on Dream and TextMesh to demonstrate the effec- tiveness of proposed components including content consistency constraint Leonsist, content initial- ization constraint Linitial and Overlapped Semantic Component Suppression (OSCS) technique, we highlight that brief qualitative ablation is given in Fig. 3.
We first present ablation results between fine-tuning directly and editing with Progressive3D based on TextMesh in Fig. 7 to demonstrate that fine-tuning with prompts cannot maintain source ob- jects prompted by overlapped semantics untouched and is unusable for progressive editing. Another visual result in Fig. 8 shows the parameter analysis of the suppression weight in OSCS. With the W decreases), the different semantics between source and target prompts play
increase of W (i.e,, & more important roles optimizations and result in more desirable 3D content. On the contrary, the progressive step edits failed results with object missing or attribute mismatching issues when we increase the influence of overlapped semantics by setting W 0.5, which further proves that our explanation of perpendicular and projection components is reasonable.
We then show the quantitative comparison in Tab. to demonstrate the effectiveness of each pro- posed component, where content consistency constraint is not involved in quantitative ablations since consistency the foundation of 3D content local editing which guarantees content beyond user-defined regions untouched. We underline that Limitial is proposed to simplify the geometry generation from empty space and OSCS is designed to alleviate the distraction of overlapped at- tributes, thus both components This has been proofed by the quantitative ablations Tab. 2: index 2 and show that applying Cinitial and OSCS alone both promote the metrics compared to the baseline in index and in- dex 4 shows that leveraging both Linitial and OSCS together can further contribute to the creation performance over CSP-100.
benefit the creation performance with conffict theoretically.
CONCLUSION
In this work, we propose general framework named Progressive3D for correctly generating 3D content when the given prompt is complex in semantics. Progressive3D decomposes the difficult creation process into series of local editing steps and progressively generates the aiming object with binding attributes with the assistance of proposed region-related constraints and the over- lapped semantic suppression technique each step. Experiments conducted on complex prompts in CSP-100 demonstrate that current text-to-3D methods suffer issues including object missing, at- tribute mismatching, and quality reduction when given prompts are complex in semantics, and the proposed Progressive3D effectively creates precise 3D content consistent with complex prompts through multi-step local editing. More discussions on the limitations and potential directions for future works provided in Appendix A.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (2022ZD01 18101), the Nat- ural Science Foundation of China (N..62202014, Shenzhen Basic Research Program under Grant JCY12022081315177600 and also sponsored by CCF Tencent Open Research Fund.
REFERENCES
Mohammadreza Armandpour, Huangjie Zheng. Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou, Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.044 049668 2023. 6
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13392-18402 2023. 3
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. arXiv preprint arXiv:2304. 12300 2023,
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.1387,, 2023. 1,3,
Xinhua Cheng, Yanmin Wu, Mengxi Jia, Qian Wang, and Jian Zhang. Panoptic compositional feature field for editable scene rendering with network-inferred labels via metric learning. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4947- 4957, 2023. 4
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, 2023. URL https:// Imsys. http://ggg/bbo///o//////
03 30 30-vicuna/. 16
DeepFloyd-Team. Deepfloyd-if, 2023,
URL https: /ott://///////////g/t///aaaaaae
IF- IF-I-XI- XI--X-vv.... 17
Weixi Feng. Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05033, 2022. 1
Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang. Yan-Pei Cao, and Song-Hai Zhang. threestu- dio: unified framework for 3d content generation, 2023. URL https: //t::///ittt //gpthub.. com/
4, 17
Ayaan Haque, Matthew Tancik. Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerff: Editing 3d scenes with instructions. arXiv preprint arXiv:2303.12788 2023.
Jonathan Ho and Tim Salimans.
Classifier-free
diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022. 18
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Pro- ceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp- 6840-6851, 2020. 18
Susung Hong, Donghoon Ahn, and Seung Wook Kim. Debiasing scores and prompts of diffusion for robust text-to-3d generation. ArXiv, abs/2303. 15413, 2023.
10

---

Published as a conference paper at ICLR 2024
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A com- prehensive benchmark for open-world compositional text-to-image image generation. arXiv preprint arXiv:2307.06350, 2023a. 7,7, 8,
Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. /2422, 2023b. 1,3,8 8, 17
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- sext-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 867-876, 2022. 3
Hiromichi Kamata, Yuiko Sakuma, Akio Hayakawa, Masato Ishii, and Takuya Narihira. Instruct 3d-to-3d: Text instruction guided 3d-to-3d conversion. arXiv preprint arXiv:2303.15780, 2023.
3
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint (rXii:2304002644 2023. 14
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping ianguage-imagee pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12906 2022. 16
Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bing- bing Ni. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint arXiv:2308, 10608, 2023. 3, 7, 14, 16
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung- Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 300-309, 2023. 1,3,5
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022. 8
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 230.:2303.05499 2023. 4, 14
Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Limscore: Unveil- ing the power of large language models in text-to-image synthesis evaluation. arXiv preprint arXiv:2305. 11.30..11116...... 2023. 1, 7,7,14
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:221 21.222211.070000 2022. 3
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13492-13502, June 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405-421, 2020. 1, 18
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Gener- ating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference papers, pp. 1-8, 2022. 3
Chong Mou, Xintao Wang, Liangbin Xie, Jing Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. ArXiy, abs/2302.08... 08845,, 2023.
11

---

Published as a conference paper at ICLR 2024
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistie image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2021.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14998, 2022. 1, 3, 5, 7, 8, 16, 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. 16
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 1, 17
Etai Sella, Gal Fiebelman, Peter Hedman, and Hadar Averbuch-Elor. Vox-e: Text-guided voxel editing of 3d objects. arXiv preprint arXiv:2303. /2048, 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yi Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021. 1
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. /65 2023. 3,8
Jascha Sohl-Dickstein, ErieL. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv: Learning, arXiv: Learning, 2015.
18
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arxiv:2304.12439 2023. 1, 3, 7, 8, 16, 17
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich, Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12619-12629, 2023a. 1,3
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint 1rXiv:2106.1066 2021.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. (6213,20 2023b. 1,
Lior Yariy, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805 4815, 2021.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE International Conference Computer Vision (ICCV), 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. 16
Jingyu Zhuang, Chen Wang. Lingjie Liu, Liang Lin, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint ar 2306. 13455, 2023. 3, 14
12

---

Published as a conference paper at ICLR 2024
A DISCUSSIONS
A.1 REALISTIC USAGE OF PROGRESSIVE3D
We note that Progressive3D contains multiple local editing steps for creating complex 3D content, which accords with user usage pipeline, i.e, creating primary object first, then adjusting its at- tribute or adding more related objects. However, Progressive3D is flexible in realistic usage since the generation capacity of basic text-to-3D method and user goals are variant. For instance in Fig. 9, desire to create the 3D content consistent with the prompt "An astronaut wearing green top hat and riding red horse". We find that MVDream fails to create the precise result while generating "An astronaut riding a red horse" correctly. Thus the desired content can be achieved by editing "An astronaut riding a red horse" within one-step editing, instead starting from "an astronaut".
MVDream
Progressive3D
"An astronaut wearing a green top hat and riding red horse."
"nn astronaut wearing a green top hat and riding red horse."
"An astronaut riding a red horse."
Figure 9: MVDream successfully creates "An astronaut riding red horse" while failing to create "An astronaut wearing green top hat and riding red horse" By leveraging one-step Progressive33 editing, correct 3D content is obtained.
A.2 OBJECT GENERATING ORDER
Different object generating orders in Progressive3D typically result in correct 3D content consistent with the complex prompts. However, the content details of the final content are impacted by created objects since Progressive3D is local editing chain started from the source content, and we give an instance in Fig. 10. With different generating orders, Progressive3D creates 3D content with different details while they both consistent with the prompt "An astronaut sitting wooden chair". In our experiments, we first generate the primary object which is desired to occupy most of the space and interact with other additional objects.
"An astronaut"
"An astronaut sitting wooden chair"
*A wooden chair"
"An astronaut sitting wooden chair"
Figure 10: Creating "An astronaut sitting on a wooden chair" from different generating orders.
13

---

Published as a conference paper at ICLR 2024
A.3 VARIOUS REGION DEFINITIONS
We highlight that Progressive3D is a general framework for various region definition forms since the corresponding 2D mask of each view be achieved. As shown in Fig. 11, Progressive3D performs correctly on various definition forms including multiple 3D bounding boxes, custom mesh, and the fine-grained 2D segmentation prompted by the keyword "helmer" through Grounded-SAM (Liu et al., 2023; Kirillov et al., 2023).
Source Content
Region Prompt
2D Mask M,
Target Content
Multiple boxes
"An astronaut wearing box gloves."
Custom Mesh
"An astronaut holding a red rifle and riding an brown wolf."
Segmentaion
"helmet"
"An astronaut wearing a golden helmet." "
Figure 11: Progressive3D supports various definition forms of regions since the corresponding 2D masks of each view can be obtained.
A.4 ATTRIBUTE EDITING
We emphasize that Progressive3D supports both modifying attributes of existing objects and creat- ing additional objects with attributes not mentioned in source prompts from scratch in user-selected regions, and provide attribute editing results in Fig. 12. Noticing that creating additional objects with attributes not mentioned in source prompts from scratch is more difficult than editing the at- tributes of existing objects. Therefore, attribute editing costs significantly less time than additional object generation.
A.5 WHY ADOPTING 2D CONSTRAINTS
Compared to directly maintaining 3D content beyond editable regions unchanged on 3D represen- tations, adopt 2D constraints for achieving such goal from the perspective of generalization.
We notice that current text-to-3D methods are developed based on various neural 3D representa- tions, thus most 3D editing methods propose careful designs for specific representation and are unusable on other representations. For instance, DreamEditor (Zhuang et al., 2023) distills the orig- inal NeRF into mesh-based radiance field to localize the editable regions, and FocalDreamer (Li
14

---

Published as a conference paper at ICLR 2024
orange cat wearing yellow suit and blue snearkers and red chef's 's hat."
"A silver tea pot and origami golden table.
"An orange cat wearing a yellow and green snearkers and chef's hat."
"A ceramic tea pot and an origami box a golden table." *
"An orange wearing cyan suit and green snearkers and chef's
silver tea pot and origami box wooden table."
Figure 12: Prgressive3D supports attribute editing on existing generated objects.
et al., 2023) proposes multiple regularized losses specifically designed for DMTet to maintain the undesired regions unchanged. In addition, their specific designs also limit the available definition forms of editable regions.
However, we underline that the optimization core of most text-to-3D is the SDS loss which super- vised rendered views 2D way, and different neural 3D representations can be rendered 2D projected color, opacity, and depth maps through volume rendering or rasterization. Therefore, our proposed 2D region-related constraints can effectively bridge different representations and various user-provides definition forms, and the strength weights of our region region-related constraints are easy to adjust since our constraints and SDS loss are all imposed on 2D views.
A.6 LIMITATIONS
Our Progressive3D efficiently promotes the generation capacity for current text-to-3d methods when facing complex prompts. However, Progressive3D still faces several challenges.
Firstly, Progressive3D decomposes a difficult generation into series of editing processes, which leads to multiplying time costs and more human intervention. A potential future direction is further introducing layout generation into the text-to-3d area, e.g., creating 3D content with complex se- mantics in one generation by inputting global prompt, a string of pre-defined 3D regions, and their corresponding local prompts. Whereas 3D layout generation intuitive suffers more difficulties and requires further exploration.
Another problem is that the creation quality of Progressive3D highly determined by the generative capacity of the base method. We believe our framework can achieve better results when stronger 2D text-to- text-to-image diffusion models and neural 3D representations are available, and we leave the adap- tion between possible improved text-to-3D creation methods and Progressive3D in future works.
B EXPERIMENTS SETTINGS
B. CSP-100
CSP-100 can be divided into four sub-categories prompts including color (e.g. red, green, blue), shape (e.g. round, square, hexagonal), material (e.g. golden, wooden, origami) and composition according to the attribute types appeared in prompts, as shown in Fig. 13. Each prompt in the color/shapemmateriall categories describes two objects binding with color/shape/mmtterilll attributes, and each prompt in the composition category describes at least three interacted objects with corre- sponding different attributes. We provide the detailed prompt list in both Tab. and Tab. 5.
15

---

Published as a conference paper at ICLR 2024
DreamTing TeamTinee Time
Progr Progresssive30
Dream Time
wouden dehing prigani sport car."
tonk and red flyeng fllg."
red end stor pior-stapof tre.."
rftee and hockig ergams metancyclee
Kewajatal cabie."
(o) Color
(b) Shope
(c) Material
(d) Composition
Figure 13: Prompts in CSP- 100 can be divided into four categories including Color, Shape, Material, and Composition according to appeared attributes.
B.2 METRICS
The CLIP-based metrics utilized by current text-to-3D methods (Poole et al., 2022; Tsalicoglou al., 2023; Li al., 2023) calculates the cosine similarity between text and image features ex- tracted by CLIP (Radford et al., 2021) However, recent works (Huang et al., 2023a; Lu et al., 2023) demonstrate that CLIP-based metrics can only measure the coarse text-image similarity but fail to measure the fine- ine-grained correspondences among multiple objects and their binding attributes. Therefore, we adopt two fine-gratd peee-gaaied text-to-image evaluation metrics including BLIP-VQA and mGPT-CoT proposed by (Huang al., 2023a) to show the effectiveness of Progressive3ID. We pro- vide comparisons Fig. 14 to demonstrate that the CLIP metric fails to measure the fine-grained correspondences while BLIP-VQA performs well, and we report the quantitative comparison of Dream based methods on CLIP metric over CSP-100 in Tab. 3.
Dream Time
MVDream
Table 3: Quantitative comparison CLIP over CSP-100.
Method
CLIP 1
argaeh eetoriglle wearng hig.ooooo chefchi
No h Lne and ridlng Serracoto wol.""
or
DreamTime
0.289
RLP-1Q4 CLP
0,340
0,347
+ CEBM
0.275
0.062
0.556
0.074
0.626
+ A&E
0.281
+ Progressive3D
0.292
Figure 14: Quantitative comparison for metrics including CLIP and BLIP-VQA on DreamTime and MVDream.
BLIP-VQA is proposed based on the visual question answering (VQA) ability of BLIP (Li et al. 2022). Concretely, BLIP-VQA decomposes complex prompt into several separate questions and takes the probability of answering "yes" as the score for a question. The final score of a specific prompt is the product of the probability of answering "yes" for corresponding questions. For stance, the complex prompt is "An astronaut holding red rifle.", the final score of BLIP-VQA is the product of the probability for questions including "An astronaut?" and "A red rifle?"
Since multimodal large language models such as MiniGPT-4 (Zhu et al., 2023) show impressive text-image understanding capacity, MiniGPT4 with Chain-of- ofhhuuhhth (mGPT-CoT) is proposed to leverage such cross-modal understanding performance to evaluate the fine-grained semantic sim- ilarity between query text and image. Specifically, we ask two questions sequence including "Describe the image." and "Predict the image-text alignment score.", and the multimodal LLM is required to output the evaluation mGPT-CoT score with detailed Chain-of-Ti of-ohoughtt prompts. In practice, we adopt MiniGPT4 fine-tuned from Vicuna 7B (Chiang et al., 2023) the LLM.
What's 's more, we define the preference criterion of human feedback as follows: Users are requested to judge whether the 3D creations generated by DreamTime Progressive3D consistent with the given prompts. one 3D content is acceptable the semantic aspect while the other is not, the corresponding acceptable method is considered superior. On the contrary, if both 3D creations
16

---

Published as a conference paper at ICLR 2024
considered to satisfy the description in the given prompt, users asked to prefer the 3D content with higher quality.
B.3 IMPLEMENT DETAILS
Our Progressive3D is implemented based on the Threestudio (Guo et al., 2023) project since Dream- Time (Huang al., 2023b) and TextMesh (Tsalicoglou et al., 2023) have not yet released their source code and the official Fantasia3D (Chen et al., 2023) code project, and all experiments are conducted NVIDIA A100 GPUs. We underline that the implementation in ThreeStudio (Guo et al., 2023) and details could be different from their papers, especially ThreeStudio utilizes Deep- Floyd IF (DeepFloyd- Team, 2023) the text-to-image diffusion model for more stable and higher quality in the generation, while Fantasia3D adopts Stable Diffusion (Rombach et al., 2022) as the 2D prior. The number of iterations N and batch size for Dream and TextMesh are and respectively. The training settings of Fantasia3D are consistent with officially provided train- ing configurations, e.g., N is 3000 and 2000 for geometry and appearance modeling stages, and batch size is set to 12. We leverage Adam optimizer for progressive optimization and the learning rate consistent with base methods. Therefore, one local editing step costs a similar time one generation of base methods from scratch. In most scenarios, the filter threshold is set to 0.5, the strength factor X, iteration threshold K Cconeist are set to 0.5 and 4, and the suppression weight W in overlapped semantic component suppression technique is set to 4.
10000
C QUALITATIVE RESULTS
C.I CONTENT CONSTRAINT WITH BACKGROUND
We divide the Loansist into content term and an empty term to avoid mistakenly treating back- grounds as a part foreground objects. give visual comparison restricting the backgrounds and foregrounds as an entirety in Fig. 15, and the Loonsist can be formulated as:
bcansist = Σ -((0))))))))mm)))))
PER
The visual results demonstrate that mistakenly treating backgrounds as part of foreground objects leads to significant floating.
Source Content
With Lconsist
With Cconsist
Figure 15: Mistakenly restricting background content foreground leads significant floating in editing result.
C.2 MORE PROGRESSIVE EDITING PROCESS
We here provide more progressive editing results for correctly creating 3D content for prompts with complex semantics in Fig. 16. More qualitative results with various complex prompts further demonstrate the creation precision and diversity of our Progressive3D.
17

---

Published as a conference paper at ICLR 2024
D PRELIMINARY
Neural Radiance Field (NeRF) (Mildenhall et al., 2020) uses a multi-layer perception (MLP) to implicitly represent the 3D scene continuous volumetrie radiance field. Specifically, MLP maps a spatial coordinate and a view direction to new-inddeenddent density and view-dependent color c. Given the camera ray rkk) + kd with camera position o, view direction d and depth k E [yn, the projected color of r r(k) is rendered by sampling N points along the ray:
N
C(r) Σ 14 cxppp)xpppp
(10)
1=1
where Ps and C2 denote the density and color of i-th sampled point, 12 exp( Σ Pzój) indi- cates the accumulated transmittance along the ray, and the distance between adjacent points.
Diffusion Model SSohl-Dickstein et al., 2015; Ho et al., 2020) is a generative model which defines a forward process to slowly add random noises clean data Ey ~ p(x)) and reverse process to generate desired results from random noises e ~ /0 I) within T time-steps:
q (1- (1-aa)))
(11)
(0 [24) = N (x)
(12)
where Or and are calculated by pre-defined scale factor 3r, and (0; 1) is calculated by and the noise Eg (3t, 1) predicted by a neural network, which optimized with prediction loss:
L= Eg.,,t [w[t)]] (()()))((( <|12
(13)
where w(t) is weighting function that depends the time-step t. Recently, text-to-image dif- fusion models achieve impressive success in text-guided image generation by learning co (1), y, t) conditioned by the text prompt y. Furthermore, classifier-free guidance (CFG) (Ho & Salimans, 2022) is widely leveraged to improve the quality of results via a guidance scale parameter we
én (It, y,
(1 + w e) (x,, y, t) wz)
(14)
Score Distillation Sampling (SDS) is proposed by (Poole et al., 2022) to create 3D contents from given text prompts by distilling 2D images prior from pre-trained diffusion model differen- tiable 3D representation. Concretely, the image æ = g(Φ) rendered by differentiable generator g and a representation parameterized by & and the gradient is calculated
V LoDS ( x) = Ee,, ((() (x,, y. <) dx
(15)
OΦp
18

---

Published as a conference paper at ICLR 2024
"An orange cot."
"Ad orunge cat wearing yellow surt."
"An mearing yellow suit and green onearters.""
pirk vase."
"A yellow tulp pirk vaze.
red rose and yellow tullp pink vase.
orange firemin
and
"An orange aearity a green
oronge cot wearing yellow cyan boots!"
peany pirk vodk."
tullp and yellow tulip pirk
"A stor shoped tryy."
"A hoxepoonal cop e scttr--haped tna.."
"A green coctus hesiggoaa cap star-shoped tray."
"A golden table."
"A coromic tea pot on golden table.
"A cenamic teo pot and a lego car golden table
peptert stor-shoped
hexagonal star shoped tray."
gelden table.
candboard box table."
triongular shaped tray."
white in hessgonul cop star-shaped tray."
origemi goldon
ceratic pot shoed
morkey origami sport
lege mot weoning riding golden motencycle."
driwng orgga! sport
matercycle.
golden matersycle." *
"A logo man dhiwing on enigami sport con"
metal monkey wearing golden crean and driving origatri sport
"A silven bunny riding golden motorcycle
lego mon wearing woodlst top hot riding a golden motersycle?
woodon dog driwing on origan! sport car.
"A inttl monkey secdden and driving origani sport car"
"A meoden dog mialing golden motorcycle"
"A lego mon weoning chaf's hot and rnding golden matercycle.
Figure 16: More progressive editing results created with our Progressive3D based on DreamTime.
19

---

Published as a conference paper at ICLR 2024
Table 4: Detailed prompt list CSP-100. (Part
Index
Prompt
Color (i.e., two interacted objects binding with different colors)
orange wearing yellow suit
2
orange cat wearing green shirt
an orange cal wearing a red fireman uniform
4
red rose in pink vase a blue peony in pink vase yellow tulip in pink vase a pair of red sneakers on a blue chair stack of green books on a blue chair a purple gift box on blue chair a red cake yellow tray
5
6
7
9
11
blue spoon in yellow tray a puir of green chopsticks in a yellow tray red desk
12
13
green
14
a pair of blue sncakers on a red desk
yellow tray on desk
Shape (i.e. two interacted objects binding with different shapes)
a round gift box on a hexagonal table
17
triangular cake a square tray on hexagonal table
hexagonal table
18
bexagonal cup round cabinet
20
a triangular sandwich on a round cahinet
a square bowl on round cabinet
22
hexagonal cup star-shaped tray
23
a triangular cake on star-shaped tray square pepper star-shaped tray
24
25
a model of round house with hexagonal roof model of round house with square roof
27
model of round house with spherical roof
Material (i.e.. two interacted objects binding with different materials)
28
lego man riding golden motorcycle
a silver bunny riding golden motorcycle
30
wouden dug riding golden motoreycle a lego man driving un origami sport car metal monkey driving origami sport car a wooden dog driving origami sport car
33
a ceramic tea pot on a golden table silver vase golden table
35
an origami box on a golden table model of silver house with golden roof a model of silver house with a wooden roof model of silver house with bronze roof
37
38
40
lego tank with golden gun a lego tank with silver gun
42
lego tank with wooden gun
Composition (lee, more than two interacted objects binding with different attributes)
43
orange wearing yellow suit and green sneakers
44
an orange cat wearing a yellow suit and red pumps
an orange wearing a yellow
eyan boots
46
orange cat wearing yellow suit and green sneakers and cyan top hat an orange cat wearing a yellow suit and green sneakers and pink cap orange wearing yellow suit and green sneakers and red chef's hat a blue peony and yellow tulip in pink vase red and yellow tulip pink
48
49
50
20

---

Published as a conference paper at ICLR 2024
Table 5: Detailed prompt list of C00- (Part 2)
Index
Prompt
51
white tulip and yellow tulip in pink vase
52
purple rose and a red rose and yellow tulip in pink vase
53
blue peony and red rose and yellow tulip in a pink vase
54
a white tulip and a red rose and a yellow tulip in a pink vase
55
golden cat a stack of green books a blue chair
56
wooden bird on stack green books on blue chair
57
a lego car on a stack of green books on blue chair
58
blue candle a green spoon on a red cake in a yellow tray
red cake in a yellow tray
59
60
pair of cyan chopsticks eyan cake on yellow tray on a red desk
red cake in a yellow tray
61
62
a ceramic tea pot on yellow tray on a red desk
63
green apple a square cake on square tray on a hexagonal table
yellow tray a red desk
64
65
a round cake on square tray on a hexagonal table
66
triangular sandwich a green apple in a hexagonal cup on a round cabinet pink peach in a hexagonal cup round cabinet a yellow pineapple
square tray a hexagonal table
67
68
69
hexagonal cup on a round cabinet
70
a red rose in a hexagonal cup on a star-shaped tray
71
white lily in hexagonal cup on star-shaped tray a green cactus in hexagonal cup on a star-shaped tray
72
73
lego wearing a chef's ' hat and riding golden motorcycle
74
lego man wearing a wooden top hat and riding golden motorcycle a lego man wearing a silver crown and riding golden motorcycle
75
76
metal monkey wearing golden crown and driving origami sport car metal monkey wearing chef's ' hat and driving an origami sport car a metal monkey wearing wooden top hat and driving an origami sport car ceramic tea pot and lego car on golden table a ceramic tea pot and an origami hox on a golden table ceramic tea pot and a cardboard box on a golden table ceramic tea pot and pair of wooden shoes on golden table a ceramic tea pot and an origami box and a green apple on a golden table ceramic tea pot and origami hox and a yellow tray on golden table a ceramic tea pot and an origami hox and a stack of blue hooks on a golden table model of a silver house with golden roof beside an origami coconut tree model of silver house with golden roof heside a wooden
77
78
79
80
81
82
83
84
85
86
87
88
a model of a silver house with golden roof beside a lego man
89
lego tank with golden gun and red flying flag
90
a lego tank with golden gun and blue flying flag
91
a lego tank with a golden gun and a yellow flying flag
92
model of a round house with a spherical roof hexagonal park
93
a model a round house with a spherical roof on square park
94
model of a round house with a spherical roof on an oval park
95
an astronaut holding red rifle and riding green origami motorcycle
96
an astronaut holding a red rifle and riding a cyan scooter
97
an astronaut holding red rifle and riding golden motorcycle
98
an astronaut holding a red rifle and riding a green origami motorcycle and wearing a blue top hat
99
an astronaut holding a red rifle and riding green origami motorcycle and wearing a cyan chef's hat an astronaut holding red rifle and riding green origami motorcycle and wearing pink cowboy hat
100
21


Reviews:
[{"review_id": "1TEOUL39HM", "reviewer": "Reviewer_S6PX", "summary": "This paper proposes a general framework named Progressive3D for correctly generating 3D content when the given prompt is complex in semantics. Progressive3D decomposes the difficult creation process into a series of local editing steps and progressively generates the aiming object with binding attributes. Experiments conducted on complex prompts in CSP-100 demonstrate that the proposed Progressive3D can create 3D content consistent with complex prompts. The motivation of generate correct 3D content for a complex prompt in semantics is good, but the solution is inflexible. Because users need to provide 3D bounding box prompt for each prompt, which is inflexible and difficult to define.", "questions": "- How long does it take for complex prompts?\n- Given a complex prompt, how to decompose the complex text, automatically or manually? And How many steps are required?\n- How can we provide the 3D bounding box prompt, I think it is difficult. In my opinion, I think the 3D bounding box limits the application of the proposed method.\n- Can you report the CLIP-Score?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Progressive3D can create precise 3D content prompted with complex semantics by decomposing a difficult generation process into a series of local editing steps.\n- Progressive3D could be incorporated into various text-to-3D methods driven by different 3D neural representations.", "weaknesses": "- The quality of generated 3D objects is poor.\n- The proposed method requires the 3D bounding box as the input, which is inflexible.\n- It is difficult for Progressive3D to change the attribute of the generated 3D objects, such as changing red to blue or metal to wood. If we want to edit the attribute, we might need to train the model case by case.\n- Only one dataset was used in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhYSQ6t4Up", "reviewer": "Reviewer_dBWG", "summary": "This paper tackles the challenge of aligning complex prompts with generated 3D assets by employing a progressive framework that decomposes the generation process into multiple local editing tasks. By doing so, the authors achieve the generation of semantically precise 3D assets. Additionally, the paper introduces a novel dataset designed to evaluate the outcomes of compositional Text-to-3D generation.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, with a clear structure and easily understandable language.\n\n2. The results demonstrate effective composition of objects with different attributes and relationships.\n\n3. The proposed dataset and evaluation metrics explore Text-to-3D benchmark in terms of composition and relationships, providing valuable insights for the community.", "weaknesses": "1. The main concern is the heavy reliance on human involvement throughout the pipeline. Users are required to provide prompt divisions and bounding boxes, and the process seems user-unfriendly, as users have to wait for the previous generation to finish before providing the next bounding box prompt.\n\n2. The paper mentions that current T2I diffusion models often struggle with complex prompts. However, there are existing methods [1,2,3] that address this problem. It would be beneficial to discuss why the authors did not directly utilize these methods, as it seems more straightforward and would save human labor. This discussion is currently missing from the paper.\n\n3. Figures 7 and 10 show inconsistencies with the claim that undesired regions remain unchanged. The leg of the astronaut turns green, and the foot is missing after adding the prompt \"and riding a red motorcycle.\"\n\n4. It would be more convincing if the paper showcased additional results (quantitative and qualitative) based on Fantasia3D. Given the low resolution of image space supervision in DreamTime and DreamFusion, the generated 3D assets appear blurry. Demonstrating the significant improvements offered by this method in more sophisticated Text-to-3D approaches would reinforce the paper's claims.\n\n5. Including an ablation study on the last term in the consistency loss (the one that imposes the empty region to be blank) would strengthen the paper's arguments.\n\n[1] Compositional visual generation with composable diffusion models, ECCV, 2022\n\n[2] Training-Free Structured Diffusion Guidance for Compositional Text-toImage Synthesis, ICLR, 2023\n\n[3] Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, ICLR, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBtuNSEd1o", "reviewer": "Reviewer_GY6s", "summary": "The paper introduces a method for progressively generating intricate 3D content. Each generation phase progressively generates local content using progressive semantic prompts. To maintain the consistency of the progressive generation, a consist loss is employed. Additionally, an initialization loss is utilized to swiftly generate content in selected areas. Furthermore, the proposed OVERLAPPED SEMANTIC COMPONENT SUPPRESSION ensures that each progressive generation optimizes towards additional semantic prompts.", "questions": "1.Is the progressive3D approach the only way to generate intricate 3D content? I want to see more comparative experiments to achieve a fairer comparison. For instance, one might first employ a complex semantic prompt to produce consistent images, then use these images to generate the corresponding 3D content. Alternatively, one could generate each object individually and subsequently merge them based on their bounding boxes, then the aggregated 3D content could be optimized to achieve a harmonious result. I am interested in understanding how to demonstrate that the progressive approach is a crucial and effective method for generating complex 3D content.\n\n2. How to resolve conflicts between subsequent generated results and earlier ones, such as an astronaut sitting on a red chair, when the first step generates a standing astronaut? Can the model optimize the transition from a standing astronaut to a sitting one?\n\n3.In the context of complex semantic prompts, does the order of prompt input have an impact on the generated results, and is it necessary to engage in simple semantic prompt planning?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The progressive approach is indeed a straightforward and effective method for generating complex 3D content. This ensures that each local element receives accurate optimization guidance.\n2.\"OVERLAPPED SEMANTIC COMPONENT SUPPRESSION\" can effectively optimize progressive 3D content in alignment with additional semantic prompts.\n3.The concept of \"Initial Loss\" contributes to achieving a stable and high-quality 3D content generation within 3D bounding box.", "weaknesses": "1. Further comparisons with other methods for achieving complex semantic prompt-driven 3D content generation are lacking.\n2. The overall pipeline is relatively straightforward and simplified. Independently generating each local content and optimizing it after the combining each local 3D content with 3D bounding boxes may yield improved results, particularly for the prompt like some object is on a tabletop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UZUZWuj2Pp", "reviewer": "Reviewer_EZDr", "summary": "This paper presents a locally progressive method for text-driven generation of semantically complex prompts. Existing methods cannot faithfully generate the complex prompts directly. Thus, this work takes an iterative approach: the complex prompt is broken up into segments and each segment is iteratively added to the 3D representation. However, directly optimizing for the new segment of the prompt each iteration leads to a muddled result where attributes bleed together. Thus, this work also conditions on a user specified edit region. This allows the method to only optimize a local region, thus preserving the rest of the 3D representation.", "questions": "- Since the user can only input the box center and the lengths of the box along each axis, it seems that the box will always be axis aligned. Is this not problematic for certain edits that do not line up well with the axes?\n- It would be helpful to clarify more how the semantic delta loss differs from (Armandpour et al.) [4]\n\nReferences:\n[4] Armandpour, Mohammadreza, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. \"Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond.\" arXiv preprint arXiv:2304.04968 (2023).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Generation with complex prompts is very challenging and existing methods struggle on this task, while this work excels.\n- The semantic delta loss is an interesting contribution that is important for editing.\n- Explicitly defines a region in which the edits can take place to ensure preservation of the existing model.\n- The paper compares to numerous existing approaches for text-to-image generation showing superior performance on complex prompts and gives a thorough ablation of the components of the method.\n- Clear presentation: the paper is well written and makes good use of experiments/figures to support its claims. Figure 2 is especially helpful for understanding this approach.", "weaknesses": "- The local region for each edit must be manually entered by the user. This slightly limits the intuitive, easy-to-use nature of this work as compared to most other purely text-driven approaches.\n- If I am understanding correctly (see question for more details), the edit region can only be defined as an axis-aligned bounding box. This seems like it could be problematic for edits that do not fit nicely into an axis-aligned box.\n- Limited comparisons to relevant existing work. There exist methods for focusing on different parts of the text prompt (Attend and Excite [1]) for 2D image generation and editing. These methods have been shown to address issues with attribute binding and “catastrophic neglect.” It would be helpful to see how a baseline performs using these approaches as the 2D model used for distillation. Additionally, DreamEditor [2] enables local editing on NeRFs using an explicit edit region. The region is inferred from the text description using attention maps from the diffusion model. A good baseline would be to use DreamEditor iteratively on each progressive edit as this still gives an explicit edit region, but does not require a user input bounding box.\n\nReferences:\n[1] Chefer, Hila, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. \"Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.\" ACM Transactions on Graphics (TOG) 42, no. 4 (2023): 1-10.\n[2] Zhuang, Jingyu, Chen Wang, Lingjie Liu, Liang Lin, and Guanbin Li. \"DreamEditor: Text-Driven 3D Scene Editing with Neural Fields.\" arXiv preprint arXiv:2306.13455 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)