TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields
Authors: Tianyu Huang
Yihan Zeng
Bowen Dong
Hang Xu
Songcen Xu
Rynson W. H. Lau
Wangmeng Zuo
Venue: ICLR 2024 poster (2024)
Keywords: 3D Open Vocabulary
Conditional Generation
Primary area: generative models

Abstract: Recent works learn 3D representation explicitly under text-3D guidance. However, limited text-3D data restricts the vocabulary scale and text control of generations. Generators may easily fall into a stereotype concept for certain text prompts, thus losing open-vocabulary generation ability. To tackle this issue, we introduce a conditional 3D generative model, namely TextField3D. Specifically, rather than using the text prompts as input directly, we suggest to inject dynamic noise into the latent space of given text prompts, i.e., Noisy Text Fields (NTFs). In this way, limited 3D data can be mapped to the appropriate range of textual latent space that is expanded by NTFs. To this end, an NTFGen module is proposed to model general text latent code in noisy fields. Meanwhile, an NTFBind module is proposed to align view-invariant image latent code to noisy fields, further supporting image-conditional 3D generation. To guide the conditional generation in both geometry and texture, multi-modal discrimination is constructed with a text-3D discriminator and a text-2.5D discriminator. Compared to previous methods, TextField3D includes three merits: 1) large vocabulary, 2) text consistency, and 3) low latency. Extensive experiments demonstrate that our method achieves a potential open-vocabulary 3D generation capability.

Published as a conference paper at ICLR 2024
TEXTFIELD3D: TOWARDS ENHANCING OPEN- VOCABULARY 3D GENERATION WITH NOISY TEXT
FIELDS
Tianyu Huang Yihan Zeng"
Bowen Dong' Rynson W. H. Lau" Wangmeng Zuo
Hang Xu² Songcen Xu²
'Harbin Institute of Technology "Huawei Noah's Ark Lab City University of Hong Kong
tyhuangestu. hit. edu. en, fzengy1han2, xu. hang, xusongcen]huawel chawee..com. cndongsky@gmail com, rynson www.city..uuu.............. edu hk, wmzuo@hit wwrzuu@hit.edu.... wew.tteedd.........
ABSTRACT
Generative models have shown remarkable progress in 3D aspect. Recent works learn 3D representation explicitly under text-3D guidance. However, limited text- 3D data restricts the vocabulary scale and text control of generations. Generators may easily fall into a stereotype concept for certain text prompts, thus losing open- vocabulary generation ability. To tackle this issue, we introduce conditional 3D generative model, namely prompts input directly, suggest to inject dynamic noise into the latent space of given text prompts, i.e., Noisy Text Fields (NTFs). In this way, limited 3D data be mapped the appropriate range of textual latent space that is expanded by NTFs. To this end, an NTFGen module is proposed to model general text latent code in noisy fields. Mcanwhile, an NTFBind module is proposed to align view- invariant image latent code to noisy fields, further supporting image-conditional 3D generation. To guide the conditional generation in both geometry and texture, multi-modal discrimination is constructed with text-3D discriminator and text- 2.5D discriminator. Compared to previous methods, TextField3D includes three merits: 1) large vocabulary, 2) text consistency, and 3) low latency. Extensive experiments demonstrate that our method achieves a potential open vocabuaayy 3D generation capability.
rather than using the text
INTRODUCTION
3D creative contents are in significantly increased demand for wide range of applications, including video games, virtual reality, and robotic simulation. However, manually creating 3D content is time-consuming process that requires high level of expertise. To achieve automatic generation,
previous works (Niemeyer Geiger 2021 Gu et al. 2021] CChn et al. 2022 Gao et al.] 2022) attempt several 3D representations like Voxel, NeRF, and SDD.. Nonetheless, objects from these methods are unconditionally generated in a single category, which can hardly be applied in practice.
With the success of text-to-image generative models (Ramesh et al. 2021
2022 Rombach al.
2022). text prompts become flexible variable to achieve open-vocabulary generative capability. Similar to 2D vision, text-to-3D generation has recently aroused great interest. Current research can
be generally separated into two strategies. One (Jain et al.) 2022] Mohammad Khalid et al.) 2022] Poole al. 2022 Lin al. 2022 optimizes differentiable 3D representations with vision-language (V-L) pre-trained knowledge, e.g., DreamFields (Jain et al. 2022) is optimized from knowledge of the V-L. pre-trained model CLIP (Radford al. [221]002)) and DreamFusion (Poole et al. 2022)
guided by diffusion model Imagen (Saharia et al. 2022). Since no 3D data are involved in training, these methods can easily fall into artifacts, facing serious 3D consistency problems. Meanwhile, the
cost of optimizing 3D representations is extremely high, in terms of both time and computational re- sources. The first strategy seems not to be the ultimate solution for 3D generation. The other (Nichol et al. 2022 Cheng al. 2023 Wei al. 2023 Sanghi et al. 2023 Jun & Nichol 2023) directly supervises 3D generators with paired text-31 3D data. These methods allow real-time generation but
extremely restricted by the scale of available 3D data. Nichol et al. (2022) and Jun & Nichol

---

Published as a conference paper at ICLR 2024
"A pumpkin"
"A designed dress"
"A traffic cone"
dumpater"
"A binhday cupcake"
"A vase with flowers"
Input Image
Output Result
(a) Text-to-3D
(b) Image-ta-3D
Figure 1: TextField3D is capable of text-to-3D and image-to-3D tasks. In particular, all the text prompts are from (Nichol et al. 2022 Jun & Nichol 2023). our method can generate diverse and reasonable results, exhibiting potential open-vocabulary capability.
(2023) collect millions text-3D pairs, which is still thousands of times smaller than the scale of V-L data. As such, the diversity of generated output and the complexity of text input incompati- ble with V-L optimized methods. Therefore, here is the question: With limited data, can we train a real-time 3D generator with the potential towards open-vocabulary content creation?
The common practice to address data scarcity is introducing V-L pre-trained knowledge to gen- erators. Specifically, 3D generative latent codes are initialized with pre-trained models. Cheng et al. (223) adopt BERT (Devlin et al., /018)/LIPP (Radford et al. (2021) encoders for text/image- condition tasks. (Sanghi al. (2023) propose train transformer module with CLIP image em- bedding and replace it with text embedding at inference. Wei et al. (2023) further attempt to aligm control. However, one key problem remains challenging: how to map limited 3D content to comprehensive V-L. pre-trained concepts. Some recent 3D representation learning works |Hunn et al. (2023): [iu al. (2023a) align genera- tive latent space based on limited categories and text prompts, resulting in close set understanding.
text prompts with rendered images to improve generative
To tackle this issue, we intend to expand the expression range of 3D latent space. We observe that 3D latent code can easily fall into stereotype concepts when trained in small data scale with clean data and templated prompts. For example, a simple prompt "a chair" can represent any type of chair, tall short, with without armrests. However, if supervised of chair, the generator may tend to produce this fixed shape only, losing diversity. Based on this observation, introduce conditional 3D generative model, dubbed TextField3D. TextField31 maps limited 3D data to dynamic fields V-L. concepts, which are named as Noisy Text Fields assume that there is dynamic range the latent mapping of text-to-3D, which can be formulated as an injected noise to the text embeddings. We thus propose an NTFGen module, where text latent code is formulated based on the field of noisy text embeddings. To support image conditional generation, we further propose an NTFBind module for binding image features NTFs and keeping the consistency of multi-view conditions. A view-invariant image latent code is derived for the image-to-3D task. Furthermore, multi-modal discrimination is constructed for the supervision of 3D generation, which we propose a text-3D discriminator to guide geometry generation and a text-2.5D discriminator to refine texture details.
specific type
(NTFs). Specifically,
Compared to previous methods, TextField3D simultaneously possesses three advantages: large vo- cabulary, text consistency, and low latency. Extensive experiments are conducted on various cate- gories and complicated text prompts, which exhibit the open- gpn-boccayllary potential of our model. We further employ several metrics to evaluate the generation quality and text consistency, demonstrating the effectiveness of our newly proposed modules.
Our contributions be summarised follows:
We introduce conditional 3D generative model TextField3D, in which limited 3D data is mapped to textual fields with dynamic noise, namely Noisy Text Fields (NTFs).
We propose NTFGen and NTFBind to manipulate general latent codes for conditional gen- eration, and multi-moda discriminator to guide the generation of geometry and texture.
Extensive experiments present an open- vecabulcyy potential our proposed method, in terms of large vocabulary, text consistency, and low latency.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Driven by the increasing demand for 3D creative contents, text-to-3D generation has witnessed rapid improvement recently. We can generally split current studies into two categories: V-L optimized methods and 3D supervised methods.
V-L Optimized Methods. With the success of V-L pre- sire-trained models, various works (Jain et al,,
2022
Mohammad Khalid et al. 2022 Poole et al. 2022 Lin et al., 2022 Chen et al. 2023 Wang
2023) leverage V-L pre-trained knowledge to optimize implicit functions of 3D representa- tions. Jain et al. (022)) supervise the generation of NeRF representation by the CLIP guidance of
et al.
given text prompts. Poole et al. (2022) introduce Score Distillation Sampling (SDS) to NeRF vari-
ant, adopting diffusion models optimize 3D renderings. Based on SDS, Waang al. (2023) further propose Variational Score Distillation (VSD), treating the 3D content of a given textual prompt as a random variable. Albeit approaching open-vocabulary generation, these methods highly depend on expensive and time-consuming optimization procedures and may easily produce 3D artifacts like multi-face and paper-thin outputs, which is not a practical solution for 3D generation.
3D supervised Methods. 3D supervised methods train generators with 3D data. Contrary to V-L optimized methods, they present an efficient way to generate solid 3D content. These works (Achlioptas et al.) 2018; Kosiorek et al. 2021; Chen & Wang 2022) are basically in an encoder-decoder architecture. For example. Kosiorek et al. (2021) proposes a variational autoen- coder, encoding rendered views of scene representations as a generative condition. However, gener- ations are restricted by the scarcity of 3D data. To allow various generative conditions, some recent works (Sanghi et al.] 2022 Mittal et al. 2022 Fu al. 2022 Cheng et al. 2023 Wei et al. 2023 Sanghi et al.] (023) attempt to combine pre-trained knowledge to the latent encoding, while the im- provement is limited. Nichol al. (2022) and Jun & Nichol (2023) collect several millions 3D data for training large-vocabulary generators, which is still far from the data scale of 2D diffusion.
In this work, aim to enhance the mapping of limited 3D data and comprehensive V-L pre-trained knowledge, presenting an efficient way to approach open-vocabulary 3D generation.
PRELIMINARIES
3.1 3D GENERATIVE MODEL
Regardless of those V-L optimized methods, a 3D generative model takes latent codes to the gen- erator and supervises corresponding generated contents with 3D data. GET3D (Gao et al. 2022) proposes a 3D generative adversarial network (GAN) that embeds a random vector to the latent space of geometry and texture. MeshDiffusion (Liu al. 2023b) learns to denoise simple initial distribution into the target 3D objects with a diffusion model. SDFusion (Cheng et al., (023) lever- ages vector quantised-variationall autoencoder (VQ-VAF) compress 31) shapes into latent space for further generations. Not losing generality, we can split 3D generation training into three compo- nents, i.e., a 3D latent code w, generator G in differentiable 3D representations, and a 3D-related supervision V3DG(w).
3.2 CONDITIONAL 3D GENERATION
To allow condition inputs for the generation, the common strategy is embedding the generative latent code with corresponding inputs. In 2D text-to-image synthesis (Ramesh et al. 202T 2022] Rombach et al., 2022)). generators are trained to minimize the objective V2DG(w) with large-scalo
V-L data, achieving an open-vocabulary performance. Unfortunately, reaching such a tremendous scale for 3D datasets is challenging. [Saharia et al. (2022) suggest that pre-trained text or V-L. models may encode meaningful representations relevant for the generation task. They initialize latent codes with freezed BERT (Devlin et al. 2018). T5 (Raffel et al. 2020), and CLIP (Radford et al. 2021) which may alleviate data scarcity to some extent. Therefore, some text-to-3D methods (Sanghi et al. 2022, Mittal et al. 2022 Fu et al. 2022 Cheng et al. 2023 Wei et al. 2023 Sanghi et al. 2023 utilize pre-trained knowledge to embed latent vectors w. Take textual imput as example:
The conditional latent code w, can be revised as w, = f(2, t), where t is pre-trained text features encoded from i, and f is mapping network to embed text features with noise distributions
3

---

Published as a conference paper at ICLR 2024
Code
XTPG2e Nhahit
"4 green tegghor
Eneoder
And
Text-3D
3B Coneeiar
lqpul
Diseriminator
Nats Tec Fael
NTRRle Modal:
Hegpr Love
Frage
Generator
Text-2.51)
Discriminator
Figure 2: The overall framework of TextField3D. The NTFGen module and the NTFBind module are proposed to encode conditional latent codes noisy text fields, which are fed to 3D generator. Multi-modal discrimination is then exploited to supervise the generation process, including a text- 3D discriminator and a text-2.5D discriminator.
METHODOLOGY
Albeit introducing open-vocabulary knowledge from V-L pre-trained models guidance, previ- ous 3D generative models (Cheng et al. al,] [023 Wei et al. 2023 Sanghi et al. (003) still struggle to
exhibit promising open-vocabulary capability. In this work, we present TcxtField3D
feasible
lution to approach open-vocahulary generation. To this end, we introduce Noisy Text Fields (NTFs) to boost the latent mapping between V-L concepts and 3ID representations. Meanwhile, propose multi-modal discrimination to enhance the supervision of 3D generation. The overall framework is We provide detailed explanations in the following.
illustrated in Figure
4.1 LEARNING GENERAL. LATENT SPACE FOR 3D REPRESENTATION
Current available 3D data is limited to small range of categories, which is unable to cover compre- hensive concepts of V-L pre-trained knowledge. As shown in Figure 3 previous 3D latent code trained with clean data and templated prompts, falling into a stereotype concept of certain prompts in training data. Therefore, it is a challenging task to align 3D latent space with the open-vocabular, embedding space of V-L pre-trained models.
In order to learn a general latent code from limited 3D data, we have to map 3D data to an equivalent expression range of pre-trained concepts. There are previous studies (Nukrai al., 2022 Gu
et al. 2022) addressing the issue of imbalanced data between multiple modalities. They attempt to N(,,, c) is attached to the embedded
solve V-L tasks with text-only training. An injected noise text, so that image features can be included in the textual space. In our case, text prompts and 3D objects have one-to-many correspondence, e.g., "a chair" represent all kinds of chairs in Figure 3 Accordingly, we can introduce a similar noise to the corresponding 3D latent space.
Noisy Text Latent Code. Considering the scarcity of available text-3D data, we attach a textual range to 3D latent space following (Nukrai et al. 2022 Gu et al. 2022). However, we observe that the one-to-many situation is more obvious in 3D. Difterent from 2D realistic images, rendered images of 3D data present single 3D object on a clean background. The diversity can be easily constrained if a strict text prompt is given like tall wooden chair without armrests". Therefore, propose dynamic noise injection to adjust the range of text embeddings t:
F. == t t N(), where a = Stxx(t)), E ((1,
where Lte is non-linear network. 6 and 62 are hyperparameters, which are set as 0.0002 and 0.016. t is textual field injected with dynamic noise, which is regarded the Noisy Text Field (NTF) of t. Latent code can then be calculated w _(z, i) in NTFs, while we observe that the mapping network f weakens the generative control of t.. In text-to-image synthesis, Härkönen al. (202)) and [Sauer et al. (2023) also notice the tendency that the input random vector can dominate text embeddings
Following them, we propose late concatenation, bypassing the mapping networks for text embeddings f.. We thus have the noisy text latent code W; [w, t).
4

---

Published as a conference paper at ICLR 2024
Nute
Novel
CAA chai
11A STAxeeede choer
"A deilddii"
G dotrom snece
Ft aal axeder
duin untee smrres
Figure 3: The motivation of Noisy Text Field. Previous works may easily form stereotype concepts of training data but have no idea about outoof-distrbuution prompts. Differently, the Noisy Text Field can model the semantic correspondences among cases.
To optimize the alignment of latent code and correspondingly generated contents, the common prac- tice is minimizing the cosine distance between text features and image features,
exp(t [maaa(
(T) log
exp(t (fees /T) + Eizi exp(t (fri((9)1177))
where fuis is pre-trained visual encoder. The image I is rendered by a renderer R at the viewpoint e, which is defined as 1 R(G(W,,, However, Lnce (I) is a generation-based objective to match the generated content with the ground-truth text prompt. We further propose condition- based objective Ence (f), enhancing the alignment of latent code in NTFs with ground-truth rendered images Í. The final objective for generating noisy textual fields is Lgen Lince (f) + Lnce (f). The module is thus named as NTFGen.
View-Invariant Image Latent Code. To allow image conditional 3D generation, previous works (Cheng et al. 2023 Gupta et al. 2023) replace text embeddings t with visual embeddings v as generative conditions. However, we argue that image conditions are not equivalent text conditions in the 3D domain, as images provide additional viewpoint information. In order that generated ob- jects can keep consistent with image conditions at different views, we learn view-invariant image latent code V. = (Siss ( f.),,),, c), where Sriew is a non-linear network conditioned with camera poses. Given an input latent code Ve we randomly select its positive latent code Vog and negative latent code v2z The view-invariant objective Lview expects to bind feature spaces with the same text prompt, which is formulated Lview
| - - V/ -/ Mcanwhile, we bind the corresponding generations 1 to noisy text fields t by an NTF binding loss Laef (1c)
exp(t [fview (T)),
Lns (1 log exp(t. [Seeew (T)) e) /T) + Eizt exp(t [Suvew (Sois ())r)) The final binding objective is Liind Lview + Liess and we regard the module as NTFBind.
(3)
4.2 TEXTURED MESH GENERATION WITH MULTI- DISCRIMINATION
After designing general latent code for 3D generation with injected noise, we then discuss the 3D generative model we used in TextField3D. Previous textured mesh generative models (Chan et al.]
2022 Gao et al. 2022 Gupta et al.] 2023) have shown remarkable progress in terms of generation quality. In particular, GET3D (Gao et al. 2022) attaches a texture field (Oechsle et al.) 2019) to a differentiable 3D representation DMTer (Shen al. 202T), efficiently producing high-quality tex- tured meshes. Following it, we feed the noisy texture fields to formulate a conditional generator toward voca-vocy 3D content. Furthermore, troduce multi-modal discrimination and build a generative adversarial training framework. In the following, we illustrate each component in this framework.
better guide the generation process, we in-
Textured Mesh Generator. Given two random vectors and 22, GET3D adopts two non-linear mapping networks fgeo and ftez for intermediate latent vectors W1 = fgeo (21) and w2 ftez (22), which are then used for the generation of 3D shapes and texture. By embedding noisy textual fields t and bypassing mapping networks fgeo and ftez, the generation process is formulated as G((w,, E) [w2.()) t)).
Te5t-2.5D Discriminator. The combination of rendered images and corresponding camera poses is widely used in GAN-based 3DD supervision (Gu et al. 2021 Chan et al. 2022 Gao et al. 2022
5

---

Published as a conference paper at ICLR 2024
Rendered images can provide detailed visual information that is beneficial for texture generation. Since 3D geometry is implicitly presented by camera poses, we regard such discrimination as 2.5D supervision. To further supervise the text consistency, we concatenate the camera pose e with t as a new discriminative condition. The text-2.5D discriminative objective is formulated as,
C(D2, G) = ELer (Dz (c, i)) + teriite (o(-Dx(fa (c. ()) + (D)(
(4)
where (Ix) is defined as g(x) = -log(1 + exp(x)). is the ground-truth rendered image set. À is hyperparameter. is related to either RGB image or silhouette mask. Following GET3D, we apply the supervision of both RGB and mask.
Text-3D Discriminator. Although 2.5D discrimination implies 3D spatial information, it is diffi- cult to represent geometric details of objects such normal, smoothness, and so on. Some recent works (Liu et al., 2023b; Cheng et al., [023 Gupta et al. (023)) have attempted learn the shape prior directly from 3D data. Similarly, we exploit sampled point clouds to discriminate the genera- tive quality. Given a generated mesh object, we uniformly sample point clouds p over its surface, A discriminator architecture similar to 2.5D is then adopted, while adjust its mapping network to PointNet Qi et al. 2017). We have the text-3D discriminative objective L(Dpe G).
L(D G) = Erer g g(Dpolp', (.p')))))))))))) 1)) + EE (a(-Dpe ()), + A [DDpc
(5)
where p' and øt are the generated and the ground-truth point clouds with text prompt respectively. P is the ground-truth point cloud set.
4.3 OVERALL TRAINING OBJECTIVE
Finally, we illustrate the overall training objective L of TextField3D, which is defined as follows:
L = L(D smg. G) + L(D G) + Ape Cip G) + Xgen Cyen:
(6)
where the loss weight parameters Ape = 0.01 and Agen = 2. The NTFBind module is additionally trained by the binding objective Ltind- During its training, we freeze noisy texture fields f.
EXPERIMENTS
5.1 DATASET
We train TextField3D with a large-scale 3D dataset Objaverse (Deitke et al. 2022), which collects over 800k 3D objects from various categories. We select around 175k objects among them, which are qualified for the training. To prepare training data, we render RGBA images of 3D objects from random camera viewpoints with blender engines (Community) 2018) and annotate textual descrip- tions to them with the latest image captioning models (Li et al. 2023 For ablation studies, a clean 3D dataset ShapeNet (Chang et al., 2015) is used to evaluate the generative quality. ShapeNet con- tains over 50k clean 3D objects from 55 categories. We can accurately divide the training and test sets based on categories, which is the reason why we choose ShapeNet for the ablation studies. We filter out objects without texture, leaving 32,863 3D textured shapes. The filtered data are split into training, validation, and testing sets in a ratio of 7:1:2, We additionally annotate ShapeNet with MiniGPT-4 (Zhu et al. 2023). Please refer to Appendix B for details of data preparation.
5.2 IMPLEMENTATION DETAILS
For Objaverse, we train TextField3D with the full set and evaluate it with open-vocabulary text prompts. For ShapeNet, we select the best-performing models on the validation set for the evalua- tion. The total training time is around days and I day with 8 V100 GPUs, respectively. Following GET3D, we use a StyleGAN (Karras et al. 2019) discriminator architecture. We use Adam (Kingmal & Ba 2014) optimizer and initialize the learning rate to 0.002. The training batch size is 64. We use CLIP's s VTT-B/33 to embed text/image inputs and freeze them on training. We sample 8.192 points for each mesh object. And the image resolution 512 512 for both rendering and generation.

---

Published as a conference paper at ICLR 2024
5.3 EXPERIMENTAL RESULTS
We train TextField3D on the Objaverse dataset, which is captioned with BLIP-2 (Li et al. Four state-of-the-ar methods are then selected for comparison: DreamFields (Jain et al. DreamFusion (Poole et al.) 20022) Point-E (Nichol et al. (022). and Shap-E E (Jun & Nichol Quantitative and qualitative results are presented in the following.
2023
2022
2023
Quantitative Results. To verify open-vocabulary generation ability, Jain et al. (022) propose evaluation set, collecting 153 captions from the Common Objects in Context (COCO) dataset (Lin) al., 2014). We follow the experiment setting: for each prompt, corresponding generated object, with a fixed camera pose at a 45 angle of elevation and a 30 angle of rotation. Table reporrts reports the retrieval precision with CLIPP (CLIP R-Precision). Our 1-shot results are better than Point-E and Shap-E, even though TextField3D is trained in a much smaller data scale. Moreover, since TextField3D generates 3D objects much faster than other methods, we try eight more times for each prompt and select the most similar image out of 9 (according to the cosine similarity with CLIP text features). The results (TextField3D, 9-shot) can reach higher level, almost close SDS methods. These results demonstrate that TextField3D is more generalized than previous supervised methods (Point-E and Shap-E) with much fewer training data and achieves a comparable open- vocabullyy capability with V-L optimized methods by much faster inference.
render an image from the
Qualitative Results We select seven representative text prompts for the text-to-3D generation task, including three single nouns, two compound nouns, and two adjectival nouns. As shown in Figure A the generation quality DreamFields and Point-E is relatively low. DreamFusion has several paper-thin cases, i.e., rat, chess piece, and brick chimney, knowledge rather than 3D shape prior. The view consistency is another problem for DreamFusion e.g., the generated teapot has multiple spouts. Shap-E is a 3D-optimized method that is similar to our TextField3D. However, it fails to generate a common rat, which raises questions about its open- vocabulary capability. Moreover, Shap-E performs poorly text prompts with simple attribute binding. The chimney does not have brick texture, and the teapot is not uniformly filled with green color. The results of our model TextField3D are basically consistent with text descriptions.
it is optimized V-L. pre- pre-trained
We also conduct image-to-3D generation, comparing with Point-E and Shap-E in Figure "TextField3D" denotes replacing text embeddings t with image features. "#NTFBind" denotes us- ing view-invariant image latent code V.. In (a), we provide part of a clay pot in a bottom view as input. Shap-E can generate similar shape but the color is lighter than the ground-truth image. TextField3D can reproduce the color, while the generated pot neck is too long due to the interference of the input view. The shape and texture are generated precisely only when our NTFBind module employed. The front view of donkey in (b) is much harder to reconstruct, as it looks quite different in multiple views. Shap-E E fails to reproduce the donkey, generating extra legs in lighter color. The donkey generated by TextField3D is better than Shap-E's, but it still has obvious artifacts at the neck and back. The reproduction with our NTFBind has the best visual result.
5.4 ABLATION STUDIES
NTFGen Module and Multi-Modal Diseriminator. To demonstrate the effectiveness of these two proposed modules, we compare four conditional generation solutions, i.e., (a) embedding pre-
trained V-L features into GET3D, which is similar to SDFusion (Cheng et al. 2023); (b) further
Table I: CLIP R- R-Precision on COCO evaluation prompts.
Method
VIT-B/32
ViT-L/14
Latency
Data Scale
DreamFields
78.6
82.9
200 V100-hr
DreamFusion
75.1
79.1
~ 12 V100-hr
Point-E (300M, text-only)
33.6
35.5
24 V100-sec
> IM
Shap-E (300M, text-only)
37.8
40.9
13 V100-sec
> 1M
Point-B (300M)
40.3
45.6
1.2 V100-min
> IM
Point-E (1B)
41.1
46.8
1.5 V100-min
1M
Shap-E (300M)
41.1
46.4
1.0 V100-min
> IM
TextField3D (ours, 1-shot)
45.8
49.0
~ 6.9 V100-sec
175K
TextField3D (ours, 9-shot)
63,4
67.3
1.0 V100-min
175K
7

---

Published as a conference paper at ICLR 2024
DreamField
DreamFusion
Point-E
Shap-F
TextFleld3D
"An apple"
tire"
rar"
"An cream"
picee"
"A brick chinncy"
"A teagoot
Figure 4: Text-to-DD generation. We use seven representative text prompts to exhibit open- vocabulary generative capability, including single nouns, compound nouns, and adjectival nouns.
(
Iaput Image
Stap-E
TENFIsLB
Figure 5: Image-to-3D generation. We use clay pot in the bottom view and a donkey in the front view as image inputs. Our network can reproduce a close novel view to the ground-truth image.
aligning features in (a) by pre-trained models, which is similar to TAPS3D (Wei et al. 2023 (c) embedding text features with our NTFGen module; (d) further attaching our text-to-3D discriminator to (c), which is method TextField3D. We train them the ShapeNet dataset, with both BLIP. 2 (Li ct al. 2023 and MiniGPT-4 (Zhu et al. 2023) captioning.
FID (Heusel al. 2017) and CLIP-score (Hessel et al. 2021) are used for the evaluation. The results demonstrate that proposed modules effectively improve generation quality and text control consistency.
We quantitatively compare the four solutions, well GET3D, in Table
We also provide qualitative results in Figure We use a simple text prompt "A green ball" to evaluate generation quality and complicated text prompt "A round table with red legs and blue
8

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative Analysis. FID (Heusel al., 2017) and CLIP-score (Hessel et al.
2021
adopted to evaluate the generation quality and text consistency of different solutions.
BLIP-2 (Li al. 2023)
MiniGPT-4 (Zhu et al.
2023
Method
CLIP ScoreT
CLIP-Score
FID
FID
ViT-B/32
ViT-L/14
ViT-B/32
ViT-L/14
GET3D (Gao et al.
2022)
41.66
41.66
(a)
31.67
28,33
22.79
34.67
28.59
23,44
(b)
29.02
29.80
24.39
29.91
30.03
24.05
(c)
29.73
30.36
25.43
28.69
30.68
25.17
(d)
26.94
30.35
25.57
25.46
30.89
25.77
"A preen ball"
table with
top surfice"
THT
(8)
(b)
(c)
(d)
(a)
(b)
(0)
Figure 6: Ablation studies. We use two text prompts to evaluate the generation quality and text control consistency of different solutions, demonstrating the effectiveness of our proposed modules.
top surface" to further evaluate text control consistency. In the former case, (d) presents high- quality generation thanks to the multi-modal discrimination. In the latter case, (c) and (d) generate reasonable results according the textual input, demonstrating the effectiveness Noisy Text Field.
NTFBind Module. To evaluate inwarinvrriant features VL = futew (frie (f we compare it with vanilla CLIP features ví = (I2) We propose two evaluation metrics: cosine similarity a/s and retrieval ratio r/F. We randomly select front view of and back view G of 55 generations I Ih (one object for each category, with label t). For CLIP similarity, we calculate the mean similarities, i.e., 8 55 cos(vv, v², //5 and こ = 55 cos(V', v/55 For retrieval ratio, we regard the opposite view of the same object as the positive target and views of other objects as negative targets. r and 7 are then computed as the average of retrieval ratios. As shown Table 3 both the similarity and the ratio of view-invariant features are higher than CLIP features, which indicates that Losew learns a vew-invariant representation. is more obvious in generated images, further demonstrating that view-invarian features enhance the generation consistency of novel views.
Table 3: Comparison of view-invariant features and vanilla CLIP features.
Image Source
$
3
Ground-trute 88.01%
91.91%
58.07%
81.44%
Generated
84.84%
91.32%
54.54%
76.97%
CONCLUSION
In this work, we present TextField3D enhance the open-vocabulary capability of 3D generative models. Specifically, we introduce Noisy Text Fields (NTFs) to 3D latent code, enhancing the mapping of V-L pre-trained knowledge and 3D training data. An NTFGen module is proposed to generate noisy text latent code, and an NTFBind module is further proposed to bind view-invarian image latent code to NTFs. Furthermore, to supervise the generation quality and text consistency, we propose multi-modal discrimination that includes both text-3D and text-2.5D discriminators. TextField3D can efficiently generate various 3D contents with complicated text prompts, exhibiting potential open-vocabulary generative capability.
Broader Impacts. 3D creations are in high demand for various applications. TextField3D can generate open- open-vocabulary 3D objects, greatly reducing the cost of producing 3D content. However, like other generative models, there may a risk of generating malicious content.
Limitations. TextFiel3D still depends on the vocabulary of training data, which may not fully match the general capability of V-L supervised methods. For example, fails in prompts like "A corgi is reading book". current 3D data mainly focuses shape/texture-reattee concepts but lacks action-related prompts. Fortunately, the scale of 3D data is steadily increasing. More and more novel concepts be included expand TextField3D's vocabulary.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by National Key RD Program of China under Grant No. 00212D0112100, and the National Natural Science Foundation of China (NSFC) under Grant No. U19A2073.
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference machine learning, pp. 40 49. PMLR, 2018.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry- awaar 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16123-16133, 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint 151200035 2015.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13877 2023.
Yinbo Chen and Xiaolong Wang. Transformers meta-learners for implicit neural representations. In Computer Visi 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XVII, pp. 170-187. Springer, 2022.
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4456-4465, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
www.sss...r//boooo 20tp:llo//22222222 30 30-vicuna/
Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL wwwww//wwwwwww www.ber..d..oooooooooooooo oorggdrrrrrrrr....ooooooo.o
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. arXiv preprint arXiv:2212.0805) 2022.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXhv:/810004800. 2018.
Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. Shapecrafter: recursive text-conditioned 3d shape generation model. Advances in Neural Information Processing Systems, 35:8882 8895, 2022.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeie, and Sanja Fidler. Get3d: generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-318 2022.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.089:. 2021.
Sophia Gu, Christopher Clark, and Aniruddha Kembhavi. can't believe there's 's no images! learning visual tasks using only language data. arXiv preprint urXiv:2211.09777 2022.
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05377 2023.
10

---

Published as a conference paper at ICLR 2024
Erik Härkönen, Miika Aittala, Tuomas Kynkäänniemi, Samuli Laine, Timo Aila, and Jaakko Lehti- nen. Disentangling random and cyclic effects in time-lapse sequences. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, and Samuli Laine. Appearance- driven automatic 3d model simplification. In EGSR (DL). pp. 85-97, 2021.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tianyu Huang, Bowen Dong, Yunhan Yang. Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 22157-22167, 2023.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.022 02463, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the JEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6995. 2014.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In Inter- national Conference on Machine Learning, pp. 5742-5752. PMLR, 2021.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint vrXiv:2301.12597 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:221 104440, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding, arXiv preprint arXiv:2305. 10764, 2023a.
Zhen Liu, Yao Feng, Michael J Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv:2303.08113, 2023b.
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 306-315, 2022.

---

Published as a conference paper at ICLR 2024
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Gener- ating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-8, 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: system for generating 3d point clouds from complex prompts, arXiv preprint arXiv:2212.0875// 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453- 11444, 2021.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:211..0057. 2022.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Tex- ture fields: Learning texture representations function space. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4531-4540, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2209 /4988, 2022.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 1748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1)::48555555) 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shoo text-to-imaae generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061 06120000.62..... 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pallern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Aditya Sanghi, Hang Chu, Joseph Lambourne, Wang, Chin-Yi Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18603-18613, 2022.
Aditya Sanghi, Rao Fu, Vivian Liu, Karl DD Willis, Hooman Shayani, Amir H Khasahmadi, Srinath Sridhar, and Daniel Ritchie. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18339-18348, 2023.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. arXiv preprint arXiv:2001.09515 2023.
12

---

Published as a conference paper at ICLR 2024
Tianchang Shen, Jun Gao, Kangxue Yin, Ming Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101. 2021.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 1221:33300.1611,,, 2023.
Jiacheng Wei, Hao Wang. Jiashi Feng, Guosheng Lin, and Kim-Hui Yap. Taps3d: Text-guided 3d textured shape generation from pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16805-16815, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1rX1:2234411592. 2023.
A EXPERIMENTAL DETAILS
A.1 EXPERIMENT IMPLEMENTATION
NTFBind Training and Inference. To support image-to-3D generation, we additionally train an NTFBind module with the binding objective Loind. We load the parameters of the generator, the discriminator, and the NTFGen module on text-to-3D training. We freeze the NTFGen module during the training so that the NTFBind module can be aligned to the trained noisy text fields. The overall training objective L is similar to text-to-3D training:
Limage- - to 3D
L DiDimgs G' L(D G) Apc G) Abind
where Abind is set as 0.1 in the experiment. Since the camera views are sampled uniformly in fixed range, we can randomly generate camera pose for an input image on inference.
Ablation Studies. In the ablation studies of the main text, we compare four solutions for 3D conditional generation. We additionally illustrate the latent code encoder of solutions (a) and (b) in Figure 7 In (a), the latent code is initialized by V-L pre-trained text encoder and supervised by text-2.55 discriminator, the overall loss La L(Dimg: G) maDkkkk G). In (b), rendered images are further aligned with input text features, which can be regarded Limg in our NTFGen module. Its overall loss can revised as Lb = La Limg. In (c). full version of our NTFGen module is adopted, in which the overall loss can be formulated Le L[Dsmg: G) + maDk, G' + Coind Finally, (d) presents our TextField3D. Compared with (c), it is further supervised by text-3D discriminator. Note that text features in (a)-(b) are fed to mapping network, while skip the mapping of text features (c)-(d), as we mentioned in the main text.
(b)
Remkend Image?
Limg
V-L Image Facoder
+
Test
Inpar Text
Text
Tostured
Mesh Ar
Text Encoder
Encoder
Generator
modoen vector
minden vector
Generator
Figure 7: The latent code encoders of solutions (a) and (b) in the main text ablation studies.
B DATA PREPARATION
B. IMAGE RENDERING
To generate 2D images from 3D meshes for training, we follow the rendering strategy in
GET3D (Gao al. 2022 We first scale each mesh object such that the longest edge of its bounding
13

---

Published as a conference paper at ICLR 2024
box equals 0.8. We then render the RGB images and silhouettes from 8 random camera poses, which are sampled uniformly from an elevation angle of range [0, 6 .] and a rotation angle of range [0, 2 For all camera poses, rendered in Blender (Community 2018) with fixed lighting.
fixed radius of 1.2 and the field of view (FOV) is 49.13". Images
B.2 IMAGE CAPTIONING
To provide text-3D paired training data, we adopt two image captioning methods BLIP-2 Li et al, 2023) and MiniGPT-4 (Zhu et al. 2023) caption the rendered images in ShapeNet (Chang al.
2015 and Objaverse (Deitke et al. 2022). Considering the data scale, we didn't use MiniGPT-4 caption Objaverse. In this section, we will introduce the captioning strategies and compare the captioning effects through examples.
BLIP-2. BLIP-2 leverages lightweight querying transformer to bridge the gap between pre- trained image encoders and language models. We use BLIP-2-OPT-2.7 as the captioning model. For each rendered image, BLIP-2 employed expand sentence beginning with "an object of" We then extract [tex]} from the expanded sentence "an object of [texr)" as the caption.
MiniGPT-4. MiniGPT-4 utilizes an advanced large language model (LLM) Vicuna (Chiang al, 2023) and pre-trained vision component of BLIP-2 to formulate multi-modal outputs. It aligns encoded visual features with the Vieuna language model using just one projection layer and freezes all other vision and language components. Albeit a lightweight alignment, it can possess many similar capabilities to GPT-4, including generating intricate image descriptions and answering visual questions. Thus, given input image and its category label {c}. we have question template "there a 3D {c} in the picture, use one sentence to shortly describe its texture and shape" for MiniGPT-4. Note that despite emphasizing "use sentence", MiniGPT-4 is still likely generate multiple sentences. To simplify captions, we use the first sentence of the answer as the final caption. Due to the time constraint, we only utilize MiniGPT-4 make captions for ShapeNet in this paper.
Comparison of Captioning Effects. In Figure g pare the captioning effects of different methods. Since TPAS3D (Wei et al.. 2023) also proposes pseudo caption generation method for captioning rendered images in part of ShapeNet, we include the comparison. From the three cases, we can observe that captions generated by TAPS3D and BLIP-2 are relatively similar, primarily in terms of the combination of adjectives and nouns. In contrast, MiniGPT-4 provides detailed descriptions, including information in both texture and shape.
we choose three representative images to com-
TAPS3D
BLIP-2
MiniGPT-4
"A red hatchback"
"A red sports car"
"The image is red sports with sleek and shiny body, designed with a streamlined style"
"A brown chair"
"A wooden chair"
"The wooden chair has rectangular shape with curved back and flat seat"
"A wooden desk"
wooden table"
"The image is wooden table with a rectangular top and four legs"
Figure 8: Comparison image captioning effects.
14

---

Published as a conference paper at ICLR 2024
B.3 DATASET FILTERING
Although Objaverse (Deitke et al. 2022) is the largest public 3D dataset, its data quality varies. Therefore, we have filtered out some low-quality data. The filtering process has steps: First, we filter out mesh files that contain more than one object, as our method is not suitable for multi- object/scene-level generation. Then, we filter out mesh objects according to roughness, because we find there are many objects with uneven surfaces, as well as objects that look like piece of paper. We use Laplacian Regularizer (Hasselgren al. 2021) to evaluate the roughness of an object, i.e., 6, V; [.. İjEN, V/, where N; is the set of one-ring neighbours for vertex Vs. As the average laplacian regularizer metric increases, the roughness level also increases. We select objects whose 5 E [10-1, 10 10-6) as our final training data, which are not too flat or too rough for generation.
C SUPPLEMENTARY RESULTS
C.I THREESTUDIO EVALUATION SET
We find a prompt list with 415 prompts in a public repo ThreeStudio and evaluate the CLIP R- Precision in this test set. Since we haven't 't surveyed any method that is evaluated in this set, we only compare our method with Shap-E. As shown in Table R3, although it a harder test set, our method still significantly outperforms Shap-E.
Table 4: CLIP R-Precision on ThreeStudio evaluation prompts.
Method
V/T-B/32
ViT-L/14
Latency
Data Scale
Shap-B (300M)
13.98
19.51
1.0 V 1000-min
IM
TextField3D (ours, 1-shot)
19.04
24.33
6.9 V100-sec
175K
TextField3D (ours, 9-shot)
37.83
42.41
1.0 V100-min
175K
C.2 COMPARISON WITH MORE SOTA METHODS
The reason why not compare TAPS3D and SDFusion directly is that, they fine-tuning methods based on single-categons GET3D pre-training. For example, in the implementation of the original TAPS3D, pre-trained category-wise GET3D checkpoint (e.g., car checkpoint, table checkpoint, and so on) is first loaded to the generator. The generator is then fine-tuned by the supervision of fixed discriminator. Considering that GET3D hasn't been pre-trained in open-vocabulary setting. it is hard to evaluate the original TAPS3D in our benchmark.
Nevertheless, we still provide a comparison based on their official code. We load the checkpoint of our pre-trained GET3D (Table 2) and further finetune the generator with TAPS3D. As shown in Table its FID and CLIP-Score (VTT-B/32) results are uncompetitive to our method and even below the version we reproduced (named as (b) in the main text). We think the results are reasonable, as the performance of fine-tuned TAPS3D strongly depends on the pre-trained GET3D. We have demonstrated in Table of the main text that GET3D can hardly generate high-quality 3D results in multircategory dataset. As result, TAPS3D is not an appropriate approach to open- vocabuloyyy generation.
Furthermore, since TAPS3D hasn't released the training captions and finetuned checkpoints, we cannot even make comparison single-category generation with them. We notice that SDFusion has released its text-to-shape checkpoint, which is trained on a table-chair dataset. Therefore, we further evaluate the original SDFusion with recursive text prompts. As shown in Figure [T0] SDFu- sion is a shape-only generator with low generation resolution. Besides, it is not sensitive to long sentences, failing generate curved legs in the last prompt.
C.3 ABLATION STUDY OF NOISY TEXT FIELD
To verify the noise range in noisy text fields, design four text prompts with gradually in- creased attributes for comparison with Shap-E. Jun & Nichol (2023) have mentioned that Shap-E struggles to bind multiple attributes to different objects. In Figure 9 fails in the third case when
15

---

Published as a conference paper at ICLR 2024
Table 5: Performance of the original TAPS3D. The results of GET3D and (b) are from Table 2 in the main text.
Method
FID
CLIP-scoret
GET3D
41.66
TAPS3D
35.12
28.38
(b)
29.02
29.80
26.94
30.35
two attributes are added. Shap-E generates extra chairs, despite the absence such a requirement in the text input. And it is confused by the fourth case, generating blue legs and a mixed-color top. In contrast, TextField3D performs well in all cases and meanwhile presents a high-level gen- eration diversity. Moreover, we calculated the variances of the four cases, which are 0.2479, 0.2153, 0.1674, and 0. 1355, respectively. The variances are consistent with our expectations of noise range, ie., a more detailed description has a smaller noise range.
Shup-F
TextField3D
"A table"
"A round table with red legs"
round logss and blue top surface"
Figure 9: Examples of gradually increased attributes, comparing with Shap-E (Jun & Nichol 2023
To further evaluate the generation performance on gradually increased attributes, we include ShapeCraft (Fu et al., 2022) in the comparison. We use the recursive prompts in ShapCraft and rephrase them as longer sentences. As shown in Figure [T] TextField3D can achieve comparable performance with ShapeCraft, although it hasn't been specifically trained with recursive prompts. On the contrary, Shap-E only succeeds in generating the first case. It seems that Shap-E cannot understand "with no armrest", treating the phrase as "with armrest".
C.4 ABLATION STUDY OF NTFGEN
Our NTFGen module consists of two objectives, iee,, Limg and List- To verify the effectiveness of both objectives, we conduct more experiments each objective of ShapeNet (Chang al.) 2015) with captions from MiniGPT-4. We report the quantitative results in Table [6) Comparing two objectives, Cimy has better generation quality However, List achieves a higher CLIP-score, which implies that List can enhance the mapping of noisy text fields. Moreover, there's only one difference between "+Limg", " and solution (b) in the ablation studies of the main text, i.e., the position of adding text features. "+Limg" performs better than solution (b) (FID: 29.91, CLIP-score; 30.03) terms of both FID and CLIP-Score, demonstrat- ing that bypassing the mapping network for text features is effective for generation quality and text control.
alignment of generations and text features.
C.5 ABLATION STUDY OF NTFBIND
Our NTFBind module binds image features in two dimensions: (1) to the noisy text fields; (2) to the multiple views. As illustrated in Table 7 two terms of binding evaluated separately as "+binding
16

---

Published as a conference paper at ICLR 2024
ShapeCruttr
hhs backrest
"Leg
aimruse
cuzved."
chair
wagkhaaken vabaas araroes lop ane caamed
with no artrest
duar with aettrmet actirss?
SDF ankie
lasplack
"Airdd laug back
curnedt
Cure
"Sardd chale with socernset
"huedd back sritnoe
"A rod lanp-hadk dlsir ararcee,
Figure 10: More examples of gradually increased attributes, comparing with ShapeCrafter (Fu et al. Shap-E (Jun Nichol 2023), and SDFusion Cheng al. (2233
2022
Table 6: Quantitative results of NTFGen's ablation studies.
Method
FID
CLIP-scoref
Lima
29.53
30.43
+ List
30.82
30.58
NTFGen
28.69
30.68
NTFs" and "+hinding views". It is obvious that binding NTFs can significantly improve the quality of image-to-3D generation (from 56.71 to 48. 48.84. -7.87 for FID). Although additionally adding view-binding can further improve the FID to 48.07, we find that the training of binding-views alone does not lead satisfactory results (we don't don't report its result as the model cannot even achieve convergence). think is caused by the disruption of alignment between the view-invariant latent code and the noisy text fields. Therefore, binding NTFs is necessary for the image-to- 3D task.
Table 7: Quantitative results NTFBind's ablation studies.
Method
FID
TextField3D
56.71
binding NTFs
48.84
+ binding views
+ NTFBind
48.07
C.6 ABLATION STUDY OF MULTI-MODAL DISCRIMINATION
The intention of the discrimination design is decouple the supervision of texture and shape gen- eration. We mainly have two unsuccessful attempts: (1) we split the text condition and the camera condition into two independent discriminators, expecting that the former condition would focus on texture and the latter condition would focus on geometry. However, it fails to generate high-quality content (FID is 36.1 14). As result, we combine two conditions together our final discriminator. (2) we replace the mask discriminator with depth discriminator, as depth provides more geometry information than the silhouette mask. However, all the generated objects fall into a sphere shape under depth discrimination. We think that is because depth discrimination is too sensitive to depth changes. Especially when using a mesh representation DMTet (Shen et al., 2021). the prominent vertices tend to be smoothed to the neighboring surfaces easily, formulating sphere shape. In- stead, we introduce a 3D discriminator based on point cloud, which can flexibly represent the mesh surface.
17

---

Published as a conference paper at ICLR 2024
C.7 MORE VISUALIZATION RESULTS
To better present our open-vocabulary generation capability, we visualize more generated objects in Fig 12 We also provide multiple views of these objects in Figure 13 The elevation angles for rendering are - -55,, 15", and 30 from left to right, respectively. In Figure [4 we generate examples for each prompt, showcasing TextField3D's generation diversity. Note, beer can in our training dataset is given in the left of Figure 14 to demonstrate that our results don't It overfit to training data. Furthermore, we provide more visualization results of complicated input prompts and conditioned images in Figure [5 and Figure 16. respectively.
18

---

Published as a conference paper at ICLR 2024
DreamField
DreamFusion
Point-E
Shap-E
TextField3D
"An orange"
"A lemon"
"A red mushroom"
stump"
"A deer"
giniffe"
"A pagoda"
lego model
rocket"
"An Egyptian obelisk"
Figure 11: More visualization results, comparing with previous state-of of-te--f-thearr the-art methods.
19

---

Published as a conference paper at ICLR 2024
DreamFicld
DreamFusion
Point-E
Shap-E E
TextField3D
barrel"
"A beer can"
"A rusty metal camister"
"A pottery vase"
"A weoden crate"
cartlboard box with graffiti"'
screw"
"A grenade"
T
"A large axe"
"A gold ring"
Figure 12: More visualization results, comparing with previous state-of-the-ar methods.
20

---

Published as a conference paper at ICLR 2024
"An orange"
"A lemon"
"A red mushroom"
"A tree"
"A free slump"
"A deer"
"A giraffe"
"A pagoda"
"A lego model rocket"
"An Egyptian obclisk"
"A barrel"
"A beer can""
rusty metal canister"
"A pottery vase"
"A wooden crate"
cardboard graffiii
"A Screw"
"A gremade"
"A large axe"
"A gold ring"
Figure 13: Multi-view visualization of generated objects in Figure and
12
The elevation angles
for rendering are 155,, 15", and 30 from left to right, respectively.
21

---

Published as a conference paper at ICLR 2024
(5)
(7)
(9)
(8
(9)
Figure 14: 9-shot examples. We provide 9 different bear cans, presenting the diversity of our gen- eration in given text. The left beer can is the most similar one detect in training dataset, demonstrating that our method hasn't overfitted to training data.
Shap-E
Ours
covered graffini stationary the paveeent"
jar glowing
"a bowl of meal with up top everything"
very girafie that siting ground"
Shap-E
Ours
missing some keys the bottom row"
"a like ovocado"
tv consected to ipad"
tall building"
Figure 15: Complicated prompts. To further present the open-vocabulary capability, we provide the visualization results of our generation in complicated prompts. Shap-E included in the compari- son.
22

---

Published as a conference paper at ICLR 2024
Input
Shap-E
Ours
Figure 16: Image-to-3D generation. We provide more visualization results of text-to-3D generation. Shap-E is included in the comparison.
23


Reviews:
[{"review_id": "GMstJ1MHgJ", "reviewer": "Reviewer_Fjdj", "summary": "In this paper, the authors present TextField3D, which is a conditional 3D generative model that enhances text-based 3D data generation by injecting dynamic noise into the latent space of text prompts, creating Noisy Text Fields (NTFs). This technique allows for the mapping of limited 3D data to a broader range of textual latent space, enhanced by the NTFs. The authors propose two modules to facilitate this process: NTFGen, which models the general text latent code in noisy fields, and NTFBind, which aligns view-invariant image latent code to the noisy fields, aiding image-conditional 3D generation. The model is guided through the conditional generation process in terms of both geometry and texture by a multi-modal discrimination setup, consisting of a text-3D discriminator and a text-2.5D discriminator. TextField3D is highlighted for its large vocabulary, text consistency, and low latency, positioning it as an advancement over previous methods in the field.", "questions": "1. I'm wondering how is the FID score calculated under the text-conditioned setting?\n2. Can you provide some more examples of the 9-shot experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper successfully expands the GAN-based GET3D framework to handle extensive vocabulary datasets, achieving results on par with or surpassing those of diffusion-based models like point-e and ShapE. This marks a significant step forward for large-vocabulary feed-forward generative models.\n- Since text and 3D are not one-to-one mapping, , the introduction of Noisy Text Fields and their corresponding modules seem to be reasonable to me.\n- The results, both qualitative and quantitative, look diverse and of a relatively good quality compared to other feed-forward models.\n- The ablation studies are clear and comprehensive, providing detailed insights into the impacts of various modules, discriminators, and choices in noise range.", "weaknesses": "- My main concern is the potential overfitting problem. In Figure 12 and 13, certain prompts (e.g., \"A beer can\", \"A wooden crate\", and \"A cardboard box with graffiti\") generate unusually detailed outputs, showing a much higher level of details than others. Based on my experience,  the training dataset likely contains very similar examples. I am interested in understanding how the authors have addressed and evaluated the risk of overfitting associated with their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x6NHyvAQFS", "reviewer": "Reviewer_oUD4", "summary": "This paper works on text-to-3d shape generation. The paper observes that the current text-to-shape generation methods are usually V-L Optimized or 3D supervised. These methods either suffer from the problem of long-optimization or restricted open-vocabulary capability. To resolve these problems, the paper proposes TextField3D, which generates 3D shapes in real-time, taking both open-vocabulary text prompts and image as input. Specifically, the method adopts a NFGen model, that maps a single text prompt into a noisy text field. This noisy text field enables more open-vocabulary text prompts than single category name or template-generated text prompts. It also proposes an NTFBind model, which maps images from any view into a view-invariant image feature in the noisy text field. The noisy latent feature from the noisy text field is then fed into a conditional generative model. The method also adopts a 2.5D and a 3D supervision that ensures the generated shape has high-quality texture and geometry. Experiment shows the proposed method generates shapes of higher quality than some V-L optimized and 3D supervised methods. It also shows the effectiveness of the NTFGen and NTFBind modules in design choice.", "questions": "+ Some minor questions:\n1. For noisy text latent code, the paper uses a learned noise, which is referred to as \"dynamic noise\". I wonder how the method performs with a non-dynamic noise, which could set \\sigma to a static number. \n2. The noting of L_{img} and L_{txt} looks like two different types of loss, but they are actually the same type of loss. Changing the namings to make them more consistent would be better for reading.\n3. Textured Mesh Generator. Are they training from scratch, or training from the pre-trained GET3D?\n4. Sillhoutte loss. The silhouette loss in equations (4) and (5) is not introduced clearly. For a first-time reader, it might be a little confusing.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "+ The paper is clearly written. The motivation is clear to me - design a real-time supervised method whose performance is competitive with VL-optimized methods. \n+ The NTFGen model shows that it is able, to some extent, to increase the expressiveness of a single text prompt. \n+ The NTFBind model shows that it is better at aligning image features from any views to text features. \n+ The generated shapes visually seem to have better quality and texture than the compared methods thanks to the supervision from 2.5D and 3D.", "weaknesses": "This paper fails to convince me of the effectiveness of its major component in the following aspects:\n+ Shape Diversity is limited by the training dataset. Quoting from the original paper - \"With limited 3D data, can we train a real-time generator that is equivalent to V-L optimized methods\", I think this is impossible considering the method is training with a relatively small scale 3D shape dataset compared with VLMs. VL-optimized methods clearly can generate synthesized imagined shapes, like a chair with the shape of an avocado, but given the qualitative examples that the authors provide, this method seems not able to generate imagined shapes. Even though I don't think this is a major drawback of the proposed method, I think this claim is faulty. \n+ Open-vocabulary capability - The open-vocabulary capability is the major claim of this paper. However, in the qualitative experiment section, the paper only provides very simple prompts, like category names, adjective nouns, or a phrase with two nouns. I think these simple phrases are not complicated enough to prove the open-vocabulary capability of the method, especially considering the method is training with complicated enough captions generated by BLIP-2 or MiniGPT4. I hope the authors can provide more results from complicated text prompts as in the captions generated.\n+ View-invariant experiments. The paper claims that the NTFBind model produces a view-invariant feature, but the experiment provided is not strong enough to prove the point.  The experiment uses image features across views and image features across ShapeNet categories to prove that image features across views are more reassembled than features across categories. I think this setting is not strong and persuasive enough. A better experiment setup would be comparing features across views with features across instances in the same category. See CLIP-NeRF[1] Figure 2 for more details. \n+ Comparison. Though the paper compared with VL optimized methods and 3D supervised methods, it didn't directly compare with the TAPS3D, which has the same training setting as the proposed method. Both of them methods use an image captioning model to augment text prompts and work on a 3D-generated model. Though the paper provides an ablation study that replaces the major component to the TAPS3D component, I wonder if it will outperform the TAPS3D original method. \n[1] CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohXOJwn3wT", "reviewer": "Reviewer_9TYY", "summary": "This paper aims to achieve open-vocabulary 3D generation by attempting to tackle the challenges of limited 3D data and therefore sparse text annotations, which would limit the model's ability to generalize to open-vocabulary queries. One way to approach the challenge is to add noises to text features to avoid models from overfitting. But it is not trivial to know how much noise is appropriate. Therefore, the authors propose to learn a noisy text fields, which learns the standard deviation of the Gaussian noise that can be added to each text feature. Other than the text-to-image task, this paper also proposes to learn a view-invariant image representation to facilitate better image-to-3D generation. As for the generator part, the authors use a 3D-aware GAN framework, with a GET3D-like generator and a text-3D and text-2.5D discriminator. Qualitative and quantitative experiments are reported to showcase the performance of the proposed method. For the quantitative part, TextFireld3D surpasses baseline methods, including Point-E, Shap-E and GET3D. Visual results also show reasonable generation quality both from text and image conditions.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, I like that the authors choose to use 3D-aware GAN instead of diffusion models to tackle the text-to-3D generation problem. GAN has clear advantages over diffusion model in terms of generation speed and smooth interpolation. But for the open-vocabulary text-guided generation task, GAN falls behind diffusion models. Therefore, it is exciting to see a GAN model surpasses diffusion baselines, e.g., Shap-E and Point-E. This paper can service as a strong baseline for the community of text-to-3D generation.\n2. The idea of learning the noisy text field is interesting. Experiments also suggest the effectiveness of this method.", "weaknesses": "My main concerns are around the experiments of this paper.\n\n1. I would expect to see more baseline methods being compared with. For example, in the ablation study, authors mentioned methods like SDFusion and TAPS3D.\n2. I would suggest also testing with the DreamFusion testing list, which contains over 400 prompts.\n3. For the image-to-3D generation, the evaluation is limited to showcasing two visual results, which is far from enough. I would suggest adding more visual results, especially showing multiple viewing angles of the generated object. Some quantitative evaluation is also expected.\n4. For the visual results, I would like to see more prompts from the DreamField or DreamFusion test sets, which include more challenging and complex examples like concept-mixing. It would help us better evaluate the open-vocabulary ability of the proposed method. Right now, the showcasing prompts only contain one simple concept and can be easily found in Objaverse training set. I understand that concept mixing is super hard given such limited training data. But at least, it is nice to see some failure cases and analysis on the failure modes.\n\nIf authors can provide more evaluations during the rebuttal. I would consider raising the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)