UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS
Authors: Han-Hung Lee
Angel X Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Text to 3D Generation
CLIP
NeRF
Adversarial Examples
Augmentation

Abstract: We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.

Under review as a conference paper at ICLR 2023
UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS
Anonymous authors Paper under double-blind review
"night city With vsporarr Gesthetic."
city: treating
CTT
"Apericat Mascle Car golos and moon syntheave." "on onscured Anight With wings; trerding on artatation.
"Tolyo city: tranding on arts tottonn
Figure I: Examples of multi-view images generated from the input prompts with our implicit Vox model trained resolution 224 and various CLIP models. Our model produces highly detailed 3D representations roughly matching the input text.
ABSTRACT
We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access any datasets, setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no system- atic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial gener- ation problem, and how the generated results impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural net- works provide an additional layer of regularization, resulting better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.
INTRODUCTION
Text image generation has seen major recent advances with the release of DALLE (Ramesh al. 2021) and diffusion models such as DALLE2 (Ramesh et al. (022). CogView2 (Ding et al.) 2022) and Latent Diffusion (Rombach et al. 2022). A natural next step the task of generating 3D objects from text input. However, supervised methods relying on paired image-text data are less suited to text 3D generation as large-scale paired text and 3D datasets are less available. Thus, the regime of little to no 3D data training supervision is beneficial. While this might seem daunting. recent work in text 3D generation showed promising results without using large-scale datasets by bridging the gap using guidance from pretrained vision-language models such CLIP (Radford et al. 2021)
At the same time, advances in differentiable neural rendering and the development of NeRF Milden- hall et al. 2020) now allow for direct optimization of 3D representation to match input images. Combining these approaches with CLIP guidance, we can generate 3D representations from text
Supplementary Wehsite: https://iseeee coder git.bb....tttuu..... http://aaa...a.......o.......... C113300 Supp - eementta

---

Under review as a conference paper at ICLR 2023
directly without paired text-3D data, by optimizing the similarity of the text to the rendered im- ages. Work that leverages CLIP for text to 3D generation can be grouped by the amount of 3D data required. The first set of methods train generative model on a 3D dataset, and then optimize a mapping network from text to the latent space of the generative model using CLIP guidance and differentiable rendering. The second set of methods utilizes no 3D or text supervision and has access to only the pretrained CLIP model. We refer to this latter regime as pure CLIP guidance. Given the scarcity of text-3D pair datasets, we focus on this regime.
A prominent example of the pure CLIP guidance regime is Dream Fields (Jain et al., 2022> which uses Mip-NeRF (Barron et al., 2021) and CLIP to guide the 3D optimization process for every new input text prompt. Unfortunately. this approach requires sigmificant computational resources and exhibits poor quality generation with low-density artifacts when using direct voxel grid optimization (see appendix of original paper). We also find that the quality of the results in Dream Fields is largely attributable to the LiT (Zhai et al., 2022) guidance model. When using the vanilla CLIP models as in our work, results are far worse. Optimizing the CLIP similarity is also prone to adversarial examples where generated images with high similarity according to CLIP have little perceived resemblance to the text description for human (Liu et al. 202T). Recent text to 3D methods use image baaeed augmentations as regularization to prevent these issues. However, there has been no systematic study of which of these regularizations matters and how much. In addition, there several possible design choices for the NeRF and CLIP modules, including the use of explicit voxel grids without any neural networks vs implicit neural representations. We systematically compare these and other factors that impact generation quality, and show that it is possible to generate highly detailed 3D representations with voxel grids alone.
Our main contributions are: 1) We conduct a systematic study of augmentations and their effect on text to 3D generation results with pure CLIP guidance; 2) We compare different CLIP backbones for guidance as well as model ensembles for finer 3D object detail; 3) We compare the regularization effects on geometry of explicit vs implicit voxel grids; and 4) We demonstrate generation of high- resolution grids using CLIP guidance only.
2 RELATED WORK
Text to Image. Recent text to image generation work has shown impressive results, from autoregres-
sive methods (Ramesh et al. 2021
Ding et al.) 2021: Yu et al. 2022) to diffusion methods (Nichol
let al.) 2021 Ramesh et al. 2022 Rombach al. 2022 Saharia et al., 2022| Ding et al.) 2022).
However, these methods require significant computational resources, and data supervision which is harder to obtain in large quantities in the case of 3D objects and corresponding text prompts. To alleviate this problem, several works use CLIP and a pretrained image generator for image genera- tion or manipulation without explicit supervision of corresponding pairs. VQGAN-CLIT (Crowson et al. 2022) passes a randomly initialized image through pretrained VQGAN encoder to get the latent vector. The latent vector is then fed through the decoder, and after applying several augmen- tations, CLIP similarity is calculated to use as a loss to optimize the latent vector. Fuse Dream (Liu cet al. 2021) shows that CLIP similarity scores are prone to adversarial attacks and that applying Diff Augment (Zhao et al. 2020) results in more robust CLIP scores that can be used for optimization.
Text to 3D with CLIP Guidance. Initial work in text 3D shape generation relied on paired 3D shape and text data for supervised training of joint 3D-text embedding spaces (Chen et al.) 2018 Jahan et al. 2021 Liu et al. 2022). Large pretrained image-text embeddings and differentiable ren- dering led to recent work demonstrating that CLIP and NeRF enable 3D object generation (Sanghi al. 2022, Jain al. 2022). manipulation (Michel al. 2022 Wang et al. [022]] Youwang al., 2022). and even 3D human animation generation (Hong et al. 2022) without direct supervision of text and 3D corresponding pairs. Here we distinguish between two different levels of supervision in these works. Although there are no corresponding text and 3D examples in the first category, a dataset of 3D objects is available. CLIP-Forge (Sanghi et al. 2022) first trains an implicit (oceu- pancy) autoencoder model on the ShapeNet dataset. Then in a second stage, normalizing flow model is trained with multi-view images and CLIP to project from the CLIP latent space onto the latent space of the autoencoder. CLIP-NeRF (Wang al. 2022 uses a similar approach by train- ing a conditional NeRF model and a mapping network to predict updates for the conditional codes according the input text. Text2Mesh (Michel et al. 2022) takes an input base mesh matching
2

---

Under review as a conference paper at ICLR 2023
the semantic class of the input text and trains color and displacement prediction for the base mesh according to the text prompt using CLIP similarity as the primary loss. CLIP-Mesh (Khalid et al. 2022) starts from a sphere mesh initialized with random normal and texture maps, and optimizes mesh vertices and maps through differentiable rendering and CLIP similarity between text and cur- rent object. Dream Fields (Jain al. 2022) utilizes NeRF representation instead to generate views of the 3D object from set of cameras and uses a similar optimization process, As shown in FuseDream (Liu et al. 2021), CLIP is very prone to adversarial generations. In the first class of methods using 3D dataset supervision, generations are constrained with a prior over 3D objects making the adversarial generation problem less severe. Since no such datasets are present in the second regime, augmentation techniques are utilized to prevent such adversarial generations in the case of Dream Fields and CLIP-Mesh. CLIP-Forge, CLIP-Mesh, and Dream Fields are text-to-3D generation methods, while CLIP-NeRF and Text2Mesh are text-to-D manipulation methods. Also, Text2Mesh, Dream Fields, and CLIP-Mesh require per-prompt training, while the other works do not require training/tuning for new text prompts during inference. Compared to previous work with- out 3D supervision, we systematically study different image augmentations, CLIP architectures, and ensembling of CLIP models, and the corresponding impact on generation results.
Hybrid Representation NeRFs. In the original NeRF (Mildenhall al. 2020), a coordinate- based MLP continuously represents a 3D scene compared to traditional explicit representations such as voxel grids. However, slow and memory-intensin image due to the number of forward passes required for sampled points along rays before plugging into volumetric rendering. Other work (Liu et al. 2020 Yu et al.) [22T] Garbin et al. 2021 Hedman et al. 2021: Reiser et al. 2021) have utilized hybrid implicit-explicit representation to speed up rendering. However, such methods still use coordinates and Fourier embeddings to represent the input. More recent work propose learning the input features stored directly in voxel grid structures. Due to learning better input representations, the MLPs used in these models can be smaller, and density values can be used prune calculations for voxels the grid. DVGO (Sun et al.; (022) represents the scene as an explicit density grid and learns feature grid along with a view view-dependent MLP to model color. TensoRF (Chen et al. 2022) factorizes the voxel grids as vectors and matrices, allowing for a smaller memory footprint. InstantNGP (Müller et al.] (022) utilizes multiresolution hierarchies of hash tables to represent the learned feature grids and another MLP to predict density and color. Plenoxels (Fridovich- Keil et al.) 2022) learns sparse voxel grid of density values and spherical harmonic coefficients for color. ReLU Fields (Karnewar al. 2022) also utilizes explicit density grids with spherical harmonics for color and shows that using ReL.U activation after interpo- lating query point in the grid result in competitive results. Our work builds on DVGO. In the explicit voxel grid model, the density and color grids are optimized directly with no neural networks. This is akin to learning alpha and RGB values in an image directly in the 2D case. For the implicit version, use neural networks to predict the density and color grids.
computation is required to render even a single
3 BACKGROUND
CLIP. Radford al. 2022)) implement CLIP an image encoder Enc, and text encoder Encr trained using contrastive losses. The two encoders project image σ, and text 2T into vectors in an aligned latent space fy Eny)(x7) E I=
vr = (2T) E R=.x1, Since the latent space is aligned between two modalities, the similarity between the image and text is given by the cosine similarity "/-0//1|o ||| The model is pretrained on a large-scale dataset collected by OpenAl consisting of more than 400 million image text pairs. We refer to different pretrained CLIP models by the architecture of the image encoder such as "ViT-B/32", "ViT-B/16" etc.
Dream Fields. Jain et al. (2022) introduced the use of pure CLIP guidance for 3D object gener- ation. For the backbone 3D representation, they use implicit NeRF representation, specifically Mip-NeRF(Barron et al. 2021) which uses conical frustums over sampled points to calculate an in- tegrated positional encoding (IPE) as opposed to the positional encoding (PE) in the original NeRF. CLIP similarity between the rendered image and input text prompt is used to facilitate training. They use a background augmentation scheme to prevent adversarial generations.
Direct Voxel Grid Optimization. In DVGO (Snn et al. density and color in NeRF. The density grid (density E R IXXN Na x NyxN, is modeled explicitly. The color grid is either modeled explicitly (igh) R XN= with a learnable feature grid
2022), , voxel grids are used to represent the
3

---

Under review as a conference paper at ICLR 2023
Implicit Voxel Grid
Explicit Voxel Grid
traditional buildings, trending
Raw Densily e 11x,, N,, N
CUP Text Erioddr
CLIP Image Encodor
Density MLP
Lezu
Grid Points Y2)
Diff
Color MLP
Perspective
RE N. x X N
love
Raw
We investigate the effect of different augmentations on text to 3D generation using pure CLIP guidance. We study two radiance field representations: explicit voxel grid VoXe<p with density and color, and implicit Voximp model that uses MLPs to predict the density and color. We train model using CLIP guidance by optimizing the similarity of the CLIP encoded latent vectors of the rendered image and the input text prompt. The rendered image is augmented before CLIP encoding (with Dug BackAug and PerspAug). We introduce KI. divergence loss against a spherical prior (red sphere) to enforce compact scene the box enclosing the red sphere shows the scene bounds.
Figure 2:
(feat) E Dx N and shallow MLP to predict color for query points implicitly where D is the feature length. The original DVGO had stages: a coarse stage to locate the object the grid, and fine detail stage. There is ground truth surface in our task, so we do not require coarse stage. Volumetric rendering follows the quadrature rule (Max) 1995). Concretely, Eq. i used to render the color (()) for a ray r where ò, is the interval between adjacent samples, a; the alpha compositing value, K is the number of queried points along the ray, and Tr, is the transmittance.
K
f-
(()) = Σ Tr, a,c, where a, = alpha(o,, 8,) = exp(-o,6.), and Tr, II j=1
(1-a)) - a,) (1)
The density and color are interpolated from voxel grid values the sample point coordinate and passed through a shallow MI.P for implicit feature grids. The sigmoid function is applied to the raw color value. For the density, post-activation scheme is used with the softplus function on the raw density, a (post) alpha(softplussinnten) (denssty)) 8).
METHOD
Model architecture. For our NeRF model, we implement two variations of the voxel grid repre- sentation: VOXEsp, an explicit voxel grid representation, and VoXImp. an implicit version that uses MLPs predict the density and color. Voxgsp explicitly models the two voxel grids consisting of the density (density RIIXNxXNxXX, and color (igh)) R3*N,xX,NX,, Vox is an implicit coordinate-basel MLP voxel grid representation with positional encodings (Mildenhall et al. 2020) of the grid vertex coordinates formulated as (PE) E R NN,NN where L is the channel size after positional encoding of grid vertex coordinates. Separate density and color MLPs are applied on the positional encodings to obtain the density and color predictions. We base our voxel grid model implementations on DVGO. However, note that our positional encoding feature grid (PE) is fixed and not learnable like the DVGO feature grid (feat) for color. Our overall model illustration can be found in Fig. The trainable parameters for the explicit model the parameters of the explicit grids and the bias term in the softplus activation of the density value. For the implicit voxel grid the trainable parameters are the density and color MLPs and the bias term in the softplus function. During training we also add progressive scaling of the voxel grid resolution as in DVGO.
Augmentations. We study the impact of combining three augmentation schemes: Background aug- mentation (BackAug) from Dream Fields Jain et al. 2022), Diff augment (DiffAug) Zhao et al. 2200)), and perspective augmentations (PerspAug) from Text2Mesh Michel et al. (2022). BackAug consists of alpha compositing checkerboard, textures or gaussian noise backgrounds to the image. DiffAug contains several image augmentations, including color jittering, image translation, and
4

---

Under review as a conference paper at ICLR 2023
cutout. Liu et al. (202)) used it for text image generation and showed it prevents adversarial generations. PerspAug denotes random perspective transformations applied to the image.
Losses. We combine the CLIP and transmittance losses introduced in Dream Fields, with losses from DVGO to reduce noise and promote smoothness. In addition, we introduce a spherical prior loss term to encourage a coherent object. For a model parameterized by 0, the CLIP loss (Eq. 2 enforces the cosine similarity between the NeRF generated image I p) for a camera pose p and the input caption 27 to be high in CLIP space. The transmittance loss (Eq. 3 prevents the scene from being overerowded by applying a loss when the average transmittance over the threshold and Tr(0, p) is the transmittance image.
LCLIP (0, p, 2T) (I,,r(((, (xT)
Cn min(r, mean("Tr(0 p)))
To encourage centered objects and uniform size, we introduce spherical prior (Eq. where the probability is 1 for coordinates q within a sphere of radius 1. We calculate the KL divergence between the spherical prior and the density voxel grid with the sampled point coordinates q from grid vertices (Eq. (5) This loss serves the same purpose as ray shifting in Dream Fields. However, it not trivial to shift and scale the voxel grid directly. Therefore, we promote centering and uniform size through this loss instead.
P sphere (q) = (,_
=(0
if g < otherwise
(4)
P Psphere (q)
LKI.,
Σ Pa Paphere (q)log
(5)
(<post) q, Vddesity))
In our model, we enable the ensembling of different CLIP models by adding a second similarity loss using a different CLIP model as shown in Eq. [ where Enc,, and Ency, are the image and text encoders for the second CLIP model respectively.
CCLIs (f. pT) = -Enc,, (I(O, p)) (ncr,(ET)
(6)
Following DVGO, we also add a total variation loss Liv to reduce noise and promote smoothness and background entropy loss La to encourage density values be either 0 or 1. Our complete loss is shown in Eq. where the As are the weights for the different loss terms.
L
LCUP + XTLLI + λTVLTV AKI,, LkI., + Aahhr + ACI.P, CCLP,
We find that during optimization, it is important to schedule the Lrv and LKL, loss terms so that they are turned off toward the end of the optimization. Please see Appendix A.1 A.I.oo for more discussion and details of other hyperparameter values.
EXPERIMENTS
5.1 IMPLEMENTATION
We sample camera poses following Dream Fields with a fixed elevation of 30, azimuth of 360 around the scene, and radius of 4 between the camera and origin. For the augmentations, unless otherwise specified, we turn on all three of them. For DiffAug, we use the same settings as in the original implementatior[] For BackAug, we use the same JAX implementation from Dream Field For PerspAug, we set the distortion to 0.6 with a probability of being applied to 1.0. The augmentations are applied in the order of (1) DiffAug, (2) BackAug, and (3) PerspAug. For most experiments, eight variations are applied for each image with both DiffAug and BackAug and only one for PerspAug, resulting in 64 images per optimization iteration. For larger models trained with CLIP ViT-L/14 and Vox only four variations of DiffAug is applied for given image resulting in 32 images after augmentation. Training is conducted with image resolution 224² during training using the VoXEsp model with the CLIP ViT-B/32 model for 15k iterations unless otherwise specified. The MLPs used in our implicit voxel grid are 3 layers deep with hidden layer dimensions of 128.
pytorch.py
5

---

Under review as a conference paper at ICLR 2023
"Bouquet of flowers sitting clear gLass vase."
Yh
Diff:
Diff:
Diff:
Diff:
Diff:
Diff:
Back:
Back: /
Back:
V
Back
Back:
v
Back: <
Perspective:
Perspective:
Perspective
Perspective:
Perspective:
Perspective KL;
KL:
KL:
KL:
KL:
KL:
Figure 3: Ablations showing how the prior KL divergence loss with different augmentations affect the generation results of Voxgxp- The chart below the images indicates the mean gradients with spect to the losses for the training iterations. The blue and orange lines represent the mean gradients for the density and color grids respectively. can be seen that the LKI. loss is essential for the appearance of densities when augmentations are turned on.
"Apples on tree ready to pick garden area." "
pork dissh with onions and peppers on white plate."
Diff: *
Diff: <
Diff: *
Diff: Back:
Diff:
Back: *
Back: *
Back: /
/
Back:
Perspective:
Perspective:
Perspective:
Perspective:
Perspective:
Figure 4: Ablations showing how different augmentations affect the generation results of VOXExp- We see that different augmentations can have distinct effects on the generation result and that combining them can result in better coherency.
5.2 ABLATION
We compare the effect of different augmentations and the KI. loss term on Vox In Appendix A.3 we show the augmentation ablation results on VOXimp-
Prior KL Loss Ablation. We show the effect of the CKL, loss term on the generation results in Fig Blank images indicate that the densitics in the entire scene
When all augmentations and the KL. loss is turned off (first column) the generated scene contains floating clouds of non- coherent artifacts. If augmentations voxel grid will fail to produce any densities. We hypothesize this is because the CLIP similarity loss landscape is more discontinuous when we add augmentations, making hard for densities to emerge since there are no gradients in any direction. The gradients are near zero for these cases shown in the plot below the images. When the KL loss is turned on (last columns) the densities more concentrated and doesn't 't disappear even when all augmentations are turned on. This could mean that the KI. loss term helps force densities emerge in the beginning allowing for meaningful gradients. Therefore, in addition to regularizing the size and centering of the object generated, the KL loss term also plays a vital role generating coherent results.
turned without the KL loss (2nd, 3rd columns), the
Augmentation Ablation. We show the effect of the augmentations: (1) DiffAug, (2) BackAug, and PerspAug and their combinations in Fig. 4 When all three augmentations turned off (first col-

---

Under review as a conference paper at ICLR 2023
"watte painting of bonsat tree; trending artstation.
"Bouquet of flowers sitting clear glass vose."
"A pizza that is covered in Lot of toppings,
RN50
VIT-B/32
VIT-8/1G
VIT L/14
VIT-8/32 VIT-L/14
VIT-8/16 VIT-L/14
Figure 5: Generation results using different CLIP image encoder architectures. The text below each column indicates the pretrained CLIP architecture used for guidance during training. We see that ensembled CLIP guidance can generate more detailed geometry than using one model.
umn), see the results are very noisy and incoherent for the text prompt. With only DiffAug on (col 2), textures start appearing. but the densities are sparsely distributed like point clouds. For results with only BackAug (col 3), the object is condensed and looks more like contiguous ob- ject. When both are turned on (col 4), the objects more clearly match the input prompt. Turning on PerspAug (col 5) helps to get rid of unwanted background textures in the apple tree prompt. However, it may create an overly smooth image or almost cartoonish effect in the plate of food. While we observe that turning off PerspAug has better qualitative results for many prompts, is infeasible to perform per-prompt tuning, we leave PerspAug on for following experiments. For the Iox model, we observe that the model can produce clean results without background textures even without PerspAug (see Appendix A.3
5.3 DIFFERENT CLIP MODEL EXPERIMENTS
Backbone Networks. We compare generation results from 5 pretrained CLIP models with different backbone networks in Fig. [5] Overall the ResNet (He et al.) 2016) models (ResNet50, RN50x4) are very noisy, but some features within the generations vaguely resemble the text prompt. It seems that these models have a hard time separating the main subject the text prompts from other textures. For ViT-B/32 and BT6- the generations are more coherent and have good separation from un- wanted textures compared to the ResNet models. can also be seen that ViT-B/16 can generate more details, seen in the bonsai plant and pizza. This has also been shown in the Dream Fields paper. However, when we use the biggest model, ViT-L/14,, the results are very noisy and non-coherent. We hypothesize that as the models increase to certain point, becomes easier find adversarial images within the CLIP space. This could suggest that ViT-L/14 is overfit on the texture features for images, the objects' overall structure. Discounting the ViT-L/14 model, vision transform- ers (Dosovitskiy et al. (021) show much better results than traditional convolutional architectures. We attribute this to the tokenization of image patches, which discretizes the image into regions and may help to disentangle the overall structure and textures of the image.
CLIP Model Ensemble. Fig. 5 (last two columns) shows the results of different CLIP ensembles. namely ViT-B/32+ViT-L/14 and ViT-B/16+ ViT-B/16-ViT-L/14 We see that for both ensemble models, we generate much finer details than using guidance model alone (see example for the flowers and bonsai plant). This suggests that using a smaller CLIP model to initialize the generation process helps prevent the bigger model VIT-L/14 from adversarial generations and may even help generate

---

Under review as a conference paper at ICLR 2023
Volos
ARDER
Vokerp
Dream Relds
hny full of traak victing
plastic jar honey plasinn in widele
"an wange hike leaning: pole. snow.
"The side of on rcar sirerwft shoving the
white cunted on
"A zebra is cating gress on the ground.
Figure 6: Our method compared with Dream Fields. All models shown here was trained resolu- tion 168² for 10k iterations and uses CLIP ViT-B/16 for guidance. We see that the VoX model generally has the best coherency in terms of both both object texture and geometric structure.
Table 1: We compare the accuracy for R- R-Precision@) @ 1 for our models to that of Dream Fields using different guidance models. R- R-Precision values parenthesis (right column) indicate when the CLIP model used to evaluate retrieval is the same the CLIP model used for optimization.
Retrieval model R- Precision
Method
ViT-B/32
ViT-B/IG
Dream Fields (ViT-B/16)
59.8 2.8
(93.5 + 1.4)
VOXEsp (ViT-B/16)
83.01
(90.85)
VOXimp (ViT-B/16)
88.24
(98.65 69)
more fine grain details. This is likely due to the bigger model size of ViT-L/14 as well as the smaller patch sizes takes in tokens. However, we also observe that some generation results are worse, such as the pizza having fewer details than just using ViT-B/32 or ViT-B/16. We attribute this to the two CLIP models having different maximums for the pizza, which may cause conflicting similarity losses and generate less sharp objects.
Explicit vs Implicit Voxel Grids Dream Fields. We show the results of VOXExp. VoXimp and Dream Fields in Fig. 6 All three models were trained with image resolution 168 and 10k iterations using CLIP VTT- model. Note that we turn PerspAug for VoX as unnecessary for the implicit model (see Appendix A.3 The VOxexp model is able generate textures that corresponds with the text prompt. However, the geometry does not conform with objects in the real world. The Voximp model in comparison produces objects with better geometrical structure. We see this with the honey jar, where the VoXmmp model generates a cohesive 3D object whereas we see an incomplete jar in the VOxesp model. The coherency for VoXimp is also better with respect the text prompt. For example, in the text prompt involving a bag of trash and a park bench, the Voximp model successfully generates both items whereas there is only trash bag for Voxgxp- However, we notice that for complex objects like animals (cat, zebra) all models fail at producing correct geometry. This likely due the limitation of the guidance model used. This failure mode was described as "repeated patterns multiple sides of the object" by Dream Fields. Overall this is still a very hard task and we find several prompts where the models failed to generate cohesive results as can be in the example with the orange bike and aireraft door. Dream Fields had poor results overall and many generations are unrecognizable with respect to the text prompt. This is in contrast to results shown in their paper using the LiT VIT-B/32 (Zhai et al. 2022) model trained on billions of higher resolutions (2882) captioned images.
For quantitative comparison, follow Jain et al. 2022) and report the R-Precision @1 retrieval scores for the three models Tab, [] We calculate R-Precision @ retrieval scores for the text and images of the objects validation set of 153 text prompts used in the Dream Fields paper. The
8

---

Under review as a conference paper at ICLR 2023
images rendered at held-out poses of elevation 45 following Dream Fields. We report the trieval scores for both ViT-B/16, which the models were optimized on, and different CLIP model, ViT-B/32. All models are able to achieve high retrieval scores when evaluated on the CLIP model that they were trained on. Overall, we see that the VOX model achieves the best quantitative re- sults. The VXXEsp model outperforms Dream Fields considerably when evaluated with different CLIP-model. We note that training at lower resolutions (1682) result in quality degradation com- pared to models trained at higher resolutions in Fig. D and that training at higher resolutions (224²) may result in even better results while still using less memory (see Appendix A.5) compared to Dream Fields.
DISCUSSION
3D Representation. In this paper we mainly explored how pure CLIP guidance affects NeRF mod- els. However, it would be interesting to see if findings here also apply to methods based on explicit representations such as CLIP-Mesh (Khalid et al.) (2022). Since they start with a primitive mesh for optimization the possible topology and complexity of possible generations are limited compared to NeRFs. It would be worth exploring whether this helps regularize CLIP guidance.
Augmentations. Augmentations guidance. We showed how different augmentations affect the generation results and that sometimes can even be harmful to the coherency of the results. This was shown with perspective augmen- tation. It can help eliminate unwanted textures in the background for some prompts, but can also overly smooth the textures for others. While prior work in this task has not put as much emphasis on testing different augmentations, we think it is important to explore more variations to find the best augmentations for different use cases.
essential for preventing adversarial generations from CLIP
Guidance Models. While we may try all sorts of augmentations to improve coherency, the main constricting factor lies within the guidance model such CLIP. If the guidance model is not able to learn the correct structure and geometry of objects, no augmentations will help bring those out in the results. Therefore, it important also improve the guidance models. We that while Dream Fields had worse results when using vanilla CLIP models for guidance, the LiT (Zhai et al. 2022) model used in their paper had huge impact on making their results coherent. This suggests the fine-tuning process in LiT can be helpful in learning better features for guidance. We also find using alternative CLIP models such as OpenCLIP Ihharco et al. (20TT) can result in high quality generation (see Appendix A.4.. It may also be helpful to look into more recent text- text-image contrastive models such as SLIP Mu et al. (02T) and UniCL (Yang et al., 2022]. Our model can serve as a proxy qualitative measure for text-image similarity models to see how robust the features learned are and whether the models have learned underlying structures for objects that can be used to construct geometry that conforms with the real world.
Dataset Supervision. Another avenue to help with adversarial generations is to have 3D dataset to help provide a good geometric prior for generators. Many current text to 3D works leverages this to generate higher quality results, albeit the cost of text prompt freedom. Our work orthogonal to theirs, and the techniques proposed in this paper can also be applied to their works.
Resource Usage. Overall, our models take less memory and are faster to train than Dream Fields (see Appendix AoppeniixA.5]ttte
[o55 for details). Our smallest model (Voxesp with CLIP ViT-B/32) uses 7GB and can be optimized on RTX 2080 Ti in roughly 20 minutes.
7 CONCLUSION
In this paper we have showed that is possible to train an explicit voxel grid NeRF model with pure CLIP guidance to generate coherent results with combination of image-based augmentations. We also provide empirical evaluations for how different augmentations as well as implicit voxel grids prevent adversarial examples within CLIP. Our model performs better than prior work when using the same CLIP model for guidance. It is also less compute and memory intensive while also being faster train.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
We acknowledge there are potential risks with using the CLIP (Radford et al. 2021 and Open- CLIP (Ilharco al., 202T) models for guidance pertaining to deceptive and discriminatory content. CLIP was trained by OpenAl with an in-house dataset of 400 million text-image pairs. While the dataset not publicly released, models that leverage CLIP for learning have demonstrated biases for applications like text-to-image generation (Struppek et al.) 2022). OpenCLIP was trained on the LAION (Schuhmann et al. 2021) dataset which contains offensive and explicit images as shown by Birhane et al. (2021). Our model inherits these biases as use these models priors to gen- erate 3D objects. On the one hand, text-to-3D models provide more accessible tools for the general public. On the other hand, they also be used produce harmful content by malicious actors to spread deceptive and hateful information. In addition, other potential ethical concerns raised by text-to-image (Nichol al. 2021 Ramesh et al. 2022. Rombach al. 2022; Saharia et al. 2022) models also apply. More systematic studies are needed in the space of text-to-3D models as higher quality generations are becoming possible and the potential risks need to be assessed and safeguards put in place to prevent misuse.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Brualla, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In Proc. of International Conference on Computer Vision (ICCV), pp. 5855-5864, 2021.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:21 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In Proc. of European Conference on Computer Vision (ECCV), 2022.
Kevin Chen, Christopher Choy, Manolis Savva, Angel Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Proc. of Asian Conference on Computer Vision (ACCV), 2018
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castri- cato, and Edward Raff. VQGAN-CLIP: Open domain image generation and editing with natural language guidance. In Proc. of European Conference Computer Vision (ECCV), 2022.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. CoeWiew: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34:19822-1998 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. CogView2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204. /42/7, 2022.
Alcxey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. of International Conference Learning Representations (ICLR), 2021.
Sara -ridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proc. of Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 5501-5510, 2022.
Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fast- NeRF: High-fidelity neural rendering at 200fps. In Proc. of International Conference Com- puter Vision (ICCV), pp. 14346-14355, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. of Conference on Computer Vision and Pattern Recognition ( (CVPR), pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Bak- ing neural radiance fields for real-time view synthesis. In Proc. of International Conference on Computer Vision (ICCV), pp. 5875-5884 2021.
Fangzhou Hong. Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. AvatarCLIP: Zero-shot text-driven generation and animation of 3D avatars. arXiv preprint arXiv:2205.08535, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021. URL https: //cesi///ccoo./ otgg/0..221111
zenodo. 5143773
Tansin Jahan, Yanran Guan, and Oliver van Kaick. Semantics- guided latent space exploration for shape generation. In Computer Graphics Forum. Wiley Online Library, 2021.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- eext-guided object generation with dream fields. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. ReLU fields: The little non- linearity that could. In ACM SIGGRAPHI Conference Proceedings, pp. 1-9, 2022.
Nasir Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-mesh: Generating textured meshes from text using pretrained image-text models. ACM Transactions Graphics (TOG), Proc. SIGGRAPHI Asia, 2022.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel 2020.
fields. Advances in Neural Information Processing Systems,
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. FuseDream: Training-free text-to-image generation with improved CLIP+GAN space optimization. arXiv preprint 1r. 2021.
Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. Towards implicit text-guided 3D shape generation. Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108, 1995.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13492-13502, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In Proc. of European Conference on Computer Vision (ECCV), pp. 405-421, 2020.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Sulf-supervision meets language-image pre-training. arXiv preprint 12.iv:21 12750, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), Proc. SIG- GRAPH, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2112.10744 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- fional Conference Machine Learning (ICML), 2021.

---

Under review as a conference paper at ICLR 2023
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proc. of International Conference on Learning Representations (ICLR), pp. 8821-8831, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint wrXiv:2204.06125 2022.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In Proc. of International Conference on Computer Vision (ICCV), pp. 14335-14345, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proc. of Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malckshan. CLIP- Towards zero-shot text-to-shape generation. In Proc. of Conference Computer Vision and Pattemn Recognition (CVPR), pp. 18603-18613, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:211 1 14, 2021.
Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting. The biased artist: Exploiting cultural biases via homoglyphs text- exxt-guided image generation models, arXiv preprint arXiv:2209.0888
2022.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5459-5469, 2022.
Can Wang. Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. CLIP-NeRF: Text-and- image driven manipulation of neural radiance fields. In Proc. of Conference Computer Vision and Puttern Recognition (CVPR), pp. 3835-3844, 2022.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in amage-text-labes space. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19163-19173, 2022.
Kim Youwang, Kim Ji- Yeon, and Hayn Oh. Clip-actor: Text-driven recommendation and styl- ization for animating human meshes. arXiv preprint arXiv:2206. 04382, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proc. of International Conference on Computer Vision (ICCV), pp. 3762-5761, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10738 2022.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18123-18133 2022.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. Advances neural information processing systems, 2020.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 HYPERPARAMETERS AND SCHEDULING
Our density and color MIPs are both 3-layer MIPs.. The input dimension for both are 63 which includes the raw coordinates and fourier features. The hidden feature size is 128 for all layers and the output size is and 3 for density and color MLPs respectively. The loss weight As for each of the terms can be found in Tab. ACLIP is only set for models trained with two CLIP guidance models, otherwise set to 0.
Table 2: Hyperparameters for the loss weights.
Loss Weights
Value (Voxes))
Value (VoXtmp)
A
0.01
0.01
ÀTV
0.1
0.2
\T
0.5
0.5
AKL
0.05
0.2
ACLIP
0.5
The other important hyperparameters can be found in Tab. Here the number of voxels decides the resolution of both the density and color grids. The grids are rectangular shapes with the proportions decided by the camera frustums and each side has integer number of voxels scaled so that the volume roughly has the amount of voxels as shown in the table. We also employ progressive scaling as in the original DVGO. For each scaling iteration the number of voxels is doubled with the starting number of voxels Na where Npg is the number of progressive scaling iterations.
2
We find that it is important to schedule the Lrv and LKL, so that they are only turned on for some iterations. The TV loss LrV is applied only during the iterations defined in the table and turned off otherwise. Since the density values are near zero in the beginning, we find that the TV loss that encourages smoothness will prevent any densities from appearing. Also TV loss will make the object too smooth so we also turn off towards the end to help the formation of sharp textures. For the KL loss LKL, turn on only in the beginning iterations. This is because the KL loss term helps to make the densities emerge, but may also encourage noisy densities appear outside the object within the sphere prior. For the *CLIP ensemble iterations', it is only applied to the second CLIP guidance model CCLI, for ensembled guidance models.
Table 3: Hyperparameters for other values.
Image Resolution
Other Hyperparameters
1682
224
Training Iterations
10k
15k
Number of Voxels (N₂ x N, x N₂) 1403 (VoX 160 (VoXimp)
160
Progressive Scaling Scheduling
[4000, 6000, 8000]
[5000, 7000, 9000, 11000]
TV Loss Iterations
4000 ~ 9000
5000 ~ 13000
KL Loss Iterations
7000
0 0~ 8000
CLIP Ensemble Iterations
4000 ~ 10000
5000 ~ 15000
In Fig. 7 we illustrate the effects of both KL and TV losses. For KL loss we see very clear circular hue around the generated object where the sphere prior locates. We turn this off in the later iterations to eliminate some noise densities around the object. For TV loss when it is turned on it helps to smooth the object. Again we turn it off towards the end of training. It can be seen that when it is turned off textures and edges appear more clearly to make the object look sharper.
A.2 AUGMENTATION, ARCHITECTURE R- PRECISION
Augmentation Ablation The R-Precision for augmentation ablations can be found in Tab, 4 Here Voxexp is trained using the ViT-B/16 CLIP model. Note that we add variant of ViT-B/32 that was
13

---

Under review as a conference paper at ICLR 2023
1000
2000
3000
4000
5000
6000
KL Loss On
Loss
7000
8000
9000
10000
11000
12000
15000
TV Loss OfF
We show the rendered images from random pose during the training process our model with the prompt Bouquet of flowers sitfing in clear glass and Vox using ensemble ViT-B/32 + ViT-L/14 CLIP model as guidance. The numher above image represents the training iterations. The orange bar and blue bar shows the duration CKL, and LTv was turned on.
Figure 7:
Table 4: We compare the R-Precision score for various augmentation configurations using different guidance models. The checks and erosses indicate which augmentations turned Precision values in parentheses (middle column) indicate when the CLIP model used to evaluate retrieval is the same the CLIP model used for optimization. We that the DiffAug augmentation is important for high R-P R-Precision and that PerspAug can hurt performance. Note that R- R-Precision is fairly crude metric that only measures whether the text prompt can be retrieved from the generation and does not directly measure the quality of the generation.
off. R-
Retrieval model R-Precision
Method (Diff/BackkPersseettiv)
ViT-B/32
ViT-B/16
ViT-B/32(OpenCLIIP
VoXe<p (x/x/x)
49.02
(95.42)
39.22
VOXExP (VIx/x)
83.01
(98.69)
80.39
VOXExp (xIVIx)
49.67
(91.50)
45.10
VOXExp (VIVIX)
84.97
(98.69)
92.16
VOXExp (VIVI))
83.01
(90.85)
84.97
trained the LAION-2B (Schuhmann al. 2021 dataset the evaluation, indicate this model as ViT-B/32(OpenCLIIP) We see from the results when we add DiffAug, the R-Precision value goes up significantly. This indicates the texture of the object plays a large role in the coherency of the object according to CLIP. When just adding BackAug, the scores do not go by much. However, when we add both DiffAug and BackAug the R-Precision scores the R- R-Precision score when added, this is probably due to the reduction in texture sharpness we saw Fig. 4 However, qualitatively for some text prompts helps to remove noise and unwanted background textures making the object have better geometric coherency. We notice that for many objects with broken geometry as long as the texture resembles the text, the R-Precision will still be high. This is big limitation of CLIP R- R-Precision as it is not good metric for whether not the generated object has good geometric structure.
the best. PerspAug decreases
Architecture Ablation The R-Precision for VOXexP and Voxmm trained with different CLIP archi- tectures be found in Tab. 5 For Voxgsp all augmentations turned on, for Vox DiffAug and BackAug are turned on. We refer to the ViT-B/32(OpenCLIPP column to compare methods as the models are trained either using ViT-B/32 or ViT-B/16. The VoXeap model trained with ViT-B/16
14

---

Under review as a conference paper at ICLR 2023
Table 5: Here calculate the R-Precision for different guidance models that used to train VXxexp and VoXimp- R-Precision values in parentheses indicate when the CLIP model used to evalu- retrieval the same as the CLIP model used for optimization.
Retrieval model R- R-Precision
Method (CLIP)
ViT-B/32
ViT-B/16
ViI-B/32(OpenCLIIP
VXXexp VVT--B322
(87.58)
73.20
80.39
VOXEsp ViT-B/16)
83.01
(90.85)
84.97
VOXExp (ViT-B/32 + ViT-L/14)
(71.90)
61.44
61.44
VoX (ViT-B/32)
(99.34)
90.20
94.12
VoXimp (ViT-B/16)
88.24
(98.69)
85.62
"Apples tree ready to pick in garden area."
"A pork dish with onions and peppers white plate."
Diff:
Diff:
Diff:
Diff:
Diff:
Back:
Back: *
Back:
Back:
Back:
Perspective:
Perspective:
Perspective:
Perspective:
Perspective:
Figure 8:
Augmentation ablation for the VoXimp model. The implicit model is able to produce results without unwanted background textures even without perspective augmentation.
has the highest score for explicit models. We see that the ensemble model VVTT-B/32+ViT-L/U has much lower scores compared to the other explicit models. Qualitatively, the ensemble models have higher texture details for most objects. However, we also notice that for some prompts, subjects mentioned in the input text not present in the generated object of the ensemble model that was otherwise present when just using ViT-B/32 or ViT-B/16 for guidance. For VoXImp- the model trained with ViT-B/32 had the highest performance. Qualitatively, objects generated by VoX trained with ViT-B/16 can generate more detailed geometry. In Dream Fields, their ViT-B/32 model also outper- formed their ViT-B/16 model in terms of R- R-Precision.
CLIP R-Precision While the R-Precision is correlated with the faithfulness of the generated object with respect to the prompt. higher scores do not always yield the best qualitative results. The R- Precision score itself is also prone to adversarial examples. This suggests that quantitative evaluation for Text-to-3D models is still very challenging and better metrics required.
A.3 VOXIMP AUGMENTATION ABLATION
We show the augmentation ablation for the VoXimp model with KL loss in Fig. 8 Overall we find that compared to Voxesp the VOXimp model is less noisy even in cases where less augmentations are There seem to be unwanted background textures when just Diff and Back augmentations are on. Since the perspective augmentation can sometimes hurt the texture sharpness we turn it off for Vox as it seems unnecessary.
15

---

Under review as a conference paper at ICLR 2023
A.4
OPENCLIP MODEL RESULTS
OpenCLIP VIT-B/32
OpenCLIP ViT-B/16
CLIP ViT-B/16
"eedteval people celebrating festival stolls; trending on artstation.
fontasy costlej trending artstation."
souts dark fontasy cityscope; trending on ortstation.
"s "stempu city; trending on grtstution. Lon."
"Tokyo city; trending artstation.
Figure 9: In this figure we test out OpenCLIP's model trained on the LAION dataset as guidance with the Voximp model. We can see that there is not one model that performs the best across all the prompts.
In this section tested out another implementation of CLIP, namely OpenCLIP IIharco et al. (202I) which was trained on the LAION Schuhmaan et al. (202I) dataset. Particularly we use the VTT-B/32 variant trained on LAION-2B and the VIT-B/I6 variant trained on LAION-400M, We use the VoXimp model and the image resolution is set to 224 during training. Our results are shown in that there is not model that performed the best across the board. For some prompts, all three models generated coherent results and the variation between them can be desirable for creative applications. would be interesting explore further how varying datasets as well as size can have an impact on the quality and diversity of the generated results.
Fig. 9 Overall,
A.5 RESOURCE USAGE
Tab, the memory usage and time consumption is shown across the devices we conducted training All of our model variations consumed less memory compared to Dream Fields and was faster to train. Their model takes a lot of memory because of the 8-layer MLP with residual connection used for the NeRF model. Since mostly used RTX A6000s with 48 GB of VRAM we didn't 't optimize hyperparameters such as voxel grid resolution much unless we ran out of memory. For code release will provide both VOXesp and VOX model settings that will worth noting that the VOXtmp model trained image resolutions 1682 and 224 consumes about the same amount memory. This because evaluate the forward pass on the entire (PE) grid and since both variations has the same voxel grid resolution their memory consumption is about the same. We plan to update this the future so that regions of the voxels that are not hit by rays will be masked out from the calculation of the forward pass. We were also not able to run Dream Fields higher resolutions in their paper because of availability issues with TPU v3-8.
more commodity GPUs. It
16

---

Under review as a conference paper at ICLR 2023
Table 6: In this table we show the memory consumption as well as training time per prompt for various resolution and training iteration settings in comparison with Dream Fields across the devices that we used.
Method
Res + Iter
Device
Memory
Time
VOXgxp
(ViT-B/32)
168 + 10k
RTX 2080 Ti
7.01 GB
21 min
VoXesp
(ViT-B/32)
224² + 15k
RTX 2080
9.76 GB
35 min
VoXesp (ViT-B/16)
1682 + 10k
RTX A6000
12.39 GB
42 min
VOXexp
(VIT-B/16)
2242+155)
RTX A6000
14.92 GB
~ hr 6 min
Voxexp (ViT-L/14)
224 224²+151 15k
RTX A6000
21.48 GB
I hr 54 min
VOXesp
VVTT-1/32+
224 15kk
RTX A6000
33.06 GB
~ hr 40 min
VOXe<p ViTT-/16+
2242+155
RTX A6000
35.37 GB
1 hr 54 min
VoXtmp (ViT-B/16)
1682 10k
RTX A6000
41.39 GB
40 min
VoXimp (ViT-B/16)
224 + 15k
RTX A6000
40.75 GB
1 hr 4 min
Dream Fields (ViT-B/16)
1682 10k
RTX A6000
0OM (> 48 GB)
Dream Fields (VIT-B/16)
1682 + 10k
TPU v2-8
~2 hr 17 min
17


Reviews:
[{"review_id": "rkJQaB10St", "reviewer": "Reviewer_XGe4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a systematic study on several design choices for pure clip guided text to 3D NeRF generation. It shows how different image augmentations improves the results. Different CLIP architectures are compared and it is observed that the ensemble of CLIP models performs better. Authors also show that implicit grid model parameterized by MLPs perform better than explicit grid. Additionally, this work introduces a spherical prior that facilitate training in this task. The final model achieves satisfactory grid synthesis using CLIP guidance only, which outperforms previous methods.", "summary_of_the_review": "This paper, by studying a number of design factors, finds a good setting that achieves impressive 3D object synthesis from CLIP guidance only. The performance is much better than previous methods. However, the technical novelty is somewhat limited and the quantitative study is weak. Considering the impressive results and that this work focus on a systematic study, the novelty is not a big issue and I am inclined to accept this paper. But I suggest adding more quantitative evaluation to make the results more solid.", "strength_and_weaknesses": "Strength:\n\n- This paper presents a systematic study on several design choices for pure clip guided text to 3D NeRF generation, which has not been done before.  In general, the study summarizes several techniques that would stabilize training and prevent adversarial results. The experimental results would serve as a useful guidance in related tasks.\n\n- Combining several designs, the final model achieves impressive 3D grid synthesis using CLIP guidance only. Compared to DreamField, it achieves more coherent results with higher memory efficiency and faster training speed.\n\n\nWeaknesses:\n\n- As a work aiming for \"systematic study\", the quantitative study is weak. Quantitative results are only provided for the final models but not for ablation study. It is unclear how consistent is the ablation study as only a few qualitative examples are provided. Thus, I suggest also providing quantitative results for the ablation studies on data augmentations and backbones.\n\n- The design choices studied in the paper (different augmentations, backbones, voxel grids) are simply some common techniques that exist before and there is not much new techniques in the paper. The results are also not surprising, as it is intuitive that augmentations, model ensemble, and implicit fields can improve results.\n\n- (Minor) For the implicit model, it is not explained why it is used to model only grid coordinates instead of continuous coordinates like NeRF. My guess is that this is more efficient, but it's better to mention it.\n\n- Typo:  \nPage 5 line 7: ',' -> '.'  \nSection 6 line 7: 'effect' -> 'affect'", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MhWNrihN3t", "reviewer": "Reviewer_hRKS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work conducts a systematic study of augmentations and their effect on text to 3D generation results with pure CLIP guidance.\nThey compare different CLIP backbones for guidance as well as model ensembles for finer 3D object detail, and compare the regularization effects on geometry of explicit vs implicit voxel grids.\nThey demonstrate generation of high-resolution grids using CLIP guidance only.", "summary_of_the_review": "As mentioned above, some deeper analysis and theoretical explanation are expected for the experiment results.\nSome implementation is not clear.", "strength_and_weaknesses": "Strength:\nThis paper is well written.\n\nWeaknesses:\n1. The technical contribution is very limited in this work. \nThey explores the effect of different augmentations, clip backbone, scene representation forms, which may contribute to community.\nHowever, only simply giving the conclusion & findings is not enough, some deeper analysis and theoretical explanation for the findings are also expected, or the paper will tend to be an experiment report instead of research paper. For instance, why KL loss is essential when augmentations are turned on in Fig. 3?\n\n2. Some implementation is not clear. For the KL divergence computation, eq. (4) and (5) are given but I cannot understand eq. (5) for the KL divergence computation (if the formulation is correct?) and it is also not clear how (5) is computed or optimized.\n\n3. The improvement compared with Dream Fields tends to be incremental as shown in Fig. 6.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2QJJ7GOAjc", "reviewer": "Reviewer_3Rq7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of using CLIP for text-guided 3D shape generation. The authors experimented on different image-based augmentations, different CLIP backbones and analyzed their influences toward final results. They also studies the regularization effects of explicit vs. implicit voxel grid models. Results shown that by combining those design considerations, this paper demonstrated better visual quality with higher memory efficiency and less training time.", "summary_of_the_review": "I think this paper offers a great study for text-to-3D synthesis by studying the CLIP architecture, 3D representation and image augmentation, and the results outperforms quite a lot upon previous methods. The only thing I am skeptical is the technique contributions and overall contribution to the research community. Therefore, currently I think currently it is below the acceptance threshold. But if the authors provides more clarification on my questions, I am willing to discuss and change my rating.", "strength_and_weaknesses": "Strength:\n- This paper provides a systematic study of the design choices towards the text-to-3D methods, which I think may guide the future work in this topic\n- The authors propose a novel spherical prior for optimization\n- By applying the strategies the authors studies, this paper achieves state-of-the-art results on text-to-3D synthesis\n\nWeaknesses:\nAs a technical paper, I think this paper might lack enough technical contributions, which makes my consider this a \"benchmark paper\". However, in this case, I found the studied a fairly narrow topic - only text-to-3D with CLIP and only on a few aspects like augmentation, CLIP architecture. I think there are many other aspects the authors might consider: synthesizing meshes as included in the related work, other feature encodings like instant-ngp. Also, I am not sure why the authors didn't use the Plenoxel model, which is also a discrete voxel representation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject