# GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space

Souhaib Attaiki, Paul Guerrero, Duygu Ceylan, Niloy J Mitra, Maks Ovsjanikov

► To cite this version:

Souhaib Attaiki, Paul Guerrero, Duygu Ceylan, Niloy J Mitra, Maks Ovsjanikov. GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space. WACV 2025 - IEEE/CVF Winter Conference on Applications of Computer Vision, Feb 2025, Tucson, United States. 10.48550/arXiv.2412.16717. [hal-05440251]

HAL Id: hal-05440251

https://hal.science/hal-05440251v1

Submitted on 4 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2412.16717v1 [cs.CV] 21 Dec 2024

# GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space

Souhaib Attaiki¹,*, Paul Guerrero², Duygu Ceylan², Niloy J. Mitra²,³, Maks Ovsjanikov¹

¹LIX, École Polytechnique, IPP Paris, ²Adobe Research, ³University College London (UCL)

* Work done at Adobe Research

https://ganfusion.github.io/

## Abstract

We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image and/or video generative models. State-of-the-art 3D generators are either trained with explicit 3D supervision and are thus limited by the volume and diversity of existing 3D data. Meanwhile, generators that can be trained with only 2D data as supervision typically produce coarser results, cannot be text-conditioned, and/or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast, denoising diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion that starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN, caption them, and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects. We evaluate the proposed method in the context of text-conditioned full-body human generation and show improvements over possible alternatives.

## 1. Introduction

Text-to-image diffusion models that operate in pixel [24] or latent spaces [59] are still relatively recent developments, but the underlying architectures have been studied extensively, resulting in multiple commercial systems across the industry that have given rise to new creative workflows.

A similar success story is yet to be repeated for 3D data. Diffusion-based methods are known to require significant amounts of training data in order to produce high-quality

generative results. Unfortunately obtaining such amounts of data in 3D is still challenging. Thus relatively few such methods have been developed with explicit 3D supervision (e.g. [17, 39, 67, 73] among others).

An attractive alternative is to train 3D generative models with easier-to-obtain 2D data. In this context, one possibility is to use 2D images as a supervisory signal for a 3D generative model [3]. Unfortunately, methods that use this approach tend to generate 3D models with lower quality and/or diversity. A different option is to produce high-quality 3D output by distilling pretrained 2D image diffusion priors [57]. However, such methods require costly optimization, e.g., using Score Distillation Sampling, at inference time, making them slow and difficult to scale. On the other hand, GAN-based methods have shown to be possible to train with only 2D data for supervision, and capable of producing high-quality 3D results efficiently [16]. Unfortunately, conditioning GANs, e.g., on text is far from straightforward since they can be unstable during training, and lack the convenient formalism of score-based (diffusion) generative models, which enable conditioning via classifier or classifier-free guidance [25].

We present GANFusion as the first method that successfully trains a feed-forward text-to-3D diffusion-generator using only single-view 2D data for supervision. We achieve this by combining the power of unconditional GAN-based generation to enable learning from only 2D data with the power of diffusion models to enable text conditioning. Specifically, we first utilize a GAN to learn a category-specific latent space in the form of triplane features [9], along with the corresponding triplane decoder. This stage requires only single-view 2D supervision. We then distill the extracted GAN latent space into a text-conditioned diffusion model by using the unconditional GAN to generate triplane samples, generating corresponding text captions, and training a text-to-triplane diffusion model on this dataset. Crucially, our method is feed-forward, in that it completely avoids test-time optimization, e.g., with
<figure><img src="image_2.png" /><figcaption>Figure 1. We propose GANFusion, a text-guided feed-forward 3D generator that is trained with only single-view image supervision. Unlike previous methods such as AG3D [16] which do not enable text conditioning, GANFusion can be conditioned on text while still achieving high generation quality compared to other text conditioned generators such as RenderDiffusion [3]. We note that GANFusion, unlike SDS-based optimization methods that use image diffusion priors [57], does not require any test-time optimization.</figcaption></figure>

Score Distillation Sampling, and instead generates results via learned denoising diffusion in the latent (triplane) space. Overall, GANFusion combines the strengths of GANs for efficient training and sample generation with the learned diffusion model, enabling principled and effective conditioning.

We evaluate GANFusion in the context of generating 3D human models. We leverage the power of text-to-image foundational models [59] to create a diverse and large-scale synthetic 2D dataset to demonstrate our method. Specifically, we render pose and depth condition images from a deformable human template model [45] and map these to realistic and high-quality images conditioned on procedurally generated text prompts via a pre-trained text-to-image model. This enables to create a high-quality dataset that is much larger compared to those commonly used in prior work [16, 26]. We compare our approach to possible baselines that can operate with only 2D data for training but leave out methods requiring access to 3D object meshes. We also evaluate our method for generating real-world faces and cats using the FFHQ and AFHQ dataset [9], as well as generating realistic 3D people using the DeepFashion dataset [44], and demonstrate its compatibility with two GAN architectures, AG3D [16] and EG3D [9].

We are unaware of any other generative method that supports feed-forward text-conditioned 3D generation and can be trained with only 2D data. Fig. 1 shows results obtained using our approach compared to the most closely related baselines. Our method is the only one that can generate high quality *text-guided* 3D models, while avoiding test-time optimization or explicit 3D supervision.

In summary, our key contributions are as follows:

1. We highlight the complementary strengths of GANs and denoising diffusion models for 3D generation. The former can be trained effectively with 2D supervision, while the latter allows *guided* generative modeling in the space of learned features.

2. We propose a novel framework that combines the aforementioned complementary strengths to present a text-guided diffusion-based 3D generator, trained using only single view 2D supervision.

3. We demonstrate that the resulting approach produces high-quality output and avoids costly test-time optimization, outperforming strong recent baselines.

## 2. Related Work

**3D generation with 3D data** Following the success of 2D generation models in high quality image generation [34,59], there have been various attempts to learn 3D generation models. A wide variety of methods have been proposed using 3D data as supervision. Different generator architectures such as GANs [40, 70], normalizing flows [72], and more recently diffusion models [63,73] have been extended to the 3D domain. Unlike images that are universally represented as 2D pixel arrays, 3D data can be represented in various forms. Hence, 3D generators have also explored different options such as point clouds [1, 5], voxels [68], implicit representations [55], and radiance fields [4, 17]. While promising, these methods have been limited because the amount of 3D data is orders of magnitude less than the image counterparts (despite recent datasets such as Object-verse [14]).

**3D generation with 2D data** To tackle the 3D data scarcity problem, a parallel line of work focuses on leveraging 2D data for 3D generation. In this context, 3D-aware GANs [8,15] have been in particular effective where a GAN is used to generate an implicit representation [18, 23, 51, 62, 69] including a NeRF [21], a volumetric field [50], or a tri-plane [9]. These representations are rendered from different viewpoints for which an image-based discriminator is used for adversarial supervision. Our method leverages the power of such unconditional generators in the first stage. However, we demonstrate that it is not trivial to add text
conditioning to such methods and hence we resort to diffu-
sion models in the second stage. More recently, there have
been efforts to train 3D diffusion models using 2D images
only. HoloDiffusion [33] creates feature grids from input
videos which are then used to generate different multi-view
images via a diffusion model. The follow up work, Holo-
Fusion [32], extends this setup with a superresolution mod-
ule. RenderDiffusion [3] generates a triplane representa-
tion from a single image using reconstruction losses. While
promising, such diffusion based 3D generators trained on
2D data only have not reached the quality of their GAN
counterparts. Hence, our method leverages a 3D-aware
GAN to generate 3D samples for training a text-conditioned
diffusion based generator.

**Text-to-3D generation** Breakthrough advances such as CLIP [58] have proposed ways to obtain common latent spaces between text and image modalities, enabling text-guided generation and editing applications. Several works [19] have utilized loss functions in the CLIP embedding space directly for text-guided 3D generation and editing. The seminal work of DreamFusion [57] distills a text-to-image model to generate 3D objects using Score Distillation Sampling (SDS), with follow-ups improving quality [10, 41] and optimization efficiency [46]. Other work [20] also leverages a GAN to obtain a latent triplane representation, but requires test-time optimization for text conditioning. Despite the advances, these approaches still require a per-prompt optimization step, which can be costly thus limiting their efficiency and scalability. Another line of work [27, 42, 43] generate multi-view images that can be reconstructed into 3D objects without optimization, by fine-tuning text-to-image models. However, fine-tuning requires synthetic 3D data that introduces bias and reduces the generality of these methods.

**Human-specific 3D generation** In the context of hu-
mans, statistical template models such as SMPL [45] can
be considered as early examples of 3D generators, which
are learned from collections of human scans with mini-
mal/tight clothing. For with clothing, Cape [47] trains a
graph CNN architecture from scans of clothed humans.
With the increasing effectiveness of implicit representa-
tions, follow up work has presented neural implicit gen-
erators [11, 53, 54]. More recently, in an impressive ef-
fort, Rodin [67] has presented a diffusion-based architec-
ture to generate text-conditioned humans. The above mod-
els, however, require access to 3D human data, and un-
fortunately, existing datasets [6, 12] are limited in quantity
and diversity. Hence, methods like Get3DHuman [71] pro-
pose using pseudo ground truth labels obtained from sin-
gle image human reconstruction methods to overcome the
data challenge. In another line of work, researchers [16, 65]
have trained 3D aware GANs specifically on human image

datasets. While showing impressive results, these methods
are not straightforward to extend to accept new guidance
or conditioning, e.g., via text prompts, due to the known
stability issues in GAN training. Hence, our method takes
a different route and utilizes the knowledge captured in
such a GAN framework, i.e., AG3D [16], to train a text-
conditioned diffusion model. Finally, there has been a series
of efforts [28, 30] that extend the SDS based optimization
approaches to avatar generation. Such methods utilize a de-
formable template body such as SMPL [35] or GHUM [36]
to regularize the optimization process. More recently, ap-
proaches that also exploit additional conditioning strate-
gies for the base image generation model, such as Control-
Net [7, 74], have been proposed. Our method also takes ad-
vantage of a ControlNet architecture to generate a large set
of human images. However, we use these images to train
a *feed forward* generation model instead. In a concurrent
effort, DiffusionGAN3D [37] performs domain adaption of
a 3D aware GAN (e.g., EG3D [9]) using SDS loss and en-
ables text conditioning by searching for a latent code in the
latent space at test time. In contrast, our method directly
uses the GAN latent space to train a text-guided diffusion
model and avoids any test time optimization.

**3D generation in a learned latent space** Finally, we note that our approach is related to existing methods that first pre-train a latent representation for 3D data and then train diffusion models in this learned space, e.g., [13, 17, 22, 49, 52,63,73]. However, these approaches are based on an *auto-encoding strategy* and build the latent space using explicit 3D or multi-view supervision. Instead, we only use unorganized *single view 2D data* without camera poses for supervision. This is a significantly harder problem as it lacks explicit 3D supervision, but at the same time allows us to greatly enlarge the training corpus. Overall we demonstrate that our approach leads to an efficient *feed-forward* text-guided 3D generative model that does not require *any* 3D training data.

**3. Motivation and Overview**

Our approach is motivated by several considerations.
First, as mentioned above, our key objective is to design
a method capable of generating text-conditioned 3D geom-
etry, while using only 2D images for supervision. Further-
more, we aim to avoid costly test-time optimization, e.g.,
via Score Distillation Sampling, and instead perform syn-
thesis through standard denoising diffusion at inference. A
key question when applying denoising diffusion is decid-
ing in *which space* diffusion is performed. This question is
especially prominent in a setup like ours since our supervi-
sory signal (2D images) differs from the target output (3D
geometry), while denoising diffusion is typically performed
in some fixed space.
<figure><img src="image_3.png" /><figcaption>Figure 2. **Method Overview.** We train a feed-forward text-to-3D diffusion model in two stages. First, we train an unconditional GAN-based 3D object generator like AG3D [16] with a single-view image dataset. 3D objects are represented as triplanes and rendered with a renderer followed by an upsampler. We generate a large set of triplanes and caption them using BLIP [38]. The resulting (triplane, caption) dataset is then used to train a text-to-3D diffusion model, effectively distilling the GAN generator into a diffusion model, while also allowing for text conditioning. Triplanes generated by the text-to-3D model are rendered using the renderer and upsampler trained in the first stage.</figcaption></figure>

One general possibility is to train an image-based encoder to produce latent features, from which 3D geometry can be generated while performing denoising diffusion in this latent space. Perhaps the most natural option would be to train the entire network end-to-end, with both the feature encoder-decoder as well as the latent denoising networks using, e.g., a rendering-based loss. This option was recently explored in RenderDiffusion [3] where a denoising network learns to simultaneously denoise input images, encode tri-plane features that are used to generate 3D geometry, and decode triplane features for rendering. Unfortunately, as we demonstrate in Sec. 5, we observe that such an approach has limited accuracy, which we attribute to three key reasons: (i) training *both* denoising diffusion *and* triplane encoding tends to be error-prone, given the very significant degrees of freedom; (ii) due to the mismatch in the input (images) and output (triplane features), the UNet architecture cannot exploit skip-connections, forcing the denoising network to operate purely on the bottleneck representation, thus significantly limiting the quality of the output; and finally, (iii) input and rendered viewpoints need to match, making it possible for this approach to overfit to the input viewpoints and produce triplanes that are lower quality when rendered from other viewpoints.

To overcome these challenges, we combine the strengths of two major approaches. First, following [16] we use a GAN to train a latent triplane-based representation using single view 2D images as supervision, but from which 3D geometry can be generated. Importantly, training a GAN does not require the input and output to be in the same space. Moreover, it also does not assume access to rendering viewpoints, which it could potentially overfit to, making it well-adapted for our scenario.

On the other hand, *conditioning* GANs with text is

known to be challenging, owing in part to the lack of interpretation via score matching enjoyed by diffusion models, which enables, e.g., classifier-free guidance [25]. Thus, after training the GAN as a triplane generator, we train a separate diffusion model operating on a set of generated triplanes.

This two-stage training with a GAN and a diffusion model allows text conditioning and leads to high-quality results. In the following sections, we provide the details of our method and compare it to strong recent baselines.

## 4. Method

As mentioned above, our goal is to train a feed-forward generator for 3D objects using single-view 2D images $\mathcal{I} = \{I_1, \dots, I_N\}$ as supervision, with $I \in \mathbb{R}^{3 \times H \times W}$. We define 3D objects as neural fields that encode densities and colors, and use triplanes [9, 56] as representation for these fields. This representation is described in Sec. 4.1.

To reach our objective, we proceed in two training stages (see Fig. 2): First, we train a GAN as an *unconditional* generator for triplane representations of 3D objects, using the 2D image dataset $\mathcal{I}$ as supervision. Previous work has shown that class-specific GANs can be trained successfully to generate 3D objects with only 2D supervision [9, 16]. We use the architecture proposed in AG3D [16] for this step. This unconditional generator produces high-quality 3D objects, but we found it hard to introduce text conditioning into a GAN architecture, due to the inherently unstable training process. This observation is supported by the absence of text-conditioned GANs in recent literature, with the exception of a few carefully tuned 2D GANs [31, 60, 75]. Using the trained GAN, we then create a large dataset of 3D objects and caption them using BLIP [38]. See Sec. 4.2 for
details on the first stage.

In the second stage, we train a text-conditioned diffusion model on the dataset generated in the first stage, using an architecture based on StableDiffusion [59]. We remove the encoder, as our 3D objects have already been encoded into triplanes by the GAN, and carefully normalize the triplanes. We found that training the diffusion model with 3D objects encoded via triplane features, as supervision, rather than using 2D supervision as in RenderDiffusion [3], is essential to obtain high-quality outputs. We show a comparison to a RenderDiffusion-based setup in Sec. 5. We describe the details of the second stage in Sec. 4.3.

## 4.1. 3D Object Representation

We define 3D objects as neural fields $s$, consisting of an RGB color field representing the albedo, and a pseudo Signed Distance Function (SDF) representing the geometry of the 3D object. The neural field can be queried at a position $x \in \mathbb{R}^3$ to give a pseudo-SDF value $d$ and an albedo color $c$ at that position: $(d, c) = s(x)$.

We use a triplane representation [9, 56], consisting of 3 orthogonal 2D feature grids $T_{xy}$, $T_{xz}$, $T_{yz}$, one for each coordinate plane, with $T_{**} \in \mathbb{R}^{n \times h \times w}$. The three feature grids can conveniently be concatenated into a single multi-channel image $T \in \mathbb{R}^{3n \times h \times w}$. A feature $T(x) \in \mathbb{R}^{3n}$ for a given query point $x$ is obtained by projecting $x$ to each coordinate plane, querying the corresponding feature grid with bilinear interpolation, and concatenating the three resulting feature vectors. This feature is converted to a pseudo-SDF and color value using an MLP $h_{\phi}$ with parameters $\phi$ as decoder:

$$ (d, c) = s(x) := h_{\phi}(T(x)). \quad (1) $$

**3D object rendering** We use a volumetric renderer based on previous work [9, 16, 48]. Samples are placed along each camera ray using a two-pass strategy: stratified sampling followed by importance sampling. Samples along a ray $v$ are then accumulated as:

$$ \mathbf{c}^v = \sum_i \mathbf{c}_i^v a_i^v \prod_{j<i} (1 - a_j^v), \quad (2) $$

$$ \text{with } a_i^v = 1 - \exp(-\sigma_i^v \delta_i^v), $$

$$ \sigma_i^v = \text{sigmoid}(d_i^v), $$

$$ (d_i^v, \mathbf{c}_i^v) = h_{\phi}(T(x_i^v)). $$

The sample locations $x_i^v$ along ray $v$ are indexed from the camera outwards. At each sample location, $\sigma_i^v$ are computed from the pseudo-SDF values $d_i^v$ using a sigmoid function, and $\delta_i^r$ is the local spacing between samples along the ray. For convenience, we also define a rendering function $r$ that outputs a rendered RGB image given a triplane $T$ and camera parameters $C$:

$$ r_{\phi}(T, C) = [\mathbf{c}^v]_{v \in \mathcal{V}_C}, \quad (3) $$

where $\mathcal{V}_C$ is the set of rays for camera $C$ and the square brackets denote concatenation and reshaping into an image $r_{\phi}(T, C) \in \mathbb{R}^{3 \times H \times W}$. Note that the rendering function depends on the learned parameters $\phi$ of the decoder $h_{\phi}$.

## 4.2. Training an Unconditional GAN for 3D Objects

Given a latent vector sampled from the standard normal distribution $z \sim N(0, 1)$, the unconditional GAN $g_{\theta}$ creates a triplane representation of a 3D object: $T = g_{\theta}(z)$.

**Architecture** We use an architecture based on AG3D [16]. It consists of a generator $g_{\theta}$, a renderer $r_{\phi}$, an upsampler $u_{\rho}$, and a discriminator $q_{\psi}$. The generator and discriminator are based on StyleGAN2 [34] and the upsampler on EG3D [9].

**Rendering** To render a triplane $T$, we use the renderer $r_{\phi}(T, C)$. As high-resolution renders have prohibitive time and memory requirements, we render in low resolution $(128^2)$ and use the upsampler $u_{\rho}$ to get higher-resolution renders $(256^2)$. The upsampler uses two inputs in addition to a low-resolution color render: i) a low-resolution render of triplane features, providing additional information about local details of the 3D object, and ii) a style vector $w$ from the StyleGAN-based generator, providing global information about the entire 3D object:

$$ I^{\text{hi}} = u_{\rho}(r_{\phi}(T, C), r(T, C), w), \quad (4) $$

where $I^{\text{hi}}$ is the high-resolution image. Triplane features $r(T, C)$ are rendered by replacing the color values $c$ in Eq. (2) and Eq. (3) with sampled triplane features $T(x)$. The style $w$ is an intermediate result of the generator $g_{\theta}$, a non-linear transformation of the latent vector $z$.

**Training** The generator, renderer, upsampler, and discriminator are trained jointly with the standard adversarial loss described in StyleGAN2 [34], using single-view 2D training from $\mathcal{I}$ as real images and upsampled renders $I^{\text{hi}}$ from the generated objects as fake images. Random viewpoints are used for the camera parameters $C$.

As demonstrated, e.g., in [16,23] GANs can be trained as 3D generators without 3D or multi-view supervision. The adversarial loss does not require ground truth for any specific render, and instead only requires the distribution of renders in $\mathcal{I}^R$ to match the distribution of images in $\mathcal{I}$. This allows generating objects unconditionally at training time and rendering them from arbitrary viewpoints, even if we do not have ground truth for these specific objects or viewpoints.

## 4.3. Text-to-3D Diffusion Model for 3D Objects

We use the GAN generator $g_{\theta}$ to create a dataset of triplanes $T$ and caption them using BLIP [38]. Details are
described in Sec. 5. We then use this dataset of (caption, triplane) pairs to train a text-conditioned diffusion model as our feed-forward text-to-3D generator, effectively distilling the unconditional distribution learned by the GAN into a text-conditioned version of the distribution.

**Triplane pre-processing** Before training on our dataset of triplanes, we normalize the triplanes by subtracting a per-channel mean and dividing by a per-channel standard deviation. Both mean and standard deviation are computed over the full dataset of triplanes. Additionally, we remove outlier values by scaling triplanes with a factor of 1/16 and clamping to [-1, 1].

**Training and Architecture** For each training sample, we randomly pick a triplane $T^0$ and a corresponding caption $Y$ from the dataset. We then add noise to a triplane according to a noise schedule $\alpha_t$:

$$T^t = \sqrt{\alpha_t} T^0 + \sqrt{1 - \alpha_t} \epsilon \quad \text{with } \epsilon \sim \mathcal{N}(0, 1), \quad (5)$$

The noise schedule $\alpha_t$ determines the amount of noise added to the triplane given a time step $t$. We use a Sigmoid noise schedule [29], which has shown good performance on high-resolution images. For each training sample, we randomly pick the time step $t$ with uniform probability in $[0, T]$. Given the noisy image, we train a denoiser network $k_\chi$ with parameters $\chi$ to predict the denoised image using an $L_2$ loss:

$$\mathcal{L}_{\text{diff}} = \|k_{\chi}(T^t, Y, t) - T^0\|_2^2. \quad (6)$$

In addition to the noisy triplane, the denoiser takes as input the text prompt $Y$ and the current time step $t$. We randomly replace $Y$ with empty text with 20% probability to allow for unconditional generation when $Y$ is empty; this is required to support classifier-free guidance at inference time. The denoiser $k_\chi$ is implemented as a UNet with the same architecture as in StableDiffusion [59].

**Inference** Given a triplane sampled from a pure noise distribution $T^T \sim \mathcal{N}(0, 1)$ and a text caption $Y$, we predict a denoised triplane $T^0$ with multiple denoising iterations using the denoiser $k_\chi$. In each iteration, the denoiser outputs a prediction of the denoised image $\hat{T}^t := k_\chi(T^t, Y, t)$. We use this prediction in a deterministic DDIM sampler [64]:

$$T^{t-1} = \sqrt{\alpha_{t-1}} \hat{T}^t + \sqrt{1 - \alpha_{t-1}} \hat{\epsilon}^t, \quad (7)$$

where $\hat{\epsilon}^t$ is the predicted noise:

$$\hat{\epsilon}^t := (T^t - \sqrt{\alpha_t} \hat{T}^t) / \sqrt{1 - \alpha_t}, \quad (8)$$

and $\alpha_t$ is the noise schedule. We additionally use classifier-free guidance [25] with a guidance scale of 7.5 to better align generated triplanes with text prompts.

**Rendering** At inference time, we render triplanes using the renderer $r_\phi$ and upsampler $u_\rho$ that was trained by the GAN. While the upsampler $u_\rho$ requires the style vector $w$ as input, which is not available in this setting, we found that the upsampler is not sensitive to this input. We pass a vector of all-ones, giving us similar upsampling quality.

## 5. Results

**Single-view image dataset.** To obtain a sufficiently large training set of 2D images that we can use both for our method and for baselines, we create the dataset of synthetic images with StableDiffusion 1.5, conditioned on depth maps, pose maps, and prompts. To obtain depth and pose maps, we take the 3D human body model SMPL [45] in A-pose and render a depth map as well as a 2D pose map from a viewpoint sampled around the 3D body. In practice, we use a fixed distance of 2.34 to the origin, a fixed elevation of 40 degrees, and randomly sample an azimuth. To obtain the prompts, we use a procedural approach that is aimed to describe the appearance of a person. In particular, we generate prompts that include a gender keyword, types of upper and lower-body clothing, and their corresponding colors. We select these clothing types and colors from a predefined set, which consists of 28 upper-body clothing types, 12 lower-body clothing types, 11 footwear types, and 23 colors. Given such a prompt and the depth/pose map renderings, we generate the corresponding RGB image resulting in a dataset of 300k images in total. A few examples are shown in Fig. 5. While Stable Diffusion generates a plausible image, we observe that it cannot perfectly follow the text prompt. For example, the generated image might depict a different clothing type or color than instructed in the prompt. While we do not need text prompts matching the images for our method, the baselines we compare to do require matching prompts. Hence, in a second pass, we refine the captions of the generated images using the same BLIP-based [38] captioning we use to caption the generated triplanes (described below). We emphasize that these prompts are not used in the first stage of training of our method. Finally, we also generate an additional set of 1k (prompt, image) pairs that we use for evaluations using the described procedural technique.

**Triplane dataset.** To train the diffusion model in the second stage of our method, we use the GAN generator $g_0$ trained in stage 1 to create 49.5k triplanes. We render these triplanes from random viewpoints following the same viewpoint distribution as used in the 2D dataset and caption them using BLIP-based captioning.

**BLIP-based captioning.** We create captions from images of persons using a visual question-answering approach based on BLIP [38]. We define a set of questions regarding the gender and clothing of a person (for example, "What is the person wearing on their feet?") and also define a set of
<figure><img src="image_4.png" /><figcaption>Figure 3. **Qualitative comparison.** We provide visual results from our method as well as the baselines. We provide random generations from AG3D [16] which is unconditional. While random samples are of high quality, it is not straightforward to extend AG3D to enable text conditioning even with the addition of CLIP based losses during training. RenderDiffusion [3] can follow the text prompts but lacks in terms of quality. GANFusion matches the quality of AG3D while enabling text conditioning. For AG3D and our results, we visualize the results with the 2D upsampler trained jointly with the 3D GAN generator in the first stage of our method.</figcaption></figure>

possible answers for each question for BLIP to choose from (for example, “sneakers”, “loafers”, “boots”, etc.). BLIP returns the best fitting answers for each of the questions, which we use to construct the prompt by filling in blanks in a template. Appendix A provides additional details.

**Baselines.** We compare our method to (i) AG3D [16], which we adopt in the first part of our method. (ii) We attempt to extend AG3D directly with additional text conditioning (AG3D+text), using an approach based on cross-attention layers following the implementation of GIGAN [31]. However, due to GAN training instabilities, we were unable to get a working version of this baseline. To improve stability, we introduce (iii) a modified version of AG3D+text, called AG3DC+text where we encourage adherence to the text prompt by adding a CLIP loss between the text prompt and the rendered generator output instead of conditioning the discriminator with the prompt. (iv) We compare to RenderDiffusion [3] where we train a diffusion network to directly predict the triplane parameters. While training this baseline, given an image from our dataset and the corresponding text prompt, we add noise to it based on a sampled diffusion timestep *t* and predict the final triplane features directly conditioned on the prompt. We then render the triplane from the corresponding viewpoint and define an L1-based reconstruction loss. In this baseline, the triplane decoder is jointly trained with the diffusion model. We train all the baselines on the same dataset as our method. To perform a more fair evaluation of the triplane generation quality, we show results both with and without the image upsampler component of AG3D and our method. Note that

AG3D used a more realistic dataset, as reflected in their results, while we opt for a synthetic dataset to obtain more diverse prompts that better demonstrate text conditioning.

**Metrics.** We use two quantitative metrics. First, the Frechet Inception Distance (FID) to measure the fidelity of the generation results. Specifically, treating our generated image dataset as the real distribution, we use InceptionV3 [66] to obtain features. Second, we use CLIP similarity to measure the text alignment between the input prompts and generations, computed as the cosine similarity between the CLIP embeddings of text prompt and rendered outputs. For both metrics, we render images from random viewpoints, following the same viewpoint distribution used in the 2D dataset.

**Qualitative comparisons.** We show a qualitative comparison to all baselines in Fig. 3 (except the unsuccessful text-conditioned version of AG3D, which we show in Fig. 10). We can see that AG3D produces high-quality results, but being an unconditional generator, it cannot adhere to the input prompts. The CLIP loss in AG3DC introduces minimal prompt adherence but decreases quality slightly. RenderDiffusion better follows the prompt but suffers from low quality and view inconsistency resulting in artifacts and missing parts when viewed from different directions. As discussed in Sec. 3, we attribute the lack of quality to the challenges RenderDiffusion faces when training a single denoiser network to jointly learn the tasks of (i) triplane encoding from a 2D image, and (ii) denoising the 2D image. Our method follows the prompt as closely as RenderDiffusion, but with significantly improved qual-
<figure><img src="image_5.png" /><figcaption>Figure 4. **Qualitative results on the FFHQ dataset [9]**. We replace AG3D [16] with EG3D [9] as the generator in our first stage to effectively enable text-conditioning on real-world face data.</figcaption></figure>

<table><caption>Table 1. **Quantitative comparisons.** We compare our method to the unconditional generator AG3D [16], a version of AG3D where text conditioning is added, and the RenderDiffusion [3] baseline. For the AG3D and our method, we evaluate the generation quality both with (high) and without (low) the use of the upsampler. Our method achieves good generation quality while also enabling text conditioning.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Unconditional</th><th colspan="6">Text Conditioned</th></tr><tr><th>AG3D (low)</th><th>AG3D (high)</th><th>AG3D+text</th><th>AG3DC+text</th><th>RenderDiff.</th><th>GANFusion (low)</th><th>GANFusion (high)</th></tr></thead><tbody><tr><td>FID↓</td><td>104.3</td><td><b>35.8</b></td><td>289.6</td><td>92.0</td><td>135.7</td><td>88.23</td><td><b>68.8</b></td></tr><tr><td>CLIP sim.↑</td><td>×</td><td>×</td><td>0.174</td><td>0.240</td><td>0.263</td><td><b>0.296</b></td><td><b>0.293</b></td></tr></tbody></table>

ity, getting close to the quality of the unconditional AG3D. As RenderDiffusion does not use an image upsampler, we also compare the three methods without upsampling, which are provided in Fig. 6. While this reduces the image detail in both AG3D and our method, both still perform far better than RenderDiffusion. Also, it can be seen that our distillation actually improves the FID compared to AG3D without the upsampler, while, at the same time, enabling text conditioning. We attribute this improvement to the fact our distillation helps to remove low-quality outliers from the distribution. In contrast, since we do not retrain the upsampler it has less benefit after our distillation. In Appendix B, we extend the qualitative comparisons, and include the text-conditioned AG3D.

**Results on FFHQ.** We also tested our method on a real-world human face dataset, FFHQ [9]. For the unconditional GAN training, we adopted the EG3D architecture [9] instead of AG3D. Qualitative results are provided in Fig. 4. Our method successfully enables text conditioning, with a quality similar to the original EG3D paper. This substitution of AG3D with EG3D demonstrates the versatility of our method to use different 3D GAN architectures that are trained with single-view data, as it is agnostic to the methods used in the initial training stage. Additionally, this suggests that our method can handle both synthetic and real-world data. In Appendix B, we provide **additional results** on the AFHQ [9] dataset (real-world cat faces), as well as the DeepFashion [44] dataset (realistic 3D human figures).

**Quantitative comparisons.** Tab. 1 provides a quantitative comparison of all baselines on our test set of 1k images. We divide the table into unconditional methods (AG3D with and without upsampling) and text-conditioned

methods. The results support our findings from the qualitative results: AG3D with upsampler achieves the best quality (low FID), but is unconditional and has, therefore, no prompt adherence compared to the text-conditioned methods. Naively adding text conditioning to AG3D results in training instability, giving both low quality and prompt adherence. Introducing a CLIP loss in AG3DC lowers the quality and only introduces a minimal amount of prompt adherence. RenderDiffusion has higher prompt adherence but low quality, and our GANFusion can combine high quality with high prompt adherence due to our two-stage training strategy. Note that while the CLIP similarity is a cosine similarity and this has a maximum of 1, we cannot reach this maximum, even with perfect prompt adherence, as text embeddings and image embeddings are separated in CLIP space [2]. On the lower bound, we would also not expect the CLIP similarity to drop to 0 without text conditioning, as all images in the dataset show humans and thus have some adherence to the prompt.

## 6. Conclusion, Limitations and Future Work

In this work, we presented a novel text-to-3D generative model, which combines the strengths of GANs and Denoising Diffusion frameworks. Specifically, we demonstrate that the latent features learned with an unconditional GAN (EG3D or AG3D) using only 2D images as supervision can be exploited to train a text-guided denoising diffusion model. The resulting method allows to generate 3D geometry from text without test-time optimization by simply using standard diffusion in the learned triplane feature space. Overall, ours is the first text-guided feed-forward 3D
generative model trained only with unstructured 2D data.

While our approach breaks new ground in efficient text-
guided 3D synthesis, it comes with several limitations.
First, any error in training the GAN will be inherited by
the denoising diffusion network, which assumes a fixed la-
tent space. Secondly, our current model works in a par-
ticular category, such as humans (or faces), and it would
therefore be interesting to consider a general-purpose text-
guided 3D generator, capable of producing geometry across
arbitrary classes or scenes. It will also be interesting to in-
vestigate other ways to condition the output, apart from text,
using, e.g., directly providing depth or specific appearance,
material or shape pose properties. Finally, exploring other
encodings, apart from the triplane features as done in our
work, and using pre-trained denoising diffusion networks,
will potentially help to bridge the gap in terms of quality of
output between the best image and 3D generative models.

**Acknowledgements** The authors would like to thank the anonymous reviewers for their valuable suggestions. Parts of this work were supported by the ERC Consolidator Grant No. 101087347 (VEGA) and the ANR AI Chair AI-GRETTE.

References

[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In *ICML*, 2018. 2

[2] Pranav Aggarwal, Hareesh Ravi, Naveen Marri, Sachin Kelkar, Fengbin Chen, Vinh Khuc, Midhun Harikumar, Ritiz Tambi, Sudharshan Reddy Kakumanu, Purvak Lapsiya, Alvin Ghouas, Sarah Saber, Malavika Ramprasad, Baldo Faieta, and Ajinkya Kale. Controlled and conditional text to image generation with diffusion prior, 2023. 8

[3] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Rend diffusion: Image diffusion for 3d reconstruction, inpainting and generation. In *CVPR*, pages 12608–12618, 2023. 1, 2, 3, 4, 5, 7, 8

[4] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Josh Susskind. Gaudi: A neural architect for immersive 3d scene generation. In *NeurIPS*, 2022. 2

[5] Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In *ECCV*, 2020. 2

[6] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu. HuMMan: Multi-modal 4d human dataset for versatile sensing and modeling. In *ECCV*, pages 557–577. Springer, 2022. 3

[7] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. *arXiv preprint arXiv:2304.00916*, 2023. 3

[8] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In *CVPR*, 2020. 2

[9] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 16123–16133, 2022. 1, 2, 3, 4, 5, 8, 12, 14

[10] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. *arXiv preprint arXiv:2303.13873*, 2023. 3

[11] Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avatars. In *CVPR*, 2022. 3

[12] Wei Cheng, Ruixiang Chen, Wanqi Yin, Siming Fan, Keyu Chen, Honglin He, Huiwen Luo, Zhongang Cai, Jingbo Wang, Yang Gao, Zhengming Yu, Zhengyu Lin, Daxuan Ren, Lei Yang, Ziwei Liu, Chen Change Loy, Chen Qian, Wayne Wu, Dahua Lin, Bo Dai, and Kwan-Yee Lin. Dna-rendering: A diverse neural actor repository for high-fidelity human-centric rendering. *arXiv preprint*, arXiv:2307.10173, 2023. 3

[13] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 2262–2272, 2023. 3

[14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In *CVPR*, pages 13142–13153, 2023. 2

[15] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In *CVPR*, 2022. 2

[16] Zijian Dong, Xu Chen, Jinlong Yang, Michael J. Black, Otmar Hilliges, and Andreas Geiger. AG3D: Learning to generate 3D avatars from 2D image collections. In *ICCV*, 2023. 1, 2, 3, 4, 5, 7, 8, 12, 14, 15

[17] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In *ICCV*, 2023. 1, 2, 3

[18] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In *NeurIPS*, 2022. 2

[19] William Gao, Noam Aigerman, Groueix Thibault, Vladimir Kim, and Rana Hanocka. Textdeformer: Geometry manipulation using text guidance. In *ACM Transactions on Graphics (SIGGRAPH)*, 2023. 3
[20] Jiatao Gu, Qingzhe Gao, Shuangfei Zhai, Baoquan Chen, Lingjie Liu, and Josh Susskind. Learning controllable 3d diffusion models from single-view images. *arXiv preprint arXiv:2304.06700*, 2023. 3

[21] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In *ICLR*, 2022. 2

[22] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oğuz. 3dgen: Triplane latent diffusion for textured mesh generation. *arXiv preprint arXiv:2303.05371*, 2023. 3

[23] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In *Computer Vision (ICCV), 2019 IEEE International Conference on Computer Vision*, 2019. 2, 5

[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. *Advances in neural information processing systems*, 33:6840–6851, 2020. 1

[25] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In *NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications*, 2021. 1, 4, 6

[26] Fangzhou Hong, Zhaoxi Chen, Yushi Lan, Liang Pan, and Ziwei Liu. Eva3d: Compositional 3d human generation from 2d image collections. *arXiv preprint arXiv:2210.04888*, 2022. 2

[27] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. *ICLR*, 2024. 3

[28] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies. TeCH: Text-guided Reconstruction of Lifelike Clothed Humans. In *International Conference on 3D Vision (3DV)*, 2024. 3

[29] Allan Jabri, David J. Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In *ICML*, ICML'23, 2023. 6

[30] Ruixiang Jiang, Can Wang, Jingbo Zhang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. In *ICCV*, 2023. 3

[31] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In *CVPR*, 2023. 4, 7

[32] Animesh Karnewar, Niloy J Mitra, Andrea Vedaldi, and David Novotny. Holofusion: Towards photo-realistic 3d generative modeling. In *ICCV*, 2023. 3

[33] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training a 3D diffusion model using 2D images. In *CVPR*, 2023. 3

[34] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In *CVPR*, 2020. 2, 5

[35] Byungjun Kim, Patrick Kwon, Kwangho Lee, Myunggi Lee, Sookwan Han, Daesik Kim, and Hanbyul Joo. Chupa: Carving 3d clothed humans from skinned shape priors using 2d diffusion probabilistic models. In *ICCV*, pages 15965–15976, October 2023. 3

[36] Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Fieraru, and Cristian Sminchisescu. Dreamhuman: Animatable 3d avatars from text. *ArXiv*, abs/2306.09329, 2023. 3

[37] Biwen Lei, Kai Yu, Mengyang Feng, Miaomiao Cui, and Xuansong Xie. Diffusiongan3d: Boosting text-guided 3d generation and domain adaption by combining 3d gans and diffusion priors, 2023. 3

[38] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In *ICML*, pages 12888–12900. PMLR, 2022. 4, 5, 6, 12

[39] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. https://arxiv.org/abs/2311.06214, 2023. 1

[40] Ruihui Li, Xianzhi Li, Ke-Hei Hui, and Chi-Wing Fu. SP-GAN:sphere-guided 3d shape generation and manipulation. *SIGGRAPH*, 40(4), 2021. 2

[41] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In *CVPR*, 2023. 3

[42] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. *NeurIPS*, 36, 2024. 3

[43] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. *ICLR*, 2024. 3

[44] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In *Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2016. 2, 8, 12, 14

[45] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. *ACM Trans. Graphics (Proc. SIGGRAPH Asia)*, 34(6):248:1–248:16, Oct. 2015. 2, 3, 6

[46] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. In *ICCV*, 2023. 3

[47] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to dress 3d people in generative clothing. In *CVPR*, June 2020. 3

[48] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In *ECCV*, 2020. 5

[49] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3D-LDM: Neural implicit 3d shape generation with latent diffusion models. *arXiv preprint arXiv:2212.00842*, 2022. 3
[50] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. *2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)*, pages 2037–2040, 2019. 2

[51] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In *CVPR*, 2021. 2

[52] Evangelos Ntavelis, Aliaksandr Siarohin, Kyle Olszewski, Chaoyang Wang, Luc Van Gool, and Sergey Tulyakov. Autodecoding latent 3d diffusion models, 2023. 3

[53] Pablo Palafox, Nikolaos Sarafianos, Tony Tung, and Angela Dai. Spams: Structured implicit parametric models. *CVPR*, 2022. 3

[54] Pablo Rodríguez Palafox, Aljavz Bovzivc, Justus Thies, Matthias Nießner, and Angela Dai. Npms: Neural parametric models for 3d deformable shapes. In *ICCV*, pages 12675–12685, 2021. 3

[55] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In *CVPR*, June 2019. 2

[56] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In *ECCV*, pages 523–540. Springer, 2020. 4, 5

[57] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. *arXiv*, 2022. 1, 2, 3

[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In *Marina Meila and Tong Zhang, editors, ICML*, volume 139, pages 8748–8763, 2021. 3

[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, June 2022. 1, 2, 5, 6

[60] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-image synthesis. In *ICML*, 2023. 4

[61] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. 13

[62] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In *NeurIPS*, 2020. 2

[63] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In *CVPR*, pages 20875–20886, 2023. 2, 3

[64] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In *ICLR*, 2021. 6

[65] Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu. Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis. *ACM Transactions on Graphics (TOG)*, 41(6):1–10, 2022. 3

[66] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2818–2826, 2016. 7

[67] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrušaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In *CVPR*, pages 4563–4573, 2022. 1, 3

[68] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In *NeurIPS*, NIPS’16, page 82–90, 2016. 2

[69] Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. In *ICCV*, pages 2195–2205, October 2023. 2

[70] Chulin Xie, Chuxin Wang, Bo Zhang, Hao Yang, Dong Chen, and Fang Wen. Style-based point generator with adversarial rendering for point cloud completion. In *CVPR*, pages 4619–4628, June 2021. 2

[71] Zhangyang Xiong, Di Kang, Derong Jin, Weikai Chen, Linchao Bao, Shuguang Cui, and Xiaoguang Han. Get3dhuman: Lifting stylegan-human into a 3d generative model using pixel-aligned reconstruction priors. In *ICCV*, pages 9287–9297, October 2023. 3

[72] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In *ICCV*, 2019. 2

[73] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In *NeurIPS*, 2022. 1, 2, 3

[74] Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang, Li Chen, Chao Long, Feida Zhu, Kang Du, and Min Zheng. Avatarverse: High-quality & stable 3d avatar creation from text and pose, 2023. 3

[75] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In *CVPR*, pages 17907–17917, 2022. 4
# Supplementary Materials

In the following, we provide a detailed description of the BLIP [38]-based captioning process. Additionally, we include further qualitative examples and evaluate on additional datasets to illustrate the performance of our method in comparison to other baselines.

Alongside this document, the project webpage presents a qualitative comparison encompassing a broader set of randomly selected results from our method and all baselines. This webpage also includes videos for several examples.

## A. Additional details for BLIP-based captioning

We ask BLIP [38] the following questions:

* What is the gender of this person?

* What is this person wearing in top?

* What is the person wearing on their lower half?

* What color is this person wearing in top?

* What color is the clothing the person is wearing on their lower half?

* Is the person barefoot?

* What is the person wearing on their feet?

* What is the color of the thing the person is wearing on their feet?

and define a set of possible answers for each question:

* **Gender:** 'man', 'woman'

* **Color:** 'Red', 'Blue', 'Green', 'Yellow', 'Purple', 'Orange', 'Black', 'White', 'Gray', 'Pink', 'Brown', 'Gold', 'Silver', 'Beige', 'Maroon', 'Teal', 'Olive', 'Navy', 'Coral', 'Turquoise', 'Indigo', 'Khaki'

* **Upper body clothing:** 'T-shirt', 'Polo shirt', 'Dress shirt', 'Tank top', 'Crop top', 'Blouse', 'Sweater', 'Hoodie', 'Jacket', 'Coat', 'Blazer', 'Vest', 'Sweatshirt', 'Pullover', 'Cardigan', 'Tunic', 'dress', 'Jumpsuit', 'Romper', 'Suit', 'Pajamas', 'Raincoats', 'Windbreakers', 'Parkas', 'Puffer jackets', 'Trench coats', 'Pea coats', 'Duffle coats'

* **Lower body clothing:** 'Jean', 'pant', 'Sweat-pant', 'Legging', 'Short', 'Skirt', 'Capri', 'Chino', 'Jogger'

* **Footwear:** 'Sneakers', 'Loafers', 'boots', 'heels', 'Flats', 'Sandals', 'Flip flops', 'Espadrilles', 'Ox-fords', 'Clogs', 'socks'

We use the answers chosen by BLIP to fill in the following prompt template:

[Prefix] [Gender] wearing a [Upper Body Color] [Upper body clothing], [Lower body color] [Lower body clothing], [Footwear color] [Footwear].

[Prefix] is randomly chosen from either "a photo of a" or "a full-body photo of a", and [Gender] is selected randomly between the response of BLIP and "person." At test time, we randomly drop some of the items in the prompt template, such as the upper body color and clothing, or the footwear color and clothing.

## B. Additional Results

In this section, additional qualitative and quantitative results are provided.

In Fig. 4, we present qualitative results obtained by training our model on the FFHQ dataset [9]. We also evaluate our model quantitatively by measuring the FID of the generated samples and comparing our results to those of EG3D [9]. Our model achieves an FID score of 49.4, compared to 26.7 achieved by EG3D. While our FID score is higher than that of EG3D, we emphasize that our model is conditioned on text, a capability that EG3D does not possess. We attribute the increased FID to the automatic labeling process used in stage 2. Specifically, the VQA model employed produces a limited set of labels, which restricts the variety of faces learned by our model compared to the full FFHQ dataset. This limitation could be addressed by utilizing a more powerful labeling algorithm.

Beyond learning human faces, we also trained our model to generate realistic cat faces from the AFHQ dataset [9] using the EG3D [9] backbone in stage 1, and realistic 3D human figures from the DeepFashion dataset [44] using the AG3D [16] backbone in stage 1. Qualitative results for these experiments are provided in Fig. 7 and Fig. 8, respectively. These results demonstrate that our model generates high-quality, realistic images while closely following the provided prompts. This highlights the adaptability of our model to multiple domains and its ability to leverage different backbones in stage 1.

Fig. 5 presents a selection of example images from the single-view image dataset we created, alongside their original captions and the refined captions, which were used to train our network.

Since RenderDiffusion does not employ an image upsampler, we also compared the results of the three main baselines without upsampling, as shown in Fig. 6. Although omitting the upsampler reduces image detail in both AG3D
<figure><img src="image_6.png" /><figcaption>Figure 5. **Dataset examples.** We show a few images and corresponding prompts from our single-view image dataset. Prompts that were initially used to generate the images are often not accurate, we refine them with our BLIP-based captioning approach.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6. **Qualitative comparison without upsampling.** We provide visual results from our method as well as the baselines. We disable the 2D upsampler for AG3D and our method and provide renderings of the triplane features directly for all methods.</figcaption></figure>

and our method, both still significantly outperform RendDiffusion.

Fig. 9 showcases a qualitative comparison with all baselines, excluding the unsuccessful text-conditioned version of AG3D, which is presented in Fig. 10. Due to training instability, experiments involving text-conditioning consistently led to mode collapse or other forms of training instabilities. The figure illustrates a case of severe mode collapse.

In Fig. 11 (top row and bottom left), we showcase images produced by our method using identical prompts but different seeds, demonstrating the generation of diverse samples for the same prompt.

Furthermore, our primary objective is to demonstrate the feasibility of our approach using varied prompts that differ in properties such as colors and clothing items, rather than presenting a production-ready model. Generalizing to arbitrary prompts would require large-scale training on datasets like LAION [61], which is outside the scope of this study. Nonetheless, our model exhibits some generalization to out-of-distribution (OOD) prompts, facilitated by the

pre-trained CLIP encoder. For instance, in Fig. 11 (bottom right), our model correctly handles new colors like lavender, which are absent from the training dataset, and supports some degree of grammatical rearrangement in sentences.
<figure><img src="image_8.png" /><figcaption>Figure 7. **Qualitative results on the AFHQ dataset [9].** We replace AG3D [16] with EG3D [9] as the generator in our first stage to effectively enable text-conditioning on real-world cat data.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 8. **Qualitative results on the DeepFashion dataset [44].** We use AG3D [16] as the generator in our first stage to effectively enable text-conditioning on realistic 3D human figures.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 9. **Qualitative comparison.** We provide visual results from our method as well as the baselines.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 10. **Text-conditional AG3D.** We attempt to add text conditioning to AG3D [16] by providing text embeddings as additional input both to the generator and the discriminator. However, we find that the training is not stable and does not converge.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 11. **Top & Bottom left:** Diverse generations produced by our model using the same prompt but different seeds. **Bottom right:** Images showcasing our model's ability to generalize to some out-of-distribution prompts not encountered during training.</figcaption></figure>