Text-guided 3D Human Generation from 2D Collections
Authors: Tsu-Jui Fu
Wenhan Xiong
Yixin Nie
Jingyu Liu
Barlas Oguz
William Yang Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Text-guided Visual Generation
3D Human Generation
TL;DR: Compositional Cross-modal Human (CCH) to fuses compositional human rendering with fashion description and achieves Text-guided 3D Human Generation (T3H)

Abstract: 3D human modeling has been widely used for engaging interaction in gaming, film, and animation. The customization of these characters is crucial for creativity and scalability, which highlights the importance of controllability. In this work, we introduce Text-guided 3D Human Generation (T3H), where a model is to generate a 3D human, guided by the fashion description. There are two goals: 1) the 3D human should render articulately, and 2) its outfit is controlled by the given text. To address this T3H task, we propose Compositional Cross-modal Human (CCH). CCH adopts cross-modal attention to fuse compositional human rendering with the extracted fashion semantics. Each human body part perceives relevant textual guidance as its visual patterns. We incorporate the human prior and semantic discrimination to enhance 3D geometry transformation and fine-grained consistency, enabling it to learn from 2D collections for data efficiency. We conduct evaluations on DeepFashion and SHHQ with diverse fashion attributes covering the shape, fabric, and color of upper and lower clothing. Extensive experiments demonstrate that CCH achieves superior results for T3H with high efficiency.

Text-guided 3D Human Generation from 2D Collections
Tsu-Jui Fu', Wenhan Xiong', Yixin Nie² Jingyu Liu', Barlas Oguz', William Yang Wang'
UC Santa Barbara Meta
(tsu-juifu, william}@cs. edu {xwhan, ynie, jingyuliu, bar jarlaso]@meta com.mttccoommmmmm......mommmmm..
Abstract
The man sporting short- sleeved cotton t-shirt and pure-colored medium shorts
The lady wearing graphic- potterned tonk top, paired with three-point denim pants
3D human modeling has been widely used for engaging interaction in gaming, film, and ani- mation. The customization of these characters crucial for creativity and scalability, which highlights the importance of controllability. In this work, we introduce Text-guided 3D Human Generation (T 3H), where a model to generate 3D human, guided by the fashion description. There are two goals: 1) the 3D human should render articulately, and 2) its outfit controlled by the given text. To address this T3H task, we propose Compositional Cross-modal Human (CCH). CCH adopts cross-modal attention to fuse compositional human rendering with the extracted fashion semantics. Each human body part perceives relevant textual guidance its visual patterns. We incorporate the human prior and semantic discrimination to enhance 3D ge- ometry transformation and fine-grained consis- tency, enabling this to learn from 2D collections for data efficiency. We conduct evaluations DeepFashion and SHHQ with diverse fashion attributes covering the shape, fabric, and color of upper and lower clothing. Extensive experi- ments demonstrate that CCH achieves superior results for T3H with high efficiency.
The woman wearing floral-patterned long-sleeved sweater and long denim pants
She dressed solid- colored sleeveless shirt with long cotton trousers
Figure 1: Text-guided 3D Human Generation (T3H).
the 3D human model, it will significantly improve controllability and meet the considerable demand. We thus introduce Text-guided 3D Human Gen- eration (T3H) to generate 3D human with the cus- tomized outfit, guided via the fashion description. Previous works (Kolotouros et al., 2019; Gao et al., 2022) depend on multi-view videos to learn 3D hu- modeling, but these data difficult to obtain and are not language controllable. Text-to-D (Jain et al., 2022: Poole et al., 2023) has shown attractive 3D generation results through the success of neural rendering (Mildenhall et al., 2020). However, these methods apply iterative inference optimization by external guidance, which is inefficient for usage.
1 Introduction
Our world is inherently three-dimensional, where this nature highlights the importance of 3D applica- tions in various fields, including architecture, prod- uct design, and scientific simulation. The capability of 3D content generation helps bridge the gap be- tween physical and virtual domains, providing an engaging interaction within digital media. Further- more, realistic 3D humans have vast practical value, especially in gaming, film, and animation. Despite enhancing the user experience, the customization of the character is crucial for creativity and scalability. Language is the most direct way of communication. If a system follows the description and establishes
To tackle these above issues, we propose Com- positional Cross-modal Human (CCH) learn T3H from 2D collections. CCH divides the human body into different parts and employs individual volume rendering, inspired by EVA3D (Hong et al., 2023).
Project website: https:// htops://text-3000 adh..githu..f io

---

We extract the fashion semantics from the descrip- tion and adopt cross-modal attention fuse body volumes with textual features, where each part can learn to perceive its correlated fashion patterns. To support various angles of view, CCH leverages the human prior (Bogo et al., 2016) to guide the geome- try transformation for concrete human architecture. Then these compositional volumes can jointly ren- der 3D human with the desired fashion efficiently. The semantic discrimination further considers com- positional distinguishment over each human part, which improves the fine-grained alignment with its description through adversarial training.
training (Goodfellow et al., 2015; Reed et al., 2016) to produce images (Xu et al., 2018; El-Nouby al., 2019; Fu et al., 2020, 2022c) or videos (Marwah et al., 2017; Li et al., 20: Fu et al., 2022b) condi- tioned on given descriptions. With sequential mod- cling from Transformer, vector quantization (Esser et al., 2021) can generate high-quality visual con- tent as discrete tokens (Ramesh et al., 2021; Ding et al., 2021; Fu et al., 2023). The denoising dif- fusion framework (Ho al., 2020; Ramesh et al., 2022; Saharia et al., 2022; Feng et al., 2023) gains much attention as its diversity and scalability via large-scale text-visual pre- mre-trainng. Beyond images and videos, 3D content creation is more challeng- ing due to the increasing complexity of the depth dimension and spatial consistency. In this paper, we consider text-guideds 3D human generation (T3H), which has vast applications in animated characters and virtual assistants
We perform experiments on DeepFashion (Liu et al., 2016; Jiang et al., 2022) and SHHQ (Fu et al., 2022a), which contain human images with diverse fashion descriptions. The patterns include various types of shapes (sleeveless, medium short, long, etc.), fabrics (demim, cotton, furry, etc.), and colors (floral, graphic, pure color, etc.) for the upper and lower clothing, To study the performance of T3H, we conduct thorough evaluation from both visual and semantic aspects. We treat overall realism, ge- ometry measure, and pose correctness to assess the quality of generated 3D humans. For the alignment with the assigned fashion, we apply text-visual rel- evance from CLIP and fine-grained accuracy by a trained fashion classifier.
3D Generation. Different representations have been explored for 3D shapes, such as mesh (Gao et al., 2019; Nash et al., 2020; Henderson et al., 2020), voxel grid (Tatarchenko et al., 2017; Li et al., 2017), point cloud (Li et al.. 2018a; Yang et al.. 2019; Luo et al., 2021), and implicit field (Chen and Zhang, 2019; Park et al., 2019; Zheng et al., 2022). Neural Radiance Field (NeRF) (Mildenhall et al., 2020; Barron al., 2022; Muller et al., 2022) has shown remarkable results in novel view synthe- sis (Schwarz et al., 2021; Chan et al., 2021; Sko- rokhodov et al., 2023) and 3D reconstruction (Yariv et al., 2021; Zhang et al., 2021). With the differ- entiable neural rendering, NeRF can be guided by various objectives. Text-to-3D draws appreciable attraction these days, which adopts external text- visual alignments (Jain et al., 2022: Khalid et al., 2022; Michel et al., 2022: Wang al., 2022a; Hong et al., 2022) and pre-trained text-to-image (Wang et al., 2022b; Nam al., 2022; Poole al., 2023; Metzer et al., 2023; Tang et al., 2023; Seo et al., 2023). However, existing methods take numerous iterations to optimize NeRF model, which is time- consuming for practical usage. Our CCH learns to extract fashion semantics with NeRF rendering and incorporates the human prior for concrete human body, achieving effective and efficient T3H.
The experiments indicate that language is nec- essary make 3D human generation controllable. Our proposed CCH adopts modal attention fuse compositional neural rendering with textual fashion as 3D humans, and semantic discrimination further helps fine-grained consistency. In summary, our contributions are three-fold:
We introduce T3H to control 3D human genera- tion via fashion description, learning from collec- tions of 2D images.
We propose CCH to extract fashion semantics in the text and fuse with 3D rendering in one shot, leading to an effective and efficient T3H.
Extensive experiments show that CCH exhibits sharp 3D humans with clear textual-related fash- ion patterns. We advocate that T3H can become new field of vision-and-lange lan-uuge research.
2 Related Work
3D Human Representation. To reconstruct 3D human, early works (Collet al., 2015; Guo al., 2019: 9; Su et al., 2020) count on off-the-shelf tools to predict the camera depth. As mitigating the costly hardware requirement, they estimate a 3D human
Text-guided Visual Generation. Using human- understandable language to guide visual generation can enhance controllability and benefit creative vi- sual design. Previous works built upon adversarial

---

Text Encoder
Cross-modal Attention
The woman wearing
a medium-skee redluum-leeveed shirt
Discriminator
Real / Fake
and long denim pants
SMPL (B, 8)
Inverse
LBS
F
Compositional Human
Volume Rendering
Figure 2: Compositional Cross-modal Human (CCH), CCH extracts fashion semantics from the description and adopts cross- attention in compositional body volumes for controllable 3D human rendering. The human prior (SMPL) provides robust geometry transformation, enabling CCH learn from 2D collections for data efficiency. The semantic discrimination further helps find-gres und-grained consistency through adversarial training.
texture (Xu and Loy, 2021; Gomes al., 2022) via the UV mapping (Shysheya al., 2019; Yoon et al., 2021). With the promising success of NeRF, recent works (Peng et al., 202 bb,;; Su et al., 2021) adopt volume rendering for 3D humans from multi-view videos (Weng et al., 2022; Chen et al., 2022). Since the data are difficult to collect, the 3D-aware gener- ation (Chan et al., 2022; Gu al., 2022; Noguchi al., 2022) learns 3D modeling from the collection of human images (Yang et al., 2022: Hong al., 2023). In place of arbitrary outputs, we introduce the first controllable 3D human generation that also learns from 2D collection, and the presented fash- ion patterns should align with the description.
bound to the far bound ty:
T(t) = exp( L: [o(r(s))as),
C(r) ((r)=J. S.M T())(()))((())
(1)
where [t)) stands for their accumulated transmit- tance. StyleSDF (Or-El et al., 2022) then replaces with single distance field (SDF) d(x) for a better surface, where (xx) = a sigmoid(( d(a) and a is lcarnable scalar that controls the tightness of the density around the surface boundary.
SMPL (Bogo et al., 2016) defines the human body as {,3 0}. where 3 E R and 0 E 3 control its shape and pose. We consider Linear Blend Skin- ning (LBS) as the transformation from the canon- ical into the observation space for the point to Σ Hy, J)2, where h& E R is the scalar of the blend weight and HA IR 4x4 is the transforma- tion matrix of the kth joint. Inverse LBS transforms the observation back to the canonical space as a similar equation but with an inverted H.
3 Text-guided 3D Human Generation
3.1 Task Definition
We present text-guided 3D human generation (T3H)
create 3D humans via fashion descriptions. For data efficiency, a 2D collection D=(V,T] TJ pro- vided, where V is the human image, and T is its fashion description. Our goal to learn the neural rendcring that maps T into an articulate 3D human with the fashion patterns of V.
3.3 Compositional Cross-modal Human
Following EVA3D (Hong et al., 2023), we split the human body into 16 parts. As shown in Fig. 2, each body part holds its own bounding box {o o }. To leverage the human prior for target pose A, we transform these pre-defined bounding boxes with SMPL's transformation matrices Hy. Ray r(1) is sampled for each pixel on the canvas. For ray that intersects bounding boxes, we pick up its near and far bounds (tn and tx) and sample points as fol- lows: t; ~ 24 [t. + ' (tr tn), t, + / (tr tn)]
3.2 Background
Neural Radiance Field (NeRF) (Mildenhall et al., 2020) defines implicit 3D as (c,a)=F(xx, d). The query point æ in the viewing direction d holds the emitted radiance e and the volume density a. To get the RGB value C(r) of certain rays r(t), volume rendering is calculated along a ray r from the near

---

We then transform these sampled points back the canonical space with inverse LBS. For shape gener- alization, we consider not only pose transformation but also blend shapes (0) and B*) (Zheng et al., 2021). N contains nearest vertices v of the target SMPL mesh for the sample point ray r(4):
Algorithm 1 Compositional cross-modal Human
1: D: 2D collection of human images/ fashion descriptions
2:
3: G. D: the generator discriminatoo model for T3H
while TRAIN LCC___CHooa do
V. <- sampled human/ description from 2
(8,0) estimated SMPL parameters of V
{T.) <- canonical points via inverse I.BS
Eq. b Eq. 3
8:
J <- rendering features inside the bth box
9k.
||r(tt))
-ull''
9:
{un) < extracted textual features of '
10:
CA < fusion via cross-moda attention
> Eq. 4
I
D +
B
11:
{c,a.) <- mixture radiance/ density
b Eq. 5
M₁
Σ ghHlk
12:
R + final rendering human
b Rq.
I
13:
14:
S + segmentation map of V
[xi]] =
[r(4)]
15:
Q t fashion map between S and T
Di Eq. 6
(M²)
(2)
16:
Casr + adversarial loss from !)
> Eqq 8 Eg. 9
EypeN Ok
MEN
17:
Car, Lea <- offset and derivation loss
18:
Canl 4- overall training loss
Eq. 10
where Ok C R is the inverse weight of the vertex Uk
19:
Update G' by minimizing Lin
and Mj E Rxx4 is the transformation matrix. The can be used for further volume rendering.
20:
Update D by maximizing C.n
21:
end while
Cross-modal Attention. During rendering, if the canonical point z; with the viewing direction d, is inside the 6th bounding box, will be treated as:
where m and n are hyperparameters. With {G,, oi). we adopt Eq. 1 to render the RGB value of ray r(t). Through all sampled rays r, we then have our final human rendering R, where the overall process can be simplified as R = G(B, 0 (T)- To summarize, CCH leverages the human prior and adopts inverse LBS to acquire the canonical space for the target pose. The human body is divided into 16 parts, and each of them fuses its correlated fashion semantics via cross-modal attention. Finally, compositional bodies jointly render the target 3D human.
2/ 2r; (omax + omin)
df db
5 Linear (2), d:),
(3)
where a linear mapping is applicd to acquire pre- liminary features 1 E IR 161881122 To exhibit the desired fashion in the final rendering, we extract the word features by the text encoder as [wi E 5R5)2) from T. We then fuse the textual features with f via cross moosl attention:
exp(Se W wF)
Semantic Discrimination. With the SMPL prior, our CCH contains robust geometry transformation for humans and can learn from 2D images without actual 3D guidance. For a ground-truth truth (V, T,, we parse the 2D human image as the segmentation S (MMHuman3D, 2021), which provides the reliable body architecture. To obtain its fashion map Q. we apply cross-modal attention between S and T:
P1 E exp(J W6 wi)
CA(P {w)) Σ pru.
(4)
where L is the length of and W is the learnable matrix. In this way, each point can learn to perceive relevant textual guidance for the 6th human body part and depict corresponding fashion patterns.
{eis) =Conv(S),
Each body part has its individual volume render- ing Fb, which consists of stacked multilayer per- ceptrons (MLPs) with the SIREN activation (Sitz- mann et al., 2020). Since the point 2; may fall into multiple boxes B, follow EVA3D to apply the mixture function (Lombardi al., 2021):
L
W wf)
Qu Σ 1=1 Σ exp(e:,) W uo?)
W/, (6)
where e is the same dimension as f. W is the learn- able attention matrix, and Q perceives which hu- man body part should showcase what fashion pat- terns. We concatenate the rendered human R (or the ground- -ruth V) with Q and feed them into our discriminator to perform binary classification:
{ o.) = (CA( de {w))),
" = exp(-m(E((x))" +1(()) ++(()))))
01) EDED ub
Σ Eubldio.1
(5)
D(R T) BC([Conv(RR) Q1).
(7)
bCB
In consequence, D can provide alignments of both

---

DeepFashion
SHHQ
Method
FIDL
Depth
PCKT
CLIP-S
FAT
FID4
Depth
PCK1
CLIP-ST
FA1
Latent-NeRF
69.654
0.0298
74.211
22.500
65.883
72.256
0.0381
73.401
22.210
67.427
TEXTure
37.058
0.0165
86,354
23.385
67,508
48.618
0.0216
85.502
24.456
68.233
CLIP-O
25,488
0.0133
87.892
21.887
61.964
34.212
0.0164
87,312
21.401
66.808
CCH
21.136
0.0121
88.355
25.023
72.038
32,858
0.0165
87.624
27.855
76.194
Table 1: Overall results of pose-guided T3H.
the human pose and fashion semantics, which im- proves the fine-grained consistency of our CCH.
culate visual and geometry similarity, compared to the ground- truth image. We treat Percentage of Cor- rect Keypoints (PCK (.5) (Andriluka et al., 201 4) as the correctness of the generated pose. To investi- gate the textual relevance of T3H results, we follow CLIP Score (CLIP-S) (Hessel et al.. 2021) for the text-visual similarity. We fine-tume CLIP (Radford et al., 2021) on DeepFashion for a more accurate alignment in this specific domain. To have the fine- grained evaluation, we train a fashion classifier on DeepFashion labels' and assess Fashion Accuracy (FA) of the generated human.
Learning of CCH. We include the non non-saturating loss with RI regularization (Mescheder et al., 2018) for adversarial learning over the ground-truth (V):
U(u) - log(1 + exp(-u)),
Ladv = ((G(3,0 1 IT
(8)
+ U(-D(V T)) + AVV T)P. T)P.
Following EVA3D, we also append the minimum offset loss Cor to maintain a plausible human shape as the template mesh. Leik penalizes the derivation of delta SDFs to zero and makes the estimated SDF physically valid (Gropp et al., 2020):
Baselines, As new task, we consider the follow- ing methods as the compared baselines.
Letent-NeRF (Metzer al., 2023) brings NeRF to the latent space and guides its generation by the given object and text-to-image prior. TEXTure (Richardson et al., 2023) paints a 3D object from different viewpoints via leveraging the pre-trained depth-to-image diffusion model. CLIP-O 0 is inspired by AvatarCLIP (Hong et al., 2022), which customizes human avatar from the description with CLIP text-visual alignment. We apply the guided loss to optimize a pre-trained EVA3D (Hong et al., 2023) for faster inference. Texformer (Xu and Loy, 2021) estimates the hu- man texture from an image. Tex12Human (Jiang et al., 2022) predicts the target human image, and we treat Texformer to further build its 3D model. For a fair comparison, all baselines are re-trained on face-blurred datasets and cannot produce identi- fiable human faces.
off = || Il (x)
Leik | (Ad(2))
(9)
The learning process of our CCH is also illustrated as Algo. 1, where the overall optimization can be:
Call Ladv + 1.5 1.5--on + 0.5 Leik,
(10)
min max Call- G D
4 Experiemnts
4.1 Experimental Setup
Datasets. We coduct experiments on DeepFash- ion (Jiang et al., 2022) and SHHQ (Fu et al., 2022a) for T3H. DeepFashion contains 12K human images with upper and lower clothing descriptions. Since there are no annotations SHHQ, we first fine-tune GIT (Wang et al., 2022c) on DeepFashion and then label for 40K text-human pairs. We follow Open- Pose (Cao et al., 2019) and SMPLify-X (Pavlakos et al., 2019) to estimate the human keypoints and its SMPL parameters. The resolution is resized into 512x256 in our experiments. Note that all faces in datasets are blurred prior training, and the model is not able to generate human faces.
Implementation Detail. We divide a human body into 16 parts and deploy individual StyleSDF (Or- El et al., 2022) for each volume rendering, and two following MLPs then estimate SDF and RGB val- ues. We adopt the same discriminator as StyleSDF over fashion maps to distinguish between fake ren- dered humans and real images. We sample N=28 points for each ray and set (m, n) (4, 8) for mix-
Evaluation Metries. We apply metries from both visual and semantic prospects. Following EVA3D, we adopt Frechet Inception Distance (FID) (Heusel et al., 2017) and Depth (Ranftl et al., 2020) to cal-
'There are six targets for FA, including the shape, fabric, and color of the upper and lower clothing. The fashion classi- fier has +95% accuracy, which provides a precise evaluation.

---

DeepFashion
DeepFashion
Method
FIDJ
CLIP-St
FAT
Train
FT.
R1
R5
R10
Texformer
45.844
20.546
66.679
OpenAI-400M
4.2
20.4
28.0
CLIP-O
25.579
20.112
61.298
LAION-2B
X
13.4
33.4
46.4
CCH
21.355
24.920
70.771
LAION-2B
45.0
83.0
93.8
Table 2: Overall results of pose-free T3H.
Table 4:
Text-to-Fashion retrieval (sample 500 pairs) by CLIP with different fine-tunings (FT.).
Ablation Settings
DeepFashion
Text
CA
SD
FID
CLIP-S
FA
DeepFashion
x
x
x
25.671
9.632
36.634
Method
Quality
Relevance
x
24.624
21.079
69.173
Latent-NeRF
1.82
2.37
21.966
24.103
80.028
TEXTure
2.38
2.51
21.275
25.211
80.776
CLIP-O
2.93
2.20
Table 3: Ablation study (256x 128) of Cross-modal At-
CCH
2.87
2.92
tention (CA) and Semantic Discrimination (SD).
Table 5: Human evaluation for T3H with aspects of 3D
quality and fashion relevance.
ture rendering. The text encoder is initialized from CLIP and subsequently trained with CCH. We treat Adam (Kingma and Ba, 2015) with a batch size of 1, where the learning rates are 2e-5 for G and 2e-4 for D. We apply visual augmentations by randomly panning, scaling, and rotating within small ranges. All trainings are done using PyTorch (Paszke et al.. 2017) on NVIDIA A100 GPUs for 1M iterations.
with the correct pose, but not the assigned fashion (lower CLIP-S and FA) by the inexplicit alignment from CLIP. Table 2 presents the pose-free results. With the guided 2D image. Texformer contains the assigned clothing in the text (higher FA than CLIP- O). But the 3D reconstruction is unable to handle spatial rendering, resulting in low-quality humans (higher FID). With cross-modal attention and mantic discrimination, CCH exceeds baselines both visual and textual relevance, making concrete human rendering with the corresponding fashion.
4.2 Quantitative Results
Table shows the pose-guided T3H results on Deep- Fashion and SHHQ. where we feed the estimated human mesh as the input object into Latent-NeRF and TEXTure. Although Latent-NeRF can portray body shapes in multiple angles from its latent NeRF space, the rendering is clearly counterfeit (higher FID and Depth). For TEXTure, the human architec- ture is constructed well by the given mesh (higher PCK). However, the estimated texture is still spa- tially inconsistent and contains inevitable artifacts (still higher FID). From the semantic aspect, Latent- NeRF and TEXTure borrow those trained diffusion models and depict the assigned appearance in the description (higher CLIP-S than CLIP-O). CLIP-O relies on EVA3D produce feasible 3D humans (lower FID). While the external CLIP loss attempts to guide the fashion, the global alignment is insuf- ficient to demonstrate detailed patterns (lower FA). Without those above drawbacks, our CCH learns to extract fashion semantics along with the composi- tional human generation, lcading to comprchensive superiority across all metrics.
4.3 Ablation Study
We study each component effect CCH in Table 3. Without the guided description, the model lacks the target fashion and results in a poor FA. This further highlights the importance of textual guidance for controllable human generation. When applying the traditional training (Reed et al., 2016), conditional GAN insufficient to extract fashion semantics for effective T3H (not good enough CLIP-S). On the other hand, our cross-modal attention constructs a better fusion between fashion patterns and volume rendering, facilitating a significant improvement in depicting the desired human appearance. Moreover, semantic discrimination benefits fine-grained align- ment and leads to comprehensive advancement.
Fine-tune CLIP-S as Evaluator. CLIP has shown promising text-visual alignment, which can calcu- late feature similarity between the generated human and the given text CLIP-S (Hessel et al., 2021). Since our T3H specific fashion domain, we consider the larger-scaled trained checkpoint from OpenCLIP (Ilharco et al., 2021) and fine-tune as
A similar trend can be found on SHHQ. Latent- N and TEXTure exhibit related fashion patterns but are hard to present realistic humans (higher FID and Depth). CLIP-O produces a sharp human body

---

The woman is wearing short-sleeved t-shirt, paired with three-point denim pants
DeepFashion
The gentleman is sporting long- sleeved cotton sweater and solid-colored long trousers
SHHQ
Latent-NeRF
TEXTure
CLIP-O
CCH
Figure 3: Qualitative comparison of pose-guue pose-guided T3H.
She is dressed sleeveless tank top with long denim pants
Method
Time (sec)
GPU (MB)
Latent-NeRF
755.7
11250
TEXTure
103.7
12530
CLIP-O
181.6
15988
CCH
0.372
6258
Table 6: Time and GPU cost perform T3H.
a more precise evaluator. Table presents text-to- fashion retrieval results, where higher recall leads to a better alignment. Whether the original CLIP or OpenCLIP, both result in poor performance and is insufficient for our evaluation. By perceiving Deep- Fashion, fine-tuning helps bring reliable alignment and is treated as the final evaluator.
Texformer
CCH
Figure 4: Qualitative comparison of pose-free T3H.
our CCH also contains a higher efficiency. Table 6 shows the inference time and GPU cost on single NVIDIA TITAN RTX. All baselines take more than 100 seconds since they require multiple iterations to optimize the 3D model from an external alignment. In contrast, extract fashion semantics and carry out T3H in shot. Without updating the model, save the most GPU memory. summary, CCH surpasses baselines on both quality and efficiency, leading to an effective and practical T3H.
Human Evaluation. Apart from automatic met- rics, we conduct the human evaluation with aspects of 3D quality and fashion relevance. We randomly sample 75 T3H results and consider MTurk2 to rank over baselines and our CCH. To avoid the potential ranking bias, we hire MTurkers for each example. Table shows the mean ranking score (from to the higher is the better). CLIP-O and CCH are built upon EVA3D, which provides an articulate human body for superior 3D quality. Even if Latent-NeRF and TuXTure take pre-trained diffusion models to acquire visual guidance, CCH exhibits more corre- sponding fashion via cross modal fusion. This per- formance trend is similar our evaluation, which supports the usage of CLIP-S and FA as metrics.
4.4 Qualitative Results.
We demonstrate the qualitative comparison of pose- guided T3H in Fig. 3. Although Latent-NeRF can portray the 3D human based on the given mesh, it only presents inauthentic rendering. TEXTure gen- erates concrete humans, but there are still obvious cracks and inconsistent textures from different an- gles of view. Morcover, both of them fail to capture "three-point", where the rendered lower clothing is
Inference Efficiency. In addition to T3H quality,
AAmazon MTuur:: https: / /www. com

---

This mon is wearing o striped medium sleeve and long cotton trousers
She is dressed in long-sleeved chiffon shirt with striped three-point shorts
raise arms
walking
Figure 5: Qualitative examples of animatable T3H, where the motion is also controlled by the text.
The woman is wearing striped cotton
short-sleeve cotton
long-sleeve
shirt, paired with long cotton pants
knitted
striped
graphic
three-point denim
long cotton
color block
pure color
Figure 7: Qualitative examples of our fashion classifier, which provides fine-grained labels for real/fake humans.
She is sporting a graphic- patterned tank top with floral- patterned three-point shorts
human body present diverse appearances; different poses then guide the character to express rich body language. This flexibility in controlling appearance and pose allows for better practical customization.
Animatable T3H. In addition to static poses, CCH can benefit from dynamic motions to achieve ami- matable T3H. Fig. 5 adopts MotionDiffuse (Zhang et al., 2022) to create the assigned action also from the text and apply it to our produced 3D models. In this way, prompt them to "raise arms" "walk" for favorable dynamic scenarios.
Figure 6: Qualitative examples of pose-control T3H.
incorrectly depicted as long pants. Because CLIP provides an overall but inexplicit alignment to the description, CLIP-O is limited and exhibits vague "denim" or "long sleeved" This observation fur- ther indicates the flaw of CLIP in detailed fashion patterns, even if has been fine-tuned on the target dataset. In contrast, our CCH adopts cross-modee modal attention with NeRF, contributing to high-quality T3H with fine -reigga fashion controllability. Fig. 4 shows the frse-free results. Texformer relies on a 2D image to estimate its 3D texture. Despite con- taining the assigned fashion, it is still restricted by the capability of 3D reconstruction, resulting in low-resolution rendering. By learning text-to-3D directly, CCH can produce textual-related humans from random poses with clear visual patterns.
5 Conclusion
We present text-guided 3D human generation (T3H) to create 3D human by fashion description. To learn this from 2D collections, we introduce Com- positional Cross-modal Human (CCH). With cross- modal attention, CCH fuses compositional human rendering and textual semantics to build a concrete body architecture with the corresponding fashion. Experiments across various fashion attributes show that CCH effectively carries out T3H with high effi- ciency. We believe T3H helps advance a new field toward vision-and-langen lannuuuugg research.
Ethics Discussion and Limitation. Our work en- hances the controllability of 3D human generation. To prevent identity leakage, we blur out faces prior to training and avoid risks similar to DeepFake (Ko- rshumov and Marcel, 2018). Because we depend on SMPL parameters, an inaccurate estimation causes
Pose-control T3H. Since our CCH is generating 3D humans from given SMPL parameters, as illus- trated in Fig. 6, we can further control T3H with a specific pose. Different fashion descriptions make a

---

a distribution shift and quality degradation. For the datasets, they reveal narrow viewing angles, which results in visible artifacts of 3D consistency.
Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese. Hugues Hoppe. Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. In Transactions on Graphics (TOG).
Acknowledgements. We appreciate the anony- mous reviewers for constructive feedback. The re- search presented in this work was funded by Meta AL The views expressed are those of the authors and do not reflect the official policy or position of the funding agency.
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021. Cog Mastering Text-to-Imase Generation via Transform- ers. In Conference on Neural Information Processing Systems (NeurIPS).
Alaaeldin El-Nouby, Shikhar Sharma, Hamnes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W. W.Taylor. 2019. Tell. Draw, and Repeat: Generating and Moditying Images Based on Continual Linguistic Instruction. Inter- national Conference on Computer Vision (ICCV).
References
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler,
and Bernt Schiele. 2014. 2D Human Pose Estimation: New Benchmark and State of the Art Analysis. In Conference on Computer Vision and Pattern Recog- nition (CVPR).
Patrick Esser, Robin Rombach, and Björ Ommer. 2021. Taming Transformers for High-Resolution Image Synthesis. In Conference on Computer Vision and Pottern Recognition (CVPR).
Jonathan Barron, Ben Mildenhall, Matthew Tan- cik, Peter Hedman, Ricardo Martin- Brualla, and Pratul P. Srinivasan. 2022. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In Conference on Computer Vision and Pat- tern Recognition (CVPR).
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023. Training-Free Faree Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. In In- ternational Conference on Learning Representations (ICIR).
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. 2016. Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from Single Image. In European Conference on Computer Vision (ECCV).
Jianglin Fu, Shikai Li, Yuming Jiang, Kwan- Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Zi- wei Liu. 2022a. StyleGAN Human: Humman Odyssey of Human Generation. In Europan Confer- ence Computer Vision (ECCV).
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. OpenPose: Realtime Multi- Person 2D Pose Estimation using Part Affinity Fields. In Transactions on Pattern Analysis and Machine Intelligence (TPAMI).
Data-Centric
Tsu-Jui Fu, Xin Eric Wang. Scott Grafton, Miguel Eckstein, and William Yang Wang. 2020. SSCR: Iterative Language- Language-Based Supervised Counterfactual Reasoning. In Confer- ence on Empirical Methods in Natural Language Processing (EMNLP).
Eric R. Chan, Comnor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan. Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2022. Efficient Geometry-aware 3D Generative Adversarial Networks. In Conference on Computer Vision and Pattern Recognition (CVPR).
Based Image Editing via Self-
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang. 2022b. M Language-based Video Editing Multi-Modal Multi-Level Transformer. In Conference on Com- puter Vision and Pattern Recognition (CVPR).
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. 2021. pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D- Aware Image Synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR).
Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. 2022c. Language-Driven Artistic Style Transfer. In European Conference on Computer Vision (ECCV).
Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan. 2022. Guuiddd Progressive NeRF for Generaliz- able and Efficient Neural Human Rendering. In Con- ference on Computer Vision and Puttern Recognition (CVPR).
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yane Fu, Jong-Chyi Su, William Yang Wang, and Sean Bell. 2023. Tell Me What Happened: Unifying Text- guided Video Completion via Multimodal Masked Video Generation. In Conference on Computer Vi- sion and Pattern Recognition (CVPR).
Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. In Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. 2019. SDM-NET: Deep Generative Network for Structured Deformable Mesh.

---

In Special Interest Croup on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia).
Fangzhou Hong. Zhaoxi Chen. Yushi Lan, Liang Pan. and Ziwei Liu. 2023. EVA3D: Compositional 3D Human Generation from 2D Image Collections. In International Conference on Learning Representa- lions (ICLR).
Xiangjun Gao, Jiaolong Yang, Jongyoo Kim. Sida Peng, Zicheng Liu, and Xin Tong. 2022. MPS-NeRF: Gen- eralizable 3D Human Rendering from Multiview Im- ages. In Transactions on Pattern Analysis and Ma- chine Intelligence (TPAMI).
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhon- gang Cai, Lei Yang. and 7iwei Liu. 2022. Avatar- CLIP: Zero-Shot Text-Driven Generation and Ani- mation of 3D Avatars. In Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH).
Thiago L. Gomes, Thiago M. Coutinho, Rafael Azevedo, Renato Martins, and Erickson R. Nascimento. 2022. Creating and Reenacting Controllable 3D Humans with Differentiable Rendering. In Winter Conference on Applications of Computer Vision ( (WACV).
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman. Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hammaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. OpenCLIP. https://githw:/ com/ml foundations/open.cc.
lan J. Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2015. Generative Adversarial Networks. In Conference on Neural In- formation Processing Systems (NewrIPS).
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. 2022. Zero-Shot Text- Guided Object Generation with Dream Fields. In Conference on Computer Vision and Pattern Recog- nition (CVPR).
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020. Implicit Geometric Reg- ularization for Learning Shapes. In International Conference Machine Learning (ICML).
Yuming Jiang, Shuai Yang, Haonan Qiu. Wayne Wu, Chen Change Loy, and Ziwei Liu. 2022. Text2Human: Text-Driven Controllable Human Im- age Generation. In Transactions on Graphics (TOG).
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. 2022. StyleNeRF: A Style-based 3D- Aware Generator for High-resolution Image Synthe- sis. In International Conference on Learning Repre- sentations (ICLR)..
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. 2022. CLIP-Mesh: Generating textured meshes from text using pre- trained image-text models. In Special Interest Group on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia).
Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dour- garian, Matthew Vall, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emiiy Cooper, Mingsong Dou, Sean Fanellov, Graham Fyffe, Christoph Rhe- manny, Jonathan Taylor, Paul Debevec, and Shahram Izadi. 2019. The Relightables: Volumetric Perfor- mance Capture of Humans with Realistic Relighting. In Transactions on Graphics (TOG).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).
Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Danilidis. 2019. Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop. In International Conference on Computer Vision (ICCV).
Paul Henderson, Vagia Tsiminaki, and Christoph H. Lampert. 2020. Leveraging 2D Data to Learn Tex- tured 3D Mesh Generation. In Conference on Com- puter Vision and Pattern Recognition (CVPR).
Pavel Korshunov and Sebastien Marcel. 2018. Deep- Fakes: a New Threat to Face Recognition? Assess- ment and Detection. In arXiv:1812..0860
Jack Hessel, Ani Holtzman, Maxwell Forbes, Ro- nan Le Bras, and Yejin Choi. 2021. CLIPScore: A Reference-fiee free Evaluation Metrie for Image Caption- ing. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. 2018a. Point Cloud GAN. In International Conference on Learn- ing Representations (ICLR).
Martin Heusel, Hubert Ramsaver, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by Two Tcae-Scale Update Rule Converge to a Local Nash Equilibrium. In Conference on Neu- ral Information Processing Systems (NeurIPS).
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. GRASS: Generative Recursive Autoencoders for Shape Struc- tures. In Transactions on Graphics (TOG).
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising Diffusion Probabilistic Models. In Con- ference on Neural Information Processing Systems (NeurIPS).
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and Lawrence Carin. 2018b. Video Gener- ation From Text. In AAAI Conference on Artificial Intelligence (AAAT).

---

Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annota- tions. In Conference on Computer Vision and Pattern Recognition (CVPR).
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tat- suya Harada. 2022. Unsupervised Learning of Effi- cient Geometry-Aware Neural Articulated Represen- tations. In European Conference on Computer Vision (ECCV).
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer. Yaser Sheikh, and Jason Saragih. 2021. Mixture of Volumetric Primitives for Effi- cient Neural Rendering. In Special Interest Group on Computer Craphics and Interactive Techniques (SIGGRAPH).
Roy Or-El, Xuan Luo, Mengyi Shan. Eli Shechtman, Jeong Joon Park, and Ira Lemellmcher-Shiizeenne 2022. StyleSDF: High- Resh-Reslltinnn 3D-Consistent Image and Geometry Generation. In Conference on Computer Vision and Pattern Recognition (CVPR).
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Conference on Computer Vision and Pattern Recognition (CVPR).
Andrew Luo, Tianqin Li, Wen-Hao Zhang, and Tai Sing Lee. 2021. SurfGen: Adversarial 3D Shape Synthe- sis with Explicit Surface Discriminators. In Interna- tional Conference on Computer Vision (ICCV).
Tanya Marwah, Gaurav Mittal, and Vineeth N. Balasub- ramanian. 2017. Attentive Semantic Video Genera- tion using Captions. In International Conference on Computer Vision (ICCV).
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Fdward Yang. Zachary Vetito, 7eming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In In- ternational Conference on Learning Representations Workshop (ICLRW).
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018. Which Training Methods for GANs do actually Converge? International Conference on Machine Learning (ICML..
Georgios Pavlakos, Vasileios Choutas, Nima Ghor- bani, Ahmed Timo Bolkart, A. Osman, Dimitrios Tzionas, and Michael J. Black. 2019. Expressive Body Capture: 3D Hands, Face, and Body from Single Image. Conference on Computer Vision and Pattern Recognition (CVPR).
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. 2023. LeRe for Shape- Guided Generation of 3D Shapes and Textures. In Conference on Computer Vision and Pattern Recog- nition (CVPR).
Sida Peng, Junting Dong, Qianqian Wang. Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. 2021a. Animatable Neural Radiance Fields for Mod- eling Dynamic Human Bodies. In International Con- ference on Computer Vision (ICCV).
Oscar Michel, Roi On. Richard Liu, Sagie Benaim, and Rana Hanocka. 2022. Text2Mesh: Text- Text-Driven Neural Stylization for Meshes. In Conference on Computer Vision and Pattern Recognition (CVPR).
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2021b. Neural Body: Implicit Neural Representa- tions with Structured Latent Codes for Novel View Synthesis of Dynamic Humans. In Conference on Computer Vision and Pattern Recognition (CVPR).
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV).
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2023. DreamFusion: Text-to-3D using 2D Diffuusio.. In International Conference on Learn- ing Representations (ICLR).
MMHuman3D. 2021. Open 3D Human Para-
metric Model Toolbox and Benchmark. https: //github.ccmm/peee www.///////oo//////eeeaeee mm mddmmm/n..
Thomas Muller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. In Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH).
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. 2021. Learn ing Transferable Visual Models From Natural Lan- guage Supervision. In International Conference on Machine Learning (ICML).
Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Al- berto Tono, Linqi Zhou, and Paul Guerrero. 2022.
3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models, In arx0v:2212.00842.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text- Conditional Image Generation with CLIP Latents. In 204.022204
Charlie Nash, Yaroslav Ganin, Ali Eslami, and Peter W. Battaglia. 2020. PolyGen: An Autoregressive Gener- ative Model of 3D Meshes. In International Confer- ence on Machine Learning (ICML).
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and

---

Ilya Sutskever. 2021. Zero-Shot Text-to-Image Gen- eration. In International Conference on Machine Learning (ICML).
Shih- Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin. 2021. A-NeRF: Articulated Neural Radi- ance Fields for Learning Human Shape, Appearance, and Pose. In Conference on Neural Information Pro- cessing Systems (NeurlPS).
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards Ro- bust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer. In Transac- tions on Pattern Analysis and Machine Intelligence (TPAMI..
Zhuo Su, Lan Xu, Zerong Zheng, Tao Yu, Yebin Liu, and Lu Fang. 2020. RobustFusion: Human Volu- metric Capture with Data-driven Visual Cues using RGBD Camera. In Earopean Conference on Com- puter Vision (ECCV).
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. 2016. Generative Adversarial Text Image Synthesis. In International Conference on Machine Learning (ICML).
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang. Ran Yi, Lizhuang Ma, and Dong Chen. 2023. Make- It-3D: High-Fidelin 3D Creation from A Single Im- age with Diffusion Prior. In arXiv:2303. 14184.
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes. and Daniel Cohen-Or. 2023. TEXTure: Text-Guided Texturing of 3D Shapes. In Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPHI).
Maxim Tatarchenko, Alexey Dasovitskiy, and Thomas Brox. 2017. Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs. In International Conference on Computer Vision (ICCV).
Can Wang, Menglei Chai, Mingming He. Dongdong Chen, and Jing Liao. 2022a. CLIP-NeRF: Text- and- and-Image Driven Manipulation of Neural Radiance Fields. In Conference on Computer Vision and Pat- tern Recognition (CVPR).
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. 2022. Photo- realistic Text-to-Image Diffusion Models with Deep Language Understanding. In Conference on Neural Information Processing Systems (NeurIPS).
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. 2022b. Score Jaco- bian Chaining: Lifting Pretrained 2D Diffusion Mod- els for 3D Generation. In arXiv:22
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and An- dreas Geiger. 2021. GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. In Con- ference on Neural Information Processing Systems (NeurlPS).
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Li- juan Wang. 2022c. GIT: A Generative Image-to-text Transformer for Vision and Language. In Transac- tions on Machine Learning Research (TMLR).
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jae- hoon Ko. Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. 2023. Let 2D Diffusion Model Know DD-Consistency for Robust Text-to-3D Generation. In 200.07230...7737.
Chung-Yi Weng, Brian Curless, Pratul P. Srini- vasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. 2022. HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video. In Conference on Computer Vision and Pattern Recognition (CVPR).
Aliaksandra Shysheya, Egor Zakharov, Kara-All Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Alek- sei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov. Alexander Vakhitov. and Victor Lempitsky. 2019. Textured Neural Avatars. In Con- ference on Computer Vision and Pattern Recognition (CVPR).
Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. AttnGAN: Fine-Grained Text Image Generation with Attentional Generative Adversarial Networks. In Conference on Computer Vision and Pattern Recognition (CVPR).
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Conference on Neural Infor- mation Processing Systems (NeurIPS).
Xiangyu Xu and Chen Change Loy. 2021. 3D Human
Texture Estimation from a Single Image with Trans- formers. In International Conference on Computer Vision (ICCV).
Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin- Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 2023. 3D generation on ImageNet. In In- ternational Conference on Learning Representations (ICIR).
Guandao Yang. Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. Point- Flow: 3D Point Cloud Generation with Continuous Normalizing Flows. International Conference on Computer Vision (ICCV).

---

Zhuogian Yang. Shikai Li, Wayne Wu, and Bo Dai. 2022. 3DHumanGAN: Towards Photo-Realistic
3D-Aware Human Image Generation. In
arXiv:2212.07378.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021. Volume Rendering of Neural Implicit Surfaces. In Conference on Neural Information Processing Sys- tems (NeurIPS).
Jae Shin Yoon, Lingjie Liu. Vladislav Golyanik, Kri- pasindhu Sarkar, Hyun Soo Park, and Christian Theobalt. 2021. Pose-Guided Human Animation from a Single Image the Wild. Conference on Computer Vision and Pattern Recognition (CVPR).
Jason Y. Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. 2021. NeRS: Neural Reflectance Surfaces Sparse-view 3D Reconstruction the Wild. In Conference on Neural Information Process- ing Systems (NeurlPS).
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022. MotionDiffuse: Text-Driven Human Motion Genera- tion with Diffusion Model. In arXiv:2208. 15001.
Xin- Xin-Yang Zheng, Yang Liu, Peng-Shus Shuai Wang, and Xin Tong. 2022. SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generation. In Computer Graphics Forum (SGP).
Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Parametric Condi-Condi Im- plicit Representation for Image-based Human Recon- struction. In Transactions Pattern Analysis and Machine Intelligence (TPAMT).
2021. PaMIR:


Reviews:
[{"review_id": "3zBqUiqoN2", "reviewer": "Reviewer_oudz", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: range of text**\\\n**A1:** We first want to clarify that our proposed method is not limited to fashion generation. With the flexible cross-modal attention over volume rendering, the model is able to learn from diverse text-annotated human images for 3D humans through our strategy. Our T3H is motivated by Text2Human [1,2,3], which also considers the fashion scenario. As the abundant visual data and human labels of fashion datasets (our used DeepFashion and SHHQ), it can support large-scale experiments and fine-grained evaluations. On the other hand, there is also text-guided virtual try-on [4], which also presents beneficial controllability, which is aligned with the motivation in our work.\n\n[1] (SIGGRAPH’22) Text2Human: Text-Driven Controllable Human Image Generation\\\n[2] (arXiv’23) Text–Conditioned Fashion Image Editing With Guided GAN Inversion\\\n[3] (arXiv’23) Text-Driven Human Video Generation\\\n[4] (SIGGRAPH’23) Controllable Virtual Try-on with Text and Texture\n\n**Q2: video is essential**\\\n**A2:** For human evaluation, we do present the generated human GIFs for comparison. We also provide the animated qualitative examples (also in GIF) on our project website (L40).\n\n**Q3: cross-modal attention**\\\n**A3:** To the best knowledge, we are the first to integrate cross-modal attention into neural volume rendering, which can generate a 3D human in one step. In contrast, the baselines (Latent-NeRF and TEXTure) leverage a pre-trained text-to-image model, which adopts cross-modal attention only for static images, and requires numerous steps to guide a NeRF model. Hence our method shows better generated results (Tables 1 and 5) as well as significantly superior efficiency (Table 6). This further demonstrates our contribution of cross-modal attention to 3D human generation.\n\n**Q4: math notations**\\\n**A4:**\\\nNeRF:\n+ $\\alpha \\in \\mathbb{R}$ is a learned scalar in StyleSDF that controls the tightness of the density around the surface boundary. $\\alpha=0$ represents a solid, sharp, and object boundary. A larger $\\alpha$ leads to a more fluffy object boundary.\n\nSMPL:\n+ $\\beta \\in \\mathbb{R}^{10}$ controls the shape (10 parameters) of a human body.\n+ $\\theta \\in \\mathbb{R}^{69}$ controls the pose (69 parameters of 23 relative joints) of a human body.\n+ $H_k \\in \\mathbb{R}^{4 \\times 4}$ is the transformation matrix for the $k$th local joint, and $h_k \\in \\mathbb{R}$ is a scalar of the blend weight.\n\nInverse LBS:\n+ $M_k \\in \\mathbb{R}^{4 \\times 4}$ and $g_k \\in \\mathbb{R}$ are similar to $H_k$ and $h_k$ but for th inverse transformation. \n\nCross-modal Attention:\n+ $f^b \\in \\mathbb{R}^{\\text{h} \\times \\text{w} \\times \\text{c}}$ is the preliminary visual features, where $\\text{h}=16$, $\\text{w}=8$, and $\\text{c}=128$.\n+ $w_l \\in \\mathbb{R}^{\\text{x}}$ is the encoded word features from the CLIP text encoder, where $\\text{x}=512$.\n+ $W^b \\in \\mathbb{R}^{\\text{c} \\times \\text{x}}$ is the learnable attention matrix for cross-modal fusion.\n\nSemantic Discrimination:\n+ $e \\in \\mathbb{R}^{\\text{h} \\times \\text{w} \\times \\text{c}}$ is the visual features of the input segmentation map $\\mathcal{S}$, which has a similar dimension of $f$.\n+ $W \\in \\mathbb{R}^{\\text{c} \\times \\text{x}}$ is another learnable attention matrix for the Discriminator $D$.\n\nWe will add the above clarification and improve Figure 2 for a detailed architecture and training pipeline in our revision.\n\n**Q5: under the scope of EMNLP**\\\n**A5:** In this paper, we investigate the controllability of text for efficient 3D human generation, which embraces the cooperation between language and visual synthesis. As you comment that \"no study has performed 3D human generation based on a fashion description\", we believe our introduced task can advance a new field of vision-and-language research. In addition, there are many works [1,2,3,4] of text-to-image generation/manipulation presented in recent NLP conferences. Our T3H takes a forward step from 2D to 3D, which can benefit the community.\n\n[1] (ACL’23) Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed\\\n[2] (ACL’23) Resolving Ambiguities in Text-to-Image Generative Models\\\n[3] (ACL’23) I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors\\\n[4] (EACL’23) Improving Cross-modal Alignment for Text-Guided Image Inpainting", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a new task to create a 3D human based on a fashion description.\n- Since no study has performed 3D human generation based on a fashion description, the authors compared the performance with similar studies, and the experiment showed better performance based on 5 metrics in 2 datasets.\n", "reasons_to_reject": "- Customizing 3D characters using text can have advantages, but it is less persuasive as to why the range of text should be limited to fashion description. For fashion, most of them are guided by images, such as virtual try on [1], and the benefits of using text for guidance are not clear.\n[1] Choi, S., Park, S., Lee, M., & Choo, J. (2021). Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14131-14140).\n- In addition, since this study generates multi-view images based on NeRF, a video is essential for evaluating the 3D creation.\n- The paper uses a cross-modal attention to create text-reflected images in multi-view, but since cross-attention is already a widely-used method in diffusion models, I do not think that it bears clear novelty.\n- Also, the writing quality leaves room for improvement. For example, the math notations are inaccurate, reducing the readability of the paper. It is strongly recommended to provide a more detailed network architecture figure to aid the understanding of the proposed approach.\n- Finally, although this study utilizes text, I am not sure whether it falls under the scope of this society.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main task of the paper is fashion description-guided 3D human generation. Since 3D human modeling is widely used in gaming and animation, as it is necessary to customize characters, the paper suggested text as the most direct way to control it. The proposed algorithm converts SMPL into a canonical space through inverse LBS, renders it to generate features that fit the given fashion description through a cross-modal attention in that space, and then sends it back to the observation space using LBS. \nAlthough this paper utilizes text, in my opinion, it does not fit the scope of EMNLP mentioned above.\n", "excitement": 1, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In the paper, the mathematical notations are unconventional, leading to reduced readability. For instance, both vectors and scalars are denoted using the same symbols, and the dimensions of variables are not specified, further complicating comprehension."}}, {"review_id": "NE89HOCtC2", "reviewer": "Reviewer_3mSQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for pointing this out! We train a gender classifier on our blurred human dataset (validation accuracy >99.6%). We then adopt it to evaluate our generated 3D humans, and the gender accuracy is 91.3%. This indicates that our model can still capture the gender concept only from the visual pattern, even if without actual faces. We will add this result to our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1, computing efficience is better than SOTA\n2, generation quality is good\n3, the proposed text based controlling is effective\n4, it is fancy to see the proposed approch works with motion generation algorithm.", "reasons_to_reject": "the employed compositional 3D human rendering result has none identity information, it means no dicriminative input is provided for the neural rendering network, could result in lower robustness for final result generation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a text based 3D photorealistic human generation approch, the generation quality is good and the text controlling effectiveness is good.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BdpTmu7ZU5", "reviewer": "Reviewer_o3tu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Since no single criterion can cover them all, we try our best to consider various metrics to evaluate each aspect of text-guided 3D human generation.\n+ FID quantifies the general realism of visual patterns;\n+ Depth and PCK measure the geometry similarity of 3D humans;\n+ CLIP-S and FA calculate the text-visual alignment with input descriptions.\n\nWe further compute Pearson correlation between human evaluation and automatic metrics (metric scores are normalized):\n|  | 3D Quality |\n| :-: | :-: | \n| FID | 76.4 |\n| Depth | 73.8 |\n| PCK | 70.3 |\n\n|  | Fashion Relevance |\n| :-: | :-: | \n| CLIP-S | 78.0 |\n| FA | 83.2 |\n\nThe results demonstrate that they are correlated, which supports our usage of automatic metrics. We will add this discussion to our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper lie in its ability to achieve faster execution compared to conventional methods, generate high-quality 3D human models, and facilitate easier data collection for learning.\n\nThe inference time for 3D human generation, which used to take over 100 seconds in conventional methods, has been reduced to less than one second. Additionally, this reduction in computation also saves GPU memory, making it accessible to a broader audience.\nThe quality of the generated 3D humans is quantitatively evaluated using various metrics, demonstrating significantly better performance compared to conventional methods. This trend is consistent even in human evaluation.\nThe fact that the paper is capable of developing T3H, which is a new field of vision-and-language research, is the NLP community’s benefit.\n", "reasons_to_reject": "This is just a minor point, but I'm not sure if the automatic evaluation criteria used in the paper are really helpful, as they do not directly measure the quality of generated models. As the paper presents a human evaluation, this may not be a very big problem, but some discussion on this point (whether they are good enough or not) may be helpful.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the method of generating 3D human body models from fashion descriptions. Several challenges in previous studies, known as T3H, had problems with unclear rendering, the use of hard-to-obtain multi-view videos for training, lack of language control, and time-consuming iterative inference optimization.\n\nThe main contributions towards solving these issues are as follows:\n\n- Proposal of a method for generating 3D human models controlled by textual descriptions using 2D collections for training.\n- Enhancing the efficiency of 3D human model generation by extracting fashion semantics from the text and fusing them with 3D rendering using cross-modal attention.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers provided feedback for this paper. They found the work interesting, the model capable and they appreciated the faster performance by the proposed approach. There was one glaring complaint from a reviewer, that had to do with fit to EMNLP. First, the reviewer felt that text conditioning the 3D generation was not sufficient to be accepted at a top language conference. Second, the reviewer felt that image conditioning was more relevant than text conditioning for this task. And third, they questioned the novelty. The authors provided a detailed rebuttal to these points and I agree with the authors views here. I find this work to be of interest to the EMNLP audience, particularly as multimodal AI is becoming increasingly popular at EMNLP. I think that text conditioned avatar generation / 3D generation is going to be very popular as a research topic going forward, and so I think this paper is very timely. I also disagree that the paper does not deliver on novelty of interest to the reader. Given the reviews, rebuttal and the above details, I recommend acceptance.
Decision: Accept-Findings