EVA3D: Compositional 3D Human Generation from 2D Image Collections
Authors: Fangzhou Hong
Zhaoxi Chen
Yushi LAN
Liang Pan
Ziwei Liu
Venue: ICLR 2023 notable top 25% (2023)
Keywords: 3D Human Generation
Human NeRF
Inverse Graphics

Abstract: Inverse graphics aims to recover 3D models from 2D observations. Utilizing differentiable rendering, recent 3D-aware generative models have shown impressive results of rigid object generation using 2D images. However, it remains challenging to generate articulated objects, like human bodies, due to their complexity and diversity in poses and appearances. In this work, we propose, EVA3D, an unconditional 3D human generative model learned from 2D image collections only. EVA3D can sample 3D humans with detailed geometry and render high-quality images (up to 512x256) without bells and whistles (e.g. super resolution). At the core of EVA3D is a compositional human NeRF representation, which divides the human body into local parts. Each part is represented by an individual volume. This compositional representation enables 1) inherent human priors, 2) adaptive allocation of network parameters, 3) efficient training and rendering. Moreover, to accommodate for the characteristics of sparse 2D human image collections (e.g. imbalanced pose distribution), we propose a pose-guided sampling strategy for better GAN learning. Extensive experiments validate that EVA3D achieves state-of-the-art 3D human generation performance regarding both geometry and texture quality. Notably, EVA3D demonstrates great potential and scalability to "inverse-graphics" diverse human bodies with a clean framework.

Published as a conference paper at ICLR 2023
EVA3D: COMPOSITIONAL 3D HUMAN GENERATION FROM 2D IMAGE COLLECTIONS
Fangzhou Hong, Zhaoxi Chen, Yushi Lan, Liang Pan, Ziwei Liu
S-L.ab, Nanyang Technological University [fangzhcu001, zhaoxi001, yushi001, liang-pan ziwei. twei..iufentu.edi
Figure 1: EVA3D generates high-quality and diverse 3D humans with photo-realistic RGB render- ings and detailed geometry. Only 2D image collections are used for training.
ABSTRACT
Inverse graphics aims to recover 3D models from 2D observations. Utilizing dif- ferentiable rendering, recent 3D-aware generative models have shown impressive results of rigid object generation using 2D images. However, it remains challeng- ing to generate articulated objects, like human bodies, due their complexity and diversity in poses and appearances. In this work, we propose, EVA3D, an un- conditional 3D human generative model learned from 2D image collections only. EVA3D can sample 3D humans with detailed geometry and render high-quality images (up to 512 256) without bells and whistles (e.g. super resolution). At the сore of EVA3D is compositional human NeRF representation, which divides the human body into local parts. Each part is represented by an individual volume. This compositional representation enables 1) inherent human priors. 2) adaptive allocation of network parameters, 3) efficient training and rendering. Moreover, to accommodate for the characteristies of sparse 2D human image collections (e.g. imbalanced pose distribution), we propose a pose-guided sampling strategy for better GAN learning. Extensive experiments validate that EVA3D achieves state- of-the-art 3D human generation performance regarding both geometry and texture quality. Notably, EVA3D demonstrates great potential and scalability to "inverse- graphics" diverse human bodies with a clean framework, Our code is publicly available httptttt////gttt htt.t//o....ccoooggog//////000
com wwt..oooooo//////////aa
INTRODUCTION
Inverse graphics studies inverse- engrre-engineeriig of projection physics, which aims to recover the 3D world from 2D observations. is not only long-standing scientific quest, but also enables

---

Published as a conference paper at ICLR 2023
merous applications in VR/AR and VFX. Recently, 3D-aware generative models (Chan al. 2021 Or-El et al. 2022; Chan et al. 2022) demonstrate great potential in inverse graphics by learning to generate 3D rigid objects (e.g. human/animal faces, CAD models) from 2D image collections. However, human bodies, as articulated objects, have complex articulations and diverse appearances. Therefore, it is challenging learn 3D human generative models that can synthesis animatable 31> humans with high-fidelity textures and vivid geometric details.
To generate high-quality 3D humans, we argue that two main factors should be properly addressed: 1) 3D human representation; 2) generative network training strategies. Due to the articulated nature of human bodies, a desirable human representation should be able to explicitly control the pose/shape of 3D humans. With an articulated representation, a 3D human is modeled in its canon- ical pose (canonical space), and can rendered in different poses and shapes (observation space). Moreover, the efficiency of the representation matters in high-quality 3D human generation. Previ- ous methods (Ngguchi et al., 2022; Bergman et al., (022) fail to achieve high resolution generation due to their inefficient human representations.
In addition,, training strategies could also highly influence 3D human generative models. The issue mainly comes from the data characteristics. Compared with datasets used by Noguchi et al. (2022) (e.g. AIST (Tsuchida et al., 2019)), fashion datasets (e.g. DeepFashion (Liu et al. 2016)) are more aligned with real-world human image distributions, making favorable dataset choice. However, fashion datasets mostly have very limited human poses and highly imbalanced viewing angles. This imbalanced 2D) data distribution could hinder 3D GAN learning, leading to difficulties in novel view/ pose synthesis. Therefore, a proper training strategy is in need to alleviate the issue.
In this work, we propose EVA3D, an unconditional high-quality 3D human generative model from sparse 2D human image collections only. To facilitate that, we propose a compositional human NeRF representation improve the model efficiency. We divide the human body into 16 parts and assign each part an individual network, which models the corresponding local volume. Our hu- man representation mainly provides three advantages. 1) It inherently describes the human body prior, which supports explicit control over human body shapes and poses. 2) It supports adap- tively allocating computation resources. More complex body parts (e.g. heads) can be allocated with more parameters. 3) The compositional representation enables efficient rendering and achieves high-resolution generation. Rather than using one big volume (Bergman et al., 2022). our com- positional representation tightly models each body part and prevents wasting parameters on empty volumes. Moreover, thanks to the part-based modeling, we can efficiently sample rays inside lo- cal volumes and avoid sampling empty spaces. With the compact representation together with the efficient rendering algorithm, we achieve high-resolution (512 x 256) rendering and GAN training without using super-resolution modules, while existing methods can only train at native resolution of 1282. Moreover, we carefully design training strategies to address the human pose and viewing angle imbalance issue, We analyze the head-facing angle distribution and propose a pose-guided sampling strategy to help effective 3D human geometry learning.
Quantitative and qualitative experiments are performed on two fashion datasets (Liu et al. 2016 Fu et al.] (022) to demonstrate the advantages of EVA3D. We also experiment on UBCFashion (Zablot- skaia et al. 2019) and AIST (Tsuchida et al./ 2019) for comparison with prior work. Extensive ex- periments on our method designs are provided for further analysis. In conclusion, our contributions are as follows: 1) We are the first to achieve high- high-resolution high-quality 3D human generation from 2D image collections; 2) We propose a compositional human NeRF representation tailored for effi- cient GAN training; 3) Practical training strategies are introduced to address the imbalance issue of real 2D human image collections. 4) We demonstrate applications of EVA3D, i.e. interpolation and GAN inversion, which pave way for further exploration in 3D human GAN.
RELATED WORK
3D-Aware GAN. Generative Adversarial Network (GAN) Goodfellow et al. 2020) has been a great success in 2D image generation (Karras et al. 2019 2020). Many efforts have also been put on 3D-awaee generation. Nguyen-Phuur et al. (2019); Henzler et al. (2019) use voxels, and Pan et al. meshes to assist the 3D- 3D-aware generation. With recent advances in NeRF (Mildenhall
2020
ct al.
2020 Tewari et al. 202 I) many have build awaae GANs based on NeRF (Schwarz et al.
2020
Niemeyer & Geiger 2021 Chan al. 2021 Deng al. 2022). To increase the generation
2

---

Published as a conference paper at ICLR 2023
resolution, Gu et al. (2021): Or-EI al. (2022): Chan et al. (002) use 2D decoders for super
resolution. Moreover, it is desirable to lift the raw resolution, by improving the rendering efficiency, for more detailed geometry and better 3D consistency (Skorokhodov et al., 2022 Xiang al. 2022]
Schwarz et al. 2022 IZhao et al. 2022). We also propose an efficient 3D human representation to allow high resolution training.
Human Generation.
Though great success has been achieved in generating human faces, it is still
challenging generate human images for the complexity in human poses and appearances (Sarkar
et al. (222bb Lewis et al. 202T] |Sarkar et al. 2021a] Jiang et al. 2022c). Recently, Fu et al. (2022); Frühstück al. (2022) scale-up the dataset and achieve impressive 2D human generation results.
For 3D human generation, [henn et al. (2022) generate human geometry using 3D human dataset.
Some also attempt to train 3D human GANS using only 2D human image collections. Grigorev
et al. 2202); Zhang et al. (2021) use CNN-based neural renderers, which cannot guarantee 3D)
consistency. Noguchi et al. (2022) use human NeRF (Noguchi al. 2021) for this task, which only trains low resolution. Bergman et al. (2022); Zhang et al. (2022a) propose to increase the resolution by super-resolution, which still fails to produce high-quality results. Hong et al. (002b);
Zhang et al. 2022b) generate 3D avatars and motions from text inputs.
3D Human Representations. 3D human representations serve as fundamental tools for human related tasks. Loper et al. (2015); Pavlakos et al. (2019b) create parametric human models, for explicit modeling of 3D humans. To model human appearances, Habermann et al. (2021); Shysheya al. (019)); Yoon al. (202T): Liu al. (202T) further introduce UV maps. Parametric modeling gives robust control over the human model, but less realism. Palafox et al. (2021) use implicit
functions to generate realistic 3D human body shapes. Embracing the development of NeRF, the
number of works about human NeRF has also exploded (Peng et al.) 2021b; Zhao et al., [202T]
Peng et al. 2021a Xu et al. 2021; Noguchi al. 2222 Weng al., 2022; Chen al.) 2022] Su
et al. 20211] Jiang et al. [2022a] b, Wang et al.] 2022). Hong et al. (2022a) propose to learn modal-
invariant human representations for versatile down-stream tasks. Cai et al. (2022) contribute large- scale multi-modal 4D) human dataset. Some propose to model human body in a compositional way (Mihajlovic et al., 2022; Palafox et al., 2022; Su et al. 2022). where several submodules are used to model different body parts, and are more efficient than single-network ones.
Compositional NeRF The compositional representation has been long studied for its effectiveness and efficiency. It has also been applied to NeRF for object, scene and human modeling, Tancik al. (2022); ; Kundu et al.] (2022) model outdoor scene NeRF in an compositional way by splitting scenes into block or object levels. Yang al. (2021); Driess et al. (2022); Wang al. (2021) decompose multi-ohjects in a scene for further editing. Compositional NeRF generation has also been studied
in prior arts (Niemeyer & Geiger 2021 BR et al. 2022).
METHODOLOGY
3.1 PREREQUISITES
NeRF (Mildenhall al.) 2020) is implicit 3D representation, which is capable of photorealistic novel view synthesis. NeRF is defined as {c,a) = (2, d), where æ is the query point, d is the viewing direction, e is the emitted radiance (RGB value), a is the volume density. To get the RGB value C(r) of some ray r(t) = o + td, namely volume rendering, we have the following formulation, C(r) 1 (()) d)dt, where T(t) exp(- Jr, ((((s))ds) is the accumulated transmittance along the ray from tn to t. tn and t denotes the near and far bounds. To get the estimation of C(r), it is discretized as
C(r) Σ - xxp(--,0:))cc, where T. =exp(- o, 0, -141 4
For better geometry, Or-El et al. 2022) propose to replace the volume density (xx) with SDF
values d(x) to explicitly define the surface. SDF can be converted to the volume density axx) a -1 "iigooid -dd())/a,, where a is a learnable parameter. In later experiments, we mainly use SDF the implicit geometry representation, which is denoted for convenience.
3

---

Published as a conference paper at ICLR 2023
a) Sample Rays & Points at Observation Space SMPI,B, 0)
b) Inverse LBS to Canonical Space
c) Transform to Local Bounding
d) Integrate Queried Results A Along Rays
Box & Query Subnetworks
Figure 2: Rendering Process the Compositional Human NeRF Representation. For shape and pose specified by SMPL(33, 9). local bounding boxes are constructed. Rays that intersect with bounding boxes are sampled and transferred to the canonical space using inverse LBS. Subnetworks corresponding to bounding boxes queried, results of which are integrated to produce final ren- derings.
SMPL (Loper al. 2015). defined as MB3 0), is parametric human model, where B, 0 controls body shapes and poses. In this work, we use the Linear Blend Skinning (LBS) al- gorithm of SMPL for the transformation from the canonical space to observation spaces. For- mally, point æ in the canonical space is transformed to an observation space defined by pose as >x k= wy Gz, ()) J)æ, where K is the joint number, wh is the blend weight of æ against joint k, G, J) the transformation matrix of joint k. The transformation from observation spaces to the canonical space, namely inverse LBS, takes a similar formulation with inverted transformation matrices.
3.2 COMPOSITIONAL HUMAN NERF REPRESENTATION
The compositional human NeRF representation is defined as Fo. corresponding to set of local bounding boxes B. For each body part k; we use subnetwork Fg E F, to model the local bounding box {b mim b IB, as shown in Fig. b). For some point æ; in the canonical coordinate with direction d; and falling inside the k-th bounding box, the corresponding radiance ct and density of queried by
2x: -bb (inin b
(c) o (a d:), where at
b
b brui
If the point falls in multiple bounding boxes A,, window function (Lombardi et al. 202 is applicd to linearly blend queried results. The blended radiance c, and density o, of x; is calculated
0. = Ewa
Σ wa wattciioo J. where wa exp(-m(át + (t)" + (z)")))
(3)
agA:
m, n are chosen empirically. Different from Palafox et (2022): Su et (0222), we only query subnetworks whose bounding boxes contain query points. It increases the efficiency of the query process and saves computational resources.
Taking advantages of the compositional representation, we also adopt an efficient volume rendering algorithm. Previous methods need to sample points, query, and integrate for every pixel of the canvas, which wastes large amounts of computational resources on backgrounds. In contrast, for the compositional representation, we have pre-defined bounding boxes to filter useful rays, which also the key for method being able to train high resolution.
As shown in Fig. 2 for the target pose 0, shape 3 and camera setup, our rendering algorithm R(F,, 3, O, cam) described as follows. Firstly, ray r(t) on the canvas. Then we transform the pre-defined bounding boxes E to the target pose using transformation matrices G5 defined by SMPL. Rays that intersect with the transformed bounding boxes are kept for further rendering. Others are marked to be the background color. For ray r(t) = + ld that intersects with single multiple bounding boxes, get the near and far bounds L,, 4n
+ id is sampled for each pixel
4

---

Published as a conference paper at ICLR 2023
B
Pest
0 cam
Diseriminator
Sample Invers
Query Intoguase
Real Fake ?
cam
Sampking
SMPL(B.6)
Ps
Preal
Figure 3: 3D Human GAN F Framework. With the estimated SMPL and camera parameters distri- bution Pest. 3D humans are randomly sampled and rendered conditioned on z ~ Ps. The renderings are used for adversarial training against real 2D human image collections Preat-
N points are randomly sampled on each ray
+ i N (ty - tn), (tr kn))
t ~ U1
tn
(4)
Next, we transform sampled points back to the canonical space using inverse LBS. Similar to Zheng et al. (2021), we inverse not only the pose transformation, but also the shape/ pose blend shapes (s) (0) to able to generalize to different body shapes. For sampled point r(t;), the nearest points N (uI...uk are found among the vertices of the posed SMPL mesh (3, 0). The transformation point r(t,) from the observation space to the canonical space is defined as
x 1
Σ EGEN
Wj (M,) w
((4))) 1 1 where M, =
Σ
K
w Gk
B
+ BB) |
(5)
I
wy 1//]r(t,) wyl is the inverse distance weight. Mj is the transformation matrix of the SMPL vertex 1/, Then query the compositional human NeRF representation IF with point æ. to get its corresponding radiance c, and density o; as defined in Eg and Finally, we integrate the queried results for the RGB value of ray rt)), as defined in Eq.
3.3 3D HUMAN GAN FRAMEWORK
With the compositional human NeRF representation, we construct a 3D human GAN framework as shown in Fig, The generator defined as (2,33,0 cam; &G) R a. (z), 3, cam). Similar to pi-GAN (Chan et al.) (222)). each subnetwork of Fg. consists of stacked MLPs with SIREN activa- tion (Sitzmann al. (020). To generate fake samples, 2 ~ p= is sample from normal distribution. {,3, c.cmm ~ Pest are sampled from the estimated distribution from 2D image collections. We use off-the-shelf tools (Pavlakos et al. 2019a Kocabas et al., 2020 to estimate {3, 0, cam} for the 2D image collections. Unlike LAN
et al. 2022). where these variables are sampled from the distribution of motion datasets (Mahmood et al. 2019), the real 2D image collections do not necessarily share the similar pose distribution as that of motion datasets, especially for fashion datasets, e.g. DeepFashion, where the pose distribution is imbalanced. Finally, the fake samples I G(z, (3, 0, cam; PG)), along with real samples I, Preat are sent discriminator D(I; Pp) for adversarial training. For more implementation details, please refer to the supplementary material.
3.4 TRAINING
Delta SDF Prediction. Real-world 2D human image collections, especially fashion datasets, usu- ally have imbalanced pose distribution. For example, as shown in Fig. 6 we plot the distribution of facing angles of DeepFashion. Such heavily imbalanced pose distribution makes hard for the network to learn correct 3D information in unsupervised way. Therefore,
propose to introduce strong human prior by utilizing the SMPL template geometry dr (z) as the foundation of our human representation. Instead of directly predicting the SDF value d ((x), we predict an SDF offset Ad xx) from the template (Yifan et al. 2022). Then dy (x: + Ad(z) (x) is used as the actual SDF value of point
Pose-guided Sampling. To facilitate effective 3D information learning from sparse 2D image col- lections, other than introducing 3D human template, we propose to balance the input 2D images
5

---

Published as a conference paper at ICLR 2023
a) Render Views Control
b) Human Pose Control
Human Shapes Control
Figure 4: Generation Results of EVA3D. The 3D-aware nature and inherent human prior of EVA3D enable explicit control over rendering views, human poses, and shapes.
based on human poses. The intuition behind the pose-guided sampling is that different viewing an- gles should be sampled more evenly to allow effective learning of geometry. Empirically, among all human joints, we use the angle of the head guide the sampling. Moreover, facial areas contain more information than other parts of the head. Front-view angles should be sampled more than other angles. Therefore, we choose use a Gaussian distribution centered at the front-view angle Ho,, with a standard deviation of dg. Specifically, M bins are divided on the circle. For an image with the head angle falling in bin m, its probability Pm of being sampled is defined as
1
exp (-)
( mm 140
2
2mm
Pm
where 8,. M
gav 2T
08
We visualize the balanced distribution in Fig. 6 The network now has higher chances of seeing more side-views of human bodies, which helps better geometry generation.
Loss Functions. For the adversarial training, we use the non-saturating GAN loss with RI regular- ization Mescheder al. 2018), which is defined as
Lav ( D = Ex-p. (B.0.cam) -Pest (f [f(D(G(2, 0, cam; (c):(p)) EI ([-- + XIVD(I,; (
(8)
where /(x)= - + exp(-w)). Other than the adversarial loss, some regularization terms are introduced for the delta SDF prediction. Firstly, we want minimum offset from the template mesh to maintain plausible human shape, which gives the minimum offset loss Loff Ex A <d(<)11 [
Secondly, ensure that the predicted SDF values are physically valid (Gropp al. 2020). penalize the derivation of delta SDF predictions to zero Cek = E,, (Ad()) The overall loss
defined as Luiv A Loll + Aeik Leik, where are loss weights defined empirically.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments on four datasets: DeepFashion (Liu et al.) (016)) SHHQ (Fu jet al., 2022), UBCFashion (Zablotskaia et al., 2019) and AIST (Tsuchida et al. 2019). The first two sparse 2D image collections, meaning that each image has different identities and poses are sparse, which makes them more challenging. The last two are human video datasets containing different poses/ views of the same identities, which is easier for the task but lacks diversity.
Comparison Methods. We compare with three baselines. ENARF-GAN (Nguuhh et al. 2022 makes the first attempt at human NeRF generation from 2D image collections. EG3D (Chan et al. 2022) and StyleSDF (Or-El et al. 2022) are state-of-the-ee methods for 3D-aware generation, both requiring super-resolution modules achieve high-resolution generation.
Evaluation Metrics. To evaluate the quality of rendered images, we adopt Frechet Inception Dis-
tance (FID) (Heusel et al. 2017) and Kernel Inception Distance (KID) (Binkowski al. 2018

---

Published as a conference paper at ICLR 2023
DeepFashion
EG3D
b) StyleSDF
ENARF-GAN
d) Ours
Figure 5: Qualitative Comparison Between EVA3D and Baseline Methods. Rendered 2D im- ages and corresponding meshes are placed side-by-side. Both the 2D renderings and 3D meshes generated by our method achieve the best quality among SOTA methods. Zoom in for the best view.
Following ENARF-GAN, we use Percentage of Correct Keypoints (PCKh@0.5) (Andriluka et al. 2014) to evaluate the correctness of generated poses. Note that PCKh@0.5 can only be calculated on methods that can control generated poses, i.e. ENARF-GAR and EVA3D. To evaluate the cor- an off off-the-sheit tool (Ranfil et al. 2022) to estimate depth from the generated images and compare it with generated depths. 50K samples, padded to square and re- sized to the same resolution (DeepFashion, SHHQ, UBCFashion at 5122; AIST 256²), used to compute FID and KID. PCKh@0.5 and Depth are evaluated on 5K samples.
rectness of geometry,
4.2 QUALITATIVE EVALUATIONS
Generation Results and Controlling Ability of EVA3D. shown Fig. Ja), EVA3D capable of generating high-quality renderings in novel views and remain multi-vier consistency. Due the inherent human prior in our model design, EVA3D can control poses and shapes the generated 3D human by changing ( and A of SMPI. We show novel pose and shape generation results in Fig. b)& c). We refer readers to the supplementary PDF and video for more qualitative results.
Comparison with Baseline Methods, We show the renderings and corresponding meshes generated by baselines and our method in Fig. EG3D trained on DeepFashion, as well as StyleSDF trained SHHQ, generate reasonable RGB renderings and geometry. However, without explicit human modeling, complex human poses make it hard to align and model 3D humans in observation spaces, which leads distorted generation. Moreover, because of the use of super resolution, their geometry is only trained under low resolution (64") and therefore lacks details. EG3D trained on SHHQ and StyleSDF trained on DeepFashion fail to capture 3D information and collapse to the trivial solution of painting on billboards. Limited by the inefficient representation and computational resources, ENARF-GAN can only be trained at a resolution of 128", which leads to low-quality rendering results. Besides, lacking human prior makes ENARF-GAN hard to capture correct 3D information of human from sparse 2D image collections, which results broken meshes. EVA3D, in contrast, generates high-quality human renderings on both datasets. We also succeeded in learning reasonable 3D human geometry from 2D image collections with sparse viewing angles and poses, thanks to the strong human prior and the pose-guided sampling strategy. Due to space limitations, only show

---

Published as a conference paper at ICLR 2023
Table 1: Comparison with State-of-the-Art Methods. The training code of ENARF-GAN is im- plemented based on the official inference code.
DeepFashion
SHHQ
Methods, Resolution
FIDL
KIDI
PCKt
Depth
FID
KID4
PCKT
Depthl
HG3D, 512"
26.38
0.014
0.0779
32.96
0.033
0.0296
StyleSDF, 5122
92,40
0.136
0.0359
14.12
0.010
0.0300
ENARF-GAN*, 128
77.03
0.114
43.74
0.1151
80.54
0.102
40.17
0.1241
Ours, 5122
15.91
0.011
87.50
0.0272
11.99
0.009
88.95
0.0177
UBCFashion
AIST
Methods, Resolution
FID.
KIDJ
PCKT
Depth
FID
KIDI
PCK1
Depthl
HG3D, 5122
23.95
0.009
0.1163
34.76
0.022
0.1165
StyleSDF, 5122
18.52
0.011
0.0311
199.5
0.225
0.0236
ENARF-GAN*, 1282
73.07
0.075
42.85
0.1128
Ours, 5122
12.61
0.010
99.17
0.0090
19.40
0.010
83.15
0.0126
Table 2: Ablation Study. Depth is evaluated with SMPL depth. We report Depthx10 10 for sim- plicity.
Table 3: Trade-Off Between RGB and Geometry.
Ori
15-15
15=15
6g
Uni
0.04
Distribution
FID
Depth..
0.03
Original
15.62
3.69
0.02
Methods
FIDI
'Depth.
dg 15
15.91
3.04
Bascline, 256
31.14
3.57
dg = 30
19.05
2.58
Composite, 5122
17.81
5.02
de 45
19.56
2,65
60
120
180
300
+ Delta SDF, 5122
15.62
3.69
de 60
25.08
2.91
Figure 6: PDF of Different Pose-
+ Pose-guide, 512²
15.91
3.04
Uniform
25.82
2.92
Guided Sampling Distributions.
results of DecpFashion and SHHQ here. For visual comparisons on UBCFashion and AIST, please refer to the supplementary material.
4.3 QUANTITATIVE EVALUATIONS
As shown in Tab. our method leads all metrics in four datasets. EVA3D outperforms ENARF- GAN in all settings thanks to our high-resolution training ability. EG3D and StyleSDF, as the SOTA methods in the 3D generation, can achieve reasonable scores in some settings (e.g. StyleSDF achieves 18.52 FID on UBCFashion) for their super-resolution modules. But they also fail on some datasets (e.g. StyleSDF fails on AIST with 199.5 FID) for complexity in human poses. In the contrast, EVA3D achieves the best FID/KID scores under all settings. Moreover, unlike EG3D or StyleSDF, EVA3D can control the generated pose and achieve higher PCKh@0.5 score than ENARF-GAN For the geometry part, also achieve the lowest depth error, which shows the importance of natively high-resolution training.
4.4 ABLATION STUDIES
Ablation on Method Designs. To validate the effectiveness of our designs on EVA3D, we sub- sequently add different designs on a baseline method, which uses one large network to model the canonical space. Experiments are conducted on DeepFashion. The results are reported in Tab, 2 Limited by the inefficient representation, the "Baseline" can only be trained 256 128, which results in the worst FID score. Adding compositional design ("+Composite)) makes the network efficient enough to be trained higher resolution of 512 x 256 and achieve higher generation quality. We further introduce human prior by predicting delta SDF ("++elta SDF"), which gives the best FID score. Finally, using the pose-guided sampling ("*+Pos-guuide",, we further decrease the depth error. We refer readers to the supplementary material for qualitative evaluations of ablation studies.
Analysis on Pose-Guided Sampling. We analyze the importance of the sampling strategy in 3D human GAN training. Three types of distributions Pest are experimented, including the original dataset distribution ("Original"), pose-guided Gaussian distribution ("ag **), and pose- pose-guided guided uniform distribution ("Uniform"). The results are reported in Tab. 3 Firstly, uniform sampling is not good strategy for that the information density is different between different parts of human.
8

---

Published as a conference paper at ICLR 2023
a) Interpolation on Latent Space
b) Target
Inversion Result
Figure 7: Applications of EVA3D. a) Interpolation the latent space gives smooth transition between two samples. b) Inversion result (right) of the target image (left).
Secondly, the original distribution gives the best visual quality but the worst geometry. It could result the trivial solution of painting on billboards. Thirdly, the pose-guided Gaussian sampling can avoid damaging visual quality too much and improve geometry learning. As the standard deviation Bg increases, FID increases while the depth error decreases. Therefore, visual quality and geometry quality. In final experiments, choose dg 15 which is satisfying balance hetween the two factors.
trade-off between
4.5 APPLICATIONS
Interpolation on Latent Space. As shown Fig. 7 a), we linearly interpolate two latent codes to generate smooth transition between them, showing that the latent space learned by EVA3D is semantically meaningful. More results are provided in the supplementary video.
Inversion. We use Pivotal Tuning Inversion (PTI) (Roich et al., 2021) to inverse the target image and show the results (b.. Reasonable novel view synthesis results can be achieved. The geometry, however, fails to capture geometry details corresponding to RGB renderings, which can be caused by the second stage generator fine-tuning of PTI. Nevertheless, we demonstrate the potential of EVA3D in more related downstream tasks. For more results and comparison with baseline methods, please refer to the supplementary material.
5 DISCUSSION
To conclude, we propose high-quality unconditional 3D human generation model EVA3D that only requires 2D image collections for training. We design compositional human NeRF representation for efficient GAN training. To train on the challenging 2D) image collections with sparse viewing angles and human poses, e.g. DeepFashion, strong human prior and pose-guided sampling intro- duced for better GAN learning. On four large-scale 2D human datasets, we achieve state-of-the-ar generation results a high resolution of 512 x 256.
Limitations: 1) There still exists visible circular artifacts the renderings, which might be caused by the SIREN activation. A better base representation, e.g. tri-plane of EG3D, and 2D decoder might solve the issue. 2) The estimation of SMPL parameters from 2D image collections is not ac- curate, which leads distribution shift from the real pose distribution and possibly compromises generation results. Refining SMPL estimation during training would make good future work. 3) Limited by our tight 3D human representation, it is hard model loose garments, accessories or body parts (like hair). The apparent geometric line artifact around the neck and shoulder areas of samples from DeepFashion training (see Fig. 5 could be caused by the compositional represen- tation having trouble modeling long hair hanging down. Using separate modules handle loose parts might be a promising direction. 4) is known that state-of-the- 3D-aware generation meth- ods (Chan et al. 2022 Or-El et al. 2022) have not achieved comparable quality with that of 2D generation (Karras et al. 2020 2021). To investigate if that is still the case terms of human gener- ation, further train StyleGAN2 (Karras et al. 2020) DeepFashion. StyleGAN2 achieves 6.52 FID, which is much lower than our FID of 1. This indicates that 3D-aware generation still has a long way develop.
ACKNOWLEDGMENNTS
This study is supported by NTU NAP, MOE AcRF Tier 2 (MOE-T2EP20221-001)) and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well cash and kind contribution from the industry partner(s).

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Although the results of EVA3D are yet to the point where they can fake human eyes, we still need to be aware of its potential ethical issues. The generated 3D humans might be misused to create contents that are misleading. EVA3D can also used to invert real human images, which can be used to create fake videos real humans and cause negative social impacts. Moreover, the generated 3D humans might be biased, which is caused by the inherent distribution of training datasets. We make our best effort to demonstrate the impartiality of EVA3D in Fig. T
REPRODUCIBILITY STATEMENT
Our method is thoroughly described in Sec. Together with implementation details included the supplementary material, the reproducibility is ensured. Moreover, Our code is publicly available at https://github. com...ccohhhooooooooo cww/hongf/ateeooooo 2116/VVADD
REFERENCES
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state the art analysis. Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Alexander W Bergman, Petr Kellnhofer, Yifan Wang, Eric R Chan, David B Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. arXiv preprint arXiv:2206. 14314, 2022.
Mikolaj Binkkwski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 401, 2018.
Mallikarjun BR, Ayush Tewari, Xingang Pan, Mohamed Elgharib, and Christian Theobalt. georf:
Generative compositional radiance fields. arXiv preprint arXiv:2210.1777 17344, 2022.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, al. Humman: Multi-modal 4d human dataset for versatile sensing and modeling. arXiv preprint arXiv:2204. 13686, 2022.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 5799-5809, 2021.
Eric R Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awer 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133 2022.
Jianchuan Chen, Ying Zhang. Di Kang. Xuefei Zhe, Linchao Bao, Xu Jia, and Huchuan Lu. An- imatable neural radiance fields from monocular rgb videos. arXiv preprint arXiv:2106. 13629, 2021.
Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges. gdna: Towards generative detailed neural avatars. arXiv, 2022.
Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), pp. 10673-10683, June 2022.
Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Mare Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint 1881v:2202.11855, 2022.
Anna Frühstück, Krishna Kumar Singh, Eli Shechtman, Niloy J Mitra, Peter Wonka, and Jingwan Lu. Insetgan for full-body image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7723-7732, 2022.
10

---

Published as a conference paper at ICLR 2023
Jianglin Fu, Shikai Li, Yuming Jiang, Kwan- Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Ziwei Liu. Eaylegan-human A data-centric odyssey of human generation. arXiv preprint arXiv:2204.1188 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 631114113,-14,, 2020.
Artur Grigorev, Karim Iskakov, Anastasia lanina, Renat Bashirov, lla Zakharkin, Alexander Vakhi- tov, and Victor Lempitsky. Stylepeople: A generative model of fullbody human avatars. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5151- 5160, 2021.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.0895: 2021.
Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Real-time deep dynamic characters. ACM Transactions on Graphics (TOG), 40(4):
1-16, 2021.
Philipp Henzler, Niloy Mitra, and Tobias Ritschel. Escaping plato's 's cave: 3d shape from adver- sarial rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9984-99933, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Fangzhou Hong. Liang Pan, Zhongang Cai, and Ziwei Liu. Versatile multi-modal pre-training for human-centric perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 66156-16166, 2022a.
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2205.08535, 2022b.
Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction your digital avatar from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5605-5615, 2022a.
Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. arXiv preprint urXiv:2203.1257. 2022b.
Yuming Jiang, Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022c. doi: (0.11445/5282333330006
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 44100, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8110-8119 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5253-5263, 2020.

---

Published as a conference paper at ICLR 2023
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, An- drea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic Neural Fields: Semantic Abject-Aware Neural Scene Representation. In CVPR, 2022.
Kathleen M Lewis, Srivatsan Varadharajan, and Ira Seltimachhhr--Sieemmaaa Tryongan: Body- aware try-on via layered interpolation. ACM Transactions on Graphics (TOG), 40(4):1-00 2021.
Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Transactions on Graphics (TOG), 40(6);1-16, 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 5442-5451, 2019.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490. PMLR, 2018.
Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang. Coap: Compositional articulated occupancy of people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13201-13210, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European conference on computer vision, pp. 405-421. Springer, 2020.
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7588-7597, 2019.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pallern Recognition, pp. 1 11453-1 1464, 2021.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5762-5772, 2021.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-awar neural articulated representations. arXiv preprint arXiv:2204.0888 2022.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13503- 13513, 2022.
Pablo Palafox, Aljaž Božič, Justus Thics, Matthias Nießner, and Angela Dai. Npms: Neural para- metric models for 3d deformable shapes. arXiv preprint arXtv:2104.00702, 2021.
Pablo Palafox, Nikolaos Sarafianos, Tony Tung, and Angela Dai. Spams: Structured implicit para- metric models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12851-12861 2022.
12

---

Published as a conference paper at ICLR 2023
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy. and Ping Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:201 1.00844, 2020.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019a.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10975-10985, 2019b.
Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable neural radiance fields for human body modeling. arXiv e-prints, pp. arXiv- 2105, 2021a.
Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9054-9063, 2021b.
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transac- fions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.0577 05744, 2021.
Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, and Christian Theobalt. Style and pose control for image synthesis of humans from a single monocular view, arXiv preprint arXiv:2102.1 1263, 2021a.
Kripasindhu Sarkar, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. Humangan: A gener- ative model of humans images. arXiv preprint arXiv:2103.00 06902, 2021b.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154- 20166, 2020.
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d- aware image synthesis with sparse voxel grids. arXiv preprint arXiv:2206.07695, 2022.
Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, et al. Textured neural avatars. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2387-2397, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. arXiv preprint arXiv:2206. 10535, 2022.
Shih- Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radi- ance fields for learning human shape, appearance, and pose. Advances in Neural Information Processing Systems, 34:12278-12299 2021.
Shih- Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disentangled articulated neural body representations via graph neural networks. arXiv preprint arXiv:2205.016666 2022.
13

---

Published as a conference paper at ICLR 2023
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srini- vasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8248-8258, 2022.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, al. Ad- vances in neural rendering. arXiv preprint arXiv:2111.05844 2021.
Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information process- ing. In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pp. 501-510, Delft, Netherlands, November 2019.
Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In European conference on computer vision, volume 4, 2022.
Ziyan Wang, Timur Bagautdinov, Stephen Lombardi, Tomas Simon, Jason Saragih, Jessica Hod- gins. and Michael Zollhofer. Learning compositional radiance fields of dynamic human heads. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
5704-5713, 2021.
Chung-Y Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16210-16220) 2022.
Jianfeng Xiang, Jiaolong Yang, Yu Deng, and Xin Tong. Gram-hd. hd: 3d-consistent image generation at high resolution with generative radiance manifolds. arXiv preprint arXiv:2206.07555 2022.
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for render- ing and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 34, 2021.
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In International Conference on Computer Vision (ICCV), October 2021.
Wang Yifan, Lukas Rahmann, and Olga Korkine-hornung. -eossetyy neural shape repre- sentation with implicit displacement fields. In International Conference on Learning Representa- tions, 2022. URL ht: /ttp:://///peeeiieeeee www/teruuutt yhhhhhc5cyyy.
Jae Shin Yoon, Lingjie Liu, Vladislav Golyanik, Kripasindhu Sarkar, Hyun Soo Park, and Christian Theobalt. Pose-guie Pose-guided human animation from single image in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15039-15048, 2021.
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guided human video generation. arXiv preprint 1rX1v11910009139, 2019.
Jianfeng Zhang, Zihang Jiang. Dingdong Yang, Hongyi Xu, Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: a 3d generative model for animatable human avatars. arXiv preprint
200.00061,, 2022a.
Jichao Zhang, Enver Sangineto, Hao Tang. Aliaksandr Siarohin, Zhun Zhong, Nicu Sebe, and Wei Wang. 3d-aware semantic-guided generative model for human synthesis. arXiv preprint arXiv:2112.01422. 2021.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. /5001, 2022b.
Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Generalizable neural human radiance field from sparse inputs. arXiv preprint (xXi::/12.002780 2021.
14

---

Published as a conference paper at ICLR 2023
Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G Schwing, and Alex Colburn. Generative multiplane images: Making a 2d gan 3d-aware. In European Conference on Computer Vision, pp. 18-35. Springer, 2022.
Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence, ((66)3170031188 2021.
15


Reviews:
[{"review_id": "N0yCu1U1OAl", "reviewer": "Reviewer_bCYZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a 3D human generative model which can learn to generate full body from 2D image collections. It proposes to decompose the human body into 16 parts and represent each part with a separate network. It also adapts several strategies for efficient training and sampling.  It also provides  evaluations and shows improvements over  pervious SOTAs in both generated geometry and texture quality. \n\n", "summary_of_the_review": "Overall, I think the proposed method is sound with promising performance. Although it is not new to represent with multi-nerf, the method is well-designed and with efficient adaption. Hence I am in favor for acceptance.  ", "strength_and_weaknesses": "Strength:\n1.  the overall pipeline is well-designed and with promising performances.  The involving of SMPL as human  human priors and porposed pose-guided sampling strategy are adaptive and efficient. \n\n2.  given the evaluation metrics, the proposed method outperforms SOTA of 3D body generative model by a large margin. \n\n3.  detailed ablation for the method design and pose sampling. \n\nWeaknesses:\n1.  technical side, the novelty is kind limited, as it is not totally new to use a multi-nerf based methods to represent objects/scenes (it would also be better to add discussion about early works like BlockNerf in the related work).   \n\n2. the results of the interpolation and inversion are interesting but no baseline was provided and the results are a bit blur. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FwnQkCOZNp", "reviewer": "Reviewer_eUx6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of 3D-aware human generation from 2D images. To generate at a high resolution, EVA3D proposes to \nuse compositional multiple NeRFs to do the generation. To overcome difficulties in training with imbalanced human dataset, the paper proposes several training strategies. Results on several human dataset demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "The proposed approach is interesting. The paper is clearly written. Meanwhile, effectiveness of some techniques can be further discussed.", "strength_and_weaknesses": "## Strengths\n\nThe proposed approach is interesting and effective. Even though some techniques are not new, e.g., LBS mapping is commonly used in human reconstruction, to integrate them into generative model training is novel.\n\n## Weakness\n\nI do not find major issues in the paper. However, there are some questions I wish authors can clarify or add to make this paper stronger. See below.\n\n## Questions\n\n**1. Clarification about \"Baseline\" in Tab. 2**\n\nCan authors clarify the difference between the \"Baseline\" in Tab. 2 and StyleSDF in Tab. 1? I am a little bit confused because the performance differs quite a lot. My understanding is that Tab. 2's \"Baseline\" includes the mapping with LBS. Is this correct?\n\n**2. For the effectiveness of composition**\n\nIn Tab. 2, to demonstrate the efficacy of composition, authors show the results between $512^2$ and $256^2$. This comparison is great to show the power of composition for high resolution. However, I feel like it may be a more direct comparison if authors can show the results from just $256^2$ with composition.\n\n**3. For the issues arising from composition**\n\nFrom qualitative results, e.g., Fig. 4 in the supplementary, it seems like the composition can impose apparent artifacts around the edge of the bounding box. For example, there are apparent lines around the neck. It would be great if authors can provide more discussion and analysis of this.\n\n**4. For the training strategies**\n\nThe proposed training strategies, i.e., pose-guided training and LBS mapping from observation space to canonical space, are essentially generalizable to all baseline methods (StyleSDF and EG3D). Meanwhile, the Delta prediction can also be applied to StyleSDF. It would build a stronger paper if authors can utilize these training strategies in baselines's training. \n\nThis is somehow related to the effectiveness of composition in Question 2: the above experiments would give clearer view of how the composition works compared to baselines's representations.\n\n**5. Lacked references**\n\n- Schwarz et al., VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids. NeurIPS 2022.\n- Zhao et al., Generative multiplane images: making a 2D GAN 3D-aware. ECCV 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B-ML1sDrbwP", "reviewer": "Reviewer_zbyZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to learn 3D human from 2D image collections. By leveraging a 3D-aware generative model, the authors propose to integrate a compositional representation and a prior by SMPL, and an improved training strategy to enable digital human generation. The authors show that it is possible to sample high-resolution depth and color images of 3D human in a much better quality compared to the previous works. ", "summary_of_the_review": "- The paper demonstrates a good application of 3D-aware generative model for human generation, with the additional tweaks such as the compositional representation, SMPL prior, and pose-guided sampling to enable high-quality results. At the moment I am positive despite that I still have some minor concerns that would be good to address or discuss in the rebuttal period. ", "strength_and_weaknesses": "** Strength ** \n\n- The proposed method contributes toward improving 3D-aware generative human modeling, which is an active research direction. \n\n- The experiments are compelling for both qualitative and quantitative results. \n\n\n** Weakness ** \n\n1) The compositional representation is a minor improvement compared to existing techniques in NeRF and volume rendering. Applying it for human modeling is a good application, though. \n\n2) It is known that FID/KID of 3D-aware generative models is worse compared to 2D GAN models. It would be good to demonstrate whether this is still the case for human generation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%