Differentiable Rendering-based Pose-Conditioned Human Image Generation
Yusuke Horiuchi
Waseda University

Edgar Simo-Serra
Waseda University

Satoshi Iizuka
University of Tsukuba

Hiroshi Ishikawa
Waseda University

y.horiuchi@suou.waseda.jp

ess@waseda.jp

iizuka@cs.tsukuba.ac.jp

hfs@waseda.jp

TargetKeypoints

Abstractx,yx,yConditional human image generation, or generation ofhuman images with specified pose based on one or more reference images, is an inherently ill-defined problem, as there
can be multiple plausible appearance for parts that are occluded in the reference. Using multiple images can mitigate
this problem while boosting the performance. In this work,
we introduce a differentiable vertex and edge renderer forincorporating the pose information to realize human image
generation conditioned on multiple reference images. The
differentiable renderer has parameters that can be jointly
optimized with other parts of the system to obtain better
results by learning more meaningful shape representationof human pose. We evaluate our method on the Market1501 and DeepFashion datasets and comparison with existing approaches validates the effectiveness of our approach.

x,y…

Differentiable
RenderEncoder

Decoderx,y

SourceKeypoints

xN

Source Image
Encoder…
Figure 1. Overview of our method. From the input pose keypoints,
our differentiable renderer generates a pose condition heatmaps
that represent both keypoints and edges.

entiable heatmap renderer that takes the pose inputs. Unlike previous approaches, the input heatmap represent thepose not only through keypoints, but also through edges,
so that better conditional dependence can be learned fromthe rendering of heatmaps such as arms and legs as well as
elbows and knees. This richer representation is able to encode more details of the human body. The differentiable
renderer is parametrized and designed so it can learn fundamental parameters of the human body shape through backpropagation, improving the accuracy of the human image
generation using multiple reference images. An overview
of our approach is shown in Fig. 1.
We evaluate our approach on the Market-1501 and Deepfashion datasets and demonstrate the effectiveness of ourapproach.

1. Introduction
Generation of novel human images from a reference image has many different applications such as virtual tryon, virtual/augmented reality, art, video manipulation, etc.
However, there can be large ambiguities in the output from
occlusions in the reference image. In order to minimize theuncertainty and to focus on real world applications, here wefocus on conditioning the image generation on multiple reference images.
Most existing approaches for conditional human image
generation from another pose rely on heatmaps that encode
the pose information [5, 6, 7]. In general, this heatmap isgenerated from sparse points by fitting Gaussians or othersimple structures, often significantly impacting the performance. In this work, we propose to use a differentiable rendering engine that can learn the optimal way to generate theheatmaps from the data itself, instead of using the various
heuristics.
We build our approach upon Multiple-Source DeformableGAN (MS-DefGAN) [7], which is applicable tomultiple reference images, and we modify it to use a differ-

2. Related Work
Conditional human image generation, otherwise known
as the pose conversion, is a task to generate an image of aperson in given reference images but with a specified alternative pose [5]. Existing approaches differ in how the poseis specified, and are roughly divided into ones that use animage of another person to specify the pose and the ones
that use a 2D skeleton. In the former, procuring the imageof a person with the desired pose can be cumbersome or1even impossible. Also, the generated image tend to be influenced by the pose-specifying image’s appearance, fromwhich we only want the pose. On the other hand, using theskeleton to specify the pose makes the task more difficult
and, as a result, the quality of the generated image can suffer, though it is easier to use and more convenient from theuser’s point of view. There also exist methods that utilize
optical flow, which require moving images as the input, as
well as ones that perform texture mapping on a 3D model ofthe person prepared in advance. In this paper, we choose themore difficult approach, which is the use of skeleton input,
for versatility’s sake.

much overhead after learning.

3. Our Approach
Our method is based on MS-DefGAN [7], extended tohave a pose keypoint rendering module (Vertex Renderer)
and pose skeleton rendering module (Edge Renderer).
The pose information P for conditioning is a tuple of a
fixed number of two-dimensional points:
P = (p1 , p2 , . . . , pk ),(1)

where pi ∈ R2 . In the experiments we set k = 18.
Based on this pose information P , a vertex heatmap
HVi ∈ RH×W is rendered for each vertex Vi , and an edge
heatmap HEi ∈ RH×W is rendered for each edge Ei , where
H and W are the height and the width of the output image,respectively.

DefGAN [6] is known as a skeleton-based method. It
realizes high-performance pose conversion without using
optical flow or 3D model through cut-and-paste of feature maps to transform the original pose to the target pose,through a clever use of skip connection in the U-Net structure. Its variant SA-DefGAN [2] realizes higher-qualitytransformation that considers the entire image by the useof Global Self-Attention in DefGAN. However, since thecomputation cost of SA-DefGAN increases as the square of
the number of pixels, its use generalizes poorly to higherresolution images required in practical use. MS-DefGAN[7] is a more practical extension of DefGAN. It can use
single or multiple reference images. Using multiple images can help reduce ambiguities and increase performance.
However, MS-DefGAN has been reported to have a lower
SSIM score than VUnet [1, 7], which uses Variational Auto
Encoder to generate looks and poses. The VUnet, in turn,has the disadvantage of not generating very diverse outputs
relative to GAN-based methods, nor is it easily extended
to the multi-source context, as it largely abandons spatial
information, making it hard to correlate multiple input images.

3.1. Vertex Renderer
In the DefGAN [6], each keypoint heatmap is given as:
HVi (p) = exp − kp − pi k /σ 2(2)for each pixel coordinate p. The conditioning pose information is fed to the network as concatenation of the heat maps
in the channel direction for each pixel/node:
HV = VertexRender(P )= Concat(HV1 , HV2 , . . . , HVk )

(3)
(4)

In DefGAN, σ = 6.0 was set as a hyperparameter fromthe results obtained by cross-validation, and pose information was calculated using this. However, setting hyperparameters is very troublesome, and the optimal value canchange depending on the size of the image and the natureof the data set. Therefore, in our approach, the value of σ islearned for each vertex. This is implemented by preparing
and calculating a tensor with a sequence of numbers. Notethat it will be 0 if the keypoint is not detected.

In this paper, we propose a higher-performance conditional human image generation framework based on MSDefGAN, in which we emphasize the importance of learning to generate the shape more precisely. Our proposed
method can help in any of the DefGAN variations to provide
a better shape-generation ability. In existing research, thepose is given to the image-generating network as a pseudoimage, or a heatmap, depicting the shape of desired humanpose. This shape generation is often unsuccessful, as is thecase of [6], leading to the preference of texture cut-andpaste. It has been expressed in terms of dots, the fixed sizeof which is determined by a hyperparameter search. Our
method represents the human pose as dots for important
points such as the hand, elbows, knees, and feet, as wellas the edges that connect these points. It also automaticallylearns the size of the dots and the lengths and the thicknesses of the edges to create an appropriate pose heatmap.
Since this method only learns a small number of parameters,
it can be used in any of the above methods without adding

3.2. Edge Renderer
There is predetermined keypoint connection information
E = (e1 , e2 , . . . , e19 ), where each element ei = (ji , ki ) isa pair of vertex indices, meaning pji and pki are the endpoints of the edge. The edge is extended by the learnable
parameters αsi and αei , and the end points of the edge are
given as follows:
pstarti = pji + αsi pki − pji
(5)

(6)
pendi = pki + αei pki − pji .
Using the vertices, we determine the vector in the edge andthe orthogonal directions. The norm of the vector is ad2

Table 1. Comparison with other pose transfer methods on theMarket-1501 for different number of reference images n. Since[1, 6] are methods that supports only a single source, these are
evaluated only on input 1. MS-DefGAN [7] and our methods support multiple sources, these are evaluated even if there are multiple inputs. mSSIM and mIS indicate masked SSIM and masked
Inception Score, respectively.

1.0
0.8
1

0.6
0.4

-γ 0 γ

0.2

1-γ

1+γ

rrect function

output0.0

Figure 2. Visualization of the Edge Rendering Algorithm. The
blue arrow indicates the direction of the vector used to calculate
the inner products. The inner product of the vector from the startpoint of the arrow to each position and the blue arrow is shown inchanging colors. Black pixels represent values of 0 or less, andwhite pixels represent values of 1 or more. In this method, the partcontaining the value from 0 to 1 is drawn.

justed so that it is the reciprocal of the original length.
v straight =pend − pstart2

kp− pstart k + 
 end

0
1
0
v normal =v straight−1 0
v normal = βv 0normal2

kv 0normal k + 

(7)
(8)
(9)

IS

mSSIM

mIS

VUnet [1]
DefGAN [6]MS-DefGAN [7]
Ours

1
1
1
1

0.312
0.225
0.318
0.324

3.283
2.994
3.256
3.316

0.862
0.828
0.863
0.860

2.544
2.745
2.593
2.538

MS-DefGAN [7]
Ours

3
3

0.362
0.364

3.173
3.220

0.877
0.873

2.541
2.494

MS-DefGAN [7]
Ours

10
10

0.384
0.384

3.082
3.083

0.885
0.881

2.510
2.471

4.1. Datasets
We compare our method and pre-existing methods(VUnet, DefGAN, and MS-DefGAN) using the Market1501 and DeepFashion datasets. The poses of the persons
in the images in the datasets are estimated and annotated byusing OpenPose [?] . We create input-output pairs from images of the same person. For the Market-1501 dataset weonly use images of people with at least 13 instances, while
for the DeepFashion dataset we only use images of people
with at least 4 instances.
As a result, we created 215,750 training, 23,913 validation, and 23,491 evaluation pairs from the Market-1501
dataset, and 26,014 training, 2,900 validation, and 6,708
evaluation pairs from the DeepFashion dataset. Note that
VUnet is a VAE, so it is trained on a single image rather
than paired training data. VUnet is trained with 8,803 Market1501 mages and 20,902 DeepFashion images.

(10)

The complete edge heatmap tensor is then created by concatenating the individual heatmaps as:= Concat(HE1 , HE2 , . . . , HEk )SSIM

4. Experiments

HEi (p) =rrect ((p − pstart ) · v straight )HE = EdgeRenderer(P )nOur EdgeRenderer is thus able to automatically adjust thelength in the edge direction and the edge thickness in theperpendicular direction. A visualization of this method isshown in Fig. 2.
The edge elongation factor α’s and the edge thickening
factor β’s are learned with back-propagation. The slope ofthe edge γ is fixed to 0.2 in this study.

A rectangular heat map is created as the inner products ofeach positions and these vectors. We then employ a ramped
rectangular function (rrect) [3] to allow back-propagation
for the sloped part:· rrect ((p − pstart ) · v normal + 0.5) .

Model

(11)
(12)

The rrect function is defined as follows:0(x < −γ)



1

(x
+
γ)
(−γ
≤ x < γ)
 2γ
rrect(x) = 1(13)(γ ≤ x < 1−γ)
1− 2γ (x−1−γ) (1−γ ≤ x < 1+γ)




0
(1+γ ≤ x) 
1+γ − x= min max
,0 ,1
2γ


 
γ+x
+ min max
,0 ,1 − 1(14)2γ4.2. Training
The structure of the neural network and the loss function follow those of MS-DefGAN. Note that the MSDefGAN [7] model is pre-trained with two reference images and then re-trained before the testing using the same
number of input images as the test data. Here, for fairness,3xi , i ∈ [1..5]

xτ

[6]

[1]

MS-DefGAN [7]Ours

Mn = 1 Mn = 1 Mn = 1 Mn = 3 Mn = 5 Mn = 1 Mn = 3 Mn = 5

Figure 3. A qualitative comparison on the Market-1501 dataset. The first column shows the source images. [6] and [1] use only the firstsource image. The target poses are given by the ground truth images in column 2. In column 4, we show the results obtained by our modelfrom increasing numbers (Mn ) of source images. The source from the first column are added while increasing Mn from left to right. We
can see that the overall shapes of the persons look better in our results, such as the pants.

xi , i ∈ [1..3]

we train the models from the beginning to the end with afixed number of inputs and evaluate using the same model
without re-training. The Market-1501 dataset model istrained with 12 input images, and the DeepFashion dataset
model is trained with 3 inputs.
The training is terminated when the highest SSIM score
for the validation dataset is reached. The SSIM score is
calculated with the window size of 11 and in 8bit color.

ground truth [7]Ours

Figure 4. Qualitative results on DeepFashion Dataset.

4.3. Quantitative Comparison

Table 2. Quantitative comparison on DeepFashion dataset.

We quantitatively evaluate the output image quality ofthe models trained as in 4.2 in each input image number
setting.
As evaluation metrics, SSIM, masked SSIM, Inception
Score (IS), and masked Inception Score are used. The resultis shown in Tables 1 and 2. The higher SSIM represents
the higher degree of shape matching. The Inception Score
evaluates the diversity of generated images. VUnet [1], with
its use of VAE, significantly sacrifices the IS to enhance theSSIM. Since our method uses the renderer to generate theshape, it can enhance the SSIM without sacrificing the IS.
Our results have the best SSIM, albeit slightly.

Model

n

SSIM

IS

VUnet [1]
DefGAN [6]MS-DefGAN [7]
Ours

1
1
1
1

0.763
0.695
0.764
0.767

3.025
3.139
2.916
2.770

MS-DefGAN [7]
Ours

3
3

0.779
0.779

2.811
2.670

The outputs by the methods can be compared in Fig. 3 forthe Market-1501 dataset and in Fig. 4 for the DeepFashion
dataset. We can see that the overall shapes of the persons
look better in our results.

dering engine. Our approach encodes the human pose moreaccurately by representing it with both vertices and edges.
Furthermore, instead of relying on heuristics to generate thepose heatmaps, our approach is able to determine the generation parameters directly from the training data throughback-propagation. Experiments on the Market-1501 andDeepFashion datasets corroborate the effectiveness of ourapproach.

5. Conclusion

6. Acknowledgements

We have presented an approach for pose-conditioned human image generation based on using a differentiable ren-This work was partially supported by JSPS Grant-in-Aid
for Scientific Research (A) grant number 20H00615.

4.4. Qualitative Comparison

4References[1] Patrick Esser, Ekaterina Sutter, and Björn Ommer. A variational u-net for conditional appearance and shape generation.
In IEEE Conf. Comput. Vis. Pattern Recog., 2018. 2, 3, 4, 5
[2] Yusuke Horiuchi, Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Spectral normalization and relativistic adversarial training for conditional pose generation with selfattention. In 2019 16th International Conference on Machine
Vision Applications (MVA), pages 1–5, 2019. 2[3] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In IEEE Conf. Comput. Vis. Pattern
Recog., pages 3907–3916, 2018. 3[4] Ma Liqian, Sun Qianru, Georgoulis Stamatios, Van Gool Luc,
Schiele Bernt, and Fritz Mario. Disentangled person image
generation. In IEEE Conf. Comput. Vis. Pattern Recog., pages
99–108, 2018.
[5] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. In Adv. Neural Inform. Process. Syst., pages 405–415,
2017. 1
[6] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière,and Nicu Sebe. Deformable gans for pose-based human image
generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
1, 2, 3, 4, 5
[7] Lathuilière Stéphane, Sangineto Enver, Siarohin Aliaksandr,
and Sebe Nicu. Attention-based fusion for multi-source human image generation. In IEEE Winter Conf. Applic. of Comput. Vis., pages 428–437, 2020. 1, 2, 3, 4, 5
[8] Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing,13(4):600–612, 2004.

5