Spectral Normalization and Relativistic Adversarial Training for Conditional
Pose Generation with Self-Attention
Yusuke Horiuchi
Waseda UniversityTokyo, Japany.horiuchi@suou.waseda.jp

Edgar Simo-Serra
Waseda University
Tokyo, Japan
ess@waseda.jp

Satoshi Iizuka
University of Tsukuba
Tsukuba, Ibaraki
iizuka@cs.tsukuba.ac.jp

Hiroshi Ishikawa
Waseda University
Tokyo, Japanhfs@waseda.jp
Abstract
We address the problem of conditional image generation of synthesizing a new image of an individual
given a reference image and target pose. We base ourapproach on generative adversarial networks and leverage deformable skip connections to deal with pixel-topixel misalignments, self-attention to leverage complementary features in separate portions of the image, e.g.,arms or legs, and spectral normalization to improve thequality of the synthesized images. We train the synthesis model with a nearest-neighbour loss in combination
with a relativistic average hinge adversarial loss. We
evaluate on the Market-1501 dataset and show how our
proposed approach can surpass existing approaches in
conditional image synthesis performance.

1

poses by extracting the pose information of the input
image using off-the-shelf 2D pose estimators. We base
our approach on adversarial training, which has shown
good results in a variety of conditional image generation tasks, such as image inpainting [3].
Reference pose

Reference image

Target image

Generate
Target pose image
Target pose

Introduction

Conditional pose generation refers to synthesizing a
new image of a person, given a reference image of theperson and a target pose. It has diverse applications
such as generating animations from single images andvisualizing fashion outfits. Due to having to generatelarge occluded parts of the image, it poses a very challenging problem. Not only is it necessary to synthesize
new limbs, but it is also important to generate novel
views of the garments worn by the person.
In this paper, we tackle this problem by buildingon the Deformable GAN [12] model, which is an approach that uses a fully convolutional network with
deformable skip connections. We use this architecture
as a base, and propose incorporating self-attention [15],which can leverage the similarity between non-local
parts of the image, unlike standard convolution operations. Furthermore, we show the advantage of using
the nearest-neighbor loss in combination with a relativistic average hinge adversarial loss [5], jointly with
spectral normalization [9], instead of training with anearest-neighbor and adversarial loss. Our approachis able to generate new images of persons with given

Figure 1. The conditional pose generation task.
Given a reference image of a person, the pose of
the person, and a target pose, we generate animage of the person in the target pose. In thiswork, the pose of the person in the input image
is automatically extracted from the input image.
We evaluate our approach on the Market-1501
dataset, and compare against existing approaches. Results show that our model compares favorably withexisting approaches, evaluated with image generation
metrics such as the Inception score.

2Related Work

Recently,
Generative Adversarial Networks(GANs) [2] have become one of the most popular deep network-based generative models, which havebeen applied to various image generation tasks, suchas image super-resolution [7] and image inpainting [3].

same posetarget posetarget image

image

GeneratorNetworktarget pose

Fake

reference image reference poseRealreference image reference pose

estimate

Discriminator
Network

Fake orReal

attention maps
ground truth

Figure 2. Task overview. The generator network is trained using nearest-neighbor loss and adversarial loss
with correct images.

In particular, Isola et al. [4] proposed a framework ofconditional GANs for image translation, which converts a given image into another image. Although
it shows several plausible image translations such asmaps to aerial photos, it has difficulty in addressing
large deformations between the input and output images, because the network is trained to spatially sharethe low-level information between the paired images.
For image translation with large spatial deformations, Ma et al. [8] proposed the pose-guided person
generation network which allows synthesizing person
images in arbitrary pose. This approach contains twostages: the first stage focuses on pose integration andgeneration based on the U-Net [10]-like convolutional
network; and the second stage refines the initial result via adversarial training. Siarohin et al. [12] further improved the pose-based human image generationby incorporating deformable skip connections that canmove local information according to the structural deformations. They also introduced a nearest-neighborloss, instead of the more common losses such as L1and L2 , in order to match the details of the model
outputs and the target images. Unlike the previous
methods, it can be trained end-to-end, while achieving
qualitatively better results.

3.1

3

where N (p) is a neighborhood of p, and pre-trained
VGG19 [13] network with respect to the spatial position p.
In addition to the nearest-neighbor loss LNN , a conditional adversarial loss is used and defined asApproach

Our approach builds on Deformable GAN(DGAN) [12] and improves it by incorporating
spectral normalization [9] and self-attention [15],as well as changing the loss function from the standard Generative Adversarial Network (GAN) to theRelativistic Hinge GAN [5].

Deformable GAN

The Deformable GAN extends the Generative Conditional Networks by incorporating deformable skip
connections in the generator network and employs anearest-neighbor loss instead of the commonly used L1and L2 losses. Our model is based on the DeformableGAN model and unless we note otherwise, we use thesame architecture.
For a given input person image and a target pose,
DGAN first extracts the pose of the person in the formof 2D skeleton with a human pose estimator model [1].
The data is then processed with a fully convolutional
network encoder to obtain a feature representation of
the input image, the extracted pose, and the target
pose. Afterwards, using the pose information for eachspecific body part, an affine transformation is computed and applied to “move” the feature-map content
corresponding to that body part.
For training, instead of the commonly used L1 andL2 losses, a nearest neighbor loss is used. For an input
image x and a target image x∗ , it is defined asX
LNN (x, x∗ ) =min kCx∗ (p) − Cx (q)k1 , (1)p∈x∗q∈N (p)LcGAN = E(x,x∗ )∈X [log D(x, H, x∗ , H ∗ )]+ E(x,x∗ )∈X,z∈Z [log(1 − D(x, H, x̂, Ĥ) , (2)where E denotes the expectation value, X is the setof training pair images, D is the discriminator modelthat outputs a probability, x̂ = G(z, x, H, H ∗ ), G is
the generator model, Z is a random distribution, andH, H ∗ , and Ĥ are the 2D skeletons of x, x∗ , and x̂,respectively.
The objective function for optimizing the Deformable GAN model thus becomes:arg min max LcGAN (G, D) + λLNN (G) ,GD(3)

where LNN (G) = E(x,x∗ )∈X,z∈Z LNN (x̂, x∗ ) .

3.2

Spectral Normalization

Spectral normalization [9] normalizes the spectral
norm of the weight matrix W of a layer so that it satisfies the Lipschitz constraint σ(W ) = 1:
W̄SN (W ) =Wσ(W )(4)Instead of naïvely applying singular value decomposition to compute σ(W ), the power iteration method
is used to estimate σ(W ) with a small computational
footprint. The spectral normalization is used in both
the discriminator and the decoder part of the generator, and improves results by decreasing the degradationof the error signal during back-propagation.

3.3

Self-Attention

Convolutional layers process information in only alocal neighborhood, making fully convolutional networks computationally inefficient for modeling longrange dependencies in images, e.g., human arms or legs.
In a self-attention layer [15], the image features fromthe previous layer φ ∈ RC×N with N channels are firsttransformed into two features spaces g(φ) = Wg φ andh(φ) = Wh φ, where Wg and Wh are learnable matrices. Let spatial locations or regions indexed by u, v.
Then the attention map β is defined by:exp(su,v )βv,u = PN,
u=1 exp(suv )suv = g(φu )| h(φv ).

NXβv,u Wo φu + φv ,3.4

(6)

u=1

where Wo is another learnable matrix and γ is a parameter that is initialized to 0 and gradually increased
during training to learn to assign more weight to nonlocal features. The self attention learns the parameters

Relativistic Average Hinge Adversarial Loss

Training with conditional adversarial loss is donewith a discriminator that is being trained to classify
whether the input is real or fake with negative loglikelihood. Relativistic adversarial losses [5] optimize
the probability that a given real data is more realistic than a randomly sampled fake data and vice versa.
This leads to more stable and robust training in general. Relativistic adversarial training can be done bymodifying the output of the discriminator D(·) to beD(x0 ) − Ex̂∈XT D(x̂)
if x0 is real0D̄(x ) =(7)0∗
D(x ) − Ex∗ ∈XI D(x ) if x0 is fake ,where we denote the set of input images as XI and theset of target images as XT , with X = (XI , XT ).
The discriminator and generator losses can then bewritten as∗
LRHD (G, D) = Ex∗ ∈XT max(0, 1 − D̄(x ))+ Ex∈XI max(0, 1 + D̄(x̂))

LRHG (G, D) = Ex∈XI max(0, 1 − D̄(x̂))
+ Ex∗ ∈XT max(0, 1 + D̄(x∗ )) .
(8)
We note that this leads to a min-min problem, unlike
the standard adversarial loss formulation, which leads
to a min-max problem.

3.5

Training

Replacing the DGAN objective function (3), wetrain our model by using the nearest-neighbor loss LNN
in conjunction with the relativistic average hinge adversarial loss by minimizing
RHarg min LNN (G) + λ1 LRHG (G, D) + λ2 LD (G, D) (9)(5)

Then βv,u indicates to what extent the layer attends
to the u-th location when synthesizing the v-th region.
The output of the layer o = (o1 , . . . , oN ) ∈ RC×N canthen be computed asov = γWg ∈ RC̄×C , Wh ∈ RC̄×C , and Wo ∈ RC×C with backpropagation during training.
We add a self-attention after each of the last twolayers of the generator’s decoder and after each of thelast two layers of the discriminator.

G,Dwhere λ1 and λ2 are two hyperparameters.

4Experimental Results

We evaluate using the Market-1501 [16] dataset
which contains 32,668 images of 1,501 individuals captured by 6 different surveillance cameras. Following [12], the dataset is cleaned up by automaticallyremoving images in which no individual is detected using HPE [1], and training with pairs of images of thesame individual in two different poses. This results in263,631 training pairs, of which we use 2,000 for validation and 10,000 for testing. No person appears inTable 1. Comparison with existing approaches.
We compare with the results of Defomable GAN(DGAN) [12], both the result reported in theirpaper trained for 90 epochs, and the result ofretraining their model and using validation tochoose the best model. The best results are highlighted in bold.
ModelISSSIM

L1

DGAN [12] (paper)
DGAN [12] (retrained)
Ours

3.185
3.272
3.402

0.290
0.274
0.279

0.292
0.288

Real-Data

3.86

1.00

0.00

Table 2. Ablation results. We compare different
variants of our proposed approach. SN: Spectral
Normalization, RH: Relativistic Hinge loss, SA:
Self-Attention. The best results are highlighted
in bold.

5
more than one splits. In contrast to [12], which trained
all models for a fixed number of iterations, we train
for 50,000 iterations and use the model that has thelargest Inception Score (IS) [11] on the validation set
which we then use to evaluate on the testing images.
This reduces the effect of the stochastic nature of training generative adversarial network models during evaluation. We evaluate using three metrics, Inception
Score (IS) [11], SSIM [14], and L1 distance, between
the ground truth and the output of the model. We use
λ1 = λ2 = 50 for our model and train by optimizing
Eq. (9) with ADAM [6].

4.1Comparison with existing approaches

We compare against the approach of Deformable
GAN (DGAN) [12] by retraining their model and using
the validation set to choose their best model. For reference, we also show their results as reported in their paper. Results are shown in Table 1. Retraining DGAN
shows improved performance in Inception Score (IS),although SSIM decreases. Our approach outperformssignificantly in IS, and although it beats the SSIM of
the retrained model, it underperforms in comparison tothe originally reported values. However, we found thatremoving the self-attention layers improves the SSIM
at the cost of IS, outperforming the original values reported in [12].

4.2

Ablation study

We perform an ablation study to analyze the different effects of the components of our model. We show
the results in Table 2. We can see that the combination of all the elements gives a large boost in IS, while
not using Self-Attention (SA) improves SSIM.

4.3

Qualitative results

We show some qualitative results of our approach inFig. 3. The output result shows that the overall qualityof the image is improved. In particular, the quality ofbackground improved by using Self-Attention.

ModelIS

SSIM

L1

Ours SN
Ours RH
Ours SA
Ours SN+RH
Ours SA+SN+RH

3.066
3.122
3.121
2.973
3.402

0.289
0.283
0.278
0.296
0.279

0.289
0.289
0.295
0.288
0.288

Conclusions

We have presented an approach for the conditional
generation of images of individuals given a single reference image and a target pose. Our model incorporates deformable skip connections and self-attention,and is trained with a nearest-neighbour loss in combination with a relativistic average hinge adversarialloss, using spectral normalization. We have evaluatedon the Market-1501 dataset and results compare favorable with existing approach.

6Acknowledgements

This work was partially supported by JST
ACT-I (Iizuka, Grant Number: JPMJPR16U3),
JST PRESTO (Simo-Serra, Grant Number: JPMJPR1756), and JST CREST (Ishikawa, Iizuka, andSimo-Serra, Grant Number: JPMJCR14D1).

References[1] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.
Realtime multi-person 2d pose estimation using partaffinity fields. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[2] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. Generative adversarialnets. In Conference on Neural Information Processing
Systems, 2014.
[3] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.
Globally and locally consistent image completion. ACM
Transactions on Graphics, 36(4):107:1–107:14, July 2017.
[4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A.
Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conference on Computer
Vision and Pattern Recognition, pages 5967–5976, 2017.
[5] Alexia Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard gan. arXiv
preprint arXiv:1807.00734, 2018.
[6] Diederik P Kingma and Jimmy Ba. Adam: A method
for stochastic optimization. In International Conference on Learning Representations, 2015.

[7] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE Conference on Computer Vision andPattern Recognition, pages 105–114, 2017.
[8] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne
Tuytelaars, and Luc Van Gool. Pose guided person image generation. In Conference on Neural Information
Processing Systems, pages 405–415, 2017.
[9] Takeru Miyato, Toshiki Kataoka, Masanori Koyama,and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[10] O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and ComputerAssisted Intervention (MICCAI), volume 9351, pages
234–241, 2015.
[11] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki
Cheung, Alec Radford, and Xi Chen. Improved tech-niques for training gans. In Conference on Neural Information Processing Systems, 2016.
[12] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe. Deformable gans for posebased human image generation. In IEEE Conference
on Computer Vision and Pattern Recognition, 2018.
[13] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. InInternational Conference on Learning Representations,2015.
[14] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, andEero P. Simoncelli. Image quality assessment: Fromerror visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
[15] Han Zhang, Ian Goodfellow, Dimitris Metaxas, andAugustus Odena. Self-attention generative adversarialnetworks. arXiv preprint arXiv:1805.08318, 2018.
[16] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang,
Jingdong Wang, and Qi Tian. Scalable person reidentification: A benchmark. In International Conference on Computer Vision, 2015.

InputGround Truth
Baseline
Ours
Attention Map
Input
Ground Truth
Baseline
Ours
Attention Map

Figure 3. Examples of generated images and attention map. Attention map is sampled from random coordinates. Baseline is output by Deformable GAN [12]