Regularized Adversarial Training for Single-shot Virtual Try-On
Kotaro Kikuchi
Waseda University
Tokyo, JapanKota Yamaguchi
CyberAgent, Inc.
Tokyo, Japan

kotaro@pcl.cs.waseda.ac.jp

yamaguchi kota@cyberagent.co.jp

Edgar Simo-Serra
Waseda University
Tokyo, Japan

Tetsunori Kobayashi
Waseda University
Tokyo, Japan

ess@waseda.jp

koba@waseda.jp

Abstract
Spatially placing an object onto a background is an essential operation in graphic design and facilitates many different applications such as virtual try-on. The placing operation is formulated as a geometric inference problem forgiven foreground and background images, and has been approached by spatial transformer architecture.In this paper,
we propose a simple yet effective regularization techniqueto guide the geometric parameters based on user-defined
trust regions. Our approach stabilizes the training processof spatial transformer networks and achieves a high-quality
prediction with single-shot inference. Our proposed method
is independent of initial parameters, and can easily incorporate various priors to prevent different types of trivial solutions. Empirical evaluation with the Abstract Scenes andCelebA datasets shows that our approach achieves favorable results compared to baselines.

1. Introduction
In this paper, we consider the problem of naturally placing a new object onto a background image, such that theresulting composition looks realistic, enabling applications
such as virtual try-on. Geometrically placing an object is abasic operation in graphic design. The difficulty in object
placement is that subtle misalignment of the object to thebackground can severely hurts the design quality. Designers spend a lot of efforts in completing visually pleasinggraphics layout because of this quality requirement. Automating this operation using machine learning techniques
can help designers’ productivity and open the door for intelligent tools for creating magazine covers [8], posters [10],banners [9], and virtual try-on [4].

WarpRegularizations

Mask ratio
CNN

Warped corners

Real / Fake
CNN
sampleReal examples

Figure 1. Overview of our image compositing pipeline. Given
a foreground and background image, our single-shot placement
model produces plausible image warping parameters to place theforeground image on the background. We propose a simple yet
effective regularization to stabilize the adversarial training of thespatial transformer network.

Object placement can be formulated as the task of finding appropriate geometric warping given foreground andbackground images. One of the promising approaches is
Spatial Transformer Generative Adversarial Networks (STGAN) [6], which comprises of multiple spatial transformer
networks and adversarial training. The main idea is geometrically correcting the location of the foreground object
given the initial warping parameters. For this purpose, STGAN iteratively updates the initial warping parameters torefine resulting composite image. The drawbacks of thisiterative approach is an increase in computation time due
to costly multiple forward computations, initial parameters
highly affecting the result, it is unclear how many iterations
should be applied to a given input, and most importantly,
the adversarial training process becomes unstable. We observed that the ST-GAN model often suffers from falling
into trivial solutions such as excessive scaling and framing
out , because these solutions can easily fool the discriminator during training.
In this paper, we propose a simple yet effective regularization technique to guide the parameters based on userdefined trust regions. Our approach effectively stabilizes
the training process of spatial transformer networks, andenables an accurate single-shot inference (Figure 1), unlike
the iterative approach in ST-GAN [6]. Furthermore, our approach does not require any initial warping parameters, andcan easily incorporate different types of user-defined priors(e.g., avoiding excessive skewing) for further guidance. Our
experiments with the Abstract Scenes and CelebA datasets
shows that our proposed approach shows favorable resultscompared to the state-of-the-art approaches.
We summarize our main contributions in the following:• We propose a novel regularization technique to guide
warping parameters during the adversarial trainingof spatial transformer networks, enabling high-quality
single-shot inference of object placement for virtual tryon.
• We show that our approach achieves favorable results on
the Abstract Scenes and CelebA datasets in comparison
to existing approaches.

2. Related Work
Virtual try-on based on 2D image composition has recently started to attract the attention from the research community. Most recent approaches are based on pixel-leveltransformations of the image [4], however, it is challenging
in this setting to generate realistic high-resolution compositions without modifying the content of the image. Additionally, large amounts of training data limit the applicability. In
this paper, we do not assume the target is wearing a similar
garment to that being tried-on and learn to predict a transformation of the original image, allowing high-resolution
image composition.
Spatial Transformer Networks (STNs) [5] introduce
learnable image warping module within a deep learning approach, allowing overlaying a masked foreground image
onto a background image. The main components of STNs
are a neural network to predict a set of warp parameters anda differentiable warping function. We build upon STNs toimplement our single-shot inference.
Generative Adversarial Networks (GAN) [2] are generative models that learn a generator network G and a dis-

criminator network D. In GAN framework, a well-trained
generator network can reproduce a generative distributionthat matches the empirical distribution of a given data collection. One advantage of GAN is that the loss function
is defined by the discriminator network, and therefore does
not require labeled datasets. Unsupervised training by GAN
framework only requires data collections representing thedesired domain distribution.
Recently proposed ST-GAN [6] introduces the GAN
paradigm into STNs. ST-GAN generates the distribution of
possible updates to the current warping parameters. Since
the generator produces updates to warping, the overallmodel iteratively applies updates to the initial warping parameters to solve for the final warping. Although ST-GAN
nicely fits our purpose of object placement, there are several drawbacks arising from unstable training of an iterativemodel. In this paper, we propose a single-shot inference approach to object placement that overcomes the instability inSTNs training by proper regularization.

3. Spatial Transformer Generative Adversarial
Networks (ST-GAN)
We briefly introduce ST-GAN [6] in the following section. Given a background image IBG and a foreground image IFG with a corresponding alpha mask MFG , the processof image compositing is expressed by:
Icomp = IFG

MFG + IBG

(1 − MFG ) .

(1)

A realistic looking composition is then obtained by warpingthe foreground image with0IFG= warp IFG , pM0FG = warp MFG , p ,(2)where warp(.) is a differentiable warping function [5], usually comprised of a homography transformation and bilinear
interpolation, and p are the warping parameters.
Original ST-GAN [6] iteratively applies Spatial Transformer Networks (STN) to predict a series of warping updates. At the i-th iteration, given the input images and theprevious warping parameters pi−1 , the warping update ∆pi
and the new warping parameters pi can be written by:∆pi = Gi warp IFG , pi−1 , IBGpi = pi + ∆pi ,(3)where Gi is the i-th geometric prediction network.
ST-GAN learns the model parameters for the geometricprediction networks and the discriminator with Wasserstein
GAN [1] objective with a gradient penalty [3] to force thediscriminator to be a 1-Lipschitz function. The warping update ∆pi is constrained to lie within a trust region by introducing an additional penalty Lupdate = k∆pi k22 [6], whichavoids trivial solutions, e.g., removing the foreground andleaving only the background image. The final loss functionis written by:LD = E D Icomp (pi ) − E D(Ireal )+ λgrad · LgradLGi = −E D Icomp (pi ) + λupdate · Lupdate ,(4)(5)

where Icomp (pi ) denotes the composite image using IBG
0and IFG
warped by pi , Ireal is a real example sampled fromtraining data collections, Lgrad is a gradient penalty term [3],λgrad and λupdate are hyper-parameters to adjust the weights
for the gradient penalty term and the warping penalty term
respectively. For more detail, refer to [6].

4. Proposed Approach
Our approach aims at penalizing the warping parameters
p falling into undesirable regime. The main idea is to introduce plausible warping regions as a prior. To do this,
we introduce the regularization function f to enforce target parameters x to lie within the range of given minimum
value xmin to given maximum value xmax . The regularizer
f is defined by a rectifier function:f (x, xmin , xmax ) = ReLU(xmin − x)
+ ReLU(x − xmax ).

(6)

Let us consider two common issues in spatial transformer networks: excessive scaling and framing out . For
preventing excessive scaling, we can set a determinant ofthe (inverse) affine matrix as a representative parameter, assuming we are parameterizing the warp by an affine transformation. We use the ratio between the sum of the original mask MFG and the sum of the transformed mask M0FG ,where p is applied, to approximate the scaling factor:
P0i,j MFG (i, j)P.
(7)
r=i,j MFG (i, j)
Then, our scaling regularizer is defined by:
Lmask = f (r, rmin , rmax ).

(8)For preventing framing out, we can apply regularization tothe coordinates of the warped corners C, where p is applied,of the foreground object:
XLcoord =f (cx , cxmin , cxmax ) + f (cy , cymin , cymax ) (9)c∈C

Our final loss functions become:LD = E D Icomp (p) − E D(Ireal )+ λgrad · LgradLG = E D(Ireal ) − E D Icomp (p)(10)+ λmask · Lmask + λcoord · Lcoord ,(11)where λmask and λcoord are hyper-parameters to adjust theweights for respective regularization terms. We modify
the generator loss for more stable hyper-parameter tuning
since the size of Wasserstein GAN objective for a generator
changes as training proceeds. Note that the first regularization term Lmask may suppress framing out as well, however,
we found that using both Lmask and Lcoord is stable. It is alsostraight-forward to apply our regularization technique to penalize any summary statistics from the warping parametersp, such as skewing, etc.
Thanks to the stable adversarial learning by our loss
(Eq. (11)), we find thati, in contrast to ST-GAN’s iterative
updates, a single-shot inference model can produce highquality prediction of object placement, significantly lowering the computational cost. Prediction is then simplified tothe following:p = G IFG , IBG .
(12)
We emphasize that this single-shot inference model does
not converge without our regularization.

5. Experiments
We evaluate our approach quantitatively with the Abstract Scenes dataset [11], and qualitatively with the CelebA
datasets [7].

5.1. Abstract Scenes Evaluation
Dataset.
We use the Abstract Scenes dataset [11] toevaluate our approach in terms of the reproducibility of theground-truth placement. The dataset contains 11,000 clipart scenes of children playing. Here, we consider a task ofplacing glasses and hats in the scene. We split the dataset
into a training, validation and test set of 8,775, 1,111 and1,109 scenes respectively. We generate background imagesby placing all the objects under the target objects (glasses ora hat), and real images by rendering all of them including
targets. Background images are all resized to 144 × 144.
We create foreground images by placing and resizing target
objects onto the center of 144 × 144 transparent pixels.
Warping parameters.
We estimate three warping parameters: scaling, horizontal translation and vertical translation. We can directly regularize warping parameters with
our regularizing function f (Eq. (??)), but we use Lmask(Eq. (8)) and Lcoord (Eq. (9)) for emphasizing the generalityof the choice of these parameters.
Evaluation metrics.
We regard a frame of foregroundimage as a bounding box and compute the accuracy with
Intersection over Union (IoU) under different thresholds.
We denote this metric as IoU@θ where θ indicates the IoU
threshold. We show results for θ = .25, .5, .75.
Results. We summarize the IoU evaluation on the test set
in Table 1. ST-GAN (initial) is the evaluation at the initial
parameters, and ST-GAN (warp 5) is the result at the final

Figure 2. We find that our method is able to produce compelling more compelling results than ST-GAN.

6. Conclusions
We proposed an effective regularization technique toguide the warping parameters of Spatial Transformer. Experiments demonstrate that our approach achieves favorable
results compared to ST-GAN baseline. In the future, wewish to evaluate our approach in a more realistic virtual tryon scenario, and extend our approach to enable simultaneous placement of multiple objects.

Acknowledgement
This work was partially supported by Waseda University Leading Graduate Program for Embodiment Informatics, and JST PRESTO (Simo-Serra, Grant Number: JPMJPR1756).

References

Initial composite

5th update(b) Ours(a) ST-GAN

Figure 2. Virtual try-on results on the CelebA dataset.
Table 1. Image compositing evaluation in Abstract Scenes. For
ST-GAN (warp 5) we show the average value of 10 trials.

IoU@θMethod

0.25

0.50

0.75

ST-GAN (initial)
ST-GAN (warp 5)
Ours

0.13
0.41
0.47

0.03
0.36
0.43

0.00
0.25
0.32

warping parameters. Our method achieves higher scoresthan the maximum scores in ST-GAN while being much
more computationally efficient.

5.2. CelebA Evaluation
CelebA is a large dataset of facial images [7]. Here, weevaluate our approach in the virtual try-on task of placingeyeglasses onto faces. We only conduct qualitative evaluation since the dataset does not contain ground truth annotation. Following procedures in [6], we create an evaluation dataset which contains 152,249 training and 18,673 test
images without glasses, and 10,521 training images withglasses. We use images of 10 glasses provided by [6] asforeground image. Following [6], we use a homographytransformation for warping glasses. Results are shown in[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein Generative Adversarial Networks. In ICML, volume 70, pages
214–223, 2017.
[2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In NIPS, pages 2672–2680. 2014.
[3] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C.
Courville. Improved Training of Wasserstein GANs. InNIPS, pages 5767–5777. 2017.
[4] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S. Davis. Viton: An
image-based virtual try-on network. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), June
2018.
[5] M. Jaderberg, K. Simonyan, A. Zisserman, andK. Kavukcuoglu.
Spatial Transformer Networks.
InNIPS, pages 2017–2025. 2015.
[6] C.-H. Lin, E. Yumer, O. Wang, E. Shechtman, and S. Lucey.
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing. In CVPR, 2018.
[7] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face
Attributes in the Wild. In ICCV, 2015.
[8] X. Yang, T. Mei, Y.-Q. Xu, Y. Rui, and S. Li. Automatic
Generation of Visual-Textual Presentation Layout. ACM
Trans. Multimedia Comput. Commun. Appl., 12(2):33:1–33:22, 2016.
[9] Y. Zhang, K. Hu, P. Ren, C. Yang, W. Xu, and X.-S.
Hua. Layout Style Modeling for Automating Banner Design. In ACM Multimedia Thematic Workshops, Thematic
Workshops ’17, pages 451–459, New York, NY, USA, 2017.
ACM.
[10] N. Zhao, Y. Cao, and R. W. H. Lau. What Characterizes
Personalities of Graphic Designs? SIGGRAPH, 37(4), 2018.
[11] C. L. Zitnick and D. Parikh. Bringing Semantics into Focus
Using Visual Abstraction. In CVPR, pages 3009–3016, 2013.