Sequential Line Search for Generative Adversarial
Networks
Masahiro Kazama
Recruit Technologies Co.,Ltd.
Tokyo, Japanmasahiro_kazama@r.recruit.co.jp

Viviane Takahashi
Recruit Technologies Co.,Ltd.
Tokyo, Japanviviane@r.recruit.co.jp

Abstract
Generative Adversarial Networks (GAN) is a popular method to generate realistic
images. The method is used for creating new designs and artwork. However, ittakes time to generate the desired image because the relationship between theGAN’s input and output is unclear. The user generally has to try several input
vectors into the GAN in order to obtain the desired output image. In this paper,
we propose a method that can efficiently generate the desired image by applying
sequential line search to the GAN. Using this technique, the user only needs tomanipulate a single slider a few times in order to obtain the desired output image.
The utility of our method is experimented through crowdsourced experiments.

1

Introduction

Generative Adversarial Networks (GAN) [1] has recently gained attention because of their capabilityof generating very realistic images. GAN is widely used for creative tasks like creating new designsand artwork. The problem in using GAN is the uncertainty of the relationship between input andoutput. In a typical system, the input is a d-dimensional vector and the output is an image. We
propose the method of applying sequential line search to GAN, in which the user merely needs tomanipulate a single slider to obtain the desired image, as described in Figure 1. This allows the user
to create the desired image quickly. Users can use our method to create artwork.

Figure 1: Summary of the proposed method. Users manipulate a single slider several times in orderto generate the desired image (i.e., to find an optimal d-dimensional input vector).

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

2Related works

Koyama et al. [2] proposed the "sequential line search" method to improve aesthetic characteristics
of images. Using this method, users are required to manipulate a single slider several times in orderto get the desired image appearance, instead of manipulating many parameters like "contrast" or"brightness". In this paper, we apply the sequential line search method to GAN.

3Method

The purpose of our proposed method is to identify the best input vector to generate the desired image.
In our proposed method, the user moves a single slider a few times in order to obtain the output image.
The sequential line search consists of 2 steps. 1) In the first step, the user manipulates a slider. The
image changes dynamically as the slider is moved. The user should stop the manipulation at the point
when the generated image is most similar to the user’s desired image. 2) We generate a new single
slider according to the user’s previous choice by using the Bayesian optimization technique. We havethe information of the user’s past action, which indicates that the image the user chose is preferable
over other images (i.e., other points on the slider). With this information, Bayesian optimization isused to identify which points should be searched next and create a new single slider. Repeating thisprocess, users can get the desired output. The detailed process is described in the supplementarymaterial.

4Experiments

We evaluate our proposed method by crowdsourced experiments. To examine our proposed method’sutility, we ask users to choose the image they judge to be most similar to a predefined target image for5 trials. After 5 trials, we calculate the Euclidean distance between the final obtained image and thetarget image. We use this distance as an evaluation metric. For the experiments, we use the MNIST
dataset, which consists of handwritten digit data. We train the GAN for the MNIST dataset. Our
method is then compared with "Random" methods where the single slider is generated randomly.

5Results and Discussion

The result of our experiment is depicted in Table 1. We calculate the Euclidean distance between the
finally obtained image and the target image. We count the number of images under each threshold.
Hence, our proposed method can generate images which are fairly close to what the user has in mind
and we believe artists could use this method as a new creation tool.
Table 1: Results of the crowdsourced experiments. We calculate the Euclidean distance between the
finally obtained image and the target image. We count the number of images under each threshold.
Lower than 6.0
Lower than 7.0
Lower than 8.0
Lower than 9.0
Larger than 9.0

Random method
0121335

Proposed method
1352028

References[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, SherjilOzair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural
information processing systems, pages 2672–2680, 2014.
[2] Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. Sequential line search for
efficient visual design optimization by crowds. ACM Transactions on Graphics (TOG), 36(4):48,2017.

2

A

Supplementary material

A.1

Detail of method

We use the GAN generator G(z) as the image generator where z ∈ Rd is the d-dimensional input
vector and x = G(z) is the output image. Our task is to find the optimal z that generates the desired
image x. The sequential line search process for GAN is as follows.
Firstly we choose 2 random input vectors zl1 , zr1 . Next we generate a single slider whose left endpoint
1
is zl1 and right end point is zr1 . An interval point in the slider is represented as zm= azl1 + (1 − a)zr1 ,where 0 ≤ a ≤ 1.
When the user manipulates the slider, the image G(zm ) is dynamically generated. The user shouldstop the slider at the point where the output image is the most similar to the desired image. The point
11and image are represented as z+and x1+ = G(z+) respectively.
After the user chooses a point in the slider, a new single slider will be generated. The left endpoint
1zl2 in the new slider is the point chosen by the user at the previous trial (i.e., zl2 = z+). The rightendpoint will be decided from the Bradley-Terry-Luce (BTL) model and Bayesian optimization.
A.1.1

BTL model

We use the BTL model to model the user’s preference toward images.
When the user chooses the image xj = G(zj ) from the set of images S = {xi |xi = G(zi )}m
i=1 , werepresent it as zj  S \ zj . The likelihood of the situation is modeled asexp(qj /s)P (zj  S \ zj ) = Pm,
i=1 exp(qi /s)(1)where qi is the user’s preference value toward image xi and zi and s is the scaling parameter.
In our single slider situation, we can consider a slider as the set of a finite number of points.
In the experiments, we noticed that the results were not altered by a change in the number ofpoints. So we choose the minimal representation where we just use the endpoints zr , zl and thechosen point z+ , thus, S = {zr , zl , z+ }. Therefore, at each turn n we can obtain the informationnD n = {z+ zln , zrn }. Using the information D = [D 1 , D 2 , ..., D n ], we will estimate the1preference values q = [ql1 , qr1 , q+, qr2 , ...]. The estimation method is explained in the next section.
Once we have the estimated preference values q, we can use this for Bayesian optimization, being
able to calculate which points should be searched in the next trial.
A.1.2

Bayesian Optimization

Bayesian optimization used for optimizing the black box function f (z) is as follows:
max f (z).

(2)z∈Z0

Usually, the derivative f (z) cannot be easily obtained and evaluating each point of f (z) is costly.
In this scenario, Bayesian optimization is useful once it indicates which point should be evaluatednext based on the current n observed values. Therefore, optimal points can be efficiently found after
processing a few observations. f is usually modelled as a Gaussian process:0f ∼ GP (mf (z), kf (z, z )),(3)where mf is a mean function. We assume that mf = 0 because we do not have any domain specificknowledge. kf is a kernel function:()
0d1 X (zi − zi )2kf (z, z ) = θd+1 exp −,2 i=1
θi203(4)where θ is a hyperparameter, which is determined by MAP estimation, given the observed data D.
However, in this paper we predetermined the hyperparameter in advance to reduce computationalcomplexity.
The predictive distribution of f (z) is a Gaussian distribution and can be calculated analytically. f (z)is used to construct the acquisition function a(z), which determines the next point that should beevaluated. Although there are several kinds of acquisition functions, the expected improvement (EI)
is commonly used. The EI function is defined asa(z) = E [max{f (z) − f + , 0}],(5)where f is the best value among current observed data. We should evaluate z as the point thatmaximizes a(z).
+In the sequential line search, we use the estimated preference value q towards z as the f (z) andcalculate the acquisition function a(z). Then we obtain the next zEI . Finally, we set zEI as the rightendpoint of the slider.
To perform the search, we need to estimate the preference value q. The parameter can be estimatedthrough MAP estimation:q M AP=arg max p(q|D, θ)(6)q=arg max p(D|q, θ)p(q|θ)(7)

q=arg max p(D|q)p(q|θ).

(8)

q

The last equation is derived from the idea that, given the preference value q, D and θ are conditionallyindependent.
The conditional probability p(D|q) is calculated by the BTL model asYnp(D|q) =p(z+ zrn , zln ).

(9)nThe conditional probability p(q|θ) is calculated with the GP asp(q|θ) = N (q; 0, K),(10)

where 0 is the mean function and K is the covariance matrix.
The summary of sequential line search for GAN is bellow.
Algorithm 1 Sequential line search for GANfor n = 1, 2, ... doq M AP = compute_M AP _estimate(D n )zEI = arg maxz a(z)n−1Create a new single slider with the endopoints zln = z+, zrn = zEI
User manipulates the slidern
D n+1 = D n ∪ {z+ zln , zrn }end forA.2

Detail of experiments

We use the GAN generator G(z) and train it for the MNIST dataset. We set the dimension of input
vector z as 5 and the dimension of the output image as 28 × 28. We set the hyperparameter ofBayesian optimization as θi = 0.5(i = 1, ..., d + 1) and the scaling parameter of the BTL model as
s = 1. We use crowdsourcing to compare our method with the random method. The summary of therandom and proposed methods is described in Table 2.
We used Amazon Mechanical Turk and 48 workers performed the task. Our task is described inFigure 2.
4

Table 2: Comparison of random and proposed method.

N=1
N=2,3,..

RandomTwo endpoints are randomly chosen
Left endpoint is the previously chosen one
Right endpoint is randomly chosen

Proposed
Two endpoints are randomly chosen
Left endpoint is the previously chosen one
Right endpoint is zEI

Figure 2: Crowdsourced experiment.
A.3

Detail of results

Example of one user’s trial on our proposed method is described in Figure 3. We can observe that theimages generated are similar to the target image.

Figure 3: Example of one user’s trial on our proposed method.

5