Title: SEMANTICALLY DECOMPOSING THE LATENT SPACES OF GENERATIVE ADVERSARIAL NETWORKS

Abstract: We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). By fixing the identity portion of the latent codes, we can generate diverse images of the same subject, and by fixing the observation portion, we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce pairs that are photorealistic, distinct, and appear to depict the same individual. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to facilitate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm's ability to generate convincing, identity-matched photographs.

1. INTRODUCTION:

In many domains, a suitable generative process might consist of several stages. To generate a photograph of a product, we might wish to first sample from the space of products, and then from the space of photographs of that product. Given such disentangled representations in a multistage generative process, an online retailer might diversify its catalog, depicting products in a wider variety of settings. A retailer could also flip the process, imagining new products in a fixed setting. Datasets for such domains often contain many labeled identities with fewer observations of each (e.g. a collection of face portraits with thousands of people and ten photos of each). While we may know the identity of the subject in each photograph, we may not know the contingent aspects of the observation (such as lighting, pose and background). This kind of data is ubiquitous; given a set of commonalities, we might want to incorporate this structure into our latent representations.

Generative adversarial networks (GANs) learn mappings from latent codes z in some low-dimensional space Z to points in the space of natural data X (Goodfellow et al., 2014) . They achieve this power through an adversarial training scheme pitting a generative model G : Z → X against a discriminative model D : X → [0, 1] in a minimax game. While GANs are popular, owing to their ability to generate high-fidelity images, they do not, in their original form, explicitly disentangle the latent factors according to known commonalities.

In this paper, we propose Semantically Decomposed GANs (SD-GANs), which encourage a specified portion of the latent space to correspond to a known source of variation. 1,2 The technique decomposes the latent code Z into one portion Z I corresponding to identity, and the remaining portion Z O corresponding to the other contingent aspects of observations. SD-GANs learn through a pairwise training scheme in which each sample from the real dataset consists of two distinct images with a common identity. Each sample from the generator consists of a pair of images with common z I ∈ Z I but differing z O ∈ Z O . In order to fool the discriminator, the generator must not only produce diverse and photorealistic images, but also images that depict the same identity when z I is fixed. For SD-GANs, we modify the discriminator so that it can determine whether a pair of samples constitutes a match.

As a case study, we experiment with a dataset of face photographs, demonstrating that SD-GANs can generate contrasting images of the same subject (Figure 1 ; interactive web demo in footnote on previous page). The generator learns that certain properties are free to vary across observations but not identity. For example, SD-GANs learn that pose, facial expression, hirsuteness, grayscale vs. color, and lighting can all vary across different photographs of the same individual. On the other hand, the aspects that are more salient for facial verification remain consistent as we vary the observation code z O . We also train SD-GANs on a dataset of product images, containing multiple photographs of each product from various perspectives (Figure 4 ).

We demonstrate that SD-GANs trained on faces generate stylistically-contrasting, identity-matched image pairs that human annotators and a state-of-the-art face verification algorithm recognize as depicting the same subject. On measures of identity coherence and image diversity, SD-GANs perform comparably to a recent conditional GAN method (Odena et al., 2017) ; SD-GANs can also imagine new identities, while conditional GANs are limited to generating existing identities from the training data.

2. SEMANTICALLY DECOMPOSED GENERATIVE ADVERSARIAL NETWORKS:

Before introducing our algorithm, we briefly review the prerequisite concepts.

2.1. GAN PRELIMINARIES:

GANs leverage the discriminative power of neural networks to learn generative models. The generative model G ingests latent codes z, sampled from some known prior P Z , and produces G(z), a sample of an implicit distribution P G . The learning process consists of a minimax game between G, parameterized by θ G , and a discriminative model D, parameterized by θ D . In the original formulation, the discriminative model tries to maximize log likelihood, yielding

min G max D V (G, D) = E x∼P R [log D(x)] + E z∼P Z [log(1 -D(G(z)))]. (1)

Training proceeds as follows: For k iterations, sample one minibatch from the real distribution P R and one from the distribution of generated images P G , updating discriminator weights θ D to increase V (G, D) by stochastic gradient ascent. Then sample a minibatch from P Z , updating θ G to decrease V (G, D) by stochastic gradient descent. for m in 1:MinibatchSize do 3:

Sample one identity vector:

z I ∼ Uniform([-1, 1] d I ). 4: Sample two observation vectors z 1 O , z 2 O ∼ Uniform([-1, 1] d O ).

5:

z 1 ← [z I ; z 1 O ], z 2 ← [z I ; z 2 O ].

6::

Generate pair of images G(z 1 ), G(z 2 ), adding them to the minibatch with label 0 (fake).

7::

for m in 1:MinibatchSize do 8:

Sample one identity i ∈ I uniformly at random from the real data set.

9::

Sample two images of i without replacement x 1 , x 2 ∼ P R (x|I = i).

10:

Add the pair to the minibatch, assigning label 1 (real).

11:

Update discriminator weights by θ D ← θ D + ∇ θ D V (G, D) using its stochastic gradient.

12:

Sample another minibatch of identity-matched latent vectors z 1 , z 2 .

13:

Update generator weights by stochastic gradient descent Zhao et al. (2017b) propose energy-based GANs (EBGANs), in which the discriminator can be viewed as an energy function. Specifically, they devise a discriminator consisting of an autoencoder:

θ G ← θ G -∇ θ G V (G, D).

D(x) = D d (D e (x)

). In the minimax game, the discriminator's weights are updated to minimize the reconstruction error L(x) = ||x -D(x)|| for real data, while maximizing the error L(G(z)) for the generator. More recently, Berthelot et al. (2017) extend this work, introducing Boundary Equilibrium GANs (BEGANs), which optimize the Wasserstein distance (reminiscent of Wasserstein GANs (Arjovsky et al., 2017) ) between autoencoder loss distributions, yielding the formulation:

EQUATION

Additionally, they introduce a method for stabilizing training. Positing that training becomes unstable when the discriminator cannot distinguish between real and generated images, they introduce a new hyperparameter γ, updating the value function on each iteration to maintain a desired ratio between the two reconstruction errors:

E[L(G(z))] = γE[L(x)

]. The BEGAN model produces what appear to us, subjectively, to be the sharpest images of faces yet generated by a GAN. In this work, we adapt both the DCGAN (Radford et al., 2016) and BEGAN algorithms to the SD-GAN training scheme.

2.2. SD-GAN FORMULATION:

Consider the data's identity as a random variable I in a discrete index set I. We seek to learn a latent representation that conveniently decomposes the variation in the real data into two parts: 1) due to I, and 2) due to the other factors of variation in the data, packaged as a random variable O. Ideally, the decomposition of the variation in the data into I and O should correspond exactly to a decomposition of the latent space Z = Z I × Z O . This would permit convenient interpolation and other operations on the inferred subspaces Z I and Z O .

A conventional GAN samples I, O from their joint distribution. Such a GAN's generative model samples directly from an unstructured prior over the latent space. It does not disentangle the variation in O and I, for instance by modeling conditional distributions P G (O | I = i), but only models their average with respect to the prior on I.

Our SD-GAN method learns such a latent space decomposition, partitioning the coordinates of Z into two parts representing the subspaces, so that any z ∈ Z can be written as the concatenation

[z I ; z O ] of its identity representation z I ∈ R d I = Z I and its contingent aspect representation z O ∈ R d O = Z O .

SD-GANs achieve this through a pairwise training scheme in which each sample from the real data consists of x 1 , x 2 ∼ P R (x | I = i), a pair of images with a common identity i ∈ I.

Each sample from the generator consists of G(z 1 ), G(z 2 ) ∼ P G (z | Z I = z I ), a pair of images generated from a common identity vector z

I ∈ Z I but i.i.d. observation vectors z 1 O , z 2 O ∈ Z O .

We assign identity-matched pairs from P R the label 1 and z I -matched pairs from P G the label 0. The discriminator can thus learn to reject pairs for either of two primary reasons: 1) not photorealistic or 2) not plausibly depicting the same subject. See Algorithm 1 for SD-GAN training pseudocode. 

2.3. SD-GAN DISCRIMINATOR ARCHITECTURE:

With SD-GANs, there is no need to alter the architecture of the generator. However, the discriminator must now act upon two images, producing a single output. Moreover, the effects of the two input images x 1 , x 2 on the output score are not independent. Two images might be otherwise photorealistic but deserve rejection because they clearly depict different identities. To this end, we devise two novel discriminator architectures to adapt DCGAN and BEGAN respectively. In both cases, we first separately encode each image using the same convolutional neural network D e (Figure 2 ). We choose this Siamese setup (Bromley, 1994; Chopra et al., 2005) as our problem is symmetrical in the images, and thus it's sensible to share weights between the encoders.

To adapt DCGAN, we stack the feature maps D e (x 1 ) and D e (x 2 ) along the channel axis, applying one additional strided convolution. This allows the network to further aggregate information from the two images before flattening and fully connecting to a sigmoid output. For BEGAN, because the discriminator is an autoencoder, our architecture is more complicated. After encoding each image, we concatenate the representations [D e (x 1 ); D e (x 2 )] ∈ R 2(d I +d O ) and apply one fully connected bottleneck layer R 2(d I +d O ) ⇒ R d I +2d O with linear activation. In alignment with BEGAN, the SD-BEGAN bottleneck has the same dimensionality as the tuple of latent codes (z I , z 1 O , z 2 O ) that generated the pair of images. Following the bottleneck, we apply a second FC layer

R d I +2d O ⇒ R 2(d I +d O )

, taking the first d I + d O components of its output to be the input to the first decoder and the second d I + d O components to be the input to the second decoder. The shared intermediate layer gives SD-BEGAN a mechanism to push apart matched and unmatched pairs. We specify our exact architectures in full detail in Appendix E.

3. EXPERIMENTS:

We experimentally validate SD-GANs using two datasets: 1) the MS-Celeb-1M dataset of celebrity face images (Guo et al., 2016) and 2) a dataset of shoe images collected from Amazon (McAuley et al., 2015) . Both datasets contain a large number of identities (people and shoes, respectively) with multiple observations of each. The "in-the-wild" nature of the celebrity face images offers a richer test bed for our method as both identities and contingent factors are significant sources of variation. In contrast, Amazon's shoe images tend to vary only with camera perspective for a given product, making this data useful for sanity-checking our approach.

Faces From the aligned face images in the MS-Celeb-1M dataset, we select 12,500 celebrities at random and 8 associated images of each, resizing them to 64x64 pixels. We split the celebrities into subsets of 10,000 (training), 1,250 (validation) and 1,250 (test). The dataset has a small number of duplicate images and some label noise (images matched to the wrong celebrity). We detect and remove duplicates by hashing the images, but we do not rid the data of label noise. We scale the pixel values to [-1, 1], performing no additional preprocessing or data augmentation.

Shoes Synthesizing novel product images is another promising domain for our method. In our shoes dataset, product photographs are captured against white backgrounds and primarily differ in orientation and distance. Accordingly, we expect that SD-GAN training will allocate the observation latent space to capture these aspects. We choose to study shoes as a prototypical example of a category of product images. The Amazon dataset contains around 3,000 unique products with the category "Shoe" and multiple product images. We use the same 80%, 10%, 10% split and again hash the images to ensure that the splits are disjoint. There are 6.2 photos of each product on average.

3.1. TRAINING DETAILS:

We train SD-DCGANs on both of our datasets for 500,000 iterations using batches of 16 identitymatched pairs. To optimize SD-DCGAN, we use the Adam optimizer (Kingma & Ba, 2015) with hyperparameters α = 2e-4, β 1 = 0.5, β 2 = 0.999 as recommended by Radford et al. (2016) . We also consider a non-Siamese discriminator that simply stacks the channels of the pair of real or fake images before encoding (SD-DCGAN-SC).

As in (Radford et al., 2016) , we sample latent vectors z ∼ Uniform([-1, 1] 100 ). For SD-GANs, we partition the latent codes according to [25, 50, 75] . Our algorithm can be trivially applied with k-wise training (vs. pairwise). To explore the effects of using k > 2, we also experiment with an SD-DCGAN where we sample k = 4 instances each from P G (z | Z I = z I ) for some z I ∈ Z I and from P R (x | I = i) for some i ∈ I. For all experiments, unless otherwise stated, we use d I = 50 and k = 2.

z I ∈ R d I , z O ∈ R 100-d I using values of d I =

We also train an SD-BEGAN on both of our datasets. The increased complexity of the SD-BEGAN model significantly increases training time, limiting our ability to perform more-exhaustive hyperparameter validation (as we do for SD-DCGAN). We use the Adam optimizer with the default hyperparameters from (Kingma & Ba, 2015) for our SD-BEGAN experiments. While results from our SD-DCGAN k = 4 model are compelling, an experiment with a k = 4 variant of SD-BEGAN resulted in early mode collapse (Appendix F); hence, we excluded SD-BEGAN k = 4 from our evaluation.

We also compare to a DCGAN architecture trained using the auxiliary classifier GAN (AC-GAN) method (Odena et al., 2017) . AC-GAN differs from SD-GAN in two key ways: 1) random identity codes z I are replaced by a one-hot embedding over all the identities in the training set (matrix of size 10000x50); 2) the AC-GAN method encourages that generated photos depict the proper identity by tasking its discriminator with predicting the identity of the generated or real image. Unlike SD-GANs, the AC-DCGAN model cannot imagine new identities; when generating from AC-DCGAN (for our quantitative comparisons to SD-GANs), we must sample a random identity from those existing in the training data. 

3.2. EVALUATION:

The evaluation of generative models is a fraught topic. Quantitative measures of sample quality can be poorly correlated with each other (Theis et al., 2016) . Accordingly, we design an evaluation to match conceivable uses of our algorithm. Because we hope to produce diverse samples that humans deem to depict the same person, we evaluate the identity coherence of SD-GANs and baselines using both a pretrained face verification model and crowd-sourced human judgments obtained through Amazon's Mechanical Turk platform.

3.2.1. QUANTITATIVE:

Recent advancements in face verification using deep convolutional neural networks (Schroff et al., 2015; Parkhi et al., 2015; Wen et al., 2016) have yielded accuracy rivaling humans. For our evaluation, we procure FaceNet, a publicly-available face verifier based on the Inception-ResNet architecture (Szegedy et al., 2017) . The FaceNet model was pretrained on the CASIA-WebFace dataset (Yi et al., 2014) and achieves 98.6% accuracy on the LFW benchmark (Huang et al., 2012). 3 FaceNet ingests normalized, 160x160 color images and produces an embedding f (x) ∈ R 128 .

The training objective for FaceNet is to learn embeddings that minimize the L 2 distance between matched pairs of faces and maximize the distance for mismatched pairs. Accordingly, the embedding space yields a function for measuring the similarity between two faces x 1 and x 2 :

D(x 1 , x 2 ) = ||f (x 1 ) -f (x 2 )|| 2 2 .

Given two images, x 1 and x 2 , we label them as a match if D(x 1 , x 2 ) ≤ τ v where τ v is the accuracy-maximizing threshold on a class-balanced set of pairs from MS-Celeb-1M validation data. We use the same threshold for evaluating both real and synthetic data with FaceNet.

We compare the performance of FaceNet on pairs of images from the MS-Celeb-1M test set against generated samples from our trained SD-GAN models and AC-DCGAN baseline. To match FaceNet's training data, we preprocess all images by resizing from 64x64 to 160x160, normalizing each image individually. We prepare 10,000 pairs from MS-Celeb-1M, half identity-matched and half unmatched. From each generative model, we generate 5,000 pairs each with z 1 I = z 2 I and 5,000 pairs with z 1 I = z 2 I . For each sample, we draw observation vectors z O randomly. We also want to ensure that identity-matched images produced by the generative models are diverse. To this end, we propose an intra-identity sample diversity (ID-Div) metric. The multi-scale structural similarity (MS-SSIM) (Wang et al., 2004 ) metric reports the similarity of two images on a scale from 0 (no resemblance) to 1 (identical images). We report 1 minus the mean MS-SSIM for all pairs of identity-matched images as ID-Div. To measure the overall sample diversity (All-Div), we also compute 1 minus the mean similarity of 10k pairs with random identities.

In Table 1 , we report the area under the receiver operating characteristic curve (AUC), accuracy, and false accept rate (FAR) of FaceNet (at threshold τ v ) on the real and generated data. We also report our proposed diversity statistics. FaceNet verifies pairs from the real data with 87% accuracy compared to 86% on pairs from our SD-BEGAN model. Though this is comparable to the accuracy achieved on pairs from the AC-DCGAN baseline, our model produces samples that are more diverse in pixel space (as measured by ID-Div and All-Div). FaceNet has a higher but comparable FAR for pairs from SD-GANs than those from AC-DCGAN; this indicates that SD-GANs may produce images that are less semantically diverse on average than AC-DCGAN.

We also report the combined memory footprint of G and D for all methods in Table 1 . For conditional GAN approaches, the number of parameters grows linearly with the number of identities in the training data. Especially in the case of the AC-GAN, where the discriminator computes a softmax over all identities, linear scaling may be prohibitive. While our 10k-identity subset of MS-Celeb-1M requires a 131MB AC-DCGAN model, an AC-DCGAN for all 1M identities would be over 8GB, with more than 97% of the parameters devoted to the weights in the discriminator's softmax layer. In contrast, the complexity of SD-GAN is constant in the number of identities.

3.2.2. QUALITATIVE:

In addition to validating that identity-matched SD-GAN samples are verified by FaceNet, we also demonstrate that humans are similarly convinced through experiments using Mechanical Turk. For these experiments, we use balanced subsets of 1,000 pairs from MS-Celeb-1M and the most promising generative methods from our FaceNet evaluation. We ask human annotators to determine if each pair depicts the "same person" or "different people". Annotators are presented with batches of ten pairs at a time. Each pair is presented to three distinct annotators and predictions are determined by majority vote. Additionally, to provide a benchmark for assessing the quality of the Mechanical Turk ensembles, we (the authors) manually judged 200 pairs from MS-Celeb-1M. Results are in Table 1 .

For all datasets, human annotators on Mechanical Turk answered "same person" less frequently than FaceNet when the latter uses the accuracy-maximizing threshold τ v . Even on real data, balanced so that 50% of pairs are identity-matched, annotators report "same person" only 28% of the time (compared to the 41% of FaceNet). While annotators achieve higher accuracy on pairs from AC-DCGAN than pairs from SD-BEGAN, they also answer "same person" 16% more often for AC-DCGAN pairs than real data. In contrast, annotators answer "same person" at the same rate for SD-BEGAN pairs as real data. This may be attributable to the lower sample diversity produced by AC-DCGAN. Samples from SD-DCGAN and SD-BEGAN are shown in Figures 3 and 1 respectively.

4. RELATED WORK:

Style transfer and novel view synthesis are active research areas. Early attempts to disentangle style and content manifolds used factored tensor representations (Tenenbaum & Freeman, 1997; Vasilescu & Terzopoulos, 2002; Elgammal & Lee, 2004; Tang et al., 2013) , applying their results to face image synthesis. More recent work focuses on learning hierarchical feature representations using deep convolutional neural networks to separate identity and pose manifolds for faces (Zhu et al., 2013; Reed et al., 2014; Zhu et al., 2014; Yang et al., 2015; Kulkarni et al., 2015; Oord et al., 2016; Yan et al., 2016) and products (Dosovitskiy et al., 2015) . Gatys et al. (2016) use features of a convolutional network, pretrained for image recognition, as a means for discovering content and style vectors.

Since their introduction (Goodfellow et al., 2014) , GANs have been used to generate increasingly highquality images (Radford et al., 2016; Zhao et al., 2017b; Berthelot et al., 2017) . Conditional GANs (cGANs), introduced by Mirza & Osindero (2014) , extend GANs to generate class-conditional data. Odena et al. (2017) propose auxiliary classifier GANs, combining cGANs with a semi-supervised discriminator (Springenberg, 2015) . Recently, cGANs have been used to ingest text (Reed et al., 2016) and full-resolution images (Isola et al., 2017; Liu et al., 2017; Zhu et al., 2017) as conditioning information, addressing a variety of image-to-image translation and style transfer tasks. Chen et al. (2016) devise an information-theoretic extension to GANs in which they maximize the mutual information between a subset of latent variables and the generated data. Their unsupervised method 2017) propose conditional GANs for synthesizing artificially-aged faces conditioned on both a face image and an age vector. These approaches all require explicit conditioning on the relevant factor (such as rotation, lighting and age) in addition to an identity image. In contrast, SD-GANs can model these contingent factors implicitly (without supervision). 2016) combine GANs with a traditional reconstruction loss to disentangle identity. While their approach trains with an encoder-decoder generator, they enforce a variational bound on the encoder embedding, enabling them to sample from the decoder without an input image. Experiments with their method only address small (28x28) grayscale face images, and their training procedure is complex to reproduce. In contrast, our work offers a simpler approach and can synthesize higher-resolution, color photographs.

Mathieu et al. (:

One might think of our work as offering the generative view of the Siamese networks often favored for learning similarity metrics (Bromley, 1994; Chopra et al., 2005) . Such approaches are used for discriminative tasks like face or signature verification that share the many classes with few examples structure that we study here. In our work, we adopt a Siamese architecture in order to enable the discriminator to differentiate between matched and unmatched pairs. Recent work by Liu & Tuzel (2016) propose a GAN architecture with weight sharing across multiple generators and discriminators, but with a different problem formulation and objective from ours.

5. DISCUSSION:

Our evaluation demonstrates that SD-GANs can disentangle those factors of variation corresponding to identity from the rest. Moreover, with SD-GANs we can sample never-before-seen identities, a benefit not shared by conditional GANs. In Figure 3 , we demonstrate that by varying the observation vector z O , SD-GANs can change the color of clothing, add or remove sunnies, or change facial pose. They can also perturb the lighting, color saturation, and contrast of an image, all while keeping the apparent identity fixed. We note, subjectively, that samples from SD-DCGAN tend to appear less photorealistic than those from SD-BEGAN. Given a generator trained with SD-GAN, we can independently interpolate along the identity and observation manifolds (Figure 5 ).

On the shoe dataset, we find that the SD-DCGAN model produces convincing results. As desired, manipulating z I while keeping z O fixed yields distinct shoes in consistent poses (Figure 4 ). The identity code z I appears to capture the broad categories of shoes (sneakers, flip-flops, boots, etc.) . Surprisingly, neither original BEGAN nor SD-BEGAN can produce diverse shoe images (Appendix G).

In this paper, we presented SD-GANs, a new algorithm capable of disentangling factors of variation according to known commonalities. We see several promising directions for future work. One logical extension is to disentangle latent factors corresponding to more than one known commonality. We also plan to apply our approach in other domains such as identity-conditioned speech synthesis. receives either two real images with the same identity (x 1 i , x 2 i ), or (x 1 i , G d (G e (x 1 i ), z O ). We show results in Figure 8 . While these experiments are exploratory and not part of our principle investigation, we find the results to be qualitatively promising. We are not the first to propose pairwise discrimination with pairs of (real, real) or (real, fake) images in GANs (Pathak et al., 2016; Isola et al., 2017) . In addition to our AC-DCGAN baseline (Odena et al., 2017) , we tried modifying DR-GAN to only disentangle identity (rather than both identity and pose in the original paper). We used the DCGAN (Radford et al., 2016) discriminator architecture (Table 4 ) as G e , linearly projecting the final convolutional layer to G e (x) ∈ R 50 (in alignment with our SD-GAN experiments). We altered the discriminator to predict the identity of x or x, rather than pose information (which is unknown in our experimental setup). With these modifications, G e (x) is analogous to z I in the SD-GAN generator, and z is analogous to z O . Furthermore, this setup is identical to the AC-DCGAN baseline except that the embedding matrix is replaced by an encoding network G e . Unfortunately, we found that the generator quickly learned to produce a single output image x for each input x regardless of observation code z (Figure 9 ). Accordingly, we excluded this experiment from our evaluation (Section 3.2). As stated in Section 3.1, AC-GANs Odena et al. (2017) provide no obvious way to imagine new identities. For our evaluation (Section 3.2), the AC-GAN generator receives identity input z I ∈ [0, 1] 10000 : a one-hot over all identities. One possible approach to imagining new identities would be to query a trained AC-GAN generator with a random vector z I such that 10000 i=1 z I [i] = 1. We found that this strategy produced little identity variety (Figure 10 ) compared to the normal one-hot strategy (Figure 11 ) and excluded it from our evaluation.

C EXPLORATORY EXPERIMENT WITH DR-GANS:



D IMAGINING IDENTITIES WITH AC-GAN:



E ARCHITECTURE DESCRIPTIONS:

We list here the full architectural details for our SD-DCGAN and SD-BEGAN models. In these descriptions, k is the number of images that the generator produces and discriminator observes per identity (usually 2 for pairwise training), and d I is the number of dimensions in the latent space Z I (identity). In our experiments, dimensionality of Z O is always 100 -d I . As a concrete example, the bottleneck layer of the SD-BEGAN discriminator autoencoder ("fc2" in Table 6 ) with k = 2, d I = 50 has output dimensionality 150.

We emphasize that generators are parameterized by k in the tables only for clarity and symmetry with the discriminators. Implementations need not modify the generator; instead, k can be collapsed into the batch size.

For the stacked-channels versions of these discriminators, we simply change the number of input image channels from 3 to 3k and set k = 1 wherever k appears in the table. 

F FACE SAMPLES:

We present samples from each model reported in Table 1 for qualitative comparison. In each matrix, z I is the same across all images in a row and z O is the same across all images in a column. We draw identity and observation vectors randomly for these samples. et al., 2017) . This model was optimized using RMS-prop (Hinton et al.) with α = 5e-5. In our evaluation (Section 3.2), FaceNet had an AUC of .770 and an accuracy of 68.5% (at τ v ) on data generated by this model. We excluded it from Table 1 for brevity. 

:

Web demo: https://chrisdonahue.github.io/sdgan

Source code: https://github.com/chrisdonahue/sdgan

"20170214-092102" pretrained model from https://github.com/davidsandberg/facenet

Figures and Tables:

Figure 1: Generated samples from SD-BEGAN. Each of the four rows has the same identity code z I and each of the fourteen columns has the same observation code z O .

Figure 2: SD-GAN architectures and vanilla counterparts. Our SD-GAN models incorporate a decomposed latent space and Siamese discriminators. Dashed lines indicate shared weights. Discriminators also observe real samples in addition to those from the generator (not pictured for simplicity).

Figure 3: Generated samples from SD-DCGAN model trained on faces.

Figure 4: Generated samples from SD-DCGAN model trained on shoes.

Figure 5: Linear interpolation of z I (identity) and z O (observation) for three pairs using SD-BEGAN generator. In each matrix, rows share z I and columns share z O .

Figure 8: Generator with an encoder-decoder architecture trained against a pairwise discriminator. Each row shares an identity vector and each column shares an observation vector. Input image on the right.

Tran et al. (2017) propose Disentangled Representation learning-GAN (DR-GAN), an approach to face frontalization with similar setup to our SD-GAN algorithm. The (single-image) DR-GAN generator G (composition of G e and G d ) accepts an input image x, a pose code c, and a noise vector z. The DR-GAN discriminator receives either x or x = G d (G e (x), c, z). In the style of (Springenberg, 2015), the discriminator is tasked with determining not only if the image is real or fake, but also classifying the pose c, suggesting a disentangled representation to the generator. Through their experiments, they demonstrate that DR-GAN can explicitly disentangle pose and illumination (c) from the rest of the latent space (G e (x); z).

Figure 9: Generated samples from cGAN trained only to disentangle identity. Each row shares an identity vector and each column shares an observation vector; input image on the right.

Figure10: AC-DCGAN generation with random identity vectors that sum to one. Each row shares an identity vector and each column shares an observation vector.

Figure 11: AC-DCGAN generation with one-hot identity vectors. Each row shares an identity vector and each column shares an observation vector.

Figure 12: Generated samples from AC-DCGAN (four sample of real photos of ID on right)

Figure Generated samples from SD-BEGAN

Figure 21: Generated samples from SD-DCGAN

Evaluation of 10k pairs from MS-Celeb-1M (real data) and generative models; half have matched identities, half do not. The identity verification metrics demonstrate that FaceNet (FN) and human annotators on Mechanical Turk (MT) verify generated data similarly to real data. The sample diversity metrics ensure that generated samples are statistically distinct in pixel space. Data generated by our best model (SD-BEGAN) performs comparably to real data. * 1k pairs, † 200 pairs.

Table content: <table><tr><td/><td/><td/><td colspan="2">Identity Verification</td><td/><td colspan="2">Sample Diversity</td></tr><tr><td>Dataset</td><td colspan="3">Mem Judge AUC</td><td>Acc.</td><td colspan="3">FAR ID-Div All-Div</td></tr><tr><td>MS-Celeb-1M</td><td>-</td><td>FN</td><td>.913</td><td>.867</td><td>.045</td><td>.621</td><td>.699</td></tr><tr><td>AC-DCGAN</td><td>131 MB</td><td>FN</td><td>.927</td><td colspan="2">.851 .083</td><td>.497</td><td>.666</td></tr><tr><td>SD-DCGAN</td><td>57 MB</td><td>FN</td><td>.823</td><td>.749</td><td>.201</td><td>.521</td><td>.609</td></tr><tr><td>SD-DCGAN-SC</td><td>47 MB</td><td>FN</td><td>.831</td><td>.757</td><td>.180</td><td>.560</td><td>.637</td></tr><tr><td>SD-DCGAN k=4</td><td>75 MB</td><td>FN</td><td>.852</td><td>.776</td><td>.227</td><td>.523</td><td>.614</td></tr><tr><td>SD-DCGAN dI =25</td><td>57 MB</td><td>FN</td><td>.835</td><td>.764</td><td>.222</td><td>.526</td><td>.615</td></tr><tr><td>SD-DCGAN dI =75</td><td>57 MB</td><td>FN</td><td>.816</td><td>.743</td><td>.268</td><td>.517</td><td>.601</td></tr><tr><td>SD-BEGAN</td><td>68 MB</td><td>FN</td><td colspan="2">.928 .857</td><td>.110</td><td>.588</td><td>.673</td></tr><tr><td>†MS-Celeb-1M</td><td>-</td><td>Us</td><td>-</td><td>.850</td><td>.110</td><td>.621</td><td>.699</td></tr><tr><td>*MS-Celeb-1M</td><td>-</td><td>MT</td><td>-</td><td>.759</td><td>.035</td><td>.621</td><td>.699</td></tr><tr><td>*AC-DCGAN</td><td>131 MB</td><td>MT</td><td colspan="3">-.765 .090</td><td>.497</td><td>.666</td></tr><tr><td>*SD-DCGAN k=4</td><td>75 MB</td><td>MT</td><td>-</td><td>.688</td><td>.147</td><td>.523</td><td>.614</td></tr><tr><td>*SD-BEGAN</td><td>68 MB</td><td>MT</td><td>-</td><td>.723</td><td>.096</td><td>.588</td><td>.673</td></tr></table>

Input abstraction for both SD-DCGAN and SD-BEGAN generators during training (where z O is always different for every pair or set of k)

Table content: <table><tr><td colspan="2">Operation Input Shape</td><td>Kernel Size Output Shape</td></tr><tr><td>[zi; zo]</td><td>[(dI ,);(k,100-dI )]</td><td>[(dI ,);(k,100-dI )]</td></tr><tr><td>dup zi</td><td>[(dI ,);(k,100-dI )]</td><td>[(k,dI );(k,100-dI )]</td></tr><tr><td>concat</td><td>[(k,dI );(k,100-dI )]</td><td>(k,100)</td></tr></table>

SD-DCGAN generator architecture

Table content: <table><tr><td>Operation Input Shape</td></tr></table>

Peer review (iclr2018):
 Summary: This paper investigated the problem of controlled image generation. Assuming images can be disentangled by identity-related factors and style factors, this paper proposed an algorithm that produces a pair of images with the same identity. Compared to standard GAN framework, this algorithm first generated two latent variables for the pair images. The two latent variables are partially shared reflecting the shared identity information. The generator then transformed the latent variables into high-resolution images with a deconvolution decoder networks. The discriminator was used to distinguish paired images from database or paired images sampled by the algorithm. Experiments were conducted using DCGAN and BEGAN on portrait images and shoe product images. Qualitative results demonstrated that the learned style representations capture viewpoint, illumination and background color while the identity was well preserved by the identity-related representations. == Novelty & Significance == Paired image generation is an interesting topic but this has been explored to some extent. Compared to existing coupled generation pipeline such as CoGAN, I can see the proposed formulation is more application-driven. == Technical Quality == In Figure 3, the portrait images in the second row and fourth row look quite similar. I wonder if the trained model works with only limited variability (in terms of identity). In Figure 4, the viewpoint is quite limited (only 4 viewpoints are provided). I am not very convinced whether SD-GAN is a generic algorithm for controlled image generation. Based on the current results, I suspect it only works in fairly constrained settings. It would be good to know if it actually works in more challenging datasets such as SUN bedroom, CUB and Oxford Flowers. “the AC-DCGAN model cannot imagine new identities” I feel the author of this paper made an unfair argument when comparing AC-DCGAN with the proposed method. First, during training, the proposed SD-GAN needs to access the identity information and there is only limited identity in the dataset. Based on the presentation, it is not very clear how does the model generate novel identities (in contrast to simply interpolating existing identities). For example, is it possible to generate novel viewpoints in Figure 4? Missing references on conditional image generation and coupled image generation: -- Generative Adversarial Text-to-Image Synthesis. Reed et al., In ICML 2016. -- Attribute2Image: Conditional Image Generation from Visual Attributes. Yan et al., In ECCV 2016. -- Domain Separation Networks. Bousmalis et al., In NIPS 2016. -- Unsupervised Image-to-Image Translation Networks. Liu et al., In NIPS 2017. Overall, I rate this paper slightly above borderline. It showed some good visualization results on controlled image generation. But the comparison to AC-GAN is not very fair, since the identity pairs are fully supervised for the proposed method. As far as I can see, there are no clear-cut improvements quantitatively. Also, there is no comparison with CoGAN, which I believe is the most relevant work for coupled image generation.