Title: GMM-UNIT: UNSUPERVISED MULTI-DOMAIN AND MULTI-MODAL IMAGE-TO-IMAGE TRANSLATION VIA ATTRIBUTE GAUSSIAN MIXTURE MODELLING

Abstract: Unsupervised image-to-image translation aims to learn a mapping between several visual domains by using unpaired training pairs. Recent studies have shown remarkable success in image-to-image translation for multiple domains but they suffer from two main limitations: they are either built from several two-domain mappings that are required to be learned independently and/or they generate lowdiversity results, a phenomenon known as model collapse. To overcome these limitations, we propose a method named GMM-UNIT based on a content-attribute disentangled representation, where the attribute space is fitted with a GMM. Each GMM component represents a domain, and this simple assumption has two prominent advantages. First, the dimension of the attribute space does not grow linearly with the number of domains, as it is the case in the literature. Second, the continuous domain encoding allows for interpolation between domains and for extrapolation to unseen domains. Additionally, we show how GMM-UNIT can be constrained down to different methods in the literature, meaning that GMM-UNIT is a unifying framework for unsupervised image-to-image translation.

1. INTRODUCTION:

Translating images from one domain into another is a challenging task that has significant influence on many real-world applications where data are expensive, or impossible to obtain and to annotate. Image-to-Image translation models have indeed been used to increase the resolution of images (Dong et al., 2014) , fill missing parts (Pathak et al., 2016) , transfer styles (Gatys et al., 2016) , synthesize new images from labels (Liu et al., 2017) , and help domain adaptation (Bousmalis et al., 2017; Murez et al., 2018) . In many of these scenarios, it is desirable to have a model mapping one image to multiple domains, while providing visual diversity (i.e. a day scene ↔ night scene in different seasons). However, the existing models can either map an image to multiple stochastic results in a single domain, or map in the same model multiple domains in a deterministic fashion. In other words, most of the methods in the literature are either multi-domain or multi-modal.

Several reasons have hampered a stochastic translation of images to multiple domains. On the one hand, most of the Generative Adversarial Network (GAN) models assume a deterministic mapping (Choi et al., 2018; Pumarola et al., 2018; Zhu et al., 2017a) , thus failing at modelling the correct distribution of the data (Huang et al., 2018) . On the other hand, approaches based on Variational Auto-Encoders (VAEs) usually assume a shared and common zero-mean unit-variance normally distributed space (Huang et al., 2018; Zhu et al., 2017b) , limiting to two-domain translations.

In this paper, we propose a novel image-to-image translation model that disentangles the visual content from the domain attributes. The attribute latent space is assumed to follow a Gaussian mixture model (GMM), thus naming the method: GMM-UNIT (see Figure 1 ). This simple assumption allows four key properties: mode-diversity thanks to the stochastic nature of the probabilistic latent model, multi-domain translation since the domains are represented as clusters in the same attribute spaces, scalability because the domain-attribute duality allows modeling a very large number of domains without increasing the dimensionality of the attribute space, and few/zero-shot generation since the continuity of the attribute representation allows interpolating between domains and extrapolating to unseen domains with very few or almost no observed data from these domains. The code and models will be made publicly available. Figure 1 : GMM-UNIT working principle. The content is extracted from the input image (left, purple box), while the attribute (turquoise box) can be either sampled (top images) or extracted from a reference image (bottom images). Either way, the generator (blue box) is trained to output realistic images belonging to the domain encoded in the attribute vector. This is possible thanks to the disentangled attribute-content latent representation of GMM-UNIT and the generalisation properties associated to Gaussian mixture modeling.

2. RELATED WORK:

Our work is best placed in the literature of image-to-image translation, where the challenge is to translate one image from a visual domain (e.g. summer) to another one (e.g. winter). This problem is inherently ill-posed, as there could be many mappings between two images. Thus, researchers have tried to tackle the problem from many different perspectives. The most impressive results on this task are undoubtedly related to GANs, which aim to synthesize new images as similar as possible to the real data through an adversarial approach between a Discriminator and a Generator. The former continuously learns to recognize real and fake images, while the latter tries to generate new images that are indistinguishable from the real data, and thus to fool the Discriminator. These networks can be effectively conditioned and thus generate new samples from a specific class (Chen et al. (2016) ) and a latent vector extracted from the images. For example, Isola et al. (2017) and Wang et al. ( 2018) trained a conditional GAN to encode the latent features that are shared between images of the same domain and thus decode the features to images of the target domain in a one-toone mapping. However, this approach is limited to supervised settings, where pairs of corresponding images in different domains are available (e.g. a photos-sketch image pair). In many cases, it is too expensive and unrealistic to collect a large amount of paired data.

Unsupervised Domain Translation.:

Translating images from one domain to another without a paired supervision is particularly difficult, as the model has to learn how to represent both the content and the domain. Thus, constraints are needed to narrow down the space of feasible mappings between images. Taigman et al. (2017) proposed to minimize the feature-level distance between the generated and input images. Liu et al. (2017) created a shared latent space between the domains, which encourages different images to be mapped in the same latent space. Zhu et al. (2017a) proposed CycleGAN, which uses a cycle consistency loss that requires a generated image to be translated back to the original domain. Similarly, Kim et al. (2017) used a reconstruction loss applying the same approach to both the target and input domains. Mo et al. (2019) later expanded the previous approach to the problem of translating multiple instances of objects in the same image. All these methods, however, are limited to a one-to-one domain mapping, thus requiring training multiple models for cross-domain translation. Recently, Choi et al. (2018) proposed StarGAN, a unified framework to translate images in a multi-domain setting through a single GAN model. To do so, they used a conditional label and a domain classifier ensuring network consistency when translating between domains. However, StarGAN is limited to a deterministic mapping between domains.

Style transfer.:

A related problem is style transfer, which aims to transform the style of an image but not its content (e.g. from a photo to a Monet painting) to another image (Gatys et al., 2015; Huang & Belongie, 2017; Tenenbaum & Freeman, 1997; Donahue et al., 2018) . Differently from domain translation, usually the style is extracted from a single reference image. We willshow that our model could be applied to style transfer as well.

Multi-modal Domain Translation. Most existing image-to-image translation methods are deterministic, thus limiting the diversity of the translated outputs. However, even in a one-to-one domain translation such as when we want to translate people's hair from blonde to black, there could be multiple hair styles that are not modeled in a deterministic mapping. The straightforward solution would be to inject noise in the model, but it turned out to be worthless as GANs tend to ignore this injected noise (Mathieu et al., 2015; Isola et al., 2017; Zhu et al., 2017b) . To address this prob-lem, Zhu et al. (2017b) proposed BicycleGAN, which encourages the multi-modality in a paired setting through GANs and Variational Auto-Encoders (VAEs). Almahairi et al. (2018) have instead augmented CycleGAN with two latent variables for the input and target domains and showed that it is possible to increase diversity by marginalizing over these latent spaces. Huang et al. (2018) proposed MUNIT, which assumes that domains share a common content space but different style spaces. Then, they showed that by sampling from the style space and using Adaptive Instance Normalization (AdaIN) (Huang & Belongie, 2017) , it is possible to have diverse and multimodal outputs. In a similar vein, Ma et al. (2019) focused on the semantic consistency during the translation, and applied AdaIN to the feature-level space. Recently, Mao et al. (2019) proposed a mode seeking loss to encourage GANs to better explore the modes and help the network avoiding the mode collapse.

Altogether, the models in the literature are either multi-modal or multi-domain. Thus, one has to choose between generating diverse results and training one single model for multiple domains.

Here, we propose a unified model to overcome this limitation. Concurrent to our work, DRIT++ (Lee et al. (2019) ) also proposed a multi-modal and multi-domain model using a discrete domain encoding and assuming, however, a zero-mean unit-variance Gaussian shared space for multiple modes. We instead propose a content-attribute disentangled representation, where the attribute space fits a GMM distribution. A variational loss forces the latent representation to follow this GMM, where each component is associated to a domain. This is the key to provide for both multi-modal and multi-domain translation. In addition, GMM-UNIT is the first method proposing a continuous encoding of the domains, as opposed to the discrete encoding used in the literature. This is important because it allows for domain interpolation and extrapolation with very few or no data (few/zero-shot generation). The main properties of GMM-UNIT compared to the literature are shown in Table 1 . (Zhu et al., 2017b) None MUNIT (Huang et al., 2018) None StarGAN (Choi et al., 2018) Discrete DRIT++ (Lee et al., 2019) Discrete GMM-UNIT (Proposed) Continuous

3. GMM-UNIT:

GMM-UNIT is an image-to-image translation model that maps an image to multiple domains in a stochastic fashion. Following recent seminal works (Huang et al., 2018; Lee et al., 2018) , our model assumes that each image can be decomposed in a domain-invariant content space and a domainspecific attribute space. In this paper, we model the attribute latent space through Gaussian Mixture Models (GMMs), formally with a K-component Z-dimensional GMM:

EQUATION

where z ∈ R Z denotes a random attribute vector sample, φ k , µ µ µ k and Σ Σ Σ k denote respectively the weight, mean vector and covariance matrix of the k-th GMM component (φ k ≥ 0,

K k=1 φ k = 1, µ µ µ k ∈ R Z and Σ Σ Σ k ∈ R Z×Z is

symmetric and positive definite). p(z) denotes the probability density of this GMM at z. In the proposed representation, the domains are Gaussian components in a mixture. This simple yet effective model has two prominent advantages. Differently from previous works where each domain is a category and the one-hot vector representation grows linearly with the number of domains, we can encode many more domains than the dimension of the latent attribute space Z. Moreover, the continuous encoding of the domains we are introducing in this paper allows us to navigate in the attribute latent space, thus generating images corresponding to domains that have never (or very little) been observed and allowing to interpolate between two domains.

We note that the state of the art models can be traced back as a particular case of GMMs. Existing multi-domain models such as Choi et al. (2018) or Pumarola et al. (2018) can be modelled with K = |domain in the training data| and ∀k Σ Σ Σ k = 0, thus only allowing the generation of a single result per domain translation. Then, when K = 1, µ µ µ = 0, and Σ Σ Σ = I it is possible to model the state of the art approaches in multi-modal translation (Huang et al., 2018; Zhu et al., 2017b) , which share a unique latent space where every domain is overlapped, and it is thus necessary to train N (N -1) models to achieve the multi-domain translation. Finally, we can obtain the approach of Lee et al. (2019) by separating the latent space from the domain code. The former is a GMM with K = 1, µ µ µ = 0, and Σ Σ Σ = I, while the latter is another GMM with K = |domain in the training data| and ∀k Σ Σ Σ k = 0. Thus, our GMM-UNIT is a generalization of the existing state of the art. the In the next sections, we formalize our model and show that the use of GMMs for the latent space allows learning multi-modal and multi-domain mappings, and also few/zero-shot image generation.

B A E Z G B E C GMM(z) A G B E C G B L KL +L A/REC +L ISO L D/G DOM +L D/G GAN GMM(z) B E Z E C SHARED PARAMS G E Z A A L S/REC +L C/REC +L CYC Input image in A Input image

3.1. THE CONTENT-ATTRIBUTE DISENTANGLING APPROACH:

GMM-UNIT follows the generative-discriminative philosophy. The generator inputs a content latent code c ∈ C C C = R C and an attribute latent code z ∈ Z Z Z = R Z , and outputs a generated image G(c, z). This image is then fed to a discriminator that must discern between "real" or "fake" images (D r/f ), and must also recognize the domain of the generated image (D dom ). For an image x n from domain X X X n (i.e. x n ∼ p X X X n ), its latent attribute z n is assumed to follow the n-th Gaussian component of the GMM, z n ∼ N (µ µ µ n , Σ Σ Σ n ), thus K = N . The attribute and content latent representations need to be learned, and they will be modeled by two architectures, namely a content extractor E c and an attribute extractor E z . See Figure 2 for a graphical representation of GMM-UNIT.

In addition to tackling the problem of multi-domain and multi-modal translation, we would like these two extractors, content and attribute, to be disentangled. This would constrain the learning and hopefully yield better domain translation, since the content would be as independent as possible from the attributes. Formally, the following two properties must hold:

Sampled attribute translation G(E c (x m ), z n ) ∼ p X X X n ∀ z n ∼ N (µ µ µ n , Σ Σ Σ n ), x m ∼ p X X X m , n, m ∈ {1, . . . , N }. (2)

Extracted attribute translation

G(E c (x m ), E z (x n )) ∼ p X X X n ∀ x n ∼ p X X X n , x m ∼ p X X X m , n, m ∈ {1, . . . , N }.

(3)

3.2. TRAINING THE GMM-UNIT:

The encoders E c and E z , and the generator G need to be learned to satisfy three main properties. Consistency: When traveling through the network, the generated/extracted codes and images must be consistent with the original samples. Fit: The distribution of the attribute latent space must follow a GMM. Realism: The generated images must be indistinguishable of real images. In the following we discuss different losses used to force the overall pipeline to satisfy these properties.

In the textbfconsistency term, we include image, attribute and content reconstruction, as well as cycle consistency. More formally, we use the following losses:

Self-reconstruction of any input image from its extracted content and attribute vectors:

L s/rec = N n=1 E x∼p X X X n G(E c (x), E z (x)) -x 1 (4)

Content reconstruction from an image, translated into any domain:

L c/rec = N n,m=1 E x∼p X X X n ,z∼N (µ µ µ m ,Σ Σ Σ m ) E c (G(E c (x), z)) -E c (x) 1 (5)

Attribute reconstruction from an image translated with any content:

EQUATION

In practice, this loss needs to be complemented with an isometry loss:

L iso = N n,m=1 E x∼p X X X n ,z,z ∼N (µ µ µ m ,Σ Σ Σ m ) | E z (G(E c (x), z))-E z (G(E c (x), z )) 1 -z-z 1 | (7)

Cycle consistency when translating an image back to the original domain:

EQUATION

In the fit term we encourage both the attribute latent variable to follow the Gaussian mixture distribution and the generated images to follow the domain's distribution. We set two loss functions.

Kullback-Leibler divergence between the extracted latent code and the model. Since the KL divergence between two GMMs is not analytically tractable, we resort on the fact that we know from which domain are we sampling and define:

EQUATION

where D KL (p q) = -p(t) log p(t) q(t) dt is the Kullback-Leibler divergence. Domain classification of generated and original images. For any given input image x, we would like the method to classify it as its original domain, and to be able to generate from its content an image in any domain. Therefore, we need two different losses, one directly applied to the original images, and a second one applied to the generated images:

EQUATION

EQUATION

where d X X X n is the label of domain n. Importantly, while the generator is trained using the second loss only, the discriminator D dom is trained using both.

The realism term tries to making the generated images indistinguishable from real images; we adopt the adversarial loss to optimize both the real/fake discriminator D r/f and the generator G:

EQUATION

EQUATION

The full objective function of our network is:

EQUATION

)

L G =λ GAN L G GAN + λ s/rec L s/rec + λ c/rec L c/rec + λ a/rec L a/rec + λ cyc L cyc + λ KL L KL + λ iso L iso + λ dom L G dom (15)

where {λ GAN , λ s/rec , λ c/rec , λ a/rec , λ cyc , λ KL , λ iso , λ dom } are hyper-parameters of weights for corresponding loss terms. The value of most of these parameters come from the literature. We refer to Appendix A for the details.

4. EXPERIMENTS:

We perform extensive quantitative and qualitative analysis in three real-world tasks, namely: edgesshoes, digits and faces. First, we test GMM-UNIT on a simple task such as a one-to-one domain translation. Then, we move to the problem of multi-domain translation where each domain is independent from each other. Finally, we test our model on multi-domain translation where each domain is built upon different combinations of lower level attributes. Specifically, for this task, we test GMM-UNIT in a dataset containing over 40 labels related to facial attributes such as hair color, gender, and age. Each domain is then composed by combinations of these attributes, which might be mutually exclusive (e.g. either male or female) or mutually inclusive (e.g. blonde and black hair).

Additionally, we show how the learned GMM latent space can be used to interpolate attributes and generate images in previously unseen domains, thus showing the first example of few-or zero-shot generation in image-to-image translation. Finally, GMM-UNIT will be applied to the Style transfer task.

We compare our model to the state of the art of both multi-modal and multi-domain image translation problems. In the former, we select BicycleGAN (Zhu et al., 2017b) , MUNIT (Zhu et al., 2017a) and MSGAN (Mao et al., 2019) . In the latter, we compare with StarGAN (Choi et al., 2018) and DRIT++ (Lee et al., 2019) , which is the only multi-modal and multi-domain method in the literature. However, since StarGAN is not multi-modal we additionally test a simple modification of the model where we inject noise in the network. We call this version StarGAN*. More details are in Appendix A.

4.1. METRICS:

We quantitatively evaluate the performance of our method through image quality and diversity of generated images. The former is evaluated through the Fréchet Inception Distance (FID) Heusel et al. (2017) and the Inception Score (Salimans et al., 2016) . We evaluate the latter through the LPIPS Distance (Zhang et al., 2018) , NDB and JSD (Richardson & Weiss, 2018) metrics. In addition, we also show the overall number of parameters used for all domains (Params). FID We use FID to measure the distance between the generated and real distributions. Lower FID values indicate better quality of the generated images. We estimate the FID using 100 input images and 100 samples per input v.s. randomly selected 10000 images from the target domain. IS To estimate the IS, we use Inception-v3 (Szegedy et al., 2016) fine-tuned on our specific datasets as classifier for 100 input images and 100 samples per input image. Higher IS means higher generated image quality.

LPIPS The LPIPS distance is defined as the L 2 distance between the features extracted by a deep learning model of two images. This distance has been demonstrated to match well the human perceptual similarity (Zhang et al., 2018) . Thus, following Zhu et al. (2017b) ; Huang et al. (2018) ; Lee et al. (2018) , we randomly select 100 input images and translate them to different domains. For each domain translation, we generate 10 images for each input image and evaluate the average LPIPS distance between the 10 generated images. Finally, we get the average of all distances. Higher LPIPS distance indicates better diversity among the generated images. NDB and JSD These are measuring the similarity between the distributions of real and generated images. We use the same testing data as for FID. Lower NDB and JSD mean the generated data distribution approaches better the real data distribution.

4.2. EDGES ↔ SHOES: TWO-DOMAINS+ TRANSLATION:

We first evaluate our model on a simpler task than multi-domain translation: two-domain translation (e.g. edges to shoes). We use the dataset provided by Isola et al. (2017) ; Zhu et al. (2017a) containing images of shoes and their edge maps generated by HED (Xie & Tu, 2015) . We train a single model for edges ↔ shoes without using paired information. Figure 3 displays examples of shoes generated from the same sketch by GMM-UNIT. Table 2 shows the quantitative evaluation and comparison with the state-of-the-art. Our model generates images with high diversity and quality using half the parameters of the state of the art. We refer to Appendix B.1 for additional results on this task. View House Numbers (SVHN) (Netzer et al., 2011) . We compare our model with the state-of-theart on multi-domain translation, and we show in Figure 4 and Table 3 the qualitative and quantitative results respectively.

Input Outputs:

From these results we conclude that StarGAN* fails at generating diversity, thus confirming the findings of previous studies that adding noise does not increase diversity (Mathieu et al., 2015; Isola et al., 2017; Zhu et al., 2017b) . GMM-UNIT instead generates images with higher quality and diversity than all the state-of-the-art models. We note, however, that StarGAN* achieves a higher IS, probably due to the fact that it solves a simpler task. Additional experiments carried out implementing a StarGAN*-like GMM-UNIT (i.e. setting σ σ σ k = 0, ∀k) indeed produced similar results. Specifically, the StarGAN*-like GMM-UNIT tends to generate for each input image one single (deterministic) output and thus the corresponding LPIPS scores are around zero. We refer to Appendix B.2 for additional results on this task. 

4.4. FACES: MULTI-ATTRIBUTE MULTI-DOMAIN TRANSLATION:

We also evaluate GMM-UNIT in the complex setting of multi-domain translation in a dataset of facial attributes. We use the CelebFaces Attributes (CelebA) dataset (Liu et al., 2015) , which contains 202,599 face images of celebrities where each face is annotated with 40 binary attributes. We resize the initial 178×218 size images to 128×128. We randomly select 2,000 images for testing and use all remaining images for training. This dataset is composed of some attributes that are mutually exclusive (e.g. either male or female) and those that are mutually inclusive (e.g. people could have both blonde and black hair). Thus, we model each attribute as a different GMM component. For this reason, we can generate new images for all the combinations of attributes by sampling from the GMM. As aforementioned, this is not possible for state-of-the-art models such as StarGAN and DRIT++, as they use one-hot domain codes to represent the domains. For the purpose of this experiment we show five binary attributes: hair color (black, blond, brown), gender (male/female), and age (young/old). These five attributes allow GMM-UNIT to generate 32 domains.

Figure 5 shows some generated results of our model. We can see that GMM-UNIT learns to translate images to simple attributes such as blonde hair, but also to translate images with combinations of them (e.g. blonde hair and male). Moreover, we can see that the rows show different realizations of the model thus demonstrating the stochastic approach of GMM-UNIT. These results are corroborated by Table 4 that shows that our model is superior to StarGAN* in both quality and diversity of generated images. We also note in this experiment that the IS is higher in StarGAN*. Additional results are on Appendix B.3. 

4.5. STYLE TRANSFER:

We evaluate our model on style transfer, which is a specific task where the style is usually extracted from a single reference image. Thus, we randomly select two input images and synthesize new images where, instead of sampling from the GMM distribution, we extract the style (through E z ) from some reference images. Figure 6 shows that the generated images are sharp and realistic, showing that our method can also be effectively applied to Style transfer.

Input Reference:

Figure 6 : Examples of GMM-UNIT applied on the Style transfer task. The style is here extracted from a single reference images provided by the user.

4.6. DOMAIN INTERPOLATION AND EXTRAPOLATION:

In addition, we evaluate the ability of GMM-UNIT to synthesize new images with attributes that are extremely scarce or non present in the training dataset. To do so, we select three combinations of attributes consisting of less than two images in the CelebA dataset: Black hair+Blonde hair+Male+Young, Black hair+Blonde hair+Female+Young and Black hair+Blonde hair+Brown+Young.

Input Black+Blonde+Female+Young Black+Blonde+Male+Young:

Figure 7 : Generated images in previously unseen combinations of attributes.

Figure 7 shows that learning the continuous and multi-modal latent distribution of attributes allow to effectively generate images as zero-or few-shot generation. At the best of our knowledge, we are the first ones being able to translate images in previously unseen domains. This can be extremely important in tasks that are extremely imbalanced.

Finally, we show that by learning the full latent distribution of the attributes we can do attribute interpolation both intra-and inter-domains. In contrast, state of the art methods such as Lee et al. (2019) can only do intra-domain interpolations due to their discrete domain encoding. Figure 8 shows some generated images through a linear interpolation between two given attributes, while in Appendix B.3 we show that we can also do intra-domain interpolations.

4.7. ABLATION STUDY:

We compare GMM-UNIT with three variants of the model that ablate L cyc , L d/rec and L iso in the Digits dataset. Table 5 shows the results of the ablation. As expected, L cyc is needed to have higher image quality. When L d/rec is removed image quality decreases, but L iso still helps to learn the attributes space. Finally, without L iso we observe that both diversity and quality decrease, thus confirming the need of all these losses. We refer to Appendix B.4 for the additional ablation results broken down by domain. 

5. CONCLUSION:

In this paper, we present a novel image-to-image translation model that maps images to multiple domains and provides a stochastic translation. GMM-UNIT disentangles the content of an image from its attributes and represents the attribute space with a GMM, which allows us to have a continuous encoding of domains. This has two main advantages: first, it avoids the linear growth of the dimension of the attribute space with the number of domains. Second, GMM-UNIT allows for interpolation across-domains and the translation of images into previously unseen domains.

We conduct extensive experiments in three different tasks, namely two-domain translation, multidomain translation and multi-attribute multi-domain translation. We show that GMM-UNIT achieves quality and diversity superior to state of the art, most of the times with fewer parameters. Future work includes the possibility to thoroughly learn the mean vectors of the GMM from the data and extending the experiments to a higher number of GMM components per domain.

A IMPLEMENTATION DETAILS:

Our deep neural models are built upon the state-of-the-art methods MUNIT (Huang et al., 2018) , BicycleGAN (Zhu et al., 2017b) and StarGAN (Choi et al., 2018) , as shown in Table 6 with details. We apply Instance Normalization (IN) (Ulyanov et al., 2017) to the content encoder E c and Adaptive Instance Normalization (AdaIN) (Huang & Belongie, 2017) and Layer Normalization (LN) (Ba et al., 2016) for the decoder G. For the discriminator network, we use Leaky ReLU (Xu et al., 2015) with a negative slope of 0.2. We use the following notations: D: the number of domains, N: the number of output channels, K: kernel size, S: stride size, P: padding size, CONV: a convolutional layer, GAP: a global average pooling layer, UPCONV: a 2×bilinear upsampling layer followed by a convolutional layer. Note that we reduce the number of layers of the discriminator on the Digits dataset.

We use the Adam optimizer (Kingma & Ba, 2015) with β 1 = 0.5, β 2 = 0.999, and an initial learning rate of 0.0001. The learning rate is decreased by half every 100,000 iterations. In all experiments, we use a batch size of 1 for Edges2shoes and Faces and batch size of 32 for Digits. And we set the loss weights to λ GAN = 1, λ s/rec = 10, λ c/rec = 1, λ a/rec = 1, λ cyc = 10, λ KL = 0.1, λ iso = 0.1 and λ dom = 1. We use the domain-invariant perceptual loss with weight 0.1 in all experiments. Random mirroring is applied during training. In Table 9 we show the quantitative results on the CelebA datset, broken down per domain. In Figure 11 we show some generated images in comparison with StarGAN*. Figure 12 shows the possibility to do attribute interpolation inside a domain. 

B.4 ABLATION STUDY:

In Table 10 we show additional, per domain, ablation results on the Digits dataset.

C VISUALIZATION OF THE ATTRIBUTE LATENT SPACE:

Figure 13 shows that the attributes sampled from the distribution and those extracted by the encoder E z are mapped and well projected in the latent space of the attributes. Figure 13 : Visualization of the attribute vectors in a 2D space via t-SNE method. "S" refers to randomly sampling from GMM components (1: black hair, 2: blondehair, 3: brown hair) and "E" refers to extracting attribute vectors by the encoder Ez from the real data.

Input:



:



Figures and Tables:

with a reference image a) Training GMM-UNIT b) Testing with sampled attributes

Figure 2: Overview of the GMM-UNIT framework: a) Training phase to translate an image from domain A A A to B B B. The generator uses the content of the input image (extracted by Ec) and the attribute of the target image (extracted by Ez) to train the network to fit the GMM. b) Testing with target attributes sampled from the GMM distribution of the attributes of domain B B B; c) Testing with an attribute extracted from an image belonging to the target domain B B B. The style of this image is inspired from Zhu et al. (2017b).

Figure 3: Examples of edges → shoes translation with the proposed GMM-UNIT. 4.3 DIGITS: SINGLE-ATTRIBUTE MULTI-DOMAIN TRANSLATION We then evaluate our model in a multi-domain translation problem where each domain is composed by digits collected in different scenes. We use the Digits-Five dataset introduced in Xu et al. (2018), from which we select three different domains, namely MNIST (LeCun et al., 1998), MNIST-M (Ganin & Lempitsky, 2014), a colorized version of MNIST for domain adaptation, and Street View House Numbers (SVHN)(Netzer et al., 2011). We compare our model with the state-of-theart on multi-domain translation, and we show in Figure4and Table3the qualitative and quantitative results respectively.

Figure 4: Samples of domain translation of GMM-UNIT trained on Digits.

Figure 5: Facial expression synthesis results on the CelebA dataset with different attribute combinations. Each row represents a different output sampled from the model.

Figure 8: Examples of domain interpolation given an input image.

Figure 1: Examples on using reference images to provide attribute representations.

Figure 2: Examples on digits translation.

3

Figure 10: Visual comparisons of state of the art methods on the digits dataset. We note that StarGAN* is a multi-domain (deterministic) model, while DRIT++ and GMM-UNIT are multi-modal and multi-domain methods.

A comparison of the state of the art methods for domain to domain translation.

Table content: <table><tr><td>Method</td><td>Unpaired</td><td>Multi-domain Multi-modal Domain (MDom) (MMod) encoding</td></tr><tr><td>CycleGAN (Zhu et al., 2017a)</td><td/><td>None</td></tr><tr><td>BicycleGAN</td><td/><td/></tr></table>

Benchmark on the Edges → Shoes dataset. MMod/MDom stand for multi-modal/domain.

Table content: <table><tr><td>Method</td><td>MMod MDom</td><td>IS↑</td><td>FID↓</td><td>NDB↓</td><td>JSD↓</td><td>LPIPS↑</td><td>Params</td></tr></table>

Benchmark on the Digits dataset. 1 FID fails in some domains because of low diversity.

Table content: <table><tr><td>Method</td><td>MMod MDom</td><td colspan="2">IS↑ FID↓</td><td>NDB↓</td><td>JSD↓</td><td>LPIPS↑</td><td>Params ↓</td></tr><tr><td>StarGAN*</td><td/><td colspan="5">3.35 88.89 62.55±3.59 .14±.03 .005±.016</td><td>11.18×1</td></tr><tr><td>DRIT++</td><td/><td>3.06</td><td>-1</td><td colspan="4">77.2±3.49 .27±.01 .055±.046 24.49M×1</td></tr><tr><td>GMM-UNIT</td><td/><td colspan="6">3.23 64.96 58.33±3.96 .12±.00 .107±.076 16.37M×1</td></tr></table>

Benchmark on the CelebA dataset. 1 FID fails in all domains because of low diversity.

Table content: <table><tr><td>Method</td><td>MMod MDom</td><td>IS↑ FID↓</td><td>NDB↓</td><td>JSD↓</td><td>LPIPS↑</td><td>Params ↓</td></tr><tr><td>StarGAN*</td><td colspan="2">3.345 -Input Black hair Blonde hair</td><td>Brown hair</td><td colspan="2">Blonde + Male Blonde + Old</td><td/></tr></table>

Ablation study performance on the Digits dataset. Lcyc 3.03 85.78 63.44±2.48 0.174±0.004 0.138±.081 GMM-UNIT w/o Ld/rec 3.26 67.95 61.33±2.35 0.144±0.005 0.121±.008 GMM-UNIT w/o Liso 3.28 68.56 59.44±4.25 0.139±0.004 0.117±.078 GMM-UNIT 3.23 64.96 58.33±3.96 0.119±0.003 0.124±.078

Table content: <table><tr><td>Model</td><td>IS↑ FID↓</td><td>NDB↓</td><td>JSD↓</td><td>LPIPS↑</td></tr><tr><td>GMM-UNIT w/o</td><td/><td/><td/><td/></tr></table>

Comparison between GMM-UNIT and the reference of Paired, multimodal, generation (BicycleGAN) on the Edges → Shoes dataset. Number of all parameters for the models trained in all the domains. † Trained on paired data.

Table content: <table><tr><td>Metric</td><td colspan="2">BicycleGAN  † GMM-UNIT</td></tr><tr><td>IS↑</td><td>2.956</td><td>3.245</td></tr><tr><td>FID↓</td><td>47.43</td><td>80.78</td></tr><tr><td>NDB↓</td><td>27.33±1.70</td><td>31.33±2.62</td></tr><tr><td>JSD↓</td><td>.089±.005</td><td>.098±.001</td></tr><tr><td>LPIPS↑</td><td>.196±0.091</td><td>.209±.110</td></tr><tr><td>Params  § ↓</td><td>64.30M×2</td><td>23.52M×1</td></tr><tr><td>§</td><td/><td/></tr></table>

Quantitative comparison on the Digits dataset.

Table content: <table><tr><td colspan="2">Target Domain Metric</td><td/><td>Method</td></tr><tr><td/><td/><td>StarGAN*</td><td colspan="2">DRIT++ GMM-UNIT</td></tr><tr><td/><td>IS↑</td><td>2.628</td><td>1.926</td><td>2.120</td></tr><tr><td/><td>FID↓</td><td>103.76</td><td>× 1</td><td>78.19</td></tr><tr><td>MNIST</td><td>NDB↓</td><td colspan="2">62.67±3.68 82.67±0.47</td><td>57.33±4.78</td></tr><tr><td/><td>JSD↓</td><td>.132±.014</td><td>.418±.012</td><td>.101±.003</td></tr><tr><td/><td>LPIPS↑</td><td>.002±.006</td><td>.001±.004</td><td>.067±.058</td></tr><tr><td/><td>IS↑</td><td>3.576</td><td>3.463</td><td>3.293</td></tr><tr><td/><td>FID↓</td><td>77.86</td><td>89.81</td><td>46.22</td></tr><tr><td>SVHN</td><td>NDB↓</td><td colspan="2">65.67±2.49 78.33±5.44</td><td>65.33±1.70</td></tr><tr><td/><td>JSD↓</td><td>.179±.006</td><td>.252±.005</td><td>.178±.002</td></tr><tr><td/><td>LPIPS↑</td><td>.008±.023</td><td>.048±.040</td><td>.115±.078</td></tr><tr><td/><td>IS↑</td><td>3.853</td><td>3.795</td><td>4.274</td></tr><tr><td/><td>FID↓</td><td>85.07</td><td>114.51</td><td>70.48</td></tr><tr><td>MNIST-M</td><td>NDB↓</td><td colspan="2">59.33±3.30 70.67±2.62</td><td>52.33±4.64</td></tr><tr><td/><td>JSD↓</td><td>.132±.000</td><td>.132±.001</td><td>.078±.004</td></tr><tr><td/><td>LPIPS↑</td><td>.006±.014</td><td>.116±.070</td><td>.191±.095</td></tr><tr><td/><td>Params  § ↓</td><td>11.18M×1</td><td>24.49M×1</td><td>16.37M×1</td></tr></table>

Peer review (iclr2020):
This paper proposes a unifying algorithm for multi-modal multi-domain image translation with either better performance or more functionality than its predecessors. It uses a GMM distribution to model the variations both within the same domain and cross-domain differences and an encoder to extract context information invariant across domains. The setup is able to degenerate into many of its predecessors (e.g. StarGAN, pix2pix, etc.) if we choose specific sets of hyperparameters. Overall, the paper did a good survey over the literature and comparisons across multiple data sets and baselines. There are a few caveats that made me hesitant to accept this paper. a) There is a plethora of literature in image translation in recent years that works on improving image quality and diversity, avoiding model collapse, expanding to multi-domain translation, etc. Although it is worthwhile combining all those works into an overarching framework, I expect the sum to be more than the individuals all bundled together, which is unclear from this paper (see numbers in Table 2). MUNIT is a great example where the idea of AdaIN is not new, but its learned version applied to image translation yields great results and lots of future directions to explore. b) It would be better to have a clearer future direction of improvement for this paper. The authors mentioned this in the last sentence, but it would be great to have something to inspire future researchers to tackle the ill-defined unsupervised image-to-image translation problem. Questions that are left unanswered: 1. If multi-modal output is desirable, is there a tradeoff or could we get diversity for free? 2. From eq.15, such a multi-term loss function may not scale to higher resolution output where model collapse occurs more often, or it could become extremely expensive to train, which hinders the speed to iterate through ideas. Has GMM-UNIT been trained at 256x256 or higher? How long did it take to train at 128x128? 3. In image translation or style transfer domain, one of the often asked questions is *what counts as a domain/style as opposed to the content?* In this work, the content is defined as the encodable features that stays constant after translation to different domains. The style is defined as the Z modeled by the domain dependent GMM distribution. From fig. 6, it looks like local changes or simple changes like eye makeup and the overall color of the image are considered as parts of the style or noise that can vary within a domain. On the other hand, more complicated changes like hair style, hair length, jewelry, are treated as the content. What causes the network choosing to classify a feature as style versus as content? Does that change if we train from scratch again?