Style-based Encoder Pre-training for Multi-modal Image Synthesis
Authors: Moustafa Meshry
Yixuan Ren
Ricardo Martin-Brualla
Larry Davis
Abhinav Shrivastava
Venue:  (2020)
Keywords: image-to_image translation
representation learning
multi-modal image synthesis
GANs

Abstract: Image-to-image (I2I) translation aims to translate images from one domain to another. To tackle the multi-modal version of I2I translation, where input and output domains have a one-to-many relation, an extra latent input is provided to the generator to specify a particular output. Recent works propose involved training objectives to learn a latent embedding, jointly with the generator, that models the distribution of possible outputs. Alternatively, we study a simple, yet powerful pre-training strategy for multi-modal I2I translation. We first pre-train an encoder, using a proxy task, to encode the style of an image, such as color and texture, into a low-dimensional latent style vector. Then we train a generator to transform an input image along with a style-code to the output domain. Our generator achieves state-of-the-art results on several benchmarks with a training objective that includes just a GAN loss and a reconstruction loss, which simplifies and speeds up the training significantly compared to competing approaches. We further study the contribution of different loss terms to learning the task of multi-modal I2I translation, and finally we show that the learned style embedding is not dependent on the target domain and generalizes well to other domains.

Under review as a conference paper at ICLR 2020
STYLE-BASED ENCODER PRE-TRAINING FOR MULTI- MODAL IMAGE SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Image-to-inage (121) translation aims to translate images from one domain to another. To tackle the multi-modal version of I2I translation, where input and output domains have one-to-many relation, extra latent imput is provided to the generator to specify a particular output. Recent works propose involved training objectives to learn latent embedding, jointly with the generator, that models the distribution of possible outputs. Alternatively, we study a simple, yet powerful pre-training strategy for multi-modal 12I translation. We first pre-train an encoder, using proxy task, to encode the style of an image, such as color and texture, into a dwwddmmnsiooall latent style vector. Then we train generator transform an input image along with a style-code to the output domain. Our generator achieves state- of-the-art results on several benchmarks with a training objective that includes just a GAN loss and a reconstruction loss, which simplifies and speeds up the training significantly compared to competing approaches. We further study the contribution of different loss terms to learning the task of multi-modal 12I translation, and finally we show that the learned style embedding is not dependent on the target domain and generalizes well to other domains.
INTRODUCTION
Image-to-Iie Ioagto--meeeeem (12I) translation is the task of transforming images from one domain to another (e.g..
semantic maps scenes, sketches -> photo-realistic images, etc.). Many problems in computer vision and graphics can be cast as 12I translation, such as photo-realistic image synthesis (Chen)
& KKltun| (2017); Isola et al. (2017) 7): Wang al. (2018a)), super-resolution (Ledíg et al. (2017)) colorization (Zhang et al. 2016)) 2017a)), and inpainting (Pathak et al. (2016)). Therefore, I2I translation has recently received significant attention in the literature. One main challenge in 12I translation is the multi-modal nature for many such tasks the relation between an input domain and an output domain B often times one-to-many, where single input image I E A can be mapped to different output images from domain B. For example, a sketch of a shoe or handbag can be mapped to corresponding objects with different colors or styles, or a semantic map of a scene can be mapped to many scenes with different appearance, lighting and/or weather conditions. Since I2I translation networks typically learn one-to-one mappings due to their deterministic nature, an extra input is required to specify output mode to which an input image will be translated. Simply injecting extra random noise as input proved to be ineffective as shown in (Isola et al. (211)) 7): Zhu et al. (2017b)), ), where the generator network just learns ignore the extra noise and collapses few modes (which is one form of the mode collapse problem). To overcome this problem. ZZu et al. (2017b) proposed BicycleGAN, which learns to encode the distribution of different possible outputs into a Tatent vector z, and then learns deterministic mapping G : (A, z) + B. So, depending on the latent vector a single input I A can mapped to multiple outputs B. While BicycleGAN requires paired training data, several works (Lee et al. (008)); Huang et al. (2018) extended it to the unsupervised case, where images domains and are not correspondence ('unpaired'). One main component of unpaired 121 is a cross-cycle consistency constraint, where the network generates an intermediate output by swapping the styles of pair of images, then swaps the style between the intermediate output again to reconstruct the original images. This enforces that the latent vector preserves the encoded style information when translated from an image to another image j and back to image again. This constraint can also be applied to paired training data, where it encourages style/attribute transfer between images. However, training BicycleGAN (Zhu et al.) (2017b)) or its unsupervised counterparts (Huang et al. (2018): Lee et al. (2018) is not trivial. For example,
single

---

Auto-EEnoderrs Under review as a conference paper at ICLR 2020
BicycleGAN combines the objectives of both conditional Variational Auto-Encodee (cVAEs) (Sohn et al. (2015) and a conditional version of Latent Regressor GANs (cLR-GANs) (Donahue et al. (2016) ; Dumoulin et al. (2016) to train their network. The training objective of (Huang al. (2018); Lee et al. (018)) is even more involved to handle the unsupervised setup.
In this work, we aim simplify the training of general purpose multi-modal 12I translation networks, while also improving the diversity and expressiveness of different styles in the output domain. Our approach is inspired by the work Meshry al. (2019) which utilizes a staged training strategy to re-render scenes under different lighting, time of day, and weather conditions. We propose a mullii-modll I2I translation networks, which makes the
pretraining approach for style encoders, training simpler and faster by requiring fewer losses/constraint.. Our approach is also inspired by the standard training paradigm in visual recognition first pretraining on a proxy task, either large supervised datasets (e.g., ImageNet) (Krizhevsky et al.] (2012); (Sun et al. (2017); 7); Mahajan et al. (2018)) or unsupervised tasks (e.g.. Doersch et al. (2015); 5); Noroozi Favaro (2016)), and then fine-tuning (transfer learning) the desired task. Similarly, we propose to pretrain the encoder using a proxy task that encourages capturing style into a latent space. Our goal is to highlight the importance of pretraining for 121 networks and demonstrate that simple approach can be very effective for multi-modal image synthesis. In particular, we make the following contributions:
We explore style pretraining and its generalization for the task of multi-modal I21 translation, which simplifies and speeds up the training compared to competing approaches.
We provide study of the importance of different losses and regularization terms for multi-modal 12I translation networks.
We show that the pretrained latent embeddings is not dependent on the target domain and generalizes well to other domains (transfer learning).
We achieve ofe art results on several benchmarks in terms of style capture and transfer, and diversity of results.
2 RELATED WORK
Deep generative models There has been incredible progress in the field of image synthesis using deep neural networks. In its unconditional setting, a decoder network learns to map random values drawn from prior distribution (typically Gaussian) to output images. Variational (VAEs) (Kingma Welling (2014)) assume bijection mapping between output images and some latent distribution and learn to map the latent distribution to unit Gaussian using the reparameter- ization trick. Alternatively, Generative Adversarial Networks (GANs) (Goodfellow al. 22014)) directly map random values sampled from unit Gaussian to images, while using discriminator network to enforce that the distribution of generated images resembles that of real images. Recent works proposed improvements to stabilize the training (Gulrajani et al. (2017); Karnewar & Iyengar (2099) Mao et al. (2017) Radford et al. (2016)) and improve the quality and diversity of the out- put KKrras al. (2018 (019)). Other works combine both VAEs and GANs into a hybrid VAE-GAN model (Larsen et al. (2016); Rosca et al. (2017)).
(elling](2014)
Conditional image synthesis Instead of generating images from input noise, the generator be augmented with side information in the form of extra conditional inputs. For example, [Sohn et al.) (2015) extended VAEs to their conditional setup (cVAEs). Also, GANs can be conditioned on different information, like class labels (Mirza & Osindero 2014); Odena et al. (2017); Van den Oord al. 20166), language description (Mansimov al. (2016): Reed al. (2016)), or an image from another domain (Chen & Koltun (2017); Isola et al. 2017 The latter is called mmage-to-Image translation.
(Son
Image-to-Image (121) translation 12I translation is the task of transforming an image from one domain, such as sketch, into a another domain, such as photo-realistic images. While there are regression-based approaches to this problem (Chen & Koltun 2017);; |Hoshen & Wolf (2018)). significant successes in this field are based on GANs and the influential work of pix2pix (Isola et al. (2017)). Following the success of pix2pix (Isola et al. (2017)), 12I translation has since been utilized in a large number of tasks, like inpainting (Pathak et al. (2016)). colorization (Zhang et al. (2016) 2017a)), super-resolution (Ledig et al. (2017), 7),, rendering
Martin-Brualle et al. (2018): Meshry et al. (2019) Thies et al. (2019), ), and many more (Dong et al. (ang) & Gupta (2016); Zhang et al.
(011b)) There has also been works extend this task to the unsupervised setting (Hoshen & Wolf
2

---

Under review as a conference paper at ICLR 2020
Oroudd Jun
G
Gone
Gater
Closs relghtr
Ouast
For neighter
appt
Dursoo
Appearance prezaining
Generator trining
Figure I: Overview of our training pipeline. Stage 1: pretraining the style encoder E using a triplet loss. Stages 2, 3: training the generator G, and finetuning both G, F together using GAN and reconstruction losses.
(2018) ; Kim et al. (2017); >: Liu et al. (2017): Ma et al. (2019); Royer et al. (2017); Zhu et al. (017a)) to multiple domains (Choi al. (2018), ). and to videos (Chan al. (2018); Wang et al. (201 (2018b)).
Multi-modal 121 translation Image translation networks are typically deterministic function approx- imators that learn a one-to-one mapping between inputs and outputs. To extend I2I translation to the case of diverse multi mudti-modal outputs, Zhu et al. (2017b) proposed the BicycleGAN framework that learns latent distribution that encodes the variability in the output domain and conditions generator on this extra latent vector for multi-modal image synthesis. Wang et al. (201 8a b) learn instance-wisg wise latent features for different objects target image, which are clustered after training to find f fixed modes for different semantic classes. At test time, they sample one of the feature clusters for each object to achieve multi-modal synthesis. Other works extended the multi-modal 121 framework to the unpaired settings where images from the input and output domains in correspondence (Almahairi et al." 2018); Huang et al. (2018); [ee et al. (2018) by augmenting BicycleGAN with different forms of a cross-cycle consistency constraint between two unpaired images. In our work, we focus on the supervised setting of multi-modal 12I translation. We propose a simple, yet effective, pretraining strategy to learn latent distribution that encodes variability in the output domain. The learned distribution can be easily adapted to new unseen datasets with simple fine-tuning, instead of training from random initialization.
not
un
3 APPROACH
Current multi-modal image translation networks require an extra input that allows for modelling the one-to-many relation between an input domain A and an output domain B as a one-to-one relation
from a pair of inputs 2) -> B. previous approaches, there has been a trade-off between simplicity and effectiveness for providing the input z. On one hand, providing random noise as the extra input z maintains a simple training objective (same as in pix2pix (Isola et al. (2017))).
y Zhu et al. (2017)) showed that the generator has little incentive utilize the input vector 2 since it only encodes random information, and therefore the generator ends up ignoring = and collapsing
However, Isola et al. (
or few modes. On the other hand, BicycleGAN (Zhu al. (2017b) combines the objectives of both conditional Variational Auto-Encoder GANs (cVAE-GAN) and conditional Latent Regressor GANs (cLR-GAN) learn latent embedding simultaneously with the generator G. Their training enforces two cycle consistencies: B + 2 + β and + β 2. This proved to be very effective, but the training objective is more involved, which makes the training slower. Also, since the latent embedding is being trained simultancously with the the generator, hyper-parameter tuning becomes more critical and sensitive.
We aim to combine the best of both worlds: an effective training of a latent embedding that models the distribution of possible outputs, while retaining simple training objective. This would allow for faster and more efficient training, as well as less sensitivity to hyper-parameters We observe that the variability in many target domains can be represented by the style diversity of images in the target domain B. where the style is defined in terms of the gram matrices used in the Neural Style Transfer literature (Gatys et al. (2016)). Then, we learn an embedding by separately training an encoder network E on an auxiliary task to optimize for = E(I") capturing the style an image B Finally, since now have learned deterministic mapping between and the style of the target
3

---

Under review as a conference paper at ICLR 2020
Input
Ours
Input
GT
Bleyde v1
Figure 2: Qualitative comparison with baselines. Ours better matches the ground truth (GT) style.
Irou
Figure 3: Style transfer for different datasets. For each dataset, we show output for applying different styles to the same input image.
output image 1" training the generator G becomes simpler as G is just required to discover the correlation between output images and their corresponding style embedding z.
To incorporate this into BicycleGAN (Zuu et al. (2017b), we replace the simultaneous training of the encoder E and the generator G with staged training as follows:
Stage 1: Pretrain E on proxy task that optimizes embedding of images in the output domain B into low-dimensional style latent space, such that images with similar styles lie closely in that space (i.e. clustered).
Stage 2:
Train the generator network G' while fixing the encoder E, that G learns to associate the style of output images to their deterministic style embedding = = F)
Stage 3:
Fine-tune both the F and G networks together, allowing for the style embedding to be further adapted best suit the image synthesis task for the target domain.
The intuition why such staged training would be effective for multi-modal 121 translation is that the encoder pretrained to model different modes of the output distribution clusters of images with similar styles (refer to the supp. material, figures 6 7
for a visualization of pretrained latent embeddings). During stage 2, the latent space is kept fixed, and the input latent to the generator can be used to clearly distinguish the style cluster to which the output belongs, which makes the multi-modal synthesis task easier for the generator. Finally, stage finetunes the learned embedding to better serve the synthesis task at hand. Next, explain how pre-train the style encoder network E in Section [.1] and how to train the generator using the pre-learned embeddings (Section) 3.2). Finally, demonstrate the generalization of pre- trainnnnn the style encoder E in Section 3.3
3.1 PRE-TRAINING THE STYLE ENCODER E
The goal of pre-training the encoder network E is to learn a deterministic mapping from the style of a target image 1 B a latent style code z; 5) Ideally, images with similar styles should be close in the style embedding space, while images with different styles should be far from each other.
To supervise training such embedding. we utilize the style loss (Gatys et al. (0166) as a distance metric to measure the style similarity between any two given images. The style encoder network E is then trained using triplet loss (Schroff et al. (2015). where the input is a triplet of images
(Ja, Tp. In), where (Ta Ip) have similar style, while (Ta. I.)) have different style, as measured by the style loss metric. The training objective for E is given by:
("i la, Ip: In) = max ([llza 2nl12 - I|a =nl1 + a].0) + A (Zas Zpp 2n)
(1)
4

---

Under review as a conference paper at ICLR 2020
Input
Bole
Sofe
Figure 4: Style interpolation. Left column is the input to the generator G. second and last columns are input style images to the style encoder, and the middle images are linear interpolation in the embedding space.
where a is separation margin, À is relative weighting parameter between the main triplet objective and an optional regularization term (-) which is an L2 regularization to encourage learning a compact latent space.
To generate triplets for pre-training the encoder E, we compute the set of ke closest and y furthest neighbors for each anchor image Ia as measured by the style loss. Then, for each anchor image Ia. we randomly sample positive image Ip and a negative image I, from the set of closest and furthest neighbors respectively. We found that, for large datasets, is sufficient to generate triplets for subset of the training images. One challenge is the set of images with an outlier style. Such images will be furthest neighbors to most images, and can mislead the training by just projecting outlier images to separate clusters. To deal with this, sample the negative style image I, from a larger set furthest neighbors; while the positive image Ip is sampled from a small set of closest neighbors so that it would have reasonable style similarity to the anchor image.
Triplet selection.
3.2 GENERATOR TRAINING
After pre-training the style encoder E (stage 1), we have established a mapping from images in the output domain, 1 B, their style-embedding as input to the generator during training, the generator has good incentive to associate the style of output images to their corresponding style embedding instead of learning to hallucinate the style. It's important to retain the deterministic correspondence between images and their style codes to facilitate the job of the generator to discover this correlation. This is why, during stage we keep the weights of the style encoder E fixed. The forward pass reconstructs a training image 1 as jB G(TA, z)). where 2, E. The training objective is similar to that of pix2pix (Isola et al. 2017 )
E(IP). Feeding the style embedding
cime (1.33 (19) = COGAN 111),11)=00)) + (1, 118)
where the Least Square GAN loss (LSGAN) (Mao et al. (207))) for the LEGAN term, and a VGG-based perceptual loss (Johnson et al. (2016) for the reconstruction term Cree- Once the generator has learned to associate the output style with the input style embedding, stage 3 fine-tunes both the generator G and the style encoder E together using the same objective
Style sampling. To perform multimodal synthesis on a given input at test time, we can capture the latent vector 2 from any existing image and transfer the style to the generated image. However, if we wish sample styles directly from the latent distribution, we can optionally enforce prior on the latent distribution. For example, we can add KL divergence term on the latent vectors to enforce unit Gaussian prior. our experiments, we found it more effective to add an L2 regularization on the the latent vectors to enforce zero-mean embeddings and limit the variance of the latent space. We then compute an empirical standard deviation for sampling. Another alternative to enable sampling is to train a mapper network M to map the unit Gaussian to the latent distribution. This can be done as a post-pun post-processing step after the style encoder has been trained and finetuned. Specifically, we follow the nearest-neighbor based Implicit Maximum Likelihood Estimation (IMLE) training (Li & Malik (208)): Hoshen et al. (2019) ) to train the mapper network M. The training objective is given by:
M arg min
Σ 1| Me))l.
e, arg min ||
M(r,)) )
M
where {r,} } is set of random samples from the unit Gaussian prior, and for each latent code 2,, select e; that maps to the nearest neighbor M(es) to 2..
5

---

Under review as a conference paper at ICLR 2020
Ground vut
Sumpling for ongiical Ny. of
Sumpping lning the repper netwak
Figure 5: Style sampling for different datasets using our approach v3. We sample either from N where μr are computed from the train set (middle), or using the mapper network M (right).
3.3 GENERALIZING THE PRE- TRAINING STAGE
The goal of the style pretraining is to learn an embedding that mimics the style loss, where images with similar style lie closely in that space. Since the definition of image style in neural style transfer literature is general and is not dependent on a specific image domain, encoding an image I to its style embedding can also be seen as general task that independent of the output domain B. This allows for performing the pretraining stage only once using auxiliary training data. The finetuning stage eventually tweaks the embedding to better suit the specific target domain B. We show experimentally in Section 4 that pretraining the style encoder on datasets other than the target domain B doesn't 't degrade the performance. It can even improve the performance if the target dataset is small, in which case pretraining auxiliary dataset helps with the generalization of the overall model.
EXPERIMENTAL EVALUATION
Datasets. We evaluate our approach on five standard 12I translation benchmarks used in Isola et al. (2017) Zhu al. (2017b); Architectural labels -> photo, aerial -+ map, edges -> shoes/handbags and night - day. In addition, we use the Space Needle timelapse dataset (Martin-Brualla (007), which consists of 2068 paired images with a 8280 x 1080 resolution, where the input domain includes images with temporally smoothed appearance, and the output domain contains real images spanning different lighting and weather conditions.
Baselines. While we report numbers for retrained models using the code released with BicycleGAN (BicycleGAN v0) for completeness, we mainly compare to two stronger baselines:
BicycleGAN vl: we implement BicycleGAN using the same network architecture as used in our approach to have a fair comparison.
BicycleGAN v2: we augment BicycleGAN with the cross-cycle consistency constraint introduced in (Huang et al. 2018 Lee et al. 2018) as follows: the input is pair of training examples (7 18), ( T for which we obtain their respective style embeddings
E(( E ). We then apply a 2-step eyclic reconstruction of 1 1 in the G(15,21) In the the second step, we re-capture the latent style vectors & = Elu),, & = E(v)
first step, we generate both images with a swapped style u = G(L ,() 1
and generate the original images 1f 1 by swapping the style again: i G(A, i G(IA, &2). We add cyclic reconstruction term for i/s jB.
4.1 EVALUATION
Image reconstruction. We report the reconstruction quality of validation set images, using both PSNR and MlexNer-based LPIPS (Zhang et al.) (0018) metrics, in Table Note that our results without fine-tuning (stage 2) are on-par-with the baselines, which verifies the validity of our approach and

---

Under review as a conference paper at ICLR 2020
Table 1:
Validation set reconstruction quality, measured by PSNR (higher is better) and LPIPS (Zhang et al. 2018) (lower is better), for various datasets. We compare between retrain- ing BicycleGAN (Zhu et al. 2017b) authors' released code (Bicycle v0), our implementation of the two baselines (BicycleGiNN vI and BicycleGAN v2) described in Section 4 and our approach both
before finetuning (ours stage 2), and after finetuning (ours stage 3).
edge2shundbugs
edges22hoes
labeleffacade
night2day
mups
space needle
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS 4 PSNR 1
LPIPS
PSNR
LPIPS
Bicycle
17.08
0.255
20.24
0.177
12.64
0.431
13.25
0.520
14.32
0.396
Bieycle v1
18.52
0.198
21.84
0.124
13.08
0,378
13.88
0.491
14.67
0.359
19.72
0.235
Ricycle v2
19.23
0.192
22.51
0.132
13.36
0.375
14.48
0.480
16.17
0.407
19.84
ours - stage 2
18.01
0.209
21.40
0.140
13.44
0.383
14.34
0.476
15.08
0.392
21.39
0.227
ours stage 18.91
0.177
22.68
13.44
0.370
0.452
15.15
0.349
22.11
Table 2: Generalization of a pretrained style en- coder E. We report validation set reconstruction for the edges2handbags and night2day datasets when pretraining with different datasets. Stages 2. 3 show results before/after finetuning E respec- tively.
Table 3: Diversity score is the average LPIPS distance (Zhang et al. 2018). User preference score is the percentage method is preferred over Ours v4, on the edges2shoes dataset.
Appeoach
LPIPS (transfer)
LPIPS (sampling)
User
preference
Dataset
pretrain dataset
Stage
Stage
Bicycle vl
0,102
0.119
30.0%
PSNR
LPIPS
LPIPS
Bicycle v2
0,138
0,132
37.7%
Shandhhsss
18.01
0.177
edgesChandlogs
2dges2shoes
17.89
0.215
18.96
0.176
Ours v1
0.153
0.148
26.5%
space neeule
17.86
0.221
19.02
0.175
Ours v2
0.171
0.140
41.1%
nigh22ky
space needle
nighi2day
14.34
13.75
0.489
0.476
15.05
15.15
0.454
0.422
Ours v3
Ours
0.151
0.149
0.165
0.132
50.4%
50%
0.461
that style-based encoder pre-training successfully learns to distinguish different modes in the output domain, which proves effective for training multi-modal 12I networks. Fine-tuning (stage 3) further improves our results compared to the baselines. Figure 2 shows qualitatively how our approach reconstructs the target style more faithfully.
Style transfer and sampling. Figure shoows style transfer to validation set images from different datasets. We can also sample random styles directly from the the latent distribution as described in Section 3.2 Figure 5 shows results for both adhoc sampling from the assumed N N(u, Ø)) empirical distribution, as well as formally sampling from unit Gaussian using the mapper network M. While both results look good, we note that the assumption for adhoc sampling is not explicitly enforced, and thus could sample bad style codes outside the distribution (see Appendix A.5 for examples).
Style interpolation. Figure shows style interpolation by linearly interpolating between two latent vectors. Note the smooth change in lighting and cloud patterns when going from cloudy to sunny (second row).
Pre-training generalization. Since the notion of style, as defined in the Neural Style Transfer literature, is universal and not specific to certain domain, we hypothesized that style-based encoder pretraining would learn a generic style embedding that can generalize across multiple domains and be effective for multi-modal image 121 translation. Here, Table For target dataset, we train the generator G three times, each with different pre-training of the style encoder E: (1) same dataset pre-training: pre-train E using the output domain B of the target dataset. (2) similar-domain pre-training: pre-train on a different dataset, but whose output domain bears resemblance to the output domain of the target dataset (e.g., edges2shoes and edges2handbags, or day images from night2day and the Space Needle timelapse dataset). (3) different-
experimentally verify hypothesis in
pre- training: pre-train on different dataset whose output domain has different styles from that of the target dataset (e.g., edges2handbags and the Space Needle timelapse datasets, or night2day and cdges2handbags datasets). Table 2 shows that without fine-tuning (stage 2), the edges2handbags dataset shows a slight performance degradation when going from pre-training on the same dataset, to pre-training on a similar- On the other hand, the night2day dataset has only 100 unique scenes for training. So, pre-training on another dataset such as Space Needle generalizes better to new scenes in the validation set, since it helps avoid overfitting the small number of unique scenes in the training set. A After fine-tuning,
dataset, and finally pre-training on a different-domain dataset.
7

---

Under review as a conference paper at ICLR 2020
Table 4: Ablation study on the effect of different components and loss terms using the edges2handbags dataset. We study direct and cyclic reconstructions on ground truth images (dir_ recon, eyc_recon) discriminator loss on direct reconstructions and on generated images with randomly sampled style (D_ _0_dir, D_ p_rand_za (indd_),_lii latent reconstruction (z_recon), L2 and KL regularization on the latent vector (2_L2, KL), and finally the use of VAE vs. just an auto-encoder.
Loss selup
Approach
PSNR
1S1
LPIPS 1
decarecon ryc_reeon
D_dir
D_rand x 2recon 2.1.2 KL
VAE
Bicyclo v1
/
18.88
2.31 0.05
0.201 = 0.003
Bicycle
/
/
/
/
18.9 0.30
2.45
0,192
0.002
Bieycle v1.2
4
/
19.02 0.10
2.36
0.175 0.001
Bieycle v2.2
/
4
/
19.3 0.07
241 = 0.00
0.176 0.002
Bicycle v1.3
19211 0.06
2.34 = 0.08
0.177 0.002
Bicycle v2.3
/
19.44 0.09
2.33 0.04
0.180 0.001
Ours vl
v
/
/
/
18.97 0.13
2.01
0,189
0.001 0.001
Ours v2
/
/
/
/
18.91 0.10
2.13
0,183 2 0.002
Ours v3
/
18.94 0.05
2.42 0.03
0.176 + 0.001
Ours v4
18.91 + 0.02
2.46 0.03
0,177 + 0.001
performance differences further reduce to be insignificant. We also investigate the generalization of encoder pre-training using non-style distance metrics in Appendix [A.4
Ablative study. We investigate the role of different loss terms as we transition from the loss setup of baselines to that of our training approach. We first remove the variational part in both BicycleGAN v and BicycleGAN v2 baselines resulting in Bicycle v1.2, v2.2. We further remove the Gaussian prior and replace the KL loss with an L2 regularization in Bicycle v1.3, v2.3. To maintain random latent vector sampling during training without a prior, we sample a random training image, and use its style code. We define different versions of our approach (v1, v2, v3, and v4) based on different loss setup during training as follows: we start with 'Ours vl', which has the same setup as Bicycle v2.3, except that it uses pre-trained embeddings described in Section 3.1 We then remove cyclic reconstruction, random z sampling, and L2 regularization terms resulting in 'Ours v2', 'v3', and 'v4' respectively. We run each setup on the edges2handbags dataset. In order to draw more reliable conclusions, we repeat each experiment 3 times and report the mean and standard deviation in Table We notice that removing the variational part in VAES is enough to achieve good results. While VAES in general are robust to noise in the input latent, we observe that this comes at the expense of the expressiveness of the latent space (c.g., less faithful style capture and transfer), especially for low dimensional latents. We also observe that our approach generally performs better with less constraints (loss terms). For example, "Ours v1, v2" have lower results than their "Bicycle v1.3, v2.3" counterparts. This shows that the main benefit of pre-trained embeddings is when the network is less constrained.
Diversity and user study. We evaluate diversity by computing the average LPIPS distance over 1600 output images. We measure diversity on two setups: we sample 100 validation images, and (1) apply style transfer from 16 randomly sampled images, or (2) we sample 16 random codes using the mapper network M to obtain 1600 outputs. We also measure the realism and faithfulness of style transfer through a user study, where 30 participants are shown an input shoe sketch, an input style image and two style transfer outputs. They are given unlimited time choose which output looks more realistic, and if both are realistic, then which transfers the style more faithfully. We fix 'Ours v4' approach as anchor and compare other methods to it. Table Isowws that the baselines achieve lower diversity and user preference compared to our approach, specially in the style transfer setup. Different variations of our method, except for 'Ours v2' yield similar diversity and user preference scores. We observe that 'Ours v2' shows artifacts in some outputs, leading to higher diversity but lower user preference. Our diversity results for the style sampling setup have some variation and are sensitive the mapper network training, but are still either on-par better than the baselines.
Conclusion. We investigated the effectiveness of embedding pre-training for the task of multi- modal 12I translation. The pre-training can be done once on auxiliary data, and generalizes well to other domains. This allows for faster training of I2I translation networks with fewer losses and achieves more faithful style capture and transfer. Furthermore, we studied the contribution of different loss terms, where we discovered that noise added by variational auto- ento--encodrr can limit the expressiveness of fow-dimensional latent spaces. Finally, we achieved state-of- of-the--ttt-tte- results on several benchmarks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented CycleGAN: Learning many-to-many mappings from unpaired data. In ICML, 2018.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. arXiv preprint arXiv: 08888800777 2018.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In iccv, pp. 1511-1 2017.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha. Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In cvpr, pp. 8799, 2018.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430, 2015.
Jeff Donahue, Philipp Krähenbühl,, and Trevor Darrell. Adversarial feature learning. iclr, 2016.
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learning. In ICCV, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. iclr, 2016.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In nips, pp. (680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In nips, pp. 5767-5777,
Yedid Hoshen and Lior Wolf. Nam: Non-adversarial unsupervised domain mapping. In ecev, pp. 436-451, 2018.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. cvpr, pp. 5811-5819, 2019.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-inage translation. ECCV, 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Wu Jie.
Facial expression recognition.
https://github. www. www//uJlee1100.. 2018. Accessed: 2019-09-22.
Facial - Pxppressiooo.eeecommii........is
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Animesh Karnewar and Raghu Sesha Iyengar. Msg-gan: Multi-scale gradients gan for stable and synchronized multi-scale image synthesis. arXiv preprint
2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A btyle generator architecture for generative adversarial networks. cvpr, 2019.
KDEF, Karolinska directed emotional faces (kdef) dataset. lht se 2017. Accessed: 2019-09-22.

---

Under review as a conference paper at ICLR 2020
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In ICML, 2017.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. ICLR, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. icml, 2016.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.
HisinYYing Lee, Hung-Yu Yu Tseng, Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In ECCV, 2018.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv preprini arXiv: 1rXiv:1809.09087 2018.
Ming-Yu Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NeurIPS, 2017.
Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Lue Van Gool. Exemplar guided unsupervised image-to-image translation. iclr, 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pp. 181-196, 2018
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. iclr, 2016.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, 2017.
Ricardo Martin-Brutn Brualla. Space needle timelapse. 2019-09-22.
2RMKZ3B 2007. Accessed:
Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, and Sean Fanello. LookinGood: Enhancing performance capture with real-time neural re-rendering. In Proc. SIG- GRAPH Asia, 2018
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin- Martin-Brualla. Neural rerendering in the wild. cvpr, 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1411
2014.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84. Springer, 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In icml, pp. 2642-2651. JMLR. org, 2017.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. Deep face recognition. In bmve, volume 1, pp. 6, 2015.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexci A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
10

---

Under review as a conference paper at ICLR 2020
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. iclr, 2016.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lee. Generative adversarial text to image synthesis. icml, 2016.
Mihaela Rosca, Balaji Lakshminarayanan, and David Warde-Farley Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. stat, 1050:15, 2017.
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and Kevin Murphy. Xgan: Unsupervised mmage-to-image translation for many-to- to-many mappings. arXiv preprint arXiv: 1711.0513, 2017.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In cvpr, pp. 815-823, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In nips, pp. 3483-3491, 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, pp. 843-852, 2017.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. tog, 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In nips, pp. 4798, 2016.
Ting- Chug- Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018a.
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Nikolai Yakovenko, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In NeurIPS, 2018b.
Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ecev, pp. 649-666. Springer, 2016.
Richard Zhang, Jun-Ya Zhu, Phillip Isola, Xinyang Geng, Angela Lin, Tianhe Yu, and Alexei A Efros. Real-time -sedde image colorization with learned deep priors. tog, 36(4): 2017a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In cvpr, pp. 586-595, 2018.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progreessionreerrsssinn by conditional adversarial autoencoder. In CVPR, 2017b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired taage-to-imaae translation using cycle-consistent adversarial networks. In ICCV, 2017a.
Jun Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Shechtman. Toward multimodal image-to toaeeett-imagee translation. In NewrIPS, 2017b.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The generator network G has a symmetric encoder-decoder architecture based on Wang al. 8ap. with extra skip connections by concatenating feature maps of the encoder and decoder. We use multiscale-patchhi
discriminator (Wang et al. 2018a) with scales and employ LSGAN (Mao et al. 2017) loss. The mapper network M is multi-layer perceptron (MLP) with three 128- dimensional hidden layers and a tanh activation function. For the reconstruction loss, we use the perceptual loss (Johnson al. 2016) evaluated at convi,2 for i E [1,5] of VGG (Simonyan & Zisserman 2014) with linear weights of w, = 1/26-i for i [1, The architecture of the style encoder E' is adopted from Lee et al. (2018). , and we use latent style vector & E IR Our optimizers setup is similar to that in Zhu et al. 2017b). We use three Adam optimizers: one for the generator G and encoder E. another for the discriminator D, and another optimizer for the generator G alone with 31 = 0, B2 0.99 for the three optimizers, and learning rates of 0.001, 0.001 and 0.0001 respectively. We use a separate Adam optimizer for the mapper network M with 31 = 0.5, 32 = 0.99, and learning rate of 0.01 with decay rate of 0.7 applied every 50 steps. Relative weights for the loss terms are AcGAN = 1, Avee = 0.02 and AL2 = 0.01 for the GAN loss, reconstruction loss, and L2 latent vector regularization respectively. When sampling triplets for any anchor image Ic, we use ke 5, k 13 for the size of the set of close and far neighbors respectively.
A.2 MORE QUANTITATIVE COMPARISON
We report the Inception Score (IS) computed over the validation set of various datasets in Table S Surprisingly, results after finetuning ("ours- stage 3") are slightly worse than those before finetuning ("ours- stage 2"),, but both are still better than the baselines except for the maps dataset. We also note the Inception Score is not very suited to image-to-image translation tasks, since it prefers output diversity with respect to ImageNet classes, not within-class diversity as in our case.
Table 5: Inception score comparison (higher is better) for different datasets.
handbags
shoes
facades
night2day
maps
space needle
Bicycle vl
2.13
2.83
1.41
1.65
3.26
1.82
Bicycle v2
2.07
2.64
1.45
1.74
3.57
1.77
stage
2.22
2.75
1.61
1.76
3.32
1.90
ours stage
2.15
2.85
1.56
1.84
3.28
1.89
A.3 LATENT SPACE VISUALIZATION
Figure 6a| visualizes the latent space learned by the style encoder E' after pretraining and before finetuning (a), after finetuning (b), and the latent space learned by BicycleGAN (Zhu et al. 2017b) (c). The embedding learned through pretraining and before finetuning shows meaningful clusters. Finetuning further brings the embedding closer to that of BicycleGAN.
A.4 ENCODER TRAININGN WITH NON-STYLE METRICS
Pre-training the encoder using a style-based triplet loss showed to be successful for multi-modal image translation tasks where the variability in the target domain is mainly boser- This is shown in the results obtained on several benchmarks, even before the finetuning stage ("ours stage 2" in Table We note though that the usage of style-loss as distance metric for triplet sampling is just one choice and can be replaced with other distance metrics depending on the target application. Triplet sampling with style distance results in learning an embedding space where images with similar colors/styles lie closely in that space as shown in Section A.3 If. for example, we sample triplets instead based on the distance between VGG-Face (Parkhi et al. 2015) embeddings, the encoder will learn latent space which is clustered by identity. In this section, aim to validate that the proposed pre-traine training strategy can be extended to multi-modal image-to-imas to-image translation tasks with non-style variability. We inspect the task of manipulating facial expressions, where the input is
12

---

Under review as a conference paper at ICLR 2020
neutral face, and the output have other emotions facial expressions. For this task, similar emotions should embedded closely in the latent space. We therefore use an off-the-shelf facial expression recognition system compute the emotion similarity/distance between any pair of images. Specifically, we compute the emotion distance as the euclidean distance between the 512-dimensional feature map of the last layer of pretrained classification network (e.g.. (Tie) 2018)) We visualize the learned latent space in Figure 7 which shows clusters with similar emotions or facial expressions. We also show example translation results on holdout set of the front-view images of the KDEF dataset (KDEF) 2017) in Figure [ We note that the generator successfully learns to manipulate facial expressions based solely on the pre-trained embeddings (without the finetuning stage). On the other hand, the DicycleGAN-basse baselines collapsed to single mode (over 3 different runs). This shows that our
unlike the BicycleGAN baselines which will require careful hyper-parameter tuning to work properly on this task. We also point out that the poor output quality is mainly due to using a pixel-wise reconstruction loss for the generator training, while the input-output pairs in this dataset are not aligned. We didn't 't investigate improving the generator training as this is orthogonal to verifying the generalization of encoder pre-training.
approach stable and not sensitive to
A.5 STYLE SAMPLING COMPARISON
Figure 9 compares style sampling using the mapper network M vs adhoc sampling from the assumed N o of an 12-regularized latent space, where μa, a are empirically computed from the training set. Note that adhoc sampling can sometimes sample bad style codes outside the distribution (e.g. third image in first row, and first image in third row in the right side of Figure 2 since the assumption that a 1.2-regularized space would yield normally distributed latents with zero mean and low standard deviation is not explicitly enforced.
A.6 TRAINING TIME
Simplifying the training objective allows for faster training time, well larger batch size due to lower memory usage. Table Table/6]shows the processing time per 1000 training images for the baselines as well different variations of our approach defined in Table 4
Table Training time (in seconds) per 1000 images for the
baselines, well as different versions of our approach (defined
in Table
Batch size
timc/kimg. (sec)
Max batch size
time/kimg, (scc)
Approach
Bicycle v1
93.11
12
85.36
Bicycle v2
155.72
8
155.72
Ours vl
145.50
8
145.50
Ours v2
98.55
12
93.04
Ours v3
64.92
16
53.80
Ours v4
8
63.96
16
51.23
A.7 CONVERGENCE ANALYSIS
Figure 10 compares the convergence of our staged training compared to the BicycleGAN baselines. The dotted line the graph marks the transition between stages and 3 of our training (i.e, switching from fixed pre- pre-trained encoder E. to finetuning both G and E together). We measure the recon- struction error (LPIPS) of the validation set of the edges2handbags dataset as the training progresses. Results show that with fixed pre-trained encoder, staged training starts with higher error than the baselines, but quickly drops to show similar performance as the baselines, and even beats the baselines before switching stage (marked by dotted line). When starting to finetune the encoder E, we get a spike in the reconstruction error as the network adapts to the shift in the pre-trained embeddings, but then the performance of the staged training steadily widens the performance gap with the baselines.
13

---

Under review as a conference paper at ICLR 2020
This shows the importance of the finetuning stage to tweak the pre-trained embeddings to better serve the image synthesis task.
14

---

Under review as a conference paper at ICLR 2020
(a) Our approach: after style pretraining.
(b) Our approach: after finetuning.
(c) BicycleGAN v1 baseline.
Figure t-SNE plots for the latent style space learned by the style encoder E (a) after style pretraining, (b) after finetuning, and (c) using the BicycleGAN vl baseline.
15

---

Under review as a conference paper at ICLR 2020
Figure 7: I-SNE plot for the pre-trained latent space learned for facial expressions on subset the KDEF dataset.
Dfffernt emotions
Ours
Biyycle
Bicycle
Figure 8: Emotion translation results. First row shows the input image, as well as the ground truth images from which we encode the latent emotion vector for reconstruction. Our staged training approach is able to achieve multi-modal synthesis, while the baselines collapse to single mode.
Sampling using mepper
Sampling
Figure 9: Sixteen randomly sampled styles using both the mapper network M (left), as well as adhoc sampling from the empirically computed N a) distribution of 1.2-regularized latent space (right). Adhoc sampling could sample bad style codes outside the latent distribution (marked in red).
16

---

Under review as a conference paper at ICLR 2020
Bicycle v1
Bicycle v2
Staged
0.275
0.25
LPIPS
0.2
0.175
90000
Train step
Figure 10: Convergence comparison between the proposed staged training (ours v3) and the BicycleGAN baselines measured by the reconstruction error (LPIPS) of the validation set of the edges2handbags dataset. Dotted line shows the transition between stages 2 and 3 of our training (i.e, switching from fixed E to finetuning both G and E together).
17


Reviews:
[{"review_id": "rJxOGmc6cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to use a non-end-to-end approach to the problem of multi-modal I2I. Firstly, a metric learning problem is solved to embed images into space, taking into account the pairwise style discrepancy (style is defined, e.g., based on VGG Gramians). As the notion of style is universal for similar datasets, this step further is shown to be generalizable. Secondly, the generator is trained on a supervised image translation tasks: the original image and the style, extracted from the target image, are fed to the generator, and the output is a translated image. Thirdly, style encoder and generator are simultaneously finetuned.\n\nOverall, this is an incremental work in the field of supervised I2I.\n\nQuestions:\n1. Is it true that the proposed approach requires semantically aligned datasets, as the style of the whole image is described with a comparatively low-dimensional vector, and the GAN objective is applied to paired outputs only? Compare, e.g., with Gramian-based style transfer, where segmentation masks are often desired for better results [1].\n2. Can the developed pipeline be generalized to the unsupervised setting, e.g., involving a cycle consistency loss and a non-conditional GAN objective? To my mind, such generalization can show the greater importance of the described method.\n\nRemarks:\n1. Formula (1) is incorrect. I guess the first term should contain z instead of g. Otherwise, the encoder parameters are optimized using the regularizer only.\n\n[1] Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, Jung-Woo Ha. Photorealistic Style Transfer via Wavelet Transforms. ICCV 2019.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJlCmGvTFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the Image-to-Image translation task via a simplified yet more effective training procedure. \n\nCompared to the direct baseline BicycleGAN, the training procedure proposed in this paper replaces the simultaneous training of the\nencoder E and the generator G with a staged training that alternatively trains on E and G and then finetune them together. Although this appears to be a simple modification, the empirical performance for generalization and reconstruction qualities prove the effectiveness of the proposal. \n\nIt is better to provide more intuition on why this pretraining phase would help to make the results generalize better and yield better performance. The current presentation of the paper mostly consists of detailed descriptions of the proposal training procedure, without some interesting discussions about why this pretraining makes the problem easier. For instance, I'm interested in seeing with some toy distributions, what is the training progress (measured quantitatively) comparing the proposed method and traditional BicycleGAN. \n\nAlthough the results look nice, with the current presentation, there's not much inspiration one could get from the paper. I encourage the authors to make some adjustments, and I will reconsider the score. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xj2bW0YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors tackle the problem of multi-modal image-to-image translation by pre-training a style-based encoder. The style-based encoder is trained with a triplet loss that encourages similarity between images with similar styles and dissimilarity between images with different styles. The output of the encoder is a style embedding that helps differentiates different modes of image synthesis.  When training the generator for image synthesis, the input combines an image in the source and a style embedding, and the loss is essentially the sum of image conditional GAN loss and perceptual loss. Additionally, the authors propose a mapping function to sample styles from a unit Gaussian distribution.\n\nI think the idea of pre-training a style-based encoder is straightforward. I am mainly concerned about the performance of the presented approach. First, there are no many visual comparisons in the paper. The only visual comparison is in Figure 8, but results are only limited to faces. The visual results in Figure 5 do not look appealing to me. The change in the style mainly comes from the global change in color: no much change in the texture or local color. The \"night2day\" results look poor to me.  I am concerned about the diversity of the styles learned in the model.\n\nOn the other hand, I am convinced that the proposed model is better than BicycleGAN, and the approach is somehow novel. The user study in Table 5 suggests that the proposed method is somehow better than BicyleGAN in visual quality on one task. My overall rating is borderline.\n\nMinor comments:\n- In the first sentence of Section 3.2, I do not think \"one-to-one correspondence\" is the right description. The encoder is not expected to be invertible.\n- In Equation (3), \"e_i\" is a little bit misleading. It does not mean the i-th element in {\"e_j\"}. You may want to replace \"e_i\" with \"s_i\" to avoid confusion. \n- The explanation of ours v1, v2, v3, v4 is not clear. It is also difficult to find its definition.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission describes a new two-stage training scheme for multi-modal image-to-image translation. The new scheme is compared to a single-stage end-to-end baseline, and the advantage of the new scheme is demonstrated empirically. All three reviewers appreciate the proposed contribution and the quality improvement it brings over the baseline. At the same time, the reviewers see the contribution as incremental and not sufficient for an ICLR paper. The author response and paper adjustment have not changed the opinion of the reviewers, so the overall recommendation is to reject.
Decision: Reject