Style-based Encoder Pre-training for Multi-modal Image Synthesis

1 INTRODUCTION . Image-to-Image ( I2I ) translation is the task of transforming images from one domain to another ( e.g. , semantic maps→ scenes , sketches→ photo-realistic images , etc. ) . Many problems in computer vision and graphics can be cast as I2I translation , such as photo-realistic image synthesis ( Chen & Koltun ( 2017 ) ; Isola et al . ( 2017 ) ; Wang et al . ( 2018a ) ) , super-resolution ( Ledig et al . ( 2017 ) ) , colorization ( Zhang et al . ( 2016 ; 2017a ) ) , and inpainting ( Pathak et al . ( 2016 ) ) . Therefore , I2I translation has recently received significant attention in the literature . One main challenge in I2I translation is the multi-modal nature for many such tasks – the relation between an input domain A and an output domain B is often times one-to-many , where a single input image IAi ∈ A can be mapped to different output images from domain B . For example , a sketch of a shoe or a handbag can be mapped to corresponding objects with different colors or styles , or a semantic map of a scene can be mapped to many scenes with different appearance , lighting and/or weather conditions . Since I2I translation networks typically learn one-to-one mappings due to their deterministic nature , an extra input is required to specify an output mode to which an input image will be translated . Simply injecting extra random noise as input proved to be ineffective as shown in ( Isola et al . ( 2017 ) ; Zhu et al . ( 2017b ) ) , where the generator network just learns to ignore the extra noise and collapses to a single or few modes ( which is one form of the mode collapse problem ) . To overcome this problem , Zhu et al . ( 2017b ) proposed BicycleGAN , which learns to encode the distribution of different possible outputs into a latent vector z , and then learns a deterministic mapping G : ( A , z ) → B . So , depending on the latent vector z , a single input IAi ∈ A can be mapped to multiple outputs in B . While BicycleGAN requires paired training data , several works ( Lee et al . ( 2018 ) ; Huang et al . ( 2018 ) ) extended it to the unsupervised case , where images in domains A and B are not in correspondence ( ‘ unpaired ’ ) . One main component of unpaired I2I is a cross-cycle consistency constraint , where the network generates an intermediate output by swapping the styles of a pair of images , then swaps the style between the intermediate output again to reconstruct the original images . This enforces that the latent vector z preserves the encoded style information when translated from an image i to another image j and back to image i again . This constraint can also be applied to paired training data , where it encourages style/attribute transfer between images . However , training BicycleGAN ( Zhu et al . ( 2017b ) ) or its unsupervised counterparts ( Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) is not trivial . For example , BicycleGAN combines the objectives of both conditional Variational Auto-Encoders ( cVAEs ) ( Sohn et al . ( 2015 ) ) and a conditional version of Latent Regressor GANs ( cLR-GANs ) ( Donahue et al . ( 2016 ) ; Dumoulin et al . ( 2016 ) ) to train their network . The training objective of ( Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) is even more involved to handle the unsupervised setup . In this work , we aim to simplify the training of general purpose multi-modal I2I translation networks , while also improving the diversity and expressiveness of different styles in the output domain . Our approach is inspired by the work of Meshry et al . ( 2019 ) which utilizes a staged training strategy to re-render scenes under different lighting , time of day , and weather conditions . We propose a pretraining approach for style encoders , in multi-modal I2I translation networks , which makes the training simpler and faster by requiring fewer losses/constraints . Our approach is also inspired by the standard training paradigm in visual recognition of first pretraining on a proxy task , either large supervised datasets ( e.g. , ImageNet ) ( Krizhevsky et al . ( 2012 ) ; Sun et al . ( 2017 ) ; Mahajan et al . ( 2018 ) ) or unsupervised tasks ( e.g. , Doersch et al . ( 2015 ) ; Noroozi & Favaro ( 2016 ) ) , and then fine-tuning ( transfer learning ) on the desired task . Similarly , we propose to pretrain the encoder using a proxy task that encourages capturing style into a latent space . Our goal is to highlight the importance of pretraining for I2I networks and demonstrate that a simple approach can be very effective for multi-modal image synthesis . In particular , we make the following contributions : • We explore style pretraining and its generalization for the task of multi-modal I2I translation , which simplifies and speeds up the training compared to competing approaches . • We provide a study of the importance of different losses and regularization terms for multi-modal I2I translation networks . • We show that the pretrained latent embeddings is not dependent on the target domain and generalizes well to other domains ( transfer learning ) . • We achieve state-of-the art results on several benchmarks in terms of style capture and transfer , and diversity of results . 2 RELATED WORK . Deep generative models There has been incredible progress in the field of image synthesis using deep neural networks . In its unconditional setting , a decoder network learns to map random values drawn from a prior distribution ( typically Gaussian ) to output images . Variational Auto-Encoders ( VAEs ) ( Kingma & Welling ( 2014 ) ) assume a bijection mapping between output images and some latent distribution and learn to map the latent distribution to a unit Gaussian using the reparameterization trick . Alternatively , Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) directly map random values sampled from a unit Gaussian to images , while using a discriminator network to enforce that the distribution of generated images resembles that of real images . Recent works proposed improvements to stabilize the training ( Gulrajani et al . ( 2017 ) ; Karnewar & Iyengar ( 2019 ) ; Mao et al . ( 2017 ) ; Radford et al . ( 2016 ) ) and improve the quality and diversity of the output ( Karras et al . ( 2018 ; 2019 ) ) . Other works combine both VAEs and GANs into a hybrid VAE-GAN model ( Larsen et al . ( 2016 ) ; Rosca et al . ( 2017 ) ) . Conditional image synthesis Instead of generating images from input noise , the generator can be augmented with side information in the form of extra conditional inputs . For example , Sohn et al . ( 2015 ) extended VAEs to their conditional setup ( cVAEs ) . Also , GANs can be conditioned on different information , like class labels ( Mirza & Osindero ( 2014 ) ; Odena et al . ( 2017 ) ; Van den Oord et al . ( 2016 ) ) , language description ( Mansimov et al . ( 2016 ) ; Reed et al . ( 2016 ) ) , or an image from another domain ( Chen & Koltun ( 2017 ) ; Isola et al . ( 2017 ) ) . The latter is called Image-to-Image translation . Image-to-Image ( I2I ) translation I2I translation is the task of transforming an image from one domain , such as a sketch , into a another domain , such as photo-realistic images . While there are regression-based approaches to this problem ( Chen & Koltun ( 2017 ) ; Hoshen & Wolf ( 2018 ) ) , significant successes in this field are based on GANs and the influential work of pix2pix ( Isola et al . ( 2017 ) ) . Following the success of pix2pix ( Isola et al . ( 2017 ) ) , I2I translation has since been utilized in a large number of tasks , like inpainting ( Pathak et al . ( 2016 ) ) , colorization ( Zhang et al . ( 2016 ; 2017a ) ) , super-resolution ( Ledig et al . ( 2017 ) ) , rendering ( Martin-Brualla et al . ( 2018 ) ; Meshry et al . ( 2019 ) ; Thies et al . ( 2019 ) ) , and many more ( Dong et al . ( 2017 ) ; Wang & Gupta ( 2016 ) ; Zhang et al . ( 2017b ) ) . There has also been works to extend this task to the unsupervised setting ( Hoshen & Wolf ( 2018 ) ; Kim et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Ma et al . ( 2019 ) ; Royer et al . ( 2017 ) ; Zhu et al . ( 2017a ) ) , to multiple domains ( Choi et al . ( 2018 ) ) , and to videos ( Chan et al . ( 2018 ) ; Wang et al . ( 2018b ) ) . Multi-modal I2I translation Image translation networks are typically deterministic function approximators that learn a one-to-one mapping between inputs and outputs . To extend I2I translation to the case of diverse multi-modal outputs , Zhu et al . ( 2017b ) proposed the BicycleGAN framework that learns a latent distribution that encodes the variability in the output domain and conditions the generator on this extra latent vector for multi-modal image synthesis . Wang et al . ( 2018a ; b ) learn instance-wise latent features for different objects in a target image , which are clustered after training to find f fixed modes for different semantic classes . At test time , they sample one of the feature clusters for each object to achieve multi-modal synthesis . Other works extended the multi-modal I2I framework to the unpaired settings where images from the input and output domains are not in correspondence ( Almahairi et al . ( 2018 ) ; Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) by augmenting BicycleGAN with different forms of a cross-cycle consistency constraint between two unpaired images . In our work , we focus on the supervised setting of multi-modal I2I translation . We propose a simple , yet effective , pretraining strategy to learn a latent distribution that encodes variability in the output domain . The learned distribution can be easily adapted to new unseen datasets with simple fine-tuning , instead of training from random initialization . 3 APPROACH . Current multi-modal image translation networks require an extra input z that allows for modelling the one-to-many relation between an input domain A and an output domain B as a one-to-one relation from a pair of inputs ( A , z ) → B . In previous approaches , there has been a trade-off between simplicity and effectiveness for providing the input z . On one hand , providing random noise as the extra input z maintains a simple training objective ( same as in pix2pix ( Isola et al . ( 2017 ) ) ) . However , Isola et al . ( 2017 ) ; Zhu et al . ( 2017b ) showed that the generator has little incentive to utilize the input vector z since it only encodes random information , and therefore the generator ends up ignoring z and collapsing to one or few modes . On the other hand , BicycleGAN ( Zhu et al . ( 2017b ) ) combines the objectives of both conditional Variational Auto-Encoder GANs ( cVAE-GAN ) and conditional Latent Regressor GANs ( cLR-GAN ) to learn a latent embedding z simultaneously with the generator G. Their training enforces two cycle consistencies : B → z → B̂ and z → B̃ → ẑ . This proved to be very effective , but the training objective is more involved , which makes the training slower . Also , since the latent embedding is being trained simultaneously with the the generator , hyper-parameter tuning becomes more critical and sensitive . We aim to combine the best of both worlds : an effective training of a latent embedding that models the distribution of possible outputs , while retaining a simple training objective . This would allow for faster and more efficient training , as well as less sensitivity to hyper-parameters . We observe that the variability in many target domains can be represented by the style diversity of images in the target domain B , where the style is defined in terms of the gram matrices used in the Neural Style Transfer literature ( Gatys et al . ( 2016 ) ) . Then , we learn an embedding by separately training an encoder network E on an auxiliary task to optimize for z = E ( IB ) capturing the style of an image IB . Finally , since we now have learned a deterministic mapping between z and the style of the target output image IB , training the generator G becomes simpler as G is just required to discover the correlation between output images and their corresponding style embedding z . To incorporate this into BicycleGAN ( Zhu et al . ( 2017b ) ) , we replace the simultaneous training of the encoder E and the generator G with a staged training as follows : Stage 1 : Pretrain E on a proxy task that optimizes an embedding of images in the output domain B into a low-dimensional style latent space , such that images with similar styles lie closely in that space ( i.e . clustered ) . Stage 2 : Train the generator network G while fixing the encoder E , so that G learns to associate the style of output images to their deterministic style embedding z = E ( IB ) . Stage 3 : Fine-tune both the E and G networks together , allowing for the style embedding to be further adapted to best suit the image synthesis task for the target domain . The intuition why such staged training would be effective for multi-modal I2I translation is that the encoder is pretrained to model different modes of the output distribution as clusters of images with similar styles ( refer to the supp . material , figures 6,7 , for a visualization of pretrained latent embeddings ) . During stage 2 , the latent space is kept fixed , and the input latent to the generator can be used to clearly distinguish the style cluster to which the output belongs , which makes the multi-modal synthesis task easier for the generator . Finally , stage 3 finetunes the learned embedding to better serve the synthesis task at hand . Next , we explain how to pre-train the style encoder network E in Section 3.1 , and how to train the generator G using the pre-learned embeddings ( Section 3.2 ) . Finally , we demonstrate the generalization of pre-training the style encoder E in Section 3.3 .

Summary: In this paper, the authors tackle the problem of multi-modal image-to-image translation by pre-training a style-based encoder. The style-based encoder is trained with a triplet loss that encourages similarity between images with similar styles and dissimilarity between images with different styles. The output of the encoder is a style embedding that helps differentiates different modes of image synthesis.  When training the generator for image synthesis, the input combines an image in the source and a style embedding, and the loss is essentially the sum of image conditional GAN loss and perceptual loss. Additionally, the authors propose a mapping function to sample styles from a unit Gaussian distribution.