Style-based Encoder Pre-training for Multi-modal Image Synthesis

1 INTRODUCTION . Image-to-Image ( I2I ) translation is the task of transforming images from one domain to another ( e.g. , semantic maps→ scenes , sketches→ photo-realistic images , etc. ) . Many problems in computer vision and graphics can be cast as I2I translation , such as photo-realistic image synthesis ( Chen & Koltun ( 2017 ) ; Isola et al . ( 2017 ) ; Wang et al . ( 2018a ) ) , super-resolution ( Ledig et al . ( 2017 ) ) , colorization ( Zhang et al . ( 2016 ; 2017a ) ) , and inpainting ( Pathak et al . ( 2016 ) ) . Therefore , I2I translation has recently received significant attention in the literature . One main challenge in I2I translation is the multi-modal nature for many such tasks – the relation between an input domain A and an output domain B is often times one-to-many , where a single input image IAi ∈ A can be mapped to different output images from domain B . For example , a sketch of a shoe or a handbag can be mapped to corresponding objects with different colors or styles , or a semantic map of a scene can be mapped to many scenes with different appearance , lighting and/or weather conditions . Since I2I translation networks typically learn one-to-one mappings due to their deterministic nature , an extra input is required to specify an output mode to which an input image will be translated . Simply injecting extra random noise as input proved to be ineffective as shown in ( Isola et al . ( 2017 ) ; Zhu et al . ( 2017b ) ) , where the generator network just learns to ignore the extra noise and collapses to a single or few modes ( which is one form of the mode collapse problem ) . To overcome this problem , Zhu et al . ( 2017b ) proposed BicycleGAN , which learns to encode the distribution of different possible outputs into a latent vector z , and then learns a deterministic mapping G : ( A , z ) → B . So , depending on the latent vector z , a single input IAi ∈ A can be mapped to multiple outputs in B . While BicycleGAN requires paired training data , several works ( Lee et al . ( 2018 ) ; Huang et al . ( 2018 ) ) extended it to the unsupervised case , where images in domains A and B are not in correspondence ( ‘ unpaired ’ ) . One main component of unpaired I2I is a cross-cycle consistency constraint , where the network generates an intermediate output by swapping the styles of a pair of images , then swaps the style between the intermediate output again to reconstruct the original images . This enforces that the latent vector z preserves the encoded style information when translated from an image i to another image j and back to image i again . This constraint can also be applied to paired training data , where it encourages style/attribute transfer between images . However , training BicycleGAN ( Zhu et al . ( 2017b ) ) or its unsupervised counterparts ( Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) is not trivial . For example , BicycleGAN combines the objectives of both conditional Variational Auto-Encoders ( cVAEs ) ( Sohn et al . ( 2015 ) ) and a conditional version of Latent Regressor GANs ( cLR-GANs ) ( Donahue et al . ( 2016 ) ; Dumoulin et al . ( 2016 ) ) to train their network . The training objective of ( Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) is even more involved to handle the unsupervised setup . In this work , we aim to simplify the training of general purpose multi-modal I2I translation networks , while also improving the diversity and expressiveness of different styles in the output domain . Our approach is inspired by the work of Meshry et al . ( 2019 ) which utilizes a staged training strategy to re-render scenes under different lighting , time of day , and weather conditions . We propose a pretraining approach for style encoders , in multi-modal I2I translation networks , which makes the training simpler and faster by requiring fewer losses/constraints . Our approach is also inspired by the standard training paradigm in visual recognition of first pretraining on a proxy task , either large supervised datasets ( e.g. , ImageNet ) ( Krizhevsky et al . ( 2012 ) ; Sun et al . ( 2017 ) ; Mahajan et al . ( 2018 ) ) or unsupervised tasks ( e.g. , Doersch et al . ( 2015 ) ; Noroozi & Favaro ( 2016 ) ) , and then fine-tuning ( transfer learning ) on the desired task . Similarly , we propose to pretrain the encoder using a proxy task that encourages capturing style into a latent space . Our goal is to highlight the importance of pretraining for I2I networks and demonstrate that a simple approach can be very effective for multi-modal image synthesis . In particular , we make the following contributions : • We explore style pretraining and its generalization for the task of multi-modal I2I translation , which simplifies and speeds up the training compared to competing approaches . • We provide a study of the importance of different losses and regularization terms for multi-modal I2I translation networks . • We show that the pretrained latent embeddings is not dependent on the target domain and generalizes well to other domains ( transfer learning ) . • We achieve state-of-the art results on several benchmarks in terms of style capture and transfer , and diversity of results . 2 RELATED WORK . Deep generative models There has been incredible progress in the field of image synthesis using deep neural networks . In its unconditional setting , a decoder network learns to map random values drawn from a prior distribution ( typically Gaussian ) to output images . Variational Auto-Encoders ( VAEs ) ( Kingma & Welling ( 2014 ) ) assume a bijection mapping between output images and some latent distribution and learn to map the latent distribution to a unit Gaussian using the reparameterization trick . Alternatively , Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) directly map random values sampled from a unit Gaussian to images , while using a discriminator network to enforce that the distribution of generated images resembles that of real images . Recent works proposed improvements to stabilize the training ( Gulrajani et al . ( 2017 ) ; Karnewar & Iyengar ( 2019 ) ; Mao et al . ( 2017 ) ; Radford et al . ( 2016 ) ) and improve the quality and diversity of the output ( Karras et al . ( 2018 ; 2019 ) ) . Other works combine both VAEs and GANs into a hybrid VAE-GAN model ( Larsen et al . ( 2016 ) ; Rosca et al . ( 2017 ) ) . Conditional image synthesis Instead of generating images from input noise , the generator can be augmented with side information in the form of extra conditional inputs . For example , Sohn et al . ( 2015 ) extended VAEs to their conditional setup ( cVAEs ) . Also , GANs can be conditioned on different information , like class labels ( Mirza & Osindero ( 2014 ) ; Odena et al . ( 2017 ) ; Van den Oord et al . ( 2016 ) ) , language description ( Mansimov et al . ( 2016 ) ; Reed et al . ( 2016 ) ) , or an image from another domain ( Chen & Koltun ( 2017 ) ; Isola et al . ( 2017 ) ) . The latter is called Image-to-Image translation . Image-to-Image ( I2I ) translation I2I translation is the task of transforming an image from one domain , such as a sketch , into a another domain , such as photo-realistic images . While there are regression-based approaches to this problem ( Chen & Koltun ( 2017 ) ; Hoshen & Wolf ( 2018 ) ) , significant successes in this field are based on GANs and the influential work of pix2pix ( Isola et al . ( 2017 ) ) . Following the success of pix2pix ( Isola et al . ( 2017 ) ) , I2I translation has since been utilized in a large number of tasks , like inpainting ( Pathak et al . ( 2016 ) ) , colorization ( Zhang et al . ( 2016 ; 2017a ) ) , super-resolution ( Ledig et al . ( 2017 ) ) , rendering ( Martin-Brualla et al . ( 2018 ) ; Meshry et al . ( 2019 ) ; Thies et al . ( 2019 ) ) , and many more ( Dong et al . ( 2017 ) ; Wang & Gupta ( 2016 ) ; Zhang et al . ( 2017b ) ) . There has also been works to extend this task to the unsupervised setting ( Hoshen & Wolf ( 2018 ) ; Kim et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Ma et al . ( 2019 ) ; Royer et al . ( 2017 ) ; Zhu et al . ( 2017a ) ) , to multiple domains ( Choi et al . ( 2018 ) ) , and to videos ( Chan et al . ( 2018 ) ; Wang et al . ( 2018b ) ) . Multi-modal I2I translation Image translation networks are typically deterministic function approximators that learn a one-to-one mapping between inputs and outputs . To extend I2I translation to the case of diverse multi-modal outputs , Zhu et al . ( 2017b ) proposed the BicycleGAN framework that learns a latent distribution that encodes the variability in the output domain and conditions the generator on this extra latent vector for multi-modal image synthesis . Wang et al . ( 2018a ; b ) learn instance-wise latent features for different objects in a target image , which are clustered after training to find f fixed modes for different semantic classes . At test time , they sample one of the feature clusters for each object to achieve multi-modal synthesis . Other works extended the multi-modal I2I framework to the unpaired settings where images from the input and output domains are not in correspondence ( Almahairi et al . ( 2018 ) ; Huang et al . ( 2018 ) ; Lee et al . ( 2018 ) ) by augmenting BicycleGAN with different forms of a cross-cycle consistency constraint between two unpaired images . In our work , we focus on the supervised setting of multi-modal I2I translation . We propose a simple , yet effective , pretraining strategy to learn a latent distribution that encodes variability in the output domain . The learned distribution can be easily adapted to new unseen datasets with simple fine-tuning , instead of training from random initialization . 3 APPROACH . Current multi-modal image translation networks require an extra input z that allows for modelling the one-to-many relation between an input domain A and an output domain B as a one-to-one relation from a pair of inputs ( A , z ) → B . In previous approaches , there has been a trade-off between simplicity and effectiveness for providing the input z . On one hand , providing random noise as the extra input z maintains a simple training objective ( same as in pix2pix ( Isola et al . ( 2017 ) ) ) . However , Isola et al . ( 2017 ) ; Zhu et al . ( 2017b ) showed that the generator has little incentive to utilize the input vector z since it only encodes random information , and therefore the generator ends up ignoring z and collapsing to one or few modes . On the other hand , BicycleGAN ( Zhu et al . ( 2017b ) ) combines the objectives of both conditional Variational Auto-Encoder GANs ( cVAE-GAN ) and conditional Latent Regressor GANs ( cLR-GAN ) to learn a latent embedding z simultaneously with the generator G. Their training enforces two cycle consistencies : B → z → B̂ and z → B̃ → ẑ . This proved to be very effective , but the training objective is more involved , which makes the training slower . Also , since the latent embedding is being trained simultaneously with the the generator , hyper-parameter tuning becomes more critical and sensitive . We aim to combine the best of both worlds : an effective training of a latent embedding that models the distribution of possible outputs , while retaining a simple training objective . This would allow for faster and more efficient training , as well as less sensitivity to hyper-parameters . We observe that the variability in many target domains can be represented by the style diversity of images in the target domain B , where the style is defined in terms of the gram matrices used in the Neural Style Transfer literature ( Gatys et al . ( 2016 ) ) . Then , we learn an embedding by separately training an encoder network E on an auxiliary task to optimize for z = E ( IB ) capturing the style of an image IB . Finally , since we now have learned a deterministic mapping between z and the style of the target output image IB , training the generator G becomes simpler as G is just required to discover the correlation between output images and their corresponding style embedding z . To incorporate this into BicycleGAN ( Zhu et al . ( 2017b ) ) , we replace the simultaneous training of the encoder E and the generator G with a staged training as follows : Stage 1 : Pretrain E on a proxy task that optimizes an embedding of images in the output domain B into a low-dimensional style latent space , such that images with similar styles lie closely in that space ( i.e . clustered ) . Stage 2 : Train the generator network G while fixing the encoder E , so that G learns to associate the style of output images to their deterministic style embedding z = E ( IB ) . Stage 3 : Fine-tune both the E and G networks together , allowing for the style embedding to be further adapted to best suit the image synthesis task for the target domain . The intuition why such staged training would be effective for multi-modal I2I translation is that the encoder is pretrained to model different modes of the output distribution as clusters of images with similar styles ( refer to the supp . material , figures 6,7 , for a visualization of pretrained latent embeddings ) . During stage 2 , the latent space is kept fixed , and the input latent to the generator can be used to clearly distinguish the style cluster to which the output belongs , which makes the multi-modal synthesis task easier for the generator . Finally , stage 3 finetunes the learned embedding to better serve the synthesis task at hand . Next , we explain how to pre-train the style encoder network E in Section 3.1 , and how to train the generator G using the pre-learned embeddings ( Section 3.2 ) . Finally , we demonstrate the generalization of pre-training the style encoder E in Section 3.3 .

Summary: The authors propose to use a non-end-to-end approach to the problem of multi-modal I2I. Firstly, a metric learning problem is solved to embed images into space, taking into account the pairwise style discrepancy (style is defined, e.g., based on VGG Gramians). As the notion of style is universal for similar datasets, this step further is shown to be generalizable. Secondly, the generator is trained on a supervised image translation tasks: the original image and the style, extracted from the target image, are fed to the generator, and the output is a translated image. Thirdly, style encoder and generator are simultaneously finetuned.