Unsupervised Video-to-Video Translation

Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time . In this paper , we formulate a new task of unsupervised video-tovideo translation , which poses its own unique challenges . Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames . We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks , and propose a spatio-temporal 3D translator as an alternative solution to this problem . We evaluate our 3D method on multiple synthetic datasets , such as moving colorized digits , as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset . Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach , which is better able to learn the complex structure of video and motion and continuity of object appearance . 1 INTRODUCTION . Recent work on unsupervised image-to-image translation ( Zhu et al. , 2017 ; Liu and Tuzel , 2016 ; Liu et al. , 2017b ) has shown astonishing results on tasks like style transfer , aerial photo to map translation , day-to-night photo translation , unsupervised semantic image segmentation and others . Such methods learn from unpaired examples , avoiding tedious data alignment by humans . In this paper , we propose a new task of unsupervised video-to-video translation , i.e . learning a mapping from one video domain to another while preserving high-level semantic information of the original video using large numbers of unpaired videos from both domains . Many computer vision tasks can be formulated as video-to-video translation , e.g. , semantic segmentation , video colorization or quality enhancement , or translating between MRI and CT volumetric data ( illustrated in Fig . 1 ) . Moreover , motion-centered tasks such as action recognition and tracking can greatly benefit from the development of robust unsupervised video-to-video translation methods that can be used out-of-the-box for domain adaptation . Since a video can be viewed as a sequence of images , one natural approach is to use an image-toimage translation method on each frame , e.g. , applying a state-of-art method such as CycleGAN ( Zhu et al. , 2017 ) , CoGAN ( Liu and Tuzel , 2016 ) or UNIT ( Liu et al. , 2017b ) . Unfortunately , these methods can not preserve continuity and consistency of a video when applied frame-wise . For example , colorization of an object may have multiple correct solutions for a single input frame , since some objects such as cars can have different colors . Therefore , there is no guarantee that an object would preserve its color if translation is performed on the frame level frame . In this paper , we propose to translate an entire video as a three-dimensional tensor to preserve its cross-frame consistency and spatio-temporal structure . We employ multiple datasets and metrics to evaluate the performance of our proposed video-to-video translation model . Our synthetic datasets include videos of moving digits of different colors and volumetric images of digits imitating medical scans . We also perform more realistic segmentation-to-RGB and colorization experiments on the GTA dataset ( Richter et al. , 2016 ) , and propose a new MRI-to-CT dataset for medical volumetric image translation , which to our knowledge is the first open medical dataset for unsupervised volumeto-volume translation . Under review as a conference paper at ICLR 2019 Figure 1 : We propose the task of unsupervised video-to-video translation . Left : Results of MR-to-CT translation . Right : moving MNIST digits colorization . Rows show per-frame CycleGAN ( 2D ) and our spatio-temporal extension ( 3D ) . Since CycleGAN takes into account information only from the current image , it produces reasonable results on the image level but fails to preserve the shape and color of an object throughout the video . Best viewed in color . Figure 2 : Results of GTA video colorization show that per-frame translation of videos does not preserve constant colours of objects within the whole sequence . We provide more results and videos in the supplementary video : https : //bit.ly/2R5aGgo . Best viewed in color . Our extensive experiments show that the proposed 3D convolutional model provides more accurate and stable video-to-video translation compared to framewise translation with various settings . We also investigate how the structure of individual batches affects the training of framewise translation models , and find that structure of a batch is very important for stable translation contrary to an established practice of shuffling training data to avoid overfitting in deep models ( Goodfellow et al. , 2016 ) . To summarize , we make the following main contributions : 1 ) a new unsupervised video-to-video translation task together with both realistic and synthetic proof-of-concept datasets ; 2 ) a spatiotemporal video translation model based on a 3D convnet that outperforms per-frame methods in Under review as a conference paper at ICLR 2019 all experiments , according to human and automatic metrics , and 3 ) an additional analysis of how performance of per-frame methods depends on the structure of training batches . 2 RELATED WORK . In recent years , there has been increasing interest in unsupervised image-to-image translation . Aside from producing interesting graphics effects , it enables task-independent domain adaptation and unsupervised learning of per-pixel labels . Many recent translation models ( Zhu et al. , 2017 ; Liu and Tuzel , 2016 ; Liu et al. , 2017b ) use the adversarial formulation initially proposed by ( Goodfellow et al. , 2014 ) as an objective for training generative probabilistic models . The main intuition behind an adversarial approach to domain translation is that the learned cross-domain mapping F : X → Y should translate source samples to fake target samples that are indistinguishable from actual target samples , in the sense that no discriminator from a fixed hypothesis space H should be capable of distinguishing them . Many recent advances in domain translation are due to the introduction of the cycle-consistency idea ( Zhu et al. , 2017 ) . Models with a cycle consistency loss aim to learn two mappings F ( x ) and G ( y ) such that not only are the translated source samples F ( x ) indistinguishable from the target , and G ( y ) are indistinguishable from source , but they are also inverses of each other , i.e . F ( G ( y ) ) = y , G ( F ( x ) ) = x . We also employ this idea and explore how well it generalizes to video-to-video translation . The cycle-consistency constraints might not restrict semantic changes as explored by Hoffman et al . ( 2017 ) . There has been work on combining cycle-consistency with variational autoencoders that share latent space for source and target ( Liu et al. , 2017a ) , which resulted in more visually pleasing results . Both adversarial ( Isola et al. , 2016 ) and non-adversarial ( Chen and Koltun , 2017 ) supervised imageto-image translation models archive much better visual fidelity since samples in source and target datasets are paired , but their use is limited since we rarely have access to such aligned datasets for different domains . Adversarial video generation has also gained much traction over last years with frame-level models based on long short-term memory ( Srivastava et al. , 2015 ) as well as spatiotemporal convolutions ( Vondrick et al. , 2016 ) , especially since adversarial terms seem to better avoid frame blurring ( Mathieu et al. , 2015 ) than Euclidian distance . However , none of these works consider learning a conditional generative video model from unpaired data , i.e . cross domain video translation . Here , we propose this problem and a solution based on jointly translating the entire volume of video . Under review as a conference paper at ICLR 2019 3 3D CONVOLUTIONAL VIDEO-TO-VIDEO TRANSLATION . We introduce a neural approach for video-to-video translation based on a conditional GAN ( Isola et al. , 2017 ; Goodfellow et al. , 2014 ) that treats inputs and outputs as three-dimensional tensors . The network takes a volumetric image ( e.g . a video ) from domain A and produces a corresponding volume of the same shape in the domain B . The generator module aims to generate realistic volumes , while the discriminator aims to discriminate between the real and generated samples . Similarly to the CycleGAN method , we introduce two generator-discriminator pairs and add a cycle consistency loss ensuring that samples mapped from A to B and back to A are close to the originals . We implement the two generators , F and G , as 3D convolutional networks ( Ji et al. , 2013 ) that follow the architecture described in ( Johnson et al. , 2016 ) . The networks consist of three convolutional layers with 3D convolutional filters of shape 3×3×3 , nine resudual blocks and two additional convolutional layers with stride 12 . The networks receive image sequences as 3D tensors of shape d× h×w , where d is the length of sequence and h and w are image height and width if an input video is grayscale , and 4D tensor of shape d× h× w × 3 if an input video is colored and represented as a sequence of RGB images . Since 3D convolutional networks require a large amount of GPU memory , the choice of the depth d of the input video is usually limited by the the memory of a single GPU unit ; we used d = 8 , h = 108 , w = 192 for the experiments with GTA datasets and d = 30 , h = 84 , w = 84 for all experiments on MNIST . The discriminators DA and DB are PatchGANs ( Isola et al. , 2017 ) as in CycleGAN , but with 3D convolutional filters . They each receive a video of size d× h× w and classify whether the overlapping video patches are real samples from the respective domain or are created by the generator network . The overall objective of the model consists of the adversarial loss LGAN and the cycle consistency loss Lcyc . The adversarial LGAN loss forces both the generator networks to produce realistic videos and the discriminators to distinguish between real and fake samples from the domain in a min-max fashion , whereas Lcyc ensures that each sample x ∼ pA translated into domain B and back is equal to the original and vice versa , i.e . G ( F ( x ) ) ≈ x ( see Fig . 3 ) . The adversarial loss LGAN is a log-likelihood of correct classification between real and synthesized volumes : LGAN ( DB , G , X , Y ) = Ey∼pB log ( DB ( y ) ) + Ex∼pA log ( 1−DB ( G ( x ) ) where the generator G is learned in a way that minimizes LGAN , while the discriminator DB aims to maximize it . The cycle consistency loss ( Zhu et al. , 2017 ) is the L1 loss between the input volume and result of the reverse translation : Lcyc = Ex∼pA ( ‖G ( F ( x ) ) − x‖1 ) + Ey∼pb ( ‖F ( G ( y ) ) − y‖1 ) The total objective can be written as follows : L ( G , F , DA , DB ) = LGAN ( G , DB , X , Y ) + LGAN ( F , DA , Y , X ) + γLcyc ( G , F ) ( 1 ) Because we employ the cycle loss and the PatchGAN architecture also employed by CycleGAN , we refer to our model as 3D CycleGAN . More generally , we can consider other generator and discriminator implementations within the overall 3D convolutional framework for video-to-video translation .

Summary: This paper proposes a 3D convolutional neural network based architecture for video-to-video translation. The method mitigates the inconsistency problem present when image-to-image translation methods are used in the video domain. Additionally, they present a study of ways to better setting up batched for the learning steps during networks optimization for videos, and also, they propose a new MRI-to-CT dataset for medical volumetric image translation. The proposed method outperforms the image-to-image translation methods in most measures.