Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation

1 INTRODUCTION . Generative adversarial models ( GANs ) have been extremely successful at learning complex distributions such as natural images ( Zhu et al. , 2017 ; Isola et al. , 2017 ) . However , for sequence generation , directly applying GANs without carefully engineered constraints typically results in strong artifacts over time due to the significant difficulties introduced by the temporal changes . In particular , conditional video generation tasks are very challenging learning problems where generators should not only learn to represent the data distribution of the target domain , but also learn to correlate the output distribution over time with conditional inputs . Their central objective is to faithfully reproduce the temporal dynamics of the target domain and not resort to trivial solutions such as features that arbitrarily appear and disappear over time . In our work , we propose a novel adversarial learning method for a recurrent training approach that supervises both spatial content as well as temporal relationships . We apply our approach to two video-related tasks that offer substantially different challenges : video super-resolution ( VSR ) and unpaired video translation ( UVT ) . With no ground truth motion available , the spatio-temporal adversarial loss and the recurrent structure enable our model to generate realistic results while keeping the generated structures coherent over time . With the two learning tasks we demonstrate how spatio-temporal adversarial training can be employed in paired as well as unpaired data domains . In addition to the adversarial network which supervises the short-term temporal coherence , long-term consistency is self-supervised using a novel bi-directional loss formulation , which we refer to as “ Ping-Pong ” ( PP ) loss in the following . The PP loss effectively avoids the temporal accumulation of artifacts , which can potentially benefit a variety of recurrent architectures . The central contributions of our work are : a spatio-temporal discriminator unit together with a careful analysis of training objectives for realistic and coherent video generation tasks , a novel PP loss supervising long-term consistency , in addition to a set of metrics for quantifying temporal coherence based on motion estimation and perceptual distance . Together , our contributions lead to models that outperform previous work in terms of temporally-coherent detail , which we quantify with a wide range of metrics and user studies . 2 RELATED WORK . Deep learning has made great progress for image generation tasks . While regular losses such as L2 ( Kim et al. , 2016 ; Lai et al. , 2017 ) offer good performance for image super-resolution ( SR ) tasks in terms of PSNR metrics , GAN researchers found adversarial training ( Goodfellow et al. , 2014 ) to significantly improve the perceptual quality in multi-modal problems including image SR ( Ledig et al. , 2016 ) , image translations ( Zhu et al. , 2017 ; Isola et al. , 2017 ) , and others . Perceptual metrics ( Zhang et al. , 2018 ; Prashnani et al. , 2018 ) are proposed to reliably evaluate image similarity by considering semantic features instead of pixel-wise errors . Video generation tasks , on the other hand , require realistic results to change naturally over time . Recent works in VSR improve the spatial detail and temporal coherence by either using multiple low-resolution ( LR ) frames as inputs ( Jo et al. , 2018 ; Tao et al. , 2017 ; Liu et al. , 2017 ) , or recurrently using previously estimated outputs ( Sajjadi et al. , 2018 ) . The latter has the advantage to re-use high-frequency details over time . In general , adversarial learning is less explored for VSR and applying it in conjunction with a recurrent structure gives rise to a special form of temporal mode collapse , as we will explain below . For video translation tasks , GANs are more commonly used but discriminators typically only supervise the spatial content . E.g. , Zhu et al . ( 2017 ) does not employ temporal constrains and generators can fail to learn the temporal cycle-consistency . In order to learn temporal dynamics , RecycleGAN ( Bansal et al. , 2018 ) proposes to use a prediction network in addition to a generator , while a concurrent work ( Chen et al. , 2019 ) chose to learn motion translation in addition to spatial content translation . Being orthogonal to these works , we propose a spatiotemporal adversarial training for both VSR and UVT and we show that temporal self-supervision is crucial for improving spatio-temporal correlations without sacrificing spatial detail . While L2 temporal losses based on warping are used to enforce temporal smoothness in video style transfer tasks ( Ruder et al. , 2016 ; Chen et al. , 2017 ) , concurrent GAN-based VSR work ( Pérez-Pellitero et al. , 2018 ) and UVT work ( Park et al. , 2019 ) , it leads to an undesirable smooth over spatial detail and temporal changes in outputs . Likewise , the L2 temporal metric represents a sub-optimal way to quantify temporal coherence and perceptual metrics that evaluate natural temporal changes are unavailable up to now . We work on this open issue , propose two improved temporal metric and demonstrate the advantages of temporal self-supervision over direct temporal losses . Previous work , e.g . tempoGAN ( Xie et al. , 2018 ) and vid2vid ( Wang et al. , 2018b ) , have proposed adversarial temporal losses to achieve time consistency . While tempoGAN employs a second temporal discriminator with multiple aligned frames to assess the realism of temporal changes , it is not suitable for videos , as it relies on ground truth motions and employs a single-frame processing that is sub-optimal for natural images . On the other hand , vid2vid focuses on paired video translations and proposes a video discriminator based on a conditional motion input that is estimated from the paired ground-truth sequences . We focus on more difficult unpaired translation tasks instead , and demonstrate the gains in quality of our approach in the evaluation section . For tracking and optical flow estimation , L2-based time-cycle losses ( Wang et al. , 2019 ) were proposed to constrain motions and tracked correspondences using symmetric video inputs . By optimizing indirectly via motion compensation or tracking , this loss improves the accuracy of the results . For video generation , we propose a PP loss that also makes use of symmetric sequences . However , we directly constrain the PP loss via the generated video content , which successfully improves the long-term temporal consistency in the video results . 3 LEARNING TEMPORALLY COHERENT CONDITIONAL VIDEO GENERATION . Domain B VSR 𝐷 , 0/1 𝑎 FrameRecurrent Generator 𝑔 𝑤 Conditional LR Triplet 𝐼 𝑎 𝑎 𝑎 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤+ + UVT 𝐷 , 0/1 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤or or Static Triplet 𝐼 𝑔 { } x3 or Static Triplet 𝐼 𝑏 { } x3 Domain A 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → Domain A Domain B 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → a ) Domain B VSR 𝐷 , 0/1 𝑎 FrameRecurrent Generator 𝑔 𝑤 Conditional LR Triplet 𝐼 𝑎 𝑎 𝑎 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤+ + UVT 𝐷 , 0/1 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤or or Static Triplet 𝐼 𝑔 { } x3 or Static Triplet 𝐼 𝑏 { } x3 Domain A 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → Domain A Domain B 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → b ) Figure 2 : a ) G. b ) The UVT cycle link using recurrent G. Generative Network Before explaining the temporal self-supervision in more detail , we outline the generative model to be supervised . Our generator networks produce image sequences in a frame-recurrent manner with the help of a recurrent generator G and a flow estimator F . We follow previous work ( Sajjadi et al. , 2018 ) , where G produces output gt in the target domain B from conditional input frame at from the input domain A , and recursively uses the previous generated output gt−1 . F is trained to estimate the motion vt between at−1 and at , which is then used as a motion compensation that aligns gt−1 to the current frame . This procedure , also shown in Fig . 2a ) , can be summarized as : gt = G ( at , W ( gt−1 , vt ) ) , where vt = F ( at−1 , at ) and W is the warping operation . While one generator is enough to map data from A to B for paired tasks such as VSR , unpaired generation requires a second generator to establish cycle consistency . ( Zhu et al. , 2017 ) . In the UVT task , we use two recurrent generators , mapping from domain A to B and back . As shown in Fig . 2b ) , given ga→bt = Gab ( at , W ( g a→b t−1 , vt ) ) , we can use at as the labeled data of ga→b→at = Gba ( ga→bt , W ( ga→b→at−1 , vt ) ) to enforce consistency . A ResNet architecture is used for the VSR generator G and a encoder-decoder structure is applied to UVT generators and F . We intentionally keep generators simple and in line with previous work , in order to demonstrate the advantages of the temporal self-supervision that we will explain in the following paragraphs . Domain B VSR 𝐷 , 0/1 𝑎 FrameRecurrent Generator 𝑔 𝑤 Conditional LR Triplet 𝐼 𝑎 𝑎 𝑎 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤+ + UVT 𝐷 , 0/1 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤or or Static Triplet 𝐼 𝑔 { } x3 or Static Triplet 𝐼 𝑏 { } x3 Domain A 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → Domain A Domain B 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → Figure 3 : Conditional VSRDs , t . Spatio-Temporal Adversarial Self-Supervision The central building block of our approach is a novel spatio-temporal discriminator Ds , t that receives triplets of frames . This contrasts with typically used spatial discriminators which supervise only a single image . By concatenating multiple adjacent frames along the channel dimension , the frame triplets form an important building block for learning because they can provide networks with gradient information regarding the realism of spatial structures as well as short-term temporal information , such as first- and second-order time derivatives . We propose a Ds , t architecture , illustrated in Fig . 3 and Fig . 4 , that primarily receives two types of triplets : three adjacent frames and the corresponding warped ones . We warp later frames back- ward and previous ones forward . While original frames contain the full spatio-temporal infor- 3 mation , warped frames more easily yield temporal information with their aligned content . For the input variants we use the following notation : Ig = { gt−1 , gt , gt+1 } , Ib = { bt−1 , bt , bt+1 } ; Iwg = { W ( gt−1 , vt ) , gt , W ( gt+1 , v′t ) } , Iwb = { W ( bt−1 , vt ) , bt , W ( bt+1 , v′t ) } . For VSR tasks , Ds , t should guide the generator to learn the correlation between LR inputs and highresolution ( HR ) targets . Therefore , three LR frames Ia = { at−1 , at , at+1 } from the input domain are used as a conditional input . The input of Ds , t can be summarized as Ibs , t = { Ib , Iwb , Ia } labelled as real and the generated inputs Igs , t = { Ig , Iwg , Ia } labelled as fake . In this way , the conditional Ds , t will penalize G if Ig contains less spatial details or unrealistic artifacts according to Ia , Ib . At the same time , temporal relationships between the generated images Iwg and those of the ground truth Iwb should match . With our setup , the discriminator profits from the warped frames to classify realistic and unnatural temporal changes , and for situations where the motion estimation is less accurate , the discriminator can fall back to the original , i.e . not warped , images . Domain B VSR 𝐷 , 0/1 𝑎 FrameRecurrent Generator 𝑔 𝑤 Conditional LR Triplet 𝐼 𝑎 𝑎 𝑎 Original Triplet 𝐼 𝑔 𝑔 𝑔 Warped Triplet 𝐼 𝑔 𝑔 𝑤 𝑔 𝑤 Original Triplet 𝐼 𝑏 𝑏 𝑏 Warped Triplet 𝐼 𝑏 𝑏 𝑤 𝑏 𝑤+ + Domain A 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → Domain A Domain B 𝑎 𝑔 → 𝑔 → 𝐺 → 𝐺 → 𝑏 𝑔 → 𝑔 → 𝐺 → 𝐺 → For UVT tasks , we demonstrate that the temporal cycleconsistency between different domains can be established using the supervision of unconditional spatio-temporal discriminators . This is in contrast to previous work which focuses on the generative networks to form spatio-temporal cycle links . Our approach actually yields improved results , as we will show below , and Fig . 1 shows a preview of the quality that can be achieved using spatio-temporal discriminators . In practice , we found it crucial to ensure that generators first learn reasonable spatial features , and only then improve their temporal correlation . Therefore , different to the Ds , t of VST that always receives 3 concatenated triplets as an input , the unconditional Ds , t of UVT only takes one triplet at a time . Focusing on the generated data , the input for a single batch can either be a static triplet of Isg = { gt , gt , gt } , the warped triplet Iwg , or the original triplet Ig . The same holds for the reference data of the target domain , as shown in Fig . 4 . With sufficient but complex information contained in these triplets , transition techniques are applied so that the network can consider the spatio-temporal information step by step , i.e. , we initially start with 100 % static triplets Isg as the input . Then , over the course of training , 25 % of them transition to Iwg triplets with simpler temporal information , with another 25 % transition to Ig afterwards , leading to a ( 50 % ,25 % ,25 % ) distribution of triplets . Details of the transition calculations are given in Appendix D. Here , the warping is again performed via F . While non-adversarial training typically employs loss formulations with static goals , the GAN training yields dynamic goals due to discriminative networks discovering the learning objectives over the course of the training run . Therefore , their inputs have strong influence on the training process and the final results . Modifying the inputs in a controlled manner can lead to different results and substantial improvements if done correctly , as will be shown in Sec . 4 . Although the proposed concatenation of several frames seems like a simple change that has been used in a variety of projects , it is an important operation that allows discriminators to understand spatio-temporal data distributions . As will be shown below , it can effectively reduce temporal problems encountered by spatial GANs . While L2−based temporal losses are widely used in the field of video generation , the spatiotemporal adversarial loss is crucial for preventing the inference of blurred structures in multi-modal data-sets . Compared to GANs using multiple discriminators , the single Ds , t network can learn to balance the spatial and temporal aspects from the reference data and avoid inconsistent sharpness as well as overly smooth results . Additionally , by extracting shared spatio-temporal features , it allows for smaller network sizes . Self-Supervision for Long-term Temporal Consistency When relying on a previous output as input , i.e. , for frame-recurrent architectures , generated structures easily accumulate frame by frame . In an adversarial training , generators learn to heavily rely on previously generated frames and can easily converge towards strongly reinforcing spatial features over longer periods of time . For videos , this especially occurs along directions of motion , and these solutions can be seen as a special form of temporal mode collapse . We have noticed this issue in a variety of recurrent architectures , examples are shown in Fig . 5 a ) and the Dst in Fig . 1 . While this issue could be alleviated by training with longer sequences , we generally want generators to be able to work with sequences of arbitrary length for inference . To address this inherent problem of recurrent generators , we propose a new bi-directional “ Ping-Pong ” loss . For natural videos , a sequence with forward order as well as its reversed counterpart offer valid information . Thus , from any input of length n , we can construct a symmetric PP sequence in form of a1 , ... an−1 , an , an−1 , ... a1 as shown in Fig . 5 . When inferring this in a frame-recurrent manner , the generated result should not strengthen any invalid features from frame to frame . Rather , the result should stay close to valid information and be symmetric , i.e. , the forward result gt = G ( at , gt−1 ) and the one generated from the reversed part , g′t = G ( at , g′t+1 ) , should be identical . Based on this observation , we train our networks with extended PP sequences and constrain the generated outputs from both “ legs ” to be the same using the loss : Lpp = ∑n−1 i=1 ‖gt − gt′‖2 . Note that in contrast to the generator loss , the L2 norm is a correct choice here : We are not faced with multi-modal data where an L2 norm would lead to undesirable averaging , but rather aim to constrain the recurrent generator to its own , unique version over time . The PP terms provide constraints for short term consistency via ‖gn−1 − gn−1′‖2 , while terms such as ‖g1 − g1′‖2 prevent long-term drifts of the results . As shown in Fig . 5 ( b ) , this PP loss successfully removes drifting artifacts while appropriate high-frequency details are preserved . In addition , it effectively extends the training data set , and as such represents a useful form of data augmentation . A comparison is shown in Appendix E to disentangle the effects of the augmentation of PP sequences and the temporal constrains . The results show that the temporal constraint is the key to reliably suppressing the temporal accumulation of artifacts , achieving consistency , and allowing models to infer much longer sequences than seen during training . Perceptual Loss Terms As perceptual metrics , both pre-trained NNs ( Johnson et al. , 2016 ; Wang et al. , 2018a ) and in-training discriminators ( Xie et al. , 2018 ) were successfully used in previous work . Here , we use feature maps from a pre-trained VGG-19 network ( Simonyan & Zisserman , 2014 ) , as well as Ds , t itself . In the VSR task , we can encourage the generator to produce features similar to the ground truth ones by increasing the cosine similarity between their feature maps . In UVT tasks without paired ground truth data , we still want the generators to match the distribution of features in the target domain . Similar to a style loss in traditional style transfer ( Johnson et al. , 2016 ) , we here compute the Ds , t feature correlations measured by the Gram matrix instead . The feature maps of Ds , t contain both spatial and temporal information , and hence are especially well suited for the perceptual loss . Loss and Training Summary We now explain how to integrate the spatio-temporal discriminator into the paired and unpaired tasks . We use a standard discriminator loss for the Ds , t of VSR and a least-square discriminator loss for the Ds , t of UVT . Correspondingly , a non-saturated Ladv is used for the G and F of VSR , and a least-squares one is used for the UVT generators . As summarized in Table 1 , G and F are trained with the mean squared loss Lcontent , adversarial losses Ladv , perceptual losses Lφ , the PP loss LPP , and a warping loss Lwarp , where again g , b and Φ stand for generated samples , ground truth images and feature maps of VGG-19 or Ds , t . We only show losses for the mapping from A to B for UVT tasks , as the backward mapping simply mirrors the terms . We refer to our full model for both tasks as TecoGAN below.1 Training parameters and details are given in Appendix G. 1Source code , training data , and trained models will be published upon acceptance .

Summary: The paper presents a novel method for training video-to-video translation (vid2vid) models. The authors introduce a spatio-temporal adversarial discriminator for GAN training, that shows significant benefits over prior methods, in particular, parallel (as opposed to joint) spatial and temporal discriminators. In addition the authors introduce a self-supervised objective based on cycle dependency that is crucial for producing temporally consistent videos. A new set of metrics is introduced to validate the claims of the authors.