Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation
Authors: Mengyu Chu
You Xie
Jonas Mayer
Laura Leal-Taixé
Nils Thürey
Venue:  (2020)
Keywords: adversarial training
generative models
unpaired video translation
video super-resolution
temporal coherence
self-supervision
cycle-consistency

Abstract: We focus on temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationship in the generated data is much less explored. This is crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving the learning objectives and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. We also propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirms the rankings computed with these metrics.

Under review as a conference paper at ICLR 2020
LEARNING TEMPORAL COHERENCE VIA SELF- SUPERVISION FOR GAN-BASED VIDEO GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
We focus on temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for variety of areas, temporal relationship the generated data much less explored. This is crucial for sequential generation tasks e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as 1,2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temport cycle consistencies. In contrast, we focus on improving the learning objectives, and propose temporally supervupe algorithm. For both tasks, we show that temporal adversarial learning is key to achieving tem- porally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effec- tively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. We also propose first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirms the rankings computed with these metrics.
INTRODUCTION
Generative adversarial models (GANs) have been extremely successful at learning complex distribu- tions such as natural images (Zhu al. 2017 Isola et al.) (017).. However, for sequence generation, directly applying GANs without carefully engincered constraints typically results in strong artifacts over time due the significant difficulties introduced by the temporal changes. particular, con- ditional video generation tasks are very challenging learning problems where generators should not only learn represent the data distribution of the target domain, but also learn to correlate the out- put distribution over time with conditional inputs. Their central objective is to faithfully reproduce the temporal dynamics the target domain and not resort to trivial solutions such as features that arbitrarily appear and disappear over time.
In our work, we propose a novel adversarial learning method for a recurrent training approach that supervises both spatial content well as temporal relationships. We apply our approach to two video-related tasks that offer substantially different challenges: video super-resolution (VSR) and unpaired video translation (UVT). With no ground truth motion available, the spatio-temporal ad- versarial loss and the recurrent structure enable our model to generate realistic results while keep- ing the generated structures coherent over time. With the two learning tasks demonstrate how spatio-temppral adversarial training can be employed in paired as well as unpaired data domains. In addition to the adversarial network which supervises the short-term temporal coherence. long-term consistency is self- self-supervieed using a novel bi-directional loss formulation, which we refer to as Pring (PP) loss in the following. The PP loss effectively avoids the temporal accumulation of artifacts, which can potentially benefit a variety of recurrent architectures. The central contributions of our work are: a spatio-temporal discriminator unit together with a careful analysis of training objectives for realistic and coherent video generation tasks, novel PP loss supervising long-term consistency, in addition to a set of metrics for quantifying temporal coherence based on motion esti- mation and perceptual distance. Together, our contributions lead to models that outperform previous work in terms of semporally-cohere detail, which we quantify with a wide range of metrics and user studies.

---

Under review as a conference paper at ICLR 2020
Imput
CydGGAN
DOOk
RecydeLAN
TeeLAN
impuu
CyORGAN
RocydeGAN
FreGGNN
Cycle-
Teco-
Input
GAN
DsOnly
Dst
GAN
When learning a mapping between Trump and Obama, the CycleGAN model gives good spatial features, but collapses to essentially static outputs of Obama. It manages transfer facial expressions back Trump using tiny differences encoded able to establish correct temporal cycle-consistency between domains, ours RecycleGAN can generate correct blinking motions. Our model outperforms the latter in terms of coherent detail that is generated.
Figure I:
Obama outputs, instead learning meaningful mapping. Being
2 RELATED WORK
Deep learning has made great progress for image generation tasks. While regular losses such as I
(Kim al.) 2016 Lai al.) 2017) offer good performance for image super-resolution (SR) tasks
in terms of PSNR metrics, GAN researchers found ad adversarial training (Goodfellow et al.) 2014)
significantly improve the perceptual quality in multi-modal problems including image SR (Ledig
et al. (016)) image translations (Zhu et al. 2017 Isola et al. (017), and others. Perceptual met-
rics (Zhang et al., 2018) Prashnani Prashaannn al. 2018) are proposed reliably evaluate image similarity by considering semantic features instead of pixel-wise errors.
Video generation tasks, on the other hand, require realistic results to change naturally over time. Recent works in VSR improve the spatial detail and temporal coherence by either using multiple low-resolution (LR) frames as inputs (Jo et et. al. 2018 Tao et al. 2017 Liu et al. 2017). or recurrently using previously estimated outputs (Sajjadi et al. 2018). The latter has the advantage to re-use high-frequency details over time. In general, adversarial learning is less explored for VSR and applying it in conjunction with recurrent structure gives rise special form of temporal mode collapse, as we will explain below. For video translation tasks, GANs are more commonly used but discriminators typically only supervise the spatial content. E.g., Zhu al. 2017) does not employ temporal constrains and generators can fail to learn the temporal cycle-comsistency. In order to learn temporal dynamics, RecycleGAN (Bansal al. 2018) proposes use prediction network in addition to generator, while a concurrent work (Chen et al. 2019) chose to learn motion translation in addition to spatial content translation. Being orthogonal to these works, we propose a spatio- temporal adversarial training for both VSR and UVT and we show that temporal self-supervision crucial for improving spatio-temporal correlations without sacrificing spatial detail. While L temporal losses based warping used to enforce temporal smoothness in video style transfer tasks (Ruder et al.) 2016 Chen et al.) (017). concurrent BEAN-based VSR work (Pérez-Pelliten et al. 2018) and UVT work (Park et al. 2019). it leads and temporal changes in outputs. Likewise, the L temporal metric represents a sub-optimal way quantify temporal coherence and perceptual metrics that evaluate natural temporal changes
undesirable smooth over spatial detail
2

---

Under review as a conference paper at ICLR 2020
unavailable up to now. We work on this open issue, propose two improved temporal metric and demonstrate the advantages of temporal self-supervvsion over direct temporal losses.
Previous work, e.g. tempoGAN (Xie et al., 2018 and vid2vid (Wang et al. 2018b), have proposed adversarial temporal losses to achieve time consistency. While tempoGAN employs a second tem- poral discriminator with multiple aligned frames to assess the realism of temporal changes, it is not suitable for videos, as relies on ground truth motions and employs a single-frama processing that sub-optimal for natural images. On the other hand, vid2vid focuses on paired video translations and proposes a video discriminator based on a conditional motion input that is estimated from the paired ground-trune truth sequences. We focus on more difficult unpaired translation tasks instead, and demonstrate the gains in quality of our approach in the evaluation section. For tracking and optical flow estimation, 1.2-based time-cycle losses (Wang et al.) 2019) were proposed constrain motions and tracked correspondences using symmetric video inputs. By optimizing indirectly via motion compensation or tracking, this loss improves the aceuracy of the results. For video generation, we propose PP loss that also makes use of symmetric sequences. However, we directly constrain the PP loss via the generated video content, which successfully improves the long-term temporal consistency in the video results.
3 LEARNING TEMPORALLY COHERENT CONDITIONAL VIDEO GENERATION
Generative Network Before explaining the temporal self-supervision in more detail, we outline the gener- ative model to be supervised. Our generator networks produce image sequences in a crame-recurrent manner with the help a recurrent generator G and flow es- timator F. We follow previous work (Sajjadi et al.] 2018). where G produces output Gt in the target do- main B from conditional input frame 0, from the in- put domain A, and recursively uses the previous gen- erated output 9t-1. F is trained to estimate the mo- tion 2e between 4;-1 and at which is then used as motion compensation that aligns Jt-1 1 to the current frame. This procedure, also shown in Fig.Zh), can be summarized as: 9t. G(a;, vz),, where v2 = F(a,-1, (t) and W is the warping operation. While one generator is enough to map data from A to B for paired tasks such VSR, unpaired generation requires second generator to establish cycle consis- tency. (Zhu et al. 2017). In the UVT task, we use two recurrent generators, mapping from domain A to B and back. As shown in Fig. 2). given op Gar W vx)), we can a the labeled data of t6 Gg. W Wh v.)) to enforce consistency. A ResNet architecture is used for the VSR generator G and a encoder-decoder structure is applied to UVT generators and F. We intentionally keep generators simple and in line with previous work, in order demonstrate the advantages of the temporal self-supervision that we will explain in the following paragraphs.
-t-1 W
Gb-a
at
Ga-b
It-1
Frame-
Recurrent Generator
Domain
Domain B
b-ra
GD -
g
b-va
Dr
Ga-b
G
St
Figure 2: a) b) The UVT cycle link using recurrent
Spatio-Temporal Adversarial Self-Supervision The central building block of our approach is a novel spatio-temporal dis- criminator D., that receives triplets of frames. This contrasts with typically used spatial discriminators which supervise only a single image. By concatenating multiple adjacent frames along the channel dimension, the frame triplets form an important build- ing block for learning because they can provide networks with gradient information regarding the realism of spatial structures as well as short-term temporal information, such as first- and second-order time derivatives.
ar-1
a2 ac+1
Conditional LR Tripleti,
9t-1
9t
0c+3
b
b
bees
Original Triple Jy
Original Tr
81
b,
Warped Triplet lws
Warped Triplet
D&r
We propose a Da., architecture, illustrated in Fig. 3 and Fig. that primarily receives two types of triplets: three adjacent frames and the corresponding warped ones. We warp later frames back- ward and previous ones forward. While original frames contain the full spatio-temporal infor-
0/1
Figure 3: Conditional VSR De,
3

---

Under review as a conference paper at ICLR 2020
mation, warped frames more easily yield temporal information with their aligned content. For the input variants we use the following notation: I, = {g--1, 9t. 1..1). 9t.11. I {br-1, be, bet1):
Iug (W vt). st. (1 v()), Lnb. ( (bb_1,,)) v.),
b.. W W(b:11, v()).
For VSR tasks, Da,, should guide the generator to learn the correlation between LR inputs and high- resolution (HR) targets. Therefore, three LR frames La- {at-1, 0e; ox+1] } from the input domain are used as a conditional input. The input of Da,t can be summarized 12 - {Ia, Lub; Ia] labelled as real and the generated inputs 12, {I,, Iug. I,} labelled as fake. In this way. the conditional De. will penalize G if I, contains less spatial details or unrealistic artifacts according to Ia, I.. At the same time, temporal relationships between the generated images Iuy and those of the ground truth lit should match. With our setup, the discriminator profits from the warped frames to classify realistic and unnatural temporal changes, and for situations where the motion estimation is less accurate, the discriminator can fall back to the original, i.e. not warped, images.
For UVT tasks, we demonstrate that the temporal cycle- consistency between different domains can be established using the supervision of unconditional spatio-temporal discriminators. This is in contrast to previous work which focuses the genera- tive networks to form spatio-temporal cycle links. Our approach actually yields improved results, as we will show below, and Fig. D shows a preview of the quality that can be achieved using spatio-temporal discriminators. In practice, we found it crucial to ensure that generators first learn reasonable spatial features, and only then improve their temporal correlation. Therefore, different to the Da.t of VST that always receives 3 concatenated triplcts as an input, the unconditional Da.t of UVT only takes one triplet at time. Focusing on the generated data, the input for a single batch can either be a static triplet of log = {ge, Gt, gt, the warped triplet Iug. or the original triplet Ig. The same holds for the reference data of the target domain, as shown in Fig. g With suffi- cient but complex information contained in these triplets, transition techniques are applied so that the network can consider the nempioltempora information step by step, i.c., we initially start with 100% static triplets lsg as the input. Then, over the course of training. 25% of them transition to Iug triplets with simpler temporal information, with another 25% transition to lg afterwards, lead- ing to a 550%,55,,22.
{ 8c )X3
b Static Triplet Lo
Xx3
Static Triplet by
Øc-1
gt
1r+1
Dt-1
by
Original Triplct
Original Triplet
9--1
9t
9c43
b
Warpad Triplet bug
Warped Triplet L0
UVT Dzz
0/1 Figure 4: Unconditional UVT Da,..
distribution of triplets. Details of the transition calculations are given in Appendix D Here, the warping is again performed via F.
While non-adversarial training typically employs loss formulations with static goals, the GAN train- ing yields dynamic goals due discriminative networks discovering the learning objectives over the course of the training run. Therefore, their inputs have strong influence on the training process and the final results. Modifying the inputs in a controlled manner can lead to different results and substantial improvements if done correctly, as will be shown in Sec. Although the proposed con- catenation of several frames seems like a simple change that has been used in variety of projects, it is an important operation that allows discriminators to understand spatio-temporal data distribu- tions. As will be shown below, it can effectively reduce temporal problems encountered by spatial GANs. While based temporal losses are widely used in the field of video generation, the spatio- temporal adversarial loss is crucial for preventing the inference of blurred structures in multi-modal data-sets. Compared to GANs using multiple discriminators, the single Da,, network can learn to balance the spatial and temporal aspects from the reference data and avoid inconsistent sharpness as well as overly smooth results. Additionally, by extracting shared spatio-temporal features, it allows for smaller network sizes.
Self-Supervision for Long-term Temporal Consistency When relying on previous output as input, i.e., for frame-recurrent architectures, generated structures easily accumulate frame by frame. In an adversarial training, generators learn to heavily rely on previously generated frames and can easily converge towards strongly reinforcing spatial features over longer periods of time. For videos, this especially occurs along directions of motion, and these solutions can be seen as a special form of temporal mode collapse. We have noticed this issue in variety of recurrent architectures, exam- ples are shown in Fig. (5 a) and the Dst in Fig. [ While this issue could be alleviated by training with longer sequences, we generally want generators to be able to work with sequences of arbitrary length for inference. To address this inherent problem of recurrent generators, we propose a new
4

---

Under review as a conference paper at ICLR 2020
Figure 5: a) Result without PP loss. The VSR network is trained with a recurrent frame-length of 10. When inference on long sequences, frame 15 and latter frames of the foliage scene show the drifting artifacts. b) Result trained with PP loss. These artifacts are removed successfully for the latter. c) The ground-truth image. With our PP loss (shown the right), the L distance between ge and ge is minimized to remove drifting artifacts and improve temporal coherence.
bi-directional "Ping-Pong" loss. For natural videos, sequence with forward order well as its reversed counterpart offer valid information. Thus, from any input of length n,, we can construct a symmetric PP sequence in form of d1,-..u=-1; uyy On-11...01 as shown in Fig. [ When inferring this in a frame-recurred from frame frame. Rather, the result should stay close to valid information and be symmetric, i.e., the forward result gi G(as, 1) and the one generated from the reversed part, øí G(ar. 1e+1), should be identical. Based on this observation, we train our networks with extended PP sequences and constrain the generated outputs from both "legs" to be the same using the loss: Cyp i= |19. 9. 9''ll2- Note that in contrast the generator loss, the L norm is correct choice here: We are not faced with multi-modal data where an I,2 norm would lead to undesirable averaging, but rather aim to constrain the recurrent generator to its own, unique version over time. The PP terms provide constraints for short term consistency via 19
manner, the generated result should not strengthen any invalid features
9n-1'lla while terms such as [1g - 9 '122 prevent long-term drifts of the results. As shown in Fig. b), this PP loss successfully removes drifting artifacts while appropriate high-frequency details are preserved. In addition, it ef- fectively extends the training data set, and as such represents a useful form of data augmentation. A comparison is shown in Appendix B to disentangle the effects of the augmentation of PP sequences and the temporal constrains. The results show that the temporal constraint is the key to reliably suppressing the temporal accumulation of artifacts, achieving consistency, and allowing models to infer much longer sequences than seen during training.
Perceptual Loss Terms As perceptual metries, both pre-trained NNs (Johnson et 2016 Wang et al. 201 Sa) and in-training discriminators (Xie et al. 2018) were successfully used in previous work. Here, we use feature maps from pre-trained VGG-19 network (Simonyan & Zisserman 2014), as well as D..t itself. In the VSR task, we can encourage the generator to produce features similar to the ground truth ones by increasing the cosine similarity between their feature maps. In UVT tasks without paired ground truth data, still want the generators to match the distribution of features in the target domain. Similar to style loss in traditional style transfer (Johnson et al.) 2016), we here compute the D.. feature correlations measured by the Gram matrix instead. The feature maps of D..: contain both spatial and temporal information, and hence are especially well suited for the perceptual loss.
Loss and Training Summary We now explain how to integrate the into the paired and unpaired tasks. We standard discriminator loss for the Da,: of VSR and least-square discriminator loss for the Da, of UVT. Correspondingly. a non-saturated Lade is used for the G and F of VSR, and least-squares one used for the UVT generators. As summarized in Table G and F are trained with the mean squared loss Ceontent. adversarial losses Lade , perceptual losses L.. the PP loss Lyp and a warping loss Lwarps where again g. b and & stand for generated samples, ground truth images and feature maps of VGG-19 or D..t- We only show losses for the mapping from A to B for UVT tasks, as the backward mapping simply mirrors the terms. We refer to our full model for both tasks as TecoGAN below[! Training parameters and details are given in Appendix G
discriminator
Source code, training data, and trained models will be published upon acceptance.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Summary loss terms.
Loss for
VSR, Datt
UVT, D.
LD
-unpp (6) (log Dit ))
Bampptts Dog(1 D02,01
Pamppta) (o 112 Pampto)(D(2
Loss for
VSR, & F
UVT, Gab
LG.F
AeCommat + AuCror + dels
ApLen Au Cuap
Lomment
bbll2
96 1536
01
+ 96
by
Lade
Pal [log (3.)
-Bawni(a) 12,
-000000 + 1000
| .)1 (
Lyv
EE1 -=
1 gc'll2
Cwap
3 |ai W (at Flat ae))
0.0. pre-trained F is used
ENet
FRVSR
DUP
GT
DsOnly
DssD
FRVSR
Foliage full frame
Temporal profiles
DoDnnty
DsDt
DsDePP
DsD:PP
DUP
TecoGANo
TecOGAN
Figure 6: In VSR foliage scene, adversarial models (ENet, DsOnly, DsDt, DsDiPP, TecoGAN and TecoGAN) yield hetter perceptual quality than methods using I.² loss (FRVSR and DUF). In temporal profiles on the right. DsDt, DsDiPP and TecoGAN show significantly less temporal discontinuities compared to ENet and DsOnly. The temporal information of discriminators successfully suppresses these artifacts.
ANALYSIS AND EVALUATION OF LEARNING OBJECTIVES
In the following, we illustrate the effects of temporal supervision using two ablation studies. In the first one, models trained with ablated loss functions show how Lady and Lpp change the overall learning objectives. Next, full UVT models are trained with differrnnt D., inputs. This highlights how differently the corresponding discriminators converge to different apatio-tempora equilibriums, and the general importance of providing suitable data distributions from the target domain. While we provide qualitative and quantitative evaluations in the following, we also refer the reader to our supplemental himl document? with video clips that more clearly highlight the temporal differences.
Loss Ablation Study Below we compare variants of our full TecoGAN model to EnhanceNet (ENet) (Saijadi et al. 2017). FRVSR (Sajjadi et al. 2018), and DUF (Jo et al. 2018) for VSR, and CycleGAN (Zhu et al. 2017) and RecycleGAN (Bansal et al. 2018) for UVT. Specifically, ENet and CycleGAN represent sfate-of-ther of-the-att single-imas adversarial models without temporal information, FRVSR and DUF are oftte-of-the-ar VSR methods without adversarial losses, and RecycleGAN is a spatial adversarial model with a prediction network learning the temporal evolution.
For VSR, first train DsOnly model that uses frame- frame-recurren recurrent G and F with VGG-19 loss and only the regular spatial discriminator. Compared to ENet, which exhibits strong incoherence due to the lack of temporal information, DsOnly improves temporal coherence thanks to the frame- recurrent connection, but there are noticeable high- feeuueryy changes between frames. The tem- poral profiles of DsOnly in Fig. and 88, correspondingly contain sharp and broken lines. When adding a temporal discriminator in addition to the spatial one (DsDr), this version generates more
Anonymized and time-stamped supplemental material availabble at:
https: ///::///wwww. dropbox. com/ sh/n071 0 wh1le9c/ AAAVng 9x88za wwt.ssssssaassx.....ppppp pwtpppxp 10add

---

Under review as a conference paper at ICLR 2020
coherent results, and its temporal profiles are sharp and coherent. However, DsDt often produces the drifting artifacts discussed in Sec. B as the generator learns to reinforce existing details from previ- ous frames to fool D, with sharpness, and satisfying D, with good temporal coherence in the form of persistent detail. While this strategy works for generating short sequences during training, the strengthening effect can lead to very undesirable artifacts for long-sequence inferences. By adding the self-supervision for long-term temporal consistency Lpp, we arrive at the DsDiPP model, which effectively suppresses these drifting artifacts with an improved temporal coherence. In Fig. 6 and Fig. 8 DsDtPP results in continuous yet detailed temporal profiles without streaks from temporal drifting. Although DsDtPP generates good results, is difficult in practice to balance the generator and the two discriminators. The results shown here were achieved only after numerous runs manu- ally tuning the weights of the different loss terms. By using the proposed Da, discriminator instead, we get first complete model for our method, denoted as TecoGANS. This network is trained with a discriminator that achieves an excellent quality with an effectively halved network size, illus- trated on the right of Fig. 7 The single discriminator correspondingly leads to a significant reduction in resource usage. Using two discriminators requires ca. 70% more GPU memory, and leads to reduced training performance by ca. 20%. The TecoGAN model yields similar perceptual and temporal quality to DsDtPP with significantly faster and more stable training.
Since the TecoGAN model requires less training resources, we also trained a larger generator with 50% more weights. In the following we will focus this larger single-diseriminato architecture with PP loss as our full TecoGAN model for VSR. Compared to the TecoGAN T model, it can gen- erate more details, and the training process is more stable, indicating that the larger generator and Da., are more evenly balanced. Result images and temporal profiles are shown in Fig. 6 and Fig. 8 Video results are shown in Sec. of the supplemental material.
We also carry out a similar ablation study for the UVT task. Again, we start from a single-image GAN-based model, a CycleGAN variant which already has two pairs of spatial generators and dis- criminators. Then, we train the DsOnly variant by adding flow estimation via F and extending the spatial generators to frame-recurren ones. By augmenting the two discriminators to use the triplet inputs proposed in Sec. 3] we arrive at the Dst model with spatio-temporal discriminators, which does not yet use the PP loss. Although UVT tasks are substantially different from VSR tasks, the comparisons in Fig. T and Sec. 4.6 of our supplemental material yield similar conclusions. In these tests, we use renderings of 3D fluid simulations of rising smoke as our unpaired training data. These simulations are generated with randomized numerical simulations using a resolution of 64 for do- main A and 256 for domain B, and both are visualized with images of size 2562. Therefore, video translation from domain A to B is a tough task, the latter contains significantly more turbulent and small-scale motions. With no temporal information available, the CycleGAN variant generates HR smoke that strongly flickers. The DsOnly model offers better temporal coherence by relying on its frame-recurrent input, but it learns a solution that largely ignores the current input and fails to keep reasonable ppatio-temporee
sycle-consistency links between the two domains. On the contrary, our D,,, enables the Dst model to learn the correlation between the spatial and temporal aspects, thus improving the cycle-consistency. However, without Cpp= the Dst model (like the DsDt model of VSR) reinforces detail over time in an undesirable way. This manifests itself as inappropriate smoke density in empty regions. Using our full TecoGAN model which includes Cyp. yields the best results, with detailed smoke structures and very good spatio-temporal cycle-consistency.
For comparison, a DsDiPP model involving a larger number of separate networks, i.e. four discrimi- nators, two frame- recurrent generators and the F is trained. By weighting the temporal adversarial losses from Dt with 0.3 and the spatial ones from Ds with 0.5, we arrived at a balanced training run. Although this model performs similarly to the TecoGAN model on the smoke dataset, the proposed teapoo-temppral Da,, architecture represents a more preferable choice in practice, as it learns a nat- ural balance of temporal and spatial components by itself, and requires fewer resources. Continuing along this direction, it will be interesting future work to evaluate variants, such as a shared Da.; for both domains, i.c. a multi-class classifier network.
Besides the smoke dataset, an ablation study for the Obama and Trump dataset from Fig. shows a very similar behavior, as can be seen in the supplemental material.
Spatio-temporal Adversarial Equilibriums Our evaluation so far highlights that temporal adver- sarial learning crucial for achieving spatial detail that is coherent over time for VSR, and for enabling the generators to learn the spatio-temporal correlation between domains in UVT. Next, we
7

---

Under review as a conference paper at ICLR 2020
6.06
ENAI
6FRVSR
GOUF
0.025
0.07
ecteCrey
TeooGAh TTwastiN 8
BG
.Das
3M
CooGAN
0.015
edeelt
#0:piP
004
0.01
713
770
604
0.02
cubio
IM
0.01
Teem
GAN
LPi8
0.033
LRPS
664
844
1257
0.1
02 0.3
04
0.5
06
0.155
0.165
0:185
Only
Ds
D60PPP
DsEk
Toco DAN
GAN
T000
Figure 7: Visual summary of VSR models. LPIPS (x-axis) measures spatial detail and temporal coherence is measured by tLP (y-axis) and iOF (bubble size with smaller as better). The middle graph zooms in the red- dashed-box box region on the left, containing models in our ablation study. The right graph shows network sizes.
will shed light on the complex spatio-temporal adversarial learning objectives by varying the infor- mation provided to the discriminator network. The following tests D.,t networks that are identical apart from changing inputs, and we focus on the smoke dataset.
In order to learn the spatial and temporal features of the target domain as well as their correla- tion, the simplest input for Datt consists of only the original, unwarped triplets, i.e. {Tg or I.)- Using these, we train a baseline model, which yields a sub-optimal quality: it lacks sharp spatial structures, and contains coherent but dull motions. Despite containing the full information, these input triplets prevent D., from providing the desired supervision. For paired video translation tasks, the vid2vid network achieves improved temporal coherence by using a video discriminator to supervise the output sequence conditioned with the ground-truth motion. With ground-truth data available, we train a vid2vid variant by using the estimated motions and original triplets, i.e {I, + F(g--1 9t) + F( 9t) or I, + F F(b + F(be+1, be)]. as the input for D However, the result do not significantly improve. The motions are only partially reliable, and hence don't ton't help for the difficult unpaired translation task. Therefore, the discriminator still fails to fully correlate spatial and temporal features. We then train a third model, concat, using the original triplets and the warped ones, i.c. {I, +wy or Io ]. In this case, the model learns to generate more spatial details with more vivid motion. Lee., the improved temporal information from the warped triplets gives the diseriminator important cues. However, the motion still does not fully resemble the target domain. We arrive at our final TecoGAN model for UVT by controlling the composition of the input data: as outlined above, we first provide only static triplets (Ieg or [ab), and then apply the transitions of warped triplets {Ing or I and original triplets {I, or Io} over the course of training. In this way, the network can first learn to extract spatial features, and build on them to establish temporal features. Finally, discriminators learn features about the correlation of spatial and temporal content by analyzing the original triplets, and provide gradients such that the generators learn to use the mo- tion information from the input and establish correlation between the motions in the two unpaired domains. Consequently, the discriminator, despite receiving only a single triplet at once, can guide the generator produce detailed structures that move coherently. Video comparisons are shown in Sec 5. of the supplemental material.
Results and Metric Evaluation While the visual results discussed above provide first indicator of the quality our approach achieves. quantitative evaluations are crucial for automated evaluations across larger numbers of samples. Below we focus on the VSR task as ground-truth data is avail- able in this case. We conduct user studies and present evaluations of the different models w.r.t established spatial metrics. We also motivate and propose two novel temporal metrics to quantify temporal coherence. A visual summary is shown in Fig. 7
For evaluating image SR, Blau & Michaeli (018) demonstrated that there is an inherent trade-off between the perceptual quality of the result and the distortion measured with vector norms or low- level structures such as PSNR and SSIM. On the other hand, metrics based on deep feature maps such LPIPS (Zhang al. 2018) can capture more semantic similarities. We measure the PSNR and LPIPS using the Vid4 scenes. With a PSNR decrease of less than 2dB over DUF which has twice the model size of ours, TecoGAN outperforms all methods by more than 40% LPIPS.
8

---

Under review as a conference paper at ICLR 2020
Table Averaged VSR metric evaluations Vid4 data set with the following metrics, PSNR: pixel-wish wise accuracy. LPIPS (AlexNet): perceptual distance to the ground truth. T-diff: pixel-wise differences of warped frames. IOF: pixel-wise distance of estimated motions. tLP: perceptual distance between consecutive frames.
User study: Bradley-Temy scores (Bradley & Terry 1952). Performance is averaged over 500 images up-scaled from 320x 134 1280x536. More details can be found in Appendix C
Methods
PSNRT
LPIPS
T-difri
10FL
[.P4
User Smidy
Model Size (M)
Processing Time (mellrame)
X100
x100
DsOaly
24.14
1.727
6.852
2.157
2.160
0.8()))1.7(FF)
D&Dt
24.75
1.770
5.071
2.198
0.614
0.8(G)+
DsDIPP
25.77
1.733
4.369
2.103
0.489
0.8(G)+
TecoGAN
25.89
1.743
4.076
2.082
0.718
0.8(C)++1.(FF)
37.07
TecoGAN
25.57
1.623
4.961
1.897
0.668
3.258
1.3(0))11.(0))
41.92
ENet
22.31
2,458
9.281
4.009
4.848
1.616
0.8
FRVSR
26.91
2.506
3.648
2.090
0.957
2,600
0.81SRNo0+1.70F)
36.95
DUF
27.38
2.607
3,298
1,588
1.329
2,933
6.2
942.21
Bi-cubic 23.66
5.036
3.152
5.578
2.144
0.0
Table 3: For the Obama& Trump Trump dataset, the averaged tLP and tOF ovaluations closely correspond to our user studies. The table below summarizes user preferences as Bradley-Tees scores.
UVT scenes
Trump-ye
Obama
Ohama + Trump
AVG
User Studies f. ref. to
metrics
[PP
1OF]
ILP
[FF]
1LP
1OF]
original input
arbitrary target
CycleGAN
0.0176
0.7727
0,0277
1.1841
0.0234
0.9784
0.0
0.0
RecycleGAN
0.0111
0.8705
0,0248
1.1237
0.0179
0.9971
0.994
0.202
TecoGAN
0.0120
0.6155
0.0191
0.7670
0.0156
0.6913
1.817
0.822
While traditional temporal metrics based on vector norm differences of warped frames, e.g. T-diff, can be casily deceived by very blurry results, e.g. bi-cubic interpolated ones, we propose to use a tandem of two new metries, IOF and tLP, measure the consistence over time. (OF measures the pixel-wise difference of motions estimated from sequences, and tLP measures perceptual changes over time using deep feature map:
OF = 1|0 OF(g)-1999l) and iLP = (P-[((((-1,bb)) -LP (1)
where OF represents an optical flow estimation with LucasKanade (1981 and LP the perceptual LPIPS metric. In tLP, the behavior of the reference is also considered, natural videos exhibit a certain degree of changes over time. conjunction, both pixel-wise differences and perceptual changes are crucial for quantifying realistic temporal coherence. While they could be combined into a single score, we list both measurements separately, as their relative importance could vary in different application settings. Our evaluation with these temporal metrics in Table 2 shows that all temporal adversarial models outperform spatial adversarial ones, and the full TecoGAN model performs very well: With large amount of spatial detail, it still achieves good temporal coherence, on par with non-adversarial methods such as DUF and FRVSR. For VSR, we have confirmed these automated evaluations with several user studies. Across all of them, find that the majority of the participants considered the TecoGAN results to be closest to the ground truth.
For the UVT tasks, where no ground-truie -ruth data is available, we can still evaluate tOF and tLP metrics by comparing the motion and the perceptual changes of the output data w.r.t. the ones from the input data i.e., iOF [F a.) - (gf .8" - "[] me>*)],
and IIP [[ a.) LP .t" ot- a-+b With sharp spatial features and coherent motion, TecoGAN outperforms previous work on the Obama& Trump dataset, as shown in Table 3 although it is worth to point out that the IOF less informative in this case, as the motion in the target domain is not necessarily pixel-wise aligned with the input, Overall, TecoGAN achieves good tLP scores thanks to its temporal coherence, on par with RecycleGAN, and its spatial detail is on par with CycleGAN. As for VSR, perceptual evaluation by humans the right column of Table confirms our metric evaluations for the UVT task (details in Appendix C

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS AND DISCUSSION
In paired as well as unpaired data domains, we have demonstrated that it is possible to learn stable temporal functions with GANs thanks the proposed discriminator architecture and PP loss. We have shown that this yields coherent and sharp details for VSR problems that go beyond what can be achieved with direct supervision. In UVT, we have shown that our architecture guides the training process to successfully establish the spatio-temporal cycle consistency between two domains. These results are reflected the proposed metrics and user studies.
While our method generates very realistic results for wide range of natural images, our method can generate temporally coherent yet sub-optimal details in certain cases such as under-resolved faces and text in VSR, or UVT tasks with strongly different motion between two domains. For the latter case, it would be interesting to apply both our method and motion translation from concurrent work (Chen al., 2019). This can make it easier for the generator to learn from our temporal self supervision. procedure does not provide a guarantee that all goals fully reached every time. However, found our method to be stable over a large number of training runs, and we anticipate that will provide a very useful basis for wide range of generative models for temporal data sets.
our method, the interplay of the different loss terms in the non-linear training
REFERENCES
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan: Unsupervised video retargeting. In The European Conference on Computer Vision (ECCV), September 2018.
Yochai Blau and Tomer Michaeli. The perception-din -istiioonnistotiiinn tradeoff. In Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah. USA. pp. 6228-6237, 2018.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4)324-3345 1952.
(CC) Blender Foundation blendr..enn.r.org. Tears steel. lattps:/maaggo.bleeadd org/ 2011. Online; accessed Nov. 2018
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online vidco stylc transfer. In Proc. Intl. Conf. Computer Vision (ICCV), 2017.
Yang Chen, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. Mocycle-gan: Unpaired video-to-video translation. arXiv preprint arXiv: I 1908,09514, August 2019.
M-L Eckert, Wolfgang Heidrich, and Nils Thuerey. Coupled fluid density and motion from single views. In Computer Graphics Forum, volume 37(8), pp. 47-58. Wiley Online Library, 2018
Gustav Theodor Fechner and Wilhelm Max Wundt. Elemente der Psychophysik: erster Theil. Breitkopt & Härtel, 1889.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information process- ing systems, pp. 2672-26888 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville, Improved train- ing of wasserstein gans. In Advonces in neural information processing systems, pp. 5767-5777, 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Younghyun Jo. Scoung Wug Oh, Jaeyeon Kang, and Seon Kim, Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 1710. 2017.
10

---

Under review as a conference paper at ICLR 2020
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convo- lutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate superresolution. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2. pp. 5, 2017.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic single image super-resolution using a generative adversarial network. arXiv:1609.0480 008000900000... 2016.
Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, and Jiaya Jia. Vidco super-resolution via deep draft-ensemble learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 531-539, 2015.
Ce Liu and Deqing Sun. bayesian approach to adaptive video super resolution. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 209 216. IEEE, 2011.
Ding Liu, Zhaowen Wang, Yuchen Fan, Xianming Liu, Zhangyang Wang, Shiyu Chang, and Thomas Huang. Robust vidco super-resolution with learned temporal dynamics. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 2526-2534. IEEE, 2017.
Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision (darpa). In Proceedings of the 1981 DARPA Image Understanding Workshop, pp. 121- April 1981.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks, In Proceedings of the IEEE International Conference on Computer Vision, pp. 2794-2802, 2017.
Kwanyong Park. Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Preserving semantic and temporal consistency for unpaired video-to-video translation. arXiv preprint arXiv: 1XXiv:9908.0768) August 2019.
Eduardo Pérez-Pellitero, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Schölkopf. Photorealistic vidco super resolution. arXiv preprint
2018.
Ekta Prashnani, Hong Cai. Yasamin Mostofi, and Pradeep Sen. PieAPP: Perceptual Image-Enee Assessment through Pairwise Preference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1808-1817, 2018.
Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In German Conference on Pattern Recognition, pp. 26-36. Springer, 2016.
Mehdi SM Sujaadi, Bernhard Schölkopf, and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Computer Vision (ICCV), 2017 IEEE International Conference pp. 4501-4510. IEEE, 2017.
Mehdi SM Sajjadi. Raviteja Vemulapalli, and Matthew Brown. Frame-recurent video super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1 1556, 2014.
Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia. Detail-revealing deep video super-resolutio.. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Kiwon Um, Xiangyu Hu, and Nils Thuerey, Perceptual evaluation of liquid simulation methods. ACM Trans- actions on Graphics (TOG), 36(4):143, 2017.
Chaoyue Wang, Chang Xu, Chaohui Wang, and Dacheng Tao. Perceptual adversarial networks for image-to- image transformation. IEEE Transactions Image Processing, 2788):4066-4077 2018a.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. vidco-to-video synthesis. In Advances in Neural Information Processing Systems (NIPS), 2018b.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019.
11

---

Under review as a conference paper at ICLR 2020
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. lempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
Richard Zhang, Phillip Isola. Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. arXiv preprint. 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2017.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
In the following, we first provide qualitative analysis(Appendix A using multiple results that are mentioned but omitted in main document due space constraints. We then explain details of the metrics and present the quantitative analysis based on them(Appendix B The conducted user studies are in support of our TecoGAN network and proposed temporal metrics (Appendix Then, we give technical details of our spatio-temporal discriminator (Sec. DD. details of network architec- tures and training parameters (Appendix F Appendix Gp. In the end, discuss the performance of our approach in Appendix H
A QUALITATIVE ANALYSIS
For the VSR task, we test our model on a wide range video data, including the generally used Vid4 dataset shown in Fig. 8 and 12 detailed scenes from the movie Tears of Steel (ToS. 2011) shown in Fig. [2 and others shown in Fig. g
As mentioned in the main document, the TecoGAN model is trained with down-sampled inputs and it similarly work with original images that were not down-samplee sampled or filtered, such as a data-set of real-world photos (Liao et al. (201)). In Fig. [T]] compared our results the same dataset. With the help of adversarial learning, our model is able to generate improved and realistic details in down-sampled images as well as captured images.
other methods (Liao al. 2015 Tao et al., 2017) that have used
Temporal
ENet
Oursc
DsDt
ENet
FRVSR
DsDtPP
TecoGAN
TecoGAN
DUF
Teco
DUF
GAN
FRVSR
ENet
Figure 8: VSR temporal profile comparisons of the calendar (time shown along y-axis). Teco- GAN models lead to natural temporal progressions, and final model closely matches the desired ground truth behavior over time.
GT
Figure 9: Additional VSR comparisons. The TecoGAN model generates sharp details in both scenes.
13

---

Under review as a conference paper at ICLR 2020
ligence
ligrnce hard Szelisk hard Szelisk
Springer
Springer
Stuart
Stuart
Liao et al. 2015
Ours
Liao2015
Ours
Tao et al (2017)
Ours
Figure 10: Comparisons for VSR of captured maggs.
Inpul
TeeoGVA
Imput
TeooAN
Inpul
TeoGAN
TreoGN
Figure 11: Results of UVT tasks on different datasets.
FRVER
TecoGAN
Figure 12: Detail views of the VSR results of ToS scenes (first three columns) and Vid4 scenes (two right-most columns) with comparisons.
14

---

Under review as a conference paper at ICLR 2020
Table 4: Metrics evaluated for the VSR Vid4 scenes.
PSNRT
BIC
ENet
FRVSR
DUF
TecoGAN
TecoGAN
DsOnly
DsDt
DsDiPP
calendar
20.27
19.85
23.86
24.07
23.21
23.35
22.23
22.76
22.95
foliage
23.57
21.15
26.35
26.45
24.26
25.13
22.33
22.73
25.00
city
24.82
23.36
27.71
28.25
26.78
26.94
25.86
26.52
27.03
walk
25.84
24.90
29.56
30.58
28.11
28.14
26.49
27.37
28.14
average
23.66
22.31
26.91
27.38
25.57
25.89
24.14
24.75
25.77
LPIPS 1x10
BIC
ENet
FRVSR
DUF
TecoGAN
TecoGAN
DsOnly
DsDt
DsDiPP
calendar
5.935
2.191
2.989
3.086
1.511
2.142
1.532
2.111
2.112
foliage
5.338
2.663
3.242
3.492
1.902
1.984
2.113
2.092
1.902
city
5.451
3.431
2.429
2.447
2.084
1.940
2.120
1.889
1.989
walk
3.655
1.794
1.374
1.380
1.106
1.011
1.215
1.057
1.051
average
5.036
2.458
2.506
2.607
1.623
1.743
1.727
1.770
1.733
OFF Lx10
BIC
ENet
FRVSR
DUF
TecoGAN
TecoGAN
DsOnly
DsDt
DsDiPP
calendar
4.956
3,450
1.537
1.134
1.342
1.403
1.609
1.683
1,583
foliage
4.922
3.775
1.489
1,356
1.238
1.444
1.543
1.562
1.373
city
7.967
6,225
2.992
1.724
2.612
2.905
2.920
2.936
3.062
walk
5.150
3.203
2.569
2.127
2.571
2.765
2.745
2.796
2.649
average
5.578
4.009
2,090
1.588
1.897
2.082
2.157
2.198
2.103
1x100
BIC
ENet
FRVSR
TecoGAN
TecoGAN
DsOnly
DsDL
DsDiPP
calendar
3,258
2.957
1.067
1.603
0.165
1.087
0.872
0.764
0.670
foliage
2.434
6.372
1.644
2.034
0.894
0.740
3,422
0.493
0.454
city
2.193
7.953
0.752
1.399
0.974
0.347
2.660
0.490
0.140
walk
0.851
2.729
0.286
0.307
0.653
0.635
1.596
0.697
0.613
average
2.144
4.848
0.957
1.329
0.668
0.718
2.160
0.614
0.489
T-diff x 100
BIC
ENet
FRVSR
DUF
TecoGAN
TecoGAN'
DsOnly
DsDt
DsDtPP
GT
calendar
2.271
9.153
3.212
2.750
4.663
3.496
6.287
4.347
4.167
6.478
foliage
3.745
11.997
3.478
3.115
5.674
4.179
8.961
6.068
4.548
4.396
city
1.974
7.788
2.452
2.244
3.528
2.965
4.929
3.525
2.991
4.282
walk
4.101
7.576
5.028
4.687
5,460
5.234
6.454
5.714
5.305
5.525
average
3.152
9.281
3.648
3.298
4.961
4.076
6,852
5.071
4.369
5.184
For UVT tasks, we train models for Obama and Trump translations, LR- and HR- smoke simulation translations, as well as translations between smoke simulations and real-smoke captures. While smoke simulations usually contain strong numerical viscosity with details limited by the simulation resolution, the real smoke, captured using the setup from Eckert et al. (2018), contains vivid fluid motions with many vortices and high-frequency details. As shown in Fig. our method can be used to narrow the gap between simulations and weal-world phenomenon.
B METRICS AND QUANTITATIVE ANALYSIS
Spatial Metrics We evaluate all VSR methods with PSNR together with the human human-calibratte LPIPS metric (Zhang et al. 2018). While higher PSNR values indicate better pixel-wise accuracy, lower LPIPS values represent better perceptual quality and closer semantic similarity. Mean values of the Vid4 scenes Liu Sun (01T) are shown on the top of Table 4 Trained with direct vector norms losses, FRVSR and DUF achieve high PSNR scores. However, the undesirable smoothing in- duced by these losses manifests themselves in larger LPIPS distances. ENet, on the other hand, with no information from neighboring frames, yields the lowest PSNR and achieves LPIPS score that is only slightly better than DUF and FRVSR. TecoGAN model with adversarial training achieves an excellent LPIPS score, with a PSNR decrease of less than 2dB over DUF, which is very reason- able, since PSNR and perceptual quality were shown to be anti-correlated (Blau & Michaeli 2018), especially regions where PSNR is very high. Based on good perceptual quality and reasonable pixel-wise accuracy, TecoGAN outperforms all other methods by more than 40% for LPIPS.
calibrated
Temporal Metrics For both VSR and UVT, evaluating temporal coherence without ground motion is very challenging problem. The metric T-diff || W W( ex)ll, used by Chen et al. (2017) as a rough assessment of temporal differences. As shown on bottom of Table T-diff, due its local nature, is easily deceived by blurry method such the bi-cubic interrelation and
15

---

Under review as a conference paper at ICLR 2020
Table 5: Metrics evaluated for the VSR of ToS.
PSNRT
BIC
|NNet
FRVSK
DUF
TecoGAN
OOF 10
BIC
ENet
FRVSR
DUF
TecoGAN
room
26.90
25.22
29.80
30.85
29.31
room
1.735
1.625
0.861
0.901
0.737
bridge
28.34
26.40
32.56
33.02
30,81
bridge
5.485
4.037
1.614
1.348
1.492
face
33.75
32.17
39.94
40.23
38.60
face
4.302
2.255
1.782
1.577
1.667
average
29.58
27.82
34.04
34.60
32.75
average
4.110
2.845
1.460
1.296
1.340
LPIPS _x10
BIC
ENet
FRVSR
DUF
TecoGAN
ILP Lx 1x100
BIC
ENet
FRVSR
DUF
TecoGAN
room
5.167
2.427
1.917
1.987
1,358
room
1,320
2,491
0,366
0.307
0.590
bridge
4.897
2.807
1.761
1.684
1,263
bridge
2.237
6,241
0.821
0.526
0.912
face
2.241
1.784
0.586
0.517
0.590
face
1.270
1.613
0.290
0.314
0.379
average
4.169
2.395
1.449
1.414
1.086
average
1.696
3.827
0.537
0.403
0.664
can not correlate well with visual assessments of coherence. By measuring the pixel-wise motion difference using 1OF in together with the perceptual changes over time using tLP, we show the tem- poral evaluations for the VSR task in the middle of Table 4 Not surprisingly, the results of ENet show larger errors for all metrics due to their strongly flickering content. Bi-cubic up- sampling, DUF, and FRVSR achieve very low T-diff errors due their smooth results, representing but undesirable avenue for achieving coherency. However, the overly smooth changes of the for- mer two are identified by the tLP scores. While our DsOnly model generates sharper results the expense of temporal coherence, it still outperforms ENet there. By adding temporal information to discriminators, our DsDt, DsD++PP, T and TecoGAN improve in terms of temporal met- rics. Especially the full TecoGAN model stands out here. For the UVT tasks, temporal motions are evaluated by comparing to the imput sequence. With sharp spatial features and coherent motion, TecoGAN outperforms previous work on the Obama& Trump dataset, as shown in Table 3
easy,
Spatio-temporal Evaluations Since temporal metrics can trivially be reduced for blurry im- age content, we found it important to evaluate results with combination of spatial and tem- poral metrics. Given that perceptual metrics are already widely used for image evaluations, we believe it is the right time to consider perceptual changes in temporal evaluations, as we did with our proposed temporal coherence metrics. Al- though not perfect, they are not easily deceived. Specifically, tOF is more robust than a direct mixel-wise metric as it compares motions instead of image content. In the supplemental material, we visualize the motion difference and it can well reflect the visual inconsistencies. On the other hand, we found that our calculation of tLP is general concept that can work reliably with different perceptual metric: When repeating the ILP evaluation with the PicAPP metric (Prash- nani et al. 2018) instead of LP, i.c., rPieP
tPleP
ENet @ FRVSR DUF TecoGAN
0.25
ENet
0.2
0.15
0.1
0.05
FRVSR
TecoGAN
DUP
PleAPP (error)
0.7
0.8
0.9
1
1.1
rPieP]
BIC
ENet
FRVSR
DUF
TecoGAN
calendar
0.091
0.093
0.028
0.021
foliage
0.155
(..06
0..03
0.036
city
0.136
0.025
0.028300.0276
walk
0.072
0.042
0.060
[[f Vt) f(gt-1,9))1,, where f(-) indicates the perceptual error function of PieAPP, we get close identical results, listed in Fig. [3 The conclusions from (PieP also closely match the I.PIS- evaluation: our network architec- ture generate realistic and temporally coher- ent detail, and the metrics we propose allow for a stable, automated evaluation of the temporal perception of a generated video sequence.
Figure 13: Tables and visualization of perceptual
metrics computed with PieAPP (Prashnani et al., 2018) (instead LPIPS used in Fig previously) on ENet. FRVSR, DUF and TecoGAN for the VSR of Vid4. Bubble size indicates the tOF score.
15 we also get similar evaluation results the Tears of Steel data-sets (room, bridge, and face. the following referred to as ToS scenes) and corresponding results are shown in Table and Fig. [6] In all tests, we follow the procedures of previous work (Jo et al. 2018 Sajjadi et al. 2018) make the outputs of
Besides the previously evaluated the Vid4 dataset, with graphs shown in Fig. 14
16

---

Under review as a conference paper at ICLR 2020
BIC
ENet
FRVSR
DUF
TecoGAN
TecoGAN
DeOnly
DsDL
DsDIPP
GT
10F
5.00
ILP
0.26
T-Diff I
0.76
0.02
Figure 14: Bar graphs of temporal metrics for Vid4.
PSNR
10F
PSNR
00-
06
244
20
18
1550g
05
22
100m
brdgge
feee
oveeage
bridge
keorago
3.6
LPIPS
LPIPS
3.06
1LP
3.00
Milee
Welk
emaage
room
brldge
200m
brige
fack
xraaag
Figure 15: Spatial metrics for Vid4.
Figure 16: Metrics for ToS.
all methods comparable, i.e., for all result images, we first exclude spatial borders with a distance of 8 pixels to the image sides, then further shrink borders such that the LR input image is divisible by 8 and for spatial metrics, we ignore the first two and the last two frames, while for temporal metrics, we ignore first three and last two frames, as additional previous frame is required for inference. In the following, we conduct user studies for the Vid4 scenes. By comparing the user study results and the metric breakdowns shown in Table 4 we found our metries to reliably capture the human temporal perception, as shown in Appendix
C USER STUDIES
We conduct several user studies for the VSR task using five different methods, namely bi-cubic interpolation, ENet, FRVSR, DUF and our TecoGAN. The established 2AFC design (Fechner & Wundi 1889; Um et al. 2017) is applied, i.c., participants have a pair-wise choice, with the ground- truth video shown as reference. One example can be seen in Fig. 17 The videos are synchronized and looped until user made the final decision. With no control to stop videos, users Participants cannot stop or influence the playback, and hence can focus more on the whole video, instead of specific spatial details. Videos positions (left/A right/B) are randomized.
After collecting 1000 votes from users for every scene, i.e. twice for all possible pairs (5x 4/2 = 10 pairs), follow common procedure and compute scores for all models with the Bradley-Teryy model (1952). The outcomes for the Vid4 scenes can be seen in Fig. (vverall scores are listed in Table 2 of the main document).
From the Bradley-Terry scores for the Vid4 scenes we can see that the TecoGAN model performs very well, and achieves the first place in three cases, well second place in the walk scene. The latter is most likely caused by the overall slightly smoother images of the walk scene, in conjunction with the presence of several human faces, where our model can lead to the generation of unexpected
17

---

Under review as a conference paper at ICLR 2020
details. However, overall the user study shows that users preferred the TecoGAN output over the other two deep-learning methods with a 63.5% probability.
This result also matches with metric evaluations. In Table 4 while TecoGAN achieves spatial (LPIPS) improvements in all scenes, DUF and FRVSR are not far behind in the walk scene. In terms of temporal metrics tOF and tLP, TecoGAN achieves similar or lower scores compared to FRVSR and DUF for calendar, foliage and city scenes. The lower performance of our model for the walk scene is likewise captured by higher (OF and tLP scores. Overall, the metrics confirm the perfor- mance of our TecoGAN approach and match the results of the user studies, which indicate that our proposed temporal metrics successfully capture important temporal aspects of human perception.
For UVT tasks which have no ground-truth data, we carried out two sets of user studies: One uses an arbitrary sample from the target domain as the reference and the other uses the actual input from the source domain as the reference. On the Obama& Trump data-sets, we evaluate results from CycleGAN, RecycleGAN, and TecoGAN following the same modality, i.e. a 2AFC design with 50 users for each run. E.g., on the left of Fig. [9 users evaluate the generated Obama in reference with the input Trump on the y-axis, while an arbitrary Obama video is shown as the reference on the x-axis. Effectively, the y-axis is more important than the x-axis as it indicates whether the translated result preserves the original expression. A consistent ranking of TecoGAN RecycleGAN > CycleGAN is shown on the y-axis with clear separations, i.c. standard errors don't overlap. The x-axis indicates whether the inferred result matches the general spatio-temporal content of the target domain. Our TecoGAN model also receives the highest scores here, although the responses are slightly more spread out. On the right of Fig. 19 we summarize both studies in single graph highlighting that the TecoGAN model consistently preferred by the participants of our user studies.
D TECHNICAL DETAILS OF THE SPATIO-TEMPE DISCRIMINATOR
Motion Compensation Used in Warped Triplet In the TecoGAN architecture, Da,, detects the temporal relationships between IN and IN with the help of the flow estimation network F. However, at the boundary of images, the output of F is usually less accurate due to the lack of reliable neighborhood information. There is a higher chance that objects move into the field of view, or leave suddenly, which significantly affects the images warped with the inferred motion. An example is shown in Fig. 20] This increases the difficulty for Da,t, as it cannot fully rely on the images being aligned via warping. To alleviate this problem, we only use the center region of /N9, and IN s, as the input of the discriminator, and we reset a boundary of 16 pixels. Thus, for an input resolution of IN s.t. and INY of 128 128 for the VSR task, the inner part in size of 96 x 96 is left untouched, while the border regions are overwritten with zeros.
1--1
W 12 (4)
W we) vella
Near image boundaries, flow estimation is less accu- rate and warping often fails to align well. First two columns show original and warped frames and the third one shows differences after warping (ideally all black). The top row shows things move into the view with problems near lower boundaries, while the sec- ond row has objects moving out the view.
Figure 20:
The flow estimation network F with the loss La.r should only be trained to support G in reaching the output quality as determined by Da,t, but not the other way around. The latter could lead to networks that confuse Da.: with strong distortions of IN9, g.t and IN2 e. In order to avoid the this undesirable case, we stop the gradient back propagation from INg, and INY, to F. In this way, gradients from D.,, to F are only back propagated through the generated samples 9:-1, 9t and 1++1 into the generator network. In this way Da, . can guide G to improve the image content, and F learss
18

---

Under review as a conference paper at ICLR 2020
The Bradley-Teme scores (standard error)
Methods
calendar
lolluge
city
walk
Refervnce
Bi-cubic
0.000 (0:000) 0.000 (0.000)
0.000 (0.000)
0.000 (0.000)
ENst
1834 (0.228) 1.634 [0.180)
1,282 (0.205
1.773 (0.197)
FRVSR
3,043 (0.246) 2.177 (0.186)
3.173 (0.240)
2.424 (0.204)
DUF
3.468 (0.252)
2.243 (0.186)
3.302 (0.242)
3.175 (0.214)
ToooGN
4.091 (0.262)
2.769
(0.194)
4,052 (0.255)
2.693 (0.207)
Blc- m Enet
FRVSR
DUF
TocoGAN
6.0011
00000
01/20 Which one doser to the reference video?
CALENDAR
FOLIAGE
CITY
WALK
Figure 17: A sample setup of user study.
Figure 18: Tables and bar graphs of Bradley- Terry scores and standard errors for Vid4 VSR.
X-axis: Generated
Bradley-Terry scores
Sid.
X-axis: Generated
Bradley- Terry
Std.
X-axis: Generated
Bradley-Tore scores
Std.
Obama vs. Obama
Error
Obama vs. Obama
scores
Ervor
Obama Obama
Brror
CycleGAN
CycleGAN
0.806
0.177
CycleGAN
0
0
RecycleGAN
0.197
RecycleGAN
RecycleGAN
0.202
0.118
TecoGAN
1,520
0.201
TocoGAN
1.727
0,208
TecnGAN
0.822
0.123
Yaxis: Generated
Bradley-Teme scores
Std.
X-uxis: Generated
Bradley- Terry scores
Std.
X-axis: Generated
Bradley Terry scares
Sid.
Ohama vs. Trump
Ohama Ohama
Emoo
Ohama Obama
Bror
CycleGAN
CycleGAN
CycleGAN
0
RecycleGAN
1.410
0.208
RocycleGAN
0.623
0.182
RecycleGAN
0.994
0.135
TooGAN
19988
0.222
TeeGAN
1.092
0.182
TevoGAN
1817
0.150
Trump > Obama
Obama + Trump
AVG
29 avael with
Ivadd witl
Iqua
TeoG
the Iquut
vv 2015
Kacydhssit
Inl with
1f ref with arbirrrr Trump
1n re with arbiitaryy Sarget
Figure 19: Tables and graphs of Bradley-Terry scores and standard errors for Shama&Trumpp UVT.
to warp the previous frame in accordance with the detail that G can synthesize. However, F does not adjust the motion estimation only reduce the adversarial loss.
Curriculum Learning for UVT Discriminators As mentioned in the main part, we train the UVT Da,, with 100% spatial triplets at the very beginning. During training, 25% of them gradually transfer into warped triplets and another 25% transfer into original triplets. The transfer of the warped triplets can be represented as: (I- triplets, we additionally fade the "warping" operation out by using (1 - Leg + ( 2/ * 8). 9t, W (g(+1, 1/1 })}, again with growing form 0 to and decreasing from to O. We found this smooth transition to be helpful for a stable training.
with growing form t 1. For the original
E DATA AUGMENTATION AND TEMPORAL CONSTRAINS IN THE PP LOSS
Since training with sequences of arbitrary length is not possible with current hardware, problems such as the streaking artifacts discussed above generally arise for recurrent models. In the proposed PP loss, both the fing-Pang data augmentation and the temporal consistency constraint contribute to solving these problems. In order to show their separated contributions, we trained another TecoGAN variant that only employs the data augmentation without the constraint (i.e., Ap 0 in Table
19

---

Under review as a conference paper at ICLR 2020
Denoted PP-Amgeent, we show its results in comparison with the DsDt and TecoGAN models models in Fig. 21 Video results are shown the the supplemental material.
During training, the generator of DsDt receives 10 frames, and generators of
DsDt
PP-Augment
TecoGAN
PSNR: 24.75, LPIPS: 1.77.
PSNR: 24 98, LPIPS: 1.81.
PSNR: 25, I.PIP:: 1.74, t.P. 0.718, 10F 2.082
PP-Augment and TecoGAN see 19 frames. While DsDt shows strong recurrent accumulation artifacts early on, the PP-Augment version slightly reduces the artifacts. In Fig. 21 works good for frame 15, but shows artifacts from frame on. Only our regular model (TecoGAN9) success- fully avoids temporal accumulation for all 40 frames. Hence, with the PP constraint, the model avoids recurrent accumulation of artifacts and works well for sequences that are substan- tially longer than the training length.
ILP: 0.614, (OF 2.198
ILP: 0,850, IFF 1.903
the Foliage seene. While DsDt leads strong recurrent artifacts early PP- Augment shows similar artifacts later in time (2nd row, middle). TecoGAN model modell successfully removes these artifacts.
Figure 21:
1st & 2nd row: Frame 15
Among others, we have tested our model with ToS sequences of lengths 150, 166 and 233. For all of these sequences, the TecoGAN model successfully avoids temporal accumulation or streaking artifacts.
F NETWORK ARCHITECTURE
In this section, use the following notation to specify all network architectures used: conc() rep- resents the concatenation of two tensors along the channel dimension; C/CT(input, kernel size, output channel, stride size) stands for the convolution and transposed convolution operation, spectively; "+" denotes element-wise addition; BilinearUp2 up-samples input tensors by a factor of 2 using bi-linear interpolation: BicubicResizz<<inpuu increases the resolution of the input tensor to 4 times higher via bi-cubic up-sampling: Dense (input, output_size) is censely-connected layer, which uses Xavier initialization for the kernel weights.
The architecture of our VSR generator G is:
conc(æt, W ((t-1, + lin C(iin, 3, 64, 1), ReLU - lo: Residual Blockkl) 1441 with i 0, 0,.., n 1;
CT 3, 3,64,2) 2), ReLU + lup2; CT 3, 64, 2). ReLU + Lupti
O(lupt,,,3 3,3, 1). ReLU brea' BicubicResize4 (x,,) lyes + gt
In T there are 10 sequential residual blocks in the generator ( (In = Iio ) while the TecoGAN generator has 16 residual blocks hn 416 ). Each Residual Block(l) contains the following operations: C(4,3, 3, 64, 1), ReLU + rı Crr,, 3, 64, 1) + + 4+1.
The VSR D,,'' architecture is:
IN C(lo, 4, 64, 2), BatchNorm, Leaky ReLU - 11; C(I,, 4, 64, 2). BatchNorm, Leaky ReLU - lz; C(2, 4, 128, 2), BatchNorm, Leaky ReLU + lg; C(1 4, 25,26,,2 2), BatchNorm, Leaky ReLU + 44;
IN + Jan: C(inn 3, 64, 1), Leaky ReLU lo;
Dense(4,) 1), sigmoid lout
VSR discriminators used in our variant models, DsDt, DsDiPP and DsOnly, have similar architec- ture as D.,,- They only differ in terms of their inputs.
The flow estimation network F has the following architecture:
conc(z,, (1=1) lin C(Im, 3,32, Leakk ReLU + lo:
C(ly, 3, 32, 1), Leaky ReLU, MaxPooling + 11; C(1,,3, 64, 1), Leaky ReLU + 12:
C(2,, 3, 64, 1), Leaky ReLU, MaxPooling + l5; C(ly, 128, Leaky ReLU + l4
C(4,, 3, 128, 1), Leaky ReLU, MaxPooling ++ ls; C(ss,, 3,2 1), Leaky ReLU + lgi
C(le, 3,256, 1), Leaky ReLU, BilinearUp2
- 17; C(1,,,, 3, 128, 1), Leaky ReLU - ls:
20

---

Under review as a conference paper at ICLR 2020
Cllg, 128, 1), Leaky ReLU, Bilinearr - 19; C(1y, 3, 64, 1), Leaky ReLU + 410:
C(11.33, 6,,4,11 1), Leaky ReLU, BilinearUp2 + h11: C(I,,, 01.111,3.3.. 32, 1), Leaky ReLU - /125 C(112, 3, 2), tanh + louti luut Max Vel 2/
Here, Max Vel is a constant vector, which scales the network output to the normal velocity range. While is the same for UVT tasks, UVT generators have encoder-decoder structure:
can (gg vx)) + lax. C(Ian, 7,32, 1), InstanceNorm, ReLU + lo:
C(lo, 3, 64, 2), InstanceNorm, ReLU + f1; C(1, 3, 128, 2), InstanceNorm, ReLU l2;
Residual Block(|2 + + la+1 with
0,.... 0,..,
CT(1++2:3 64, 2). InstanceNorm, ReLU + 3- (T (3,+3 3, 32, 2), InstanceNorm, ReLU + h2445 CT( 3, 1), tanh + loot
Residval Blocklz + i) contains the following operations: C(2245, 3, 128, 1), InstanceNorm, ReLU - 1244 (1(44,, 3, 128, 1), InstanceNorm r244; r244 1244 + 1z44- We 10 residual blocks for all UVT generators.
Since UVT generators are larger than the VSR generator, we also use a larger Da,t architecture:
IN or IN + lin C(lin, 4,61, 21), ReLU lo: C(lo, 4, 128, 2), InstanceNorm, Leaky ReLU + 41 C(h, 4, 256, 2), InstanceNorm, Leaky ReLU l2: C(22, 1,512, 2), InstanceNorm, Leaky ReLU ls; Dense(ls, 1) lout
Again, all ablation studies use the same architecture with different inputs.
G TRAINING DETAILS
We use the non-saturated GAN for VSR and LSGAN (Mao et al. 2017) for UVT and both of them can prevent the gradient vanishing problem of vanilla GAN (Goodfellow al., 2014). While train stably with dynamic discriminator updating strategy, i.c. discriminators are not updated when there is already a large difference between D(I") and D(9),, the training process could potentially be further improved with modern GAN algorithms, e.g. Wasserstein GAN (Gulrajani et al. 2017). We train G and F together for VSR, while we simply use the pre- pre-trained F for UVT.
For the VSR task, our training data-set consists of 250 short HR videos, each with 120 frames. We use sequences with a length of 10 and a batch size of 4. A black image is used as the first previous frame of each video sequence. Lee. one batch contains 40 frames and with the PP loss formulation, the NN receives gradients from 76 frames in total for every training iteration. To improve the stability of the adversarial training, we pre-train G and F with a simple I loss of Elsa- bell2 + Cwrr for 500k batches. We use 900k batches for the adversarial training stage. The data-sets of the UVT tasks contain around 2400 to 3600 frames. We train the generators with a sequence length of 6 and a batch size of 1. Since temporal triplets are gradually faded in, do not pre-train models for UVT tasks. With smaller datasets, we train UVT models with 100k batches.
In the pre-training stage of VSR, we train the F and a generator with 10 residual blocks. An ADAM optimizer with 3 = 0.9 is used throughout. The learning rate starts from 10-4 and decays by 50% every 50k batches until it reaches 2.5 10-5 This pre- pre-trrined model is then used for all TecoGAN variants as initial state. In the adversarial training stage of VSR, all TecoGAN variants trained with a fixed learning rate of 5 10-3 The generators in DsOnly, DsDt, DsDtPP and TecoGAN 10 residual blocks, whereas the TecoGAN model has 6 additional residual blocks in its generator. Therefore, after loading 10 residual blocks from the pre-trained model, these additional residual blocks are faded in smoothly with factor of 2.5 * 10 We found this growing training methodology. first introduced by Growing GAN Karras et al., 2017). to be stable and efficient in our tests. When training the VSR D&Dt and DsDiPP, extra parameters are used to balance the two cooperating discriminators properly. Through experiments, we found D, to be stronger. Therefore, we reduce the learning rate of D, to 1.5 10-5 in order to keep both discriminators balanced. At the same time, a factor of 0.0003 is used on the temporal adversarial loss to the generator, while the spatial adversarial loss has factor of 0.001. During the VSR training, input LR video frames cropped to a size of 32 x 32. In all VSR models, the Leaky ReLU operation uses a tangent of 0.2 for the negative half space. Additional training parameters are listed in Table
21

---

Under review as a conference paper at ICLR 2020
Table 6: Training parameters
VSR Param DsOnlyDDD DsDt
DsDiPP
TecoGAN
TecoGAN
UVT Param DxOnly
Dst
TecoGAN
le-3
Ds:
le-3, DE: 3e-4
le-3
le-3
0.5
0.5
0.0
0.0
0.5
0.0
0.0
100.0
0.02 for VGG for Discriminator
Ag
from 10" decays to
1.0.
Aw
0.0, a pre-trained is used UST tasks
laaming
Se-5
Se-5
5e-5
Ac
10.0
Ae.5
For all UVT tasks, we use learning rate of 10-4 to train the first 90k batches and the last 10k batches are trained with the learning rate decay from 10 O. Images of the input domain are cropped into size of 256 x 256 when training, while the original size is 288 x 288. While the Additional training parameters are also listed in Table For UVT, Leuntent and L. are only used to improve the convergence of the training process, We fade out the Leontent in the first 10k batches and the Lg is used for the first 80k and faded out in last 20K.
H PERFORMANCE
TecoGAN is implemented in TensorFlow. While generator and discriminator are trained together, we only need the trained generator network for the inference of new outputs after training, i.e., the whole discriminator network can discarded. 1080Ti GPU with 11G memory, the resulting VSR performance for which is given in Table 2
evaluate the models on Nvidia GeForce GTX
The VSR TecoGAN model and FRVSR have the same number of weights (843587 in the SRNet, iec generator network, and 1.7M in F), and thus show very similar performance characteristics with around 37 ms spent for one frame. The larger VSR TecoGAN model with 1286723 weights in the generator is slightly slower than TecoGAN spending 42 per frame. In the UVT task, generators spend around 60 ms per frame with a size of 512 x 512. However, compared with the DUF model, with has more than million weights in total, the TecoGAN performance significantly better thanks to its reduced size.
22


Reviews:
[{"review_id": "rkxVYkeJ9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper presents a novel method for training video-to-video translation (vid2vid) models. The authors introduce a spatio-temporal adversarial discriminator for GAN training, that shows significant benefits over prior methods, in particular, parallel (as opposed to joint) spatial and temporal discriminators. In addition the authors introduce a self-supervised objective based on cycle dependency that is crucial for producing temporally consistent videos. A new set of metrics is introduced to validate the claims of the authors.\n\nI really like this paper. Although the method is a rather complex mix of multiple losses, these are justified in detail, both intuitively and empirically. The appendix is filled with much more detail about implementation, architecture and more results. Finally the results show that the proposed method is superior across the board compared to previous approaches from the literature.\n\n\nStrenghts:\n- The approach works on two distinct applications of vid2vid.\n- Detailed ablations justifying the introduction of every single part of the overall objective.\n- Strong results.\n- Clear writing and presentation.\n- A lot of additional details and results in the appendix for the more interested reader.\n\nWeaknesses:\n- Rather complex overall objective\n- Seems to need a lot of tweaking\n\nQuestions:\n- Isn't the PP loss just an incarnation of cycle consistency loss? If so, maybe there is no need for the introduction of a new name for it.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syx9r90j5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents video generation method with spacio-temporally consistent features. This is done through: a) temporal adversarial learning, b) Ping Pong loss, and c) metrics that quantify the quality. The methods are evaluated on two datasets and user studies.\n\nThe idea is interesting and the paper is well written. The results are convincing.\n\nThe originality of the concatenation of several frames is somewhat limited, since it is a standard procedure in other domains such as robotics. Nevertheless the results are positive.\n\nSeems like the metrics definitions were not included in the main body of the paper - the authors should either include them to remove from the contributions. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkxvTKqTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contribution\nAugments the loss of video generation systems with a discriminator that considers multiple frames (as opposed to single frames independently) and a new objective termed ping-pong loss which is introduced in order to deal with “artifacts” that appear in video generation. The paper also proposes a few automatic metrics with which to compare systems. Although the performance does not convincingly exceed its competitors, the contribution seems to be getting the spatio-temporal adversarial loss to work at all.\n \nOverall\nI found the video generations impressive, and the addition of the discriminator seems to improve sharpness of the individual frames over methods trained with non-adversarial losses, in particular DUF. Although TecoGAN does not beat its competitors (for example DUF) on most proposed automatic metrics, it appears to be ranked better in expectation in user studies (Table 2). However, I am concerned about generalizability of the method.\n \nDecision\nAs someone not in this field, I find it hard to judge the performance of the system from so few samples. I also found the differences in the losses between tasks to be worrisome, as it indicates this method does not generalize without heavy tweaking of the loss. My current decision is weak reject, since the generations look quite good but I have concerns about generalization of the approach to other video generation tasks as well as the justification of the ping-pong loss.\n\nQuestions\n- Is RecycleGAN unable to be applied to video super-resolution? Why is it not compared to in Table 2?\n- Is L_Phi the perceptual loss? I don't think it was mentioned in the body of the text.\n- I found the ping-pong (PP) loss to be unjustified. The loss is motivated by the issue of \"artifacts\", which I assume are poor generations due to lacking a good model of the world. The paper says this issue could be alleviated by training with longer video sequences, but that would prevent the generator from working with sequences of arbitrary length. I do not believe the ping-pong loss allows the generator to work with arbitrary sequences, as training only on short sequences and their reverse should not allow the model to generalize to longer sequences. Additionally, this approach would likely not scale to long sequences as well, since it requires doubling the sequence length. Would you be able to train on longer sequences?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bkezu8ZT5B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis paper proposes a training objective for higher quality video generation for the tasks of Video Super Resolution (VSR) and Unpaired Video Translation (UVT) and also two evaluation metrics tOF and tLP. They provide a comprehensive ablative study of the proposed method and also show comparisons against baselines for the tasks of VSR and UVT.\n\n\nPros:\n+ Novel video generation method for VSR and UVT.\n+ Novel metrics\n+ Well written paper\n\nWeaknesses / comments:\n\n- Confused about inputs to discriminator (I_{w,g} and I_{w,b})\nI am not sure I understand the inputs I_{w,g} and I_{w,b} to the discriminator network. Based on the description, I_{w,g} = {W(g_{t-1}, v_t), g_t, W(g_{t+1}, v_t^’)} = {g_t, g_t, g_t}, assuming v_t^” means reverse flow. A similar process seems to be happening with I_{w,b}. If this is the case, will the discriminator optimally get the same frame concatenated together (i.e. {g_t, g_t, g_t})? Now, if that’s the case, how is the discriminator modeling temporal consistency other than making the generator learn to put pixels forward and back in place since the “original triplets” (real data) are {g_{t-1}, g_t, g_{t+1}} and {b_{t-1}, g_t, g_{t+1}}, respectively. This is very confusing to me. It would be good if the authors can clarify this in the rebuttal.\n\n\n- How do you prevent the zero output in PP loss?\nThe PP-loss compares forward and backward prediction by || g_t - g_t^’ ||. If not careful, the neural network can just learn to output zeros or the same frame for the full video. Did the authors see any behavior like this? Or did the proposed formulations prevent this? Or was there a very small weight applied to this loss?\n\n\n\n- TecoGAN vs baselines generator parameters (rather than TecoGAN^{-}).\nBased on the experimental section, it looks like TecoGAN is the main network being compared to the baseline methods. TecoGAN has more parameters in the generator compared to TecoGAN^{-}. Did the authors make sure that the generator had the same number of parameters as the other methods? The authors mention a performance difference between TecoGAN and DUF due to DUF having more parameters, but what about the other baselines?\n\n- Evaluation metric contributions in the supplementary material?\nThe description of the two proposed metrics tOF and tLP have been placed in Appendix B. These are mentioned as contributions of the paper so their description should be in the main text. Please make sure that Appendix B is in the main text in future versions of this paper.\n\n\n- UVT task evaluation in the supplementary material?\nThe UVT task is mentioned in the abstract and as a target task in this work, however, the evaluations for this are in the Appendix. Please move them to the main text in future versions of this paper. Secondary experiments should be in the Appendix but I feel this is primary.\n\n\n- UVT task only evaluated with the proposed evaluations?\nThe UVT evaluation in the Appendix only has the proposed metrics as evaluation. I understand that, since it’s an unpaired video translation task, there is no ground truth to compare against. However, a human based study could be done where human raters would judge for the more realistic of N videos.\n\n\n- The same qualitative comparisons in the paper are not in the provided website.\nThe provided website does not have the same qualitative evaluations as the paper does. For example: There are 7 methods being compared for the same video in Figure 8 in the supplementary material, but I don’t see anything like that in the website.\n\n\n- For the UVT task, we omit the DsDtPP model because …….\nIn the paper, the authors mention that they don’t provide the DsDtPP baselines because it requires a lot of computation. However, I feel these missing baselines make the experiments incomplete since it’s a different task.\n\n\n- Figure 5 has no point of reference with which to compare the frames.\nFigure 5 points out that the video b) is better than video a). While it is true that one seems more noisy than the other, there should be a point of reference for readers to make sure of it.\n\n\nConclusion:\nIn conclusion, the paper seems to present a novel method and evaluation metrics but has many issues as stated above. It would make the submission better if the authors can address them in the rebuttal.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an architecture for conditional video generation tasks with temporal self-supervision and temporal adversarial learning. The proposed architecture is reasonable but looks somewhat complicated. In terms of technical novelty, the so-called "ping-pong" loss looks interesting and novel, but other parts are more-or-less some combinations of existing techniques. Experimental results show promise of the proposed method against selected baselines for video super-resolution (VSR) and unpaired video-to-video translation tasks (UVT). In terms of weakness, (1) the technical novelty is not very high; (2) the final loss is a combination of many losses with many hyperparameters; (3) experimentally the proposed method is not compared against recent SOTA methods on VSR and UVT. 

The proposed method should be compared against more recent SOTA baselines for VSR tasks (see examples of references below):

EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
https://arxiv.org/abs/1905.02716

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations
ICCV 2019

Recurrent Back-Projection Network for Video Super-Resolution
CVPR 2019

The same comment would apply for baselines for UVT tasks:

Mocycle-GAN: Unpaired Video-to-Video Translation
https://arxiv.org/abs/1908.09514

Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation
https://arxiv.org/abs/1908.07683

Particularly for UVT, the evaluated dataset seems limited in terms of scope as well (i.e., evaluations on more popular benchmarks, such as Viper would be needed for further validation). Overall, given that the contribution of this work is an empirical performance with a rather complex architecture/loss, more comprehensive empirical evaluations on SOTA baselines are warranted.

Decision: Reject