S-Flow GAN
Authors: Miron Yakov
Coscas Yona
Venue:  (2020)
Keywords: GAN
Image Generation
AI
Generative Models
CV

Abstract: Our work offers a new method for domain translation from semantic label maps
and Computer Graphic (CG) simulation edge map images to photo-realistic im-
ages. We train a Generative Adversarial Network (GAN) in a conditional way to
generate a photo-realistic version of a given CG scene. Existing architectures of
GANs still lack the photo-realism capabilities needed to train DNNs for computer
vision tasks, we address this issue by embedding edge maps, and training it in an
adversarial mode. We also offer an extension to our model that uses our GAN
architecture to create visually appealing and temporally coherent videos.

Under review as a conference paper at ICLR 2020
S-FLOW GAN
Anonymous authors Paper under double-blind review
ABSTRACT
Our work offers new method for domain translation from semantic label maps and Computer Graphic (CG) simulation edge map images to photo-realistic im- ages. We train Generative Adversarial Network (GAN) in a conditional way generate photo-realistic version of given CG scene. Existing architectures of GANs still lack the photo-realism capabilities needed to train DNNs for computer vision tasks, address this issue by embedding edge maps, and training it in an adversarial mode 1. We also offer an extension to our model that uses our GAN architecture create visually appealing and temporally coherent videos.
INTRODUCTION
The topic of image to image translation and more generally video to video translation of major importance for training autonomous systems. It is beneficial to train autonomous agent in real environments, but not practical, since enough data cannot be gathered Collins et al. (2018). However, using simulated scenes for training might lack details since a synthetic image will not be photo- realistic and will lack the variability and randomness of real images, causing training to succeed up certain point. This gap also referred as the reality gap Collins al. (2018). By combining non photo realiste, simulated model with an available dataset, we can generate diverse scenes containing numerous types of objects, lightning conditions, colorization etc. Chen Koltun (2017).
In this paper, depict flexible Deep Convolution Neural Network (DCNN) we called Deep Neural Edge Detector (DNED) which embed edge maps. combine embedded edge maps which act skeleton with semantic map as input to our model (fig 2.1), The model outputs a photo-realistic version of that scene. Using the skeleton by itself will generate images that lack variability specific skeleton itself. Instead, we learn to represent skeletons by neural network and at test time, we sample the closest appropriate skeleton the network has seen at training. Moreover, we have extended this idea to generate photo-realistic videos (i.e. sequence of images) with novel loss that uses the optical flow algorithm for pixel coherency between consecutive images.
approach generate images from semantic label map and
restricts the representation that
Figure I: in this paper we propose a method for generating photo-realistic images from semantic labels simulator scene. This figure provides images related the Synthia dataset Ros et al. (2016). Left semantic map of the scene. Middle generated image from pix2pixHD Wang et al. (2018b). Right- - Our generated image. The texture and color space our generated image is more natural giving the image the desired photo-realism.
Recent works in the field of image generation include pix2pix Isola et al. (2017) offering im- age generation from semantic maps, cascaded refinement networks Chen & Koltun (2017) using networks refining different resolutions in a cascade manner, pix2pixHD Wang et al. (2018b) can generate HD images in conditional manner using multi-scale discriminator and dual generator

---

Under review as a conference paper at ICLR 2020
used super resolution generator. L1 loss for image generation is known to generate low quality images as the generated images are blurred and lack details Dosovitskiy & Brox (2016). Instead, Gatys et al. (2016), Johnson et al. (2016) are using a modified version of the perceptual loss, al- lowing generation of finer details in an image. Pix2pixHI Wang et al. (2018b) and CRN Chen & Koltun (2017) are using perceptual loss as well for training their networks, e.g. VGGnet Simonyan & Zisserman (2014). Moreover, HiX2pixHD are using instance maps as well as label maps to enable the generator to separate several objects of the same semantics. This is of high importance when synthesizing images having many instances of the same semantics in single frame.
As for video generation the loss used by Wang et al. (2018a), Shahar et al. (2011) tend to be computationally expensive while our approach simpler. We are using two generators of the same architecture, and they are mutually trained using our new optical flow based loss that is fed by dense optical flow estimation. Our evaluation method is FID Heusel al. (2017) and FVD Unterthiner et al. (2018) as it is a common metric being used for image and video generation schemes. We call this work s-Flow GAN since we embed Spatial information obtained from dense optical flow in a neural network as prior for image generation and flow maps for video coherency. This optical flow available since the simulated image is accessible at test time in the case of CG2real scheme.
We make Three major contributions: First, our model can generate visually appealing photo- realistic images from semantic maps having high defimition details. Second, we incorporate a neural network to embed edge maps, thus allowing generation of diverse versions of the same scenes. Third, we offer new loss function for generating natural looking videos using the above mentioned image generation scheme. please refer to this link for videos and comparison to related work.
2
RELATED WORK
2.1
GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GAN) were introduced in 2014 Goodfellow et al. (2014). This method generate images that look authentic to human observers. They do by having two neural networks, one generating candidates while the other acts as a critique and tries to evaluate the gen- eration quality Arjovsky et al. (20o7).Radford et al. (2015)7Zhao et al. (2016) et al. (al1 imans et al. (2016). GANs are widely used for image generation; some image synthesis schemes used generate low resolution images e.g. 32x32 Isola et al. (2017) while Brock et al. (2018) were able to generate higher resolution images (up to 512x512). In addition, Wang et al. (2018b) were able generate even higher resolution images using coarse-to-fine generators. The reason generating high resolution images is challenging is the high dimensionality of the image generation task and the need to provide queues for high resolution Odena et al. (2017), Karras et al. (2017). We offer queues as edge map skeletons generated by our proposed DNED module. During training the DNED is trained to learn the representations of real image edge maps. During test the DNED shown CG (Computer Graphics) edge map, finds its best representation and provides the generator with an appropriate generated edge map sampled from real image edge maps distribution.
Figure 2: example of the images for training the model. Left is the semantic map. Middle is the edge map extracted from the real image. Right is the real image being used by the discriminator for adversarial training. Please note that the real image is not used by the generator neither training nor at test time, but only its edge map. The main issue in the CG2real model compared to the image to image models is that the simulators image available to the generator test time. We thus use the simulators image to extract the edge map allowing the generator to generate the necessary fine details in the output image.
2

---

Under review as a conference paper at ICLR 2020
2.2
IMAGE SYNTHESIS
2 IMAGE TO IMAGE TRANSLATION
In the pix2pix setting, they used Conditional GAN Mirza & Osindero (2014), where the networks input is a semantic map of the scene, and while training in adversarial mode, a fake version of the real image is given to the discriminator to distinguish. In the CG2real setting in addition to the semantic map we also have access to the simulated image. Using the CG image as is, might be counter productive since it will be trained to reconstruct CG images and not photo-realistic ones. Conversely some of the underlying CG information correlates with the real world and can provide meaningful prior to the synthesis. Since the relevant information lies in the image high frequencies Burt & Adelson (1983), we learn the distribution of edge maps in real images (high resolution details), and provide representation of to the generator test time. Some image generation tasks use label maps only, e.g. Isola et al. (2017). The label maps provide only information about the class of given pixel. In order to generate photo-realistic images, some use instance maps as well Wang et al. (2018b), This way, they can differentiate several adjacent objects of the same class. Nonetheless, while most datasets provide object level information about classes like cars, pedestrians, etc. they do not provide that information about vegetation and buildings. As a result, the generated images might not correctly separate those adjacent objects, thus degrading photo-realism.
2.2.2 LEARNING EDGES BY A NEURAL NETWORK
Generating edge maps using neural networks is a well established method. Holistically- Edge Detection (HED) provides holistic image training and prediction for multi-scale and multi- level feature learning Xie & Tu (2017). They use a composition of generated edge maps to learn a fine description of the edge scene. Inspired by their work, we train neural network to learn edge maps of real images.
As mentioned before, our generator requires edge map as input. get the edge map using spacial Laplacian operator with threshold. Providing the generator with deterministic edge map will produce the same scene, we train the DNED take input that deterministic edge map, learn its representation and produce a variant of that edge map, as a superposition of edges seen in real datasets. This way the generator will be able to produce a varaiaty of photorealistic images for the same scene.
Since approach (using edge maps) is not class dependent, do not need instance map informa- tion to generate several adjacent instances of the same semantics. Moreover, this approach addresses the problem of generating fine details within a class like buildings and vegetation as can bee seen in fig 3.2.
2.3 VIDEO TO VIDEO SYNTHESIS
Generating temporally coherent image sequences is a known challenge. Recent works use GANs to generate videos in an unconditional setting Saito et al. Tulyay.u et al. Vondrikk et al. (2016), by sampling from a random vector, but don't provide the generator with temporal constrains, thus generating non coherent sequences of images. Other works like video matting Bai et al. (2009) and video inpainting Wexler et al. (2004) translate videos to videos but rely on problem specific constrains and designs. A recent work named vid2vid Wang et al. (2018a) offers to conditionally generate video from video and is considered to one of the best approaches to date. Using FlowNet 2.0 1lg al. (2017) they predict the optical flow of the next image. In addition, they use a mask to differentiate between two parts; the hallucinated image generated from instance-level semantic segmentation masks and the predicted image from the previous frame. By adding these two parts, this method can combine the predicted details from the previously generated image, with the details from the newly generated image. Inspired by Wang et al. (2018a), we are using flow maps of consecutive images to generate temporally coherent videos. Contrary to Wang et al. (2018a) we are not using a CNN to predict the flow maps or a sequence generator, but a classical Computer vision approach. This is since pre-trained network (trained on real datasets) failed to generalize and infer on simulated datasets e.g. Synthia. This enables better temporal coherency and improve video generation robustness.
Vondrick
3

---

Under review as a conference paper at ICLR 2020
3 MODEL
Our CG2real model aims to learn the conditional distribution of an image given a semantic map. Our video generation model aims use this learned distribution for generating temporally coherent videos using the generated images from the CG2real scheme. We first depict the image generation scheme, then we review our video generation model.
3.1 IMAGE GENERATION
We use a conditional GAN to generate images from semantic maps as in Isola et al. (2017). In order to generate images, the generator receives the semantic segmentation images 8; and maps it to photo-realistic images E,. In parallel, the discriminator takes two images, The real image æ; (ground truth) and the generated image f, and learns to distinguish between them. This supervised learning scheme is trained in the well-known min max game Goodfellow et al. (2014), et al. (2016):
min max LGAN
LGAN(D.G) = Elo,s) s)] + E(s) Rog(1 Rog) D(s, G(s)))
(2)
3.2 EMBEDDING EDGE MAPS
In order to generate photo-realistic visually appealing images containing fine details, provide learnt representation of an edge map to the generator (fig 2.1), allowing to learn the conditional distribution of real images given semantic maps and edge maps, i.e.:
LGAN(D.G.)) E(a,s))log(1: s)] (((ss))-paauuss))))oollll D(s, G(sse)))
(3)
During training, given an example image 2,, we can estimate its edge map by the well-know spatial Laplacian operator Guattery & Miller (2000).
al. (2015). This edge map concatenated to the semantic label map and both are given as priors to the generator for adversarial training of the fake image S. vs. the real image r;. To allow stable training we begin training our GAN with the edge maps from the Laplacian operator. After stabilization of the generator and discriminator, we provide our generator with edge maps from the DNED. We then jointly train the GAN with the DNED.
The DNED architecture is a modified version of HED Xie & Tu (2017). In HED, they generate several sized versions of the edge map, each having different receptive field. The purpose is to create an ensemble edge maps, each allowing different level of details in the image. When superimposing all, the resulting edge map will have coarse-to-fine level of details in the generated edge map image. By changing the weights that ensemble, we can generate the desired variability in the generated edge map, thus allowing us generate diverse versions of the output. To conclude, the loss function for training the DNED is:
LDNED: = =EDNED(E(z)) = Σ a, (CE(
(4)
Where: (x(), 0 5 is the ith side output of a single scale, E(x) is the classic edge map generated by the spatial Laplacian operator, BCE the binary cross entropy loss. N in our case. 0; is the contribution of the 5th scale to the ensemble.
Increasing the resolution of the image might be challenging for GAN training. In other methods the discriminator needs large receptive field Isola et al. Seif & Androutsos (2018), 22018).imonyan & Zisserman (uo et al. (2016), requiring a deeper network or larger convolution kernels. Using deeper network is prone to overfitting and in the of GAN training, and might cause training to unstably. This challenge is usually addressed by the multi-scale approach Ghiasi & Fowikes Dentonn al. (2015)HHang et al. (2017). al. (2017)Zhhang al. (2017).
4

---

Under review as a conference paper at ICLR 2020
Since the DNED embed learnt representation of skeletons, architecture performs very well on higher resolution images. Our original generated images were size [512x256]. We have successfully trained model generate images of size [768x384] i.. 1.5 times larger in each dimension without changing the model while using a single discriminator (see 3.2).
Figure 3: comparison of 768x384 pix images generated by pix2pixHD (Left) and our model (Right). Our model can generate lower level details in the image, thus improving its photo-realism. This figure provides an example comparing (768x384 pix) resolution images of pix2pixHD (left side) compared to model (right).
We showed that generating high quality images when using single discriminator is feasible and training stable. We provide comparison using our method with multi-scale discriminator 3.2. the FM loss is computed with k=1 for single layer discriminator and k=3 for multi layer one:
Figure 4: Comparison of generated test images, when training with single discriminator and multi-scale one. The left image is generated when the generator was trained with single discrimi- nator, while the right image while using multi-scale one. This figure demonstrates that when using our model (with our skeleton), training with single discriminator might be enough.
T. Σ E (i N₁
C+ C C. (Di, G,e e)
(5)
In addition, following Dosovitskiy & Brox (006),,aty: et al. (2016),Johnson er al. (2016) al. (2017) we are using the perceptual loss for improved visual performance and encourage the discriminator distinguish real or fake samples using pre traind VGGnet Ledig et al. (2017).
P Σ 1=1
Epercep: Lepeccep (x,, G(s,e))
(FLlvcoa FLvvaa (Gtraa,G(ssnnn e)))
(6)
Where, P is the number of slices from pre-trained VGG network and Lvaa, are the features extracted by the VGG network from the 5th layer of the real and generated images respectively. To conclude, our overall objective for generating photo-realistic, diverse images in the CG2real setting is to minimize LOG2real:
Lm
LCG2real = min
max CD,
Σ
ck ck + Aeppercpp + AgLNNED (7)
G
Dr. k= 1i
5

---

Under review as a conference paper at ICLR 2020
Figure 5: By using edge maps, the model learns to separate objects of the same semantics. The most dominant example buildings. Unlike cars, pedestrians or bicycle riders, that separable using the instance map, buildings are not. The semantic label provides the pixels in which the building exists. Considering the fact that scene of adjacent buildings is somewhat common, the ability to separate them is of high value. Left the label map. Middle generated image by Wang et al. (2018b). Right our generated image. Our model can generate unique adjacent buildings from the semantic label maps of better quality compared to Wang et al. (2018b).
Figure 6: previous work test images Wang et al. (2018b) (Top) compared to our model test images (Bottom). The images generated by our model contain low level details, allowing the desired photo- realism
3.3
VIDEO GENERATION
Using pre trained CG2real networks, we generate two consecutive images, and then estimate two flow maps. The first flow map is between Ti, #141, where x; and #4+1 are two consecutive real im- ages. The second flow map is between G(s,, e,), G(s 6141) where G(s,, e,) and G(s e1+1) are two consecutive generated (fake) images. Note that the generation of G(s,, e:). G( e++1 is done independently, meaning apply our CG2real method twice, without any modifications. To conclude we enforce temporal coherency by using the following loss:
Groww = L1 Li (Freal, F fake)
(8)
Where Freat F(x,, Fyake F(G(si, ei), G( (i+1)) and F(+) is the optical flow operator. This formulation eliminates the need of using sequential generator as in Wang et al. (2018a), allowing not only using our image generation model twice, which adds more constrains to the video generation scheme, but also avoid errors accumulation arising from positive feedback by feeding generated image the generator, as can be seen in figure 3.3 and in this video.
By adding L to the LCG2real loss, the network learns to generate G(s e++1) taking the flow maps into account, thus generating temporally coherent images as depicted in 3.3.
Luidengen Litoo LCG2real

---

Under review as a conference paper at ICLR 2020
CG2real
Fake Image
Flow-Fake
c (Feaa
CG2real
Fake Image t+1
Flow-Real
block diagram of the video generation model. Two identical CG2real models generate Fake image (t) and Fake image (t+1). The two consecutive fake images are fed to the flow-fake estimator, while flow maps are trained using T,1 (Freat: Ffake) loss. This enables the pre-trained CG2real models to learn the required coherency for generating photo-realistic videos.
Figure 7:
consecutive real images are fed to the flow-real estimator. Both real and fake
RESULTS
Our goal is to generate photo-realistic images. (fig 3.2) we can find some examples from the CG2real image synthesis task, and in (fig 4) present consecutive images depicting the video to video synthesis. We use the same evaluation methods as used by previous image to image works ,e.g. pix2pix Isola et al. (2017),, pix2pixHD Wang al. (2018b) and others. The evaluation pro- cess consist performing semantic segmentation with a pre-trained seamntic segmentation network Zhao et al. (2017) synthesized images produces by our model, then calculating the semantic pixel accuracy and the mean intersection over union (mloU) over the classes in the dataset. As shown in tables 1, bellow, our network outperforms previous works. The ground-truth results are the pixel accuracy and mloU when performing the same semantic segmentation with the real images (Oracle).
Furthermore, to evaluate the image generation quality, we used another metric to evaluate dis- tances between datasets called FID (Frchet Inception Distance) Heusel et al. (2017), & Lunz (2018). It very common metric for generative models as it correlates well with the visual quality of generated samples Wang et al. (2018a). FID calculates the distance between two multi- variate Gaussians real and generated respectively; where X, ~ N(u,, E,) and X, ~ N(ug Eg) are the 2048-dimensional activations of the Inception-v3 pool3 layer Szegedy et al. (2016), and FTD |||, 12 Tr(E, + E 2(E,,gg)1/3 is the score for image distributions X, and Xg- Lower FID score better, meaning higher similarity between real and generated samples.
Cityscapes
Pix2pix
Pix2pixHD
Ours
Oracle
Pixel accuracy [%]
0.7279
0.81
0.83
0.86
Mean IoU [%]
0.5324
0.67
0.69
0.701
Table 1: semantic segmentation results on the cityscapes Cordts et al. (2016) validation set
Synthia
Pix2pix
Pix2pixHD
Ours
Oracle
Pixel accuracy [%]
0.54
0.79944
0.860753 0.740040
0.913132
Mean IoU [%]
0.36
0.55955
0,8419
Table 2:
semantic segmentation results on the Synthia Ros et al. (2016) dataset
As be seen in tables 1, 2, pix2pixHDs results are better than pix2pix for pixel accuracy and mloU. Our results are better than pix2piXHD, and almost meet the oracles results on both Synthia Ros et al. (2016) and cityscapes Cordts al. (2016). In table 3, we compare the FID score for all

---

Under review as a conference paper at ICLR 2020
FID,FVD
Pix2pix
Pix2pixHD
Vid2vid
Ours-img
Ours-vid
FID
I16.69
71.21
154.36
69.25
69.81
FVD
0.706
0.326
Table FID and FVD metric comparisson between pix2pix, pix2pixHD vid2vid and Ours.
the four image generation models w.r. the Oracle. Ours-img (Our image generation model) outper- forms both pix2pix and pix2pixHD. Moreover, adding a temporal consistency constrain to the image generation process degrades image quality. Vid2vid uses pix2pixHI as its image generation model imposes substantial degradation in the image quality (71.21 to 154.36). Our video generation uses CG2real model had marginal effect the FID score of Ours-vid (69.25 to 69.81 and even outperformed pix2pixHD) and did not degrade generated images quality (fig 4).
Our video generation evaluation method is FVD (Frchet Video Distance) proposed by Unterthiner et al. (2018). FVD is a metric for video generation models evaluation and it uses a modified version FID. we calculated the FVD score for our generated video (Ours-vid) w.r.t. the Oracle (real video) and did the same for vid2vid w.r. the same Oracle. Our FVD score on the video test set is 0.326 while vid2vid's 0.706 meaning our videos are more than twice similar to the oracle. we suggest
that this substantial margin stems from the errors accumulated in the video generation model of vid2vid (fig 4). As mentioned, Our video generation model uses our flow loss therefore does not encounter this phenomena.
Figure 8: Comparison of video generation. Up- images generated by vid2vid Wang al. (2018a), Down images generated by our video generation model. Our generated images are temporally coherent and visually appealing our images sky is more natural, road signs are clearer and build- ings have finer level of details. This example emphasizes the error propagation of vid2vid's model wile our model does not accumulate errors (see street lights in upper right corner each image). The main objective of the video generation model is to enable generating non flickering images by giving objects in consecutive images the same color and texture, i.c. sample from the same area in the latent spaces. full videos can be seen here.
5 SUMMARY
We present a CG2real conditional image generation as well as conditional video synthesis. We offer to use network learning the distribution of edge maps from real images and integrate into generator (DNED). We were able to generate highly detailed and diverse images thus enabling bet- ter photo-realism. Using the DNED enable generating diverse yet photo-realistic realizations of the same desired scene without using instance maps. As for video generation, we offer new scheme that utilizes flow maps allowing better temporal coherence videos. We compared our model to recent works and found that it outperforms both current quantitative results and more importantly generates appealing images. Furthermore, our video generation model generates temporally coher- ent and consistent videos.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jonas Adler and Sebastian Lunz. Banach wasserstein gan. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desaa-iancch,, and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 6754-6763. Curran Associates, Inc., 2018. URL http: pett::/ppapers. .nip.. cc/paper/795 /709 banach wett--awssssstteeeeee gan.pdf.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXi:: 2017.
Wasserstein gan.
arXiv preprint
Xue Bai, Jue Wang. David Simons, and Guillermo Sapiro. Video snapcut: Robust video object cutout using localized classifiers. In ACM SIGGRAPH 2009 Papers, SIGGRAPH '09, pp. 70:1- 70:11, New York, NY, USA, 2009. ACM. ISBN 978- 1-6055-- 726-44 doi: 10.1145/1576266. 1531376. URL http: / doi. acm. /10 00.....111117222100 15313766
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Peter Burt and Edward Adelson. The laplacian pyramid compact image code. IEEE Transac- tions on communications, 3144):332-540, 1983.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1511-1520, 2017.
Jack Collins, David Howard, and Jürgen Leitner. Quantifying the reality gap robotic manipulation tasks. arXiv preprint arXiv: 1et::111110010990 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2016.
Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using laplacian pyramid of adversarial networks, 2015.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based deep networks, 2016.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEFE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Golnaz Ghiasi and Charless C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. Lecture Notes in Computer Science, pp. 519534, 2016. ISSN 1611-3349, doi: 10. 11111079978-3331-44655 978-3-319-46487-2
URL http::pppwp.......oet.o 1007/
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Stephen Guattery and Gary L. Miller. Graph embedding and laplacian eigenvalues. SIAM J. Matrix Anal. Appl., 21(3):003-723, 2000.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp.
2017.
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked genera- tive adversarial networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.11099cvp..2017.222 URL http: / /xx doi org/ 10///1.10 1109/ 2PPR.2017.02

---

Under review as a conference paper at ICLR 2020
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10. 10.1109/cxp:2017..17, URL http:/ /dx. doi. org/
/ 10.11.111.0 111/1/...1111111//11 /11099////R222/ 2017 179.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style trans- fer and super-resolution. Lecture Notes in Computer Science, pp. 694711, 2016. ISSN 1611-3349. doi: 10. 978 3- 319 46775-666 6_3..
http:///xxxdd..org....ooo URL http://dxx. http:///dx..11..or1111.0.0.00
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. In Advances in neural information processing sys- tems, pp.
2016.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets, 2014.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2642 2651. JMLR. org, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. 2017 IEEE International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.11099icw.22117.3888 URL http:// / /xx.xxxxx///o///// doi org/o0gg////oooo 11191101000T 2017. 308.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
George Seif and Dimitrios Androutsos. Large receptive field networks for high-scale image super- resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 763-772, 2018.
Oded Shahar, Alon Faktor, and Michal Irani. Super-resolution from a single video. In CVPR, 2011. URL hwtt:htp//.. http:://ww, wisdom, we meeizman. ac. wtt.ic..ll//// wwtsssionSinngeee /iig/nnnneeeeeeeeeeeeeee html.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X.vv:409.1530 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 JEEE Conference Computer Vision and Pattern Recognition ( (CVPR), Jun 2016. doi: 10. 10.1109900:.0016..333.. URL http:/ doi. org/10. 1ttp1/0.119////// 201 6.16.306 3.80..
10

---

Under review as a conference paper at ICLR 2020
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1 10.1009/cyp:2201..006...
URL btttp://xx.oo doi.oi.....oooooooooooo.oo.....d org/1c..
1109/CVPR. 2018 00111.001600...0
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly, Towards accurate generative models of video: A new metric challenges. arXiv preprint arXiv:1812.01777, 2018.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics, 2016.
Ting-Chur Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis, 2018a.
Ting-Chum Wang, Ming-Yi Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018b.
Yonatan Wexler, Eli Shechtman, and Michal Trani. Space-time video completion. In Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume I, pp. I-I. IEEE, 2004.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. Internationnl Journal of Computer Vision, 125(1-3):318, Mar 2017. ISSN 1573-1405. doi: 10.1007/s1 10.10077/1266------0444- URL http: dx doi www./o.....oooooooooooooo......o 11.11111000000 100701000000000000 000007/s11233 117-1004-2.
Han Zhang, Tao Xu, and Hongsheng Li. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. 2017 IEEE International Conference on Computer sion (ICCV), Oct 2017. doi: 1..1109cccv..217..66
doi. wtt.:/....og/01100000100
ICCV. 2017. 629.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1 2017/c URL. http:/htt///t /xx. http///xxxx/oo///////o//oo/.oo 1tt....gg/...1//////// 11w./1199/...221 2017. 660.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network, 2016.
Jun-Yan Zhu, Philipp Krhenbhl, Eli Shechtman, and Alexei A. Efros. Generative visual manipula- tion on the natural image manifold. Lecture Notes in Computer Science, pp. 597613, 2016. ISSN 1611-3349. doi: 10. 778-3-319-2 1000---4444-1---
URL http:// doi wttp://.............o........
Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Bli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pp. 465-476, 2017.

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Figure 9: Additional Test images from Cityscapes Dataset. Left pix2pixHD. Right Ours. These images further demonstrate the photo-realism achieved by our model.
12

---

Under review as a conference paper at ICLR 2020
Figure 10: Additional test images on Synthia dataset. Left- pix2pixHD. Right Ours. These images demonstrate improved image quality, better and finer details in the generated objects, buildings and vegetation.
13

---

Under review as a conference paper at ICLR 2020
Figure 11: Additional test images on Synthia dataset. Left- pix2pixHD. Right Ours. These images demonstrate improved image quality, better and finer details in the generated objects, buildings and vegetation.
14

---

Under review as a conference paper at ICLR 2020
Figure 12: Test video on CityScapes. Left vid2vid. Right Ours video gen model. These images demonstrate better temporal coherency in the generated images. Moreover, the top left corner of the left video, we notice the error propagates. Better yet, the buildings in the right video are more reasonable, w.r.t. windows, shades, general texture, etc. As for image quality, the road signs in the right video are better emphasized.
15

---

Under review as a conference paper at ICLR 2020
de
Figure 13:
Test video on Scayes Left - vid2vid. Right Ours video gen model. This figure provides more images from the same video presented in fig 4. Pay attention to the error propagation on the top right images of vid2vid. Again, our model demonstrates finer road signs and higher level of details in the generated buildings.
16


Reviews:
[{"review_id": "S1lOOAZTFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method to improve the image-to-image translation. By utilizing the CG-based Synthia and embedded edge maps, it has shown some effects on the Cityscapes dataset.\n\nThis paper is not novel. Generally, it just utilized additional sources to help the generation. It is difficult to generalize to another dataset/domain as the CG-based set is not easy to get. The model side has very limited novelty. The pipeline and motivation are similar to previous work for domain adaptation for generative models and bridging domains:\n1. Unsupervised Image-to-Image Translation Networks\n2. Adaptation Across Extreme Variations using Unlabeled Domain Bridges\n\nThe experimental study is weak. Only the Cityscapes dataset is included. It is good to see if this method can work on others, like ADE20k and COCO-stuff. Also, some strong baseline, such as SPADE, is ignored. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkx9woWCFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors use edge maps to generate more realistic images and videos in GANs. They train a GAN in a conditional way to\ngenerate a photo-realistic version of a given scene.\n\nIn my opinion, the novelty of the proposed approach is not high and the improvements are incremental. I have some concerns about the method as well: what happens if the quality of edge maps is not good for some datasets? Is it realistic to assume that the edge maps of real images are provided with high quality? \n\nRegarding the video generation, authors should compare their approach with recent video GAN papers such as https://arxiv.org/abs/1907.06571\n\nThere are several typos in the paper as well. I suggest that authors proof read their paper more carefully. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxuSoZu9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY: Use HED to convert semantic maps to edge maps, then use GAN to generate realistic images, by adding several losses together. Also add a flow loss while generating videos.\n\nPaper could be written much better, it is slightly confusing to follow.\n\nThe effort behind the work is commendable, they used good baselines to compare. However, the idea does not seem different from already existing ones. Although the authors mention this as a CG2real problem, as far as I understand this is a semantic maps to image problem. Maybe I'm missing something.\n\nThe only innovation is to have a middle step of converting these semantic maps to edge maps, and then from edge maps to realistic images. This is not altogether a big change, I was hoping to see it goes in a good direction. As far as I understand, the DNED model the authors use is a slight change to the HED model the authors cite, in that they sample different weights to combine to form different outputs.\n\nThe results look good from the examples shown. However, there is a lot more work to be done. Since architecturally there is not much difference in the already existing models, maybe focus on more use cases of edge maps. What is the advantage of using edge maps as an intermediate step, as opposed to some other representation? How do we exploit this fact better? These are good questions to explore.\n\nThe video results look good, and this is one example of exploiting the fact that edge maps are an intermediate step. However, more work needs to be done to support claims of diversity.\n\nOverall, it seems intuitive to throw in a lot of losses and let the network optimize over all of them in an adversarial way, but a lot more effort needs to be put into thinking about any innovation there.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a new GAN-based method for translating from semantic maps of (synthetic) images/videos (from computer graphics) to photo-realistic images/videos with the aid of edge maps. The main innovation is the inclusion of edge maps to the generator, where the edge maps are initially computed using the spatial Laplacian operator, and later output from their DNED network. According to the authors, the edge map allows them to generate images with fine details and to generate output images at higher resolutions.  The authors use their method to generate both single images as well as videos. 

The submission received relatively low scores (2 rejects and 1 weak reject).  This was unchanged after the rebuttal (the authors did not submit a revised version of their paper).  The reviewers voiced concerns about the following:
1. Limited novelty
All of the reviewers indicated that they felt the novelty of the proposed approach of not high as the work seemed to make only small modifications on prior work.  In the author response, the authors provided some details on where they felt their innovation to be.  The paper can be improved by building on those and having experiments/examples to probe those claims in more detail. 

2. Application to other datasets
The proposed method is demonstrated only on two datasets of driving scenarios (Cityscapes and Synthia).  It is unclear how the method will generalize to other types of inputs.  Experiments on other datasets will demonstrate whether the proposed approach can work well for other types of images.

3. The overall quality of the writing.  
The overall quality of the writing is poor and hard to follow in places. The paper should also include more discussion of domain adaptation in the related work section.  It's possible that with improved writing that situates the work and explains the novel aspects of the work better, that the concern about limited novelty will be partially alleviated.   The paper also needs an editing pass as there are many grammar/spelling/capitalization issues.

Page 2: "We make Three" --> "We make three"
Page 3: "as can bee seen in fig 3.2" --> "as can be seen in ..." (it's unclear which figure "fig 3.2" refers to, as figures are labeled Figure 1, Figure 2, etc)
Page 4: Equation (3), symbol e is not explained (it is presumably the edge map)
Page 7: "bellow" --> "below"

Overall, there are interesting elements in this paper and the reviewers noted that the generated results look good.  However, the paper will need to be improved considerably.   The authors are encouraged to improve their work and submit to an appropriate venue.

Decision: Reject