ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods

1 INTRODUCTION . Climate change is a serious danger to our planet , with warming temperatures causing extreme weather events that affect the livelihood of an increasing number of people globally ( HoeghGuldberg et al. , 2018 ) . In particular , rising sea levels , increasing precipitation and faster snow melt exacerbate extreme floods , presenting a major risk to populations worldwide ( Dottori et al. , 2016 ) . One common barrier to climate action and behavioral change is distancing , a psychological phenomenon resulting in climate change being perceived as temporally and spatially distant and uncertain . Previous research has shown that showing or simulating first-person perspectives of climate change-related extreme weather events can contribute to reducing distancing ( Chapman et al. , 2016 ; Sevillano et al. , 2007 ) . Digital technologies are increasingly used for this purpose ( Herring et al. , 2017 ; Ahn et al. , 2014 ) , but they often target specific regions or render manually their effects . In this context , we have developed ClimateGAN , which can generate extreme flooding based on arbitrary street-level scenes , such as Google Street View images . We generate floods of 1 m , which is consistent with the definition of ” extreme flooding ” in the literature ( Kulp & Strauss , 2019 ) , and we divide the task of flooding into two parts : a Masker model to predict which pixel locations of a given image would be under water if a flood occurred , and a Painter model to generate contextualized water textures conditioned on both the input and the Masker ’ s prediction . Our contributions are : proposing and motivating the novel task of street-level flood generation , a data set of pairs of images with/without flooding from a virtual world , the ClimateGAN model which includes a novel multitask architecture for generating geometry- and semantics-informed binary masks and a procedure to thoroughly evaluate it in the absence of ground-truth data . We also compare our model to existing generative modeling frameworks and provide an ablation study of the components of our model . 2 RELATED WORK . While the task of generating extreme flooding in street-level images is novel , related work has been carried out in applying Deep Learning for flood segmentation ( Sazara et al. , 2019 ) and flood depth estimation ( Kharazi & Behzadan , 2021 ) . Generative modeling has also been used for transferring weather conditions on street scenes using both imagery ( Li et al. , 2021 ) and semantic maps ( Wenzel et al. , 2018 ) . For the purposes of our task and given its unique constraints , we frame our approach in the context of image-to-image translation , involving conditional image synthesis and domain adaptation . We present relevant related work from these three areas in the sections below . Image-to-image translation ( IIT ) is a computer vision task whose goal is to map a given image from one domain to another ( Liu et al. , 2017 ) . IIT approaches can either carry out the translation on the entire input image or utilize masks to guide the translation task . In the first case , initial IIT approaches relied on the existence of two aligned domains such as photographs and sketches of the same objects . CycleGAN ( Zhu et al. , 2017 ) this constraint , allowing the domains to remain unaligned , and further progress was made by architectures such as MUNIT ( Huang et al. , 2018 ) and CUT ( Park et al. , 2020 ) . The second category of IIT focuses the translation process on particular input image areas , typically by leveraging attention or segmentation masks . This more closely resembles our case , as we aim to flood only part of the image . Notable examples of this category include Attention-Guided GANs ( Tang et al. , 2019 ) and InstaGAN ( Mo et al. , 2019 ) which uses instance-level semantic masks to guide the translation process . Conditional image synthesis differs from IIT in that the input can be a label , text or a segmentation map , instead of another image ( Mirza & Osindero , 2014 ) . One approach from this category that is particularly relevant to our work is SPADE ( Park et al. , 2019 ) , a module that enables the transformation of a semantic layout—such as that of a street scene or landscape—into an image that semantically matches this layout . The idea behind SPADE is to create residual blocks where the input is first normalized and then denormalized in a spatially relevant way by small convolutional networks , functions of spatial conditioning variables . This approach also introduced the GauGAN generator , which leverages SPADE blocks to learn a spatially-adaptive transformation , enabling the synthesis of realistic images based on the input maps . Domain adaptation aims at transferring knowledge from one domain to another using different data sources ( Ganin & Lempitsky , 2015 ) . This can be particularly useful in tasks where more ( labeled ) data is available from a simulated world than in the real world , like in our case . Domain adaptation techniques can then be used to bridge the distributional gap between real and simulated scenes , learning useful tasks such as semantic segmentation and depth prediction , which function both in real and simulated scenarios . Examples of domain adaptation approaches that adopt these techniques include : CYCADA ( Hoffman et al. , 2018 ) , which leverages cycle-consistency constraints similar to those proposed by CycleGAN to improve domain adaptation , ADVENT ( Vu et al. , 2019a ) , which uses Adversarial Entropy Minimization to achieve high performance in unsupervised domain adaptation for semantic segmentation , and Depth-aware Domain Adaptation ( DADA ) ( Vu et al. , 2019b ) , which improves on ADVENT by leveraging dense depth maps . 3 CREATING IMAGES OF FLOODS . Our task of generating flooding resembles that of unsupervised image-to-image translation . However , this framework poses two major problems : first , the translation needs to be restricted to the portion of the image that would contain water , which advises against approaches which alter the im- age globally . Second , we are only concerned with adding water and not the reverse , which eliminates the need for cycle-consistent approaches . Therefore , in order to undertake this task , we developed a novel conditional image synthesis approach that consists of two models : a Masker that produces a binary mask of where water would plausibly go in the case of a flood , and a Painter that renders realistic water given a mask and an image . We provide an overview of this procedure in Fig . 2 , and detail the individual components in the remainder of this section . 3.1 DATA . Real Data First-person images of floods are scarce ; moreover , even when such pictures exist , it is exceedingly rare to find the corresponding image of the same location before and after flooding . We therefore pursued several alternate approaches for gathering real flooded street-level images , spanning from web-scraping to creating a website and a mobile app to collect crowd-sourced images . We complemented flooded images with images without floods , i.e . of normal streets and houses . We aimed to cover a broad scope of geographical regions and types of scenery : urban , suburban and rural , with an emphasis on images from the Cityscapes Cordts et al . ( 2016 ) and Mapillary Neuhold et al . ( 2017 ) data sets . We collected a total of 6740 images : 5540 non-flooded scenes to train the Masker , and 1200 flooded images to train the Painter . Simulated Data Besides the lack of paired data , another limitation of real-world images is that they do not contain scene geometry annotations and semantic segmentation labels , which we want to leverage during model training . To solve both of these problems , we created a 1.5 km2 virtual world using the Unity3D game engine . To be as realistic as possible , it contains urban , suburban and rural areas , which we flooded with 1m of water to gather ‘ with ’ and ‘ without ’ pairs ( see A ) . For each pair of images obtained , we also captured the corresponding depth map and semantic segmentation layout of the scene . In the end , we gathered approximately 20000 images from 2000 different viewpoints in the simulated world , which we used to train the Masker . We make this data set publicly available1 to enable further research . 3.2 MASKER . To leverage the information available in the simulated domain and transfer performance to real data , we adapted DADA ( Vu et al. , 2019b ) to train the Masker . The main objective of this procedure is to inform segmentation with depth . A naive use of DADA for our task would miss a crucial difference : the Masker produces information about what could be in an image , not what is present in it . The core contribution of our Masker ’ s architecture is therefore to structure it as a multi-headed network with depth and segmentation decoders to learn what is present in the current scene , then conditioning the flood mask decoder on this information ( and the input image ) to predict where water could be . In the following section , subscripts s and r respectively identify the simulated and real domains ; we use i ∈ { r , s } to refer to an arbitrary domain . E is an encoder network while D , S and M , are the depth , segmentation and flood mask decoders , respectively , as per Fig . 2 . To guide decoders in the early training stages , we used pseudo labels as supervision targets for real data . In other words , we considered predictions of ad hoc pretrained models as ground truth . However , because those labels are noisy , we limited that procedure to the beginning of training . Depth decoder We consider depth in the disparity space , and predict the normalized inverse depth di = D ( E ( xi ) ) from an input image xi . We used the scale-invariant loss from MiDaS ( Lasinger et al. , 2019 ) , which is composed of a scale and shift-invariant MSE loss term LSSIMSE and a gradient matching term LGM . We used the following targets to compute this loss : ground-truth depth maps for simulated input images and pseudo labels inferred from the MiDaS v2.1 model for real input images . The complete depth loss is : LDepth = λ1LSSIMSE + λ2LGM . ( 1 ) Segmentation decoder The segmentation decoder S is implemented such that S ◦E corresponds to the DeepLabv3+ architecture ( Chen et al. , 2018 ) . It is trained as described in DADA , leveraging 1URL hidden for peer-review depth information available in the simulated world to improve segmentation predictions by giving more attention to closer objects , producing si = S ( E ( xi ) , di ) . Two fusion mechanisms encourage this : feature fusion , which multiplies element-wise the latent vector zi = E ( xi ) by a depth vector obtained from the depth decoder , and DADA fusion , which multiplies the self-information map ( Vu et al. , 2019a ) I ( si ) = −si · log si element-wise with the depth predictions di to obtain the depthaware self-information map Î ( si ) = I ( si ) di . In addition to DADA we used pseudo labels inferred from a pre-trained segmentation model in the real domain . The rest of the training of S is however similar to DADA : 1/ to encourage real domain predictions to be confident and reduce the gap with simulated predictions an entropy minimization ( EM ) term LEM ( sr ) is added and 2/ the domain gap between the distributions of real and simulated self-information maps is shrunk using WGAN-based adversarial training ( Arjovsky et al. , 2017 ) . LSeg = λ3LCE + λ4LEM + λ5LWGAN . ( 2 ) Flood mask decoder This decoder is structured to be conditioned not only on the input image , but also on predictions di and si from other decoders . To implement this dependence , we propose a new use of SPADE ( Park et al. , 2019 ) conditional blocks . In our case , for an input xi , the conditioning variable is therefore U i = [ xi , di , si ] , where the tensors are concatenated along the channel axis . The mask mi = M ( zi , U i ) and its self-information map I ( mi ) are computed from the latent representation zi = E ( xi ) . We also implemented a total variation ( TV ) loss on the mask mi for both domains in order to encourage the predictions to be smooth , ensuring that neighboring pixels have similar values ( Johnson et al. , 2016 ) —note that ∆ is the spatial difference of the image mesh : LTV ( mi ) = En , h , w [ ( ∆hmi ) 2 + ( ∆wmi ) 2 ] . ( 3 ) In the simulated domain , we used a binary cross-entropy loss LBCE ( yms , ms ) with the groundtruth mask yms . In the real domain , absent of any ground truth , we encouraged the predicted flood mask mr to at least encompass the ground by introducing a ground intersection ( GI ) loss , penalizing masks that assign a low probability to locations where a pre-trained model detected ground gr : LGI ( gr , mr ) = En , h , w [ 1 ( gr−mr ) > 0.5 ] . ( 4 ) As per the DADA approach , we also added an entropy minimization loss to increase the mask decoder ’ s confidence in its real domain predictions : LEM ( mr ) = En , c , h , w [ −mr logmr ] . ( 5 ) Lastly , similarly to the segmentation decoder , we adversarially trained the flood mask decoder with a WGAN loss LWGAN to produce self-information maps Î ( mi ) indistinguishable by a discriminator . M ’ s total loss is a weighted sum of all of the above losses : LMask = λ6LTV + λ7LGI + λ8LBCE + λ9LEM + λ10LWGAN . ( 6 ) The Masker ’ s final loss sums the losses of the three decoders : LMasker = LDepth+LSeg+LMask .

Summary: This paper proposes to generate images of floods, that is, simulate photo-realistic floods, with a model named ClimateGAN, which consists of a Masker to generate masks for floods and a Painter to draw floods on images. The proposed ClimateGAN is a two-stage solution for generating flood masks on input images and painting the images with floods respectively, and each stage (Masker and Painter) is designed based on many popular related techniques, like DADA for depth-aware semantic segmentation, SPADE for injecting extra information into cGANs, and WGAN as well as TV, BCE and EM losses. For training ClimateGAN, this work also collects a real dataset with street-level floods and generates a simulated dataset using Unity3D. The outputs of Masker and Painter are evaluated separately compared to several previous image-to-image translation methods, also the ablation study is conducted to validate the effectiveness of each component in ClimateGAN.