ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods
Authors: Victor Schmidt
Alexandra Luccioni
Mélisande Teng
Tianyu Zhang
Alexia Reynaud
Sunand Raghupathi
Gautier Cosne
Adrien Juraver
Vahe Vardanyan
Alex Hernández-García
Yoshua Bengio
Venue: ICLR 2022 Poster (2022)
Keywords: GAN
Climate Change
Domain Adaptation
Representation Learning
Computer Vision
Application

Abstract: Climate change is a major threat to humanity and the actions required to prevent its catastrophic consequences include changes in both policy-making and individual behaviour. However, taking action requires understanding its seemingly abstract and distant consequences. Projecting the potential impacts of extreme climate events such as flooding in familiar places can help make the impacts of climate change more concrete and encourage action. As part of a larger initiative to build a website (https://thisclimatedoesnotexist.com) that projects extreme climate events onto user-chosen photos, we present our solution to simulate photo-realistic floods on authentic images. To address this complex task in the absence of suitable data, we propose ClimateGAN, a model that leverages both simulated and real data through unsupervised domain adaptation and conditional image generation. In this paper, we describe the details of our framework, thoroughly evaluate the main components of our architecture and demonstrate that our model is capable of robustly generating photo-realistic flooding on street images.

Published as a conference paper at ICLR 2022
CLIMATEGAN: RAISING CLIMATE CHANGE AWARENESS BY GENERATING IMAGES OF FLOODS
Victor Schmidt" 12, Alexandra Luccioni Mélisande Tengl2, 12, Tianyu Zhang!.. 1.2 Alexia Reynaud!.2, Sunand Raghupathi Gautier Cosne!.,, Adrien Juraver!., Vahe Vardanyan', Alex Cermnddez-Garche 12, and Yoshua Bengio!. 1.2
! Milla Québec AI Institute, Montréal, Canada Université de Montréal, Montréal, Canada Columbia University, New York City, USA
*CDRIN, Matane, Canada
ABSTRACT
Climate change is a major threat to humanity and the actions required to prevent its catastrophic consequences include changes in both policy-making and individual behaviour. However, taking action requires understanding its seemingly abstract and distant consequences. Projecting the potential impacts of extreme climate events such as flooding in familiar places can help make the impacts of climate change more concrete and encourage action. As part of a larger initiative to build a website (https: //ttts/tttieeeee 1 teeattddonottee
st. com) that projects ex- treme climate events onto user-chosen photos, we present our solution to simulate photo-realistic floods authentic images. To address this complex task in the absence of suitable data, we propose ClimateGAN, a model that leverages both simulated and real data through unsupervised domain adaptation and conditional image generation. In this paper, we describe the details of our framework, thor- oughly evaluate the main components of our architecture and demonstrate that our model is capable of robustly generating photo-realistic flooding on street images.
INTRODUCTION
Climate change is serious danger to our societies, with warming temperatures causing extreme weather events that affect the livelihood of an increasing number of people globally (Hoegh- Guldberg et al., 2018). In particular, rising sea levels, increasing precipitation and faster snow melt exacerbate extreme floods, presenting a major risk to populations worldwide (Dottori et al., 2016). One common barrier to climate action and behavioral change is distancing. a psychological phenomenon resulting in climate change being perceived as temporally and spatially distant and un- certain. Showing or simulating first-person perspectives of climate change-related extreme weather events contribute to reducing distancing (Chapman al. 2016 [Sevillano et al. 2007) and infor- mation technologies are increasingly used for this purpose (Herring et al. 2017 Ahn et al. 2014), but they often target specific regions render manually their effects.
In this context, we have developed ClimateGAN, which can generate extreme flooding based on ar- bitrary street-level scenes, such Google Street View images. We generate floods of m, realistic expected water level for climate change-related flooding events (Kulp & Strauss 2019), and we di- vide the task of flooding into two parts: Masker model predict which pixel locations of given image would be under water a flood occurred, and a Painter model to generate contextualized water textures conditioned on both the input and the Masker's prediction. Our contributions are: proposing and motivating the novel task of street-level flood generation, a data set of pairs of images with/withhut flooding from virtual world, the ClimateGAN model which includes a novel multi- task architecture for generating geometry- and immantcc-inffomed binary masks and a procedure to thoroughly evaluate it in the absence of ground generative modeling frameworks and provide an ablation study of the components of our model.
data. We also compare our model to existing
"Corresponding authors- tschmidtv, Jucci on s. 1 1a quebec

---

Published as a conference paper at ICLR 2022
Mila
Mila
Figure 1: ClimateGAN, a model that generates extreme floods (right) on street-level images (left).
RELATED WORK
While the task of generating extreme flooding in street-level images is novel, related work has been carried out applying Deep Learning for flood segmentation (Sazara et al. 2019) and flood depth estimation (Kharazi & Behzadan 2021). Generative modeling has also been used for transferring weather conditions on street scenes using both imagery (Li et al. (021) and semantic maps (Wenzel et al. 2018). For the purposes of our task and given its unique constraints, we frame approach in the context of image-to-image translation, involving conditional image synthesis and domain adaptation. We present relevant related work from these three areas in the paragraphs below.
Image-to-image translation (IIT) a computer vision task whose goal is map a given image from one domain to another (Liu et al. 2017). IIT approaches can either carry out the translation on the entire input image or utilize masks to guide the translation task. the first case, initial IIT approaches relied on the existence of two aligned domains such photographs and sketches of the same objects. CycleGAN (Zhu et al. 2017) relaxed this constraint, allowing the domains to remain unaligned, and further progress was made by architectures such MUNIT (Huang al., 2018) and CUT (Park et al.) (020). The second category of IIT focuses the translation process on particular input image areas, typically by leveraging attention or segmentation masks. This more closely resembles our case, as we aim to flood only part of the image. Notable examples of this category include Attention-Guided GANs (Tang et al. 2019) and InstaGAN (Mo al. 2019) which uses instance-level semantic masks to guide the translation process.
Conditional image synthesis differs from IIT in that the input can be a label, text segmen- tation map, instead of another image (Mirza & Osindero, 2014). One approach from this category that is particularly relevant our work is SPADE (Park al. 2019), module that enables the transformation of a semantic layout-such as that of a street scene or landscape-into an image that semantically matches this layout. The idea behind SPADE to create residual blocks where the input is first normalized and then denormalized in a spatially relevant way by small convo- lutional networks, functions of spatial conditioning variables. This approach also introduced the GauGAN (Park et al. 2019), generator, which leverages SPADE blocks to learn a spatially- transformation, enabling the synthesis of realistic images based on the input maps.
Domain adaptation aims at transferring knowledge from one domain to another using different data sources (Ganin Lempitsky 2015). This can be particularly useful in tasks where more (la- beled) data is available from a simulated world than in the real world, like in our case. Domain adaptation techniques can then be used to bridge the distributional gap between real and simulated scenes, learning useful tasks such as semantic segmentation and depth prediction, which function both real and simulated scenarios. Examples of domain adaptation approaches that adopt these techniques include: CYCADA (Hoffman al. 2018), which leverages cycle-consistency constraints similar to those proposed by CycleGAN to improve domain adaptation, ADVENT (Vu et al. 2019a), which uses Adversarial Entropy Minimization to achieve high performance in unsupervised domain adaptation for semantic segmentation, and Depth-aware Domain Adaptation (DADA) (Vu et al. 2019b), which improves ADVENT by leveraging dense depth maps.
2

---

Published as a conference paper at ICLR 2022
3 CREATING IMAGES OF FLOODS
Our task resembles that of unsupervised image-to-image translation. However, we identified three major challenging differences: first, the translation restricted to the portion of the image that would contain water rather than altering the image globally. Second, the water occludes multiple objects in the scene and typically only a part of them, which differs from the application cases of instance- aware methods. Finally, we are only concerned with adding water and not the reverse, which eliminates the need for cycle-consistent approaches. Therefore, order to undertake this task, we developed a novel conditional image synthesis method that consists of two models: a Masker that produces a binary mask of where water would plausibly go in the case of a flood, and a Painter that renders realistic water given mask and image. We provide overview of this procedure in Fig. 2 and describe the individual components in the remainder of this section.
3.1 DATA
First-person images of floods are scarce, the corresponding image before the flood is rarely available and even more so scene geometry and semantic segmentation annotations, which we want to lever- age during training. To overcome these limitations, we created a virtual world to generate annotated flooded and non-flooded pairs of images, and pursued multiple approaches to collect real photos.
Simulated Data We created a 1.5 kmn² virtual world using the U engine. To be as realistic as possible, we simulated urban, suburban and rural areas, which we flooded with Im of water to gather 'with' and 'without' pairs (see Appendix A). For each pair of images, we also captured the corresponding depth map and semantic segmentation layout of the scene. Overall, we gathered approximately 20,000 images from 2,000 different viewpoints in the simulated world, which used to train the Masker. We make this data set publicly available to enable further research.
Real Data Approaches for gathering real flooded street-level images spanned from web-scraping to crowd-soureing via website and mobile app P. We also included images without floods of typical streets and houses, aiming to cover a broad scope of geographical regions and types of scenery: urban, suburban and rural, with an emphasis on images from the Cityscapes Cordts et al. (2016) and Mapillary Neuhold et al. (017)) data sets. We collected a total of 6740 images: 5540 non-flooded scenes to train the Masker, and 1200 flooded images to train the Painter.
3.2
MASKER
To leverage the information available in the simulated domain and transfer performance to real data, we adapted DADA (Vu et al.) (019b) to train the Masker. The main objective of this procedure is to inform segmentation with depth. A naive use of DADA for our task would miss crucial difference: the Masker produces information about what could be in an image, not what is present in it. The core contribution of our Masker's architecture is therefore to structure it as a multi-headed network with depth and segmentation decoders to learn what is present in the current scene, then conditioning the flood mask decoder on this information (and the input image) predict where water could be.
In the following, subscripts s and r identify the simulated and real domains, respectively; we use i {r,s) $} to refer to an arbitrary domain. E is an encoder network while D, S and M, are the depth, segmentation and flood mask decoders, respectively, as per Fig.
Depth decoder We consider depth in the disparity space, and predict the normalized inverse depth d, (EE(z))) from an input image H;. We used the scale-in scale-invariant loss from MiDaS (Lasinger et al. 2019). which is composed of a scale and shift-invariant MSE loss term LSSIMSE and a gradient matching term LGM- We used the following targets to compute this loss: ground-truth depth maps for simulated input images and pseudo labels inferred from the MiDaS v2.1 model for real input images. The complete depth loss is:
Depth ASSSSSE + AzLGM-
(I)
https://cliiateee
3

---

Published as a conference paper at ICLR 2022
x
E
2
M
M
P
y
D
Masker
Tensors: Input,
Unconditional
SPADE-hused conditional network
Intermediate, Output
convolutional network
Figure 2: The ClimateGAN generation process: first, the input æ goes through the shared encoder E. Three decoders use the resulting representation z: D predicts depth map d, S produces depth- informed segmentation map and lastly M outputs a binary flood mask, taking 2 input and sequentially denormalizing with SPADE blocks conditionally on d, and æ. Finally, the Painter P generates an image ý of a flood, conditioned on the input image a and the binary predicted mask m. Note that the Masker and the Painter are trained independently and only combined test-time.
Segmentation decoder The segmentation decoder S is implemented such that S o E corresponds to the DeepLabb33+ architecture (Chen et al. 2018 It is trained described in DADA, leveraging depth information available the simulated world to improve segmentation predictions by giving more attention closer objects. producing 8; -E;(;;; d:). Two fusion mechanisms encourage E(x;) by a depth vector this: feature fusion, which multiplies element-wise the latent vector self-information map (Vu obtained from the depth decoder, and DADA fusion, which multiplies the
et al. 2019a) I(s;)
log dlement-wiss. with the depth predictions d: to obtain the depth- aware self-information map I (ss) (8;) de
addition to DADA, we used pseudo labels inferred from a pre-trained segmentation model in the real domain. Otherwise the training of S is similar to DADA: to encourage confident real do- main predictions and reduce the gap with simulated predictions an entropy minimization (EM) term CEM ( is added. Further, WGAN-based adversarial training (Arjovsky et al. (2017) is leveraged to shrink the domain gap between the distributions of real and simulated self-information maps:
Lseg = >gLCE + X4LEM + dgEWGAN.
(2)
Flood mask decoder This decoder is structured to be conditioned not only the input image, but also on predictions d, and 8; from other decoders. To implement this dependence, we propose of SPADE conditional blocks. In our case, for input æ;, the conditioning variable is therefore U; [2;; d,, si]. where the tensors are concatenated along the channel axis. The mask m: M (z U,) and its self-information map ((m,, are computed from the latent representation z; = E(æ,). We also implemented a total variation (TV) loss on the mask m; for both domains in order to encourage the predictions to be smooth, ensuring that neighboring pixels have similar values (Johnson et al. 2016) note that A is the spatial difference of the image mesh:
Crv = En,,,, = +
m;
In the simulated domain, we used binary cross-entropy loss Lacr(ym, m.)) with the ground- truth mask Um. In the real domain, absent any ground truth, we encouraged the predicted flood mask m, at least encompass the ground by introducing ground intersection (GD) loss, penalizing masks that assign low probability to locations where pre-trained model detected ground g,::
Ler m.)) = En, (1. (g.-mm.)20.5)
As per the DADA approach, we also added an entropy minimization loss to increase the mask decoder's confidence in its real domain predictions:
LEM (m,) En,c.h.wl -m, log m,).
(5)
4

---

Published as a conference paper at ICLR 2022
Input
Depth
Segmentation
Mask
Masked Input
Painted Input
Figure 3: Example inferences of ClimateGAN, along with intermediate outputs. The first row shows how the Masker is able to capture complex perspectives and how the Painter is able realistically contextualize the water with the appropriate sky and building reflections. On the second row, can see that close-ups with distorted objects do not prevent the Masker from appropriately contouring objects. Finally, the last row illustrates how, in unusual scenes, the Masker may imperfectly capture the exact geometry of the scene but the final rendering by the Painter produces an acceptable image of flood. More inferences including failure cases are shown in Appendix HI
Lastly, similarly to the segmentation decoder, we adversarially trained the flood mask decoder with a WGAN loss CANAN to produce self-information maps (m,) indistinguishable by a discriminator. M's s total loss is weighted sum of all of the above losses:
LMask dgCTV = X7har + A&CECE + dgCEM + XIoLWGAN.
(6)
The Masker's final loss sums the losses of the three decoders: LMsakeer = Deppeh + L Lassk
3.3 PAINTER
Given an input image and binary mask, the goal of the Painter is to generate water in the masked area while accounting for context in the input image. This is important for the realism of flooded image because water typically reflects the sky and surrounding objects, and is colored conditionally on the environment. The architecture of the Painter is based on GauGAN using SPADE conditioning blocks. These blocks learn transformations that vary both spatially and conditionally allowing the model better propagate contextual information through the generation pipeline than if it were only available at the input of the generator. We adapted GauGAN to fit our task: rather than conditioning on a semantic map, we conditioned on a masked image. Finally, we copied the non-masked area back onto the generated image, to ensure that the other objects in the image (e.g. buildings and sky) remained intact. We trained the Painter on the 1200 real flooded images, inferring m from pseudo labels of water segmented by a pre-trained DeepLabv3+ model. At test time was generated by the Masker from æ. Thus, the output ý of the Painter P is:
the input,
P(c, m)
+ @ (1-m),
N(.1)).
According to Park et al. 2019). a perceptual VGG loss (Ledig et al. 2017) and a discriminator feature-matching loss (Salimans al. 2016) are essential for good performance. Since these last two losses rely on comparing imput and generated output, we trained the Painter using images of floods, separately from the Masker (which requires non-flooded images inputs).
5

---

Published as a conference paper at ICLR 2022
4 EVALUATION METHOD
The quality and photo-realism of our model's output depend on both the accuracy of the Masker in determining realistic area for the flood and the ability of the Painter to infill the input mask with water with realistic texture matching the surrounding scene. In order to best understand the contribution of each component, we evaluated the Masker and the final flooded images separately.
4.1 MASKER EVALUATION
The lack of ground-truth data is not only an obstacle for training, but also for obtaining reliable evaluation metries to assess the performance of our model. Therefore, order obtain metries for evaluating the quality of the Masker and for comparing the individual contribution of the proposed components to its architecture, we manually labeled test set of 180 images retrieved from Google Street View. We collected images of diverse geographical provenances, levels of urban development, and compositions (crowds, vehicles, vegetation types, etc.). We manually annotated every pixel of each image with one of three classes: (1) munoo--e-flooded-aa pixels higher than 1 1.5m above the ground level; (2) foust-b-flooeed--e
with height less than 0.5 m, and (3) may-be-flooded all remaining pixels. We provide further details in Appendix C.I
4.1.1 METRICS
We propose the following metrics to evaluate the quality of masks (more details in Appendix C.2):
Error rate The perceptual quality of the generated images is highly impacted by the location and area of the predicted masks' errors. We found that both large gaps in must-be-flooded areas (i.e. false negatives, FN), and large sections of predicted masks in cannot-be cannot-be-floodee be-foooddd areas (i.e. false positives, FP) account for low perceptual quality. Thus, we propose the error rate to account for the amount of prediction errors in relation to the image size: error (F'N FP)/(H W).
F05 Score So as to consider the precision and recall of the mask predictions, we also evaluate the F8=0.5 (F05) score, which lends more weight to precision than recall (van Rijsbergen, 1979).
Edge coherence is also important to take into account the shape similarity between the predicted mask and the ground truth label. particularly in the uncertainty region defined by the may-be-floode class. We capture this property by computing the standard deviation a of the minimum of Euclidean distances .). between every pixel in the boundary of the predicted mask B and the boundary of the must-be be-oodde area B, which are computed by filtering the masks with the Sobel edge detector:
edge coherence = 1 - a min d(B., B.)/1l)
(8)
4.1.2 ABLATION STUDY
In order to assess the contribution of each the components described in Section 3.2 to the overall Masker performance, we performed ablation study by training 18 models, each with different
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
G
Pseudo labels
Depth (D)
Segmentation (S)
SPADE
DADA (S)
DADA (M)
Table 1: Summary of the ablation study of the Masker. Each numbered column corresponds to a trained model and black dots indicate which techniques (rows) were included in the model. The last two columns correspond to the baseline models: ground (G) segmentation from HRNet as flood mask and InstaGAN (I).

---

Published as a conference paper at ICLR 2022
Inpot
CycleGAN
MUNIT
InstaGAN
InstaGAN+Mask
Painter+ Ground
ClimareGAN
116
Figure 4: Example inferences of ClimateGAN and comparable approaches on three diverse street scenes from the test set. We can see that ClimateGAN is able to generate both realistic water texture and color, well as complex mask that surrounds objects such cars and buildings. Comparable approaches are often too destructive, producing artifacts in the buildings and sky (e.g. 1st row of MUNIT) or not destructive enough, resembling more rain on the ground than high floods (e.g. 3rd row of CycleGAN and 2nd row of InstaGAN).
analyzed the effect of including the fol- lowing components in the Masker architecture: training with pseudo labels, D, S, DADA for S, DADA for M, and SPADE for M. When not using SPADE, M is based on residual blocks followed by convolutional and upsampling layers, taking 2 as input, and we can also inform it with depth according to the DADA procedure just like we do for S.
combination of techniques. As illustrated in Table 1
We compared these variants of the Masker with two baselines: ground segmentation from HRNet pre-trained on Cityscapes instead of the flood mask (G) and InstaGAN (I). For each model variant in the ablation study (column in Table 1) we computed mask predictions for each of the 180 test set images and the values of the three metries: error, F05 and edge coherence. In order to determine whether each technique (row) improves the performance, we computed the difference between the metries on each image for every pair of models where the only difference the inclusion or exclu- sion of this technique. Finally, we carried out statistical inference through the percentile bootstrap method (Efron 1992) to obtain robust estimates of the performance differences and confidence inter- vals. In particular, we obtained one million bootstrap samples (via sampling with replacement) for each metric to get a distribution of the bootstrapped 20 % trimmed mean, which is robust measure of location Wilcox (20TT). We then compared the distribution against the null hypothesis, which indicates no difference in performance (see Fig. 5). We considered a technique to be beneficial for the Masker if its inclusion reduced the error rate. In case of inconclusive error rate results, we considered an increase in the F05 score and finally in the edge coherence.
4.2 COMPARABLES FOR HUMAN EVALUATION
While the nature of our task is specific to our project, we can nonetheless benchmark ClimateGAN against IIT models. In fact, in earlier iterations of our project, we leveraged the CycleGAN archi- tecture in order to achieve initial results (Schmidt et al. 2019), before adopting a more structured approach. Therefore, to be as comprehensive in our comparisons as possible, we trained the fol- lowing five models on the same data as the ClimateGAN Painter and used the same test set for the comparison: CycleGAN, MUNIT, InstaGAN, InstaGAN using the mask to constrain the trans- formation to only the masked area (similarly to Eq. (T). and the ClimateGAN Painter applied to ground segmentation masks predicted by HRNet. These models were trained using the original pa- pers' configurations, with haper-parameter tuning to be fair in our comparison. Some samples from the models can be seen in
Fig. 4 and further results provided in Appendix 1) Fig. 16
7

---

Published as a conference paper at ICLR 2022
PO5
Edge coberence
Pseudo Inbels
Depth (D)
Seg. (S)
SPADE
DADA (S)
(M)
-0.002
-0,001
0.001
20 % trimmed mean difference and hootstrapped confidence intervals
Figure 5: Statistical inference tests of the ablation study. Shaded areas indicate metric improvement. All techniques but DADA (M) significantly improved the error rate, and some further improved the P05 score and edge coherence.
5 RESULTS
This section presents the model evaluation results, including the Masker ablation study and the comparison of the overall proposed Loodel-Masker and Painter- Painter-against comparable approaches via human evaluation. Visual examples of the inferences of our model can be seen in Fig. 3
5.1 MASKER EVALUATION
The main conclusion of our ablation study is that five of the six techniques proposed to improve the quality of the Masker positively contribute to the performance. In Fig. 3] we show the median differences and confidence intervals obtained through the bootstrap. The error rate improved- 99 % confidence-iin in the models that included pseudo labels, a depth head, segmentation head, SPADE-basse M and DADA for the segmentation head. For some but not all techniques, the FO5 score and edge coherence also improved significantly. In contrast, we found that both the error and the F05 score were worse when DADA for the Masker was included.
Moddla
0.00
0.01
0.02
0.00
0.03
0.36
0.07
3:50
0.99
2.80
0.060
0.965
0,970
0.975
0. 0.900
0.503
0.500
Errur
PO5 mo0rs
Edge colerrene:
Figure 6: Evaluation metrics for subset of the models studied in the ablation study and presented in Table [ models trained without pseudo labels or with DADA for the Masker are excluded as well as the two baselines for comparison. We show the median of the distribution with bootstrapped 99 % confidence intervals. The shaded area highlights the best model.
In Fig. we show the Masker evaluation metrics for subset of the models, 1-7, selected upon the conclusions of the ablation study, as well as the two baseline models. As main conclusion, our proposed Masker largely outperforms the baselines for the three metrics, especially the error and the F05 score. Further, the metrics of the individual models support the findings of the ablation study, as the best performance-5.. 6 and 7-is achieved by the models that include all or most techniques: pseudo labels, depth and segmentation heads, DADA for 5 and SPADE. Surprisingly, model 2, which only includes pseudo labels and the depth head, also achieves high performance, In contrast, models 3 and 4, which include segmentation head without DADA or SPADE, obtain worse performance. Therefore, it seems that the contribution of the segmentation head is clearly complemented by using DADA and SPADE. A more complete analysis is provided in Appendix C The final architecture we selected for the Masker includes: pseudo labels, D, S, DADA for 5 and SPADE for M (model in Table 1).
terms of
8

---

Published as a conference paper at ICLR 2022
MUNIT
CycleGAN
Mask-InstaCAN
Painted
0.0
02
033
0.4
0.5
0.5
0.7
0.8
0.9
1.0
Figure 7: Results of the human evaluation: the blue bars indicate the rate of selection of Climate- GAN over each alternative. The error lines indicate 99 % confidence intervals.
5.2 HUMAN EVALUATION
So to compare ClimateGAN to related alternatives, we asked human participants select the image that looked more like an actual flood, given pairs in which one image was generated by ClimateGAN and the other by one of the models (Fig.g Participants chose ClimateGAN images most times in all cases Fig. 7). Further details of this study are provided in Appendix D
6 FUTURE WORK
An intuitive extension of our model would be to render floods at any chosen height. Despite the appeal of this approach, its development is compromised by data challenges. To our knowledge, there is no data set of metric height maps of street scenes, which would be necessary for converting relative depth and height maps into absolute ones. Moreover, simulated worlds-including our own-that have metric height maps do not cover a large enough range of scenes to train models that would generalize well on worldwide Google Street View images. Another promising direction for improvement would be to achieve multi-level flooding, controlling water level represented by mask, which faces the same challenges.
We also explored the integration of several multi-task learning strategies weigh the various Masker losses, including dynamic weight average (Liu et al.) (019) and weighting losses by un- certainty (Kendall al., 20188. However, we empirically found that our manual tuning of constant weights performs better than the aforementioned methods. In future work, we aim to explore other
techniques like gradient modulation methods (Yu et al. 2020 Maninis et 2019
CONCLUSION
In this research, we have proposed to leverage advances in modern generative modeling techniques to create visualizations of floods, in order to raise awareness about climate change and its dire consequences. fact, the algorithm described here has been incorporated into an interactive web- based tool that allows users input any address of their choice and visualize the consequences of extreme climate events like floods, while learning about the causes and impacts of climate change?)
Our contributions comprise data set of images and labels from 3D virtual world, novel archi- tecture ClimateGAN for the task of street-level flood generation, and a thorough evaluation method to measure the performance of our system. We found our proposed Masker-Painter dichotomy to be superior to existing comparable techniques of conditional generations two ways: first, we empir- ically showed that decoder with geometrical information from depth predictions and semantic information from seg- mentation maps, and by training the three decoders together. Second, we established, using human evaluations, that our Painter alone is able to create realistic water textures when provided ground masks, and that its performance increases even further when provided with the Masker's predic- tions. Overall, this allows ClimateGAN to produce compelling and robust visualizations of floods on a diverse of first-person views of urban, suburban and rural scenes.
able to produce more realistic flood masks by informing the flood mask
https:: i

---

Published as a conference paper at ICLR 2022
REFERENCES
Sun Joo Grace Ahn, Jeremy N. Bailenson, and Dooyeon Park. Short-and long-tene tern effects of em- bodied experiences immersive virtual environments on environmental locus of control and behavior. Computers in Human Behavior, 39, 2014.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. In ICIR, 2017.
Daniel A. Chapman, Adam Corner, Robin Webster, and Ezra M Markowitz. Climate visuals: A mixed methods investigation of public perceptions of climate images in three countries. Global Environmental Change, 41, 2016.
Liang- Chigg-Chihhhh Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 2016.
Francesco Dottori, Peter Salamon, Alessandra Bianchi, Lorenzo Alfieri, Feyera Aga Hirpa, and Luc Feyen. Development and evaluation of framework for global flood hazard mapping. Advances in Water Resources, 94, 2016. ISSN 0309- 1708.
Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics. Springer, 1992.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings. JMLR.org. 2015.
Jamie Herring, Matthew VanDyke, Glenn Cummins, and Forrest Melton. Communicating local climate risks online through an interactive data visualization. Environmental Communication, 11 (1), 2017.
Ove Hoegh-Guldberg, Daniela Jacob, M Bindi, S Brown, I Camilloni, A Diedhiou, R Djalante, K Ebi, Engelbrecht, Guiot, et al. Impacts of 1.5 e global warming on natural and human systems. Global warming of 1.5 C. An IPCC Special Report, 2018.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, adversarial domain adaptation. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüsssen, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018
and Trevor Darrell. Cycada:
Xun Huang, Ming-Yis Yu Liu, Serge Belongie, and Jan Kautz. Multimodal Unsupervised Image-to- Image Translation. vrXiv:1804.04732 [cs, stat], 2018. arXiv: 1804.04732.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolue resperressluttinn. arXiv preprint N.XY:16633301155 2016.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE Conference Computer Vision and Patlern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018.
Baharch Alizadeh Kharazi and Amir H Behzadan. Flood depth mapping in street photos with image processing and deep neural networks. Computers, Environment and Urban Systems, 88: 101628, 2021.
Scott A Kulp and Benjamin H Strauss. New elevation data triple estimates of global vulnerability to sea-level rise and coastal flooding. Nature communications, 10(1):1-12, 2019.
10

---

Published as a conference paper at ICLR 2022
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Carbon Emissions of Machine Learning. arXi::9910.09700 [cs], 2019. arXiv: 1910.09700.
Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datases transfer. CoRR, abs/1907. 0077001344 01341, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo- realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2017.
Xuelong Li, Kai Kou, and Bin Zhao. Weather gan: Multi-domain weather translation using genera- tive adversarial networks. arXiv preprint arXiv:2103.05422 2021.
S. W. Link and R. A. Heath. A sequential theory of psychological discrimination. Psychometrika, 40(1), Mar 1975. ISSN 1860-0980.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised taage-to-imagee translation networks. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multi- ple tasks. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1..1.1111111. 2014.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instance-aware image-to-image translation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 2017.
José P Ossandón, Selim Onat, and Peter König. Spatial biases in viewing behavior. Journal of vision, 14(2), February 2014. ISSN 1534-7362.
Taesung Park, Ming-Yr Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation/ / IEEE, 2019.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun Yan Zhu. Contrastive learning for condi- tional image synthesis. In ECCV, 2020.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, 2016.
Cem Sazara, Mecit Cetin, and Khan M Iftekharuddin. Detecting floodwater roadways from image data with handerafted features and deep transfer learning. In 2019 JEEE Intelligent Transportation Systems Conference (ITSC), pp. 804-809. IEFF, 2019.

---

Published as a conference paper at ICLR 2022
Victor Schmidt, Alexandra Luccioni, S. Karthik Mukkavilli, Narmada Balasooriya, Kris Sankaran, Jennifer Chayes, and Yoshua Bengio. Visualizing the Consequences of Climate Change Using Sycll-Consistent Adversarial Networks. arXiv:1905.03709 [cs], 2019. arXiv: 1905.03709.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. arXiv:1907.10597 2019.
Green Al. arXiv preprint
Verónica Sevillano, Juan I Aragonés, and P Wesley Schultz. Perspective taking, environmental concern and the moderating role of dispositional empathy. Environment and behavior, 39(5), 2007.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019. Association for Computational Linguistics.
Hao Tang, Dan Xu, Nicu Sebe, and Yan Yan. Attention-guided generative adversarial networks for unsupervised image-to-image translation. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.
CJ van Rijshergen. Information retrieval, 2nd edbutterworths, 1979.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Con- ference on Computer Vision and Patlern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation TEEE, 2019a.
Tuan- Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. DADA: depth- aware domain adaptation in semantic segmentation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019. IEFE, 2019b.
Patrick Wenzel, Qadeer Khan, Daniel Cremers, and Laura Leal-Taixé Modular vehicle control for transferring semantic information between weather conditions using gans. In Conference on Robot Learning, pp. 253-269. PMLR, 2018.
Rand R Wilcox. Introduction robust estimation and hypothesis testing. Academic press, 2011.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020.
Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li F Fei-Fei, and Michael Bern- stein. Hype: A benchmark for human eye perceptual evaluation of generative models. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEBE Computer Society, 2017.
12

---

Published as a conference paper at ICLR 2022
A SIMULATED DATA
In this section, we expand on Section 3.1 and provide details about our simulated world and the images and labels we can obtain from it.
We created a 1.5 km virtual world using the Unity3D game engine containing urban, suburban and rural areas.
The urban environment contains skyscrapers, large buildings, and roads, as well as objects such as traffic items and vehicles. Fig. 8 shows a bird's eye view of the urban area of virtual environment. The rural environment consists of a landscape of grassy hills, forests, and mountains, with sparse houses and other buildings such as church, and no roads. The rural and urban areas make up for k of our virtual world.
Figure 8: Bird's eye view the urban area (city) and rural area (outskirts of the city) of our simulated world
The suburban environment (Figure 9) is residential area of 0.5 km with many individual houses with front yards.
Figure 9: Bird's eye views of the suburban area of our simulated world
To gather the simulated dataset, captured 'before' and 'after' (without/with) flood pairs from 2000 viewpoints with the following modalities:
'before' non-flooded RGB image, depth map, segmentation map
'after" flooded RGB image, binary mask of the flooded area, segmentation map
The camera placed about 1.5m above ground, and has field of view of 120", and the resolution of the images 1200x900 At each viewpoint, we took 10 pictures, by varying slightly the position of the camera in order to augment the dataset.
13

---

Published as a conference paper at ICLR 2022
Fig. 10 shows the different modalities captured each viewpoint, and Fig. 11 shows samples of our simulated dataset in urban, suburban and rural areas.
Depth The depth maps provided RGB images for the 'before' (without flood) case, and the depth is recorded up to 1000m away from the camera, with precision of 4mm.
Segmentation
There are nine different classes of objects in the simulated world:
sky
ground: road, sidewalks, road markings, anything that is asphalt building
traffic item: lampposts, traffic signs, poles
vegetation: small bushes, trees, hedges excludes grass, lawns terrain: rocks, soil, lawns
car: cars and trucks
other: miscellancous objects such postboxes, trashcans, garbage bags, etc.
water: only present in the 'after' flooded images
While people are not included in the simulated world, the segmentation model able to learn this class from the real world due to the supervision signal given by the HRNet pseudo-labels.
Mask We also include binary masks of the flood (water segmentation) for the 'after' images. The masks are used to train the Masker with ground truth target flood masks in the simulated domain.
RGB
Segmentation
Mask
Depth
Figure 10: Sample data obtained at one spot for one camera position in our virtual world. The top row shows the modalities of the 'before' flood image: RGB image of the scene, depth map and segmentation map; and the bottom row shows those obtained in the 'after' configuration: RGB image, segmentation map and binary flood mask.
B LOSSES
For clarity, in Section 3.2 we did not detail the exact formulation of some of the losses since they are either straight- forward (as the Cross Entropy) direct applications of their definitions as per their original paper (e.g. the SSIMSE loss). We expand on those here. In particular, to guide decoders, we used pseudo labely from pre-trained models as supervision targets for real data. Because these labels are noisy, we limited this procedure to the first ten epochs of training.
B. DEPTH DECODER
In this section, we detail the depth decoder loss presented in Section 3.2 and adapted from (Lasinger al. 2019).
14

---

Published as a conference paper at ICLR 2022
Figure 11: Samples from our simulated dataset urban (top row), rural (middle row), and suburban (bottom row) areas.
Let a predicted disparity map and d' the corresponding ground truth. The aligned disparity maps with zero translation and unit scale are:
d d t-d(d) d" d' - t(d²)
s(d)
s(d²)
where and s are defined as t(d) median (d) and s(d) 1/2 En 1 ((d)-
The depth decoder loss is composed of a scale-and-shift invariant MSE loss term:
LSSIMSE
2 Em (ddm,,..)
[ - d
(10)
an a multi-scale gradient matching term to enforce smooth gradients and sharp discontinuities:
(n,h.a) + V R (n.h.)))
Lam = Em
V
(11)
h,w
where ((n.h.w)) k. (n, and R& corresponds to the difference of disparity (inverse depth) maps at scale Following (Lasinger al. 2019) consider scale levels, downsampling by factor two the image at each scale.
B.2 SEGMENTATION DECODER
This decoder computes the segmentation map s; given an input image æ;, s; being 4D tensor of shape N C x H x W. The number of channels C' corresponds to the number of classes, nine in our case: ground, building, traffic item, vegetation, terrain, car, sky, person and other.
First, we detail the two fusion mechanisms used in the DADA approach (Vu et al. 2019b):
Feature fusion: is the element-wing wise multiplication between the latent vector and a depth vector of the same size. This depth vector is obtained by convolutional layer applied to the depth head before the average pooling. The fused features are given as input to the segmentation decoder.
15

---

Published as a conference paper at ICLR 2022
DADA fusion:
Instead of giving the self-information map 1 the AdvEnt (Vu et al. 2019a) discriminator QS, we give the element-wise multiplication between 7/ and the depth predictions d, The obtained matrix is called the depth-aware map 1
We also detail all segmentation losses that are not defined in the main paper. First, the cross-entropy loss is defined as:
Ly 8:)==
= - En.,, (nihhw)
w log s (m.oA..)))
(12)
Furthermore, the entropy minimization loss, which is used as suggested by the authors of AD- VENT et al. 2019a), is computed according to the following equation:
(EM( En.cch.ul-ss. log 8,]
(13)
Finally, we detail the WGAN loss that is computed using discriminator Qs which outputs the probability of the depth-aware map coming from the real domain. If updating the decoder S, the loss tries to fool the discriminator:
LG-CAN(Q I(s,), i(s.)) -- (I..0)
(14)
updating D, the loss tries improve the discriminator's prediction:
LD-GAN(QS, [(s,), - (((s))
(15)
B.3 FLOOD MASK DECODER
Similarly to the segmentation decoder, we detail in this section the WGAN loss that is computed using a discriminator Qd in order to train M. This discriminator outputs the probability of the self- information map / (m,, coming from the real domain. When updating the decoder M, the loss tries to fool the discriminator:
LG GG-GAN(Q" , [(m.)) = =- E,, (I(m,))
((m.)
(16)
When updating QM, the loss tries to improve the discriminator's predictions:
CA-CAN(Q ((m.))((m)))
(17)
-E,(Q" (I(m.)) - Q² (T(m,)))
C MASKER EVALUATION
In Section 4.1 we introduced the method for evaluating the quality of Masker's output. Here, we provide details of the annotation procedure for generating Masker (Appendix C.I) and the metrics proposed to assess the performance (Appendix C.2).
C.I TEST SET COLLECTION AND ANNOTATION
In this section, we provide more details about our test set data collection procedure, and the annota- tion guidelines we followed.
We collected the 180 images of our test from Google Street View with the following selection guidelines:
Geographical diversity: we selected images in a variety of cities on all continents, with different architectural styles and general landscapes.
16

---

Published as a conference paper at ICLR 2022
Varied levels of urban development: both urban and rural areas were included. Variety of scenes content: we endeavored to cover wide range of objects in our test set, including images containing bike racks, dense crowds, and various vehicles in urban areas, and different vegetation types in rural areas.
We also carefully collected challenging images to determine the limitations of our model, including images facing slopes (going up or down), or stairs, with various ground textures, around areas under construction, and in areas near canals among others. The images were manually annotated, and the pixels of each image were categorized in one of three classes according to the following instructions:
Must be flooded: This class contains the minimal region that should be flooded. We want to represent floods of height at least 0.5 m. Typically, this corresponds to flooding up to the knees of adult pedestrians or up the top cars' wheels. In cases when such reference objects were available, the annotator would make the 0.5m estimate based on other cues, such as doorsteps, traffic signs and vegetation.
Cannot be flooded: This label indicates regions absolutely do We generally put any pixel corresponding to an object with a height greater than 1.5 m above ground in this category. This includes car roofs and adult pedestrians' heads, which we used as reference to determine the lower limit of this region.
be flooded.
May be flooded: This category contains any pixel not assigned to the other classes. reflects the fact that do not enforce flooding at specifie height yet value plausible flood location.
We show examples of labeled test images Fig.
Must be flooee
May be Gloded
Cannot be flooded
Figure 12: Examples of labeled images from test set.
17

---

Published as a conference paper at ICLR 2022
C.2 METRICS
In Section 4.1.1 we proposed three metrics to compare the masks predicted by the Masker and the labeled test images: error, FOS score and edge coherence (Eq. (8) Here, we delve into the reasons why we proposed more than one metric and the contribution of each metric.
We proposed the error-number of erroncously predicted pixels divided by the size of the image-as the main evaluation metric, since it characterizes the size of the errors in the image, which directly impacts the perceived mask quality.
However, we argue that the error alone does not capture all aspects of the performance of a Masker model the test set. For example, the error does not take into account the size of the labeled areas (see image "D" in Fig. 19). While the size of the labels may have smaller perceptual impact, in order to characterize the precision and sensitivity of the model, additional metrics are useful. We proposed the FO5 score:
1.25 x precision x recall
1.25
TP+FP X TP4FN
F05
(18)
0.25 precision recoll
0.25 TP4FP + TP4FN
which computes the weighted harmonic mean of precision and recall also known sensitivity or true positive rate. We used 3 = 0.5 order weigh precision more than recall, that penalize more false positives than false negatives. In our context, this translates into setting higher penalty pixels-foo instance heads of pedestrians, automobile roofs and high areas of the image, in general- than for missing areas that should be flooded. While both types of errors should penalized, the former has a higher perceptual impact.
for flooding cannot-be-floode
1.000
A, C, E
0.975
0.98
0.950
0.96
0.925
0.94
Edge coherence
F05 score
0.900
0.92
0.875
0.90
0.850
0.88
0.825
F
0.86
0.800
0.84
0.00
0.02
0.04
0.06
0.08
0.10
0.12
Error
Figure 13: Distribution of the three metrics for the selected best Masker. The annotations correspond to the images in Fig. 19
Finally, we proposed an edge coherence metric (Eq. (8) in order to take into consideration the shape of the predicted mask, with respect to the shape of the blloode--llooed label. Note that neither the enoor or the FO5 score account for the shape, as only the amount of correct or incorrect pixels matter, regardless of the position. As explained in the previous section, we defined may-be-flooded class in order to allow for different levels of flooding in the mask prediction. That is, higher levels of flooding in a prediction should not be penalized necessarily, as long as the mask is consistent with the semantics in the image. Our proposed metric of edge coherence is based on the assumption that predictions whose border is roughly parallel to the border of the must- be flooldd label should be less penalized than highly dissimilar shapes.
Fig. 13 shows the value of the three metrics for all the images in the test set. While there is certain correlation between the metrics, especially because the bulk of the distributions is around images for
18

---

Published as a conference paper at ICLR 2022
Error
P
Pscudo labels
-6.9 10 [-9 10
4.5
10
0.0
Depth
x
-1.4x110
10
<0.01
Seg. (S)
-3.6 10
-
<0.0001
SPADE
-2.6 x2.6xx100 [-..7 10 -5.1
10
<0.01
DADA (S)
-3.5 [- 10
-1.7x -
10-
0.0
DADA (M)
+2.1 10-3 [+1.5 x 10 +2.9
10-3] 0.0
Table 2: Details of the results of the ablation study. The values in each cell are the 20 % trimmed mean error difference between models with and without given technique, in brackets the 99 % confidence intervals, and the p value.
FO5 score
Pseudo labels
+7.4 x 10-4 [+4.3 10
+1.1x
10-3]
0.0
Depth
-3.9 x 1-01
10-9
<0.1
Seg. (S)
+2.9 (H12
+4.1
<0.001
SPADE
10-4
0001
DADA (S)
+3.8 x 10 [+1.3 10
+6.5 10
<0,0001
DADA (M)
-2.6 x 10-3 [-3.5 X 10 -1. 30
0.0
Table 3: The analogue to Table for the F05 score
which the predictions are very accurate and hence all the metrics are near perfect, the plot shows that various metrics are useful for identifying few images for which not all metrics are low. We further illustrate the meaning of our proposed metrics Fig. 19 where we show images that obtained the lowest-2ne quantile and highest-999 quantile -values of each metric.
C.3 ABLATION STUDY
Here, we detail the methodology used for the ablation study of the Masker, presented in Sec- tion 4.1.2. well extend the set of results provided in Section 3.1
In the ablation study, we studied the contribution to the Masker evaluation metrics each technique: training with pseudo labels, depth head (D), a segmentation head (S), SPADE, DADA for the segmentation head and DADA for the masker head. For every technique I, we considered all models m. which included such technique, and the paired models m which differed only by the absence of't. Then, for every metric we constructed datasets of metric differences for every image i:
di = rmm)) - r(m)")) -
(19)
and obtained I million bootstrapped samples. On each bootstrap sample computed the 20 % trimmed mean, which forms a bootstrap distribution from which we derived the confidence intervals
Edge coherence
p
Pseudo labels
+3.7 x [+1.2 10 10
10
<0.01
Depth
-1.4
=.0
Seg. (S)
+2.8 x [=11
>0.1
SPADE
-9.3 x
= 0.0
DADA (S)
-1.0 10- 10-4
0001
DADA (M)
+4.6 x 10 1-1 10- 10
10-31
0.01
Table 4: The analogue to Table for the edge coherence
19

---

Published as a conference paper at ICLR 2022
99 confidence interval
value / 0.0019
2300
Null hypothesis 0.0
2000
Dereity
Bootstrap mean 0.0004
1,500
5D 0.0001 SE
1000
0.0000
0.0007
0.0000
Bootstrap estimates
Figure 14: Bootstrapped distribution of the 20 % trimmed means of the difference in edge coherence between models that included pseudo labels and their counterparts. Equivalent distributions were obtained for all other techniques and metrics in the ablation study.
340000
13
12
15
17
5.000
0.011
0.056
0.003
0.000
0.012
0.800
0.900
0.992
0.991
0.995
0.580
0.974 0.072
0.972
0.980 0.982 0,904 0.905 0.503
Error
FOG 5c000
Edge colieressoe
Figure 15: Distribution of the metrics-entt FO5 score and edge coherence of the complete set of models tested in the ablation study, 1-18, excluding the two baselines, whose distributions are shown in Fig. 6 The solid symbols indicate the median of the distribution and the error lines the bootstrapped 99 % confidence intervals. The shaded area highlights the best model-7.
we reported graphically in Fig. 3 In particular, we computed the 99 % confidence intervals, that the lower bound is the 0.5th quantile and the upper bound is the 99.5th quantile of the bootstrap distribution. In Table 2] we provide the numerical details of all the tests in the ablation study. One advantage of the bootstrap driven distribution rather than a binary test. In order to illustrate this, we provide the distribution for one of the techniques- pseudo labels-and of the metrics-edge coherence-in Fig. 14
other statistical inference methods is that the outcome is data-
In Fig. 15
we extend Fig. where we show the distribution of the metrics for the complete set of models in the ablation study. We do not include the baseline models, which obtained significantly worse metrics, in order to better visualize the differences within the models of the ablation study. From the figure, it becomes apparent that models trained without pseudo labels (9-18), and models trained with DADA for the masker head (8, 9, 17, 18) achieved worse performance, in general. This consistent with the results of the ablation study obtained through the bootstrap.
D HUMAN EVALUATION RESULTS
147 evaluators took part in the study (done via the Mechanical Turk platform) and we gathered 2700 total evaluations, 3 for each pair of images. For each image generated by ClimateGAN, com-
20

---

Published as a conference paper at ICLR 2022
pared it to 5 other approaches (MUNIT, InstaGAN, CycleGAN, InstaGAN with Mask and Painted Ground), and each one of these pairs was seen by evaluators.
During the human evaluation of our results, we found that overall, evaluators primarily preferred ClimateGAN to other comparable approaches to varying degrees (a complete presentation of results can be found in Section 5.2). However, there were number of cases when evaluators preferred other approaches to ClimateGANN. We present such images in Fig. 16 where the images outlined in red are those that are those for which all evaluators preferred comparable approach ClimateGAN.
It can be observed that it is hard to consider these approaches as being better than ClimateGAN i.e,, they do not appear to be more realistic neither in terms of the quality of generated water nor in the portion of the image that is flooded. However, the concept of realism and quality of generated imagery is, indeed, hard to define and quantify, tematically preferred by evaluators. In order to carry out more thorough human evaluation of our results, would be necessary to have more evaluators per pair of images and, ideally, different ex- perimental setups for instance, different captions and different designs of the comparison interface. Overall, is hard to draw a conclusion for the motivation behind the preferences of individual eval- uators. It can nonetheless be observed that in the case of some models, such as InstaGAN, images of light flooding (i.c. that after moderate rain) were preferred to that of more severe flooding generated by ClimateGAN. In other cases, such that of InstaGAN+Mask, images with more pronounced reflections were chosen over those of the murkier, rippled water produced by ClimateGAAN
unsurprising that some images were sys-
Finally, we feel that the best evaluation of the impact of images generated by ClimateGANN is to compare them with other mediums of climate communication. For this purpose, we are working with a group of researchers in psychology test the effect that our images have on climate risk perception. The experimental setup involves providing subjects with a text regarding the risks of climate change-induced flooding. One group of subjects will only see the text, second will see the text accompanied by an image of generic flooded house, whereas the third will see an image of their address of residence flooded using ClimateGAN. Preliminary results have indicated that individuals who saw their own place of residence flooded are more likely to perceive climate change as real threat to their livelihood, and more likely to take action to fight it.
E MORE DISCUSSION
E. FAILURE CASES
The Painter seems to systematically generate realistic water textures, successfully incorporating the context of the scene such as reflections of buildings on the water, provided the mask is reasonably accurate, In other words, we have not identified failure patterns of the Painter, specifically. What's more, the output of the Painter often mitigates small failures in the accuracy of the mask: even when the mask is not perfect, the image output by the Painter may look perceptually realistic. The main source of failures, where the final image does not look perceptually realistic, is instead the Masker. This is expected given the more challenging nature of the task -eeeeiciii a mask for a hypothetical flood, taking into account the geometry of the scene, semantics, etc. The following the kinds of images where we have found the Masker to perform sub-optimally at times:
Images where the sky or the ground are not visible
Images with a large object, such as a car or a truck, taking a large fraction of the foreground
Wide open views of grass fields the edges of the water mask are sometimes incorrect
Scenes with multiple people in the scene the edges of the water mask around all objects are sometimes inaccurate, although the perceptual error is mitigated by the Painter
Scenes with much vegetation, such as bushes
Urban scenes with a pronounced slope
Surprisingly, the painted image sometimes looks realistic in these cases, despite an inaccurate Mask. We hypothesize that the reason for lower accuracy of the Masker in these cases is the lack of such images in the training set, beside the additional difficulty intrinsic to some of these images, even to human annotator.
21

---

Published as a conference paper at ICLR 2022
Input
CycleGAN
MUNIT
InstaGAN
Figure 16: Example inferences of ClimateGAN and comparable approaches on images from the test set. Images outlined in red are those that were systematically (i.e. 3 out of times) preferred over ClimateGAN.
22

---

Published as a conference paper at ICLR 2022
E.2 END-TO-END ARCHITECTURE
We studied the possibility to train the Masker and the Painter jointly. One major constraint to it is that of the Painter's losses (Feature Matching Loss and Perceptual Loss) require the input data and generated data to be compared, and should therefore come from the same "category": flooded scenes. In other words, both those losses compare the input and the output to encourage the Painter to produce more realistic water. On the other hand, the Masker never processes such data: it produces masks from non-flooded images. Thus, we tried the following approaches:
Remove the two aforementioned losses from the Painter's training procedure, and train it jointly with the masker from non-flooded data
(zx) = P(e, (1 - Masker(z) @ x) @ Masker(z) + x @ (1 Masker(z))
(20)
As we expected, this procedure not only makes training the masker more difficult but we could not make the Painter to produce realistic water.
Keep the two losses, train the Painter and Masker from different data sources (flooded/non- flooded) but use the loss from the Discriminator of the Painter train the Masker. In other words, train P and M as described in the main body but add loss (P is frozen here) LGAND (Dp(ū(x)) where LGAN, is a GAN loss (cross-entropy or L2 for instance), and ý is defined above. Since the Discriminator Dr in the Painter's adversarial training procedure is supposed to assess how "realistic" its output is, then this takes into account not only how the water is painted, but also where it is. Unfortunately this proved to be very challenging in terms of computational complexity, memory footprint and most of all, convergence. We did not find proper scheduling/scaling mechanism for LGAN, to be informative enough to improve the Masker's performance.
E.3 CLIMATEGAN VS ALL EVALUATION
In the human evaluation section, readers may be interested in 1-vs-all comparisons, comparing Cli- mateGAN outputs to all comparable methods at once. However, such a procedure would introduce methodological complications, both in the statistical tools at hand and because of the biases known to affect human evaluators when performing multi-way comparisons. For instance, laterality is well known for largely impacting visual inspection (Ossandón et al.. 2014) and two-alternative forced choices is known be an easier and more reliable task for human evaluators, which is why it is the common approach to measure sensitivity to stimuli or preferences in the cognitive sciences (Link & Heath 1975). In addition, recent research in GAN evaluations also recommends making pairwise comparisons in order to measure user preference (Zhou et al./ (019). Furthermore, in our case we are mainly interested in knowing whether the images produced by our model are preferred over other alternatives, rather than comparing these alternatives against each other.
F CARBON IMPACT
The environmental impact of machine learning is becoming an increasingly major issue for our field, given the extensive experimentation and hyper-parameterr tuning required to successfully train large neural networks (Strubell al.] 2019 Schwartz et al. 2019 Lacoste al., 2019). In order to estimate our own carbon footprint, we counted all of the GPU hours (weighted by GPU usage) that entire team used during the whole course of our project, from initial stages to model selection to the ablation study and prrer-parameter tuning on our final ClimateGAN model. We used the MLCO, Emissions Calculato (Lacoste al. 2019) obtain a final estimate of 362.72 kilograms of CO.eq.. which is comparable to 900 miles driven by an average passenger vehicle, 42% of a US household's yearly energy consumption or round trips between Paris, France and Saint-Petersburg. Russia (369kg COyeq.) or between Boston, MA and Miami, FL (345kg CO2eq) This does not include the rest of the computational infrastructure's 's energy consumption (CPUs, data transfers and storage etc.) nor the full Life Cycle Analysis of any of the hardware used.
iibbco,.iihuub.oiiaacc
SSurce: EPA
"Source: ICAO
23

---

Published as a conference paper at ICLR 2022
We incredibly lucky that power grid is powered predominantly by renewable energy. If this were not the case and we were using coal-powered energy, the figure stated above would be 50-80 times larger, and thereby more problematie. We hope that our colleagues will also start tallying and sharing the carhon footprint their research using tools such as CodeCCrbonn and the Experiment Impact Tracker and that community will start being mindful regarding the trade-off between scientific progress and environmental impact.
G FULL CLIMATEGAN ARCHITECTURE
In order to better illustrate the overall training procedure of ClimateeAN, we provide more detailed overview in Fig. 17
The Masker can be seen in the top part of the Figure, encompassing the three decoders described in(Section 3.2] Depth (D) Segmentation (S) and Flood-Mask (M). The Painter shown the lower part of the image, with its SPADE-basee encoder P. The losses of each component are indicated in the rounded white boxes within each decoder. Tensors are represented using squares, with different colors for input, label, intermediate and output tensors, and pseudo-labelt with dotted outlines. To emphasize the SPADE-based conditional architectures of M and P, they are colored in green. As per Fig. 2 the output of ClimateGAN, i.e. the flooded image y. is in blue. Note that the dotted and dashed line from m, to P is not leveraged during training and only used for test-time inferences. Discriminators conceptually included in the WGAN losses.
H SUPPLEMENTARY IMAGES
In order to further illustrate the performance and capabilities of ClimateGAN, provide additional inferences in Fig. 18] complementing those presented in Fig. 3 The first two rows were chosen to be successful inferences, middle two were selected at random and bottom two failure cases.
24

---

Published as a conference paper at ICLR 2022
Masker
d
Depth Decoder
X
SSIMSE GM
E
D
SSIMSE GM
CB
Segmentattion Decoder
WGAN
S
EM
m
Flood- WloddMMsk Decoder
EM
M
m
WGAN
Test-time
inference only
GI
Painter
y
WGAN
P
y
VGG
FeatMatch
Unconditional
SPADE-based conditional network
Tensors: Input, Label, Intermediate, Output
convolutional network
Loss from pseudo-labels
Pseudo Labels
Loss
Figure 17: Detailed diagram of the training procedure of ClimateGAN, We highlight how the sim- ulated and real data paths in the model are similar yet different. We use dashed arrows and boxes to emphasize that pseudo labels are only used as noisy signal in the beginning of training. All the losses represented in the white rounded boxes are detailed in Section 3.2 and Appendix B.
25

---

Published as a conference paper at ICLR 2022
Input
Depth
Segmentation
Mask
Masked Input
Painted Input
Figure 18: More samples of the full ClimateGAN forward pass: input image, inferences from the depth, segmentation and flood mask decoders of the Masker, followed by the masked input image fed to the Painter and finally the flooded image output by the Painter. Notably, the Painter is able to produce consistently contextualized water, with color and reflections relevant to the surrounding objects and to the sky. While generally able to appropriately understand a scene's perspective and circumvent objects, the Masker however sometimes unable to predict plausible flood masks (as illustrated the bottom two rows). difficult to exactly understand the source of this because both the depth and segmentation maps look generally appropriate.
26

---

Published as a conference paper at ICLR 2022
Labels
Prediction
Metric
Low error rate TP
A
May-be-foodee
FO5: 1.0000
Edge coberence: 0.9662
High error rate
TN
B
May-be-flodeed
Error: 0.1138
FO5: 0.8856
Edge coherence:
High score
TN
C
Error: 0.0000
FO5; 1.0000
Edge coberence: 0.9832
Low FO5 score
TP
FP fleeee
D
0.0055
FO5: 0.8589
Edge coherence: 0.8859
High edge coherence
TP
Mast-be flooded
E
Error: 0.0017
0.9992
Edge coherence: 0.9960
Low edge coherence
Prediction
Mest-be-floded
F
Error: 0.0516
FO5: 0,7971
Edge coherence: 0.8504
Figure 19: Examples of images that obtained good and bad-within the 2nd and 98th quantiles, respectively-Maskee Masker's predictions metries. From top two rows bottom two: error, F05 score and edge coherence. The first row of each metric corresponds to image with good values of the metric. The white segments in the images illustrating the edge coherence indicate the shortest distance between the predicted and the ground truth mask. The legend of the column "Labels" is the same as Fig. 12 and the images can be identified in Fig. 13 the letters on the left.
27


Reviews:
[{"review_id": "P7RtGnfTrV1", "reviewer": "Reviewer_TBDz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ethical and societal motivation of the paper is clear and commendable.\n\nI appreciate that the images generated from the virtual world with and without flooding are available, but I do not understand why the authors do not make available the real dataset. I understand that it contains images from Cityscapes and Mapillary, but it also contains new real images that the authors collected (as described in the “real data” section). I encourage the authors to make this other dataset available.\n\nThe main weakness of the paper is the limited technical contribution. The proposed ClimateGAN is given by combining two already existing models (DADA and GauGAN), with minor modifications. In section 3.4, the authors state that “A naive use of DADA for our task would miss a crucial difference” and “contribution of our Masker’s architecture is therefore to structure it as a multi-headed network”, but it is not clear to me if and how the authors have modified the architecture of DADA. By looking at Vu et al, 2019b, it seems to me that the architecture of the Masker is the same as DADA, apart from the last SPADE block “M” added after DADA.\n\nI appreciate that the authors report in Figure 3 failure cases, but I it would be nice if the authors could comment more on such failures. Do they happen when the input image has a particular geometry or perspective? I would also include more failure cases in the paper (like the two in figure 18 in the appendix), where the Painter also fails at producing plausible flooding.\n\nIn the paper, the Masker and the Painter are trained independently and only combined at test-time. It would be interesting to see what happens if the Masker and the Painter are trained jointly and compare this to the results in the paper.\n\nIn section 4.1.2, the authors state “We considered a technique to be beneficial for the Masker if its inclusion reduced the error rate. In case of inconclusive error rate results, we considered an increase in the F05 score and finally in the edge coherence”. Why is this hierarchy considered for the three metrics? I think that an explanation is required.\n\nThe evaluation and results section are very dense in terms of information and their structure does not help the reader in grasping the key points. I would suggest to restructure them and to provide a bullet lists of the main takeaways/findings from the quantitative and qualitative analysis.\n\n\nHuman evaluation\n\nImportant details are missing about the set up of the human evaluation. The most important I would say is the number of evaluators. From the appendix, it seems that only three people took part in the evaluation. If this is the case, I believe that the results would not represent an accurate estimate of the performance comparison between the ClimateGAn and the other models.\n\nAbout the “human evaluation” section it seems that ClimateGAN was compared against one model at a time. It would be interesting to compare all the considered models together and see how many times ClimateGAN produces the most realistic images among all models.\n\nIn the conclusion, the authors state that “we established, using human evaluations,  that our Painter alone is able to create realistic water textures when provided ground masks, and that its performance increases even further when provided with the Masker’s predictions”. However, I could not find such evaluation. From the paper, it seems that the human evaluation is for the Masker+Painter, not for the painter alone.\n\n\nMINOR COMMENTS\n-\tSpecify what the lambdas in the loss function represent.\n-\tIn the “image-to-image translation” section, this sentence lacks the verb “CycleGAN (Zhu et al., 2017) this constraint. Also, what is the first category of IIT? Please clarify it in the text.\n-\tIn the “conditional image synthesis”, I would add a reference where GauGAN is first mentioned.\n-\tI think it would be interesting to know what is the number of new real images that you were able to collect (apart from the ones taken from Cityscapes and Mapillary)\n-\tIf my understanding is correct, the “comparables” section refers to other models used for comparison in the human evaluation section. I think it should be clearly stated in the section. \n\n\n", "summary_of_the_paper": "In this paper, the authors introduce ClimateGAN, a framework for the generation of images of flooded scenarios in order to raise climate change awareness and prompt action. In particular, the authors consider the realistic case of scarcity of training data and propose and unsupervised approach. An evaluation of each ClimateGAN components is performed, both quantitative (through metrics) and qualitative (through human feedback). ClimateGAN is also compared against other generative modeling frameworks.\n", "summary_of_the_review": "I believe that the purpose of the proposed model is topical and of interest. However, from a technical viewpoint, I believe that the contribution is not novel enough for acceptance at ICLR as ClimateGAN is given by combining two already existing models (DADA and GauGAN) with minor modifications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GhY6b6PJInG", "reviewer": "Reviewer_PcPf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - This is a good application by leveraging the cutting-edge GAN-related techniques on an interesting area of generating flooded images.\n - The two-stage design with a Masker and a Painter is useful for synthesizing floods on images.\n\nWeaknesses:\n - The motivation of this work is described confusingly. First, the meaning of generating the flooded images is not well explained. Second, generating flooded images is not well related to the climate change. Thus, this work seems not that meaningful for climate change.\n - The main contribution of proposed ClimateGAN is the two-stage design for synthesizing floods on images, but each stage (Masker/Painter)  is designed with limited novelty based on some popular models and losses (refer to the ablation study in Section 4.1.2), making the whole work is not strong enough.\n - The comparison is not convincing. First, to evaluate the whole ClimateGAN, it's better to consider the state-of-the-art image-to-image translation methods, such as DRIT++, GauGAN, TSIT, DUNIT, etc. Second, to evaluate the Masker for flooded mask estimation, it's better to consider the state-of-the-art methods on both image-to-image translation and image segmentation (or maybe other related techniques). Besides, the data for training and testing should also be well considered for fair comparison.\n - If the two-stage design is the most important concept of this work, this should be well demonstrated with more insightful analysis theoretically and empirically.", "summary_of_the_paper": "This paper proposes to generate images of floods, that is, simulate photo-realistic floods, with a model named ClimateGAN, which consists of a Masker to generate masks for floods and a Painter to draw floods on images. The proposed ClimateGAN is a two-stage solution for generating flood masks on input images and painting the images with floods respectively, and each stage (Masker and Painter) is designed based on many popular related techniques, like DADA for depth-aware semantic segmentation, SPADE for injecting extra information into cGANs, and WGAN as well as TV, BCE and EM losses. For training ClimateGAN, this work also collects a real dataset with street-level floods and generates a simulated dataset using Unity3D. The outputs of Masker and Painter are evaluated separately compared to several previous image-to-image translation methods, also the ablation study is conducted to validate the effectiveness of each component in ClimateGAN.", "summary_of_the_review": "The paper can be regarded as a particular application for synthesizing flooded images with street-level cityscape images, however, since this work is not well motivated, yet the contribution and comparison are not strong enough, overall, I vote for rejecting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0zeownr6Zp", "reviewer": "Reviewer_6bHT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\nThe creation of a simulated flood dataset\nThe use of segmentation \"Masker\" conditioned on the depth of an image is something I haven't seen before\n\nThe qualitative results appear improved over other methods, and there is a positive human evaluation also \n\nNegatives\n\nthe system is a collection of black boxes, depth estimation, segmentation estimation etc\n\nLimited to a single depth of flood value (although this is discussed in future work)", "summary_of_the_paper": "This topical paper claims to raise awareness of climate change by projecting flooding images of popular places. To achieve this, they use a GAN based image generator placing water on street view style images.", "summary_of_the_review": "This paper is different  and topical, with still has a contribution to the research community and hopefully will provoke discussions", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims at raising awareness of climate change by GAN-projecting flooding images of popular places. This is an interesting case. While all reviews agree that this is an interesting direction, they also value the contributions differently. Two of them would like to see more methodological contributions, two focus more on the contribution to the (psychological) fight of climate change. Nevertheless, the rolling discussion helped to clarify several of the issues raised by the reviewers, and (in my opinion) "combining" existing methods to realize an important model that is one little step towards making people more aware how climate change may impact their own lifes is highly creative and useful. I therefor overall suggest strongly to accept the paper. As the ICLR CfP reads, ICLR is not just about methodological contributions. Societal considerations of representation learning are explicitly mentioned.
Decision: Accept (Poster)