Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This topical paper claims to raise awareness of climate change by projecting flooding images of popular places. To achieve this, they use a GAN based image generator placing water on street view style images.
main_review: Positives
The creation of a simulated flood dataset
The use of segmentation "Masker" conditioned on the depth of an image is something I haven't seen before

The qualitative results appear improved over other methods, and there is a positive human evaluation also 

Negatives

the system is a collection of black boxes, depth estimation, segmentation estimation etc

Limited to a single depth of flood value (although this is discussed in future work)
summary_of_the_review: This paper is different  and topical, with still has a contribution to the research community and hopefully will provoke discussions
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to generate images of floods, that is, simulate photo-realistic floods, with a model named ClimateGAN, which consists of a Masker to generate masks for floods and a Painter to draw floods on images. The proposed ClimateGAN is a two-stage solution for generating flood masks on input images and painting the images with floods respectively, and each stage (Masker and Painter) is designed based on many popular related techniques, like DADA for depth-aware semantic segmentation, SPADE for injecting extra information into cGANs, and WGAN as well as TV, BCE and EM losses. For training ClimateGAN, this work also collects a real dataset with street-level floods and generates a simulated dataset using Unity3D. The outputs of Masker and Painter are evaluated separately compared to several previous image-to-image translation methods, also the ablation study is conducted to validate the effectiveness of each component in ClimateGAN.
main_review: Strengths:
 - This is a good application by leveraging the cutting-edge GAN-related techniques on an interesting area of generating flooded images.
 - The two-stage design with a Masker and a Painter is useful for synthesizing floods on images.

Weaknesses:
 - The motivation of this work is described confusingly. First, the meaning of generating the flooded images is not well explained. Second, generating flooded images is not well related to the climate change. Thus, this work seems not that meaningful for climate change.
 - The main contribution of proposed ClimateGAN is the two-stage design for synthesizing floods on images, but each stage (Masker/Painter)  is designed with limited novelty based on some popular models and losses (refer to the ablation study in Section 4.1.2), making the whole work is not strong enough.
 - The comparison is not convincing. First, to evaluate the whole ClimateGAN, it's better to consider the state-of-the-art image-to-image translation methods, such as DRIT++, GauGAN, TSIT, DUNIT, etc. Second, to evaluate the Masker for flooded mask estimation, it's better to consider the state-of-the-art methods on both image-to-image translation and image segmentation (or maybe other related techniques). Besides, the data for training and testing should also be well considered for fair comparison.
 - If the two-stage design is the most important concept of this work, this should be well demonstrated with more insightful analysis theoretically and empirically.
summary_of_the_review: The paper can be regarded as a particular application for synthesizing flooded images with street-level cityscape images, however, since this work is not well motivated, yet the contribution and comparison are not strong enough, overall, I vote for rejecting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce ClimateGAN, a framework for the generation of images of flooded scenarios in order to raise climate change awareness and prompt action. In particular, the authors consider the realistic case of scarcity of training data and propose and unsupervised approach. An evaluation of each ClimateGAN components is performed, both quantitative (through metrics) and qualitative (through human feedback). ClimateGAN is also compared against other generative modeling frameworks.

main_review: The ethical and societal motivation of the paper is clear and commendable.

I appreciate that the images generated from the virtual world with and without flooding are available, but I do not understand why the authors do not make available the real dataset. I understand that it contains images from Cityscapes and Mapillary, but it also contains new real images that the authors collected (as described in the “real data” section). I encourage the authors to make this other dataset available.

The main weakness of the paper is the limited technical contribution. The proposed ClimateGAN is given by combining two already existing models (DADA and GauGAN), with minor modifications. In section 3.4, the authors state that “A naive use of DADA for our task would miss a crucial difference” and “contribution of our Masker’s architecture is therefore to structure it as a multi-headed network”, but it is not clear to me if and how the authors have modified the architecture of DADA. By looking at Vu et al, 2019b, it seems to me that the architecture of the Masker is the same as DADA, apart from the last SPADE block “M” added after DADA.

I appreciate that the authors report in Figure 3 failure cases, but I it would be nice if the authors could comment more on such failures. Do they happen when the input image has a particular geometry or perspective? I would also include more failure cases in the paper (like the two in figure 18 in the appendix), where the Painter also fails at producing plausible flooding.

In the paper, the Masker and the Painter are trained independently and only combined at test-time. It would be interesting to see what happens if the Masker and the Painter are trained jointly and compare this to the results in the paper.

In section 4.1.2, the authors state “We considered a technique to be beneficial for the Masker if its inclusion reduced the error rate. In case of inconclusive error rate results, we considered an increase in the F05 score and finally in the edge coherence”. Why is this hierarchy considered for the three metrics? I think that an explanation is required.

The evaluation and results section are very dense in terms of information and their structure does not help the reader in grasping the key points. I would suggest to restructure them and to provide a bullet lists of the main takeaways/findings from the quantitative and qualitative analysis.


Human evaluation

Important details are missing about the set up of the human evaluation. The most important I would say is the number of evaluators. From the appendix, it seems that only three people took part in the evaluation. If this is the case, I believe that the results would not represent an accurate estimate of the performance comparison between the ClimateGAn and the other models.

About the “human evaluation” section it seems that ClimateGAN was compared against one model at a time. It would be interesting to compare all the considered models together and see how many times ClimateGAN produces the most realistic images among all models.

In the conclusion, the authors state that “we established, using human evaluations,  that our Painter alone is able to create realistic water textures when provided ground masks, and that its performance increases even further when provided with the Masker’s predictions”. However, I could not find such evaluation. From the paper, it seems that the human evaluation is for the Masker+Painter, not for the painter alone.


MINOR COMMENTS
-	Specify what the lambdas in the loss function represent.
-	In the “image-to-image translation” section, this sentence lacks the verb “CycleGAN (Zhu et al., 2017) this constraint. Also, what is the first category of IIT? Please clarify it in the text.
-	In the “conditional image synthesis”, I would add a reference where GauGAN is first mentioned.
-	I think it would be interesting to know what is the number of new real images that you were able to collect (apart from the ones taken from Cityscapes and Mapillary)
-	If my understanding is correct, the “comparables” section refers to other models used for comparison in the human evaluation section. I think it should be clearly stated in the section. 



summary_of_the_review: I believe that the purpose of the proposed model is topical and of interest. However, from a technical viewpoint, I believe that the contribution is not novel enough for acceptance at ICLR as ClimateGAN is given by combining two already existing models (DADA and GauGAN) with minor modifications.
review_title: null

Meta-review: This paper aims at raising awareness of climate change by GAN-projecting flooding images of popular places. This is an interesting case. While all reviews agree that this is an interesting direction, they also value the contributions differently. Two of them would like to see more methodological contributions, two focus more on the contribution to the (psychological) fight of climate change. Nevertheless, the rolling discussion helped to clarify several of the issues raised by the reviewers, and (in my opinion) "combining" existing methods to realize an important model that is one little step towards making people more aware how climate change may impact their own lifes is highly creative and useful. I therefor overall suggest strongly to accept the paper. As the ICLR CfP reads, ICLR is not just about methodological contributions. Societal considerations of representation learning are explicitly mentioned.
Decision: Accept (Poster)