Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new method for unpaired image to image translation that addresses uncertainty estimation and robustness. The method is based on an adaptive loss function that accounts for spatially-varying noise levels. This loss function motivated by Bayesian modelling and it is incorporated in a cycle-consistent loss. The experimental setting consists in training the proposed method and other competitors on clean data and test it on noisy data. The method is evaluated on different dataset including street scene images and medical images. According to a measure of performance on noisy data, the method is shown to outperform existing methods. Last, the method outputs uncertainty maps which are the parameters of the spatially adaptive loss function, these are shown to positively correlate with the residual losses.
main_review: The motivation of the paper is clear and is of definite interest. Quantification of uncertainty matters for practical applications. At runtime, uncertainty comes from at least two sources: the uncertainty on the learned model parameters and the uncertainty in the input. This paper incorporates the uncertainty of the input directly in the loss, via a weighted $L^p$ loss in which $p$ and the weight are pixel dependent. The actual form is motivated by a Bayesian analysis. These loss parameters are learnt and depend on the entire image. 
The paper is well written and easy to read. 


Questions: 
- Figure 3 shows that the method outperforms the other methods even at $0$ noise level. Since the method is made to take learn noise in the loss function, it is not intuitive why a performance improvement can be achieved at $0$ noise level. Can the authors elaborate on this?
- Data augmentation can be used to gain robustness with respect to noise in the data. How would it compare with the proposed method?
- In the situation where the parameters of the data noise is known, it is rather unclear that the model would be able to retrieve these parameters. Would it be possible to test the model in this situation? My guess is that the found parameters will be very dependent on the image and will not recover the true underlying parameters. As a conclusion, it is non easily interpretable what is learnt in this loss function. 



Typos: 
- page 4 : equation 9 and 5 a dot is missing
- "boarder impact"



limitations_and_societal_impact: The limitations of the work is absent from the conclusion. It is of importance that the authors discuss the potential pitfalls of the method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper achieves improved and more robust unpaired image-to-image translation results by modeling uncertainty of the outputs. The proposed method generalized and treats the original pixel cycle loss of CycleGAN as the special case in which the uncertainty distribution is uniformly fixed. Furthermore, the predicted uncertainty can be used as a way to measure risk at inference time. 
main_review: The paper proposes a clear idea that improves on previous methods in terms of both output quality and robustness. The paper is well motivated and explained. Moreover, modeling uncertainty can be beneficial in assessing the risk when processing out-of-distribution, corrupted images, and the paper proves this point on medical imaging. The model uncertainty is sufficiently analyzed in various types of artifical noises via several graphs and visualizations.

For the biggest weakness, something seems wrong with the qualitative comparison results. The RGB channels seem shifted and not aligned (clean Facade results of Figure 4). Images are also quite blurry. In Fig3 of Supp Mat, the input Facade segmentation layouts are greyscale. They also look too low resolution for 256x256 images. If these images are indeed faulty, I wonder if the evaluations are also based on these images. Since there is possibility of faulty evaluation, I am hesitant to conclude a positive rating for this submission. 

The evaluations for the generated images in image-to-image translation tasks often involve FID. Including FID to the paper can also make the paper more comprehensive and comparable to existing papers in this field. 

The architecture of the proposed method is based on cascaded U-Net, and some changes may be coming from the architectural changes. I am curious of the results of using this architecture, but using the original cycle loss without uncertainty modeling. 

Lastly, it would be interesting to see which input images of the dataset are considered as uncertain. Visualizing the top-10 most certain and uncertain examples would be interesting. 
limitations_and_societal_impact: I believe improving robustness and modeling uncertainty of image-to-image translation methods is certainly beneficial, especially for applications like medical imaging. The authors also adequately discuss this in Broader Impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a relaxation of the cycle consistency loss that takes into account per-pixel prediction uncertainty. More specifically, authors assume that cycle-reconstruction residuals are distributed as iid generalized Gaussian random variables, and maximize likelihood of these reconstruction residuals given predicted per-pixel parameters. Authors report two metrics of sensitivity to noise and three segmentation accuracy metrics across four real datasets with known GT and seven baseline methods, confirming that in all cases their method outperforms or performs on par with SotA. Authors also show that restricting parameters of their generalized Gaussian log-likelihood loss to constant values (effectively reducing it to standard CycleGAN) degrades the performance of their method according to these performance metrics. Moreover, authors showed that the overall prediction uncertainty strongly correlates with the magnitude of the true prediction error computed using GT.
main_review: Writing is very clear, and the overall presentation is great. Figure 1 might need some more love - it made understanding the paper harder, not easier, but I still managed to understand everything well from text, so it is not a deal breaker. 

Authors quantitatively verified all claims made in the paper and provide an exhaustive set of [baselines x datasets x metrics]. Additional meaningful ablations and visualizations (e.g. residual-vs-uncertainty plot) supporting their claims.

While I support accepting this paper, and reported results are already conclusive, I believe that following additions and clarifications would help in making these results even more convincing. 

First of all, authors should directly point readers to Section A.6 of the supplementary for results of all methods on all dataset both in text and in figure captions, because my initial review assumed that these cases were not reported at all. While talking about the supplementary, “set all the CycleGAN parameters to the default ones provided in the implementation except for the weights of the cycle-consistency loss” - could you elaborate on why/how you decided to change baseline hyper-parameters from default ones? 

Many images (including tested g/nCyGAN and proposed UGAC) in Figure 4 (and Figure 2 in supplementary) look like their color channels were displaced (e.g. three displaced r/g/b blobs instead of a single white blob where the car logo goes). This kind of artifact seems to be not very typical for image-to-image models and not present in any prior work. Could you please elaborate on why you think it showed up in your results (including prior work g/nCyGAN)? It is just a visualization issue or it might have affected reported results?

How does the amplitude of the noise used during training of nCyGAN relate to NL0-3 levels used during evaluation? Does changing the training noise level in nCyGAN affect the results reported in Table 1?

I understand that the generalized Gaussian is used to model heavy tales of the residuals, but does this choice matter in practise? For example, if you modeled per-pixel reconstruction residuals as iid Gaussian/Laplace rvs with predicted _per-pixel standard deviations_, did you check how that would affect the performance experimentally?

“Residual scores vs. uncertainty scores” (Figure 7) - while it is clear from this picture that there must be some significant correlation between the two, the p-value of a simple statistical test verifying that the observed deviation of the correlation coefficient from zero is significant (given your sample size) would support this point quantitatively. 

Both the original cyclegan loss and the proposed loss assume that all residuals are independent, which is clearly not the case in practise. I would appreciate an extended discussion of consequences of such choice versus maximizing the likelihood of a proper conditional image density model. In other words: why modeling the residual likelihood as iid noise (gaussian or otherwise) is preferred over modeling it as a proper density model that takes pixel interactions into account.

Definitions of AMSE and ASSIM need some clarification. Is it \int |MSE(GA(ai), bi) - MSE(GA(ai + noise), bi)| or |MSE(GA(ai), GA(ai + noise))| or something else entirely?

Fig 5 - colored lines in the legend are too thin and therefore hard to see.
limitations_and_societal_impact: Authors do not explicitly discuss scenarios in which their method might fail. One possible cause for concern are over-confident predictions in the context of human computer interaction: if a machine _usually_ accurately predicts how uncertain its predictions are, and human learns to rely on there prediction uncertainty estimates, how do we mitigate the consequences of (inevitably, eventually) machine wrongly over-estimating its confidence for an unusual input, and human blindly relying on it (because "the machine said it was confident, so we decided that double-checking is not necessary")?
review_title: null

Meta-review: After discussion, the reviewers are all for accepting the work. It is well-motivated and presents its ideas well. There are some concerns regarding image artifacts (particularly, the color channels being distorted). The rebuttal does not really address this concern as no prior work which use very similar architectures suffer from this phenomena. I highly recommend the authors look into resolving or clarifying why this happens, and incorporating this discussion in their paper.
Decision: Accept (Poster)