Robustness via Uncertainty-aware Cycle Consistency
Authors: Uddeshya Upadhyay
Yanbei Chen
Zeynep Akata
Venue: NeurIPS 2021 Poster (2021)
Keywords: Robustness
Uncertainty
Image-to-Image translation
Cycle Consistency

Abstract: Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen perturbations at test time. To address this, we propose a novel probabilistic method based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed distributions. We compare our model with a wide variety of state-of-the-art methods on various challenging tasks including unpaired image translation of natural images spanning autonomous driving, maps, facades, and also in the medical imaging domain consisting of MRI. Experimental results demonstrate that our method exhibits stronger robustness towards unseen perturbations in test data. Code is released here: https://github.com/ExplainableML/UncertaintyAwareCycleConsistency.

Robustness via Uncertainty-aware Cycle Consistency
Uddeshya Upadhyay'
Yanbei Chen'
Zeynep Akata
University of Tübingen
Max Planck Institute for Intelligent Systems
Abstract
Unpaired image-to-ing to-immge translation refers learning sere-image-dmmaiee map- ping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness outliers predictive uncertainty, leading to performance degradation when encountering unseen per- turbations at test time. To address this, we propose a novel probabilistic method Generalized Adaptive Cycle Consistency (UGAC),
based on Uncertainty-awar which models the per-pixel residual by generalized Gaussian distribution, ca- pable of modelling heavy-tailed distributions. We compare our model with a wide variety of state-of-the-an methods on various challenging tasks including unpaired image translation of natural images, using standard datasets, spanning autonomous driving, maps, facades, and also in medical imaging domain consist- ing of MRI. Experimental results demonstrate that our method exhibits stronger robustness towards unseen perturbations test data. Code is released here: https: //gi tiub........ com/ http://phhaaaaaaaeteeeerreaaaata
Introduction
Translating an image from a distribution, i.e. source domain, to an image in another distribution, i.e. target domain, with a distribution shift is an ill-posed problem as unique deterministic one-to-one mapping may not exist between the two domains. Furthermore, since the correspondence between inter-domain samples may be missing, their joint-distribution needs to be inferred from set of marginal distributions. However, as infinitely many joint distributions can decomposed into a fixed set of marginal distributions [1, 2, 3], the problem is ill-posed in the absence of additional constraints.
Deep dearning-basee methods tackle the image-to-image translation task by learning inter-domain
mappings in paired or unpaired manner. Paired image translation methods [4, 5, 6, 7, 8, 9] exploit the inter-domain correspondence by penalizing the per-pixel residual (using 1 l2 norm) between the output and corresponding ground-truth sample. Unpaired image translation approaches [1, 10, 11, 12, 13, 14] often use adversarial networks with an additional constraint on the image or feature space imposing structure on the underlying joint distribution of the images from the different domains.
Both paired and unpaired image translation approaches often learn a deterministic mapping between the domains where every pixel in the input domain is mapped to a fixed pixel value in the output domain. However, such a deterministic formulation lead to mode collapse while at the same time not being able to quantify the model predictive uncertainty important for critical applications, e.g.. medical image analysis. is desirable test the performance of the model on unseen perturbed input at test-time, to improve their applicability in the real world. While robustness to outliers is a focus in some domains [15, 16, 17, 18), it has not attracted as much attention unpaired translation.
To address these limitations, we propose an unpaired (unsupervised) probabilistic image-to-image translation method trained without inter-domain correspondence in end-to-end manner. The probabilistic nature of this method provides uncertainty estimates for the predictions. Moreover, modelling the residuals between the predictions and the ground-truth with heavy-tailed distributions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

makes our model robust to outliers and various unseen data. Accordingly, we compare various state-of-the-arr models and our model in their capacity handle samples from similar distribution as taining-dataser as well as perturbed samples, in the context of unpaired translation.
Our contributions framework based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC). Our framework models the residuals between the predictions and the ground-truths with heavy-tailed distributions improving robustness to outliers. Probabilistic nature of UGAC also provides uncertainty estimates for the predictions. (ii) We evaluate UGAC on multiple challenging datasets: natural images consisting Cityscapes [19]. Google aerial maps and photos [4], CMP Facade [20] and medical images consisting of MRI from IXI [21]. We compare our model to seven state-of-the-ar image-to-ine -o-ggee translation methods [12, 22, 1, 11. 10, 23]. Our results demonstrate that while UGAC performs competitively when tested on unperturbed images, improves state-of-the-art methods substantially when tested on unseen perturbations, establishing its robustness. (iii) We show that our estimated uncertainty scores correlate with the model predictive errors (i.e., residual between model prediction and the ground-truth suggesting that it acts as good proxy for the model's reliability at test time.
follows. (i) We propose unpaired probabilistic image-to-image translation
2 Related Work
Image-to-image translation. Image-to-image translation is often formulated as per-pixel deter- ministic regression between two image domains of [24, 25, 26). In [4], this is done in a puired manner using conditional adversarial networks, while in [10, 1, 12, 22, 11] this is done in an umpaired manner by enforcing additional constraints on the joint distribution of the images from separate domains. Both CycleGAN [10] and UNIT [1] learn bi-directional mappings, whereas other recent methods [12, 22, 11] learn uni-directional mappings.
Quantification of uncertainty in the predictions made by the unpaired image-to-image translation models largely remains unexplored. Our proposed method operates at the intersection of uncertainty estimation and unsupervised translation. Critical applications such as medical image-to tongee-o-inng to-image translation [27, 28, 29, 30, 31, 32] is an excellent testbed for our model as confidence in the network's predictions is desirable [33, 34] especially under the influence of missing imaging modalities.
Uncertainty estimation. Among two broad categories of uncertainties that be associated with a model's prediction, epistemic uncertainty in the model parameters is learned with finite data whereas aleatoric uncertainty captures the noise/uncertainty inherent in the data [35, 36]. For image-to-image translation, various uncertainties can be estimated using Bayesian deep learning techniques [36, 37, 38, 39, 40]. In critical areas like medical imaging, the errors in the predictions deter the adoption of such frameworks in clinical contexts. Uncertainty estimates for the predictions would allow subsequent revision by clinicians [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51].
Existing methods model the per-pixel heteroscedasstictyy as Gaussian distribution for regression tasks [36]. This is not optimal in the presence of outliers that often tend to follow heavy-tailed distri- butions [52, 53]. Therefore, we enhance the above setup by modelling per-pixel heteroscedasticity as generalized Gaussian distribution, which can model wide variety of distributions, including Gaussian, Laplace, and heavier-tailed distribution.
3 Uncertainty-awarr Generalized Adaptive Cycle-consistency (UGAC)
We present the formulation of the unpaired image-to-image translation problem. We discuss the short- comings of the existing solution involving the cycle consistency loss called CycleGAN [10]. Finally, we present our novel probabilistic framework (UGAC) that overcomes the described shortcomings.
3.1 Preliminaries
Formulation. Let there be two image domains A and B. Let the set of images from domain A and B be defined by (i) SA := {a1,d2..aa, az...an), where a; ~ PA Vi and (ii) SB == [b,, b2..bm }, where by ~ PB Vi, respectively. The elements a, and by represent the 7th image from domain and respec- tively, and are drawn from underlying unknown probability distribution PA and Pa respectively.
2

---

Ca of, a.)
b;
D
bi
20
b
La, o By, b.)
Figure 1: Our UGAC framework with the cycle between two generators. For translating from to B (A + B), the input a; is mapped to generalized Gaussian distribution parameterized by {b,, a; 8b) The backward cycle (A + B - A) reconstructs the image distribution parameterized by (ā,, 0 39). UGAC uses Las objective function in Eq. 8 and adversarial losses in Eq. 10 and 11.
Let each image have K pixels, and Was represent the Athh pixel of particular image u, We are interested in learning a mapping from domain to B (A (A B) and B to (B + A) an unpaired manner that the correspondence between the samples from Pa and Pg is not required at the learning stage. In other words, we want to learn the underlying joint distribution PAB from the given marginal distributions PA and PB. This work utilizes CycleGANs that leverage the eycle consistency to learn mappings from both directions (A - B and B + A), but often we are only interested in one direction and the second direction is the auxiliary mapping that aids in learning process. We define the mapping +
B as primary and + A as auxiliary.
Cycle consistency. Learning joint distribution from the marginal distributions is an ill-posed problem with infinitely many solutions [3]. CycleGAN [10] enforces an additional structure on the joint distribution using set of primary networks (forming GAN) and set of auxiliary networks. The primary networks are represented by (9 0% DA(: (2)), where GA represents a generator and DA represents discriminator. The auxiliary networks represented by {= DB(:; (R): While the primary networks learn the mapping A B, the auxiliary networks learn - A (see Figure 1). Let the output of the generator GΛ translating samples from domain A (say a;) to domain B be called b. Similarly, for the generator Gn translating samples from domain (say b;) to domain A be called à, i.e., b, G (a,; ( and à; GB (b;; (g) To simplify the notation, we will omit writing parameters of the networks in the equation. The cycle consistency constraint [10] re- re-translates the above predictions (bs, â;) to get back the reconstruction in the original domain (a,,be), where, a GB(b,) and b; (â.), and attempts to make reconstructed images (ā;, bi) similar to original input (a,, by) by penalizing the residuals with L1 norm between the reconstructions and the original input images, giving the cycle consistency Laye (ā,, b,, a,, b.) La (ā, a,) + La (b, b.).
Limitations of cycle consistency. The underlying assumption when penalizing with the L, norm is that the residual at every pixel between the reconstruction and the input follow zero-mean and fixed-varine variance Laplace distribution, i.e., ūej = aij + ei and bij big + cg with,
Gi Laplace(c: O. /2 ) 22
v2l--0
(1)
where o² represents the fixed-variame variance of the distribution, a. represents the thh pixel in image a, and et represents the noise in the jth pixel for the estimated image aij. This assumption on the residuals between the reconstruction and the input enforces the likelihood (i.e., (())x) = O)) and X == SA SB) to follow a faacooed Laplace distribution:
where ④ = 0% u og UoR U og
L(O|x') & II
valeg mggl
<2 bpal
(2)
1P
where minimizing the Segative-log-likelhooe yields Leye with the following limitations. The resid- uals in the presence of outliers may not follow the Laplace distribution but instead heavy-tailed distribution, whereas the i.i.d assumption leads to fixed variance distributions for the residuals that do not allow modelling of heteroscedasticity to aid in uncertainty estimation.
3

---

3.2 Building Uncertainty-aware Cycle Consistency
We propose to alleviate the mentioned issues by modelling the underlying per-pixel residual distribution as independent but non-identically distributed zero-mean generalized Gaussian distribution (GGD) (Figure 2), i.e., with no fixed shape (3 > 0) and scale (ox 0) parameters. Instead, all the shape and scale parameters of the distributions are predicted from the networks and formulated as follows:
e-18-05
g=16=1
s=1p-15
0=1,=2
2-05,5-05
0=05,02
0=5.0=05
0.2
0.0
-2
Big
o
cj G, B(j)
15
Figure 2: Probability density func- tion (pdf) for generalized Gaussian distribution. Different scale (a) and shape (8) parameters lead to differ-
20, 1 3g
For each Eij. the parameters of the distribution (eyj,Be) may not be the same as parameters for other 6,&8; therefore, they are non-identically distributed allowing modelling with heavier tail distributions. The likelihood for our proposed model is,
ent tail behaviour. (a,, 3) (1,2) represents Gaussian distribution.
L(01x)
II 9 (B% Cij. asj. aiz (5)) agg bpg bpg): sjpg
(4)
where (3%)) represents the jth pixel of domain A's shape parameter 8 (similarly for others). 9 (B d.. Maj, usy) is the pixel-likelihood at jth pixel of image u, (that can represent images of both domain A and B) formulated as,
o (3%j o.jy. Usg. uny) GGD Bay a.y. Bij),
(5)
The negative-log-liklliooe is given by.
[8g 0gg
#8 In
18ng Spg
- In 1(0)() -
Σ PAIN
ln 20
13
20
pq
Pg
(6)
minimizing the negative-log-likeliee yields new cycle consistency loss, which we call as the uncertainty-awee generalized adaptive cycle consistency loss Luuyes given of {ās, 0i 39 az and S (b at, bi).
Luyel B) Lag( + (.)
where (af( = Laslās, af, 3r, a;) is the new objective function corresponding to domain A,
Las(ā;, 0f. PT, 04)
1 K
Σ
Lāaj -ais]
airl
log 0
30 Uj
+ log T(
(8)
ai
oa
where (ā,, be) are the reconstructions for (a;, b,) and (a B9)) (a. F.) are scale and shape parameters for the reconstruction (ā,, b,), respectively.
The L, norm-based cycle consistency (Ceyc) special case of Luye with (afj, Bi a Bi))
(1, 1, 1, j. To utilize Lucye, one must have the maps and the & maps for the reconstructions of the inputs. To obtain the reconstructed image, a (scale map). and 3 (shape map), we modify the head of the generators (the last few convolutional layers) and split them into three heads, connected to common backbone. Therefore, for inputs and b; to the generator GA and GB. the outputs are:
(br, ah. (.) = (a,) and (ās, a (2)) = GB (h:) (a., aG Bi) = (b.) and (b,, af. 3.) = (ā.))
The estimates are plugged into Eq. (7) and the networks are trained to estimate all the parameters of the GGD modelling domain A and B, i.c. (āig, aty, (9 and (bij. aiy. Vij.

---

Furthermore, we apply adversarial losses [10] to the mapping functions, (i) GA A - B and (ii) GB B + A, using the discriminators Da and DB. The discriminators are inspired from patchGANs [4, 10] that classify whether 70x70 overlapping patches are real or not. The adversarial loss for the generators (L [10]) is.
F C)
(10)
The loss for discriminators (C [10]) is,
ead (2(DDA Le L2( (2(DB((a))) L2(B (
(11)
To train the networks we update the generator and discriminator sequentially at every step [10, 4, 54]. The generators and discriminators are trained to minimize L and L as follows:
C Cucy + A2 and L CD.
(12)
Closed-form solution for aleatoric uncertainty. Although predicting parameters of the output image distribution allows to sample multiple images for the same input and compute the uncertainty, modelling the distribution as GGD gives us the uncertainty (a (aaletoric) without sampling from the
r²r(())
distribution closed form solution exists, aeeatoric
Epistemic uncertainty (Tepistemic) is
calculated by multiple forward passes (T = 50 times) with dropouts activated for the same input and computing the variance across the outputs (ūz), i.e., o depistemic = (E, E,
4 )/TT. We define
the total uncertainty (a) ²² o aelaatoric + o² epistemic-
4 Experiments
In this section, we first describe our experimental setup and implementation details. We compare our model to a wide variety state-of-the-an methods quantitatively and qualitatively. Finally we provide ablation analysis study the rationale of our model formulation.
4.1 Experimental Setup
Tasks. We study the robustness of unpaired image-to to tome-ggt translation methods, where different methods are first trained on clean images and then evaluated on perturbed images The clean images referred noise-level 0 (NLO); while the perturbed images with increasing noise are referred as NL1, NI.2, and NL.3. We test three types of perturbation including Gaussian, Uniform, and Impulse. From NLO NL3, the standard deviation of the additive Gaussian noise is gradually increased. Similarly, for additive uniform noise, different levels are obtained by gradually increase the upper-bound of the uniform sampling interval [55] and for impulse noise we gradually increase the probability of pixel-value replacement [56]. Details of constructing various NLS are in supplementary.
Datasets. We evaluate on four standard datasets used for image-to-image translation: (i)
Cityscapes [19] contains street scene images with segmentation maps. including 2.975 training and 500 validation and test images; (ii) Google maps [4] contains 1,096 training and test images
scraped from Google maps with acrial photographs and maps; (iii) CMP Facade [20] contains 400 images from the CMP Facade Database including architectural facades labels and photos. (iv) IXI [21]
a medical imaging dataset with 15, 5,000/5, 00000000000000000000000000 training/ten weaiiingggett//llliie vallidaiitiiion images, including Tl MRI and T2 MRI. More preprocessing details for all the datasets are in supplementary.
Translation quality metrics. Following [23]. we evaluate the translation quality of the generated segmentation maps and images, for the datasets with segmentation maps (e.g.. Cityscapes). First, to evaluate the generated segmentation maps, we compute the Intersection over union TOUUSEGN and mean class-wise accuracy (Acc. SEGM) between the generated segmentation maps and the ground- truth segmentation maps. Second, to evaluate the generated images, we first feed the generated images X,r a pre-trained pix2pix model [4] (denoted as p2p, which trained translate images to segmentation maps) to obtain the segmentation maps (2rr). Then, we feed the original images Xorg to the same pix2pix model to obtain another segmentation maps p2p( X Nrrg), and compute the IoU between two outputs 22p(X,,) and p2p(X Xorg) (IoU. P2P).
5

---

disGAN
0.250
0.200
0.200
83GAN
ACC. SEGM
0.200
SEGM
0.150
0.150
UNIT
0.150
0.100
0.100
CUT
CyGAA
0.050
0.050
0.050
8CyGAN
0.000
NL1
ML2
0.000
NLO
NLI
0.000
NLO
UGAC(ous)
DOYGAN
Figure 3: Evaluation of different methods on Cityscapes with Gaussian perturbation under varying noise levels. NLO denotes clean images without noise, NL1, NL2, NL3 are unseen noise levels.
ACC.Segm IoU. Segm, RoU.P2P are three metrics for evaluating translation quality. Higher is better.
Cityscapes
Maps
Facade
IXI
P Methods
AMSE (sid), ASSIM (sid)1
AMSE (sid),
ASSIM (sid)
AMSE (sdy)
ASSIM (sid)T
AMSE (sid))
ASSIM (sud)
goGAN [22]
107.83 (10.8)
(0.09)
117.21 (10.6)
0.43 (0.07)
(38.21(115))
0,41 (0.05)
(8.7)
0.67 (0.12)
Ganssinn
[UT
108.34 (8.7)
0.51 (0.12)
119.32 (8.9)
0.51 (0.11)
123.22 (17.6)
0.58 (0.09)
87.12 (10.4)
0.64 (0.07)
Cy.GAN [10]
121.32 (10.3)
0.31 (0.13)
107.32 (7.5)
0.61 (0.13)
(15.3)
0.45 (0.07)
(9.14
(0.09)
GAN [23]
107.76 (11.2)
0.60 (0.08)
(9.3)
(0.05)
109.32 (10.4)
0.68 (0.06)
88.36 (8.2)
0.77 (0.09)
UGAC (ours)
80.19 (10.4)
0.78 (0.09)
72.32 (8.4)
0.82 (0.07)
95.37 (9.3)
0.77 (0.04)
68.38 (9.8)
0.87 (0.11)
geGAN [22]
96.76 (18.2)
0.66 (0.03)
104.83 (11.7)
0.49 (0.09)
129.54 (15.1)
0.47 (0.09)
91.45 (13.3)
0.71 (0.08)
Uniform
CUT [11]
98.45 (9.8)
0.59 (0.09)
108.21 (7.5)
0.53 (0.14)
114.45 (21.9)
0.55 (0.12)
75.31 (8.3)
0.78 (0.15)
CyGAN [10]
111.17 (15.4)
0.35 (0.08)
91.47 (10.8)
0.70 (0.10)
158.57 (25.2)
0.39 (0.16)
85.24 (9.5)
0.72 (0.05)
Cy [23]
97.89 (12.1)
0.64 (0.04)
75.97 (10.7)
0.78 (0.16)
106.79 (18.7)
0.69 (0. 14)
70.89 (8.8)
0.81 (0.09)
UGAC (ours)
63.77 (8.5)
0.83 (0.07)
51.24 (6.6)
0.88 (0.11)
92.77 (13.2)
0.78 (0.07)
43.54 (6.2)
0.89 (0.05)
geGAN [22]
105.64 (17.3)
0.60 (0.07)
116.55 (15.8)
0.45 (0.11)
134.56 (10.7)
0.40 (0.11)
121.31 (17.4)
0.66 (0.13)
Impulse
CUT [1]]
90.56 (11.6)
0.52 (0.11)
97.21 (7.8)
0.65 (0.09)
118.89 (15.9)
0.52 (0.07)
98.66 (9.7)
0.69 (0.09)
Cy.GAN [10]
122.48 (19.6)
0.30 (0.12)
112.38 (9.8)
0.62 (0.13)
174.65 (19.2)
0.33 (0.14)
106.16
0.67 (0.12)
GAN [23]
95.78 (10.6)
0.61 (0.05)
90.17 (13.2)
0.77 (0.08)
119 89 (12.8)
0.57 (0.09)
96.91 (10.57)
0.73 (0.06)
UGAC (ours)
78.85 (6.9)
0.80 (0.10)
66.58 (10.4)
0.86 (0.05)
103.83 (9:4)
0.72 (0.09)
70.54 (10.4)
0.85 (0.07)
Table 1: Evaluating methods on four datasets under Gaussian, Uniform and Impulse perturbations, evaluated with AMSE (lower better) and ASSIM (higher better) across varying noise levels. "P" perturbation. We show results with best performing four methods (other three are in supplementary).
Metrics for model robustness. We define two metrics similar to [23] to test model robustness towards noisy inputs. (i) AMSE is the area under the curve measuring the MSE between the outputs of
the noisy input and the clean input under different levels of noise, i.e., AMSE _ (MSE(Ga +
Thin
n), GA (a;)))) where 7) is the noise level, GA denotes the generator that maps domain sample a; (from domain A) to domain B. (ii) ASSIM is the area under the curve measuring the SSIM [57] between the outputs of the noisy input and the clean input under different levels of noise, i.e., + n)) GA (d1)). These two metrics show how much the output deviates when fed with the corrupted input from the output corresponding to clean input, averaged over multiple corruption/noise levels. Computational details are in supplementary.
ASSIM- = Iam (SSIM(AA(
Implementation details. In our framework, the generator is a cascaded U-Net that progressively improves the intermediate features to yield high-quality output [30], we use patch discriminator [4]. All the networks were trained using Adam optimizer [58] with a mini-batch size of 2. The initial learning rate was set to 2e-4 and cosine annealing was used to decay the learning rate over 1000 epochs. The paper-parameters, (A,, X2) (Eq. (12)) were set to (10, 2). For numerical stability, the proposed network produces 1 instead of a. The positivity constraint on the output (for predicted o. 3))
is enforced by applying the ReLU at the end of the output layers in the network. The architecture details and the training scheme are in supplementary.
4.2 Comparing with the State of the Art
Compared methods. We compare our model seven tf-h-ar-the-art methods for unpaired image- to-image translation, including (1) distanceGAN [59] (disGAN): a uni-directional method to map different domains by maintaining a distance metric between samples of the domains. (2) geometry consistent GAN [22] (geGAN): uni-directional method that imposes pairwise distance and geometric constraints. (3) UNIT [1]: bi-directional method that matches the latent representations of the two
domain. (4) CUT [1]]: a uni-directional method that uses contrastive learning to match the patches in

---

(g) Ingut
(1 disGAN
@ GCGNN
DO UNIT
64 OUT
C CAGAN
(6 BAAGAN
M GAN
(9) UGAC
GT
Clean
Perturbed
Clean
Perturbed
Clean
Perturbed
Clean
Perturbed
Figure 4: Qualitative results Cityscapes, Google Maps, CMP Facade, and IXI. Outputs of clean image (at NLO) and perturbed image (at NL3) are shown. (a) input, (1)-() outputs from compared methods, and (8) output from UGAC, (b) ground-truth images. Outputs of UGAC are much closer groundtruth images (better in quality) than the other methods in the presence of noise perturbations.
the same locations in both domains. (5) CycleGAN [10] (Cy.GAN): a bi-directional method that uses additional guess discriminator that "guesses" at random which of the image is fake in the collection of input and reconstruction images. (7) adversarial noise CycleGAN [23] (nCy.GAN): another variant of
eycle consistency loss. (6) guess CycleGAN [23]: variant of CycleGAN that
CycleGAN that introduces noise in the cycle consistency loss. Note that both guess CycleGAN [23] and adversarial noise CycleGAN [23] improve the model robustness noise.
Quantitative evaluation. As described in Section 4.1, trained the models using the clean images (NLO) and evaluated them at varying noise levels (NLO, NLI, NL2, NL3), results are detailed next.
Figure shows the quantitative results Cityscapes dataset with Gaussian perturbation. When in- creasing the noise levels, we observe that the performance compared methods degrade significantly, while method remains more robust to noise e.g., the mean IoU SEGM values are changed from around 0.24 to 0.2 for our model but degrades from around 0.24 to 0.05 for the baseline Cy.GAN. Similarly, our model outperforms two strong competitors (gCy. GAN, nCy.GAN) that are built to defend noise perturbation on higher noise levels. Similar trends are observed for other datasets (in supplementary). This indicates that model offers better translation quality at higher noise levels.
To evaluate model robustness, we tested different methods using the metrics AMSE and ASSIM to quantify the overall image quality under increasing noise levels as defined in Section 4.1. Table shows the performance of all the models on different datasets for three types of perturbations, ie.. Gaussian, Uniform, and Impulse. We can that the proposed UGAC model performs better
7

---

than other methods. For instance, when adding Gaussian noise, UGAC obtains much better ASSIM of 0.78/0.82/0.77/00 vs. 0.60/0.68/0.68/0. yielded by the best competitor aCy.GAN on Cityscapee/Map//acade///
When adding Uniform noise or Impulse noise, we can also find that our model outperforms the other methods by substantial margins. Overall, the better performance of UGAC on different datasets suggests its stronger robustness towards various types of perturbations.
Qualitative results. Figure 4 visualizes the generated output images for Cityscapes, Google Maps, CMP Facade, and IXI datasets where all the models are trained with clean images and tested with either clean images or perturbed images. The test-time perturbation is of type Gaussian and corresponds to noise-level NI.2. We see that, while all the methods generate samples of high quality when tested on unperturbed clean input; whereas when tested with perturbed inputs, observe results with artifacts but the artifacts are imperceptible in our UCAC method.
The results on Cityscapes dataset (with the primary direction, translating from segmentation maps to real photo) demonstrate that with perturbed input, methods such as disGAN, geGAN, UNIT generate images with high frequency artifacts (col.1 to 3), whereas methods such as CUT, Cy.GAN, gCy.GAN and JCy.GAN (col.4 to 7) generate images with low frequency artefacts. Both kinds of artefact lead to degradation of the visual quality of the output. Our method (col.8) generates output images that still preserve all the high frequency details and are visually appealing, even with perturbed input. Similar trends are observed for other datasets including Maps (with primary translation from maps to photo) and Facade (with primary translation from segmentation maps to real photo).
For the IXI dataset (with primary translation from TI to T2 MRI scans), we observe that the other models fail reconstruct medically relevant structures like trigeminal-nerve (in the centre) present in the input Ti MRI scans. Moreover, high-freque freeefrey details throughout the white and grey matter in the brain are missing. In contrast, our method gracefully reconstructs many of the details. More qualitative results are in supplementary, with similar trends as in Figure 4. It shows that our model capable of generating images of good quality at higher noise levels.
4.3 Analyzing the Model Uncertainty
Evaluating the generalized adaptive norm. We study the performance of our method by mod- elling the per-pixel residuals in three ways on IXI dataset. First, i.idd Gaussian distribution, i.e., (Bg) is manually set to (1, 2), which is equivalent to using fixed l2 norm at every pixel in cycle consistency loss LLasl == visual quality when given perturhed input. Second, i.i.d Laplace distribution, i.e., (akj, Bij) is manually set to (1, 1)vi,,,, J. which is equivalent to using fixed I norm at Third,
0.900
0.800
0.700
every pixel in cycle consistency loss (Casla=1.3=11 independent but non-identically distributed generalized Gaus- sian distribution (UGAC), which is equivalent to using spatially varying ly quasi-norms where q is predicted by the network for every pixel ( (La.) (paa)
0.600
0.500
NEO
NL3
(ar, B) (1, 2)
(a, B) =)==0000 (1,1) (a, B) pred
Figure 5: Adaptive (a, 3) predd
Fig shows the quantitative performance of these three vari- ants across different noise levels for IXI datasets. We see that spatially adaptive quasi-norms perform better than fixed norms, even at higher noise levels (i.e., presence of outliers). Note that our GGD based heteroscedastic model subsumes the Gaussian (a = 1, = 2) and Laplacian (a = 1, B = 1). Moreover, the heteroscedastic versions of Gaussian and Laplacian can be obtained by fixing B, i..., for Laplacian (3 = 1) and for Gaussian (3 = 2), and varying a. Modeling residuals Gaussian/Laplacian distribution because it is distributions (along with all possible Gaussian/ Laplacian aaplccian
vs. fixed (a,, (1, and (a, (1,, norm.
as GGD is more liberal than both homo/hetero-sceeeeeee able to capture all the heavier/lighter distributions) that are beyond the modeling capabilities of Gaussian/ aaplsiannnLaplaiian alone.
Visualizing uncertainty maps. We visualize uncertainty maps for the Tlw MRI (domain A) to T2w MRI (domain B) translation task, on IXI dataset, with perturbations in the imput (NL3).
Figure 6-(a) shows input axial slices (Tlw at NL3). The perturbations have degraded the high- frequency features (see green ROI). Figure 6-(b) shows the corresponding ground-truth axial slice (T2w MRI). Figure 6-(c) shows that our method recovers high- high-frequency details. However,
8

---

(a) Noicy T1 impat
(b) GT T2
(4) Prediction
(d)
(c)
f d
( |Prediction - GT
0.0 >0.05
+0.39 10,05
<0,30
<0.00
Figure 6: Visualization of uncertainty maps for noisy input at NL3 (sample from IXI test-set). (a) Noisy Tlw MRI input. (b) Corresponding ground-truth T2w MRI. (c) Predicted T2w MRI. (d)-(e) Predicted a and 3 maps. (f) Uncertainty maps derived from predicted a and 3 maps. (g) Absolute residual between the prediction and the ground-trute
observe higher contrast (compared ground-truth) (green ROI). The subtle disparity between the contrast has been picked up by our scale-map (a) and shape-map (3) as shown in Figure 6-(d) and (e), respectively. Moreover, non-identically) likelihood model for the pixel level residuals, the structure in the a and the B (Figure 6-(d) and (e)) shows that the model learns to exploit the correlation the neighbourhood pixels. The pixel-level variation in the and / yields pixel-level uncertainty values in the predictions as described in Section 4.1.
that, although our formulation assumes independent (but
Figure 6-(f) shows the uncertainty map (σ) for the predictions made by the network. We see that the disparity in the contrasts between the prediction and the ground-truth is reflected as high uncertainty in the disparity region, i.e., uncertainty is high where the reconstruction of inferior quality, indicated by high-residual values shown in Figure 6-(g). The correspondence between uncertainty maps (Figure 6-(1)) and residual maps (Figure 6-(g)) suggests that uncertainty maps can be used as the proxy to residual maps (that are unavailable the test time, as the ground-truth images will not be available) and can serve as an indicator of image quality. More samples showing the translation results with high uncertainty estimated using a, (9) are shown in supplementary.
Residual scores vs. uncertainty scores. To further study the relationship between the uncer- tainty maps and the residual maps across a wide variety of images, we analyze the results on IXI test set. We show the density and the scatter-plot between the residual score and uncertainty score in Figure 7, where every point represents single image. For an image, the mean residual score (on the y-axis) is derived as the mean of absolute residual values for all the pixels in the image. Similarly, the uncertainty score (on the araxis) is calculated as the mean of uncertainty values of all the pixels in that image. From the plot, we see that across the test-set mean uncertainty score correlates positively with the mean residual score, i.e., higher uncertainty score corresponds to higher residual. An image with higher residual score represents poor quality image. This further supports the idea that uncertainty maps derived from our method can be used as a proxy to residual that indicates the overall image quality of the output generated by our network.
0.07
0.06
0.05
res
0.04
0.03
0.02
0.06
0.00
0.10
2.12
0.14
9.16
0.0
mean uncertainty score"
Figure 7: Residual scores vs. uncertainty scores.
5 Discussion and Conclusion
In this work, we propose an uncertainty-aware generalized cycle consistency for unpaired image translation along with uncertainty estimation. Our formulation assumes the pixel-wise independent (but non-identically) distributed likelihood model for the residuals, relaxing the i..d. assumption made by the previous work. However, our experiments also show that the model learns the underlying structure between the neighbourhood pixels and predicts the structured/coorelatee parameters for the output distribution (i.e., a, 3 for MRI translation shown in Figure 6-(d) and (e)).
We demonstrate the efficacy of the proposed method on robust unpaired image translation on various datasets spanning autonomous driving, maps, facades, and medical images consisting of MRT scans. We also demonstrate the robustness of our method by evaluating the performance in different kinds of perturbations in the input with varying severity and show that our method outperforms all the basclines by generating superior images in terms of quantitative metrics and appearance. In addition,

---

we show that the uncertainty estimates (derived from the estimated parameters of the distribution, and (3) are faithful proxy the residuals between the predictions and the ground truth.
worth noting that robustness towards various kinds of perturbation can also be achieved by data augmentation techniques that include the perturbed images in the training phase. However, this is orthogonal to the concept proposed in this work that achieves robustness via the a new modeling technique. principle, one could combine both the augmentation techniques and modeling techniques to obtain more robust models. In this work, we used relatively small neural networks (in terms of parameters based on UNet), while this network has not been used previously for this problem, we employ it to train our models with limited compute with reasonable training time and a lower memory footprint (details of the networks available in the Appendix A.5). This however affects the performance of the networks, and leads to images with artifacts/distortions (specially with small datasets consisting few hundred samples). Our method can be applied to deeper neural networks with more parameters/higher capacity and trained with higher resolution images, which would lead to significantly better performance, given enough compute.
An interesting avenue for further exploration is the analysis of uncertainty maps when presented with anomalous inputs, beyond perturbations, with stronger shifts between training and test data distribution which will be investigated in future.
Broader Impact
Modern deep-learning-basee image translation schemes are becoming more popular. They allow the generation of synthetic datasets, e.g., for the segmentation case in autonomous driving, faster image acquisition via algorithmic super-resolution, image enhancement in computational photography, faster and cheaper medical diagnosis by translating between different imaging modalities. However, critical areas like medical imaging and autonomous driving require methods that are robust towards various perturbations and, the same time, can also provide uncertainty estimates in the predictions. Estimating the uncertainty in the prediction can help trigger expert intervention preventing fatal scenarios.
We introduced a novel image-to-image translation model capable of estimating uncertainty along with the predictions and is shown to be beneficial in ensuring good image translation quality and good model performance on downstream tasks even in the presence unseen noisy patterns in input images inference time. Furthermore, the images. These merits could bring positive socictal impacts to various critical application domains, such medical imaging and autonomous driving.
method is potentially applicable to detect ambiguities in
Acknowledgements This work has been partially funded by the ERC (853489. DEXIM) and by the DFG (2064/1- Project number 390727645). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Uddeshya Upadhyay.
References
[1] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to- -oo-maae translation networks, In NIPS, 2017.
[2] Charlie Frogner and Tomaso Poggio. Fast and flexible inference of joint distributions from their marginals. In ICML, 2019.
[3] Torgny Lindvall. Lectures on the coupling method. Courier Corporation, 2002.
[4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
[5] yaxing wang. Lu Yu, and Joost van de Weijer. DeepiZi: Enabling deep hierarchical image-to poaeeett-imagee translation by transferring from gans. In NewrIPS, 2020.
[6] Kai Zhang, Lue Van Gool, and Radu Timofte. Deep unfolding network for image super-resolution. In CVPR, 2020.
[7] Shawn Mathew, Saad Nadeem, Sruti Kumari, and Arie Kaufman. Augmenting colonoscopy using extended and directional cyclegan for lossy image translation. In CVPR, 2020.
10

---

[8] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Dino: A conditional energy-based gan for domain translation. In ICLR, 2021.
[9] Tamar Rott Shaham, Michael Gharbi, Richard Zhang, Eli Shechtman, and Tomer Michaeli. Spatially- adaptive pixelwise networks for fast image translation. CVPR, 2021.
[10] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[11] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020.
[12] Ngoc-Trung Tran, Tuan-Anh Bui, and Ngai-Man Cheung. Dist-gam: An improved gan using distance constraints. In ECCV, 2018.
[13] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The spatially-correlative loss for various image translation tasks. In CVPR, 2021.
[14] Somi Jeong, Youngjung Kim, Eungbean Lee, and Kwanghoon Sohn. Memory-guided unsupervised image-to-image translation. In CVPR, 2021.
[15] Peter Huber al. The 1972 wald lecture robust statistics: review, Annals of Mathematical Statistics, 1972.
[16] Claudio Gentile, The robustness the p-norm algorithms. Machine Learning, 2003.
[17] T. Hastie, R. Tibshirani, and M. Wainwright. Statistical Learning with Sparsity: The Lasso and General- izations. Chapman & Hall/CRC Monographs on Statisties & Applied Probability. CRC Press, 2015.
[18] Michael Black and Anand Rangarajan. On the unification of line processes, outlier rejection, and robust statistics with applications in early vision. IJCV, 1996.
[19] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscupes dataset for semantic urban understanding. In CVPR, 2016.
[20] Radim Tyleček and Radim Šára. Spatial pattern templates for recognition of objects with regular structure. In GCPR, 2013.
[21] Emma C. Robinson, Alexander Hammers, Anders Ericsson, A. David Edwards, and Daniel Rueckert, Identifying population differences in whole-brain structural networks: A machine learning approach. Neurolmage, 2010.
[22] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consisten generative adversarial networks for one-sided unsupervised domain mapping. In CVPR, 2019.
[23] Dina Bashkirova, Ben Usman, and Kate Saenko. Adversarial self-defense for cycle-consistent gans. NewrIPS, 2019.
[24] Saining Xie and Zhuowen Tu. Holistically-nnetted edge detection. In ICCV, 2015.
[25] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[26] Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Lct there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultancous classification. ACM Transactions Graphics, 2016.
[27] Hien Van Nguyen, Kevin Zhou, and Raviteja Vemulapalli. Cross-domain synthesis of medical images using efficient location-sensitive deep network. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro Frangi, editors, MICCAI, 2015.
[28] Qianye Yang. Nannan Li, Zixu Zhao, Xingyu Fan, I Eric, Chao Chang, and Yan Xu. Mri cross-modality image-to-image translation. Scientific reports, 2020.
[29] Salman UH Dar, Mahmut Yurt, Levent Karacan, Aykut Erdem, Erkut Erdem, and Tolga Çukur. Image synthesis in multi contrast mri with conditional generavvvee adversarial networks. IEEE TMI. 2019.
11

---

[30] Karim Armanious, Chenming Jiang. Mare Fischer, Thomas Küstner, Tobias Hepp. Konstantin Nikolaou, Sergios Gatidis, and Bin Yang. Medgan: Medical image translation using gans. Computerized Medical Imaging and Graphics, 2020.
[31]
Uddeshya Upadhyay and Suyash Awate. Robust super-resolution gan, with manifold-based and perception loss. In International Symposium on Biomedical Imaging (ISBI), 2019.
[32] Uddeshya Upadhyay and Suyash P Awate. A mixed-supervisionn multilevel gan framework for image quality enhancement. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2019.
[33] Raghav Mchta, Angelos Filos, Yarin Gal, and Tal Arbel. Uncertainty evaluation metric for brain tumour segmentation. In MIDL, 2020.
[34] Philipp Seebück, José Ignacio Orlando, Thomas Schlegl, Sebastian M Waldstein, Hrvoje Bogunović, Sophie Klimscha, Georg Langs, and Ursula Schmidt-Erfurthe Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal oct. IEEE TMI, 2019.
[35] HM Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Sacid Nahavandi. Neural network-based uncertainty quantification: A survey of methodologies and applications. IEEE access, 2018.
[36] Alex Kendall and Yarin Gal. What uncertaintics do we need bayesian deep learning for computer vision? In NIPS, 2017.
[37] Simon Kohl, Bernardino Comera-Paredes Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus H Maicr-Hein, SM Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In NeurIPS, 2018.
[38] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In JCML, 2016.
[39] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
[40] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modemn neural networks. In ICML, 2017.
[41] Qingping Zhou, Tengchao Yu, Xiaoqun Zhang, and Jinglai Li. Bayesian inference and uncertainty quantification for medical image reconstruction with poisson data, SIAM Journal on Imaging Sciences, 2020.
[42] Shi Hu, Daniel Worrall, Stefan Knegt, Bas Veeling, Henkjan Huisman, and Max Welling. Supervised uncertainty quantification for segmentation with multiple annotations. In MICCAI, 2019.
[43] Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence, 2019.
[44] Chuyang Ye, Yuxing Li, and Xiangzhu Zeng. An improved deep network for tissue microstructure estimation with uncertainty quantification. Medical image analysis, 2020.
[45] Tanya Nair, Doina Precup, Douglas L Arnold, and Tal Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical image analysis, 2020.
[46] Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing, 2019.
[47] Alain Jungo and Mauricio Reyes. Assessing reliability and challenges of uncertainty estimations for medical image segmentation. In MICCAI, 2019.
[48] Uddeshya Upadhyay, Viswanath P Sudarshan, and Suyash P Awate. Uncertainty aware gan with adaptive loss for robust mri image enhancement. ICCV workshop on Computer Vision for Automated Medical Diagnosis, 2021.
[49] Viswanath Sudarshan, Uddeshya Upadhyay, Gary F Egan, Zhaolin Chen, and Suyash P Awate. Towards lower-dose pet using physics- based uncertainty- aware multimodal learning with robustness to out-of- distribution data, Medical Image Analysis, 2021.
12

---

[50] Ryutaro Tanno, Daniel E Worrall, Aurobrata Ghosh, Enrico Kaden. Stamatios N Sotiropoulos. Antonio Criminisi, and Daniel C Alexander. Bayesian image quality transfer with cnns: exploring uncertainty in dmri super-resolution. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017.
[51] Ryutaro Tanno, Daniel E Worrall, Enrico Kaden, Aurobrata Ghosh, Francesco Grussu, Alberto Bizzi, Stamatios N Sotiropoulos, Antonio Criminisi, and Daniel Alexander. Uncertainty modelling in deep learning for safer neuroimage enhancement: Demonstration in diffusion mri. Neurolmage, 2021.
[52] Jiyong Oh and Nojun Kwak. Generalized mean for robust principal component analysis. Pattern Recogni- tion, 2016.
[53] Charles Bouman and Ken Sauer. A generalized gaussian image model for edge-preserving map estimation. IEEE TIP, 1993.
[54] lan J Goodfellow, Jean Rbudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. NIPS, 2014.
[55] P. Refregier and F. Goudail. Statistical Image Processing Techniques for Noisy Images: An Application- Oriented Approach. Springer US, 2013.
[56] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In NeurlPS, 2019.
[57] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE TIP, 2004,
[58] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[59] Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. In NIPS, 2017.
13


Reviews:
[{"review_id": "iDholkMfE8n", "reviewer": "Reviewer_TSoT", "summary": "Authors propose a relaxation of the cycle consistency loss that takes into account per-pixel prediction uncertainty. More specifically, authors assume that cycle-reconstruction residuals are distributed as iid generalized Gaussian random variables, and maximize likelihood of these reconstruction residuals given predicted per-pixel parameters. Authors report two metrics of sensitivity to noise and three segmentation accuracy metrics across four real datasets with known GT and seven baseline methods, confirming that in all cases their method outperforms or performs on par with SotA. Authors also show that restricting parameters of their generalized Gaussian log-likelihood loss to constant values (effectively reducing it to standard CycleGAN) degrades the performance of their method according to these performance metrics. Moreover, authors showed that the overall prediction uncertainty strongly correlates with the magnitude of the true prediction error computed using GT.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Writing is very clear, and the overall presentation is great. Figure 1 might need some more love - it made understanding the paper harder, not easier, but I still managed to understand everything well from text, so it is not a deal breaker. \n\nAuthors quantitatively verified all claims made in the paper and provide an exhaustive set of [baselines x datasets x metrics]. Additional meaningful ablations and visualizations (e.g. residual-vs-uncertainty plot) supporting their claims.\n\nWhile I support accepting this paper, and reported results are already conclusive, I believe that following additions and clarifications would help in making these results even more convincing. \n\nFirst of all, authors should directly point readers to Section A.6 of the supplementary for results of all methods on all dataset both in text and in figure captions, because my initial review assumed that these cases were not reported at all. While talking about the supplementary, “set all the CycleGAN parameters to the default ones provided in the implementation except for the weights of the cycle-consistency loss” - could you elaborate on why/how you decided to change baseline hyper-parameters from default ones? \n\nMany images (including tested g/nCyGAN and proposed UGAC) in Figure 4 (and Figure 2 in supplementary) look like their color channels were displaced (e.g. three displaced r/g/b blobs instead of a single white blob where the car logo goes). This kind of artifact seems to be not very typical for image-to-image models and not present in any prior work. Could you please elaborate on why you think it showed up in your results (including prior work g/nCyGAN)? It is just a visualization issue or it might have affected reported results?\n\nHow does the amplitude of the noise used during training of nCyGAN relate to NL0-3 levels used during evaluation? Does changing the training noise level in nCyGAN affect the results reported in Table 1?\n\nI understand that the generalized Gaussian is used to model heavy tales of the residuals, but does this choice matter in practise? For example, if you modeled per-pixel reconstruction residuals as iid Gaussian/Laplace rvs with predicted _per-pixel standard deviations_, did you check how that would affect the performance experimentally?\n\n“Residual scores vs. uncertainty scores” (Figure 7) - while it is clear from this picture that there must be some significant correlation between the two, the p-value of a simple statistical test verifying that the observed deviation of the correlation coefficient from zero is significant (given your sample size) would support this point quantitatively. \n\nBoth the original cyclegan loss and the proposed loss assume that all residuals are independent, which is clearly not the case in practise. I would appreciate an extended discussion of consequences of such choice versus maximizing the likelihood of a proper conditional image density model. In other words: why modeling the residual likelihood as iid noise (gaussian or otherwise) is preferred over modeling it as a proper density model that takes pixel interactions into account.\n\nDefinitions of AMSE and ASSIM need some clarification. Is it \\int |MSE(GA(ai), bi) - MSE(GA(ai + noise), bi)| or |MSE(GA(ai), GA(ai + noise))| or something else entirely?\n\nFig 5 - colored lines in the legend are too thin and therefore hard to see.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do not explicitly discuss scenarios in which their method might fail. One possible cause for concern are over-confident predictions in the context of human computer interaction: if a machine _usually_ accurately predicts how uncertain its predictions are, and human learns to rely on there prediction uncertainty estimates, how do we mitigate the consequences of (inevitably, eventually) machine wrongly over-estimating its confidence for an unusual input, and human blindly relying on it (because \"the machine said it was confident, so we decided that double-checking is not necessary\")?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6h"}, "extra_text": {}}, {"review_id": "EC7TB0iu_07", "reviewer": "Reviewer_xn61", "summary": "The paper achieves improved and more robust unpaired image-to-image translation results by modeling uncertainty of the outputs. The proposed method generalized and treats the original pixel cycle loss of CycleGAN as the special case in which the uncertainty distribution is uniformly fixed. Furthermore, the predicted uncertainty can be used as a way to measure risk at inference time. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a clear idea that improves on previous methods in terms of both output quality and robustness. The paper is well motivated and explained. Moreover, modeling uncertainty can be beneficial in assessing the risk when processing out-of-distribution, corrupted images, and the paper proves this point on medical imaging. The model uncertainty is sufficiently analyzed in various types of artifical noises via several graphs and visualizations.\n\nFor the biggest weakness, something seems wrong with the qualitative comparison results. The RGB channels seem shifted and not aligned (clean Facade results of Figure 4). Images are also quite blurry. In Fig3 of Supp Mat, the input Facade segmentation layouts are greyscale. They also look too low resolution for 256x256 images. If these images are indeed faulty, I wonder if the evaluations are also based on these images. Since there is possibility of faulty evaluation, I am hesitant to conclude a positive rating for this submission. \n\nThe evaluations for the generated images in image-to-image translation tasks often involve FID. Including FID to the paper can also make the paper more comprehensive and comparable to existing papers in this field. \n\nThe architecture of the proposed method is based on cascaded U-Net, and some changes may be coming from the architectural changes. I am curious of the results of using this architecture, but using the original cycle loss without uncertainty modeling. \n\nLastly, it would be interesting to see which input images of the dataset are considered as uncertain. Visualizing the top-10 most certain and uncertain examples would be interesting. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe improving robustness and modeling uncertainty of image-to-image translation methods is certainly beneficial, especially for applications like medical imaging. The authors also adequately discuss this in Broader Impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "0WlxI8oawWG", "reviewer": "Reviewer_e4fm", "summary": "This paper introduces a new method for unpaired image to image translation that addresses uncertainty estimation and robustness. The method is based on an adaptive loss function that accounts for spatially-varying noise levels. This loss function motivated by Bayesian modelling and it is incorporated in a cycle-consistent loss. The experimental setting consists in training the proposed method and other competitors on clean data and test it on noisy data. The method is evaluated on different dataset including street scene images and medical images. According to a measure of performance on noisy data, the method is shown to outperform existing methods. Last, the method outputs uncertainty maps which are the parameters of the spatially adaptive loss function, these are shown to positively correlate with the residual losses.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper is clear and is of definite interest. Quantification of uncertainty matters for practical applications. At runtime, uncertainty comes from at least two sources: the uncertainty on the learned model parameters and the uncertainty in the input. This paper incorporates the uncertainty of the input directly in the loss, via a weighted $L^p$ loss in which $p$ and the weight are pixel dependent. The actual form is motivated by a Bayesian analysis. These loss parameters are learnt and depend on the entire image. \nThe paper is well written and easy to read. \n\n\nQuestions: \n- Figure 3 shows that the method outperforms the other methods even at $0$ noise level. Since the method is made to take learn noise in the loss function, it is not intuitive why a performance improvement can be achieved at $0$ noise level. Can the authors elaborate on this?\n- Data augmentation can be used to gain robustness with respect to noise in the data. How would it compare with the proposed method?\n- In the situation where the parameters of the data noise is known, it is rather unclear that the model would be able to retrieve these parameters. Would it be possible to test the model in this situation? My guess is that the found parameters will be very dependent on the image and will not recover the true underlying parameters. As a conclusion, it is non easily interpretable what is learnt in this loss function. \n\n\n\nTypos: \n- page 4 : equation 9 and 5 a dot is missing\n- \"boarder impact\"\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work is absent from the conclusion. It is of importance that the authors discuss the potential pitfalls of the method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, the reviewers are all for accepting the work. It is well-motivated and presents its ideas well. There are some concerns regarding image artifacts (particularly, the color channels being distorted). The rebuttal does not really address this concern as no prior work which use very similar architectures suffer from this phenomena. I highly recommend the authors look into resolving or clarifying why this happens, and incorporating this discussion in their paper.
Decision: Accept (Poster)