Robustness via Uncertainty-aware Cycle Consistency
Year: 2021

Abstract:
Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen perturbations at test time. To address this, we propose a novel probabilistic method based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed distributions. We compare our model with a wide variety of state-of-the-art methods on various challenging tasks including unpaired image translation of natural images, using standard datasets, spanning autonomous driving, maps, facades, and also in medical imaging domain consisting of MRI. Experimental results demonstrate that our method exhibits stronger robustness towards unseen perturbations in test data. Code is released here: https: //github.com/ExplainableML/UncertaintyAwareCycleConsistency.

text: Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty, leading to performance degradation when encountering unseen perturbations at test time. To address this, we propose a novel probabilistic method based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC), which models the per-pixel residual by generalized Gaussian distribution, capable of modelling heavy-tailed distributions. We compare our model with a wide variety of state-of-the-art methods on various challenging tasks including unpaired image translation of natural images, using standard datasets, spanning autonomous driving, maps, facades, and also in medical imaging domain consisting of MRI. Experimental results demonstrate that our method exhibits stronger robustness towards unseen perturbations in test data. Code is released here: https: //github.com/ExplainableML/UncertaintyAwareCycleConsistency.

heading: 1 Introduction
text: Translating an image from a distribution, i.e. source domain, to an image in another distribution, i.e. target domain, with a distribution shift is an ill-posed problem as a unique deterministic one-to-one mapping may not exist between the two domains. Furthermore, since the correspondence between inter-domain samples may be missing, their joint-distribution needs to be inferred from a set of marginal distributions. However, as infinitely many joint distributions can be decomposed into a fixed set of marginal distributions [1, 2, 3], the problem is ill-posed in the absence of additional constraints.
Deep learning-based methods tackle the image-to-image translation task by learning inter-domain mappings in a paired or unpaired manner. Paired image translation methods [4, 5, 6, 7, 8, 9] exploit the inter-domain correspondence by penalizing the per-pixel residual (using l1 or l2 norm) between the output and corresponding ground-truth sample. Unpaired image translation approaches [1, 10, 11, 12, 13, 14] often use adversarial networks with an additional constraint on the image or feature space imposing structure on the underlying joint distribution of the images from the different domains.
Both paired and unpaired image translation approaches often learn a deterministic mapping between the domains where every pixel in the input domain is mapped to a fixed pixel value in the output domain. However, such a deterministic formulation can lead to mode collapse while at the same time not being able to quantify the model predictive uncertainty important for critical applications, e.g., medical image analysis. It is desirable to test the performance of the model on unseen perturbed input at test-time, to improve their applicability in the real world. While robustness to outliers is a focus in some domains [15, 16, 17, 18], it has not attracted as much attention in unpaired translation.
To address these limitations, we propose an unpaired (unsupervised) probabilistic image-to-image translation method trained without inter-domain correspondence in an end-to-end manner. The probabilistic nature of this method provides uncertainty estimates for the predictions. Moreover, modelling the residuals between the predictions and the ground-truth with heavy-tailed distributions
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
makes our model robust to outliers and various unseen data. Accordingly, we compare various state-of-the-art models and our model in their capacity to handle samples from similar distribution as training-dataset as well as perturbed samples, in the context of unpaired translation.
Our contributions are as follows. (i) We propose an unpaired probabilistic image-to-image translation framework based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC). Our framework models the residuals between the predictions and the ground-truths with heavy-tailed distributions improving robustness to outliers. Probabilistic nature of UGAC also provides uncertainty estimates for the predictions. (ii) We evaluate UGAC on multiple challenging datasets: natural images consisting Cityscapes [19], Google aerial maps and photos [4], CMP Facade [20] and medical images consisting of MRI from IXI [21]. We compare our model to seven state-of-the-art image-to-image translation methods [12, 22, 1, 11, 10, 23]. Our results demonstrate that while UGAC performs competitively when tested on unperturbed images, it improves state-of-the-art methods substantially when tested on unseen perturbations, establishing its robustness. (iii) We show that our estimated uncertainty scores correlate with the model predictive errors (i.e., residual between model prediction and the ground-truth) suggesting that it acts as a good proxy for the model’s reliability at test time.

heading: 2 Related Work
text: Image-to-image translation. Image-to-image translation is often formulated as per-pixel deterministic regression between two image domains of [24, 25, 26]. In [4], this is done in a paired manner using conditional adversarial networks, while in [10, 1, 12, 22, 11] this is done in an unpaired manner by enforcing additional constraints on the joint distribution of the images from separate domains. Both CycleGAN [10] and UNIT [1] learn bi-directional mappings, whereas other recent methods [12, 22, 11] learn uni-directional mappings.
Quantification of uncertainty in the predictions made by the unpaired image-to-image translation models largely remains unexplored. Our proposed method operates at the intersection of uncertainty estimation and unsupervised translation. Critical applications such as medical image-to-image translation [27, 28, 29, 30, 31, 32] is an excellent testbed for our model as confidence in the network’s predictions is desirable [33, 34] especially under the influence of missing imaging modalities.
Uncertainty estimation. Among two broad categories of uncertainties that can be associated with a model’s prediction, epistemic uncertainty in the model parameters is learned with finite data whereas aleatoric uncertainty captures the noise/uncertainty inherent in the data [35, 36]. For image-to-image translation, various uncertainties can be estimated using Bayesian deep learning techniques [36, 37, 38, 39, 40]. In critical areas like medical imaging, the errors in the predictions deter the adoption of such frameworks in clinical contexts. Uncertainty estimates for the predictions would allow subsequent revision by clinicians [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51].
Existing methods model the per-pixel heteroscedasticity as Gaussian distribution for regression tasks [36]. This is not optimal in the presence of outliers that often tend to follow heavy-tailed distributions [52, 53]. Therefore, we enhance the above setup by modelling per-pixel heteroscedasticity as generalized Gaussian distribution, which can model a wide variety of distributions, including Gaussian, Laplace, and heavier-tailed distribution.

heading: 3 Uncertainty-aware Generalized Adaptive Cycle-consistency (UGAC)
text: We present the formulation of the unpaired image-to-image translation problem. We discuss the shortcomings of the existing solution involving the cycle consistency loss called CycleGAN [10]. Finally, we present our novel probabilistic framework (UGAC) that overcomes the described shortcomings.

heading: 3.1 Preliminaries
text: Formulation. Let there be two image domains A and B. Let the set of images from domain A and B be defined by (i) SA := {a1, a2...an}, where ai ∼ PA ∀i and (ii) SB := {b1, b2...bm}, where bi ∼ PB ∀i, respectively. The elements ai and bi represent the ith image from domain A and B respectively, and are drawn from an underlying unknown probability distribution PA and PB respectively.
Let each image have K pixels, and uik represent the kth pixel of a particular image ui. We are interested in learning a mapping from domain A to B (A→ B) and B to A (B → A) in an unpaired manner so that the correspondence between the samples from PA and PB is not required at the learning stage. In other words, we want to learn the underlying joint distribution PAB from the given marginal distributions PA and PB . This work utilizes CycleGANs that leverage the cycle consistency to learn mappings from both directions (A→ B and B → A), but often we are only interested in one direction and the second direction is the auxiliary mapping that aids in learning process. We define the mapping A→ B as primary and B → A as auxiliary. Cycle consistency. Learning a joint distribution from the marginal distributions is an ill-posed problem with infinitely many solutions [3]. CycleGAN [10] enforces an additional structure on the joint distribution using a set of primary networks (forming a GAN) and a set of auxiliary networks. The primary networks are represented by {GA(·; θGA),DA(·; θDA )}, where GA represents a generator and DA represents a discriminator. The auxiliary networks are represented by {GB(·; θGB),DB(·; θDB )}. While the primary networks learn the mapping A→ B, the auxiliary networks learn B → A (see Figure 1). Let the output of the generator GA translating samples from domain A (say ai) to domain B be called b̂i. Similarly, for the generator GB translating samples from domain B (say bi) to domain A be called âi, i.e., b̂i = GA(ai; θGA) and âi = GB(bi; θ G B). To simplify the notation, we will omit writing parameters of the networks in the equation. The cycle consistency constraint [10] re-translates the above predictions (b̂i, âi) to get back the reconstruction in the original domain (āi,b̄i), where, āi = GB(b̂i) and b̄i = GA(âi), and attempts to make reconstructed images (āi, b̄i) similar to original input (ai, bi) by penalizing the residuals with L1 norm between the reconstructions and the original input images, giving the cycle consistency Lcyc(āi, b̄i, ai, bi) = L1(āi, ai) + L1(b̄i, bi). Limitations of cycle consistency. The underlying assumption when penalizing with the L1 norm is that the residual at every pixel between the reconstruction and the input follow zero-mean and fixed-variance Laplace distribution, i.e., āij = aij + aij and b̄ij = bij + b ij with,
aij , b ij ∼ Laplace( ; 0, σ√ 2 ) ≡ 1√ 2σ2
e− √ 2 | −0| σ , (1)
where σ2 represents the fixed-variance of the distribution, aij represents the jth pixel in image ai, and aij represents the noise in the j
th pixel for the estimated image āij . This assumption on the residuals between the reconstruction and the input enforces the likelihood (i.e., L (Θ|X ) = P(X|Θ), where Θ := θGA ∪ θ G B ∪ θDA ∪ θDB and X := SA ∪ SB) to follow a factored Laplace distribution:
L (Θ|X ) ∝ ∏ ijpq e− √ 2|āij−aij | σ e− √ 2|b̄pq−bpq| σ , (2)
where minimizing the negative-log-likelihood yields Lcyc with the following limitations. The residuals in the presence of outliers may not follow the Laplace distribution but instead a heavy-tailed distribution, whereas the i.i.d assumption leads to fixed variance distributions for the residuals that do not allow modelling of heteroscedasticity to aid in uncertainty estimation.

heading: 3.2 Building Uncertainty-aware Cycle Consistency
text: We propose to alleviate the mentioned issues by modelling the underlying per-pixel residual distribution as independent but non-identically distributed zero-mean generalized Gaussian distribution (GGD) (Figure 2), i.e., with no fixed shape (β > 0) and scale (α > 0) parameters. Instead, all the shape and scale parameters of the distributions are predicted from the networks and formulated as follows:
aij , b ij ∼ GGD( ; 0, ᾱij , β̄ij) ≡
β̄ij
2ᾱijΓ( 1 β̄ij
) e − ( | −0| ᾱij )β̄ij .
(3)
For each ij , the parameters of the distribution {ᾱij , β̄ij} may not be the same as parameters for other iks; therefore, they are non-identically distributed allowing modelling with heavier tail distributions. The likelihood for our proposed model is,
L (Θ|X ) = ∏ ijpq G (β̄aij , ᾱ a ij , āij , aij)G (β̄ b pq, ᾱ b pq, b̄pq, bpq), (4)
where (β̄aij) represents the j th pixel of domain A’s shape parameter βai (similarly for others). G (β̄uij , ᾱ u ij , ūij , uij) is the pixel-likelihood at j
th pixel of image ui (that can represent images of both domain A and B) formulated as,
G (β̄uij , ᾱ u ij , ūij , uij) = GGD(uij ; ūij , ᾱ u ij , β̄ u ij), (5)
The negative-log-likelihood is given by,
− ln L (Θ|X ) = − ∑ ijpq ln β̄aij 2ᾱaijΓ( 1 β̄aij ) e − ( |āij−aij | ᾱa ij )β̄aij + ln
β̄bpq
2ᾱbpqΓ( 1 β̄bpq
) e − ( |b̄pq−bpq| ᾱbpq )β̄bpq (6)
minimizing the negative-log-likelihood yields a new cycle consistency loss, which we call as the uncertainty-aware generalized adaptive cycle consistency loss Lucyc, given A = {āi, ᾱai , β̄ai , ai} and B = {b̄i, ᾱbi , β̄bi , bi},
Lucyc(A ,B) = Lαβ(A ) + Lαβ(B), (7)
where Lαβ(A ) = Lαβ(āi, ᾱai , β̄ai , ai) is the new objective function corresponding to domain A,
Lαβ(āi, ᾱai , β̄ai , ai) = 1
K ∑ j
( |āij − aij |
ᾱaij
)β̄aij − log
β̄aij ᾱaij + log Γ( 1 β̄aij ), (8)
where (āi, b̄i) are the reconstructions for (ai, bi) and (ᾱai , β̄ a i ), (ᾱ b i , β̄ b i ) are scale and shape parameters for the reconstruction (āi, b̄i), respectively.
The L1 norm-based cycle consistency (Lcyc) is a special case of Lucyc with (ᾱaij , β̄aij , ᾱbij , β̄bij) = (1, 1, 1, 1)∀i, j. To utilize Lucyc, one must have the α maps and the β maps for the reconstructions of the inputs. To obtain the reconstructed image, α (scale map), and β (shape map), we modify the head of the generators (the last few convolutional layers) and split them into three heads, connected to a common backbone. Therefore, for inputs ai and bi to the generator GA and GB , the outputs are:
(b̂i, α̂ b i , β̂ b i ) = GA(ai) and (āi, ᾱai , β̄ai ) = GB(b̂i)
(âi, α̂ a i , β̂ a i ) = GB(bi) and (b̄i, ᾱbi , β̄bi ) = GA(âi), (9)
The estimates are plugged into Eq. (7) and the networks are trained to estimate all the parameters of the GGD modelling domain A and B, i.e. (āij , ᾱaij , β̄ a ij) and (b̄ij , ᾱ b ij , β̄ b ij) ∀ij.
Furthermore, we apply adversarial losses [10] to the mapping functions, (i) GA : A → B and (ii) GB : B → A, using the discriminators DA and DB . The discriminators are inspired from patchGANs [4, 10] that classify whether 70x70 overlapping patches are real or not. The adversarial loss for the generators (LGadv [10]) is,
LGadv = L2(DA(b̂i), 1) + L2(DB(âi), 1). (10)
The loss for discriminators (LDadv [10]) is,
LDadv = L2(DA(bi), 1) + L2(DA(b̂i), 0) + L2(DB(ai), 1) + L2(DB(âi), 0). (11)
To train the networks we update the generator and discriminator sequentially at every step [10, 4, 54]. The generators and discriminators are trained to minimize LG and LD as follows:
LG = λ1Lucyc + λ2LGadv and LD = LDadv. (12)
Closed-form solution for aleatoric uncertainty. Although predicting parameters of the output image distribution allows to sample multiple images for the same input and compute the uncertainty, modelling the distribution as GGD gives us the uncertainty (σaleatoric) without sampling from the distribution as a closed form solution exists, σ2aleatoric = α2Γ( 3β )
Γ( 1β ) . Epistemic uncertainty (σepistemic) is
calculated by multiple forward passes (T = 50 times) with dropouts activated for the same input and computing the variance across the outputs (ût), i.e., σ2epistemic = ( ∑ t(ût − ∑ t ût T )
2)/T . We define the total uncertainty (σ) as σ2 = σ2aleatoric + σ 2 epistemic.

heading: 4 Experiments
text: In this section, we first describe our experimental setup and implementation details.We compare our model to a wide variety of state-of-the-art methods quantitatively and qualitatively. Finally we provide an ablation analysis to study the rationale of our model formulation.

heading: 4.1 Experimental Setup
text: Tasks. We study the robustness of unpaired image-to-image translation methods, where different methods are first trained on clean images and then evaluated on perturbed images The clean images are referred as noise-level 0 (NL0); while the perturbed images with increasing noise are referred as NL1, NL2, and NL3. We test three types of perturbation including Gaussian, Uniform, and Impulse. From NL0 to NL3, the standard deviation of the additive Gaussian noise is gradually increased. Similarly, for additive uniform noise, different levels are obtained by gradually increase the upper-bound of the uniform sampling interval [55] and for impulse noise we gradually increase the probability of pixel-value replacement [56]. Details of constructing various NLs are in supplementary.
Datasets. We evaluate on four standard datasets used for image-to-image translation: (i) Cityscapes [19] contains street scene images with segmentation maps, including 2,975 training and 500 validation and test images; (ii) Google maps [4] contains 1,096 training and test images scraped from Google maps with aerial photographs and maps; (iii) CMP Facade [20] contains 400 images from the CMP Facade Database including architectural facades labels and photos. (iv) IXI [21] is a medical imaging dataset with 15,000/5,000/10,000 training/test/validation images, including T1 MRI and T2 MRI. More preprocessing details for all the datasets are in supplementary.
Translation quality metrics. Following [23], we evaluate the translation quality of the generated segmentation maps and images, for the datasets with segmentation maps (e.g., Cityscapes). First, to evaluate the generated segmentation maps, we compute the Intersection over union (IoU.SEGM) and mean class-wise accuracy (Acc.SEGM) between the generated segmentation maps and the groundtruth segmentation maps. Second, to evaluate the generated images, we first feed the generated images Xtr to a pre-trained pix2pix model [4] (denoted as p2p, which is trained to translate images to segmentation maps) to obtain the segmentation maps p2p(Xtr). Then, we feed the original images Xorg to the same pix2pix model to obtain another segmentation maps p2p(Xorg), and compute the IoU between two outputs p2p(Xtr) and p2p(Xorg) (IoU.P2P).
Metrics for model robustness. We define two metrics similar to [23] to test model robustness towards noisy inputs. (i) AMSE is the area under the curve measuring the MSE between the outputs of the noisy input and the clean input under different levels of noise, i.e., AMSE = ∫ ηmax ηmin
(MSE(GA(ai + η),GA(ai)))dη, where η is the noise level, GA denotes the generator that maps domain sample ai (from domain A) to domain B. (ii) ASSIM is the area under the curve measuring the SSIM [57] between the outputs of the noisy input and the clean input under different levels of noise, i.e., ASSIM = ∫ ηmax ηmin
(SSIM(GA(ai + η),GA(ai))dη. These two metrics show how much the output deviates when fed with the corrupted input from the output corresponding to clean input, averaged over multiple corruption/noise levels. Computational details are in supplementary.
Implementation details. In our framework, the generator is a cascaded U-Net that progressively improves the intermediate features to yield high-quality output [30], we use a patch discriminator [4]. All the networks were trained using Adam optimizer [58] with a mini-batch size of 2. The initial learning rate was set to 2e−4 and cosine annealing was used to decay the learning rate over 1000 epochs. The hyper-parameters, (λ1, λ2) (Eq. (12)) were set to (10, 2). For numerical stability, the proposed network produces 1α instead of α. The positivity constraint on the output (for predicted α, β) is enforced by applying the ReLU at the end of the output layers in the network. The architecture details and the training scheme are in supplementary.

heading: 4.2 Comparing with the State of the Art
text: Compared methods. We compare our model to seven state-of-the-art methods for unpaired imageto-image translation, including (1) distanceGAN [59] (disGAN): a uni-directional method to map different domains by maintaining a distance metric between samples of the domains. (2) geometry consistent GAN [22] (gcGAN): a uni-directional method that imposes pairwise distance and geometric constraints. (3) UNIT [1]: a bi-directional method that matches the latent representations of the two domain. (4) CUT [11]: a uni-directional method that uses contrastive learning to match the patches in
the same locations in both domains. (5) CycleGAN [10] (Cy.GAN): a bi-directional method that uses cycle consistency loss. (6) guess CycleGAN [23]: a variant of CycleGAN that uses an additional guess discriminator that “guesses" at random which of the image is fake in the collection of input and reconstruction images. (7) adversarial noise CycleGAN [23] (nCy.GAN): another variant of CycleGAN that introduces noise in the cycle consistency loss. Note that both guess CycleGAN [23] and adversarial noise CycleGAN [23] improve the model robustness to noise.
Quantitative evaluation. As described in Section 4.1, we trained the models using the clean images (NL0) and evaluated them at varying noise levels (NL0, NL1, NL2, NL3), results are detailed next.
Figure 3 shows the quantitative results on Cityscapes dataset with Gaussian perturbation. When increasing the noise levels, we observe that the performance of compared methods degrade significantly, while our method remains more robust to noise – e.g., the mean IoU.SEGM values are changed from around 0.24 to 0.2 for our model but degrades from around 0.24 to 0.05 for the baseline Cy.GAN. Similarly, our model outperforms two strong competitors (gCy.GAN, nCy.GAN) that are built to defend noise perturbation on higher noise levels. Similar trends are observed for other datasets (in supplementary). This indicates that our model offers better translation quality at higher noise levels.
To evaluate model robustness, we tested different methods using the metrics AMSE and ASSIM to quantify the overall image quality under increasing noise levels as defined in Section 4.1. Table 1 shows the performance of all the models on different datasets for three types of perturbations, i.e., Gaussian, Uniform, and Impulse. We can see that the proposed UGAC model performs better
than other methods. For instance, when adding Gaussian noise, UGAC obtains a much better ASSIM of 0.78/0.82/0.77/0.87 vs. 0.60/0.68/0.68/0.77 yielded by the best competitor nCy.GAN on Cityscapes/Maps/Facade/IXI. When adding Uniform noise or Impulse noise, we can also find that our model outperforms the other methods by substantial margins. Overall, the better performance of UGAC on different datasets suggests its stronger robustness towards various types of perturbations.
Qualitative results. Figure 4 visualizes the generated output images for Cityscapes, Google Maps, CMP Facade, and IXI datasets where all the models are trained with clean images and tested with either clean images or perturbed images. The test-time perturbation is of type Gaussian and corresponds to noise-level NL2. We see that, while all the methods generate samples of high quality when tested on unperturbed clean input; whereas when tested with perturbed inputs, we observe results with artifacts but the artifacts are imperceptible in our UCAC method.
The results on Cityscapes dataset (with the primary direction, translating from segmentation maps to real photo) demonstrate that with perturbed input, methods such as disGAN, gcGAN, UNIT generate images with high frequency artifacts (col.1 to 3), whereas methods such as CUT, Cy.GAN, gCy.GAN and nCy.GAN (col.4 to 7) generate images with low frequency artefacts. Both kinds of artefact lead to degradation of the visual quality of the output. Our method (col.8) generates output images that still preserve all the high frequency details and are visually appealing, even with perturbed input. Similar trends are observed for other datasets including Maps (with primary translation from maps to photo) and Facade (with primary translation from segmentation maps to real photo).
For the IXI dataset (with primary translation from T1 to T2 MRI scans), we observe that the other models fail to reconstruct medically relevant structures like trigeminal-nerve (in the centre) present in the input T1 MRI scans. Moreover, high-frequency details throughout the white and grey matter in the brain are missing. In contrast, our method gracefully reconstructs many of the high-frequency details. More qualitative results are in supplementary, with similar trends as in Figure 4. It shows that our model is capable of generating images of good quality at higher noise levels.

heading: 4.3 Analyzing the Model Uncertainty
text: Evaluating the generalized adaptive norm. We study the performance of our method by modelling the per-pixel residuals in three ways on IXI dataset. First, i.i.d Gaussian distribution, i.e.,
(αij , βij) is manually set to (1, 2)∀i, j, which is equivalent to using fixed l2 norm at every pixel in cycle consistency loss (Lαβ |α=1,β=2). visual quality when given perturbed input. Second, i.i.d Laplace distribution, i.e., (αij , βij) is manually set to (1, 1)∀i, j, which is equivalent to using fixed l1 norm at every pixel in cycle consistency loss (Lαβ |α=1,β=1). Third, independent but non-identically distributed generalized Gaussian distribution (UGAC), which is equivalent to using spatially varying lq quasi-norms where q is predicted by the network for every pixel (Lαβ |pred).
Fig 5 shows the quantitative performance of these three variants across different noise levels for IXI datasets. We see that spatially adaptive quasi-norms perform better than fixed norms, even at higher noise levels (i.e., presence of outliers). Note
that our GGD based heteroscedastic model subsumes the Gaussian (α = 1, β = 2) and Laplacian (α = 1, β = 1). Moreover, the heteroscedastic versions of Gaussian and Laplacian can be obtained by fixing β, i.e., for Laplacian (β = 1) and for Gaussian (β = 2), and varying α. Modeling residuals as GGD is more liberal than both homo/hetero-scedastic Gaussian/Laplacian distribution because it is able to capture all the heavier/lighter-tailed distributions (along with all possible Gaussian/Laplacian distributions) that are beyond the modeling capabilities of Gaussian/Laplacian alone.
Visualizing uncertainty maps. We visualize our uncertainty maps for the T1w MRI (domain A) to T2w MRI (domain B) translation task, on IXI dataset, with perturbations in the input (NL3).
Figure 6-(a) shows input axial slices (T1w at NL3). The perturbations have degraded the highfrequency features (see green ROI). Figure 6-(b) shows the corresponding ground-truth axial slice (T2w MRI). Figure 6-(c) shows that our method recovers high-frequency details. However, we
observe a higher contrast (compared to ground-truth) (green ROI). The subtle disparity between the contrast has been picked up by our scale-map (α) and shape-map (β) as shown in Figure 6-(d) and (e), respectively. Moreover, we see that, although our formulation assumes independent (but non-identically) likelihood model for the pixel level residuals, the structure in the α and the β (Figure 6-(d) and (e)) shows that the model learns to exploit the correlation in the neighbourhood pixels. The pixel-level variation in the α and β yields pixel-level uncertainty values in the predictions as described in Section 4.1.
Figure 6-(f) shows the uncertainty map (σ) for the predictions made by the network. We see that the disparity in the contrasts between the prediction and the ground-truth is reflected as high uncertainty in the disparity region, i.e., uncertainty is high where the reconstruction is of inferior quality, indicated by high-residual values shown in Figure 6-(g). The correspondence between uncertainty maps (Figure 6-(f)) and residual maps (Figure 6-(g)) suggests that uncertainty maps can be used as the proxy to residual maps (that are unavailable at the test time, as the ground-truth images will not be available) and can serve as an indicator of image quality. More samples showing the translation results with high uncertainty (σ estimated using α, β) are shown in supplementary.
Residual scores vs. uncertainty scores. To further study the relationship between the uncertainty maps and the residual maps across a wide variety of images, we analyze the results on
IXI test set. We show the density and the scatter-plot between the residual score and uncertainty score in Figure 7, where every point represents a single image. For an image, the mean residual score (on the y-axis) is derived as the mean of absolute residual values for all the pixels in the image. Similarly, the uncertainty score (on the x-axis) is calculated as the mean of uncertainty values of all the pixels in that image. From the plot, we see that across the test-set mean uncertainty score correlates positively with the mean residual score, i.e., higher uncertainty score corresponds to higher residual. An image with higher residual score represents a poor quality image. This further supports the idea that uncertainty maps derived from our method can be used as a proxy to residual that indicates the overall image quality of the output generated by our network.

heading: 5 Discussion and Conclusion
text: In this work, we propose an uncertainty-aware generalized cycle consistency for unpaired image translation along with uncertainty estimation. Our formulation assumes the pixel-wise independent (but non-identically) distributed likelihood model for the residuals, relaxing the i.i.d. assumption made by the previous work. However, our experiments also show that the model learns the underlying structure between the neighbourhood pixels and predicts the structured/correlated parameters for the output distribution (i.e., α, β for MRI translation shown in Figure 6-(d) and (e)).
We demonstrate the efficacy of the proposed method on robust unpaired image translation on various datasets spanning autonomous driving, maps, facades, and medical images consisting of MRI scans. We also demonstrate the robustness of our method by evaluating the performance in different kinds of perturbations in the input with varying severity and show that our method outperforms all the baselines by generating superior images in terms of quantitative metrics and appearance. In addition,
we show that the uncertainty estimates (derived from the estimated parameters of the distribution, α and β) are faithful proxy to the residuals between the predictions and the ground truth.
It is worth noting that robustness towards various kinds of perturbation can also be achieved by data augmentation techniques that include the perturbed images in the training phase. However, this is orthogonal to the concept proposed in this work that achieves robustness via the a new modeling technique. In principle, one could combine both the augmentation techniques and modeling techniques to obtain more robust models. In this work, we used relatively small neural networks (in terms of parameters based on UNet), while this network has not been used previously for this problem, we employ it to train our models with limited compute with reasonable training time and a lower memory footprint (details of the networks available in the Appendix A.5). This however affects the performance of the networks, and leads to images with artifacts/distortions (specially with small datasets consisting few hundred samples). Our method can be applied to deeper neural networks with more parameters/higher capacity and trained with higher resolution images, which would lead to significantly better performance, given enough compute.
An interesting avenue for further exploration is the analysis of uncertainty maps when presented with anomalous inputs, beyond perturbations, with stronger shifts between training and test data distribution which will be investigated in future.
Broader Impact
Modern deep-learning-based image translation schemes are becoming more popular. They allow the generation of synthetic datasets, e.g., for the segmentation use case in autonomous driving, faster image acquisition via algorithmic super-resolution, image enhancement in computational photography, faster and cheaper medical diagnosis by translating between different imaging modalities. However, critical areas like medical imaging and autonomous driving require methods that are robust towards various perturbations and, at the same time, can also provide uncertainty estimates in the predictions. Estimating the uncertainty in the prediction can help trigger expert intervention preventing fatal scenarios.
We introduced a novel image-to-image translation model capable of estimating uncertainty along with the predictions and is shown to be beneficial in ensuring good image translation quality and good model performance on downstream tasks even in the presence of unseen noisy patterns in input images at inference time. Furthermore, our method is potentially applicable to detect ambiguities in the images. These merits could bring positive societal impacts to various critical application domains, such as medical imaging and autonomous driving.
Acknowledgements This work has been partially funded by the ERC (853489 - DEXIM) and by the DFG (2064/1 – Project number 390727645). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Uddeshya Upadhyay.

Review summaries:
This paper introduces a new method for unpaired image to image translation that addresses uncertainty estimation and robustness. The method is based on an adaptive loss function that accounts for spatially-varying noise levels. This loss function motivated by Bayesian modelling and it is incorporated in a cycle-consistent loss. The experimental setting consists in training the proposed method and other competitors on clean data and test it on noisy data. The method is evaluated on different dataset including street scene images and medical images. According to a measure of performance on noisy data, the method is shown to outperform existing methods. Last, the method outputs uncertainty maps which are the parameters of the spatially adaptive loss function, these are shown to positively correlate with the residual losses.

The paper achieves improved and more robust unpaired image-to-image translation results by modeling uncertainty of the outputs. The proposed method generalized and treats the original pixel cycle loss of CycleGAN as the special case in which the uncertainty distribution is uniformly fixed. Furthermore, the predicted uncertainty can be used as a way to measure risk at inference time. 

Authors propose a relaxation of the cycle consistency loss that takes into account per-pixel prediction uncertainty. More specifically, authors assume that cycle-reconstruction residuals are distributed as iid generalized Gaussian random variables, and maximize likelihood of these reconstruction residuals given predicted per-pixel parameters. Authors report two metrics of sensitivity to noise and three segmentation accuracy metrics across four real datasets with known GT and seven baseline methods, confirming that in all cases their method outperforms or performs on par with SotA. Authors also show that restricting parameters of their generalized Gaussian log-likelihood loss to constant values (effectively reducing it to standard CycleGAN) degrades the performance of their method according to these performance metrics. Moreover, authors showed that the overall prediction uncertainty strongly correlates with the magnitude of the true prediction error computed using GT.