What's Behind the Mask: Estimating Uncertainty in Image-to-Image Problems
Authors: Gilad Kutiel
Regev Cohen
Michael Elad
Daniel Freedman
Venue: Submitted to ICLR 2023 (2023)
Keywords: uncertainty estimation
image-to-image

Abstract: Estimating uncertainty in image-to-image networks is an important task, particularly as such networks are being increasingly deployed in the biological and medical imaging realms.  In this paper, we introduce a new approach to this problem based on masking.  Given an existing image-to-image network, our approach computes a mask such that the distance between the masked reconstructed image and the masked true image is guaranteed to be less than a specified threshold, with high probability.  The mask thus identifies the more certain regions of the reconstructed image.  Our approach is agnostic to the underlying image-to-image network, and only requires triples of the input (degraded), reconstructed and true images for training.  Furthermore, our method is agnostic to the distance metric used.  As a result, one can use $L_p$-style distances or perceptual distances like LPIPS, which contrasts with interval-based approaches to uncertainty.  Our theoretical guarantees derive from a conformal calibration procedure.  We evaluate our mask-based approach to uncertainty on image colorization, image completion, and super-resolution tasks, demonstrating high quality performance on each.

Under review as a conference paper at ICLR 2023
WHAT'S BEHIND THE MASK: ESTIMATING UNCERTAINTY IN IMAGE-TO-IMAN PROBLEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Estimating uncertainty in image-to-image networks is an important task, particu- larly as such networks are being increasingly deployed in the biological and med- ical imaging realms. In this paper, we introduce new approach this problem based on masking. Given an existing image-to-image network, our approach com- putes mask such that the distance between the masked reconstructed image and the masked true image is guaranteed to be less than a specified threshold, with high probability. The mask thus identifies the more certain regions of the re- constructed image. Our approach is agnostic to the underlying image-to-image network, and only requires triples of the input (degraded), reconstructed and true images for training. Furthermore, our method is agnostic the distance met- ric used. As a result, one can use -y-st distances or perceptual distances like LPIPS, which contrasts with interval-based approaches uncertainty. Our theo- retical guarantees derive from a conformal calibration procedure. We evaluate our mask-based approach uncertainty on image colorization, image completion, and super-resolution tasks, demonstrating high quality performance on each.
INTRODUCTION
Deep Learning has been very successful in many applications, spanning computer vision, speech recognition, natural language processing, and beyond. For many years, researchers were mainly content to develop new techniques to achieve unprecedented accuracy, without concern for under- standing the uncertainty implicit such models. More recently, however, there has been concerted effort within the research community to understand and quantify the uncertainty of deep models.
This paper addresses the problem of estimating uncertainty in the realm of image-to-image (some- times referred to as image reconstruction) tasks. Such tasks include super-resolution, deblurring, colorization, and image completion, amongst others. Computing the uncertainty is important gener- in application domains such biological and medical imaging, in which
ally, but is particularly fidelity to the ground truth is paramount, If there is an area of the reconstructed image where such fidelity unlikely or unreliable due high uncertainty, this crucial to convey.
Our approach to uncertainty estimation based on masking. Specifically, we are interested in the possibility of computing mask such that the uncertain regions in the image are masked out. More formally, we would like the distance between the masked reconstructed image and the masked true image be small in expectation. Ideally, the method should be agnostic to the choice of distance function, which should be dictated by the application. A high level overview of our approach is illustrated in Figure
We show a direct connection between the mask and theoretically well-founded definition of un- certainty that matches image-to-image tasks. We then derive an algorithm for computing such a mask which can apply to any existing (i.e. pre-trained) image-to-image network, and any distance function between image pairs. All that is required for training the mask network is triplets of the input (degraded), reconstructed and true images. Using procedure based on conformal prediction (Angelopoulos & Bates 2021a), we can guarantee that the masks so produced satisfy the following criterion: the distance between the masked reconstructed image and the masked true image is guar- anteed to be less than a specified threshold, with high probability. We demonstrate the power of the method image colorization, image completion, and super-resolution tasks.

---

Under review as a conference paper at ICLR 2023
Colorization Model
y
y
8
778
Figure 1: A high level view of approach: is the degraded input (colorization in this example) to the regression model. x and ÿ (the colorized image) serve as the input to the masking model which predicts the uncertainty mask The ground-truth image is denoted by y. With high probability, the distortion between the masked images, y @ m and O m is low.
Our contributions are as follows:
We present approach to uncertainty computation based masking. We show that mov- ing from binary to continuous masks preserves the connection with a theoretically well- founded definition of uncertainty that is relevant for image-to-image tasks.
We derive an algorithm for computing the masks which works for arbitrary image-to-image networks, and any distance function between image pairs. The masks are guaranteed to yield small distance between masked ground truth and reconstructed images, with high probability.
We demonstrate the effectiveness of the method on image colorization, image completion, and super-resolution tasks attaining high quality performance on each.
NOTATIONS
Throughout the paper we represent an image as a column-stacked vector & E R", We use &k to denote the kth image collection, while T(t marks the ith entry of a. We define E T and T E RR to be vectors of all zeros and ones respectively. The operation @ y stands for the Hadamard (point-wise) product between x and y. For 2 1, we denote the fp-norm of by |ا 2 En [.) The symbol d(.,-) represents general distortion measure between two images. We define a continuous mask as a vector m E [0, 1]", where the average size of a mask equals | / such that the mask (no-masking) has size 0 while the mask 0 has size of Given a mask m, we define dm (æ, y) & d(m O I, m @ y) as the distortion measure between the masked versions of images x and y. For natural number n 2 we define the set [n] a {1,..., n).
RELATED WORK
Bayesian Uncertainty Quantification The Bayesian paradigm defines uncertainty by assuming a distribution over the model parameters and/or activation functions. The most prevalent approach is Bayesian neural networks (MacKay) 1992] Valentin Jospin et al.] 2020|Izmailov Izmallov et al.) (2020). which stochastic models trained using Bayesian inference. Yet, the number of model parameters has
2

---

Under review as a conference paper at ICLR 2023
grown rapidly, computing the exact posteriors has became computationally intractable. This short- coming has led to the development of approximation methods such as Monte Carlo dropout (Gal & Ghahramani] 2016 Gal al. 2017a), stochastic gradient Markov chain Monte Carlo (Salimans et al.] 2015] Chen et al. 2014), apllccian approximations (Ritter et al. 2018) and variational infer-
ence (Blundell et al. 2015 Louizos & Welling 2017 Posch et al. 20T9). Alternative Bayesian tech-
niques include deep Gaussian processes (Damianou & Lawrence 2013). deep ensembles (Ashukha)
al. 2020; Hu et al. 2019), and deep Bayesian active learning (Gal et al. 2017b), to name just
few. For a recent comprehensive review on Bayesian uncertainty quantification we refer the readers
to Abdar et al. (202T)
Distribution-Free Methods and Conformal Prediction Unlike Bayesian methods, the frequentist
approach assumes the true model parameters are fixed with no underlying distribution. Examples
of such distribution-free techniques are model ensembles (Lakshminarayananl et al.) 2017 Pearce
all.
2018), bootstrap (Kim et al.] | Alaa Van Der Schaar, 2020), interval regression Pearce
ee al. 2018; Kivaranovic et al. 2020 Wu et al.) 1 and quantile regression (Gasthaus et al.
2019
Romano et al. 2019). An important distribution-free technique which is most relevant to
work is conformal prediction (Angelopoulos & Bates 2021b Shafer & Vovk 2008). The con- formal approach relies on a labeled calibration dataset to convert point estimations into prediction regions. Conformal methods
be used with any estimator, require no retraining, are compu- tationally efficient and provide coverage guarantees in finite samples (Lei et al. 2018). Recent development includes conformalized quantile regression (Romano et al.] 2019 Sesia & Candès Angelopoulos et al. 2022b), conformal risk control (Angelopoulos et al. 2022a] Bates et al,
2020
2021
Angelopoulos et al., 2021 and semantic uncertainty intervals for generative adversarial net-
works (Sankaranarayanan et al. 2022 Sun 2022) provides an extensive survey on distribution-frem
conformal prediction methods.
Uncertainty for Image-to-image Regression Tasks A work that stands out in this realm is (An- gelopoulos et al. 2022b). which experiments with the conformal prediction technique. They eval- uate and compare several models that address both the prediction of value per each pixel, and an accompanying confidence interval that should contain the true value. After calibration, a guaran- tee is provided such that with high probability the true values of the pixels are indeed within these predicted confidence intervals. They find that quantile regression gives the best results among the four methods considered. As work addresses the same breed of image-to-image problems, provide extensive comparison to this work.
MASK-BASED UNCERTAINTY ESTIMATION
3.1 PROBLEM FORMULATION
We consider the problem of recovering an image E [0, 1J" from its observations x E [0, 1| and denote by (xx) E [0, lh given estimator of y. We define the uncertainty of ş given æ as follow:
U(] A Eygz [d(v, ú)],
where -): denotes an arbitrary distortion measure between the ground truth image y and its esti- mation Φ. and E is the expectation over the random variable y|æ. Ideally, we would like to design g such that
P P(U(O))sa)28
(2)
where 0 S and 0 < < 1 are user-defined, and the probability P(-) is computed over the joint
probability of the problem pairs (y, x)). However, in practice, condition (2) might not hold. Hence, our goal is to identify areas within ý which have the most adverse effect on the uncertainty and lead
to the violation of (2). Formally, we aim to construct an uncertainty map given by continuous mask me (0, 1]"d such that
p (Eyy (v. 1) S a 3
where dm (y. 9) d(m O y. m @ 9). Notice we must avoid trivial and undesired solutions, e.g. O, which satisfy (3) but provide true information regarding the uncertainty. To that end,
3

---

Under review as a conference paper at ICLR 2023
consider solving the following problem
max |mm]] subject
Eyla dm (y. 0) S a,
(P1)
o S m - T.
In words, wish to find the minimal masking required satisfy the distortion constraint. Notice that Problem (PI) does not depend on as we consider an instance regression problem for a given and originating y. We bring back the probabilistic condition using in the next section, when addressing the whole image manifold.
Under certain conditions, the optimal solution is readily given in closed-form expression as shown by the next theorem.
Theorem 1. Consider Problem P1] with the distortion measure d(y, 1) -
0ll where p 1. Then, for sufficiently small o, the optimal mask m admits closed-form solution given by
4(3)
mie = 0F
(3)1())
where for all [n]]
(()) A
(Bylell0b) - (6)1p)) 1/(-1))
Proof. See Appendix C
Note that the above implies the following:
Corollary 1. The optimal mask, as defined by the solution of PI is directly related to the uncer- tainry measure Equation (7) via
U(il)) x
(mzg)) 10-1)
1
Proof.
n
n
Σ (mp))-(--1))
(0-1) α E(a()) -
Bytellii - M6) = U (t1a)).
Thus, the above establishes a correspondence between the optimal mask and uncertainty as defined in (T). Furthermore, it can be shown that the result of Theorem can be extended p 1. leading to binary mask (see Appendix A).
The result Teeveem[[serves only as general motivation our work, since obtaining the optimal mask directly via the above equations is impractical for a number of reasons. First, a may vary depending on the user and the application. Second, for arbitrary distortion measure Problem (PI) does not admit a closed-form solution. Last and most importantly, at inference time the ground truth y is unknown. Hence, our next step is to provide an estimation the mask which requires no knowledge of y. can be computed for any differentiable distortion measure, and is easily adapted for various values of a and B.
3.2 METHOO
Here, aim to design estimator for the the uncertainty mask defined by Problem PI). We start by modeling the mask as a neural network mo(#, ÿ) with parameters 0, which we refer to as
the masking model (see details in Section 4.2p Similar Problem (PT), we define the following optimization problem
min [ Ø) Tll
subject to
Ey [dmo (u,, 9) S a.
(P2)
σ < mo (x. 0) S T..
4

---

Under review as a conference paper at ICLR 2023
Ignoring (for now) the second constraint, the Lagrangian of Problem P2
is given by
L(O, ()) [ 9) - T12 + / [Eyl (v. 0) a
(4)
where μ 0 is the dual variable which is considered an hyperparameter. Given μ,, the optimal mask can be found by minimizing L(0, (4) with respect to A, which is equivalent to minimizing
|| (z, 0) - 1112 + pul a (dmo (v. ü)]
(5)
since a does not depend on 0. Thus, motivated by the above, we train our masking model using the following loss function:
((D,))
E [mm (..0.))D
I11 pd (v. 9)
(6)
where D & {( Dk } is a dataset of triplets of the degraded input, recovered output and the true image respectively. To enforce that the mask is within [0, 1] we simply clip the output of masking model. Note that the proposed approach facilitates the use of any differentiable distortion measure. Furthermore, at inference time, masking model provides approximation of the optimal uncertainty mask - the solution of (PI) without requiring y.
Following the above, notice that the loss function is independent of and B. hence, cannot guarantee that my satisfies condition (3). To overcome this, we follow previous work on conformal prediction and consider the output of our masking model initial estimation of the uncertainty which can be calibrated to obtain strong statistical guarantees. In general, the calibrated mask ma follows the form
mx(() F(ma(t): A); Vi = 1,..,,
where F(; A) is monotonic non-decreasing function, parameterized by global scalar which is tuned throughout the calibration process. As our goal is to reduce the distortion below a prede- fined fixed level, the calibrated mask need be inversely proportional the distortion. Since our predicted mask is proportional to the distortion we use the following formula"
A
mx(i) c c+1-mat)
Vi 1, 1,..,.... n
which found empirically to perform well in our experiments. To set the value of À we assume given calibration dataset {(2xx Uk. y/)hh and perform the following calibration procedure:
1. For each pair (Ix, 0v)) we predict a mask m. = =mg (t..
2. We define calibrated mask mAr according (8) where A the maximal value such that
d (mxx y. mA ②) a.
3. Given all }. set À to be the 1 - 3 quantile of {A+}, i.c the maximal value for which at least 3 fraction of the calibration set satisfies condition (PI).
The final mask, calibrated according to a and B, is guaranteed to satisfy condition above.
The benefits of masking approach as follows. First, it provides measure of uncertainty which can be easily interpreted. In addition, our training process can accept any distortion measure and learns the relationships and correlations between different pixels in the image. Note thatour model is trained only once, irrespective of the values of a and 8. Thus, our model can be easily adapted for different values of and 3 via our simple calibration process. Finally, at inference, can produce our uncertainty map without the knowledge of the ground truth while still providing strong statistical guarantees on our output mask.
We add small value, e, to the denominator for numerical stability and clip the value of m to he in (0. 1] formal description of our algorithm can he seen in Appendix B
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASETS AND TASKS
Datasets Two data-sets are used in our experiments:
Places365 (Zhou al. 2017): A large collection of 256x256 images from 365 scene categories. We use 1,803,460 images for training and 36,500 images for validation/test.
Rat Astrocyte Cells (Ljosa et al., 2012): A dataset of 1,200 uncompressed images of scanned rat cells of resolution x We crop the images into x56x tiles, and randomly split them into train and validation/test sets of sizes 573,744 and 11,621 respectively. The tiles are partially overlapped as we use stride of 32 pixels when cropping the images.
Tasks
We consider the following pagge-toimmaee tasks:
Completion: Using grayscale version of Places365, we remove middle vertical and horizontal stripe of 32 pixel width, and aim reconstruct the missing part.
Super Resolution: We experiment with this task on the two data-sets. The images are scaled down to 64 64 images and the goal is to reconstruct the original image.
Colorization: We convert the Places365 images to grayscale and aim to recover their colors.
Figure shows sample input and output for each of the above tasks.
4.2 EXPERIMENTAL DETAILS AND SETTINGS
Image-to-Imare Models We start by training models for the above three tasks. Note that these models not intended to be state of the art, but rather used to demonstrate the uncertainty estima- tion technique proposed in this work. We use the same model architecture for all tasks: an 8 layer U-Net. For each task we train two versions of the network: (i) A simple regressor; and (ii) A con- ditional GAN, where the generator plays the role of the reconstruction network. For the GAN, the discriminator implemented as 4 layer CNN. We use the LI loss the objective for the regressor, and augment it with an adversarial loss in the conditional GAN (A 20). as in Isola et al. (017)). All models are trained for ten epochs using the Adam optimizer with learning rate equal 1c-5 and batch size equal to 50.
Mask Model For our masking model also use an 8 layer U-Net architecture. This choice was made for simplicity and compatibility with previous work. The input for the mask model is the concatenation of the degraded image and the predicated image on the channel axis. It's output is mask having the same shape as the predicted image, with values in the range [0, The masking model trained using the loss function described in Section 3.2 with μ 2, learning rate of 1e-5 and batch size of 25.
Experiments We consider the LI, L2, SSIM and LPIPS distances. We set aside 1, 1,000 samples from each validation set for calibration and use the remaining samples for evaluation. We demon- strate the flexibility of our approach by conducting experiments on a variety of 12 settings: (i) Image Completion: (Regressor, GAN) x (L1, LPIPS); (ii) Super Resolution: (Regressor, GAN) x (L1, SSIM); and (iii) Colorization: [Regressor, GAN)- x L2).
Thresholds Recall that given a predicted image, goal is find mask that, when applied to both the prediction and the (unknown) reference image, reduces the distortion between them to a predefined threshold with high probability B3. Here fix 3 0.9 and choose the threshold to be the 0. lqquante of each measure computed on a random sample from the validation set, i.c. roughly 10% of the predictions already considered sufficiently good and do not require masking at all.
4.3 COMPETITOR TECHNIQUES
Quantile - interval-Based Technique We compare our method to the quantile regression option presented in (Angelopoulos et al. 2022b). As described in Section 2 their uncertainty is formulated by pixel-wise wise confidence intervals, calibrated so as to guarantee that with high probability the true value of the image is indeed within the specified range. While their predicted confidence intervals

---

Under review as a conference paper at ICLR 2023
ANGRO
ENCAIIIN
Exss
BAGARASIT
SCOTTI8N BAGACRERT
Figure 2: Example of uncertainty masks for image completion (top), colorization (middle) and super resolution (bottom): the images from left to right are the input to the model, the reference image, the output of the model, the calibrated uncertainty mask trained on the LI loss and the L1 distance map. In the image completion task the bottom left corner is richer in details and thus there is high uncertainty regarding this part the reconstructed image. For the colorization task, the clothes and the colored area of the bus are the most uncertained regions respectively these regions can be colorized with large variaty of colors. In the super resolution task the uncertained regions are around the edges and text while smooth surfaces are more certain parts. Note that the actual error (right most image) is not equivalent to our definition of uncertainty and we only present it as a reference.
are markedly different from the expected distortion we consider, we can use these intervals as heuristic mask. For completeness, we also report the performance of the quantile regression even when it is less suitable, i.e. when the underlying model is GAN and when the distortion loss is different from LI. We note again that for the sake of a fair comparison, our implementation of the masking model uses exactly the same architecture as the quantile regressor.
Opt Oracle We also compare our method with an oracle, denoted Opt, which computes an optimal mask using the following optimization procedure that has access to the reference image y:
{(m) min Il - 712 + pdm (y. 9)-
This minimization is obtained via gradient descent, using the Adam optimizer with a step-size 0.01, iterating until the destination distortion value is attained. Clearly, this approach does not require calibration, as the above procedure ensures that all the masked samples are below the target distortion threshold.
4.4 RESULTS AND DISCUSSION
We now show series of results that demonstrate our proposed uncertainty masking approach, and its comparison with Opt and Quantile We begin with representative visual illustration of our proposed mask for several test cases, see Figure 2 As can be seen, the mask aligns well with the true prediction error, while still identifying sub-regions high certainty.
In Figure 3 we explore the performance of the three compared methods with respect to (i) the prob- abilistic guarantee, (ii) the size of the masks obtained, and (iii) the correlation between mask sizes
Due space limitations, we show more extensive experimental results in Appendix B, while presenting selected portion of them here.
7

---

Under review as a conference paper at ICLR 2023
[M] Ours
C(M. D) (1)
C(M.
Mopt) (1)
Network
Distance
Opt
Quantile
Ours
Quantile
Ours
Quantile
Regression
L1
0.09
0.10
0.15
0.89
0.78
0.89
0.76
Regression
LPIPS
0.01
0.01
0.20
0.54
0.51
0.89
0.77
GAN
L1
0.09
0.09
0.14
0.95
0.85
0.94
0.80
GAN
LPIPS
0.01
0.01
0.08
0.31
0.24
0.50
0.23
Rat Astrocyte Cells
Regression
L1
0.24
0.26
0.28
0.99
0.54
0.95
0.88
Regression
SSIM
0.03
0.03
0.13
0.66
0.64
0.82
0.57
GAN L1
0.26
0.30
0.40
0.94
0.63
0.80
0.72
GAN SSIM
0.03
0.03
0.13
0.79
0.63
0.83
0.63
Places365
Regression
L1
0.30
0.36
0.39
0.99
0.97
0.95
0.94
Regression
SSIM
0.10
0.23
0.48
0.89
0.85
0.94
0.84
GAN
LI
0.37
0.38
0.47
0.97
0.81
0.95
0.67
GAN SSIM
0.10
0.12
0.51
0.86
0.81
0.92
0.86
Regression
L1
0.27
0.37
0.40
0.68
0.43
0.57
0.46
Regression
1.2
0.18
0.37
0.38
0.57
0.30
0.60
0.48
GAN
L1
0.27
0.38
0.40
0.58
0.40
0.60
0.52
GAN
L2
0.18
0.36
0.38
0.42
0.28
0.59
0.49
Table I: Image completion (top), super-resolution (middle) and colorization (bottom), results. | indicates average mask size; C(M, D) the correlation between predicted mask size and the amount of distortion; and C(M, Mopt) the correlation between the size of the calibrated mask and the size of the optimal mask. Arrows indicate which direction better. Best results shown in blue. 95% confidence intervals were calculated for all the experiments and are omitted due to lack of space. All presented results were verified for their statistical significance.
Quantile and Ours versus Opt. As can be seen from the top row, all three methods meet the distortion threshold with fewer than 10% exceptions, as required. Naturally, Opt does not have outliers since each mask is optimally calibrated by its computation. The spread of loss values tends to be higher with Quantile, indicating weaker performance. The colorization results, here and below, seem to be more challenging, with a smaller performance increase for our method.
The above probabilistic results are not surprising as they are the outcome of the calibration stage. The main question remaining relates to the size of the mask created as to meet this threshold distortion. The middle row in FigureBpreseee presents histograms of these sizes, showing that our approach tends produce masks that are close in size to those of Opt; while Quantile produces larger, and thus inferior, masked areas. Again, our colorization results show smaller if any improvement.
The bottom row in Figure S shows the correlation between mask sizes, and can be observed, high such correlation exists between our method and Opt, while Quantile produces less correlated and much heavier masks.
Quantitative results are presented in Table I for the completion, super resolution, and colorization tasks. First and foremost, the results obtained by our method surpass those of Quantile across the board. Our method exhibits smaller mask size || aligned well with the masks obtained by Opt. Quantile, as expected, produces larger masks. In terms of the correlation between the obtained mask-size and the unmasked distortion value C(M, D). our method shows high such agreement, while Quantile lags behind. This correlation indicates a much desired adaptivity of the estimated mask to the complexity of image content and thus to the corresponding uncertainty. The correlation C(M, Mopt) between Opt's 's mask size and the size of the mask obtained by our method or Quantile show similar behavior, aligned well with behavior see in the bottom of Figure 3
8

---

Under review as a conference paper at ICLR 2023
Image Complation Ragression LPIPS
Supar Resolution GAN
Colorization Regression
Our
Quarttee
Optimal
Our
Quantte
Opt
Our
Quantre
0.70
Model
0.060
Method
n.c018
Model
Optimal
0.18-
0.055
0.0018
0.16
0.250
0.0014
0.045
0.14
0.0012
0.040
0.32
0.0010
1055
0.035
1055
0.30-
0.0000
0.300
0.28
0.0006
0.025
0.08
0.300
0.0004
0.5
0.215
0.0000
0.5
0.100
0.0200
Modd
Medel
Modd
Ou
3.5
4.0
600
312
920
65
Ska
Mast Size
50c
Madd
Node
Mcdi
6.0
300
Figure 3: Summary of results for image completion (left), super-resolution (middle) and colorization (right). The top row shows the distribution of distortion values after masking versus the user-chosen threshold. The middle The bottom row describes the inter-relation between Opt's mask size and the sizes obtained by Our method and Quantile's.
presents histograms of mask sizes for the three compared methods.
5 CONCLUSIONS
Uncertainty assessment in image-to-image regression problems is a challenging task, due to the implied complexity, the high dimensions involved, and the need to offer effective and meaningful visualization of the estimated results. This work proposes a novel approach towards these challenges by constructing a continuous mask that highlights the trust-worthy regions in the estimated image. This mask provides a measure of uncertainty accompanied by an accuracy guarantee, stating that with high probability, the distance between the original and the estimated images over the non- masked regions is below a user- user-specified value. The presented paradigm is flexible, being agnostic to the choice of distance function used and the regression method employed, while yielding masks of small area.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Moloud Abdar, Farhad Pourpanah, Sadig Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges.
Information Fusion, 76:243-297, 2021.
Ahmed Alaa and Mihacla Van Der Schaar. Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In International Conference on Machine Learning, pp. 175-190.
PMLR, 2020.
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-fire uncertainty quantification. CoRR, abs/2107.07511, 2021a. URL https:
/aaxiv..rr//bb//2111..1
Anastasios Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021b.
Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021.
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint htt.::200.00811444 2022a.
Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates, Michael Jordan, Jitendra Ma- lik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. arXiv preprint 2XXiv:2202.05263 2022b.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06477, 2020.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1-34, 2021.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. processing systems, 30, 2017a
Advances in neural information
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192. PMLR, 2017b.
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function rnns. In The 22nd international conference artificial intelligence and statistics, pp. 1901-1910.
PMLR, 2019.
Ruihan Hu, Qijun Huang, Sheng Chang, Hao Wang, and Jin He. The MBPEP: deep ensemble pruning algorithm providing high quality uncertainty prediction. Applied Intelligence, 49(8): 2942-2955, 2019.
10

---

Under review as a conference paper at ICLR 2023
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-ine -immage translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Pavel Izmailov, Wesley Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace inference for Bayesian deep learning. Uncertainty in Artificial Intelligence, pp. 1169-11 1169-117,, PMLR, 2020.
Byol Kim, Chen Xu, and Rina Barber. Predictive inference is free with the jackknife+-after- bootstrap. Advances in Neural Information Processing Systems, 33:4138-4415 2020.
Danijel Kivaramovic, Kory D Johnson, and Hannes Leeb. Adaptive, distribution prediction intervals for deep networks. In International Conference Artificial Intelligence and Statistics, pp. 4346-4356. PMLR, 2020.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 (523):109411111, 2018.
Vebjorn Ljosa, Katherine L Sokolnicki, and Anne E Carpenter. Annotated high-throughput mi- croscopy image sets for validation. Nature methods, 9(7):637-637, 2012.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp. 2218-2227, PMLR, 2017.
David JC MacKay. Bayesian interpolation. Neural computation, 4(3):415 447, 1992.
Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Necly. High-quality prediction inter- vals for deep learning: A distribution-fres ensembled approach. In International conference machine learning, pp. 4075-4084. PMLR, 2018.
Konstantin Posch, Jan Steinbrener, and Jürgen Pilz. Variational inference to measure model uncer- tainty in deep neural networks. arXiv preprint
2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, volume 6. International Conference Representation Learning. 2018
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International conference machine learning, pp. 1218-12211 1226. PMLR, 2015.
Swami Sankaranarayanan, Anastasios Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola. Semantic uncertainty intervals for disentangled latent spaces. arXiv preprint arXiv:2207.1007 2022.
Matteo Sesia and Emmanuel Candès. A comparison of some conformal quantile regression meth- ods. Stat, 9(1 :e261, 2020.
Glenn Shafer and Vladimir Vovk. A tutorial conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Sophia Sun. Conformal methods for quantifying uncertainty in spatiotemporal data: A survey, arXiv preprint Ttt.:2209.035800 2022.
Laurent Valentin Jospin, Wray Buntine, Farid Boussaid, Hamid Laga, and Mohammed Bennamoun. Hands-on Bayesian neural networks-ag tutorial for deep learning users. arXiv e-prints, pp. arXiv- 2007, 2020.
11

---

Under review as a conference paper at ICLR 2023
Dongxia Wu, Liyao Gao, Xinyue Xiong, Matteo Chinazzi, Alessandro Vespignani, Yi-An Ma, and Rose Yu. Quantifying uncertainty in deep spatiotemporal forecasting. arXiv preprint arXiv:2105.11982 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
12

---

Under review as a conference paper at ICLR 2023
A TASK ILLUSTRATIONS
Figure 4: The three tasks we experimented with: 1) Image Colorization on the left column 2) Gray-seale Image Completion on the middle column, and Super Resolution on the right column.
B CALIBRATION ALGORITHM
Algorithm Finding X;
def scale (lambda_, mask, eps=le-6):
return (mask lambda_ (eps + mask (1 mask)) clip (0,
def find lambda (mask, y, y hat, eps=le-4):
1, h, lambda_ = eps, 1 eps, 0
while h -1 eps: lambda_ (h + 1)
loss loss_ fn scale (a, mask),
hat * scale mask))
if loss > alpha: lambda_
else: lambda
return lambda_
C PROOF THEOREM
Recall that we have defined the following optimization problem defining the optimal mask:
max [|m]]] subject to
Eyja (d (y. 0) S a,
(P1)
7 < - T.
13

---

Under review as a conference paper at ICLR 2023
Theorem 1: Consider Problem (P1) with the distortion measure d(y. 0) 1 where 1. Then, for sufficiently small the optimal mask m² admits a closed-form solution given by
-ll
mio a²
(E)=19))
where for all E [n]]
4(5) A
(Eyle (10 v()) Ir)) 1/(-1))
Proof. We start with reformulating Problem PI
as follows:
min -||m|| subject to Eyl [dm (v, 0)1 a,
m(n) E [0, 1].
Next, we define the Lagrangian of the above problem,
L(m, 1)) - -|mll + a [Byx @ (v 5)11 - -a):
where μ is the dual variable, and we invoke the chosen metric d(y, 0) 11l ØI for p According to the Karush-Kuhn-Tuuce (KKT) conditions, the optimal solution (m" (m=",²") must satisfy
0m()) = 0,
Vi 1,...,.. n,
aL
=
Furthermore, notice that the Lagrangian is separable with respect to the coordinates of m, thus, can write
0L
0
[m()] + (Byjee (v(0) i())) [P] ox)
dm())
Om
i
|maa) + u|m/) Byl==lli 2(3) 0(0)))) P1)
Om)())
0
m()) + um (Eyirllyg 0(0))²)
Om(()
+ pplE (1((3) - 0(a1') 1P] m 0.
In the above assume that m(3) 2 0, fact that should be verified once solution is formed. Hence, we obtain
m p-1 1 (upEgtelleco - θ(o) -()) p (1)
To simplify our derivation, we define
d( Eygeellts D(0)
(()) di0 11(1-11
Thus, can rewrite the above result equivalently as
p-1
(upd(()) or m(a) = (pp) -1/0-11) 463)
(6)
14

---

Under review as a conference paper at ICLR 2023
Now, consider the second KKT condition:
dL
By =lllm @ (y - =)] [p) a
0p
=
E Pyllllm (9G) (1)))l9
Σ m ( d() -a
Σ m,) (p)d())()))) _d(s) -
= (pp)
mij - a
j=1
(up)) -1/(p-1) 1)
{()) a 0.
j=1
11/p
+ pp
Σラ=1 9(3)
Therefore, the optimal solution m² is given by
9(1)
mie (E 46)
Note that assumed, the obtained solution mig non-negative. Finally, assuming is sufficiently small such that
4
a < min
(E)=1 (())
the solution satisfies the second constraint, mi) S which completes the proof.
We now turn to present a closely related result, referring to the case d(y:ū) 9) = |1y -0l1- Theorem 2: Consider Problem P1 with the distortion measure d(y. 9) 11 0 Then the optimal mask m² admits a closed-form solution given by
[1, d(g <
m
0 otherwise.
where dey A Eyyeelloo ا and d= is defined by
Ed (d() I _d = a.
Proof. Consider the Lagrangian of the above problem when d(y, 9) = 11y أ|l L(m, 4) & -Imm|| + p (1y @ (y 1) a):
where y >> 0 the dual variable. As before, the problem separable with respect to the coordinates of m. leading to the following mneddinnniioall problem
m6a mo (udio 1).
min -m() + umigddo min osm())1
where d() & EEyoz (1bt)) 1. Thus,
1, drp <
m()
0, otherwise.
15

---

Under review as a conference paper at ICLR 2023
Observe that this result is intuitive in retrospect, smaller distances d correspond to un-masked areas. In addition, regardless of μ. the choice the mask value dictated by the distance, a fact that we leverage hereafter. Recall that the solution should satisfy
Σ m = a,
where mix) are I-es for the smaller values of d(s). Thus,
Σ mo
Σ d(s) a.
(d((E) <d
Hence, the above provides a definition for the threshold value d", and 1/d", which completes the proof.
D MORE RESULTS
In this appendix we bring the results corresponding to all 12 settings explored in our work, corre- sponding to three inverse problems, two regression techniques and two metrics per each, along the following breakdown:
Image Completion: (Regressor, GAN] {LI, LPIPS:
Super Resolution: (Regressor, GAN] x SSIM): and
Colorization: [Regressor, GAN} x (LI,22)
We start in Figures 5 7 with the obtained distributions masked distortion values for Opt, Ours and Quantile. The goal here is to show that all three methods meet the required distortion threshold with exceptions that do not surpass the destination probability 8 0.1.
Figures 10 present histograms of the obtained mask sizes for the three methods. The goal is get minimal areas in these masks, so as to keep most the image content unmasked.
Figures 3 conclude these results by bringing graphs showing the inter-relation between the Opt mask size and the ones given by Ours and Quantile.
Analysis and conclusions drawn from these graphs brought in the experimental part of the paper.
16

---

Under review as a conference paper at ICLR 2023
Image Completion GAN LPIPS
Image Completion GAN L1
Opt
Our
Quantile
Opt
Our
Quantile
0.026
Model
0.20
Model
Opt Our
Our Quantile
Quantile
0.022
0.16
0.020
0.018
0.14
0.016
loss
0.12
0.014
0.012
0.10
0.008
0.06
0.006
0.004
0.04
Image Completion Regression L1
Image Completion Regression LPIPS
Opt
Our
Quantile
Opt
Our
Quantile
0.020
Model
0.20
Model
Opt
Opt
Our
0.18
Our
Quantile
Quantile
0.016
0.16
0.014
0.14
0.012
0.12
loss
0.010
0.30
0.008
0.08
0.006
0.004
0.04
0.002
0.02
Figure 5: Image Completion The distribution of the masked distortion values versus the chosen threshold (shown as horizontal dashed line) for the three tested methods Opt, Ours and Quantile.
17

---

Under review as a conference paper at ICLR 2023
Super Resolution GAN
Super Resolution GAN SSIM
Our
Quantile
Opt
Quantile
Model
0.50
Model
Opt
Opt
Our
0.45
Our
Quantile
Quantile
0.40
0.05
0.35
0.04
0.30
loss
0.25
0.20
0.02
0.15
0.01
0.00
Super Resolution Regression L1
Super Resolution Regression SSIM
Opt
Our
Quantile
Opt
Our
Quantile
0.045
Model
0.45
Model
Opt
Opt
0.040
Our
Quantile
Quantile
0.035
0.35
0.030
0.025
0.25
loss
0.020
0.015
0.15
0.010
0.005
0.05
0.000
0.00
Figure 6: Super Resolution The distribution of the masked distortion values versus the chosen threshold (shown as a horizontal dashed line) for the three tested methods - Opt, Ours and Quantile.
18

---

Under review as a conference paper at ICLR 2023
Colorization GAN
Colorization GAN L2
Our
Quantile
Opt
Quantile
Model
0.025
Opt
Our
Opt
Our
Quantile
Quantile
0.0016
0.020
0.0014
0.0012
0.015
loss
0.0010
0.0008
0.010
0.0002
0.000
0.0000
Colorization Regression
Colorization Regression L2
Opt
Quantile
Opt
Our
Quantile
0.030
Model
0.0018
Model
Opt
Opt
0.0016
Quantile
Quantile
0.0014
0.0012
0.0010
loss
0.0008
0.0006
0.005
0.0002
Figure 7: Colorization The distribution of the masked distortion values versus the chosen threshold
(shown as a horizontal dashed line) for the three tested methods Opt, Ours and Quantile.
19

---

Under review as a conference paper at ICLR 2023
Image Completion
Image Completion
Model
Oyt Quartile
Ost
Quartie
0.00
0.06
0:08
0.17
0.36
0.70
0.24
0.00
0.00
0.54
0.06
0.08
0.10
0.12
0.14
0.16
018
Mask Size
Mank Size
Image Complation Regression L1
Image Completion Regression LPIPS
Model
Model
0.34
0.20
0.55
0.35
0.40
0.45
Mask: Size
Mank Size
Figure 8: Image Completion Histograms of the calibrated mask sizes for the three tested methods Opt, Ours and Quantile.
20

---

Under review as a conference paper at ICLR 2023
Super Resolution
Super
Model
Oyt Quartie
Oct
Quartie
0.0
8.1
03
0.4
0.5
0.6
0.7
0.8
0.00
0.10
0.20
0.30
0.40
0.50
0.00
0.70
Mask Size
Mank Size
Super Resolution Regression
Super Resolution Regression SSIM
Model
Model
6.1
0.5
0:30
0.00
Mank Size
Mask Size
Figure 9: Super Resolution Histograms of the calibrated mask sizes for the three tested methods Opt, Ours and Quantile.
21

---

Under review as a conference paper at ICLR 2023
Colorization
Model
Model
Oyt
Ost Quartie
Quartie
0.0
6.3
6.4
0.5
0.6
57
08
69
1.0
6.0
0.3
0.4
0.6
07
0.8
0.
1.0
Mask Size
Mank Size
Colorization Regression L1
Colorization Regression L2
Model
Model
Our
0.6
3.2
0.
0.4
07
0.6
Mank Size
Mank Size
Figure 10: Colorization Histograms of the calibrated mask sizes for the three tested methods Opt, Ours and Quantile.
22

---

Under review as a conference paper at ICLR 2023
Image Completion GAN L1
LPIPS
022
Model
Model
Dur
Our
020-
0.3
0
0.30
3.0
0.06
0.56
0.00
0.10
0.3
6.5
0.88
035
000
3.000
3.50
3:004
0.353
0.00
000
Opt Size
Opt Size
Image Completion Regression L1
Image Completion Regression UPIPS
Model
Model
035
Our
Our
020
5180
0.35
0:30
0.00
0.0
6.0
4.22
0.64
0.08
0.00
0.22
0.36
0.25
0.2
0.00
0.000
0.002
0.004
0,006
2.000
0.000
0:02
0004
Opt
Sizee
Opt Size
Figure 11: Image Completion The correlation between the Ours and Quantile mask sizes versus Opt's mask size.
Super Resolution GAN L1
Super Resolution GAN SSIM
68
0.5
Quartls
6.
9168
0.4
0.3
0.2
0.1
3.00
0.0
0.20
3300
0.40
0.50
0.62
0.70
0.000 0.235
6d10
0.015
0.000
0.025
0.000
0.005
0.6400245
Opt Size
Opt Size
Super Regression
Super Resolution Regression
Model
Model
077
0.5
0.6
02-
0.0
0.0
odoo
Figure/ 12: Super Resolution The correlation between the Ours and Quantile mask sizes versus Opt's mask size.
23

---

Under review as a conference paper at ICLR 2023
Colorization GAN L1
Colorization GAN L2
10-
Model
Model
67.
0.7
0.3
0.1
0.4
0.5
0.6
3.6
5.4
0.5
06
Opt Size
Opt Sizo
Colorization Regression
Colorization Ragression
Model
Model
Quart
Quardla
0.0
0.8
0.0
0.5
0.7
3.
035
0.6
033
3.6
0.7
2.5
3.0
50
0.1
02
0.3
2.4
0.3
2.5
03
Opt
Opt
Figure 13: Colorization The correlation between the Ours and Quantile mask sizes versus Opt's mask size.
24


Reviews:
[{"review_id": "x0RS2eabQA5", "reviewer": "Reviewer_DEta", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "     Authors present a new method for estimating the uncertainty in\n     image-to-image problems. Given a model that is already trained to do\n     image-to-image prediction, authors train a model that predicts a mask that\n     only retains the accurate areas in the prediction. The inaccurate areas are\n     masked out, therefore reducing the overall prediction error. The mask is\n     said to retain ``certain'' regions in the prediction. \n\n     Authors compare their method with two alternatives. First, is a method that\n     predicts confidence intervals. The confidence intervals are modified by the\n     authors of this work to generate a mask. The resulting mask is compared to\n     that given by the method here. The second alternative is an upper-bound. A\n     method that is aware of the prediction and the ground-truth. \n\n     Results show that the proposed method is indeed better than the\n     alternative and close to the upper-bound. ", "summary_of_the_review": "     The work is interesting but there are some drawbacks. \n\n     The most important drawback is that authors effectively present a method\n     that predicts error maps rather than uncertainty in the prediction.", "strength_and_weaknesses": "     Strengths: \n     1. The proposed method is a post-hoc analysis technique and, therefore, it\n        can be applied to any pre-trained network. This is a clear strength. \n     2. Predicted masks are not as good as the upper bound, but they are much\n        better than the alternative technique. \n     3. Authors demonstrate the method on three different tasks colorization,\n        super-resolution and image completion. \n     4. Authors demonstrate their method on biological imaging, where\n        uncertainty quantification may indeed be important. \n\n     Weaknesses: \n     1. The proposed technique does not seem to predict uncertainty. To the best\n        of my understanding, it predicts where the original model is making\n        errors. An easy way to see this would be to consider a point where the\n        model is confident but inaccurate, i.e., a bias. The predicted mask\n        should cover this point, even though it is not an uncertain prediction\n        of the model. The problem is actually in the definition of\n        ``uncertainty''. Equation 1 (or 3) do not necessarily only correspond to\n        uncertainty. It corresponds to expected error for a sample. It also\n        includes the contribution of bias. In this regard, I think the\n        uncertainty positioning may not be very accurate. Consequently, the\n        comparison with a method that computes confidence intervals may not be\n        appropriate. Here, I should note that predicting expected error or the error is not a\n        bad target. However, the difference between this and uncertainty\n        estimation should probably be made clear. \n\n     2. I am not sure about the contribution of the corollary 1. The result is\n        not very surprising in my opinion. The mask that aims to minimize the\n        ``uncertainty'' is bound to be related to the ``uncertainty''. \n\t\n     3. It is unclear how authors estimate the expectation in Equations 4\n        or 5. This is over y|x variable. However, in Equation 6, they drop the\n        expectation and simply take only one sample to compute this\n        expectation. Using this I am assuming the model learns to predict the\n        error and not the expected error. As a result, the model may not be able\n        to identify an uncertain prediction at a pixel for a given sample since\n        for that pixel, there may be only 1 output that happens to be close to\n        the ground truth. \n\n     4. Equation 8 yields a suspicious behavior. When m_{\\theta} = 0,\n        m_{\\lambda} = \\lambda. However, when m_{\\theta} = 1, m_{\\lambda} =\n        \\infty. It is unclear how authors deal with this. Furthermore, the\n        intuition of this specific calibration form is unclear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ea52_COv5r", "reviewer": "Reviewer_GH3K", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to uncertainty estimation for image-to-image tasks based on continuous masking. The approach can work with arbitrary networks, and distance metrics between images and is guaranteed to yield distances within a small threshold with high probability. Experiments are conducted on image colorization, image completion, and super-resolution tasks. ", "summary_of_the_review": "The paper presents a novel method that connects uncertainty estimation with masks. The method is highly flexible and has a theoretical guarantee. In the experiments, the method is tested on three different image-to-image tasks, which are comprehensive. While the results and comparison are not very intuitive and clear, which can be further improved. ", "strength_and_weaknesses": "Strength\n+ theoretically proved that the uncertainty mask is with an accuracy guarantee, and the user can set the threshold to control the ratio of trust-worthy regions.\n+ experiments done on three image-to-image tasks prove the adaptability of the proposed method.\n\nWeaknesses\n+ in the mask of the colorization task, there is some inexplicable circular artifact pattern, which would depreciate the uncertainty mask.\n+ in table 1, the proposed method produces comparable results to that of Opt in some settings, it’s not mentioned whether significant tests are performed or not.  It’s said that a smaller average mask size ||M|| indicates better performance, why Opt’s results are not the best?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XBBxATfxW7t", "reviewer": "Reviewer_xoZ2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Quantifying uncertain regions when performing image-to-image tasks such as inpainting or super resolution can be useful in identifying low confidence regions. This work presents a mask minimization approach that strives to obtain the smallest regions that correspond to the the most uncertain regions, as defined by user defined probabilistic thresholds. The method is implemented as a neural network and is modular in which distance- or distortion measures can be used in assessing the quality of masks. Experiments are reported on three image-to-image tasks : inpainting, colorization, super-resolution on two datasets using multiple distance measures. The experimental evaluation on two datasets are compared with an oracle based method (Opt) and a related conformal prediction method (Quantile), showing improvements compared to the latter.  \n", "summary_of_the_review": "A method to estimate uncertainty in image-to-image tasks is presented within a theoretical formulation. However, the problem formulations are simplified extensively to obtain the final objectives; implications of these simplifications are not explained in relation to the original problem formulation. Assessing uncertainty in the experiments is done using mask size as a proxy, which is unconventional and does not immediately translate into established notions of uncertainty estimation methods. \n", "strength_and_weaknesses": "**Strengths:**\n\n* The problem formulations are clearly presented with elegant theoretical justifications. The proof of Theorem 1 solved using a dual formulation of P1 with KKT conditions look correct. The limitations of formulating the problem P1 are discussed clearly, and the motivation for simplifying it to P2 is reasonable.\n\n* The overall task of estimating uncertain regions in image-to-image tasks is interesting, and the diversity of tasks addressed in this work is comprehensive. \n\n* The resulting method is simple and looks like can be easily integrated into other image-to-image tasks. \n\n* Experimental results show the proposed method to consistently perform better than the baseline. There are some issues with the evaluation itself which is discussed further below.\n\n**Weaknesses:**\n\n* **Simplification from P1 to P2**: The simplification from P1 to P2, and finally the objective in Eq. 6 used to train the model assumes a series of strong simplifications. While I see the need for making these simplifications; for instance, going from P1 to P2 removes the reliance on ground truth masks at inference, the implications of making these simplifications remains unclear. One could ideally arrive at the same objective in Eq. 6 without theorising about P1/P2, as it is in the end trying to minimize the span of a mask and distortion. I appreciate the authors presenting theorems and proofs in Appendix A, but I am not entirely convinced this to be useful in the end. I would like clarifications on how the final objective in Eq. 6 still is a reasonable solution to P1. \n\n* **Distortion measures**: The modularity of the proposed method, that it is not tied to specific distance or distortion measures is appealing. However, there is no specification of the distortion measures at all (or that I could easily find). In the end, for the three experiments which are the distortion measures used? Is there a difference in which distortion measure $d(\\cdot)$ is used for the three tasks. \n\n* **Loss function in Eq. 6**: During training the loss function in Eq. 6 still uses the ground truth. If this method were to be an unsupervised one is there a further simplification of Eq. 6 that could remove the reliance on ground truth? For instance, the distortion measure could instead be based on some variation of pixel-level entropy of the reconstructed data.  \n\n* **Pixel-wise uncertainty measures**: One other simplification used in this work is that of the pixels being independent and their uncertainty measures are estimated/modelled at the pixel level (please correct me if I have misunderstood this part). If this is the case, how reliable are these uncertainty masks? At least from the visualizations in Fig. 3 the mask seems largely spatially correlated. How is this ensured in the model?\n\n* **Mask as a neural network**: In Sec 3.2 the authors state _modelling the mask as a neural network_. What does this exactly mean? The neural network $m_\\theta(x,\\hat{y})$ takes the degraded input and the reconstructed output and predicts the mask, as a pixel grid with [0,1] values? This is not entirely clear and also the neural network itself is later described as the mask model (implemented as a U-net); are these two the same? If the mask model is indeed a U-net what are the inputs to it? Are the degraded input and reconstruction both provided as the input?  \n\n* **Mask-size as a measure of uncertainty**: One of the main concerns with the evaluations reported in this work is using mask size as measure of uncertainty or the correlation of mask size with distortion. This is firstly an unconventional way of assessing uncertainty; perhaps also due to the lack of clarity on what distortion measures are used it seems to be not the most obvious way of representing uncertain regions. Additional results are reported in Fig. 4 but are not discussed in the context of uncertainty assessment, making it harder to wrap them in understanding the proposed work.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9My4PBoRRu", "reviewer": "Reviewer_pBvq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new approach for uncertainty estimation in image reconstruction, which uses masking mechanism to identify the more certain regions of the reconstructed image, thus narrowing the distance between masked ground truth and reconstructed images. The experiments are conducted on three reconstruction tasks: image colorization, image completion and super-resolution.", "summary_of_the_review": "The paper studied the problem of uncertainty estimation in image-to-image domain. The idea seems somewhat interesting, but the effectiveness is not well verified. I think this paper meets the margins of acceptance.", "strength_and_weaknesses": "Strength:\n1. The proposed method introduced a continuous mask to replace the previous binary mask for better modeling the uncertainty between the ground truth image and the reconstructed image.\n2. The conformal prediction strategy is employed to calibrate the masking model to obtain strong statistical guarantees. \n3. The paper is good writing and easy to follow.\n\nWeakness:\n1. The generation way of the continuous mask m is not clear, is it the same as the prevalent implicit neural representation? And the ablation study compared to the binary mask should be provided. \n2. The visualization results for super-resolution are not included in the paper.\n3. It seems that Figure 2 is dispensable, as it doesn’t provide more meaningful information to understand the experiment setups.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject