Looks Too Good To Be True: An Information-Theoretic Analysis of Hallucinations in Generative Restoration Models
Authors: Regev Cohen
Idan Kligvasser
Ehud Rivlin
Daniel Freedman
Venue: NeurIPS 2024 poster (2024)
Keywords: uncertainty
hallucinations
perception
tradeoff
distortion
restoration tasks
inverse problems.
Primary area: safety_in_machine_learning
TL;DR: We uncover an inherent limitation of generative image restoration: higher perceptual quality comes at the expense of greater uncertainty.

Abstract: The pursuit of high perceptual quality in image restoration has driven the development of revolutionary generative models, capable of producing results often visually indistinguishable from real data.
However, as their perceptual quality continues to improve, these models also exhibit a growing tendency to generate hallucinations – realistic-looking details that do not exist in the ground truth images.
Hallucinations in these models create uncertainty about their reliability, raising major concerns about their practical application.
This paper investigates this phenomenon through the lens of information theory, revealing a fundamental tradeoff between uncertainty and perception. We rigorously analyze the relationship between these two factors, proving that the global minimal uncertainty in generative models grows in tandem with perception. 
In particular, we define the inherent uncertainty of the restoration problem and show that attaining perfect perceptual quality entails at least twice this uncertainty. Additionally, we establish a relation between distortion, uncertainty and perception, through which we prove the aforementioned uncertainly-perception tradeoff induces the well-known perception-distortion tradeoff.
We demonstrate our theoretical findings through experiments with super-resolution and inpainting algorithms.
This work uncovers fundamental limitations of generative models in achieving both high perceptual quality and reliable predictions for image restoration. 
Thus, we aim to raise awareness among practitioners about this inherent tradeoff, empowering them to make informed decisions and potentially prioritize safety over perceptual performance.

Looks Too Good To Be True:
An Information-Theoree Analysis of Hallucinations in Generative Restoration Models
Regev Cohen
Idan Kligvasser
Ehud Rivlin
Daniel Freedman
Verily Al (Google Life Sciences), Israel
regevcohen@google.com
Abstract
The pursuit of high perceptual quality in image restoration has driven the devel- opment of revolutionary generative models, capable of producing results often visually indistinguishable from real data. However, as their perceptual quality continues to improve, these models also exhibit growing tendency to generate hallucinations- rcalistic-looking details that do not exist in the ground truth im- ages. Hallucinations in these models create uncertainty about their reliability, raising major concerns about their practical application. This paper investigates this phenomenon through the lens of information theory, revealing fundamental tradeoff between uncertainty and perception. We rigorously analyze the relation- ship between these two factors, proving that the global minimal uncertainty in generative models grows in tandem with perception. In particular, we define the inherent uncertainty of the restoration problem and show that attaining perfect perceptual quality entails at least twice this uncertainty. Additionally, we estab- lish relation between distortion, uncertainty and perception, through which prove the aforementioned umcertainly-percepiie tradeoff induces the well-known perception-distortin tradeofT. We demonstrate our theoretical findings through experiments with super-resolution and inpainting algorithms. This work uncovers fundamental limitations of generative models in achieving both high perceptual quality and reliable predictions for image restoration. Thus, we aim to raise aware- ness among practitioners about this inherent tradeoff, empowering them make informed decisions and potentially prioritize safety over perceptual performance.
20mnaent
Uncertainty Lower Bound
(- Better)
Ussomer
Impossible Region
(-
Better)
Perception
Figure 1: Illustration of Theorem In restoration tasks, the minimal attainable uncertainty is lower bounded by a function that begins at the inherent uncertainty Unnneren of the problem (Definition 2 and graudally increases up twice this value as the recovery approaches perfect perceptual quality.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Better Perception
Higher Uncertainty (Hallucinations)
Image inpainting results. Algorithms ordered from low to high perception (left to right). Note the corresponding increased hallucinations and distortion. See Section for details.
Figure 2:
Introduction
Restoration tasks and inverse problems impact many scientific and engineering disciplines, as well as healthcare, education, communication and art. Generative artificial intelligence [0 38 I0] has transformed the field of inverse problems due its unprecedented ability to infer missing information and restore corrupted data. In the realm of image restoration, the quest for high perceptual quality has led to a new generation of generative models, capable of producing outputs of remarkable realism, virtually indistinguishable from true images.
While powerful, growing empirical evidence indicates that generative models are susceptible to hallucinations [30], characterized by the generation of seemingly authentic content that deviates from the original input data, hindering applications where faithfulness is crucial. The root cause of hallucination lies in the ill-posed nature of restoration problems, where multiple possible solutions can explain the observed measurements, leading to uncertainty in the estimation process.
Concerns surrounding hallucinations have prompted the development of uncertainty quantification methods, designed to evaluate the reliability of generated outputs. These approaches offer crucial insights into the model's s confidence in its predictions, empowering users to assess potential deviations from the original data and make informed decisions. Despite this progress, the relationship between achieving high perceptual quality and the extent of uncertainty remains an understudied area.
This paper establishes the theoretical relationship between uncertainty and perception, demonstrating through rigorous analysis that the global minimal uncertainty in generative models increases with the level of desired perceptual quality (see illustration in Figure Tp. Leveraging information theory, we quantify uncertainty using the entropy of the recovery error [9]] while we measure perceptual quality via conditional divergence between the distributions of the true and recovered images (58 Our main contribution are as follows:
1. We introduce definition for the inherent uncertainty Uniberent of an inverse problem, and formulate the encertainty-perceppie (UP) function, seeking the minimal attainable uncertainty for a given perceptual index. We prove the UP function is globally lower-bounded by Uniherent (Theorem [
2. We prove fundamental trade-off between uncertainty and perception under any underlying data distribution, restoration problem or model (Theorem []. Specifically, the entropy power of the recovery enror exhibits lower bound inversely related to the Rényi divergence between the true and recovered image distributions (Theorem 3). This shows that perfect perceptual quality requires at least twice the inherent uncertainty Unherent-
We establish a relationship between uncertainty and mean squared error (MSE) distortion, demon- strating that the uncertainty perceepioon trade-off induces the well-known distortion- perrepponn trade-off [4 (Theorem
4. We empirically validate all theoretical findings through experiments on image super-resolution and inpainting (Section covering broad spectrum of recovery algorithms, diverse metrics and data distributions. Our experimental results for image inpainting are illustrated in Figure

---

We aim to provide practitioners with deeper understanding of the tradeoff between uncertainty and perceptual quality, allowing them strategically navigate this balance and prioritize safety when deploying generative models in real-world, sensitive applications.
Related Work
Recent work in image restoration has made significant strides both perceptual quality assessment and uncertainty quantification, largely independently. Below, we outline the main trends in research on these topics, laying the foundation for our framework.
Perception Quantification Perceptual quality in restoration tasks encompasses how humans perceive the output, considering visual fidelity, similarity to the original, and absence of artifacts. While traditional metrics like PSNR and SSIM [82] capture basic similarity, they miss finer details and higher-level structures. Learned metries like LPIPS 87,, VGG-loss (72), and DISTS [22] offer improvements but still operate on pixel or patch level, potentially overlooking holistic aspects. Recently, researchers have leveraged image-level embeddings from large vision models like DINO 117 and CLIP [62] to capture high-level similarity. Further advancements include HyperIQA 774) that leverages self-adaptive hyper networks to blindly assess image quality in the wild, while LIQE (88] and QAlign (4 utilize large language models to capture high-level semantic similarity and alignment between the restored and original images. Here, we follow previous works [58] 14 3T and adopt a mathematical notion of perceptual quality defined as the divergence between probability densities.
Uncertainty Quantification Uncertainty quantification techniques can be broadly categorized into two main paradigms: Bayesian estimation and frequentist approaches. The Bayesian paradigm defines uncertainty by assuming distribution over the model parameters and/or activation functions [I] The most prevalent approach is Bayesian neural networks [52 78 [4], which are stochastic models trained using Bayesian inference. To improve efficiency, approximation methods have been developed, including Monte Carlo dropout [24] 25 stochastic gradient Markov chain Monte Carlo 667 18).
Laplacian approximations (33 and variational inference [6 [1
60 Alternative Bayesian techniques
encompass deep Gaussian processes (20), deep ensembles 7 33,, and deep Bayesian active learning
26). In contrast to Bayesian methods, firequentist approaches operate assume fixed model parameters with underlying distribution. Examples of such distribution-fiee techniques model ensembles
59], bootstrap [36 21, interval regression [59 37 83 and quantile regression [27 64].
An emerging approach in recent years is conformal prediction B 70 which leverages a labeled calibration dataset to convert point estimates into prediction regions. Conformal methods require no retraining, computationally efficient, and provide coverage guarantees in finite samples [4 These works include conformalized quantile regression (64 69 (6) conformal risk control
and semantic uncertainty intervals for generative adversarial networks (68). The authors of introduce the notion of conformal prediction masks, interpretable image masks with rigorous statistical guarantees for image restoration, highlighting regions of high uncertainty in the recovered images. Please see [75] for an extensive survey of distribution-fri conformal prediction methods. A recent approach [T]] introduces principal uncertainty quantification method for image restoration that considers spatial relationships within the image to derive uncertainty intervals that are guaranteed to include the true unseen image with user-defined confidence probabilities. While the above studies offer a variety of approaches for quantifying uncertainty, a rigours analysis of the relationship between uncertainty and perception remains underexplored in the context of image restoration.
The Distortion- tradeoff 11] and its follow-ups 23
Distortion-Perceptioo Tradcoff The most relevant studies to our research are the work on the
15 K A key finding in [14] establishes a convex tradcoff between perceptual quality and distortion in image restoration, applicable to any distortion measure and distribution. Morcover, perfect perceptual quality comes at the expense of no more than 3dB in PSNR. The work in 23 extends this, providing closed-fome expressions for the tradcoff' when MSE distortion and Wasserstein-2 distance are considered as distortion and perception measures respectively. In [58], it is shown that the Lipschitz constant of any deterministic estimator grows to infinity as it approaches perfect perception.
This work uniquely emphasizes uncertainty image restoration, distinguishing it from distortion. While distortion measures how close restored image is to the original, uncertainty quantifies the confidence in the restoration itself. This distinction crucial for decision-making, as high uncertainty can hinder informed choices, complementing existing research on perceptual quality and robustness.
3

---

3 Problem Formulation
We adopt a Bayesian perspective address inverse problems, wherein we seek to recover a random vector X IR from its observations, represented by another random vector M(X) E R
Here M I2 + IRd is a non non-invertible degradation function, implying X cannot be perfectly recovered from Y. Formally:
Definition 1. A degradationnincae
M said to be invariable if. the conditionnl probability Pxly (-ly) a Dirac delta function for almost every y in the support of the distribution pY of Y..
The restoration process involves constructing a estimator X E2 to estimate X from Y, inducing conditional probability Pxyy- The estimation process forms Markov chain X + - X, implying that X and X are statistically independent given Y.
In this paper, we analyze estimators X with respect two performance criteria: perception and uncertainty. To assess perceptual quality, we follow a theoretical approach, similar to previous works 85 14), and measure perception using conditional divergence between X and X defined as
Dx X) A Ey-py (Do(VXIY=NPPIII-))
where Du stands for general divergence function. When estimator attains a low value of the metric above, we say it exhibits high perceptual quality. When it comes to uncertainty, there are diverse practical methods to quantify it [28] []] However, for analysis, we aim to identify fundamental understanding of uncertainty. Therefore, we adopt the concept of entropy power from information theory, which assesses the statistical spread of random variable. For the definition of entropy power and other relevant background, we refer the reader to Appendix [ Utilizing entropy power, we formally define the inherent uncertainty intrinsic to the restoration problem as follows
Definition 2. The inherent uncertainty in estimating X from Y is defined as:
1 e
U Univerent (((XY)=
2Te
where h(X(Y) denotes the entropy of X given Y.
The inherent uncertainty quantifies the information irrevocably lost during observation, acting as a fundamental limit the recovery of X from Y, regardless of the estimation method. Notably, when the degradation process is invertible, this inherent uncertainty becomes zero Untsesent = 0, reflecting the possibility of perfect recovery of X with complete confidence.
We now turn our attention to the main focus of this paper, the succetaiity-perceptiiii (UP) function:
U(P) min {N(* ( X(Y): D)(x PXY
In essence, U(P) represents the minimum uncertainty achievable by estimator with perception quality of at least P, given the side information within the observation Y. In contrast to the perception-distortinn functioon [14), the above objective prioritizes the information content of error signals over their mere energy, and its minimization promotes concentrated errors for robust and reliable predictions. The following example offers intuition into the typical behavior of this function.
Example 1. Consider Y X + W where X N 1) and W ~ -N o²) are independent. Let the perception measure be the symmetric Kullback-Leibler (KL) divergence DsKL and assume stochastic estimators of the form X = E [ + Z where Z (O, o?) is independent of As derived in Appendix C the UP function admits closed form expression in this case, given by
U(P) NN(XX) (P+1) (P-1-v(P+12D)) )J
where N(X = (1+1100) o
The above result, illustrated in Appendix C demonstrates the minimal attainable uncertainty in- creases as the perception quality improves. Moreover, The above example suggests a structure for perreepton function U(P), which fundamentally relies on the inherent uncertainty
'ee Appendix for brief explanation of how conditional divergence relates human perception.

---

(X Remarkably, the following section shows that this dependency generalizes beyond the spe- cific example presented here, where particular form is determined by the underlying distributions. along with the specific perception measure employed.
Remark One may consider the following alternative formulation
(P) A min P {N(*
The alternative objective quantifies uncertainty as the entropy power of the error, independent of the side information Y. While potentially insightful, this approach may overestimate uncertainty since
N x N( X) where equality holds and only if the error & X - - is independent of Y. Although further investigation is warranted, we hypothesize that the behavior of function (3)
mirrors that of the UP function which we examine detail in the following section.
The Uncertainty-Percepiion Tradeoff
Thus far, have formulated the rationale. We now proceed to derive its key properties, including detailed analysis for the case where Rényi divergence serves as the measure of perceptual quality, Subsequently, we establish direct link between the UP function and the well-known distortion- distortion-perceptiin perception tradeoff. Finally, we demonstrate our theoretical findings through experiments on image super-resolution.
function and elucidated its underlying
4.1 The Ennerttainty-ereepttiin Plane
The following theorem establishes general properties of the uncertainty- function, UP)), irrespective the specific distributions and divergence measures chosen.
Theorem 1. The uncertainty- pereeettion function (P) displays the following properties
1.
(monotonically non-increasing and continuous):
min (U(P)).U(PP)) su UU(A + (1 max (U(P)),U(P)), AA e [0,1]
2. Boundlessness:
N(XY)) S U(P) 2N(X)lY).
where XG is zero-mean Gaussian random variable with covariance identical to X. The inherent uncertainty is upper bounded by NXGalY), which depends the deviation of X from Gaussianity.
The theorem establishes a fundamental tradeoff' between perceptual quality and uncertainty in image restoration, regardless of the specific divergence measure, data distributions, or restoration model employed. This tradeoff is fundamentally linked to the inherent uncertainty [[X ( arising from the information loss during the observation process. Notably, the upper bound can be expressed
N(XalY) (( Pxx Pxx (X,
This shows that as X approaches Gaussianity. N( approaches N().lY) However, concur- rently, it implies general higher values of N (X]Y) due to Lemma I of Appendix B This finding yields surprising insight: for multivariate Gaussian distributions, perfect perceptual quality comes the expense of exactly twice the inherent uncertainty the problem.
Next, we show that for a fixed perceptual index P, the optimal algorithms lie on the boundary of the constraint set. This facilitates the optimization,
restricts the search space to the boundary points.
Theorem 2. Assume D (X, is convex in its second argument. Then, for any P 2 0, the minimum is attained on the boundary where (X, =
Note that the assumption of the convexity of D. in its second argument is not restrictive condition. In fact, most widely-used divergence functions, notably f'divergennes (such as KI. divergence, total variation distance, Hellinger distance, and Chi-square divergence), exhibit this property.
5

---

While the above theorems describe important characteristics of the uncertainty perecpttion function, additional assumptions are needed to gain deeper insights. Therefore, we now focus on Rényi divergence as our perception measure. Rényi divergence is versatile family of divergence functions parameterized by an order O S r, encompassing the well-known KI. divergence as a special case when r = 1. This divergence plays critical role in in analyzing Bayesian estimators and numerous information theory calculations [79]. Importanly, is also closely related to other distance metrics used in probability and statistics, such as the Wasserstein and Hellinger distances. Focusing on the case where r = 1/2, we arrive at:
U(P) = min NN NN x(Y) X)) PXIY
(5)
While we set r 1/2 to facilitate our derivations, it is important to note that all orders (0,1) are equivalent (see Appendix [B. Consequently, given this equivalence and the close relationship between Rényi divergence and other metrics, analyzing the specific formulation provided by ( may yield valuable insights applicable to a wide range of divergence measures. The following theorem provides lower and upper bounds for the UP function.
Theorem 3. The uncertainty- pereeetion function is confined to the following region n)P) ((JY) S UP) S (P) -N(x)lY)
where 1 S (P) 2 is a convex function w.r.t the perception index and is given by
(y()) (2e 2P
(2e 24 1)3
Noteworthy, Theorem 3 holdd true regardless of the underlying distributions of X and Y. thereby providing a universal characterization of the UP function in terms of perception. Furthermore, as depicted in Figure [3] Theorem [3] gives rise the uncertainty- pereeettionn plane, which divides the space into three distinct regions:
1. Impossible region, where no estimator reach.
2. Optimal region, encompassing all estimators that are optimal according to (5)
3. Suboptimal region of estimators which exhibit overly high uncertainty.
The existence of an impossible region highlights the mucertainty-perceplle tradeoff, proving no estimator can achieve both high perception and low uncertainty simultaneously. This finding under- scores the importance of practitioners being aware of this tradeoff, enabling them to make informed decisions when prioritizing between perceptual quality and uncertainty in their applications. The uncertainty-percepiiin plane could serve as a valuable framework for evaluating estimator perfor- mance in this context. While not a comprehensive metric, it may offer insights into areas where improvements can be made, guiding practitioners towards estimators that strike a more desirable balance between perception and uncertainty. For certain estimators residing in the suboptimal region, it may be possible to achieve lower uncertainty without sacrificing perceptual quality. Thus, we believe that our proposed further research and practical applications, ultimately leading to the development of safer and reliable image restoration algorithms.
seerraintyyprrcettoon plane serve as valuable starting point for
Next, we analyze how the dimensionality of the underlying data affects the uncertainty-perrepiion tradeoff. To achieve this, we extend the function r7(P) to include dimension parameter d, denoted as ()(P:)) d). As shown Fig. 4 Ty T)(Pdt d) exhibits a rapid incline as perception improves and attain higher values in higher dimensions. This observation suggests that in high-dimensional settings, the eecertainty-perceptii tradcoff becomes more severe, implying that any marginal improvement in perception for an algorithm is accompanied by a dramatic increase in uncertainty.
Finally, we conjecture that the general form of the tradeoff, given by the inequality Theorem 3 holds for different divergence measures, with the specific form of (P) capturing the nuances of each chosen measure. For instance, considering the Hellinger distance as our perception measure, we obtain the same inequality in Theorem but with (P) defined for 0 < P < 1 a
2
2 ( V ( Pyl/d -1) - 1.
2
T/Hlllnngr (P)
(6)
Py4/
The case of P = is obtained by taking the limit lim (?P) =
p+1

---

2NCXelFi
Suboptimal
Optimal Impossible
NIX X1M
2N0
N(AP)
1.0
2.5
(122 X(X)
Figure 3: The enpertainty-perceplee plane (Theorem 3) The impossible region demonstrates the inherent tradeoff between perception and uncertainty, while other regions may guide practitioners toward estimators that better balance the two factors, highlighting potential areas for improvement.
1.95
1.90
1.85
1.80
1.75
nf-64
1.70
d-256
1.65
(f-512
1.60
d-2048
0.00
0.2
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Perceptual
Figure 4: Impact of dimensionality, as revealed Theorem demonstrates that the uncertainty- perception tradeoff intensifies in higher dimensions. This implies that even minor improvements in perceptual quality for algorithm may come the cost significant increase in uncertainty.
4.2 Revisiting the Distortion-Percoption Tradcoff
Having established the uncertainty--eereppiin tradeoff and its characteristics, we broaden our analysis to estimation distortion, particularly the mean squared-error. A well-known result in estimation theory states that for any random variable X and for any estimator X based upon side information Y. the following holds true (19
E
| -11--11P 2
2M(XXY)
2тE
This inequality, related to the uncertainty principle, serves as a fundamental limit to the minimal MSE achieved by any estimator. However, it does not consider the estimation uncertainty of X as the right hand side independent of X. Thus, we extend the above in the following theorem.
Theorem 4. For any random variable X, observation and unbiased estimator X, it holds that
E |
x12] N (x -x(xx)
Notice that for any estimator X we have N(X XIY) > implying
implying
ElJ x11 > N(XIY) 2
(8)

---

The above result aligns with equation 7). demonstrating that Theorem 4 serves as generalization of inequality Z incorporating the uncertainty associated with the estimation. Furthermore, by viewing the estimator X as a function of perception index P. we arrive at the next corollary.
Corollary 1. Define the following fiistottion-percettonn
D(P)
A min
B [IX-
[|]IX-IIp] DA X
PRIY
Then, for any perceptual index P, we have D(P) 2 U(P).
As uncertainty increases with improving perception, the corollary implies that distortion also increases. Thus, when utilizing MSE as a measure of distortion, the uncertainty-perceppe tradeoff induces a peterpionn tradeoff [44 offering novel interpretation of this well-known phenomenon.
Experiments
Setup. Our theoretical framework is grounded in empirical observations, leading to validate findings through experiments on common benchmark tasks: image super-resolution and inpainting. We analyze performance through the lens of uncertainty, alongside established measures of perceptual quality and distortion. To assess perceptuall quality, we employ state-of-the-ae metrics including
HyperlQA (74). LIQE [88] and Q-ALIGN [84). Distortion is evaluated using traditional measures:
MSE, peak signal-to-noies ratio (PSNR), and structural similarity index (SSIM) 82) Accurately
estimating entropy in high-dimensional spaces presents significant challenges 46 hence, we utilize
an upper bound for uncertainty, N (Xc- Xxa Y as detailed in Appendix F This practical alternative simplifies computation to calculating the geometric mean of the singular values of the error covariance.
For super-resolution, resolution image from its low-resolution counterpart obtained via 4x bicubic downsampling. Our evaluation spans diverse range of recovery algorithms, including EDSR (50), ESRGAN [8]], SinGAN 771). SANGAN (39, DIP 77). SRResNet/SRGAN variants [4] EnhanceNet [66], and Latent Diffusion Models (LDMs) with parameter 3 [0, 1]
utilize the BSD100 benchmark dataset [53], aiming predict high-
65 where / 0 recovers DDIM
32 and 3 recovers DDPM 733. In the context of image inpainting, we leverage the SeeTTrue dataset 86). an image-text alignment benchmark known for its diverse collection of real and synthetic text-image pairs. Here, we focus our analysis on diffusion models due to their state-of-the-art performance and growing popularity in the field.
Results. Jpuueess our super-resolution analysis. As observed in the top row, across various perceptual measures, unattainable blank region exists in the lower right corner, indicating that no model simultaneously achieves both low uncertainty and high perceptual quality. Furthermore, an anti-correlation emerges near this region, where modest improvements in perceptual quality translate to dramatic increases in uncertainty. This observation suggests the existence of a tradeoff between uncertainty and perception. Additionally, the bottom showcases strong relationship between uncertainty and distortion across diverse measures, demonstrating that any increase in uncertainty leads a significant rise in distortion Figure 6 displays similar trends for image inpainting, consistent with our super-resolution analysis and reinforcing the validity of our findings across diverse restoration tasks and data distributions. This is further visualized in Figure 2 which presents outputs from selected algorithms ordered by perceptual quality. The results clearly demonstrate an increase in hallucination (uncertainty) and distortion with increasing perceptual quality. Finally, Appendix
H presents additional results obtained via direct estimation of statistics in high dimensions, further supporting our theoretical analysis.
Conclusion
This study established the eeverttinnty-prrepppin tradeoff in generative restoration, demonstrating that high perceptual quality leads to increased hallucination (uncertainty), particularly in high dimensions. We characterized this tradeoff and its fundamental relation the inherent uncertainty of the problem,
'Note that MSE is a measure of distortion, whereas PSNR and SSIM are measures of inverse distortion; this accounts for the negative slope in the first two figures, and the positive slope in the third.
8

---

CON
INEAMATION
Ses LEM
Tro
23
Uncartannt
Uncartannt
SHGW
SASSON
28-
23
HYPERI04
36-3
255
2.5
JONC
152
2
14-
pr WRR0e010222
6.
MAE
Jassas
SONDAN
geaac
JERCAN
pott
jenas
(2M
LON
Emeedee
230
200
1o%.
24-
Joucte Mi
Uncortainty
L--3
Uncertainty
28-2
Uncertainty
Figure 5: Evaluation of SR algorithms. Top: Uncertainty-pereepe plane showing the tradeoff between perceptual quality and uncertainty (y-axis) for various perceptual measures. Bottom: Uncertainty-distoree plane showing the relationship between uncertainty and various distortion measures. Axis placement differs in the rows to highlight the distinet roles of uncertainty.
introducing the eeeertainty-perceptiin plane which may guide practitioners in understanding estimator performance. By extending our analysis MSE distortion, we showed that the distortion- -prrrppttnnn tradeoff emerges as direct consequence of the uncertainty-perceplee tradeoff. Experimental results confirmed our theoretical findings, highlighting the importance of this tradeoff image restoration.
7 Limitations
Our analysis is grounded in the theoretical framework of entropy as measure of uncertainty. Information theory offers powerful framework for quantifying uncertainty and dependencies in data, handling multivariate and heterogeneous data types, and capturing complex patterns. However, its wider adoption has been limited by the challenge of estimating information -eteatiin--thoettic measures in high dimensions. The curse of dimensionality makes accurate density estimation infeasible [2 leading many to rely simpler second-order statistics.
The development of practical tools for estimating statisties in high- figh-dimensional data remains an active area of research [76]. While initial approaches assumed exponential family distributions (e.g..
Gaussian) for tractable calculations [57], their performance degrades for long-tailed distributions. Non-parametric methods like binning strategies, including KDE and kNN estimators [6] 40 29), offer more flexibility are data-dependent and sensitive to parameter choices. Alternative approaches involve ensemble estimation 43 or von Mises Expansions 35 the distributional analog of the
Taylor expansion. Rotation- -Basd Iterative Gaussianization 46 presents a promising direction by transforming data into multivariate Gaussian domain, simplifying density estimation. However, its application to images has been limited to small patches due to the computational challenges of learning rotations based principal independent component analysis. A recent extension addresses this by utilizing convolutional rotations, enabling efficient processing of entire images 45
While accurately estimating high- digeesimenii entropy remains an active research area, Section utilizes a tractable upper bound. This alternative calls for further investigation into its potential for quantifying uncertainty and analyzing algorithm performance. Moreover, incorporating this bound into the design of new algorithms could enable explicit control over the eeeertaintt--ereeptii trade-off, potentially leading more reliable solutions.

---

pox giass
222
122
220
120
127
227
127
225
JOM
2.25
225
220
UML
222
HYPERIOA
1e-3
2.52
241
10%
3.55
2.50
PASE
E375
2.2
200
0:00
J.
325
235
0:3
232 235
3.44
2.54
2:35 Uncertainty
235
2.44
Uncortainty
Uncortainty
Figure 6: Evaluation of LDMs on image inpainting, highlighting the trade-off between uncertainty and perceptual quality (top) and the incertainty-distortion relationship (bottom). No model achieves both low uncertainty and high perceptual quality, with higher uncertainty generally leading to increased distortion. Differing axis placements emphasize the distinet roles of uncertainty.
Lastly, we focused our empirical validation on image super-resolution and inpainting, two benchmark problems in image restoration. Our analysis, however, applies any restoration task with non- invertible degradation. Hence, expanding the experiments to additional image-to-image tasks and domains such as audio, video, and text may reveal broader implications and applications of our work.
8 Broader Impact
Our work revealing a fundamental tradeoff' between uncertainty and perception in image restora- tion carries significant societal impact. Developers across various fields, including healthcare and autonomous systems, often integrate cutting-edge models into their applications, prioritizing state- of-the-art performance and perceptual quality. However, our work aims highlight a crucial factor often overlooked: the inherent tradeoff between uncertainty and perception. By raising awareness of this tradeoff, we empower developers to make informed decisions that prioritize safery and reliability over purely perceptual enhancements. For instance, in healthcare, potential restoration algorithms can be evaluated by plotting them on the peccertinty-perreppee of methods that strike the optimal balance for specific clinical needs. Furthermore, by understanding this inherent trade-off, practitioners can consider trading performance for better safety and resilience against potential misuse and misinterpretations.
plane, facilitating the identification
While primarily theoretical, our analysis yields a practical measure of uncertainty (or entropy), used in our experiments to visually and quantitatively illustrate our findings. This tractable uncertainty measure, or any differentiable alternative, can be incorporated into loss function during the training of generative models like GANs or as an optimization objective to guide the reverse process in diffusion models. This approach enables the development of algorithms that explicitly optimize for the tradcoff between uncertainty and perception.
10

---

References
[1] Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth,
P., Cao, X., Khosravi, A., Acharya, U.R., et al.: A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion 76, 243-297 (2021)
[2] Alaa, A., Van Der Schaar, M.: Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In: International Conference on Machine Learning. pp. 175-190. PMLR
(2020)
[3] Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.0551 (2021)
[4] Angelopoulos, A.N., Bates. S., Candès, E.J., Jordan, M.I., Lei, L.: Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.0100 (2021)
[5] Angelopoulos, A.N., Bates, S., Fisch, A., Lei, I.., Schuster, T.: Conformal risk control. arXiv
preprint ar2i 02814 (2022)
[6] Angelopoulos, A.N., Kohli, A.P., Bates, S., Jordan, M.I., Malik, J., Alshaabi, T., Upadhyayula,
S., Romano, Y:: Image-to- image-to-image to-emaae regression with distribution- free uncertainty quantification and applications in imaging. arXiv preprint arXiv:2202.05228 052665 (2022)
[7] Ashukha, A., Lyzhov, A., Molchanov, D., Vetrov, D.: Pitfalls of in-domain uncertainty estima- tion and ensembling in deep learning. arXiv preprint arXiv:2002. (2020)
[8] Bates, S., Angelopoulos, A., Lei, L.., Malik, J., Jordan, M.: Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM) 68(6), 1-34 (2021)
[9] Beirlant, J., Dudewicz, E.J., Györfi, L., Van der Meulen, E.C., et al.: Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences 6(1),
17-39 (1997)
[10] Belhasin, O., Kligvasser, I., Leifiman, G., Cohen, R., Rainaldi, E., Cheng, L.F., Verma, N., Varghese, P., Rivlin, E., Elad, M.: Uncertainty-awan opg-2-ecg for enhanced cardiovascular diagnosis using diffusion models. arXiv preprint arXiv:2405. 11566 (2024)
[11] Belhasin, O., Romano, Y., Freedman, D., Rivlin, E., Elad, M.: Principal uncertainty quantifica-
tion with spatial correlation for image restoration problems. arXiv preprint arXiv:2305. (2023)
[12] Bellman, R.: A mathematical formulation of variational processes of adaptive type. In: Proceed- ings of the Berkeley Symposium on Mathematical Statistics and Probability, p. 37. University of California Press (1961)
[13] Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., Zelnik-Manor, L.: The 2018 pirm challenge on perceptual image super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0-0 (2018)
[14] Blau, Y., Michaeli, T.: The perception-diitorrion tradeoff. In: Proceedings of the IEEE confer- computer vision and pattern recognition. pp. 6228-6237 (2018)
[15] Blau, Y., Michaeli, T.: Rethinking lossy compression: The rate-distortion titeddditortioonpprreptonn tradeoff.
In: International Conference on Machine Learning. pp. 675-685. 685. PMLR (2019)
[16] Blundell, C., Cornebise, J., Kavukeuoglu, K., Wierstra, D.: Weight uncertainty in neural
network. In: International conference on machine learning. pp. 1613-1622. PMLR (2015)
[17] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin. A.: Emerging properties in self-supervised vision transformers. In: Proceedings the IEEE/CVF international conference on computer vision. pp. 9650-9660 (2021)
[18] Chen, T., Fox, E., Guestrin, C.: Stochastic gradient hamiltonian monte carlo. In: International conference on machine learning. pp. 1683-1691. PMLR (2014)
[19] Cover, T.M.: Flements of information theory. John Wiley & Sons (1999)
[20] Damianou, A., Lawrence, N.D.: Deep gaussian processes. In: Artificial intelligence and statistics. pp. 207-215. PMLR (2013)
[21] Delattre, S., Fournier, N.: On the kozachenko-leonenke entropy estimator. Journal of Statistical Planning and Inference 185, 69-93 (2017)

---

[22] Ding, K., Ma, K., Wang. S., Simoncelli, E.P.: Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence 44(5), 2567-2581 (2020)
[23] Freirich, D., Michaeli, T., Meir, R.: A theory of the distortion--errcetioon tradeoff in wasserstein space. Advances in Neural Information Processing Systems 34, 25661-25672 (2021)
[24] Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: international conference machine learning. pp. 1050-1059. PMLR (2016)
[25] Gal, Y., Hron, J., Kendall, A.: Concrete dropout. Advances in neural information processing systems 30 (2017)
[26] Gal, Y., Islam, R., Ghahramani, Z.: Deep bayesian active learning with image data, In: Interna- tional Conference on Machine Learning. pp. 118--1192. PMLR (2017)
[27] Gasthaus, J., Benidis, K., Wang, Y., Rangapuram, S.S., Salinas, D., Flunkert, V., Januchhwskki,
T.: Probabilistic forecasting with spline quantile function rnns. In: The 22nd international conference on artificial intelligence and statistics, pp. 1901-1910, PMLR (2019)
(28)
Gawlikowski, J., Tassi, C.R.N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triehel, R., Jung, P., Roscher, R., et al.: A survey of uncertainty in deep neural networks. Artificial Intelligence Review pp. 1-77 (2023)
[29] Goria, M.N., Leonenko, N.N., Mergel, V.V., Novi Inverardi, P.L.: A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Journal of Nonparametric Statistics 17(3), 277-297 (2005)
[30] Gottschling, N. N.M., Antun, V., Hansen, A.C., Adcock, B.: The troublesome kernel-on hallucina- tions, no free lunches and the accuracy-stability trade-off in inverse problems. arXiv preprint arXiv:2001. 1.0258 (2020)
[31] Hepburn, A., Laparra, V., Santos-Rodriguez, R., Ballé, J., Malo, J.: On the relation between
statistical learning and perceptual distances. In: International Conference on Learning Repre- sentations (2021)
[32] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840-6851 (2020)
[33] Hu, R., Huang, Q., Chang, S., Wang, H., He, J.: The MBPEP: a deep ensemble pruning
algorithm providing high quality uncertainty prediction. Applied Intelligence 49(8), 2942-2955 (2019)
[34] Izmailov, P., Maddox, W.J., Kirichenko, P., Garipov, T., Vetrov, D., Wilson, A.G.: Subspace inference for Bayesian deep learning. In: Uncertainty in Artificial Intelligence. pp. 1169-11799 PMLR (2020)
[35] Kandasamy, K., Krishnamurthy, A., Poczos, B., Wasserman, L., et al.: Nonparametric von mises estimators for entropies, divergences and mutual informations. Advances in Neural Information Processing Systems 28 (2015)
[36] Kim, B., Xu, C.. Barber, R.: Predictive inference is free with the jackknife+ Advances in Neural Information Processing Systems 33, 4138-4149 (2020)
[37] Kivaranovic, D., Johnson, K.D., Leeb, H.: Adaptive, distribution-fres prediction intervals for deep networks. In: International Conference on Artificial Intelligence and Statistics. pp. 4346-4356. PMLR (2020)
[38] Kligvasser, I., Cohen, R., Leifman, G., Rivlin, E., Elad, M.: Anchored diffusion for video face reenactment. arXiv preprint arXiv:2407.15153 (2024)
[39] Kligvasser, Michaeli, T.: Sparsity aware normalization for gans. In: Proceedings of the AAAI
Conference on Artificial Intelligence. vol. 35, pp. 8181-8190 (2021)
[40] Kozachenko, L., Leonenko, N.: On statistical estimation of entropy of a random vector, problems inform. Transmission 23, 95101-16 (1987)
[41] Kozachenko, I.F., Leonenko, N.N.: Sample estimate of the entropy of a random vector. Prob- lemy Peredachi Informatsii 23(2), 9-16 (1987)
12

---

[42] Kutiel, G., Cohen, R., Elad, M.. Freedman, D., Rivlin, E.: Conformal prediction masks: Visualizing uncertainty in medical imaging. In: ICLR 2023 Workshop on Trustworthy Machine Learning for Healthcare (2023)
[43] Kybic, J.: High-dimensional mutual information estimation for image registration. In: 2004 International Conference on Image Processing. 2004, ICIP"04. vol. 3, pp. 1779-1782. IEEE (2004)
[44] Lakshminarayanan. B., Pritzel, A., Blundell, C.:
Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017)
[45] Laparra, V., Hepburn A., Johnson, J.E., Malo, J.: Orthonormal convolutions for the rotation based itcrative gaussianization. In: 2022 IEEE International Conference on Image Processing (ICIP). pp. 4018-4022 IEEE (2022)
[46]
Laparra, V. Johnson, J.F., Camps- Valls, G., Santos-Rodrrguez, R., Malo, J:: Information theory measures via multidimensionall gaussianization. arXiv preprint arXiv:2010.03807 (2020)
[47] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.
(2017)
[48]
Lee, J.A., Verleysen, M., et al.: Nonlinear dimensionality reduction, vol. 1. Springer (2007)
[49] Lei, J., Sell, M., Rinaldo, A., Tibshirani, R.J., Wasserman, L.: Distribution free predictive inference for regression. Journal of the American Statistical Association 113(523), 1094-1111
(2018)
[50] Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference computer vision and pattern recognition workshops. pp. 136-144 (2017)
[51] Louizos, C., Welling, M.: Multiplicativee normalizing flows for variational bayesian neural
networks. In: International Conference on Machine Learning. pp. 2218 1122722222222 PMLR (2017)
[52] MacKay, DJJ.: Bayesian interpolation. Neural computation 4(3), 415 447 (1992)
[53] Madiman, M., Melbourne, J., Xu, P.: Forward and reverse entropy power inequalities in convex geometry. In: Convexity and concentration, pp. 427-485. Springer (2017)
[54] Marin-Franch, I., Foster, D.H.: Estimating information from image colors: An application to digital cameras and natural scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(1), 78-91 (2012)
[55] Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statisties. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001. vol. 2, pp. 416 423. IEEE (2001)
[56] Nielsen, F: Hypothesis testing, information divergence and computational geometry. In: Inter- national Conference on Geometric Science of Information. pp. 241-248. Springer (2013)
[57] Nielsen, F., Nock, R.: closed-form expression for the sharma- mittal entropy of exponential families. Journal of Physics A: Mathematical and Theoretical 45(3), 032003 (2011)
[58] Ohayon, G., Michaeli, T., Elad, M.: The perception-robusttes, tradeoff in deterministic image restoration. arXiv preprint ar Xiv:2231 .09253 (2023)
[59] Pearce, T., Brintrup, A., Zaki, M., Neely, A.: High-quality prediction intervals for deep learning:
A distribution-free ensembled approach. In: International conference on machine learning. pp. 4075-4084, PMLR (2018)
[60] Posch, K. Steinbreenrr, J., Pilz, Variational inference to measure model uncertainty in deep neural networks. XiXiv preprint arXiv: 1902. 10189 (2019)
[61] Pothapakula, P.K., Primo, C., Ahrens, B.: Quantification of information exchange in idealized and climate system applications. Entropy 21(11), 1094 (2019)
[62] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning, pp. 7748-8763.. PMLR (2021)
13

---

[63] Ritter, H., Botev, A., Barber, D.: A scalable Laplace approximation for neural networks. In: 6th International Conference on Learning Representations, ICLR C01-Cooneeeecee Track Proceedings. vol. 6. International Conference on Representation Learning (2018)
[64] Romano, Y., Patterson, E.,, Candes, E.: Conformalized quantile regression. Advances in neural information processing systems 32 (2019)
[65] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 10684-10695 (2022)
[66] Sajjadi, M.S., Scholkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of the IEEE international conference on computer vision. pp.
(2017)
[67] Salimans, T., Kingma, D., Welling, M.: Markov chain monte carlo and variational inference: Bridging the gap. In: International conference machine learning. pp. 1218-1228 PMLR (2015)
[68] Sankaranarayanan, S., Angelopoulos, A.N., Bates, S., Romano, Y., Isola, P.: Semantic uncer- tainty intervals for disentangled latent spaces. arXiv preprint arXiv:2207.10077 (2022)
[69] Sesia, M., Candès, E.J.: A comparison of some conformal quantile regression methods. Stat 9(), e261 (2020)
[70] Shafer. G., Vovk, V: A tutorial on conformal prediction. Journal of Machine Learning Research 9(3) (2008)
[71] Shaham, T.R., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4514455-4555 4580 (2019)
[72] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recogni- tion. arxiv preprint arXiv:1409.1555 (2014)
[73] Song, J., Meng. C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
[74] Su, S., Yan, Q., Zhu, Y., Zhang, C., Ge, X., Sun, J., Zhang, Y.: Blindly assess image quality in the wild guided by self-adaptive hyper network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3667-3676 (2020)
[75] Sun, S.: Conformal methods for quantifying uncertainty in spatiotemporal data: A survey. arXiv preprint
.03580 (2022)
[76] Szabó. Z.: Information theoretical estimators toolbox. The Journal of Machine Learning
Research 15(1). 283-287 (2014)
[77] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9446-9454 (2018)
[78] Valentin Jospin, I.., Buntine, W., Boussaid, F., Laga, H., Bennamoun, M.: Hands-on Bayesian neural networks-a tutorial for deep learning users. arXiv e-prints pp. arXiv-2007 (2020)
[79] Van Erven, T., Harremos, P.: Rényi divergence and kullback-leibler divergence. IEEE Transac- tions on Information Theory 60(7), 3797-3820 (2014)
[80] Varshavsky-Hassin Hassid, M., Hirsch, R., Cohen, R., Golany, T., Freedman, D., Rivlin, E.: On the semantic latent space of diffusion-based text-to-speech models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 246 255 (2024)
[81] Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops. pp. 0-0 (2018)
[82] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600-612 (2004)
[83] Wu, D., Gao, I.., Xiong, X., Chinazzi, M., Vespignani, A., Ma, Y.A., Yu, R.: Quantifying uncertainty in deep spatiotemporal forecasting. arXiv preprint arXiv:2105. 1182 (2021)
14

---

[84] Wu, H., Zhang, Z., Zhang, W., Chen, C., Liao, L., Li, C., Gao, Y., Wang, A., Zhang, E., Sun, W., al.: Q-align: Teaching lmms for visual scoring discrete text-defined levels. arXiv preprint arXiv:2312. 17090 (2023)
[85] Xu, T., Zhang, Q. Li, Y., He, D., Wang, Z., Wang, Y., Qin, H., Wang, Y., Liu, J., Zhang, Y.Q.: Conditional perceptual quality preserving image compression. arXiv preprint arXiv:2308.08154 (2023)
[86] Yarom, M., Bitton, Y. Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E., Szpektor, I:: What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems 36 (2024)
[87] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.:
The unreasonable effectiveness perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586-595 (2018)
of deep features
[88] Zhang, W., Zhai, G., Wei, Y., Yang. X., Ma, K.: Blind image quality assessment via vision- language correspondence: A multitask learning perspective. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14071-10 14081 (2023)
15

---

A Conditional Divergence and Human Perception
In our context, perception is defined as the probability Psuccess of human observer successfully distinguishing between pair of natural and degraded images, drawn from Px,y), and pair of restored and degraded images drawn from Px,y), From a Bayesian perspective, the optimal decision rule maximizing Pruccess yields (156) Section 2):
1 1 Div(pxXY,Px.)
Psuccess
+ 2
where Div(px.Y,P3 (Px.)) is the total-variation (TV) distance. When D(pxy,, PPx.) 0, the two pairs are indistinguishable (Pauccess = 0.5), implying perfect perception quality. We generalize this beyond the total-variation (TV) distance to any conditional divergence, recognizing that the divergence that best relates to human perception remains an open question.
B Information-Theory Preliminaries
To make the paper self-contained, we briefly overview the essential definitions and results in information-theory Let X, Y and Z be continuous random variables with probability density functions px (I), PY (y) and (2) respectively. The space of probability density functions denoted by Q. We assume the quantities described below, which involve integrals, are well-defined and finite.
Definition (Entropy). The differential entropy of X, whose support is a set Sz, is defined by
M(X) - )(X)=1) [sx px (x) px)))..x
Definition 4 (Rényi Entropy). The Rényi entropy of order r 2 0 of X is defined by
1 log 1
(x)
The above quantity generalizes various notions of entropy, including Hartley entropy, collision entropy, and min-entropy. In particular, for we have
(1(X) & lim (x) = m-11
Definition 5 (Entropy Power). Let be 1(X) be the differential entropy of X Rdd. Then, the entropy Power of X is given by
N(X)
3b(X)
2Te
Definition (Divergence). statistical divergence any function D. Q R* which satisfies the following conditions for all p, q Ω:
1. (p(p.)) 2.
2. (p. iff p almost everywhere.
Table 1: Formulas for Multivariate Gaussian Distribution
Distribution
Quantity
Closed-Form Expression
X N(un, Ex)
A(X)
I (2 [En]
X N(J1, Ex)
N(X)
Σ 1/n
X N(Her Exd
h Xx)
2 In (n(()))) [Er]]-
X ~ N(px, Ex),
D1/2(X,8)
[ue - 2²y²² (In
(Mz My) + In
Y NWpy En)
16

---

Definition 7 (Rényi Divergence). The Rényi divergence of order r 2 0 between px and is
D, -- log (e(r))p"(x)ddk
The above establishes a spectrum of divergence measures, generalising the Kullback-Leiller diver- gence D1(X,Y) DxL(X, Y). Furthermore, it is important note that all orders (0, 1) are equivalent (79), since
1 r1-t - D D((.), VO <rrtte -l-r
Definition 8 (Conditioning). Consider the joint probability pXY and the conditional probabilities Pxly (aly) and Pzly (zlu). The conditional differential entropy of X I2 given Y is defined as
1(XI))
Pxy (2, v) log Y (aly)dady
5xy
Byy-py [N(X|V =y]
where Sxy is the support of pxy. Then, the conditional entropy power of X given Y is
N(X = 1 (
2n
Similarly, the conditional divergence between X and Z given Y is defined as
(x Y) & Ey-py [Xx = y, Z = y)l
For example, the conditional Rényi divergence is given by
(x
((-2 1 log y (alw)) pydy.
Table [
summarizes closed-form expressions for several quantities relevant to the multivariate Gaussian distribution. Below we present two fundamental results that form the basis of our analysis.
Lemma 1 (Maximum Entropy Principle [1)). Let X Rd be a continuous random variable with zero mean and covariance Eg. Define Xa ~ NO0 Ez) to be a Gaussian random variable, independent of X, with the identical covariance matrix Exzo Ex.. Then,
h(X) & h(XG),
N(X) S ((Xa) =1En11/4
Lemma 2 (Entropy Power Inequality (53). Let X and Y be independent continuous random variables. Then, the following inequality holds
N(X) N(Y) S N(X+Y)
where equality holds iff X and Y are multivariate Gaussian random variables with proportional covariance matrices. Equivalently, let X,, and Yg be defined as independent, isotropic multivariate Gaussian random variables satisfying b(Xgg) -((X) and h(y h(Y). Then,
h(X) + h(x,) h(Yg) h(gg + Yg) S 3(X+Y)
C Derivation of Example 1
Since x E [X]Y] + Z, then X|Y A/EE [X]Y] of). Moreover, X|Y ~ ([ o o2 where of 1+
Thus, the conditional error entropy is given by N(} X of + o and the
symmetric KI. divergence is DskL(X, (ox x
Lt 20.0,
1, leading the following problem
0 +
U(P) min a
0 + 0²
= P].
(10)
20.00
17

---

MAIx MIX
RAAM
(x XIX)
Figure 7: The Envertainty-Perceptiin function for Example As perception quality improves, the minimal achievable uncertainty increases, suggesting tradeoff governed by the inherent uncertainty.
Therefore, seek the minimal value of o that satisfies the constraint. Note that the minimal value attained at the boundary the constraint set, where the inequality becomes an equality
0 + o
1 P = -2 1)0= + og =0. O.
(11)
20
The solution to the aforementioned quadratic problem is o. 0g ( Substituting the later into the objective function, we obtain
U(P) =0 1+ (p+1-V(P+1)) 1)
(12)
Finally, the entropy power of an univariate Gaussian distribution equals its variance og (( Figure 7 visualizes the resulting eecertainty-perceplii tradeoff.
D Proof of Theorem
First, the constraint C(P) (X
Dx, X P) defines compact set which is continuous in
P. Hence, by the Maximum Theorem 191. U(P) is continuous. In addition, U(P) is the minimal eтror entropy power obtained over a constraint set whose size does not decrease with P, thus, U(P) non-increasing in P. Any continuous non non-increasing function is quasi-lines linear, For the lower bound consider the case where P = co, leading to the following unconstrained problem
U(xo) A min N(X X(Y).
(13)
PXJY
For any P> 0t holds that U(co) < UP),, and by Lemma 2 we have
N(X + min N(XIV) < U(x)).
(14)
PRI
Since minpgry N(XY)) 2 0 we obtain
VP 0::0: N(XY)) S UP).
(15)
Next, we have U(P) (0)) N(E, X(Y) where PX,,Y = Pxly. Define 2 8, 8 then
Σely Σgyy + Σzly = 2Exly where we use that X and X are independent given Y. Thus.
((0) N(IY)) (Vvl)) E.yll
1/4
|2 1/d
Er yl 1/d (N (16)
where the first inequality due Lemma Finally, for any 2 it holds that U(P) < U()) which implies (()) < 2N (No (Y), completing the proof.
18

---

E Proof of Theorem 2
Assuming (X(,, X X is convex in its second argument, the constraint represent a compact, convex set. Moreover, 1xx - X is strietly-conave w.r.t P as a composition of a linear function (convolution) with strictly-coneave function (entropy). Therefore, we minimize log-concave function over convex domain and thus the global minimum is attained on the set boundary where Dr (X, Y)) P.
F
Proof of Theorem 3
We begin with applying Lemma] 1 and Lemma to bound the objective function as follows
N(X, + N(x = N(X, - X,IV) & N(} x & N(8G - XalV). (17)
Note that the bounds are tight as the upper bound is attained when X Y and X|Y are multivariate Gaussian random variables, while the lower bound is attained if we further assume they isotropic. Thus, we can bound the ppeeepttn function as follows
(g U(P) < Uc(P)
(18)
where we define
(P) A min ((X,)))+N(X,, D1/2(X x s SP]
&
PXyIY
(19)
Uc(P) & min
(N(XG- -
Xc s
PRair
The above quantities can be expressed in closed form. We start with minimization problem of the upper bound which can be written as
Xe E &ol -v)
v) s { (20)
Uc(P) = min
{
PXclY
2Te
where the expectation is over y Y. Substituting the expressions for h(Xo X = y) and
XolY y). we get
log (2 Eeppt)
(Eel) + Enju) /2
Uc(P) min {Eny)
log
2
Eajpl
|Em|a|
(21)
Notice the optimization is with respect to the covariance matrices { (E Simplifying the above, we can equivalently solve the following minimization
log (Enly + Emjy) /2|
min Eeny
E log Egly +
s.t.
E
<.P. P.
(22)
|> |
The solution of a constrained optimization problem can be found by minimization the Lagrangian
[(Eely + Exlr) /2
L ((Ean),)) A) A E [log Σajn + Enlv|] + A
E log
P
(23)
v Eetpl
E
Since expectation is linear operation and using that P = E [P], we rewrite the above
[(Ealy + Ez/p)
L ((Emy).A) log
Eajy
Ezjo
log
(24)
v //aayll EDevvl
The expression within the expectation can be written
1
log |izlo] -P): (25)
log ΣEyy + Ezlvl +
log (Ealy + Exly) /2|
log |
2
2
19

---

Next, according to KKT conditions the solutions should satisfy
OL 8E
= 0. Using the linearity of the
expectation and differentiating 25 w.r.t Egyy y we obtain
(Ealy + Es(u) +t ((Ea)+ + Enlo)
3-1 = 0
(26)
2 51y
Multiplying both sides by (Ealy + Ex/u) : we have
+ 2
Exly D3 = 0
2
2 1 1= E2 5ky
(1 + 2
2
(27)
(A 2)IBAy = VΣ
X
3 Ezgy x+2
Define 7 = A42 80 Egly
= 7 Egyy- Substituting the latter into the constraint we get
log (7Ez1l + Eg(u) /2
log Mnllv
2 log
Emly = P
1 1
n
= noo
log? = P 2
(1 + 7)²
(28)
4y
+ 7 27 + 4ye
+ *P() 203 - V /2eedr - -1)2 - 1.
Thus, we obtain that
Uo(P) -P() N(XG|V)
(29)
where
= +1= = 2 12002- 10c 112 11
(30)
Notice ((0) = 2, while limp- (PP) so 1 S (PP) 22 Following similar steps where replace Eg4y and Enjy with N(( and N(X respectively, we derive
Up(P) = (P)) -(())))
(31)
G Proof of Theorem
Define & A X X. Then,
E ||8 x|]
E
E || x|||||]
IE
1 1 [|]|||\]]
E 1
[
g|y)
(H [EE*|Y])
1 2 Tr (Zelw)
E
Tr
E
1/0
E [iee + Exir 1/4
(6)
1
e 3h18 - xx
E
(
2Tee
-
1
e (xx - 3 = =N -N(==N(X-00 -xx-xlx).
(d)
2xe
2xe
where (a) by the law of total expectation, (b) is due to the inequality of arithmetic and geometric
means, (c) follows Lemma and (d) is according to Jensen's inequality.
20

---

H Results via Direct Estimation
we used practical measures for perceptual quality and a tractable upper bound for uncertainty. Here, we supplement those results with direct computations of entropy and divergence work [14 [3], we treat images as stationary sources and extract 9 x 9 patches. To estimate Rényi divergence for perceptual quality assessment, we first model the probability density functions using kernel density estimation. Subsequently, we compute the divergence through empirical expectations. Baachenko-LLenentee estimator, which calculates the patch
Estimating high-dimensional statisties is prone errors 46
dige-simeniional setting. Following prior
Uncertainty is estimated using the
sample differential entropy based nearest neighbor distances 41 21 9
54). Results, shown in
Figure strongly align with the trends observed in Figure
1e-2
10-2
1.2
IDMc
DOIN
2.8
LDM DOIN
DMss
2.6
LDMs7
LDM
DOPR
1.0
SinGAN
DOPH
2.4
Enhance
Enhancellet
SAGAN-V6G54
SRGAN-V6054
Uncertainty
0.9
Distortion
ESRGANSINGAN
ESRGAN
SRGAN-VGG22
0.8
SANGAN
2.0
SRGAN-1GG22
0.7
1.8
Dicubic
SRResNet-WGG22
VRGG2Ner-VGG22
DIP
Bio,bic
14
SRResNet
RRS
0.4
1.2
DSR
0.8
Perception
Uncertainty
1e-2
Figure 8: Evaluation of SR algorithms via direct estimation of high-dimensional statisties. Left: Uncertainty-perception plane demonstrating the tradeoff between perceptual quality and uncertainty. plane illustrating the relation between uncertainty and distortion.
Right: Uncertainty-distorii Results are consistent with the finding in Figure
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We ensured the abstract and introduction the abstract and introduction describe our major contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations of our work are discussed extensively in dedicated section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: A detailed problem formulation, including all assumptions, is provided om dedicated section. We have taken great care to ensure the clarity and correctness of our proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Although our main contribution is theoretical, we complement with an empirical analysis of existing open models. This analysis is presented with complete details to ensure full reprodueibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
23

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our experimental analysis utilizes open-source models and datasets. While current submission does not include the code, provide comprehensive description of our experimental setup to facilitate reproduction of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.co wttpccppppppcccceteoec
/eweee/eeeeeaiaaeeceeeeeeeeeeaee cobomiission pessoonooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our analysis centers on pre-trained models applied to open datasets. Section and Appendix (Hlpoiidd the technical details necessary for reproducing results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Our experimental analysis involves applying pre-trained models to a fixed set of open datasets, resulting in deterministic outputs. As there is inherent randomness or variation in the experimental process, traditional statistical significance measures like error bars confidence intervals are not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our primary contribution is theoretical, and the accompanying experimental analysis is computationally lightweight, requiring only basic processing on standard CPU given the ground-trudh -trut, distorted, and recovered images. Therefore, detailed compute resource specifications are not essential for reproducing the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp::/neurippp: http::////rriiscceeeee httpp//ps...pppooeccctcccccclala
Answer:
[Yes]
Justification:
We confirm that our study aligns with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification:
Broader Impacts of our work are discussed in their own dedicated section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses analyzing existing open-source models and datasets, and therefore does not introduce new models or datasets that require specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
available models and datasets utilized our work are properly cited and acknowledged in the paper.
Justification: All relevant
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ comddatasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "u5gtyY4m5f", "reviewer": "Reviewer_W9yk", "summary": "Deep generative models have achieved remarkable performance in image restoration, resulting in generated images of high visual quality. However, these models often produce high-frequency details that are not consistent with the ground-truth images. Such hallucinations introduce uncertainty in the generated content and affect the reliability of model predictions. This paper defines the uncertainty of image restoration and the uncertainty-perception (UP) function, and reveals an uncertainly-perceptual trade-off. The paper theoretically analysis the relationship between the uncertainly-perceptual trade-off and the perceptual-distortion trade-off. The theoretical findings are validated through experiments with image super-resolution algorithms. Results show that no model can achieve both low uncertainty and high perceptual quality simultaneously.", "questions": "1. The paper [1] quantifies the structural uncertainty for image restoration. The intuitive explanation is better for readers to understand the inherent uncertainty mentioned in the paper.\n\n2. The practical estimation for computing divergence and uncertainty should be elaborated, which is challenging for real-world images that are typically high-dimensional.\n\n3. While the theoretical analysis is robust, the paper could benefit from more concrete examples of how the findings apply to real-world scenarios, like healthcare or autonomous systems.\n\n[1] Belhasin O, Romano Y, Freedman D, Rivlin E, Elad M. Principal uncertainty quantification with spatial correlation for image restoration problems. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023 Dec 14.", "limitations": "The paper aims to quantify the potential limitation (e.g., hallucination) of generative models. At the end of the paper, it discusses the limitations of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  This is a timely paper that analyses the hallucination of deep generative models. It proposes a novel insight of the phenomenon of hallucinations in generative models, a critical issue that affects the reliability of image restoration tasks.\n\n2. The paper adopts the Bayesian framework to analyze the tradeoff between uncertainty and perception. This framework helps in quantifying the inherent uncertainty in generative models and establishes a theoretical foundation for understanding the limitations of these models.\n\n3. The theoretical findings are empirically validated using single-image super-resolution algorithms. This strengthens the credibility of the theoretical analysis of the study.\n\n4. Following the definition, a concrete example (e.g., example 1) is illustrated which can help the readers better understand the concept.", "weaknesses": "1. Experiments only provide results in image super-resolution, how about applying the proposed method on other restoration tasks?\n2. The experiments primarily use synthesized datasets, such as BSD100. The paper would benefit from including experiments on more diverse and real-world datasets to validate the findings under different conditions and data distributions.\n3. The paper relies on entropy and Rényi divergence for theoretical analysis. However, the practical estimation of high-dimensional entropy is challenging. Although the authors use a tractable upper bound for uncertainty, the practical estimation methods and their limitations are not thoroughly discussed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort the reviewer has dedicated to reviewing our manuscript. Below we address in detail the major concerns raised by the reviewer.\n\n**Weaknesses**\n\n1-2. *Experiments* - We acknowledge the reviewer's concerns regarding the experiments on image super-resolution using the BSD100 dataset. While our primary focus is theoretical, we sought to validate our findings via numerical experiments. We initially chose super-resolution due to its status as a benchmark in image restoration, the wide availability of datasets and models, and its extensive study in related literature on the perception-distortion tradeoff. However, in response to the reviewer's criticism, we have expanded our experiments to include an evaluation of latent diffusion models in image inpainting using the SeeTrue* dataset, a diverse benchmark for image-text alignment containing both real and synthetic text-image pairs. These new results, detailed in the 'Author Rebuttal' and the accompanying PDF, align with our existing findings, showing their validity across different restoration tasks and under diverse conditions and data distributions.\n\n*Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E. and Szpektor, I., “*What you see is what you read? improving text-image alignment evaluation*”. NeurIPS 2023.\n\n3. Our work primarily focuses on the theoretical analysis of the relationship between uncertainty and perception, leveraging the properties of information-theoretic measures like entropy and divergence. However, we acknowledge the reviewer's valid concern regarding the challenges of estimating these statistics in high dimensions. In response, we have expanded our discussion on this topic in the revised manuscript to include a comprehensive review of practical estimation methods and their limitations. The following discussion, along with the accompanying full references, will be included in the revised manuscript:\n\nInformation theory offers a powerful framework for quantifying uncertainty and dependencies in data, handling multivariate and heterogeneous data types, and capturing complex patterns. However, its wider adoption has been limited by the challenge of estimating information-theoretic measures in high dimensions. The curse of dimensionality makes accurate density estimation infeasible [1, 2], leading many to rely on simpler second-order statistics.\n\nThe development of practical tools for estimating statistics in high-dimensional data remains an active area of research [3]. While initial approaches assumed exponential family distributions (e.g., Gaussian) for tractable calculations [4], their performance degrades for long-tailed distributions. Non-parametric methods like binning strategies, including KDE and kNN estimators [5, 6, 7], offer more flexibility but are data-dependent and sensitive to parameter choices. Alternative approaches involve ensemble estimation [8] or von Mises Expansions [9], the distributional analog of the Taylor expansion. Rotation-Based Iterative Gaussianization (RBIG) [10] presents a promising direction by transforming data into a multivariate Gaussian domain, simplifying density estimation. However, its application to images has been limited to small patches due to the computational challenges of learning rotations based on principal or independent component analysis. A recent extension addresses this by utilizing convolutional rotations, enabling efficient processing of entire images [11].\n\nOur theoretical study investigates the relationship between uncertainty, perception, and distortion in image restoration through the lens of information theory. We uncover a novel uncertainty-perception tradeoff and its connection to the well-known distortion-perception tradeoff. While primarily theoretical, our analysis yields a practical measure of uncertainty (or entropy), used to visually and quantitatively illustrate our findings. This measure potentially can be replaced by the aforementioned estimators for broader applications.\n\n\n**Questions**\n\n1. Our work utilizes the Bayesian framework, treating recovery error as a random variable and employing entropy as a natural measure of uncertainty**. However, estimating entropy in high dimensions presents challenges. Over time, practical methods have emerged that rely on alternative definitions of uncertainty. One such method, proposed by Belhasin et al., quantifies uncertainty volume using principal components of the empirical posterior probability density function. We agree with the reviewer that this novel definition is intuitive and leads to a practical method for uncertainty quantification, which we briefly discuss in the current manuscript. In the revised version, we will expand our discussion and description of this approach, acknowledging its merits and relevance to our work. \n\n** Cover, Thomas M.; Thomas, Joy A. (1991). *Elements of Information Theory*. \n\n2. Please see the responses to weakness above. We have carefully considered your feedback and, in response, have expanded our discussion on advancements in the practical estimation of statistics in high dimensions.\n\n3. Developers across various fields, including healthcare and autonomous systems, often integrate cutting-edge models into their applications, prioritizing state-of-the-art performance and perceptual quality. However, our work aims to highlight a crucial factor often overlooked: the inherent tradeoff between uncertainty and perception. By raising awareness of this tradeoff, we empower developers to make informed decisions that prioritize safety and reliability over purely perceptual enhancements. For instance, in healthcare, potential restoration algorithms can be evaluated by plotting them on the uncertainty-perception plane, facilitating the identification of methods that strike the optimal balance for specific clinical needs. In light of your feedback, we will address the points mentioned above in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9owcxvmLK", "reviewer": "Reviewer_xyrt", "summary": "The paper employs information-theory tools to characterize a tradeoff between uncertainty and perception in image restoration. They prove that high perceptual quality leads to increased uncertainty and the uncertainty-perception trade-off induces the distortion-perception trade-off. The theoretical results are illustrated with experiments in image super-resolution tasks.", "questions": "1. The uncertainty-perception plane is based on SFID, PDL and LPIPS, the relationship is still exists on stronger vision-language IQA like LIQE[1], Q-ALIGN[2]? \n2. Some recent SR methods[3,4] explore a better trade-off between perception and artifacts. How they perform in uncertainty-perception and uncertainty-distortion measurement?\n\nRefs:\n[1] Blind image quality assessment via vision-language correspondence: A multitask learning perspective. CVPR, 2023.\n[2] Q-ALIGN: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. arXiv:2312.17090.\n[3] DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models. ICML, 2023.\n[4] Details or artifacts: A locally discriminative learning approach to realistic image super resolution. CVPR, 2022.", "limitations": "See weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The authors provide clear theoretical and quantitative demonstrations.", "weaknesses": "1. The trade-off between perception and distortion has been discussed in some previous works. The authors should clarify their contribution especially compare to previous works[1,2].\n2. This paper establishes the theoretical relationship between uncertainty and perception. However, the authors do not provide practical applications, e.g. how to use this relationship in restoration task.\n\nRefs:\n[1] The Perception-Distortion Tradeoff. CVPR, 2018.\n[2] The Perception-Robustness Tradeoff in Deterministic Image Restoration. arXiv:2311.09253.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and constructive feedback. Below we address in detail the major concerns raised by the reviewer.\n\n**Weaknesses** \n\n1. We acknowledge the extensive literature on the perception-distortion tradeoff, particularly the seminal work demonstrating the inherent conflict between these two factors [1]. This work quantifies perception using a theoretical divergence between true and estimated distributions, establishing a tradeoff valid for any distortion measure. More recent research [2] has introduced a tradeoff between perception and robustness in restoration algorithms, utilizing the Lipschitz constant as a measure of robustness and defining \"joint perceptual quality\" based on the Wasserstein distance between joint distributions. This study reveals that improving joint perception leads to increased sensitivity in algorithms.\n\nWhile these works provide valuable insights into the relationship between perceptual quality, distortion, and robustness, crucial aspects for understanding generative models, our work offers a distinct contribution by focusing on the concept of uncertainty. Although related, uncertainty plays a different role than distortion in image restoration. Distortion metrics quantify the fidelity of the restored image to the ground truth, typically through error size. Uncertainty metrics, on the other hand, quantify the range of possible solutions, characterizing confidence in the restoration itself. This distinction is vital for decision-making, as high distortion may lead to incorrect choices, while high uncertainty implies low confidence, making it difficult to make any informed decision. Our work thus complements existing research by incorporating uncertainty into the broader discussion of image restoration quality.\n\nIn the revised manuscript, we will expand the discussion on these references [1, 2] to further emphasize our unique contribution and the points mentioned above.\n\n2. We understand the reviewer's concerns regarding the practical implications of our work. Our primary goal is to raise awareness among developers in various fields about the inherent tradeoff between perception and uncertainty. This understanding allows them to prioritize safety and reliability alongside perceptual enhancements when integrating cutting-edge models into desired applications.\nIn practice, our proposed uncertainty-perception plane can serve as a valuable tool for evaluating potential restoration algorithms, facilitating the identification of methods that achieve the desired balance for specific applications. Moreover, the tractable uncertainty measure used in our experiments, or any differentiable alternative, can be incorporated into a loss function during the training of generative models like GANs or as an optimization objective to guide the reverse process in diffusion models. This approach enables the development of algorithms that explicitly optimize for the tradeoff between uncertainty and perception.\n\nIn the revised manuscript, we will expand the discussion on the practical implications of our work on the development and evaluation of image restoration models.\n\n**Questions**\n\n1. We thank the reviewer for raising this important point. While we initially used SFID, PDL, and LPIPS as common and widely accepted measures of perception, we acknowledge the significant progress in developing new metrics like LIQE and Q-ALIGN that better align with human perception. In response to the reviewer’s feedback, we have conducted additional experiments on image inpainting, measuring perception with LIQE and Q-ALIGN. The results, provided in the 'Author Rebuttal' and accompanying PDF, support our existing findings, showing that an increase in perceptual quality generally correlates with increased uncertainty. We will include these additional experiments in the revised manuscript.\n\n2. We appreciate the reviewer highlighting the recent advancements in super-resolution (SR) methods that explore the trade-off between perception and artifacts [3, 4]. While time constraints prevented us from evaluating these specific methods within the rebuttal period, we are confident that our theoretical analysis remains valid regardless of the specific recovery algorithm. Our framework establishes a fundamental relationship between uncertainty and perception, independent of the algorithm's implementation details. Therefore, we anticipate that the mentioned SR methods would exhibit behavior consistent with our findings, demonstrating a similar relationship between uncertainty and perception. In the revised manuscript, we plan to include an evaluation of these recent SR methods to further validate and expand upon our theoretical framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ys4jumffOR", "reviewer": "Reviewer_WuZt", "summary": "This paper presents a theoretical perspective towards hallucinations and reveals a tradeoff between uncertainty and perception for image restoration problem. Additionally, the paper points out that uncertainly-perception tradeoff can induce the well-known perception-distortion tradeoff.", "questions": "see weakness", "limitations": "The authors discuss the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a theoretical interpretation about the hallucinations problems of inverse problem, which may offer useful guidance for further practical research.\n2. The writing is good and the paper is easy to follow.\n3. Rich theoretical results about uncertainly-perception tradeoff and its relationship with perception-distortion tradeoff.", "weaknesses": "1. Typically, perception can be measured by criteria like LPIPS.  Why can the conditional convergence (Eq 1) provide measurement for perception? Can the authors provide some intuitive explanation?\n2. The theorem 2 is based on a strict assumption that $D_v$ is convex in its second argument. Can you proof this directly?\n3. The visual results are not enough. The authors should provide some examples that contain hallucinations. It seems that in Figure 5, the details are realistic-looking details but not hallucinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort  in evaluating our manuscript. We have taken the feedback into consideration and responded to the specific points raised below.\n\n**Weaknesses** \n\n1. In our context, perception is defined as the probability $p_\\text{success}$ of a human observer to successfully distinguish between a pair of natural and degraded images (drawn from $p_{X,Y}$) and a pair of restored and degraded images (drawn from $p_{\\hat X, Y}$). From a Bayesian perspective, the optimal decision rule maximizing $p_\\text{success}$ yields**:\n$$p_\\text{success}=\\frac{1}{2}+\\frac{1}{2}D_\\text{TV}(p_{X,Y},p_{\\hat X,Y})$$\nwhere $D_\\text{TV}(p_{X,Y},p_{\\hat X,Y})$ is the total-variation (TV) distance.\nWhen $D(p_{X,Y},p_{\\hat X,Y})=0$, the two pairs are indistinguishable ($p_\\text{success}=0.5$), implying perfect perception quality. We generalize this beyond the total-variation distance to any conditional divergence, recognizing that the divergence that best relates to human perception remains an open question. However, computing divergences in high dimensions is challenging, leading to the use of practical alternatives like LPIPS. Following the reviewer’s comment we will include the above explanation in the revised manuscript.\n\n** See section 2 in the following paper: ​​Nielsen, F., 2013, August. Hypothesis testing, information divergence and computational geometry. In International Conference on Geometric Science of Information (pp. 241-248). Berlin, Heidelberg: Springer Berlin Heidelberg.\n\n2. While Theorem 2 assumes convexity of $D_v$ in its second argument, this is not a restrictive condition. In fact, most widely-used divergence functions, notably all $f$-divergences (such as KL divergence, total variation distance, Hellinger distance, and Chi-square divergence), exhibit this property.  This broad applicability is further highlighted by the convexity of the Rényi divergence, used in our work, in its second argument, which we prove next.\nThe Renyi divergence with $r\\geq 0, r\\neq 1$ between two probabilities $p$ and $q$ is given by\n$$D_r(p,q)=\\frac{1}{r-1}\\int p(x)^rq(x)^{1-r}dx.$$\nNotice that the function $f(q)=\\frac{1}{r-1}q^{1-r}$ is convex, therefore, for any $0\\leq\\lambda\\leq 1$ and distributions $q_1$ and $q_2$ we have\n\\begin{align*}\n    D_r(p,\\lambda q_1+(1-\\lambda)q_2)\n    &=\\frac{1}{r-1}\\int p(x)^r(\\lambda q_1(x)+(1-\\lambda)q_2(x))^{1-r}dx \\\\\\\\\n    &=\\int p(x)^r\\frac{1}{r-1}(\\lambda q_1(x)+(1-\\lambda)q_2(x))^{1-r}dx \\\\\\\\\n    &\\leq \\int p(x)^r\\frac{1}{r-1}\\lambda q_1(x)^{1-r}+ \\int p(x)^r\\frac{1}{r-1}(1-\\lambda) q_2(x)^{1-r} \\\\\\\\\n    &=\\lambda\\frac{1}{r-1} \\int p(x)^rq_1(x)^{1-r}+(1-\\lambda)\\frac{1}{r-1} \\int p(x)^rq_2(x)^{1-r} \\\\\\\\\n    &= \\lambda D_r(p,q_1)+(1-\\lambda)D_r(p,q_2),\n\\end{align*}\nimplying the Renyi divergence is convex. \nNote that the proof above omits the case of $r=1$ for simplicity. For a comprehensive proof and further properties, see Theorem 12 in\nVan Erven, T. and Harremos, P., 2014. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory, 60(7), pp.3797-3820.\n\n3. We acknowledge the reviewer's concerns regarding the visual results and agree with the suggestion to provide additional examples. To address this, we have conducted further experiments on image inpainting, with new visual results presented in the 'Author Rebuttal' and the accompanying PDF. These results clearly demonstrate the phenomenon of hallucination in image restoration, where the degree of hallucination increases with the improvement in the perceptual quality of the restoration algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PiGnYOARBS", "reviewer": "Reviewer_DvxH", "summary": "This work aims to provide a theoretical analysis about the uncertainty-perception trade-off in generative models, corresponding to the fidelity-naturalness trade-off of the generated images. By defining the inherent uncertainty and formulating a uncertainty-perception （UP) function, the authors proves that the UP function is globally lower-bounded by the inherenet uncertainty. Additionally, they derive that perfect perceptual quality requires at least twice the inherent uncertainty. The proposed theoretical framework establish a relationship between uncertainty and MSE, resembling the well-known perception-distortion trade-off. All theoretical findings are empirically verified with image super-resolution algorithms.", "questions": "I think it would be more convincing if the authors can provide more experimental results in real-world SR tasks.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ This is a timely theoretical analysis about the hallucination phenomenon widely occurs in generative models.\n+ The proposed theoretical framework helps practitioners better understand the tradeoff between uncertainty and perceptual quality, guiding them to tune the models in real-world safety-sensitive applications.\n+ Detailed proofs are provided.", "weaknesses": "- In my opinion, LPIPS, MSE, PSNR, SSIM are all full-reference image quality metrics that quantify the fidelity of the restorted images with respect to the ground-truths. Would it be more reasonable to adopt no-reference image quality metrics to quantify the perception ?\n- The experiment part is relatively weak, where more quantitative examples are expected.\n- The GT image is not presented in Fig. 5, making it difficult to assess the fidelity of the resorted results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our manuscript and providing constructive feedback. Below, we respond to the specific points raised by the reviewer. \n\n**Weaknesses**\n\n1. While our initial analysis utilized full-reference metrics (LPIPS, MSE, PSNR, SSIM) due to their widespread acceptance, we acknowledge the reviewer's suggestion regarding no-reference metric to quantify perception. We have therefore performed new experiments on image inpainting using state-of-the-art perceptual measures LIQE and Q-ALIGN. The results, available in the 'Author Rebuttal' and accompanying PDF, reinforce our conclusions, demonstrating a clear link between perceptual quality and increased uncertainty. We will revise the manuscript to incorporate these findings.\n\n2. We acknowledge the reviewer's critique regarding the experimental aspect of our manuscript. In response, we have expanded our evaluation to include latent diffusion models in the context of image inpainting, utilizing the SeeTrue** dataset. This dataset provides a diverse benchmark for image-text alignment, encompassing both real and synthetic text-image pairs. The new results, detailed in the 'Author Rebuttal' and the accompanying PDF, further validate our findings across different restoration tasks and under a wider range of conditions and data distributions.\n\n** Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E. and Szpektor, I., “What you see is what you read? improving text-image alignment evaluation”. NeurIPS 2023.\n\n3. We thank the reviewer for highlighting this important point. We will add the ground truth image to the revised manuscript to provide a clearer reference point. Additionally, in response to the reviewer's feedback, we have included new visual results in the 'Author Rebuttal' and the accompanying PDF. These results more effectively illustrate the phenomenon of hallucination in image restoration, clearly demonstrating that the degree of hallucination increases as the perceptual quality of the restored image improves.\n\n\n**Questions** \n1. We appreciate the reviewer's suggestion to include additional experimental results on real-world super-resolution tasks. While we acknowledge the value of such experiments in demonstrating the real-world applicability of our theoretical findings, time constraints prevented us from conducting them within the scope of this rebuttal. However, we believe that the expanded experiments presented in the 'Author Rebuttal' and accompanying PDF, which include a diverse dataset and the use of no-reference image quality metrics, significantly strengthen the empirical support for our theoretical framework.\nWe have included these results to demonstrate the robustness of our findings across different image restoration tasks and under a wider range of conditions and data distributions. We recognize that further validation on real-world SR tasks would be valuable, and we hope to incorporate such experiments in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their efforts in evaluating our manuscript, their overall positive feedback, and their constructive criticisms, which have significantly strengthened our contribution. We have provided detailed responses to each reviewer individually. In the following, we summarize the major revisions made in response to the collective feedback:

**Extended Experiments**

* Image Inpainting: In addition to our initial super-resolution experiments, we have conducted extensive new experiments on image inpainting. We specifically chose latent diffusion models for this task due to their state-of-the-art performance and growing popularity.
* Diverse Dataset: We utilized the SeeTrue* dataset, a benchmark known for its diverse range of real and synthetic text-image pairs. This allows us to assess the validity of our findings across different restoration tasks and under a broader spectrum of conditions and data distributions.

*Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E. and Szpektor, I., “What you see is what you read? improving text-image alignment evaluation”. NeurIPS 2023.

* No-Reference Metrics: We incorporated the state-of-the-art no-reference perceptual quality metrics LIQE and Q-ALIGN to quantify perception in these new experiments.

* Visual and Quantitative Results: We have included new visual and quantitative results in the attached PDF. These results support our existing findings, showing that an increase in perceptual quality generally correlates with increased uncertainty.  Furthermore, the visual results more effectively illustrate the phenomenon of hallucination in image restoration, clearly demonstrating the increase in hallucination with improved perceptual quality.

**Extended Discussion and Theoretical Context**

* Practical Implications: We have expanded the discussion on the practical implications of our theoretical analysis, emphasizing how our findings can guide the development and evaluation of image restoration models, particularly in safety-critical applications.
* Estimation of High-Dimensional Statistics: We have provided a more comprehensive discussion on the challenges and advancements in estimating statistics like entropy and divergence in high dimensions. This includes an overview of practical estimators and their limitations, offering valuable insights for researchers and practitioners.
* Related Work: We have further clarified our unique contribution by expanding the discussion on related previous works and highlighting the distinct role that uncertainty plays in image restoration compared to distortion or robustness.

We are confident that these revisions address the reviewers' concerns and enhance the overall quality and impact of our work. We once again thank the reviewers for their valuable feedback and hope that our responses adequately address their questions and suggestions.

Meta-review: This paper provides a theoretical analysis of the tradeoff between perceptual quality and uncertainty in generative image restoration models, revealing that higher perceptual quality leads to increased uncertainty, often manifesting as hallucinations. The work is grounded in information theory, with the authors establishing a connection between the uncertainty-perception tradeoff and the perception-distortion tradeoff. The theoretical findings are empirically validated through experiments, initially focused on single-image super-resolution.

Reviewers appreciated the paper's theoretical rigor and its timely contribution to understanding the limitations of generative models. However, concerns were raised about the limited scope of the experiments and the practical applicability of the findings. In response, the authors extended their experiments to include image inpainting and employed no-reference perceptual quality metrics, which addressed these concerns effectively. They also provided a more comprehensive discussion on the challenges of estimating high-dimensional statistics and added additional visual results to better illustrate the phenomenon of hallucinations.

Overall, the paper was well-received after the discussions, with reviewers acknowledging the strengthened empirical support and the practical relevance of the work. The recommendation is to accept the paper as a poster presentation, given its solid theoretical contribution and the effective responses to reviewer concerns.
Decision: Accept (poster)