A Theory of the Distortion-Perception Tradeoff in Wasserstein Space
Authors: Dror Freirich
Tomer Michaeli
Ron Meir
Venue: NeurIPS 2021 Poster (2021)
Keywords: perception-distortion tradeoff
optimal transport
statistical estimation theory
image restoration

Abstract: The lower the distortion of an estimator, the more the distribution of its outputs generally deviates from the distribution of the signals it attempts to estimate. This phenomenon, known as the perception-distortion tradeoff, has captured significant attention in image restoration, where it implies that fidelity to ground truth images comes on the expense of perceptual quality (deviation from statistics of natural images). However, despite the increasing popularity of performing comparisons on the perception-distortion plane, there remains an important open question: what is the minimal distortion that can be achieved under a given perception constraint? In this paper, we derive a closed form expression for this distortion-perception (DP) function for the mean squared-error (MSE) distortion and Wasserstein-2 perception index. We prove that the DP function is always quadratic, regardless of the underlying distribution. This stems from the fact that estimators on the DP curve form a geodesic in Wasserstein space. In the Gaussian setting, we further provide a closed form expression for such estimators. For general distributions, we show how these estimators can be constructed from the estimators at the two extremes of the tradeoff: The global MSE minimizer, and a  minimizer of the MSE under a perfect perceptual quality constraint. The latter can be obtained as a stochastic transformation of the former.


A Theory of the Distortion-Perception Tradeoff in
Wasserstein Space
Dror Freirich
Tomer Michaeli
Technion Israel Institute of Technology drorfre@gmail.com com
Technion - Israel Institute of Technology
tomer m@ee. technion, ac
Ron Meir
Technion Israel Institute of Technology rmeir@ee. technion.ac. ac.
Abstract
The lower the distortion of an estimator, the more the distribution of its outputs generally deviates from the distribution of the signals it attempts to estimate. This phenomenon, known as the perception-distee -distortio tradeoff, has captured significant attention in image restoration, where it implies that fidelity to ground truth images comes the expense of perceptual quality (deviation from statisties of natural images). However, despite the increasing popularity of performing comparisons on the perception-distortion plane, there remains an important open question: what the minimal distortion that can be achieved under given perception constraint? this paper, we derive a closed form expression for this distortion- perrrpptneeo (DP) function for the mean squared-error (MSE) distortion and the Wasserstein-2 perception index. We prove that the DP function is always quadratic, regardless of the underlying distribution. This stems from the fact that estimators the DP curve form a geodesic in Wasserstein space. In the Gaussian setting, we further provide closed form expression for such estimators. For general distributions, we show how these estimators can be constructed from the estimators at the two extremes of the tradeoff:: The global MSE minimizer, and minimizer of the MSE under a perfect perceptual quality constraint. The latter can be obtained as a stochastic transformation of the former.
Introduction
Inverse problems that involve signal reconstruction from partial noisy measurements, arise in numerous scientific domains. Examples range from medical imaging to tomography, microscopy, astronomy and audio enhancement. In many such problems is desired design estimator that (i) has a small reconstruction error (low distortion), and (ii) outputs reconstructions that cannot be told apart from valid signals (good perceptual quality). Interestingly, however, it has been shown that the lower the average distortion an estimator, the more the distribution of its outputs generally deviates from the distribution of the signals it attempts estimate (4. In other words, low distortion generally comes at the price of poor perceptual quality, and vice versa. This phenomenon, known as the perception-diste -diittrttonntttiiinnn tradeoff. has found particular interest in the image restoration domain (see Fig. D. where algorithms are now commonly being evaluated using both distortion measures and perception indices [5].
Unfortunately, despite the increasing popularity of performing comparisons on the perception- distortion plane, the minimal distortion that can be achieved under given perception constraint (red curve in Fig. remains an open question. Blau and Michaeli investigated several properties
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

D
Distortion Ejd(X, Xa)
PSRGAN
BDSR
Unattainable
Low resolution imput
EDSR
ESRGAN (high MSE, low FID)
Peception dp(px-Px)
(low MSE, high
Figure 1: Illustration of the pistortionn-erreptionn tradeoff in super-resolution low resolution
image (left) is fed to two state-of-the-ar super-resolution algorithms (middle). EDSR [14] achieves low MSE distortion, but produces blurry reconstructions with high FID values [TO. ESRGAN BI outputs photo-realistic recoveries with low FID, but its MSE is significantly higher. This is result of the cettoottio-perceepttn distortion and have their outputs distributed like the signals they are designed to estimate. In this paper, we derive a closed form expression for the iistortion-perceptin function (red curve) for the MSE distortion and Wasserstein-2 perception index.
tradeoff (right). Namely, estimators cannot simulatenously achieve a low
of this distortion-ppereppin function, such as monotonicity and convexity. But beyond this rather general characterization, little is known about its precise nature. In this paper, we derive closed form expression for the distortion- pererppoon (DP) function for the special case where distortion is measured by mean squared-error (MSE) and perception is measured by the Wasserstein-2 distance between the probability laws of the estimate and the estimand.
Our main contributions are: (i) We prove that the DP function is always quadratic in the perception constraint P. regardless of the underlying distribution (Theorem I). (ii) We show that it is possible to construct estimators on the DP curve from the estimators at the two extremes of the tradeoff (Theorem 3 The one that globally minimizes the MSE, and minimizer of the MSE under a perfect perceptual quality constraint. The latter can be obtained as a stochastic transformation of the former. (iii) the Gaussian setting, we further provide closed form expression for optimal estimators and for the corresponding DP curve (Theorems 4 and (5). We show this Gaussian DP curve is lower bound on the DP curve of any distribution having the same second order statisties. Finally, we illustrate our results, numerically and visually, in super-resolution setting in Section 5 of our theorems are provided in Appendix B
The proofs
Our theoretical results shed light on several topics that are subject to much practical activity. Particu- larly, many recent works adress the task of diverse perceptual image reconstruction, by employing randomization among possible restorations [15 11 Commonly, such works attempt to sample from the posterior distribution of natural images given the degraded input image. This is done, for
example, using priors over image patches (7], conditional generative models [18 21 or implicit
priors induced by deep denoiser networks [T]. Theoretically, posterior sampling leads to perfect perceptual quality (the restored outputs are distributed like the prior). However, fundamental question is whether this optimal in terms of distortion. we show Section [3.]] posterior sampling is often not an optimal strategy, in the sense that there exist perfect perceptual quality estimators that achieve lower distortion.
Another topic of practical interest is the ability to traverse the distortion-perceptt tradeoff at test time, without having to train different model for each working point. Recently, interpolation between -orrettn- models and perception-orienter ones, has been suggested for this end. Existing methods include interpolation
pixel space [I]] or some latent space (6),, interpolation between (22] and style transfer between a now-distortion reconstruction and high perceptual quality one 6 In light of this plethora of approaches, natural to ask which strategy is tradeoff, linear interpolation in pixel space leads to optimal estimators. We also discuss geometric connection between interpolation and the fact that estimators on the DP curve form geodesic in Wasserstein space.
network weights [31
optimal. In Section 3.2 we show that for the

---

2 Problem setting and preliminaries
2.1 The distortion-perreption tradeoff
Let X, Y be random vectors taking values in R and IR"v, respectively. We consider the problem of constructing estimator X of X based on Y. Namely, we are interested in determining conditional distribution Px such that X constitutes good estimate of X. For example, in the super-resolution setting shown in Fig. T Y is the low resolution image (left), X is the corresponding ground-truth high-resolution image (not shown), and X is super-resolution reconstruction generated from Y (e.g. the EDSR or ESRGAN estimators in the middle).
In many practical cases, the goodness of an estimator is associated with two factors: (i) the degree to which X is close to X average (low distortion), and (ii) the degree to which the distribution of X is close to that of X (good perceptual quality). An important question, then, is what is the minimal distortion that can be achieved under a given level of percepiual quality? and how can we construct estimators that achieve this lower bound? In mathematical language, we are interested in analyzing the distortion-perccptiion (DP) function (defined similarly to the perception-distortion function of 4
D(P) min PRIY
B[d(X, dp(px,P&) S sP]
Here, d: R. X R" R+ U {0} is some distortion criterion, dp(-,)) is some divergence between probability measures, and Py is the probability measure on RT induced by P&ly and py. The expectation is taken w.r.t. the measure P induced by P&j and pxy, where we assume that X is independent of X given
As discussed in 4]. the function D(P) is monotonically non-increasing and convex whenever .) is convex in its second argument (which is the case for most popular divergences). However, without further concrete assumptions on the distortion measure d(.,.) and the perception index dp(.)), .). little can be said about the precise nature of D(P).
Here, we focus our attention on the squared-erry error distortion d(x:, .)- = ||>~ -+1 and the Wasserstein-2 distance (px) (2(xx. Px)) with which (T) reads
D(P) = min Pxiy
[ElLXX x1 Px( S p
(2)
We assume that all distributions have finite first and second moments. In addition, from Theorem below it will follow that the minimum is indeed attained, so that 2 is well defined.
It is well known that the estimator minimizing the mean squared error (MSE) without any constraints, given by X* [[][Y]. This implies that D(P) monotonically decreases until P reaches p & W2(px, Px- beyond which point ((P) takes the constant value D* و - X This is illustrated in Fig. 2 It is also known that (0) < 2D* since the posterior sampling estimator and [[] - = 2D* 4 However, apart from these
-
PXi- Pxjy achieves W2(px. rather general properties, the precise shape of the DP curve has not been determined to date, and neither have the estimators that achieve the optimum in (2). This is our goal in this paper.
2.2 The Wasserstein and Gelbrich Distances
Before we present our main results, we briefly survey a few properties of the Wasserstein distance, mostly taken from (20). The Wasserstein-pl (p 2 1) distance between measures μ and y on a separable Banach space X' with norm || defined by
W 7) A inf (E(,.v)~ IIU VIP)
(T(
(3)
where I1(p, 7) is the set of all probabilities on X x X with marginals μ and 7. A joint probability achieving the optimum in (3) is often referred to as optimal plan. The Wasserstein space of probability measures is defined as
wo(x)) 2 {7: || < co)
3

---

D(P)
Source dist.
2D
Optimal dist.
Estimator dist.
X
Yp
D
px- μ P
P
P
px = Y
X
0
P
P
Figure 2: The ASSE-Wasserstein22 tradeoff and the geometry of optimal estimators. The left pane depicts the distortion distortion-perceptionn function for the MSE distortion and the Wasserstein-2 per- ception index. The minimal possible distortion, D is achieved by the estimator X F. The perception index attained by this estimator P* At the other extreme the tradeoff, we know that D(0) < 2D". The right pane shows the geometry of the distributions of optimal estimators in Wasserstein space. The minimal distortion D(P) can be achieved by an estimator with distri- bution 7p. which lies straight line (or geodesic) between px and px. Its distance from the former is (px,, 7P) = P and its distance from the latter is W2(xx= :P)) = P P. Therefore, D(P) D' W2 ( D (P* - P)?
and W, constitutes metric on (x))..
For any (m,, Σ1), (m2, E2) E Rd x s (where st is the set of symmetric positive semidefinite matrices in (d ) the Gelbrich distance is defined as
GZ ((m1, En), (m2, E2)) - [mm - m 112 + Tr Σ + En 2 $1
(4)
The root PSD matrix always taken to PSD. For any two probability measures μl, H2 on Rd with means and covariances (mi, E1), (m2, Σ2), from [8 Thm. 2.1] we have that
(21, P2) ) 2 (2 E1), (m2, E2)).
(5)
When µn (m,,, ; En and #2 (m,, Σ2) are Gaussian distributions on Rd, we have that W2(n, 122) = G((m,, E1), (m², E2)). This equality obvious for non-singular measures but true for any two Gaussian distributions 20 18). If E, and En are non-singular, then the distribution attaining the optimum in corresponds to
U N~N(m, E1), V = m2 + TU - -m1),
(6)
where
Ti-22 = E (I) E2 Σ
is the optimal transformation pushing forward from (0, Σ1) to (O, Mn) [12). This transformation satisfies 2g 71-22,17--2 For discussion on singular distributions, please see App. A
3
Main results
3.1 The MSE-Wasserstein-2 tradeoff
The DP function (2) depends, of course, on the underlying joint probability Pxy of the signal and measurements Y. Our first key result is that this dependence can be expressed solely in terms of D* and P In other words, knowing the distortion and perception index attained by the minimum MSE estimator X", suffices for determining D(P) for any P.

---

Theorem (The DP function), The DP function
2 is given by
D(P) D* + (PP'
(8)
where (ar)+ max(0, Furthermore, an estimator achieving perception index P and distortion DPP) can always be constructed by applying (possibly stochastic) transformation to X".
Theorem Tis of practical importance because in many cases constructing estimator that achieves a low MSE ei.e. an approximation of X)) is a rather simple task. This is the case, for example, in image restoration with deep neural networks. There, it is common practice to train network by minimizing its average squared error on a training set. Measuring the MSE of such a network on a large test set provides an approximation for D We also obtain an approximation of at least lower bound on P by estimating the second order statistics of and X". Specifically, recall that P" is lower bounded by the Gelbrich distance between (mx, Ex) and (mx*, Ex+), which is given by (G*)² &
Tr(Ex + Σx- 2 Ex -= 2/231/22 (see (5). Given approximations for D and
G we can approximate lower bound the DP function for any P.
D(P) 2 D + [[G P)+1
The bound is attained when X and Y are jointly Gaussian.
Uniqueness A remark is in place regarding the uniqueness of estimator achieving (8). As we discuss below, what defines an optimal estimator X is its joint distribution with X This joint distribution may not be unique, in which case the optimal estimator is not unique. Moreover, even pxxx is unique, the uniqueness of the estimator is not guaranteed because there may be different conditional distributions y that lead to the same optimal xxx* In other words, given the optimal xxx- one can choose any joint probability Pxyx- that has marginals xxx- and x- One option is take the estimator X be a (possibly stochastic) transformation of X*, namely P = X-x--- But there may be other options. cases where either Y or X are a deterministic transformation of X (e.g. when X* has a density, or is an invertible function of Y), there is unique joint distribution P with the given marginals [ Lemma 5.3.2]. In this case, if xxx- is unique then so is the estimator P
Randomness Under the settings of image restoration, many methods encourage diversity in their output by adding randomness [15] 3 22 In our setting, we may ask under what conditions there exists optimal estimator X which is deterministic function of Y. For example, when py óg but X has some non-atomic distribution, it is clear that no deterministic function of Y can attain perfect perceptual quality. It turns out that sufficient condition for the optimal X to be deterministic function of Y is that X" have a density. We discuss this in App. B and explicitly illustrate it in the Gaussian case (see Sec. 3.3), where X has a non-singular covariance matrix then X is a deterministic function of Y.
When posterior sampling optimal? Many recent image restoration methods attempt to produce diverse high perceptual quality reconstructions by sampling from the posterior distribution px [8 IT. As discussed in [4) the posterior sampling estimator attains perception index of 0) and distortion 2D". But an interesting question is: when is this strategy optimal? In other words, in what cases do we have that the DP function at P 0 equals precisely 2D* and is not strictly smaller? Note from the definition of the Wasserstein distance (3), that
(namely W2(xx,
(P*)² W Px.) < E[|[X
X X = D Using this in (8) shows that the DP function at
P = 0 is upper bounded by
D(0) = D' + (P*)² 2D'
(10)
and the upper bound is attained when (P=)² D To see when this happens, observe that
[r Ex + - -
= - <
ITEx (P)
(11)
We can see that when Tr(Ex - -T<(E>/'> Σ 1/111)) the leftmost and rightmost sides become equal, and thus (p*)²= D To understand the meaning of this condition, let us focus on the case where Ex and Σx- are jointly diagonalizable. This is reasonable assumption for natural images,
5

---

where inftrtiivariance induces diagonalization by the Fourier basis [30] In this case, the condition can be written in terms of the eigenvalues of the matrices, namely Σ, (Exx = E, ( }^, (Ex) ). This condition satisfied when each xx(Ex-) equals either >, or Namely, the ith eigenvalue of the error covariance of X", which is given by Σx Ex-, is either xx or O. We conclude that posterior sampling is optimal when there exists a subspace spanned by some of the eigenvectors of Σx, such that the projection of X onto S can be recovered from Y with zero error, but the projection of onto S+ cannot be recovered at all (the optimal estimator is trivial). This is likely not the case in most practical scenarios. Therefore, it seems that posterior sampling is often not optimal. That is, posterior sampling can be improved upon in terms of MSE without any sacrifice in perceptual quality.
3.2 Optimal estimators
While Theorem D reveals the shape of the DP function, it does not provide recipe for constructing optimal estimators on the DP tradeoff. We now discuss the nature of such estimators.
Our first observation that since X is independent of X given Y. its MSE can be decomposed as E|||X - [|] = E[| X 1 + E|| ||1 (see App. Therefore, the DP function (2)
can
be equivalently written as
D(P) = D' + min - - x-11 Px(P Pxly
(12)
Note that the objective in (2) depends on the MSE between X' and X", so that we can perform the minimization on Pxxx- rather than on P (once we determine the optimal Pxxx- we can construct a consistent Pxjy as discussed above).
Now, let us start by examining the leftmost side of the curve D(P), which corresponds to a perfect perceptual quality estimator (i.e. P = 0). In this case, the constraint becomes px px. Therefore,
D(0) D" + min {
EIIL - xx PXx. E (I(xXPxx))
(13)
PXx-
where II(xx: px.) ) is the set of all probabilities on T x R with marginals px, px, One may readily recognize this as the optimization problem underlying the Wasserstein-2 distance between Px and px. This leads us to the following conclusion.
Theorem 2 (Optimal estimator for P = 0). Let Xo be an estimator achieving perception index 0 and MSE D(0). Then its joint distribution with X attains the optimum in the definition of W2(xx, (x-), Namely, P x- is an optimal plan between px and px=.
Having understood the estimator Xo at the leftmost end of the tradeoff, we now turn to study optimal estimators for arbitrary P. Interestingly,
show that Problem 12 is equivalent to (see App. B)
D(P) = D* + min { (W(Pp..x-)) W2(p, S P).
(14)
ps
Namely, an optimal y is closest to px- among all distributions within a ball of radius P around px. as illustrated in Fig. 2 Morcover, P&x. is optimal plan between Px and px- As it turns out, this somewhat abstract viewpoint leads to a rather practical construction for X from the estimators Xo and X at the two extremes of the tradeoff. Specifically, we have the following result.
Theorem 3 (Optimal estimators for arbitrary P). Let Xo be an estimator achieving perception index O and MSE (0). Then for any P E Po) the estimator
Sp=
P X0
P
Xp -1111p p
p X
(15)
is optimal for perception index P. Namely, it achieves perception index P and distortion D(P).
Theorem 3 has important implications for perceptual signal restoration. For example, in the task of image super-resolution, there exist many deep network based methods that achieve low MSE 14 29 25). These provide an approximation for X*. Moreover, there an abundance of methods that achieve good perceptual quality at the price of reasonable degradation in MSE (often by

---

incorporating a GAN-based loss) [3] 31, 24). These constitute approximations for Xy. However, achieving results that strike other prescribed balances between MSE and perceptual quality commonly require training a different model for each setting. Shoshan et al. [26] and Navarrete Michelini et al. [17] tried to address this difficulty by introducing new training techniques that allow traversing the distortion-perces -prceeption tradeoff at test time. However, Theorem shows that such specialized training methods are not required in our setting. Having a model that leads to low MSE and one that leads to good perceptual quality, it is possible to construct any other estimator on the DP tradeoff, by simply averaging the outputs of these two models with appropriate weights. We illustrate this in Sec. S
3.3 The Gaussian setting
When X and Y are jointly Gaussian, it is well known that the minimum MSE estimator X is linear function of the measurements! Y. However, it not a-priori clear whether all estimators along the DP tradeoff are linear in this case, and what kind of randomness they possess. As we now show, equipped with Theorem 3] we can obtain closed form expressions for optimal estimators for any P. For simplicity, we assume here that X and Y have zero means and that Σx, Σy > 0.
It is instructive start by considering the simple case, where Ex- is non-singular (in Theorem
below we address the more general case of a possibly singular Ex-). ). It is well known that
X 2x Y,
Σx Ex Σyx.
(16)
Now, since we assumed that Ex, Σx- > 0. we have from Theorem 2 and
6 that
Xo Σزز (MX. xx xxxx*
(17)
Finally, we know that P G", which is given by the left-hand side of II Substituting these expressions into (5), we obtain that an optimal estimator for perception P E GO) is given by
Xp= = 2s
P
1 Ex +
- ExyEP'y.
(18)
As can be seen, this optimal estimaato is deterministic linear transformation of Y for any P. The setting just described does not cover the case where Y is of lower dimensionality than X because in that case Xx- is necessarily singular (it is nr x R2 matrix of rank at most Ny: see (T6). In this case, any deterministic linear function of Y would result in an estimator X with rank-n, covariance. Obviously, the distribution of such an estimator cannot be arbitrarily close to that of X, whose covariance has rank 1zz- What is the optimal estimator in this more general setting, then?
Theorem 4 (Optimal estimators in the Gaussian case). Assume X and Y are zero-mean jointly Gaus- sian random vectors with Xx, Ey O. Denote 1 & Tpx-pxx- > 1/2 x112 3>)/2(2x>// (2>222xx Exx /
Then for any P E [0, G"), an estimator with perception index P and MSE D(P) can be constructed
as
Xp= (11-p))
P (.PP) ΣX
EX xX)
Σx EX
+ ) EXTEp'Y (1-
P
W. (19)
G.
G
where W is zero-mean Gaussian noise with covariance Σw = -x222 - which is independent of X, and E'x- is the gseudo-inverss of Σx-
22/22/00 x-
1/2
1 X
Note that in this case, we indeed have random noise component that shapes the covariance of Xp to become closer to Σx as P gets closer to 0. It can be shown (see App. BB) that when Σx- is invertible,
Σw = 0 and (9) reduces to (18). Also note that, as in (88). the dependence of Xx on Y in (9) is only through X = ExyE Y.
As mentioned in Sec. 3.1 the optimal estimator is generally not unique. Interestingly, in the Gaussian setting we can explicitly characterize a set of optimal estimators.
Theorem 5 (A set of optimal estimators in the Gaussian case). Consider the setting of Theorem
Let E x.Y E Xy satisfy
Σ Ey Erx 2
(20)
7

---

and Wo be a zero-mean Gaussian noise with covariance
Ew, Ex Exox by
E NoY 00
(21)
that is independent of X, Y. Then, for any P E [0, G*), an optimal estimator with perception index P can be obtained by
Xp (1- Exor
P
Ey'r+ + (1-
P. Wo
+
IxY G
Wo. but generally not unique.
(22)
The estimator given in is one solution to 20
21 2
3.4 A Comment on the ASE-Wassersteing tradeoff
While our results concern the MSE - W22 tradeoff, they can be used to draw conclusions regarding the DP tradeoff with other divergences. In particular, (8) constitutes lower bound on the MSE- Wasserstein-p tradeoff for any 2 2. Furthermore, we can show that the MSE - W1 DP function is lower bounded by D + [P[ P)+ where P - Wi Wi(px, Px.).
Note that at the point P = 0, the DP function coincides with (8) for any plausible divergence. For detailed discussion, we kindly refer the reader to the Appendix.
geometric perspective on the distortion tradeoff
In this section provide geometric point of view main results. Specifically, show that the results of Theorems andd]] 3 are a consequence a more general geometric property the space W2(R"*). ). In the Gaussian case, this simplified a geometry of covariance matrices.
Recall from (T) that the optimal p is the one closest to px- (in terms of Wasserstein distance) among all measures at distance P from px This implies that to determine Px- we should traverse the geodesic between px- and px until reaching a distance P from px- Furthermore, Xxx- should be the optimal plan between P and px- Interestingly, geodesics in Wasserstein spaces take particularly simple form, and their explicit construction also turns out to satisfy the latter requirement. Specifically, let Y, l be measures in W2 (IRd), let [I(', (4) be optimal plan attaining W2(7, pt), and let denote the projection n; Rd Rd + Rd such that m; 22)) I;, 1,2. Then, the curve
7 A [1 l))1 + 1T2) #V,
LE [0.
(23)
is constant-speed geodesic from y to μ in W2 (Rd) 2 where # is the push-forward operation Particularly,
W2(ne 7/) 21=s|W2(7..0
(24)
and it follows that We(11, 7) = (W2(7, μ4) and W2(1, n) = (1 () u). Furthermore,
Te,4 E [0, 1] is constant-speed geodesic with no -,,7 ns then the optimal plans between 7 and between Te, are given by
[TT, (1-t)m t)= #V,
[(1 t) + tm2, T2. #V,
(25)
respectively, where v E IT(T,11) is some optimal plan. Applying (23 to (No, X*) with
t = PPPP", we obtain (5). where we show that the obtained estimator achieves E|||Xp - X x+
(1 (xx Px-). This explains the result of Theorem
worth mentioning that this geometric interpretation simplified under some common settings. For example, when 7 is absolutely continuous (w.r.t. the Lebesgue measure), we have measurable map Ta which is the solution to the optimal transport problem with the quadratic cost 20) Thm
1.6.2, 16]. The geodesic
23 then takes the form
Yt [ld + t(T - Id)] E [0.
(26)
Therefore, in our setting, if = px- has a density, then we can obtain Xp by the deterministic transformation [X* ~ + (1 ) (Tpx- -px (X*) X+]] (see Remark about randomness in Sec. 3.1
For measures /4 X,Y, we say that measurable transform T'; X + y pushes y forward to μ (denoted μuB) for any measurable B C
T T#y #) iff x(1 (B)) (BB)
8

---

Further simplification arises when y. are centered non-singular Gaussian measures, in which case T- is the linear and symmetric transformation 77). Then, V/t is Gaussian measure with covariance Σ Tr where T, _ [T + T(T - f]]- Therefore, in the Gaussian case, the shortest path (33 between distributions is reduced to trajectory in the geometry of covariance matrices induced by the Gelbrich distance [27]. additionally Σ, and En commute, then the Gelbrich distance is further reduced to the ddditannce between matrices, as we discuss in App. D
5 Numerical illustration
We now experimentally illustrate the results of Theorems and We compute distortion and perception indices for 13 super resolution algorithms in a 4x magnification task the BSD100 datase P [6]. We then demonstrate the efficiency of approximating the lower bound (9) on the distortion (P), and of the estimators suggested in (15), in this practical setting. The evaluated algorithms include EDSR [14), ESRGAN [3I, SinGAN [24], ZSSR (25). DIP [29], SRResNet variants which optimize MSE and VGG2.2. SRGAN variants which optimize MSE, VGG2,2 and
VGGs,4 in addition to an adversarial loss [33, ENet [23] ("PAT" and "E" variants), and the stochastic explorable SR method of 33 (ExpSR). Low resolution images obtained by 4x downsampling of BSD100 images using a bicubic kernel.
In Fig. 3 we plot each method on the (and reconstructed) images to be stationary random sources, and use x 9 patches (totally 1.6 x 106 patches) to empirically estimate the mean and covariance matrix for the ground-truth images, and for the reconstructions produced by each method. We then use the estimated Gelbrich distances (4) between the patch distribution of each method and that of ground-truth images, perceptual quality index. Recall this is lower bound on the Wasserstein distance. In the right pane of Fig. 3 we measure perception using the popular FID index [TO), which relies on the Fréchet distance between deep feature distributions of ground-trute and reconstructed images (assuming they are normally distributed). FID is known correlate well with visual quality, and while it is not dircctly related to our theory, we can see a qualitatively similar behavior to that depicted in the left pane.
plane. In the left pane, we consider natural
We consider the EDSR method [14] to constitute a good approximation for the minimum MSE estimator X since it achieves the lowest MSE among the evaluated methods. We therefore estimate the lower bound (9) as
D(P) DEDSR + [(PEDSR P)+1²,
where DEDSR is the MSE of EDSR, and PEDSR is the estimated Gelbrich distance between EDSR reconstructions and ground-truth images. Note the unoccupied region under the estimated curve in Fig. 3 which is indeed unattainable according to the theory.
The figure also shows 9 estimators X., which we construct by interpolation between EDSR and ESRGAN, X, t.XEDSR + (1 XESRGAN with t e [0, 10. We observe that estimators constructed using these two extreme points are closer to the optimal DP tradeoff than the other evaluated methods. This is true both for the Gelbrich perception index and for FID. In Fig. 4 we present visual comparison between SRGAN-VGG,,2 13 and our interpolated estimator Xo.12- Both achieve roughly the same RMSE distortion (18.08 for SRGAN, 18.14 for Xo.12), 12, but our estimator achieves lower perception index. Namely, by using interpolation, perceptual quality, without degradation in distortion. The improvement in visual quality is also apparent in the figure. Additional visual comparisons including more points along the DP curve and ground ground-truth images can be found in the Appendix.
manage to achieve improvement in
6 Conclusion
In this paper we provide full characterization of the cestortion-perceptioon tradeoff for the MSE distortion and the Wasserstein-2 perception index. We show that optimal estimators are obtained by interpolation between the minimum MSE estimator and an optimal perfect perceptual quality estimator. In the Gaussian case, explicitly formulate these estimators. To the best of our knowledge, this is the first work to derive such closed-form expressions. Our work paves the way towards fully
2 All codes are freely available and provided by the authors.

---

ExpSR ENet PAT
ExpSR EN
20
SRGAN VGG
D(P)
SRGAN
SUGAAN
VGO
SiGGAN
ESRGAN
ESRGAN
XO
SRGAN
SRGAN
18
RMSR
SRGAN MIS
RMSE
SRGAN
17
SeReaNes VGG
XAL
MSE
MPP
Xn
BRRRNNe VGG
DIP
16-
16-
2SSR
Sas
ZSSR
15
RNot B
SRResNNe
15-
SHtReNNt
ENet
MSI
14-
EDSR
14-
EDSR
0
10
11
12
FIT)
Figure 3: Evaluation of SR algorithms. We plot 13 algorithms (blue) on the Distortion Distortion-Prreettoon plane. the left pane, perception measured using the Gelbrich distance between empirical means and covariances of patches from the ground-truti images and the reconstructed images. In the right pane, we measure perception using FID. In orange is the estimated lower bound ( where we consider EDSR to be the global minimizer X Note the unoccupied region under the estimated curve, which is unattainable. We also plot 9 estimators X, (Green) created by interpolation between EDSR and ESRGAN reconstructions, using different relative weights t. Note that estimators constructed using these two extreme estimators are closer to the optimal DP curve than the compared methods.
SRGAN
VGG22
X0.12
Figure 4: A visual comparison between estimators with approximately the same MSE. Upper:
SRGAN-VGG2,2 Lower: Xo.12, an interpolation between EDSR and ESRGAN using / = 0.12. Observe the improvement in perceptual quality, without any significant degradation in distortion.
understanding the DP tradeoff under more general distortions and perceptual criteria, and bridging between fidelity and visual quality at test-time, without training different models.
Broader impact Synthesis of photo-realistic visual contents may raise concerns of inappropriate and malicious use. This is true for image generation in general (e.g. with GANs), but to some extent also for image restoration tasks like super-resolution. However, even without malicious intent, the outputs of a high perceptual quality algorithm can often not be very close to the ground truth images. In this paper quantify this effect, by studying the best similarity (lowest distortion) can hope to achieve with an algorithm having a prescribed level of perceptual quality.
Acknoledgments This work was partially supported by grants 451/17 and 852/17 from the Israel Science Foundation, by the Ollendorff Center of the Viterbi Faculty of Electrical and Computer Engineering at the Technion, and by the Skillman chair in biomedical sciences.
10

---

References
[1] Mohamed Abderrahmen Abid, Ihsen Hedhli, and Christian Gagné. A generative model for hallucinating diverse versions of super resolution images. arXiv preprint arXiv:2102.06624 2021.
[2] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science Business Media, 2008.
[3] Yuval Bahat and Tomer Michaeli. Explorable super resolution. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2716-2725, 2020.
[4] Yochai Blau and Tomer Michaeli. The perception-distortinn tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018.
[5] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zeinik-Manor, The 2018 PIRM challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0 0-0, 2018.
[6] Xin Deng. Enhancing image quality via style transfer for single image super-resolution. IEEE Signal Processing Letters, 25(4):571-575, 2018.
[7] Roy Friedman and Yair Weiss. Posterior sampling for image restoration using explicit patch priors arXiv preprint arXiv: 20X.:::0040099955 2021.
Matthias Gelbrich. On a formula for the 12 wasserstein metric between measures euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185-203, 1990.
[9] Robert M Gray. Toeplitz and circulant matrices: A review. 2006.
[10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cur- Associates, Inc., 2017. http:/tttttp/cceeeeeeeee .pff pdf
[11] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. arXiv preprint arXiv:2101.09552 2021.
[12] Martin Knott and Cyril S Smith. On the optimal mapping of distributions. Journal of Optimiza- tion Theory and Applications, 43(1):39-49, 1984.
[13] Christian Ledig, Lueas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 468 4681-4690, 2017.
[14] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[15] Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte. Srilow: Learning the super-resolution space with normalizing flow. In European Conference on Computer Vision, pages 715-732. Springer, 2020.
[16] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423. IEEE, 2001.
[17] Pablo Navarrete Michelini, Dan Zhu, and Hanwen Liu. Multi-scale recursive and perception- distortion controllable image super-resolution. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.

---

[18] Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High percep- tual quality image denoising with posterior sampling egan, arXiv preprint arXiv:2103.04192. 2021.
[19] Felix Otto and Cédric Villani. Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173(2)361-400) 2000.
[20] Victor M Panaretos and Yoav Zemel. An invitation to statistics in Wasserstein space. Springer Nature, 2020.
[21] Mangal Prakash, Alexander Krull, and Florian Jug. Divnoising: diversity denoising with fully convolutional variational autoencoders. arXiv preprint arXiv:2006. 06072, 2020.
[22] Mangal Prakash, Mauricio Delbracio, Peyman Milanfar, and Florian Jug. Removing pixel noises and spatial artifacts with generative diversity denoising methods. arXiv preprint arXiv:2/04.01377 2021.
[23] Mchdi SM Sajjadi, Berhaard Scholkopf, and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference Computer Vision, pages 4491-4500, 2017.
[24] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning generative model from single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4570-4580, 2019.
[25] Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3118-3126 -126, 2018.
[26] Alon Shoshan, Roey Mechrez, and Lihi Zelnik-Manor. Dynamic-net: Tuning the objective with- out re-training for synthesis tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3215-3223, 2019.
[27] Asuka Takatsu. On wasserstein geometry of gaussian measures. In Probabilistic approuch to geometry, pages 463 472. Mathematical Society of Japan, 2010.
[28] Michel Talagrand. Transportation cost for gaussian and other product measures. Geometric & Functional Analysis GAFA, 6(3):587-600, 1996.
[29] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446-9454 9454,945,, 2018.
[30] Michael Unser. On the approximation of the discrete karhunen-loeve transform for stationary processes. Signal Processing, 7(3):231-22) 1984.
[31] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. ESRGAN: Enhanced super-resolution generative adversarial networks. In Pro- ceedings of the European Conference Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[32] Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. Deep network interpo- lation for continuous imagery effect transition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1692-1701, 2019.
[33] Zhen Zhang, Mianzhi Wang, and Arye Nehorai. Optimal transport in reproducing kernel hilbert spaces: Theory and applications. IEEE transactions on pattern analysis and machine intelligence,
2019.
12


Reviews:
[{"review_id": "vyTNRAVaYVK", "reviewer": "Reviewer_McgJ", "summary": "The authors of this paper suggest that they show four novel contributions to the image compression literature that hold true for MSE distortion and Wasserstein-2 perception index:\n1. They prove that the Distortion-perception function is always quadratic in the perception constraint P regardless of the underlying distribution\n2.  The show that it is possible to construct estimators of the Distortion-perception curve from estimators at the two extremes of the tradeoff (the one that globally minimizes MSE and the one that minimizes MSE under a perfect perceptual quality constraint).\n3.In the gaussian setting, they provide a closed form expression for optimal estimators and the corresponding Distortion-perception curve, they also show that this is a lower bound on the curve of any distribution having the same second-order stats.\n4.They illustrate their results conclusively with super-resolution as their test case.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that this is a very good paper, that makes novel contributions to an exciting area of the compression field.  To me the biggest contribution of this paper is Theorem 3 (Equation 15) which says that a linear interpolation between the two edge estimators can construct an estimator of the distortion-perception curve that outperforms other estimators at locations along the curve that are not at either extreme.\n\nThis is a strong, novel contribution worthy of NeurIPS.\n\nThe main weakness of the paper is in the visual comparisons presented, the authors don't show the ground truth image patch with the decoded patches, so even though the results seem to suggest their outcome is better than the comparisons, it is not entirely clear.  In addition, visual comparison at more points along the curve would also be useful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "gpehNbEumDr", "reviewer": "Reviewer_WGqG", "summary": "The paper characterizes the perception-distortion in the context of (image) enhancement tradeoff, complementing prior results. For example, in image super resolution, this tradeoff describes the relationship between the point-wise reconstruction quality (distortion) and how well the distribution of the reconstructions match the distribution of the high-resolution data one aims to reconstruct. The paper derives this general tradeoff explicitly and characterizes certain aspects specifically for Gaussian distributions. The theoretical results are illustrated with experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To my knowledge, the paper presents novel results characterizing the perception-distortion tradeoff for image (or more generally signal) enhancement as introduced by [4]. The paper focuses on MSE as a distortion and the Wasserstein-2 distance to measure perceptual quality. I particularly like that the authors derive Theorem 3 which provides practical guidance as to how to navigate the distortion-perception tradeoff. The paper is well-written.\n\nI have the following questions and suggestions:\n- MSE and the Wasserstein-2 distance are very specific choices of rate and distortion, and most super-resolution models are trained with different distortion metrics and/or GAN losses approximating a different divergence. What can be said about these cases? Any intuition whether similar properties as those proven here could be expected? This could also be empirically evaluated in the context of the experiments described in Sec 5. For example the authors could in addition use the Frechet Inception Distance as an additional perceptual index.\n- The experiments in Sec 5 are done for deterministic decoders. How do the practical implications of Thm 3 (15) change for stochastic decoders as in [15, 3, 21]?\n- It is sometimes unclear whether the m_i and \\Sigma_i just refer to the first and second moments of a distribution (that could e.g. model natural images) or whether specifically Gaussian distributions are meant. This is in particular important because Guassians are a crude model for natural images (in the context of super resolution).\n- In Sec 3.1 the authors argue that D^\\star can be estimated using the loss of an MSE trained network after training. However, this value is usually an upper bound on D^\\star and plugging it into (9) will invalidate the inequality. This should be pointed out clearly in the paper. If possible, it would be great to make a corresponding practical statement where one fixes the predictor and then derives a lower bound on D(P) for that predictor.\n- I feel that the first statement in the abstract sounds too general. It should be mentioned that it is in the context of image restoration (or at least a problem that admits a distortion-perception tradeoff)\n\nOverall I think the paper makes an interesting contribution.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For limitations please see the main review. In terms of societal impact it might make sense to briefly mention the potential issues inherent with perceptual super-resolution as possibly unwanted image content might be synthesized. In that context, the present paper is an advancement as it derives a method to adjust the D(P) tradeoff.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1AVky_S_YT_", "reviewer": "Reviewer_7S5X", "summary": "Distortion-perception is the phenomenon that the better a criterion (on natural images) is optimized (for instance SNR in image denoising) the stronger the deviation to the distribution of the space of natural images. This paper characterises this phenomenon under Mean Squared Error (MSE) as the criterion under a Wasserstein bound on the conditional distribution. The result is an interpolation between the distribution of the conditional distribution and the distribution itself. An explicit formulation of the optimizer of their Distortion-perception function is provided. Simulations are made, showing that interpolating different models that approximates the quantities in their theoretical formulation can achieve better visual aspect.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Studying criterion of optimization may appear of increasing interest for the community and it seems to be already the case. \n\nMain concerns: *More background and perspectives are needed in the presentation of the work*\n\nMy main concern with the paper is that the technical results are quite straightforward and take too much space in the paper. \nDiscussing the distortion-perception function in the introduction would have been more informative. In particular, what are the challenges to estimates the quantities involved in this distortion-perception function. I reckon that the Wasserstein framework is nice to obtain this simple interpolation formula which is almost immediate consequence of the definition. So, I would have expected more discussion on why Wasserstein, what about other divergences, why not at least discussing other divergences in order to highlight the different bottlenecks. How to estimate the quantities involved and gentle introduction to these questions would have been a plus. \n\nUnderstanding the experimental section was difficult, although I am not an expert in this area. Explaining what is done with a more pedagogical presentation would be beneficial.\n\nOther comments: \n- About the technical part, I think the authors should not spend so much space of the results and I would suggest to put the proof of the main result in the main text in one theorem and one corollary at most.\n- The Gelbrich distance is most often called the Bures-Wasserstein metric. Bures simply is a metric introduced in quantum physics, long before Gelbrich found the closed form although without translation term.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No particular comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "-up8FcR2kU7", "reviewer": "Reviewer_LmPX", "summary": "The paper studies the trade-off between minimizing the mean-square error (distortion) and the\nWasserstein-2 distance (perception) when constructing an estimator \\hat{X} of a random variable X.\nIn particular, the work analyzes a distortion-perception function, which given the radius P of a Wasserstein-2\nball around X returns the minimal mean-square error of \\hat{X} constrained to be in that ball. \n\nThe analysis reveals that for estimators with small Wasserstein-2 distance to the true distribution, one can often\nobtain an estimator with much better mean-square error by a simple linear interpolation in the output space. \n\nThe theoretical results are applied to deep image super-resolution, where it is shown that the perceptual quality of\nan estimator with small MSE can be improved by interpolating it with one having small Wasserstein-2 distance. Remarkably,\nthe interpolated estimator often still has low MSE, while significantly lower Wasserstein-2 distance, leading to a greatly\nimproved perceptual quality. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, and I have no major objections regarding the overall clarity of the work. \n\nOne concern is that the approach is in some sense an incremental step compared to [4], which already introduces the perception-distortion tradeoff in image restoration. The main contribution over that work is the closed-form expression for the distortion-perception function for the specific combination of MSE and Wasserstein-2 distance. \n\nTo me, it appears that the significance / impact of the paper is overall rather limited, as the final estimator suggested by the theory is just a convex combination of the two estimator in output space. This seems to stem from the perhaps overly simplistic choice of MSE / Wasserstein-2, which ignores the fact that the manifold of natural images is a highly nonlinear space. Therefore, I feel that this strategy comes with obvious limitations and suggests that the considered measure of perception might not be too meaningful. \n\nI apologize if the above criticisms are based on some basic misconception / incomplete understanding of the contributions and I am willing to raise my score in such a case.\n\nFinally, I found the experimental evaluation a bit short. Qualitative results are only shown on four different images. For Figure 3, I do not agree that there is an improvement in perceptual quality as claimed, especially for the first image. Such claims could be more rigorously evaluated using a small-scale study involving humans.\n\n### Minor comments.\n- Due to the concise choice of notation I had difficulties following the mathematical\ndetails starting from Section 2.1. There, X, \\hat{X} and Y are introduced as random vectors, which mathematically are\nmeasurable functions mapping from a space which being left unspecified into R^{n_x} and R^{n_y}. These introduce the pushforward measures\np_X, p_{\\hat{X}} and a certain conditional distribution p_{\\hat X | Y}. Since these objects are introduced in a quite implicit/indirect way, \nmentioning the exact signature and definition of the density functions (or Borel probability measures?) would be helpful. \n\n- It might be beneficial for the reader also to explain what X, \\hat X and Y specialize to in the setting of a concrete example,\nperhaps for the case of image super-resolution considered in the applications sections. Maybe such an example can be included after line 76 in the paper, or in the appendix. \n\n- I felt a bit uneasy about the precise definition of Eq. (1). In particular, it is unclear with respect to which probability distribution\nthe expectation is taken (Is it wrt the unspecified distribution that X and \\hat X pushforward or is it the conditional distribution?). What space does p_{\\hat X | Y} live in? \nPerhaps these things could be expanded on in a bit more detail in Appendix A.1? \n\n- I can understand why an economical/concise notation is chosen (as is often in machine learning papers). But for me, it made checking the details in a rigorous fashion a bit exhausting. Overall I don't have doubts about the correctness of the results though. \n\n- The distortion-perception trade-off is studied on the example of image restoration. This is a fascinating and important field,  but perhaps a slightly niche topic for the NeurIPS audience, limiting the overall significance and impact of the work. Could the proposed framework perhaps \nconsidered in a broader context? For example, empirical risk minimization problems in machine learning optimize for low distortion; could generalization capabilities of ML methods perhaps be improved by a similar interpolation technique? \n\n- l. 154. I did not understand what a deterministic function is.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned in the main review, a discussion on the limitations of performing a simple linear interpolation in output space is missing. \nAre there for example \"failure cases\", where perception and distortion measured by the Wasserstein-distance and MSE are good\nbut the visual quality is worse than of the two individual estimators?\n\nI don't believe that the work needs to address negative impacts on society, as it is largely theoretical. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found this paper to provide an interesting and welcome application for optimal transport, even if the theoretical contribution is limited from a technical perspective.  After some deliberation, we converged on accepting this paper.

In the revision, please provide the following:

* More experimental evidence corroborating that a convex combination of two estimators results in visually pleasing results.
* Better introduction to the perception-distortion tradeoff for non-experts
* Optionally:  Detailed discussion (and additional theoretical results if possible) for distortions/divergences beyond the MSE/Wasserstein-2 case.
Decision: Accept (Poster)