Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Distortion-perception is the phenomenon that the better a criterion (on natural images) is optimized (for instance SNR in image denoising) the stronger the deviation to the distribution of the space of natural images. This paper characterises this phenomenon under Mean Squared Error (MSE) as the criterion under a Wasserstein bound on the conditional distribution. The result is an interpolation between the distribution of the conditional distribution and the distribution itself. An explicit formulation of the optimizer of their Distortion-perception function is provided. Simulations are made, showing that interpolating different models that approximates the quantities in their theoretical formulation can achieve better visual aspect.
main_review: Studying criterion of optimization may appear of increasing interest for the community and it seems to be already the case. 

Main concerns: *More background and perspectives are needed in the presentation of the work*

My main concern with the paper is that the technical results are quite straightforward and take too much space in the paper. 
Discussing the distortion-perception function in the introduction would have been more informative. In particular, what are the challenges to estimates the quantities involved in this distortion-perception function. I reckon that the Wasserstein framework is nice to obtain this simple interpolation formula which is almost immediate consequence of the definition. So, I would have expected more discussion on why Wasserstein, what about other divergences, why not at least discussing other divergences in order to highlight the different bottlenecks. How to estimate the quantities involved and gentle introduction to these questions would have been a plus. 

Understanding the experimental section was difficult, although I am not an expert in this area. Explaining what is done with a more pedagogical presentation would be beneficial.

Other comments: 
- About the technical part, I think the authors should not spend so much space of the results and I would suggest to put the proof of the main result in the main text in one theorem and one corollary at most.
- The Gelbrich distance is most often called the Bures-Wasserstein metric. Bures simply is a metric introduced in quantum physics, long before Gelbrich found the closed form although without translation term.

limitations_and_societal_impact: No particular comments.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors of this paper suggest that they show four novel contributions to the image compression literature that hold true for MSE distortion and Wasserstein-2 perception index:
1. They prove that the Distortion-perception function is always quadratic in the perception constraint P regardless of the underlying distribution
2.  The show that it is possible to construct estimators of the Distortion-perception curve from estimators at the two extremes of the tradeoff (the one that globally minimizes MSE and the one that minimizes MSE under a perfect perceptual quality constraint).
3.In the gaussian setting, they provide a closed form expression for optimal estimators and the corresponding Distortion-perception curve, they also show that this is a lower bound on the curve of any distribution having the same second-order stats.
4.They illustrate their results conclusively with super-resolution as their test case.
main_review: I believe that this is a very good paper, that makes novel contributions to an exciting area of the compression field.  To me the biggest contribution of this paper is Theorem 3 (Equation 15) which says that a linear interpolation between the two edge estimators can construct an estimator of the distortion-perception curve that outperforms other estimators at locations along the curve that are not at either extreme.

This is a strong, novel contribution worthy of NeurIPS.

The main weakness of the paper is in the visual comparisons presented, the authors don't show the ground truth image patch with the decoded patches, so even though the results seem to suggest their outcome is better than the comparisons, it is not entirely clear.  In addition, visual comparison at more points along the curve would also be useful.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper characterizes the perception-distortion in the context of (image) enhancement tradeoff, complementing prior results. For example, in image super resolution, this tradeoff describes the relationship between the point-wise reconstruction quality (distortion) and how well the distribution of the reconstructions match the distribution of the high-resolution data one aims to reconstruct. The paper derives this general tradeoff explicitly and characterizes certain aspects specifically for Gaussian distributions. The theoretical results are illustrated with experiments.
main_review: To my knowledge, the paper presents novel results characterizing the perception-distortion tradeoff for image (or more generally signal) enhancement as introduced by [4]. The paper focuses on MSE as a distortion and the Wasserstein-2 distance to measure perceptual quality. I particularly like that the authors derive Theorem 3 which provides practical guidance as to how to navigate the distortion-perception tradeoff. The paper is well-written.

I have the following questions and suggestions:
- MSE and the Wasserstein-2 distance are very specific choices of rate and distortion, and most super-resolution models are trained with different distortion metrics and/or GAN losses approximating a different divergence. What can be said about these cases? Any intuition whether similar properties as those proven here could be expected? This could also be empirically evaluated in the context of the experiments described in Sec 5. For example the authors could in addition use the Frechet Inception Distance as an additional perceptual index.
- The experiments in Sec 5 are done for deterministic decoders. How do the practical implications of Thm 3 (15) change for stochastic decoders as in [15, 3, 21]?
- It is sometimes unclear whether the m_i and \Sigma_i just refer to the first and second moments of a distribution (that could e.g. model natural images) or whether specifically Gaussian distributions are meant. This is in particular important because Guassians are a crude model for natural images (in the context of super resolution).
- In Sec 3.1 the authors argue that D^\star can be estimated using the loss of an MSE trained network after training. However, this value is usually an upper bound on D^\star and plugging it into (9) will invalidate the inequality. This should be pointed out clearly in the paper. If possible, it would be great to make a corresponding practical statement where one fixes the predictor and then derives a lower bound on D(P) for that predictor.
- I feel that the first statement in the abstract sounds too general. It should be mentioned that it is in the context of image restoration (or at least a problem that admits a distortion-perception tradeoff)

Overall I think the paper makes an interesting contribution.

limitations_and_societal_impact: For limitations please see the main review. In terms of societal impact it might make sense to briefly mention the potential issues inherent with perceptual super-resolution as possibly unwanted image content might be synthesized. In that context, the present paper is an advancement as it derives a method to adjust the D(P) tradeoff.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the trade-off between minimizing the mean-square error (distortion) and the
Wasserstein-2 distance (perception) when constructing an estimator \hat{X} of a random variable X.
In particular, the work analyzes a distortion-perception function, which given the radius P of a Wasserstein-2
ball around X returns the minimal mean-square error of \hat{X} constrained to be in that ball. 

The analysis reveals that for estimators with small Wasserstein-2 distance to the true distribution, one can often
obtain an estimator with much better mean-square error by a simple linear interpolation in the output space. 

The theoretical results are applied to deep image super-resolution, where it is shown that the perceptual quality of
an estimator with small MSE can be improved by interpolating it with one having small Wasserstein-2 distance. Remarkably,
the interpolated estimator often still has low MSE, while significantly lower Wasserstein-2 distance, leading to a greatly
improved perceptual quality. 
main_review: The paper is well-written, and I have no major objections regarding the overall clarity of the work. 

One concern is that the approach is in some sense an incremental step compared to [4], which already introduces the perception-distortion tradeoff in image restoration. The main contribution over that work is the closed-form expression for the distortion-perception function for the specific combination of MSE and Wasserstein-2 distance. 

To me, it appears that the significance / impact of the paper is overall rather limited, as the final estimator suggested by the theory is just a convex combination of the two estimator in output space. This seems to stem from the perhaps overly simplistic choice of MSE / Wasserstein-2, which ignores the fact that the manifold of natural images is a highly nonlinear space. Therefore, I feel that this strategy comes with obvious limitations and suggests that the considered measure of perception might not be too meaningful. 

I apologize if the above criticisms are based on some basic misconception / incomplete understanding of the contributions and I am willing to raise my score in such a case.

Finally, I found the experimental evaluation a bit short. Qualitative results are only shown on four different images. For Figure 3, I do not agree that there is an improvement in perceptual quality as claimed, especially for the first image. Such claims could be more rigorously evaluated using a small-scale study involving humans.

### Minor comments.
- Due to the concise choice of notation I had difficulties following the mathematical
details starting from Section 2.1. There, X, \hat{X} and Y are introduced as random vectors, which mathematically are
measurable functions mapping from a space which being left unspecified into R^{n_x} and R^{n_y}. These introduce the pushforward measures
p_X, p_{\hat{X}} and a certain conditional distribution p_{\hat X | Y}. Since these objects are introduced in a quite implicit/indirect way, 
mentioning the exact signature and definition of the density functions (or Borel probability measures?) would be helpful. 

- It might be beneficial for the reader also to explain what X, \hat X and Y specialize to in the setting of a concrete example,
perhaps for the case of image super-resolution considered in the applications sections. Maybe such an example can be included after line 76 in the paper, or in the appendix. 

- I felt a bit uneasy about the precise definition of Eq. (1). In particular, it is unclear with respect to which probability distribution
the expectation is taken (Is it wrt the unspecified distribution that X and \hat X pushforward or is it the conditional distribution?). What space does p_{\hat X | Y} live in? 
Perhaps these things could be expanded on in a bit more detail in Appendix A.1? 

- I can understand why an economical/concise notation is chosen (as is often in machine learning papers). But for me, it made checking the details in a rigorous fashion a bit exhausting. Overall I don't have doubts about the correctness of the results though. 

- The distortion-perception trade-off is studied on the example of image restoration. This is a fascinating and important field,  but perhaps a slightly niche topic for the NeurIPS audience, limiting the overall significance and impact of the work. Could the proposed framework perhaps 
considered in a broader context? For example, empirical risk minimization problems in machine learning optimize for low distortion; could generalization capabilities of ML methods perhaps be improved by a similar interpolation technique? 

- l. 154. I did not understand what a deterministic function is.

limitations_and_societal_impact: As mentioned in the main review, a discussion on the limitations of performing a simple linear interpolation in output space is missing. 
Are there for example "failure cases", where perception and distortion measured by the Wasserstein-distance and MSE are good
but the visual quality is worse than of the two individual estimators?

I don't believe that the work needs to address negative impacts on society, as it is largely theoretical.
review_title: null

Meta-review: Reviewers found this paper to provide an interesting and welcome application for optimal transport, even if the theoretical contribution is limited from a technical perspective.  After some deliberation, we converged on accepting this paper.

In the revision, please provide the following:

* More experimental evidence corroborating that a convex combination of two estimators results in visually pleasing results.
* Better introduction to the perception-distortion tradeoff for non-experts
* Optionally:  Detailed discussion (and additional theoretical results if possible) for distortions/divergences beyond the MSE/Wasserstein-2 case.
Decision: Accept (Poster)