Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a measure, namely Image Downscaling Assessment by Rate-Distortion (IDA-RD), to quantitatively evaluate image downscaling algorithms.
strength_and_weaknesses: The authors use traditional downscaling algorithms such as bicubic, bilinear, and nearest neighbor interpolation, as well as state-of-the-art downscaling algorithms to test the proposed measure. However, the reviewer does not recommend accepting the current version, because of the following concerns:

1. There are plenty of datasets for real-world super-resolution, i.e., the datasets contain the paired LR and HR images. Therefore, the performance of downsampling can be measured on these datasets by calculating the PSNR/LPIPS between the downsampled images and the original LR images. In comparison with this, what is the benefit of the proposed method?

2. The IDA-RD scores for DPID, Perceptual and L0-reg. are not able to indicate anything, since there is no evidence showing that the predicted score of the proposed measure is correct or not on these methods.

3. What is the relationship between the proposed measure and "rate-distortion"?

4. Why use StyleGAN on PULSE, but SRFlow on other datasets? It doesn't make sense to change the network for different datasets.


clarity,_quality,_novelty_and_reproducibility: The proposed method has some novelty, however, the authors should address the questions above. 
summary_of_the_review: The authors use traditional downscaling algorithms such as bicubic, bilinear, and nearest neighbor interpolation, as well as state-of-the-art downscaling algorithms to test the proposed measure. However, the reviewer does not recommend accepting the current version, because of the concerns in "Weaknesses" above.

----- After responses -----

The authors addressed some of my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is interesting. It gives an indepth analysis of down-scaling methods and degradations for gan sr networks. It uses rate distortion for evalutions and gives some interesting results. 
strength_and_weaknesses: Strenght:
1. Rate distortion for evluating down-scaler is interesting.
2. Comprehsive analysis of different down-scalers and different kinds of degradations.

Weaknesses
1. Only SRflow and Pulse sr networks are tested.
2. No realworld datasets are tested. only sythesis datasets.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good to follow and understand. 
Quality: High quality writing and presentation. 
Novelty: Somehow novel to use rate distoration model.
Reproduce: This work seems to be reproducedable. 
summary_of_the_review: This work worths published given its in depth analysis of down-scaling algorthms using rate distortion. Such results can be useful for scalable video coding in the future codec designs.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new method to measure image downscaling algorithms. The core idea is to upsample the downsampled image using blind/one-to-many generative-model-based super-resolution methods and compare the upsampled image with the original image. The similarity will be used as the final result. The authors show results on some synthetic data and compare different methods. 
strength_and_weaknesses: Weaknesses:
1. The statement of Eq (2) is very confusing. $Q$ denotes conditional distribution. The distribution of what? Of the lossy image reconstruction function $f$? Is the $Q$ can be written by $Q(f|f_{ds})$? Or should it be $Q(x’|x)$ according to the $P(z)\to Q(x’|x)$. I don’t know what the random variable is and I don’t know why there is a distribution. From any aspect, Eq (2) seems to make no sense. Moreover, is $x’$ follows a distribution? How can D measure between $x$ and $x’$? Is D a metric of distributions?
2. The many-to-many part is also very confusing. I know that the generative model generates a distribution or something, but it is not one-to-many. For everything that is generated by the model, we need a latent code. With the latent code, it is not one-to-many. And I don’t think we need a many-to-many statement here.
3. The above two questions make Eq (4) even more confusing.
4. In Eq (5), what is $z*$? And the mathematical is not presented in the right way.
5. For the experiments, There is no conclusive evidence proving that the proposed method is consistent with the human perception results. It is not clear whether the proposed method is linear in evaluation, or whether it is order-preserving. Experiments on synthetic images can provide limited evidence. However, due to the very large degradation of the synthetic images, it is difficult to make such a big difference when evaluating the downsampling method. So these experiments are not enough to provide evidence that their methods are consistent with human perception results.
6. Are there any other applications of the proposed method? For example as a clue to know new downsampling methods?
clarity,_quality,_novelty_and_reproducibility: This paper is very sloppy in its mathematical presentation. After many readings, I can probably understand the author's intention. But it's not simple. I need to guess what the author actually intended.
summary_of_the_review: The main problem with the paper is the writing, and imperfect experimental proof of the performance and significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new metric, namely IDA-RD, to evaluate image downscaling algorithms quantitatively. To this end, the authors employ the idea that a downscaling algorithm that preserves more details in the resulting low-resolution images should lead to less distorted high-resolution images constructed by state-of-the-art super-resolution methods. Experimental comparisons are made over various downscaling operations with different levels of distortions to demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths
 
1. The paper addresses an interesting problem of measuring the downscaling methods
2. The paper does contribute some new ideas to tackle the problem at hand
3. The proposed methods outperform some of the other existing metrics on domain-specific image datasets

Weaknesses

1. Missing comparisons: There are some other upscaling super-resolution (SR) functions that are likely to handle the unseen degradations well, for example, methods in [p1-p3]. The experiments should be conducted with these methods as well.

[p1] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution, ICCV 2021
[p2] Real-esrgan: Training real-world blind super-resolution with pure synthetic data, ICCVW 2021
[p3] Generalized Real-World Super-Resolution Through Adversarial Robustness, ICCVW 2021


2. Conclusions from little pieces of evidence: (Refer: Section 4.3) Based on the results, authors argue that SOTA downscaling methods (Weber et al., 2016; Liu et al., 2017; Oeztireli & Gross, 2015) actually lose more information than Bilinear downscaling. However, this conclusion comes from the assumption that the proposed method is optimal for measuring the distortions of any arbitrary downscaling operation. This assumption is unlikely to be true and needs further evaluation, analysis, and comparison before coming to a conclusion. To this end, I would recommend adding “comparison: with mean opinion score (i..e, feedback from humans based on perceptual understanding)”, “analysis: such as picking some examples that best reveal the distortion effects and then arguing why the SOTA methods perform poorly in comparison to bilinear”, and  “evaluation: ensure fair evaluation since the SOTA methods might have fine-tuned for say natural images or specific types of images or distortions”. Altogether we need much more experimental evidence in support of this kind of argument that actually contradicts the common beliefs in the literature. Moreover, the experiments are done only with domain-specific images, which also makes adds to the fact that the proposed method is yet an “incomplete metric”.

3. Clarity on the choices of methods: 
- Regarding the GAN generator-based method: Based on Eq. 5, the proposed method employs the downscaling method that we wish to evaluate to generate the LR image for training the super-resolution (SR) method. Such training can be done with any fully-supervised SR method. Henceforth, it is not clear why other SR methods would fail to do the same job or what the limitations are in using other methods. There should also be experiments to support such arguments/failure cases. There seem to be some comparisons under “Invalidity of Non-blind and Non-stochastic SR method” it is not clear if the experiments were done with pre-trained networks or the ones trained with “f_ds” as in Eq. 5.
- Regarding the SRflow method: This method employs bicubic downscaling for training the SR network is more attractive since we can avoid retraining the SR network to evaluate any new downscaling method. However, as I have mentioned before, a more “generalizable to unseen degradations SR network” could be [p1-p3] and should therefore be included in experiments and comparisons. 

4. Why only domain-specific datasets in experiments and comparisons?

The authors seem to have used only the face images for all the comparisons. Why are the natural images not included in the study? Or is it like the authors were trying to tackle metrics for certain domain-specific images alone?
clarity,_quality,_novelty_and_reproducibility: The work contributes some novel ideas, such as the notion of using "information loss during downscaling via SR networks as a quantitative metric." Some of the descriptions are missing with respect to the experiments (for example, under "Invalidity of Non-blind and Non-stochastic SR method," it is not clear if the author used pre-trained networks for ESRGAN (Wang et al., 2018) and SR3 (Saharia et al., 2022))
summary_of_the_review: While the work presents some interesting ideas, many comparisons and experiments are either missing or are not done fairly. The overall evaluation of the method seems to lack multiple aspects, which makes the paper below the acceptance threshold. 

Comments after Author Response:
Based on the author's response I am changing my rating from Reject to Accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new metric to evaluate image downscaling algorithms, using the lens of rate distortion theory.

Strengths:
- Use of rate distortion theory to evaluate down-scalers is interesting
- Comprehensive evaluation of different down-scalers under different types of image degradations

Weaknesses:
- Unclear what the advantage of this method is over evaluating PSNR on ground-truth (hi res, lo res) paired data. Authors responded saying that in these paired datasets, lo res data is *not* ground truth because it is produced via some process such as bicubic downsampling. The reviewer responded saying actually, no, both lo-ris and hi-res images are captured by cameras, so they *are* ground truth. Author did not follow up to this.
- Evaluation is only on synthetic data; no real-world data. Authors did upload a revised version of the paper with real-world results that also supports their claims.
- Missing comparisons to some other down-scalers (which the authors added in the revision)

All reviewers were negative. A couple responded to author responses and were not swayed. Seems like clear reject.


justification_for_why_not_higher_score: All reviewers were negative about this paper. Those who acknowledged the author responses were not swayed by those responses. It is especially concerning that the authors seem to misunderstand how ground truth (low res, high res) image pairs are used for evaluating downscaling algorithms: given that such ground truth pairs exist, the usefulness of the proposed method is questionable.
Decision: Reject