Deep Generative Model based Rate-Distortion for Image Downscaling Assessment
Authors: Yuanbang Liang
Bhavesh Garg
Paul L Rosin
Yipeng Qin
Venue: Submitted to ICLR 2023 (2023)

Abstract: In this paper, we propose a novel measure, namely Image Downscaling Assessment by Rate-Distortion (IDA-RD), to quantitatively evaluate image downscaling algorithms. In contrast to image-based methods that measure the quality of downscaled images, ours is process-based that draws ideas from the rate-distortion theory to measure the distortion incurred during downscaling. Our main idea is that downscaling and super-resolution (SR) can be viewed as the encoding and decoding processes in the rate-distortion model, respectively, and that a downscaling algorithm that preserves more details in the resulting low-resolution (LR) images should lead to less distorted high-resolution (HR) images in SR. In other words, the distortion should increase as the downscaling algorithm deteriorates. However, it is non-trivial to measure this distortion as it requires the SR algorithm to be blind and stochastic. Our key insight is that such requirements can be met by recent SR algorithms based on deep generative models that can find all matching HR images for a given LR image on their learned image manifolds. Empirically, we first validate our IDA-RD measure with synthetic downscaling algorithms which simulate distortions by adding various types and levels of degradations to the downscaled images. We then test our measure on traditional downscaling algorithms such as bicubic, bilinear, nearest neighbor interpolation as well as state-of-the-art downscaling algorithms such as DPID, L0-regularized downscaling, and Perceptual downscaling. Experimental results show the effectiveness of our IDA-RD in evaluating image downscaling algorithms.

Under review as a conference paper at ICLR 2023
DEEP GENERATIVE MODEL BASED RATE-DISTORTION FOR IMAGE DOWNSCALING ASSESSMENT
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a novel measure, namely Image Downscaling Assess- ment by Rate-Distortion (IDA -DD,, to quantitatively evaluate image downscaling algorithms. In contrast to image-based methods that measure the quality of down- scaled images, ours is process-based that draws ideas from the rate-distortion the- ory measure the distortion incurred during downscaling. Our main idea is that downscaling and super-resolution (SR) can be viewed as the encoding and decod- ing processes in the rate-distortion model, respectively, and that a downscaling algorithm that preserves more details in the resulting low-resolution (LR) images should lead to less distorted high-ress nigh-resolution (HR) images in SR. In other words, the distortion should increase the downscaling algorithm deteriorates. However, it is non-trivial to measure this distortion as requires the SR algorithm to be blind and stochastic. Our key insight is that such requirements can be met by recent SR algorithms based on deep generative models that can find all matching HR im- ages for given LR image on their learned image manifolds. Empirically, we first validate our IDA-RD measure with synthetic downscaling algorithms which sim- ulate distortions by adding various types and levels of degradations the down- scaled images. We then test our measure on traditional downscaling algorithms such as bicubic, bilinear, nearest neighbor interpolation as well as state-of-the-ee downscaling algorithms such DPID (Weber et al. (2016), Lo-regularized down- scaling (Liu et al. 2017). and Perceptual downscaling (Oeztireli & Gross 2015). Experimental results show the effectiveness of our IDA-RD in evaluating image downscaling algorithms.
INTRODUCTION
Image downscaling is fundamental problem in image processing and computer vision. To ad- dress the diverse application scenarios, various digital devices with different resolutions, such as smartphones, iPads, and desktop monitors, co-exist, which makes this problem even more impor- tant. contrast to image super-resolution (SR), which aims to "add" information to low-resolution (LR) images, image downscaling algorithms focus "preserving" information present in the high- resolution (HR) images, which is particularly important for applications and devices with very lim- ited screen spaces.
Traditional image downscaling algorithms low-pass filter an image before resampling it. While this prevents aliasing in the downscaled LR image, important high-frequency details of the HR image removed simultaneously. resulting in blurred or overly-smooth LR image. To improve the quality of downscaled images, several sophisticated approaches have been proposed recently, including remapping of high-frequency information (Gastal & Oliveira, 2017). optimization of perceptual priors (Liu et al. 2017). and pixelizing the HR image (Gerstner al. 2012 Han et al.) 2018 8 Kuang et al.] | & Wong, 2021). Nevertheless, research in image downscaling algorithms has significantly slowed down due to the lack of a quantitative measure to evaluate them. Specifically, standard distance measures (e.g. L1, L2 norm) and full-reference image quality assessment (QQA) methods are not applicable here due to the absence of ground truth LR images; existing No-Reference IQA (NR-IQA) metrics (Mittal jet al.) 2012bbe a Bosse et al. 2017) cannot be applied either as they rely on the "naturalness" of HR images, which is not present in L.R images (we will verify this in our experiments).
image quality metrics (Oeztirel & Gross 2015). using L.0rreullardi

---

Under review as a conference paper at ICLR 2023
In this paper, we propose new quantitative measure for image downscaling based Claude Shannon's 's rate-distortion theory (Berger 2003), namely Image Downscaling Assessment by Rate- Distortion (IDA-RD). The main idea of IDA-RD measure is that superior image downscaling algorithm would try to retain as much information as possible in the LR image, thereby reducing the distortion when being up-sealed (a..a. super-resolved) to the size of the original HR image. However, such an upscaling method is non-trivial as it must satisfy two challenging requirements: i) blindness, i.e. must apply to all kinds of downscaling algorithms without knowing them in advance; ii) stochasticity, i.e. it must be able to generate manifold of HR images that captures the conditional distribution of the super-resolution process. Our key insight is that both such requirements can be satisfied by the recent success of deep generative models in blind and stochastic super-resolution. To demonstrate the flexibility of our IDA-RD measure, we show that it can be successfully implemented with two mainstream generative models: Generative Adversarial Networks (Menon et al. (2220) and Normalizing Flows (Lugmayr et al. 2020). Extensive experiments demonstrate the effectiveness of our IDA-RD measure in evaluating image downscaling algorithms. Our contributions include:
Drawing on Claude Shannon's 's rate-distortion theory (Berger 2003 we propose the Image
Downscaling Assessment by Rate-Distortin (IDA-RD) measure to quantitatively evaluate image downscaling algorithms, which fills a gap in existing image downscaling research.
We demonstrate the effectiveness of our IDA-RD measure with extensive experiments on both synthetic and real-world image downscaling algorithms.
2 RELATED WORK
Image Downscaling has a long history and its traditional methods (e.g. bicubic) have now become the standard for image processing and computer vision software, making it difficult to trace their origins. To this end, we only review recent attempts in developing better image downscaling algo- rithms. For example, Gastal & Oliveira (2077) conducted a discrete Gabor frequency analysis and propose to remap the high-freyuency information of HR images to the representable range of the downsampled spectrum, thereby preserving high frequency details in image downscaling. Oeztireli & Gross (2015) model image downscaling as an optimization problem and minimize a perceptual metric (SSIM) between the input and downscaled image. However, the limitations of SSIM are also carried over to their approach. DPID (Weber et al.) (016)) preserves small details by assigning higher weights the input pixels whose color deviates from their local neighborhood within the convolu- tional filter. Liu et al. (017)) propose an optimization framework using two L. regularized priors that addresses two issues of image downscaling, i.e. salient feature preservation and downscaled im- age construction. Image thumbnailing, special case of image downscaling, has been studied by [Sun) & Ling (2013). Their two-component thumbnailing framework, named as Scale and Object Aware Thumbnailing (SOAT) focuses on saliency measure and thumbnail cropping. Li et al. (018)) term image downscaling as image Compact Resolution (CR) and address it with a Convolutional Neural Network (CNN). Inspired by the success of CNNs in image super-resolution (SR), they introduce the CNN-CR model for image downscaling that can be jointly trained with any CNN-SR model. Although their CNN-CR model results in better reconstruction quality than other downscaling al- gorithms, they only demonstrate results for small downscaling factors ( (x2). However, the majority of both image downscaling and super-resolution algorithms tend to focus on larger scaling factors (e.g. x8). Despite the aforementioned works, there does not exist a good quantitative measure for the evaluation of image downscaling methods, which impedes the research on them.
Image Quality Assessment (IQA) can be subjective or objective. Between them, subjective meth- ods rely on the visual inspection by human assessors while objective methods resort to quantitative measures, e.g., image statistics. Examples of the most commonly used objective IQA metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Multi-Scale SSIM (MS-SSIM) (Wang et al.) 2003) and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang ctt al. 2018). However, such IQA metrics are not applicable in the evaluation of image downscaling algorithms there are ground truth LR images for comparison. Thus, most researchers rely on subjective evaluation of downscaled images, which is costly and time-come mine-consumngg.
No-Reference Image Quality Assessment -R-IQQA addresses IQA in the absence of reference (i.e. ground truth) image. For example, Mittal et al. (2012a) propose BRISQUE, an NR-IQA metric that uses the natural scene statistics (NSS) quantify loss of "naturalness" in distorted images.
2

---

Under review as a conference paper at ICLR 2023
Imput Downsraling Method
f
f Jus
Ex,ol D
X
IDA-RD
Figure 1: Illustration of the proposed IDA-RD measure. Given downscaling method fas to be evaluated, i) we first use it to downscale several HR images; ii) then, we upscale them back to the original resolution with fus and measure the distortion from the corresponding HR images. Such an upscaling method leverages the recent success in deep generative models and thus can i) apply to arbitrarily down-scaled images and ii) output manifold of HR images that captures the conditional distribution given downscaled image.
Using locally normalized luminances, BRISQUE models regressor which maps the feature space to image quality scores. Based on their NSS, Mittal et al. (2012b) further devised an Opinion Unaware (OU) and Distortion Unaware (DU) model for blind NR-IQA, which is named NIQE, Bosse et al. (2077) follow a data-driven approach for NR-IQA. Inspired by Siamese networks, they train a deep neural network for feature extraction and regression in end-to-end manner. However, due the lack of a large enough training dataset, their model does not generalize well. However, such NQ- IQA metrics also not applicable, the "naturalness" they rely exists only in HR but not LR images. To this end, we borrow ideas from Claude Shannon's 's rate-distortion theory and propose a new measure called Image Downscaling Assessment by Rate-Distortion (IDA-RD). Our IDA-RD measure leverages the recent success in deep generative models and shows promising results in the quantitative evaluation of image downscaling methods.
Deep Generative Models We refer interested readers to Bond Taylor et al. (2021) for detailed survey on deep generative modeling. Here, we review the two deep generative models used in our work, i.e. Generative Adversarial Networks (GANs) and normalizing flows. Since the pioneering work by Gooofflloow et al. (2014), GANs have experienced significant improvements. For exam- plc. Radford al. (2015) proposed DCGAN, which incorporates convolutional neural networks for better image synthesis. Arjovsky et al. (2017) addressed the notorious instability of GAN training by employing novel loss function, i.e. the Wasserstein distance loss. To date, the StyleGAN ries (Karras et al. 2019; 2020 2021) developed by Nvidia has shown impressive (maybe even the best) results in high-resolution and high-quality image synthesis, leading to various applications in image processing and manipulation (Abdal et al. 2019 2020 Zhu et al.) 2020). In this paper, we with a StyleGAN generator pre-trained portrait images. Nevertheless, normalizing flows (Rezende & Mohamed] (2015] Papamakarios et al.) 2021 Keller al., 202I) that construct complex distributions by transforming probability density function through a series of invertible mappings have attracted increasing attention in the past sev- eral years. In this paper, we employ the SRFlow (Lugmayr et al., 2020) model to implement our measure, which directly learns the conditional distribution of the HR output given the LR input.
follow Menon et al. (2020) and implement
3 OUR APPROACH
In this section, we first introduce the definition of our metric derived from Claude Shannon's 's rate- distortion theory (Berger 2000). and then detail how deep generative models help to sidestep the data scarcity challenge that impedes the application of the proposed metric.
3.1 METRIC DEFINITION
We create a proxy task, namely the lossy compression problem underpinned by Claude Shannon's 's rate dite-ditortion theory (Berger 2003). and formulate image downscaling as its encoding process:
inf (X,, X)] s.t. (x( s R (8](r)
(I)
3

---

Under review as a conference paper at ICLR 2023
where X is the set of input high-resolution images, X is the set of output reconstructed images, R is a rate constraint determined by the downscaling process Q (0)z) x) or Q for short is the probability density function (PDF) of reconstructed HR images è conditioned on an input HR image with respect to a given lossy image reconstruction function f that à f(x) = [fs((a(())) where fus and fas denote image upscaling and downscaling functions respectively, Do distortion metric between two image sets where the image correspondence is determined by Q. Thus, we propose to use the expectation of the distortion an evaluation metric for image downscaling:
S(fa.) = (X, = E., (a (D(,,
where x E X E X, D is distortion metric between two images, e.g., s., LPIPS Zhang et al. 2018 The lower S, the better the downscaling algorithm fas- Although straightforward, the application of such a metric remained a challenge in the past as it requires a strong upscaling function fus that can:
Reconstruct the input image x regardless of the input downscaling algorithm fas-
Generate a conditional distribution of reconstructed images #|z for each rr
Between them, the first is commonly known blind image super- resootresotttion that is essentially many-to-one mapping problem that aims to map different distorted downscaled images to the same high-resolution image; the second is commonly known one-to-many super-resolution due to its ill-posed nature caused by the information loss during downscaling (Lugmayr et al. 2020).
Data Scarcity Challenge Combining the above two requirements makes the desired fus an ex- tremely challenging many-to-many mapping problem that has remained unsolved for decades. Specifically, the numerous kinds of distorted downscaled images and the corresponding countless high-resolution images for each of them makes it infeasible to collect sufficient data for supervised leamning methods:
fue arg min Eisn (Eins) [HR fo(ILR)]l)
to
where IHR and LRR denote the high resoluesotttion (HR) and low-resolution (LR) training images re- spectively, ERR indicates that there are many HR corresponding to the same LRR, ET indicates that there are many LA obtained by different image downscaling methods fas-
3.2 EVALUATION WITH DEEP GENERATIVE MODELS
Our key insight is that the above-mentioned data scarcity challenge (Eq. can be overcome by the recent successes in deep generative modeling (Goodfellow et al.) 2014; Radford et al.) 20155] Arrovsky et al., 2017; Karras et al.) 2019; 2020; 2021; Rezende & Mohamed 2015 Papamakarios et 2021 Keller et a 2001 In deep generative modeling, neural network model is trained to learn manifold of natural and high-resolution (HR) images from samples in the training dataset. This has been successfully applied to various image processing tasks (Abdal et al. 2019 2020) Zhu et al. 2020). To demonstrate the flexibility of our metric, we show its two imp implementations using two mainstream deep generative models: i) Generative Adversarial Networks (GANs) and ii) Normalizing Flows respectively as follows.
Implementation with GAN generator. Similar Menon al. (2020). we implement the upsam- pling function fus in our metric using an optimization-base based GAN inversion method (Abdal et al.) 2019 2020). Leveraging the power of a pre-trained StyleGAN (Krraas et al. 2019 generator G, we define our GAN-based fus (Eq. (2 as locating the optimized StyleGAN latent code z so that its
corresponding HR image G(zţ) synthesized by G' shares the same downscaled image as an input LR image Inn fas (x):
(uo(RR G(z]) = arg min |R - Sás (G(z)))
(4)
G(z))
where ILR = fas denotes the input LR image downscaled by fae. z₁ denotes the i-th randomly initialized latent code to be optimized to get the i-th sample from àil Tnn (L.e, G(z )), i 1,
is the index. It can be observed that i) our fas sidesteps the data scarcity challenge (Eq. 3 by using
Note that in image downscaling, such constraint on R is always satisfied as the downscaled images of fixed resolution defined by users.
4

---

Under review as a conference paper at ICLR 2023
StyleGAN generator that is trained with HR images only (i.e., without any many-to many-to-mam many LR-HR training pairs); ii) it relocates the supervision to downscaling (i.. enforcing different HR images to be downscaled to the same LR image) and thus outputs high quality HR images G(zţ) ) that applies to an arbitrary choice of fas; iii) it is inherently stochastic given the random choices of 24-
Implementation with Flow model. We employ the SRFlow model (Lugmayr et al. 2020) and implement the fus in our metric with a conditional invertible neural network. Leveraging its invert- ible nature. Sus is trained to explicitly learn the conditional distribution L|IER R. by minimizing the negative log-likelihood:
fus arg min log Pa (f (RR))
(5)
fo
where ILR = fäs fhicubic(x)) a bicubic downscaled image of HR imput z, z denotes a random latent variable whose distribution encodes ž IL with 'reparameterization trick'. Although trained with only bicubic downscaling, surprisingly, we observed that the resulting fus can also be applied to evaluate other downscaling methods.
We use SRFlow in the final version of our metric as it shares similar performance as the GAN-based implementation but has a much lower time cost. Please see Sec. 4.4 for detailed ablation study.
4 EXPERIMENTS
To validate the effectiveness of our IDA-RD measure, we first test it with synthetic image downscal- ing methods whose performance are known beforehand (Sec. (Sec.4.2), (42). Specifically, we simulate differ- ent types and levels of downscaling distortions by adding controllable degradations (e.g., Gaussian Blur, Contrast Change) to bicubic-downscaled images. In principle, the heavier the degradation, the worse the results of downscaling, and the higher our measure should be. We also validate the effectiveness of our IDA-RD measure across different scaling factors. Then, we show that our mea- sure can also be used to evaluate real-world image downscaling methods like Bicubic, Bilinear, Nearest Neighbour, and state-of thett- downscaling methods like LO-regularized (Liu et al.) 2017)) Perceptual (Oeztireli Gross| 2015) and DPID (Weber al., 2016) (Sec. 4.3). Third, we perform a thorough ablation study to justify the algorithmic choices of our measure (Sec. 4.4). Finally, we empirically justify our motivation in Sec. [4.5 Please see the appendix for additional experiments and examples of downscaled images (Appendix A.1
4.1 EXPERIMENTAL SETUP
Dataset Unless specified, we use a balanced subset of 900 images from the FFHQ dataset (Karras al. 2019). including face images at 1024 x 1024 resolution, as the set of input high-resolution images X in Eq. for our IDA-RD measure. Please see Appendix A.2 for more details on how we construct balanced subsets of images from FFHQ. The results on other datasets, including NPRpor- trait 1.0 (Rosin et al. (022)) and AFHQ-Cat (Choi et al. 2020), are shown in Sec. 4.4 Note that we use these domain-
datasets as they are more stable for SRFlow. Please see Appendix A.8 for the results and discussions real-world datasets, e.g., DIV2K (Agustsson & Timolte 2017
Flickr30k (Young et al., 2014).
Image Upscaling Algorithms We use SRFlow (Lummayr al. 2020 as the fus in Eq Specifi- cally, we used the models provided by the authors for 4x and 8 x super resolution that are pre-trained on DIV2K (Agustsson & Timofte 2017) and Flickr2K datasets") Unless specified,
the 8 model for all experiments. Note that we also tested PULSE (Menon et al.) 2020) as an alternative in Sec. 4.4 For PULSE, we use the same StyleGAN generator pre-trained with FFHQ (Karras al.) 2019). This model generates face images of size x024x 1024. We use a learning rate of 0.4, and stop the optimization for each image after 200 steps of spherical gradient descent. The noise signals of the StyleGAN generator were kept fixed.
Hyperparameters Unless specified, we use Ng 5 as the number of images upscaled from
single downscaled image for the estimation of Q in Eq. [ we use LPIPS (Zhang et al.) 2018)) as the distortion measure 1) in Eq. 2, we use Nx = 900 as the number of images in the set of high-
resolution image X in Eq. 2
gttps:://itthbb.coo http://ttt..ub.cccmmmm//eeeeee 121/ SRF
5

---

Under review as a conference paper at ICLR 2023
Table IDA-RD scores for synthetic image downscaling with different types and levels of degrada- tions (a), (b); with mixed degradations (c). The numbers in parentheses denote degradation param- eters. As a reference, the IDA-RD score for the bicubie-downsealed image without degradation is 0.11 It is best to Zoom In to view the examples of downscaled images with different types and levels of degradations.
Gauss, Blur
Gauss. Noise
Contrast Inc.
Contrast Dec.
(1.0)
0.320+0.048
(0.05)
0.482-000051
(1.5)
0.231=0.042
(0.75)
0.330=0.004
(2.0)
0.434=0.057
(0.10)
0.640=00.52
(2.0)
0.317=0.041
(0.50)
0.644=00077
(4.0)
0.579100005
(0.20)
0.65911.0022
(2.5)
0.462-0.043
(0.25)
0.669:10.034
(a)
Quantization
Gauss. Blur (1)
0.320++0.048
(15)
0, 0.4400000002
+ Gauss. Noise (0.05)
0.585:10.062
(10)
0.205
Contrast Dec. (0.75)
0.664+0.046
(5)
0.463
+ Quantization (10)
0.795+#0.063
(b)
(c)
Table 2: IDA-RD scores for synthetic image downscaling methods with different scaling factors.
()): the resolution of downscaled images. Bicubic: bicubicdoownscleed image without degradation.
G.B.: Gaussian Blur. The 32x super-resolution is achieved by a concatenation 8x and 4x upscaling implemented by pretrained SRFlow models.
Scaling Factor 4x (256 x 256)
Bicubic
G.B. (a 1.0)
G.B. (o 2.0)
G.B. (a 4.0)
0.058:0.144
0.146:F0.032
0.269-00.4
0.055
8x (128 x 128)
0.110+0.145
0.320=0.048
0.434
32x (32 x 32)
0.228:100566
0.61410.008
0.680=0.066
0.741=0.065
4.2 TEST WITH SYNTHETIC DOWNSCALING METHODS
In this section, we demonstrate the effectiveness our IDA-RD measure by testing its performance on synthetic downscaling methods, which simulate the effects of different downscaling methods by adding controllable degradations bicubie-donnscaled images.
4.2.1 EFFECTIVENESS ACROSS DEGRADATION TYPES AND LEVELS
As detailed below, we test our IDA-RD measure with four sets of synthetic downscaling methods that apply different types and levels of degradations to bicubic-downseall images respectively.
Gaussian Blur.
We apply Gaussian blur to the -oweseed images. The standard deviation of the blur kernel are shown in Table (a).
chosen from {1.0, 2.0, 4.0). The kernel size was set as (3o + 1). The results
Gaussian Noise. We add Gaussian noise to the bicubie-downscaled images. The standard deviation of the noise is chosen from {0.05, 0.1, 0.2). The results are shown in Table (a).
Contrast Change. apply contrast change to bicubie-downscaled images. To increase the con- trast, we select the scale factor from (1.5, 2.0, 2.5). Note that such scaling can cause degradation due to the clipping of extreme intensity values. Similarly, to decrease the contrast, we select the contrast parameter from (0.25, 0.50, 0.75). The results are shown in Table (a).
Quantization. We apply pixel quantization to bicubic-downscaled images and select the number of color thresholds from (5, 10, 15). Specifically, we apply Otsu's multilevel thresholding algo- rithm (Otsu) 1979) to the graylevel histogram which is derived from the color image, and then apply these thresholds uniformly to each of the RGB color channels. The results shown in Table (b).
Mixed Degradations. addition to single degradations mentioned above, we also demonstrate the effectiveness of IDARRD measure their mixtures. The results shown in Table (c).

---

Under review as a conference paper at ICLR 2023
Table 3: IDA-RD scores for real-world image downscaling methods with different scaling factors. S.F.: Scaling Factor, the resolutions of downscaled images (e.g., 512 for 2x, 64x64 for 16x), are omitted for simplicity. N.N.: Nearest Neighbour. L0-reg.: LO-regularized. Note that the rela- tively large standard deviations in some cases (especially when the scaling factors are small) indicate the algorithmic biases of image downscaling methods against individual images, egg., flat images with large color blocks may suffer less from information loss. The 32x super-resolution is achieved by a concatenation of 8x and a 4x upscaling implemented by pretrained SRFlow models.
S.F.
Bicubic
Bilincar
N.N.
DPID
Perceptual
L0-reg.
4x
0.042
0.031=0.053
0.335+003100
0.122
0.388+0.321
0.136=t0.251
8x
0.11010.145 0.208
0.090=0.067 0.272=0.056
0.512:10.300
0.127:1.294 0.601=0.163
0.213-103301 0.398:10.377 0.291
32x
0.51440.152
0.307=0.050
Table 4:
Ablation study of Sus. the image upscaling algorithms. PULSE (Menon et al. 2020 and
SRFlow (Lugmayr al. 2020) have similar results but those of SRFlow are more distinguishable.
Bicubic
Bilinear
N.N.
DPID
Perceptual
LO-reg.
PULSE
0.17150.015 0.164 0.090=0.067
0.254+0.018
0.179=0.016
0.2234:0017 0.398+0.337
0.205=0.0016
SRFlow
0.110#0.145
0.512+0,340
0.127=0.294
0.213=00301
can be observed that our IDA-RD measure works as expected (i.e., the stronger the degradation, the worse the downscaling algorithm, and the higher the IDA-RD) for all synthetic image downscaling methods, which demonstrates its effectiveness.
4.2.2 EFFECTIVENESS ACROSS SCALE FACTORS
We further demonstrate the effectiveness of our IDA-RD measure on synthetic downscaling algo- rithms across different scaling factors. As Table shows, we test our IDA-RD on synthetic down- scaling algorithms of different levels of Gaussian Blur degradation as mentioned above. It can be observed that: i) the larger the scaling factor, the more the information loss, and the higher the IDA-RD; ii) the stronger the degradation, the worse the downscaling algorithm, and the higher the IDA-RD; which justifies the validity our IDA-RD measure.
4.3 EVALUATING EXISTING DOWNSCALING METHODS
We apply our method to compare six existing downscaling algorithms, consisting of three traditional methods: Bicubic, Bilinear, Nearest Neighbor (N.N.), and three state of the methods: DPID (We- ber et al. 2016). LO-regularized downscaling (Liu et al. 2017). and Perceptual (Oeztireli & Gross) 2015) downscaling. The results are shown in Table classical downscaling algorithms (i.e., Bicubic, Bilinear, and N.N.), our IDA-RD measure identifies the quality of these algorithms the correct order (Bilinear > Bicubic > N.N.), although the dif- ference between the results of Bicubic and Bilinear downscaling is not significant as expected; ii) when applied to SOTA ones, the common belief that these algorithms should perform better than Bilinear downscaling. However, none of these methods achieve better in IDA-RD. suggesting that although SOTA image downscaling methods excel perceptual quality, they actually lose more in- formation than Bilinear downscaling. Nevertheless, it can be observed that DPID and I .0-regularized methods are slightly better than Perceptual downscaling on our IDA-RD measure, which is consis- tent with previous understanding. These indicate that our IDA-RD measure to visual inspection, i.e., good image downscaling algorithm should be both visually satisfying and achieve low IDA-RD score, which further validates the role of our measure providing new insights into image downscaling algorithms. Please see Appendix A.10 A.10 for a qualitative comparison.
can observed that: i) when applied to
useful complement
4.4 ABLATION STUDY
In this experiment, justify the algorithmic choices of our IDA-RD measure, i.e., fuss D, the number of images used to estimate Q and in X, and the content of X in Eq. 2 by performing a thorough ablation study them.
7

---

Under review as a conference paper at ICLR 2023
Table Ablation study of No. the number of images required for robust estimation of in Eq
No 1
3
5
10
15
Bicubic
0.103=0.141
0.109=0..12
0.110 +0.145
0.111#0.145
1.10==0.14
Bilinear 0.090=0.069
0.090=000077
0.090
+0.067
0.090+00.067
0.090=00.067 0.512:10.340
N.N.
0.513=0.341
0.312:10.340
0.512 +0.340
1..1400
Table 6: Ablation study D, the distortion measure in Eq. 2
Dec.: Decrease. Param.: Parameter.
Please see Table 14 in Appendix A.3 for experiments with other synthetic downscaling methods.
Param.
PSNR
SSIM
MS-SSIM
LPIPS
0.75
22.137:+4.020
0.834
0.881#0.101
0.330=10.047
Contrast Dec.
0.50
17.814122.148
0.71410.087
0.81911.0800
0.64411.007
0.25
14.790:11.461
0.578
0.579-10.028
0.669-10.034
1.00
25.159
0.744
0.929
0.320+0.008
Gaussian Blur
2.00
22.365
0.646+00073
0.849+00.03
0.434+00.057
4.00
1..739
0.558
0.715#0.051
0.579#0.065
Choice of fus. As Table 4 shows, both PULSE (Menon et al. 2020) and SRFlow Lugmayr et al. 2020j have similar results when used as sas our IDA-RD measure, ie., N.N. Perceptual > LO-regularized > DPID > Bicubic > Bilinear. However, since SRFlow yields more distinguishable results and runs much faster (Table 15 in Appendix (A.4), we use it in our IDA-RD measure. Never- theless, our IDA-RD is very flexible (i.e., not restricted to PULSE or SRFlow) and will benefit from future progresses of blind and stochastic super-resolution methods (please see Appendix A.9
Number of Images used to Estimate Q. As Table 5 shows, for downscaled image, we investigate how many images are required to be upscaled from it (by fue ) to achieve robust estimation of the conditional distribution Q and thus IDA-RD, namely No. It can be observed that the results become stable when NQ 2 5, so we choose No = for our IDA-RD measure.
Choice of D. As Table 6 shows, test different choices of D including multiple image distortion metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) (Wang al. 2004), MS-SSIM (Multi-Scale SSIM), and LPIPS (Zhang al., 2018). Experimental results demonstrate a similar trend across all of them, indicating the flexibility of our IDA-RD measure. Nevertheless, since LPIPS is a more advanced metric that has been shown be more consistent with human perception, we use it in the final version of our IDA-RD measure.
Number of Images in X. As Table 7 we investigate how many images are required in the test dataset X consisting of high- high-resolution images to achieve robust estimation of IDA-RD, namely Nx. can be observed that the results become stable when Nx Z 900, so we choose Nx 900 for our IDA-RD measure.
Table 7: Ablation study of Nx, the number of images in test dataset in Eq. Synthetic image downscaling methods with Contrast Decrease with 0.75 (DGI); Gaussian Noise with 0.05 (DG2); mixed noise consisting of Gaussian Blur with a 1.0, Contrast Decrease with a 0.75. and Gaussian Noise with 0.05 (DG3); are used in the experiments.
Nx
30
300
600
900
1200
1500
DGI 0.320+0.026
0.32140.047
0.32140.046
0.330=0.047
0.325+0.047
0.329:0.047
DG2
0.50110.005
0.473110051
0.48110.050
0.48210.051
0.4831100051
0.484:100051
DG3
0.483++0088
0.312++0008
0.321+00005
0.320+0.008
0.321
0.322+00.04
The Content of X. As Table in addition to FFHQ (Karras et al. 2019), we test our IDA-
RD measure on another two datasets: the NPRportrait 1.0 benchmark set (Rosin et al. 2022) and AFHQ-Cat (Choi et al. 2020). Between them. we use all 60 images at around 800x 1024 resolution from the NPRportrait controlled diversity of gender, age and ethnicity: we use a random sample of 900 images at 512x512
benchmark set X, which was carefully constructed so to include
resolution from the AFHQ-Cat dataset as X. We test them with 4x image downscaling. It can be
8

---

Under review as a conference paper at ICLR 2023
Table 8: Ablation study of the contents of dataset X in Eq.
(1) Bicubic (2) Bilinear (3) Nearest
Neighbor (N.N.) (4) DPID (5) Perceptual (6) L0-regularized.
FFHQ
NPRportrait 1.0
AFHQ-Cat
(I)
0.110=0.145
0.119
0.107
(2)
0.090:10.067
0.100:10.101
0.091+:0033
0.312:10.40
0.3291:.292
0.277+0.1103
03
(4)
0.127=0.294
0.119:10.099
0.152
02
0.398-1.337
0.391
0.289=:0.067
(6)
0.213+0.301
0.166+00.23
0.211+00025
observed that our conclusions hold for all datasets, which further verifies the flexibility of our method against the content of X. Without loss of generality, we use FFHQ in our IDA-RD measure.
4.5 MOTIVATION JUSTIFICATION
Invalidity of NR-IQA Metrics As Table. 9 shows, existing NR-IQA metrics, such as NIQE (Mittal et al. 20126) and BRISQUE (Mittal et al. 2012a), are not suitable for the image downscaling can be observed that i) NIQE struggles calculate
problem, especially extreme downscaling. proper scores at all resolutions below 128x 128; ii) BRISQUE does not provide the correct scores at resolution of 32x32.
Please Appendix A.5 for results higher resolutions.
Table 9: NIQE and BRISQUE scores at different resolutions. The test image was randomly selected from the FFHQ dataset and -ewusee-ownneeaeed to different resolutions (LR). Different levels of Gaussian Blur with kernel = 1.0, 2.0, 4.0 were applied as synthetic image downscaling methods.
Resolution
LR
a = 1.0
σ 2.0
4.0
128x 128
18.873
18.872
18.870
18.869
NIQEL
64
18.872
18.872
18.870
18.869
32x32
18.873
18.869
18,870
18.867
128x 128
16.045
34.423
47.017
55.166
BRISQUEJ
64
41.360
42.417
43.346
54.344
32x32
43.458
43.458
44.015
43.668
Invalidity of Non-blind and Non-stochastic SR method As Table shows, non-blind and non- stochastic SR methods like ESRGAN (Wang al. 2018) and SR3 (Saharia al. (2022) fail to distinguish among image downscaling algorithms, which justifies the choice of blind and stochastic SR methods in our IDA-RD.
Bicubic
Bilinear
N.N.
DPID
Perceptual
L0-reg-
ESRGAN
0.02240.012
0.017=0.006
0.008 0.179+0.000 0.0047
0.025-00.009
0.024:0.004 0.172+0.043
0.024:0.007 0.17140.049
SR3
0.16940.048
0.171+0.004
Table 10: Invalidity of using ESRGAN and SR3 in our IDA-RD measure.
CONCLUSION
In this paper, we presented Image Downscaling Assessment by Rate Distortion (IDA-RD), quan- titative measure for the evaluation of image downscaling algorithms. Our measure circumvents the requirement
ground-truth LR image by measuring the distortion the HR space, which is enabled by the recent success of blind and stochastic super-resolution algorithms based on deep generative models. We validate our approach by testing various synthetic downscaling algorithms, simulated by adding degradations, various datasets. We also test downscaling algorithms, which further validates the role of our measure providing new insights into image downscaling algorithms. Please Appendix for Limitation and Future Work.
rorl-world image

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to embed images into the StyleGAN latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN++: How to edit the embedded im- ages? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-8305, 2020.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings the IEEE conference computer vision and pattern recog- nition workshops, pp. 126-135, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223. PMLR, 2017.
Toby Berger. Rate-distortion theory. Wiley Encyclopedia of TeTeecommmiccttinn, 2003.
Sam Raylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. arXiv preprint arXiv:2103.00 044922 2021.
Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojeiech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEFE Trans- actions on Image Processing, 27(1):206-219 2017.
Angela Castillo, María Escobar, Juan C Pérez, Andrés Romero, Radu Timofte, Luc Van Gool, and Pablo Arbelaez. Generalized real-world super-resolution through adversarial robustness. Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 1855-1865, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wop Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2020.
Deng- Deng-Ping Fan, ShengChuan Zhang, Yu-Huan Wu, Yun Liu, Ming- Ming Cheng, Bo Ren, Paul L Rosin, and Rongrong Ji. Scoot: A perceptual metric for facial sketches. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5612-5622, 2019.
Eduardo SL Gastal and Manuel M Oliveira. Spectral remapping for image downscaling. ACM
Transactions on Graphics (TOG), 16(4):1-16, 2017.
Timothy Gerstner, Doug DeCarlo, Marc Alexa, Adam Finkelstein, Yotam Gingold, and Andrew Nealen. Pixelated image abstraction. In NPAR@ Expressive, pp. 29-36, 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- tion Processing Systems, 27, 2014.
Chu Han, Qiang Wen, Shengfeng He, Qianshu Zhu, Yinjie Tan, Guoqiang Han, and Tien-Tsin Wong. Deep unsupervised pixelization. ACM Transactions Graphics (70G), 3766); 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119, 2020.
Tero Karras, Miia Aittala. Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852-8633 863, 2021.
10

---

Under review as a conference paper at ICLR 2023
Thomas A Keller, Jorn WT Peters, Priyank Jaini, Emiel Hoogeboom, Patrick Forré, and Max Welling, Self normalizing flows. In International Conference on Machine Learning, pp. 5378- 5387. PMLR. 2021.
Hailan Kuang, Nan Huang, Shuchang Xu, and Shunpeng Du. A pixel image generation algorithm based cyclegan. In 2021 IEEE 4th Advanced Information Management, Communicates, Elec- fronic and Automation Control Conference (IMCEC), volume 4, pp. 476-480. IEEE, 2021.
Yue Li, Dong Liu, Houqiang Li, Li Li. Zhu Li, and Feng Wu. Learning a convolutional neural network for image compact-resolution. IEEE Transactions on Image Processing. 28(3):1092- 1107, 2018.
Junjie Liu, Shengfeng He, and Rynson WH Lau. Lo-regularized image downscaling. IEEE Trans- actions on Image Processing, 27(3):107-11085, 2017.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In European Conference on Computer Vision, pp. 715- 732. Springer, 2020.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In Proceedings the the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2437-2445, 2020.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assess- ment in the spatial domain. IEEE Transactions on Image Processing, 21((2):4695 2012a.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-213 2012b.
Cengiz Oeztireli and Markus Gross. Perceptually based downscaling of images. ACM Transac- tions Graphics (TOG), 34(4):1-10, 2015.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 91):622-6,, 1979.
George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanann Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 2021.
Jordi Pont-Tuset and Ferran Marques. Measures and meta-measures for the supervised evaluation of image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2131-2138, 2013.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint (tt::/5/1..443 2015.
Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows Interna- fional Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Paul Rosin, Yu-Kun Lai, David Mould, Ran Yi, Itamar Berger, Lars Doyle, Seungyong Lee, Chuan Li, Yong- Jin Liu, Amir Semmo, Ariel Shamir, Minjung Son, and Holger Winnemöller. NPRpor- trait L.O: three-level benchmark for non-photorealistic rendering portraits. Computational Visual Media, 8(3):445-455 465, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yunyi Shang and Hon-Cheng Wong. Automatic portrait image pixelization. Computers & &Gaphics, 95:47-59, 2021.
Jin Sun and Haibin Ling. Scale and object aware image thumbnailing. International Journal of Computer Vision, 104(2):33--153, 2013.

---

Under review as a conference paper at ICLR 2023
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Interaational Conference on Computer Vision Work- shops (ICCVW).
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 0-0, 2018.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality Asilomar Conference on Signals, Systems Computers, 2003,
assessment. In The volume 2. pp. 398-14022 IEEE, 2003.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 13(4):600- 612, 2004.
Nicolas Weber, Michael Waechter, Sandra C Amend, Stefan Guthe, and Michael Goesele. Rapid, detail-preserving image downscaling. ACM Transactions on Graphics (TOG), 35(6): 2016.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Kai Zhang. Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In IEEE International Conference on Computer 2021.
Vision, pp.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. SEAN: Image synthesis with semantic region- adaative normalization. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5104-5113, 2020.
A APPENDIX
A.1 EXAMPLES OF DOWNSCALED IMAGES USED IN OUR EXPERIMENTS
Table II and Table 12 show examples of images downscaled by synthetic and real-world image downscaling methods used in our experiments, respectively.
A.2 BALANCING FFHQ INTO AGE-, GENDER-, AND RACE- -ALLACED SUBSETS
We balance the FFHQ dataset Karras et al. (2019) into subsets (i.e., X in Eq. 2 that are balanced in age, gender and ethnicity for a fair evaluation of our IDA-RD measure. For the gender and age labels of FFHQ images, we use those offered by the FrHO-feature--datsse
for the ethnicity labels of FFHQ images, we use the recognition results of DeepFace According to the above, we define i) four age groups: Minors (0- 18), Youth (19-36), Middle Aged (36-54) and Seniors (54+); ii) three major ethnic groups: Asian, White and Black; iii) two gender groups: Male and Female. We apply K-means to cluster FFHQ images in 24 (4 x x2) groups and select images from them evenly to generate the subsets used in our experiments. As Table 13 shows, the subsets used in our experiments are highly-balancee in terms of age, gender and ethnicity.
A.3 ADDITIONAL ABLATION STUDY ON D THE DISTORTION MEASURE
As complement to Table 6 in the main paper, Table 14] shows additional results for the ablation
study of D, which further justifies our choice of LPIPS as the distortion measure in our IDA-RD.
http::///tthab.cco////////ffffff dataset
12

---

Under review as a conference paper at ICLR 2023
Table 11: Examples of images downscaled by synthetic image downscaling methods, i.e. those adds controllable degradations to srrubic-dwnssclled images (Sec. 4.2). The numbers below images are the degradation parameters. LR: bicubic-downscaled images, Dec.: decrease, Inc.: increase, Gauss.: Gaussian.
Guass, Blur
LR
0 0= 1.0
0 = 2.0
4.0
Contrast Dec.
LR
0.75
0.5 Contrast Inc.
0,25
LR
1.5 Gauss.
2.0
2.5
Noise
LR
0.05 Quantization
0.1
0.2
LR
10
Mixed Degradations
LR
+Contrast Dec, +Gauss. Noise +Quantization +Gauss. Blur
A.4 TIME COMPLEXITY
Table 15 shows the running times of our IDA-RD measure using PULSE and SRFlow fue (Eq. 2) on an Nvidia RTX3090 GPU, respectively. It can be observed that the SRFlow implementation runs much faster, which justifies choice of using it in our IDA-RD measure.
13

---

Under review as a conference paper at ICLR 2023
Table 12: Examples of images downscaled by real-world image downscaling methods. N.N.: Near- est Neighbour; LO-reg.: LO-regularized.
Bicubic
Bilinear
N.N.
DPID
Perceptual
LO-reg.
Table 13: Statistics of our balanced FFHQ subsets. MI: Minors, Youth, MA: Middle Aged, S: Senior; A: Asian, W: White, B: Black; M: Male, F: Female. J.E.: Joint Entropy, which measures the extent to which a subset is balanced. As reference, tully-balanced subset has joint entropy of -24 (1/24) loga( (1/521122) (1/24) x 4.5850.
Size
Age
Ethnicity
Gender
J.E.
MI
MA
A
W
M
30
9
7
10
10
15
15
4.2817
300
76
75
70
79
102
100
98
150
150
4.4998
600
168
142
141
149
200
194
206
329
271
4.5245
900
222
227
215
236
304
295
301
452
448
4.5343
1200
445
442
453
460
608
591
601
902
898
4.5375
1500
684
664
673
679
909
887
904
1352
1348
4.5386
A.5 ADDITIONAL RESULTS WITH NIQE AND BRISQUE
As complement to Table 9 in the main paper. Table 16a and Table 16b
show additional results of
NIQE (Mittal et al. 2012b and BRISQUE (Mittal et al. 2012a) at higher resolutions where the two scores work better.
A.6 LIMITATION AND FUTURE WORK
Limitations. Since our measure makes use of GAN- and Flow-based super-resolution (SR) models, the limitations of these models are carried over as well. First of all, we cannot use test data beyond the learnt distribution of the SR model. For example, unlike the SRFlow (Lugmayr et al. 2020) model trained on general images that are used in the main paper, our GAN-based implementation uses StyleGAN generator pre-trained on portrait images, which only allows for the use of portrait face images to evaluate downscaling algorithms. Also, although highly unlikely to occur, we cannot
Table 14: Ablation study of D, the distortion measure Eq. 2 Dec.: Decrease. Param.: Parameter.
Param.
PSNR
SSIM
MS-SSIM
LPIPS
0.75
22.13714.020
0.83411..19
0.88110.101
0.330=0.047
Contrast Dec.
0.50
17.81442.148
0.714:0.087
0.819-10.000
0.644-0.074
0.25
14.790::1.461
0.578+0.072
0.57910.028
0.669-0.034
1.50
14.019
0..03-
0.772:10.150
0.041
Contrast Inc.
2,00
13.450+3.539
0.482-+0.192
0.693+0.131
0.317=0.041
2.50
11.032-42.993
0.357
0.602+0.120
0.462-0.043
0.05
10.78410.160
0.59740.004
6.6071
0.482
Gaussian Noise
0.10
18.12111.713
0.563+0.029
0.57610.006
0,640-0.052
0.20
1..120+1.751
0.500
0.376-10.66
0.659
1.00
25.19911.999
0.0099
0.92910.017
0.320
Gaussian Blur
2.00
22.865
0.646-
0.849-0.033
0.057
4.00
19.738
0.558+0.080
0.71540.051
0.579=0.065
14

---

Under review as a conference paper at ICLR 2023
Table 15: Running times of our IDA-RD with PULSE and SRFlow fus (Eq. respectively. Nx: the number of images in test dataset X in Eq.
Nx
PULSE
SRFlow
300
3h08min
18min
600
6h10min
35min
900
9h08min
55min
Table 16: Additional results of NIQE and BRISQUE at higher resolutions (lower is better).
Resolution
LR
0
1.0
o
2.0
= 4.0
x024x 1024
3.700
4.158
5.173
6.471
512x512
2.406
3.959
5.574
6.299
256x256
3.047
4.611
7.133
6.792
128x 128
18.873
18.872
18.870
18.869
64x64
18.872
18.872
18.870
18.869
32x32
18.873
18,869
18.870
18.867
(a) NIQE scores
Resolution
LR
0 1.0
0 2.0
4.0
x 1024
26.792
32.827
48.971
59.043
512x512
19.536
33.391
57.447
63.144
256x256
28.582
39,282
55.747
65.990
128x 128
16.045
34.423
47.017
55.166
64x64
41.360
42.417
43.346
54.344
32x32
43.458
43.458
44.015
43.668
(b) BRISQUE scores
evaluate downscaling algorithms whose output images are of higher quality than those generated by SR model (i.e,, no distortion).
Future work. Our framework still requires ground truth HR image. However, believe the distortion can be calculated without such ground truth image. To further validate our IDA-RD measure, in the future we will use the meta-measure methodology (Pont-Tuset & Marques 2013 Fan et al.) 2019), in which secondary, easily quantifiable measures are constructed to quantify the performance of less easily quantifiable measure.
A.7 ABLATION STUDY OF Nx FOR IDA-RO IMPLEMENTED WITH PULSE
As Table 17 shows, we also investigate how many images are required in the test dataset X con- sisting high-resolution images to achieve robust estimation of IDA-RD implemented with PULSE (Menon et al. 2020). Similar to those in the main paper, can be observed that the sults become stable when Nx 2 900, which further justifies our choice Nx = 900 for IDA-RD.
Table 17: Ablation study of Nx for IDA-RD implemented with PULSE. Synthetic image down- scaling methods with Contrast Decrease with 0.75 (DGI); Gaussian Noise with 0.05 (DG2); mixed noise consisting of Gaussian Blur with o 1.0, Contrast Decrease with o = 0.75, and Gaussian Noise with 0.05 (DG3); are used in the experiments.
Nx
30
300
600
900
1200
1500
DGI 0.35110.014
0.34210.019
0.34310.012
0.33910.022
0.33910.021
0.339110.02
DG2
0.361+0.011
0.383+00011
0.374+0.012
0.351+0.023
0.353+0.002
0.352+0.021
DG3
0.47110.011
0.48311.012
0.39110.013
0.29310.019
0.289-0.022
0.29110.021
15

---

Under review as a conference paper at ICLR 2023
A.8
EXPERIMENTAL RESULTS ON REAL-WORLD DATASETS
Table 18 shows our IDA-RD scores on two real-world datasets: DIV2K (Agustsson & Timotte
2017 and Flickr30k (Young et al. 2014). can be observed that our conclusions still hold (N.N. > Perceptual > LO-regularized > DPID > Bicubic >> Bilinear), which further justifies the validity of the proposed IDA-RD measure. Note that both experiments are conducted with scaling factor of 4x as we observed SRFlow become unstable for higher scaling factors (Fig. 2 For stable uses of SRFlow, we intentionally used domain- art image downscaling methods (i.e., Perceptual, L0-regularized, DPID) used in our experiments are general ones that are applicable to all domains (í.e. not tuned for specific domains).
datasets in the main paper. Note that all state-of-the-
Table 18: IDA-RD scores on two real-world datasets: DIV2K (Agustsson Timofte 2017) and Flickr30k (Young et al. 2014). UD: "unknown downscaled" images provided by DIV2K. A scaling factor of 4x is used for both datasets as we observed higher scaling factors makes SRFlow unstable.
Ricuhic
Bilinear
N.N.
DPID>
Perceptual
L0-reg.
UD 0.35520.128
DIV2K 0.157=0.073
0.129:0.089 0.239+0.112
0.37420.079
0.216:E0.057
0.226:10002 0.33620:068 0.367=0.080
Flickr30K
0.263=0.102
0.452=0.105
0.357+00.97
0.364=0.103
Figure 2: SRFlow becomes unstable for scaling factor of 8x on real-world datasets, e,g., DIV2K (Row 1), while such cases never happen for domain-speciff datasets, e.g., FFHQ (Row 2).
(a) N.N.
(b) DPID
(c) Perceptual
(d) LO-reg.
(c) N.N.
(f) DPID
(g) Perceptual
(b) LO-reg.
A.9 ADDITIONAL ABLATION STUDY OF fas
As Table 19 shows, we tested our IDA-RD measure with some other choices of state-of-the-art SR methods: BSRGAN (Zhang et al. 202T), RSR (Castillo et al.) 202T) and Real-ESRGAN (Wang et al.). However, all these methods are blind but non-stochastic (Sec, 4.5p. which do not satisfy the requirement of our IDA-RD measure and generate less informative results. Specifically, the results of BSRGAN and Real-ESRGAN are less distinguishable among different downscaling methods; the results of RSR are slightly better but still not comparable to SRFlow.
A. QUALITATIVE EVALUATION OF EXISTING DOWNSCALING METHODS
As Fig. 3 shows, ofate-of-the-art image downscaling methods achieve better perceptual quality by "exaggerating" perceptually important features in the original image (e.g., building lights, water
16

---

Under review as a conference paper at ICLR 2023
Table 19: Additional ablation study of fas. the image upscaling algorithms. Following Sec. A.8 we
use the DIV2K dataset and scaling factor of 4x. BSRGAN (Zhang et al. 2021). RSR (Castillo al., 2 and Real-ESRGAN (Wang et al.) are blind but non-stochastic SR methods (Sec. 4.5 which do not satisfy the requirement of our IDA-RD measure and generate less informative results.
Bicubic
Bilinear
N.N.
DPID
Perceptual
Z0-reg.
BSRGAN
0.000
0.011 0.008
0.02440.022 0.423:10.132 0.026:t0.022
0.013+0.011
0.025=000188
0.000
RSR
0.231=0.071
0.208=0.095 0.015=0.011
0.379=0.123 0.288=10099 0.016:10.012
0,231+00071 0.026=0.017
Real-ESRGAN
0.014:1.0110
0.017+:0013
reflections), thus leading over-exaggeratio in the upscaled images. As result, they have lower IDA-RD scores than bicubic and bilinear downscaling.
Orighnal
212
Orginal
SR1
S12
S13
(a) Bicubic
(h) Bilinear
Onginal
$8.1
2
$1.5
Orginal
S8.1
$1.3
MD2
MD
2
0.3830
(c) N.N.
(d) DPID
Orgital
MB3
MD22
MD
L.2
140
MD2
MD
0.377
3357
3.3342
2.2012
522113
5
22792
(e) Perceptual
(1) LO-reg.
Figure 3: Qualitative evaluation of existing image downscaling methods. Original: the input HR
image; LR: the downscaled LR image; SR1, SR2, SR3: three instances of upscaled images: MD1, MD2, MD3: difference map visualizations of (SR1, Original), (SR2, Original), and (SR3, Original), respectively. The white numbers on the left-top corners: the corresponding LPIPS scores of the difference map visualizations. State-of-the-an image downscaling methods (DPID, Perceptual and LO-reg.) achieve better perceptual quality by "exaggerating" perceptually important features in the original image (e.g., building lights, water reflections), thus leading to over-exaggeration in the upscaled images and lower IDA-RD scores.
17


Reviews:
[{"review_id": "mF6Crb_beQK", "reviewer": "Reviewer_mBvw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to measure image downscaling algorithms. The core idea is to upsample the downsampled image using blind/one-to-many generative-model-based super-resolution methods and compare the upsampled image with the original image. The similarity will be used as the final result. The authors show results on some synthetic data and compare different methods. ", "summary_of_the_review": "The main problem with the paper is the writing, and imperfect experimental proof of the performance and significance. ", "strength_and_weaknesses": "Weaknesses:\n1. The statement of Eq (2) is very confusing. $Q$ denotes conditional distribution. The distribution of what? Of the lossy image reconstruction function $f$? Is the $Q$ can be written by $Q(f|f_{ds})$? Or should it be $Q(x’|x)$ according to the $P(z)\\to Q(x’|x)$. I don’t know what the random variable is and I don’t know why there is a distribution. From any aspect, Eq (2) seems to make no sense. Moreover, is $x’$ follows a distribution? How can D measure between $x$ and $x’$? Is D a metric of distributions?\n2. The many-to-many part is also very confusing. I know that the generative model generates a distribution or something, but it is not one-to-many. For everything that is generated by the model, we need a latent code. With the latent code, it is not one-to-many. And I don’t think we need a many-to-many statement here.\n3. The above two questions make Eq (4) even more confusing.\n4. In Eq (5), what is $z*$? And the mathematical is not presented in the right way.\n5. For the experiments, There is no conclusive evidence proving that the proposed method is consistent with the human perception results. It is not clear whether the proposed method is linear in evaluation, or whether it is order-preserving. Experiments on synthetic images can provide limited evidence. However, due to the very large degradation of the synthetic images, it is difficult to make such a big difference when evaluating the downsampling method. So these experiments are not enough to provide evidence that their methods are consistent with human perception results.\n6. Are there any other applications of the proposed method? For example as a clue to know new downsampling methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZdxY5BbQ9Wu", "reviewer": "Reviewer_o28g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a measure, namely Image Downscaling Assessment by Rate-Distortion (IDA-RD), to quantitatively evaluate image downscaling algorithms.", "summary_of_the_review": "The authors use traditional downscaling algorithms such as bicubic, bilinear, and nearest neighbor interpolation, as well as state-of-the-art downscaling algorithms to test the proposed measure. However, the reviewer does not recommend accepting the current version, because of the concerns in \"Weaknesses\" above.\n\n----- After responses -----\n\nThe authors addressed some of my concerns.", "strength_and_weaknesses": "The authors use traditional downscaling algorithms such as bicubic, bilinear, and nearest neighbor interpolation, as well as state-of-the-art downscaling algorithms to test the proposed measure. However, the reviewer does not recommend accepting the current version, because of the following concerns:\n\n1. There are plenty of datasets for real-world super-resolution, i.e., the datasets contain the paired LR and HR images. Therefore, the performance of downsampling can be measured on these datasets by calculating the PSNR/LPIPS between the downsampled images and the original LR images. In comparison with this, what is the benefit of the proposed method?\n\n2. The IDA-RD scores for DPID, Perceptual and L0-reg. are not able to indicate anything, since there is no evidence showing that the predicted score of the proposed measure is correct or not on these methods.\n\n3. What is the relationship between the proposed measure and \"rate-distortion\"?\n\n4. Why use StyleGAN on PULSE, but SRFlow on other datasets? It doesn't make sense to change the network for different datasets.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XYor5P9AFas", "reviewer": "Reviewer_CnsW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new metric, namely IDA-RD, to evaluate image downscaling algorithms quantitatively. To this end, the authors employ the idea that a downscaling algorithm that preserves more details in the resulting low-resolution images should lead to less distorted high-resolution images constructed by state-of-the-art super-resolution methods. Experimental comparisons are made over various downscaling operations with different levels of distortions to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "While the work presents some interesting ideas, many comparisons and experiments are either missing or are not done fairly. The overall evaluation of the method seems to lack multiple aspects, which makes the paper below the acceptance threshold. \n\nComments after Author Response:\nBased on the author's response I am changing my rating from Reject to Accept.", "strength_and_weaknesses": "Strengths\n \n1. The paper addresses an interesting problem of measuring the downscaling methods\n2. The paper does contribute some new ideas to tackle the problem at hand\n3. The proposed methods outperform some of the other existing metrics on domain-specific image datasets\n\nWeaknesses\n\n1. Missing comparisons: There are some other upscaling super-resolution (SR) functions that are likely to handle the unseen degradations well, for example, methods in [p1-p3]. The experiments should be conducted with these methods as well.\n\n[p1] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution, ICCV 2021\n[p2] Real-esrgan: Training real-world blind super-resolution with pure synthetic data, ICCVW 2021\n[p3] Generalized Real-World Super-Resolution Through Adversarial Robustness, ICCVW 2021\n\n\n2. Conclusions from little pieces of evidence: (Refer: Section 4.3) Based on the results, authors argue that SOTA downscaling methods (Weber et al., 2016; Liu et al., 2017; Oeztireli & Gross, 2015) actually lose more information than Bilinear downscaling. However, this conclusion comes from the assumption that the proposed method is optimal for measuring the distortions of any arbitrary downscaling operation. This assumption is unlikely to be true and needs further evaluation, analysis, and comparison before coming to a conclusion. To this end, I would recommend adding “comparison: with mean opinion score (i..e, feedback from humans based on perceptual understanding)”, “analysis: such as picking some examples that best reveal the distortion effects and then arguing why the SOTA methods perform poorly in comparison to bilinear”, and  “evaluation: ensure fair evaluation since the SOTA methods might have fine-tuned for say natural images or specific types of images or distortions”. Altogether we need much more experimental evidence in support of this kind of argument that actually contradicts the common beliefs in the literature. Moreover, the experiments are done only with domain-specific images, which also makes adds to the fact that the proposed method is yet an “incomplete metric”.\n\n3. Clarity on the choices of methods: \n- Regarding the GAN generator-based method: Based on Eq. 5, the proposed method employs the downscaling method that we wish to evaluate to generate the LR image for training the super-resolution (SR) method. Such training can be done with any fully-supervised SR method. Henceforth, it is not clear why other SR methods would fail to do the same job or what the limitations are in using other methods. There should also be experiments to support such arguments/failure cases. There seem to be some comparisons under “Invalidity of Non-blind and Non-stochastic SR method” it is not clear if the experiments were done with pre-trained networks or the ones trained with “f_ds” as in Eq. 5.\n- Regarding the SRflow method: This method employs bicubic downscaling for training the SR network is more attractive since we can avoid retraining the SR network to evaluate any new downscaling method. However, as I have mentioned before, a more “generalizable to unseen degradations SR network” could be [p1-p3] and should therefore be included in experiments and comparisons. \n\n4. Why only domain-specific datasets in experiments and comparisons?\n\nThe authors seem to have used only the face images for all the comparisons. Why are the natural images not included in the study? Or is it like the authors were trying to tackle metrics for certain domain-specific images alone?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "78BvRFqsX9", "reviewer": "Reviewer_SxNM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is interesting. It gives an indepth analysis of down-scaling methods and degradations for gan sr networks. It uses rate distortion for evalutions and gives some interesting results. ", "summary_of_the_review": "This work worths published given its in depth analysis of down-scaling algorthms using rate distortion. Such results can be useful for scalable video coding in the future codec designs. ", "strength_and_weaknesses": "Strenght:\n1. Rate distortion for evluating down-scaler is interesting.\n2. Comprehsive analysis of different down-scalers and different kinds of degradations.\n\nWeaknesses\n1. Only SRflow and Pulse sr networks are tested.\n2. No realworld datasets are tested. only sythesis datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject