Title: Neural Image Compression: Generalization, Robustness, and Spectral Biases
Venue: NeurIPS 2023
Accepted: yes
Keywords: image compression

robustness

generalization
Abstract: Recent advances in neural image compression (NIC) have produced models that are starting to outperform classic codecs. While this has led to growing excitement about using NIC in real-world applications, the successful adoption of any machine learning system in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand NIC performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we provide CLIC-C and Kodak-C by introducing 15 corruptions to the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspired inspection tools to gain deeper insight into errors introduced by image compression methods as well as their OOD performance. We then carry out a detailed performance comparison of several classic codecs and NIC variants, revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC and its dependence on the spectral properties of the data. Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing robust and generalizable NIC methods. Code and data will be made available at https://github.com/klieberman/ood_nic.

Neural Image Compression: Generalization, Robustness, and Spectral Biases

Abstract

Recent advances in neural image compression (NIC) have produced models that are starting to outperform classic codecs. While this has led to growing excitement about using NIC in real-world applications, the successful adoption of any machine learning system in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand NIC performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we provide CLIC-C and Kodak-C by introducing 15 corruptions to the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspired inspection tools to gain deeper insight into errors introduced by image compression methods as well as their OOD performance. We then carry out a detailed performance comparison of several classic codecs and NIC variants, revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC and its dependence on the spectral properties of the data. Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing robust and generalizable NIC methods. Code and data will be made available at https://github.com/klieberman/ood_nic .

1 Introduction

Consider the Mars Exploration Rover, whose scientific objective is to search for clues to past activity of water (and perhaps life) on Mars. To achieve this, the rover collects images of interesting rocks and soils to be analyzed by the scientists on Earth. Sending these images down the Earth-bound data stream in their original form is too slow and expensive due to limited bandwidth. Thus, it is well accepted that image compression could play a key role in producing scientific breakthroughs [44].

Employing image compression in such a setting is challenging for three main reasons: 1) a high compression ratio is desired due to low communication bandwidth, 2) given the battery-operated nature of these devices, the compression module has to be lightweight so it consumes less memory and power, and 3) robustness and generalization to environmental noises and domain shifts, respectively, are desired due to limited Mars-specific training data. These requirements are not specific to the planetary exploration use case but also arise in a wide range of scientific applications which use image compression in the wild [33].

Recently, neural image compression (NIC) has demonstrated remarkable performance in terms of rate-distortion and runtime overhead on in-distribution (IND) data [9, 46]-satisfying requirements 1) and 2). However, there is limited work on understanding the out-of-distribution (OOD) robustness and generalization performance of image compression methods (requirement 3) [42]. Our work is driven by several open fundamental empirical and theoretical questions around this crucial issue.

How can the expected OOD performance of image compression models be reliably assessed? Can we gain a deeper understanding of the modus operandi of different image compression methods? How do training data properties and biases impact data-driven compression methods?

Main Contributions: This paper takes a critical view of the state of image compression and makes several contributions toward answering the aforementioned questions. ❶ First, we design comprehensive benchmark datasets for evaluating the OOD performance of image compression methods. Inspired by existing OOD benchmarks for classification and detection [26, 28, 57, 56], we design CLIC-C and Kodak-C by introducing 15 common shifts emulating train-deployment distribution mismatch to the popular CLIC and Kodak datasets. ❷ Next, we focus on understanding the image compression performance. The de-facto approach is to use rate-distortion (RD) curves measured with perceptual quality metrics, such as PSNR. Such scalar metrics, although easy to compute, are known to be extremely limited in what they can capture and sometimes can even be misleading [63, 61]. To complement RD curves, we propose spectrally-inspired inspection tools that provide a more nuanced picture of (a) compression error, and (b) OOD performance of a given method. Specifically, we introduce a power spectral density (PSD) based approach to understand the reconstruction error. Our approach not only quantifies how much error was made but also highlights precisely where it was made (in the frequency domain). Similarly, to understand the OOD performance of a compression method in unseen deployment scenarios, we propose Fourier error heatmaps -avisualization tool for highlighting the sensitivity of the reconstruction performance of a compression method to different perturbations in the frequency domain. ❸ Using our benchmark datasets and inspection tools, we carry out a systematic empirical comparison of classic codecs (i.e., JPEG2000, JPEG, and VTM) with various NIC models (fixed rate, variable rate, and pruned versions of the scale hyperprior model [9], as well as efficient learned image compression (ELIC) [24]). ❹ Finally, we develop theoretical tools to connect NIC OOD performance with its training data properties.

Main Findings: Our analysis resulted in some invaluable insights about the state of image compression. We summarize some of our findings below.

Compression methods yielding the same PSNR (or bpp) can produce very different spectral artifacts. Our tools help uncover hidden spectral biases and highlight the limitations of de-facto RD curve-based performance comparison.

As the compression rate increases, different codecs prioritize different parts of the frequency spectrum. By precisely characterizing this behavior, our tools help in advancing the current understanding of the modus operandi of image compression methods.

Image compression models generalize to low- and mid-frequency shifts better than high-frequency shifts. This finding calibrates our expectations for image compression performance on OOD data.

NIC models are better at denoising high-frequency corruptions than classic codecs. This finding reveals that unlike classic codecs, NIC models have an inherent spectral bias due to the image frequencies present in their training data.

Identifying the most suitable compression method becomes exceptionally challenging without the knowledge of spectral characteristics of OOD shifts. Our systematic evaluation identifies this open issue with current compression methods and suggests the design of next-generation NIC models that can adapt themselves at runtime based on the spectral nature of the data to be a potentially worthwhile direction to pursue in the future.

(a)

Low

Snow, Frost, Fog,

Brightness, Contrast

Med

Motion blur, Zoom blur,

Defocus blur, Glass blur,

Elastic transform, Pixelate,

JPEG compression

High

Gaussian noise, Shot noise,

Impulse noise

(b)

Original

Snow

ShotNoise

700

Q

We corroborate our findings with a detailed theoretical analysis, showing that multiple overarching trends in our experimental results can be attributed to neural compression models' spectral bias.

Appendix A has a detailed related work discussion, while all references are listed in the main paper.

2 Out-of-distribution image compression datasets

To evaluate NIC in the presence of environmental or digital distribution shifts, we generated variants of the CLIC and Kodak datasets, which we refer to as CLIC-C and Kodak-C. Following the techniques presented in [26] for studying the performance of DNN classifiers encountering distributional shifts 'in the wild", our -C datasets consist of images augmented by 15 common corruptions. For each image in the original dataset, the -C dataset contains a corrupted version of the image for each of the 15 common corruptions 2 , and for each of five corruption severity levels, with 1 being the lowest severity and 5 being the highest. A sample of some corruptions on CLIC-C is provided in Figure 1a.

While each -C dataset offers a broad sampling of environmental or digital image corruptions, it also provides a spectrally diverse collection of corruptions, in the sense that each corruption can be categorized as low, medium, or high frequency based on the frequency content used for perturbations. We will write PSD ( · ) to denote the function that converts the input image from the spatial to the frequency domain by computing the power spectral density of the input. Practically, computing PSD ( · ) is done by applying the fast Fourier transform (FFT) [11], followed by a shift operation to center the zero-frequency component, then taking the absolute value. Now suppose we have a set X = { X k } N k =1 of uncorrupted images and some corruption function c ( · ) (e.g., frost, gaussian noise, etc.). We analyze the spectrum each corruption c ( · ) by computing 1 N ∑ N i =1 PSD ( X i -c ( X i )) (see Figure 1a). We classify the corruptions into types of frequencies: low, medium, and high using the characterization from [39], which has been adopted by the neural network robustness community. Note that categorizing a corruption as high is more akin to saying it 'contains substantial high-frequency content' rather than saying it 'exclusively consists of high-frequency content.'

3 Spectral inspection tools

While existing scalar metrics, such as PSNR, are able to summarize the visual similarity of reconstructed images to the original, we will demonstrate that such metrics can provide an incomplete (and sometimes misleading) picture when measuring the impact of compression in OOD settings. Notably, existing tools do not consider the impact of compression on different frequency ranges of images within a dataset. To more thoroughly analyze the effects of image compression, we propose to measure and visualize the effect of image compression in the spectral domain. Given an image compression model C that returns reconstructed images, we introduce tools for analyzing compression

error in the Fourier domain to better understand ( i ) which spectral frequencies are distorted by C , ( ii ) the OOD generalization error, and ( iii ) the robustness error in the presence of distributional shifts.

Definition 3.1 (Spectral Measure of Distortion Error) . To analyze ( i ), we evaluate the image compression model C 's ability to reconstruct components of an image across a range of frequencies. To quantify this, we compute the average PSD of the difference between each image X k in a dataset X and the reconstructed version C ( X k ) of X k : D ( C , X ) := 1 N ∑ N k =1 PSD ( X k -C ( X k )) .

Definition 3.2 (Spectral Measure of OOD Generalization Error) . For ( ii ), we evaluate C 's ability to faithfully reconstruct OOD images. To quantify this, we extend the metric D ( C , X ) to account for a corrupted version c ( X ) of X as follows: G ( C , X , c ) := 1 N ∑ N k =1 PSD ( c ( X k ) -C ( c ( X k ))) .

Definition 3.3 (Spectral Measure of OOD Robustness Error) . For ( iii ), we evaluate C 's denoising ability. To quantify this, we compute the average PSD of the difference between each uncorrupted image X k and the reconstructed version C ( c ( X k )) of the corresponding corrupted image c ( X k ) : R ( C , X , c ) := 1 N ∑ N k =1 PSD ( X k -C ( c ( X k ))) .

The PSD function used in the above formulas converts the input image from the spatial to frequency domain by applying the Fast Fourier Transform (FFT) followed by a shift operation to center the zero-frequency component. The proposed tools are computed for a set of images by averaging the power spectral density (PSD) of the difference between the compressed image and the original for each image in the set of images. For simplicity, when ( C , X , c ) is clear from the context, we will just write D , G , or R . Note that G provides insight into the compression model C 's ability to generalize to a distribution shift c while R visualizes the denoising effect (or lack thereof) of C across the frequency domain.

In Appendix B, we present results using an additional tool, the Fourier heatmap , that utilizes Fourier basis perturbations as corruptions and is used to corroborate our findings for the specific -C datasets and corruptions we consider. This tool can be leveraged when specific OOD data is unavailable.

4 Experiments and findings

Using our spectral inspection tools and OOD datasets, we analyze the performance of several image compression methods and identify several inter- and intra-class differences between classic codecs and NIC methods. We show results for two classic codecs, JPEG and JPEG2000, and two NIC models, SH NIC and ELIC, in the main body and report the remaining results in Appendices C and D.

Classic codecs. We apply each of the following algorithms over several compression rates q .

JPEG [60]

JPEG2000 [53]

Versatile Video Coding (VVC) (equivalently h.266) using the VVC Test Model (VTM) software [30, 3]

Neural Image Compressors (NIC) Models. NIC optimization uses a hyperparameter λ to control the relative weight of distortion (quality of reconstruction) and rate (level of compression) terms in the objective function. We train each NIC model over several values of λ . All models were optimized on the train split of the 2020 CLIC dataset [59]. Further details on their model architectures and training can be found in Appendix I.

SH NIC : Scale-hyperprior model from Ballé et al. optimized for PSNR ( i.e., distortion objective is mean squared error) [9].

ELIC : Efficient Learned Image Compression optimized for PSNR [24].

Variant of SH NIC optimized for MS-SSIM [9].

Variable-rate version of SH NIC optimized using Loss Conditional Training [17].

Variable-rate model above with 80-95% of weights pruned using gradual magnitude pruning [70].

Evaluation setup. We compare distortion, robustness, and generalization error of different image compression methods under three constraints: (a) no constraint , (b) fixed-bpp , and (c) fixed-PSNR . In (a), we compare methods over their full range of rate-distortion tradeoffs by generating ratedistortion curves. In (b), we compare models with hyper-parameters which give a very similar bits per pixel (bpp) result on a particular dataset. For example, we find that on the CLIC dataset, SH NIC with

D(C, x): Fixed-bpp

RD Curves

D(C,x):Fixed-psnr

lowbpp

med.bpp

high bpp

42

med. psnr

high psnr

SH NIC

SH NIC

40

(0.25, 31.54)

(0.77, 36.85)

(1.21, 38.98)

38

(0.25, 31.54)

(0.37, 33.17)

(0.77, 36.85)

ELIC

high

ELIC

PSNR (dB)

36

(0.25, 33.23)

(0.75, 37.97)

(1.27, 41.26)

34

(0.16, 31.32)

(0.25, 33.23)

(0.53, 36.69)

JPEG2000

med

JPEG2000

(0.24, 30.37)

(0.80, 35.34)

(1.20, 37.31)

30

SH NIC

(0.34, 31.72)

(0.48, 33.09)

(1.09, 36.83)

JPEG

ELIC

JPEG

JPEG2000

28

JPEG

(0.22, 24.57)

(0.76, 32.88)

(1.22, 35.49)

0.0

low

0.5

med.

1.0

high

1.5

2.0

(0.59, 31.57)

(0.81, 33.23)

(1.56,36.82)

0

600

15 0

6

Image size (bpp)

0

400

300

λ = 0 . 15 , ELIC with λ = 0 . 15 , JPEG2000 with q = 10 , and JPEG with q = 75 all give a bpp close to 1.21. Thus, comparing these three models with those hyper-parameters on CLIC under a fixed-bpp constraint, emulates a setting in which a fixed budget is available to store images . Analogously, in (c) we compare models with hyper-parameters yielding a fixed PSNR. This allows the comparison of spectral properties of data distributions that achieve the same quality according to the scalar PSNR metric . Scenarios (b) and (c) are used when evaluating D , G , R , Fourier heatmaps, and accuracy on a downstream task.

Test data. All models are tested on (a) in-distribution (IND) and (b) corrupted (or OOD) datasets. For (a), we use the 2020 CLIC test split, the full Kodak dataset, and the ImageNet validation split. For (b), we use the corresponding -C datasets for each of the datasets in (a). The main body contains results for the CLIC/CLIC-C dataset. Analogous results for Kodak are in Appendix G.

4.1 Evaluating spectral distortion on IND data

On in-distribution (IND) data, the existing RD curve metrics in the center of Figure 2 highlight the established trend that these NIC models outperform the JPEG2000 and JPEG across the compression rates that the NIC model is trained on (bpp ∈ [0 . 1 , 1 . 5] ), with ELIC outperforming SH NIC and JPEG2000 outperforming JPEG.

Next, we use our spectral inspection tool D to better understand the effects of different image compression methods. Specifically, Figure 2 shows plots of D under three fixed-bpp and three fixed-PSNR scenarios on the clean CLIC dataset. We highlight some surprising insights below.

Two methods yielding the same PSNR can produce very different spectral artifacts. Under the fixed-PSNR constraint (right side of Figure 2), each column consists of methods with hyperparameters selected to give very similar PSNRs on the CLIC test set ( e.g. , models on the 'high psnr' column all have PSNR ≈ 36 . 8 ). Despite having comparable PSNRs, the plots of D vary greatly between the four models. In particular, the SH NIC models distort high frequencies more than medium frequencies (notice the warmer-colored rings around the edges of the D plots with cooler-colored centers). JPEG2000, on the other hand, distorts low and medium frequencies more than high frequencies (notice the large rectangles of warmer colors). ELIC distorts all frequencies relatively evenly (the color gradients in the plots are very narrow). 3 JPEG severely distorts one ring, but this ring includes lower frequencies (has a smaller radius) than the rings left by SH NIC. This same pattern holds under the fixed-bpp constraint (left side of Figure 2). These observations demonstrate that PSNR is not a comprehensive metric .

Lowfreq.:Snow

Med.freq.:Glass Blur

High freq.:Shot Noise

40.0

24 -

SHNIC

37.5

Rw.r.t. c(x) (dB)

45

PSNR w.r.t. c(x) (dB)

ELIC

JPEG2000

35.0

32.5

40

18

JPEG

27.5

30.0

ELIC

SH NIC

PSNR

35

SH NIC

ELIC

16

14

25.0

JPEG2000

30

JPEG2000

12

JPEG

JPEG

22.5 -

10

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

0

1

2

3

4

5

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

Lowfreq.:Snow

Med.freq.:Glass Blur

High freq.:Shot Noise

15

SHNIC

26

SH NIC

27.5

SH NIC

ELIC

ELIC

ELIC

PSNR w.r.t. x (dB)

JPEG2000

w.r.t. x (dB)

25

JPEG2000

JPEG2000

JPEG

JPEG

JPEG

24

20.0

11

PSNR

23

PSNR

17.5 -

10

22

15.0

9

12.5

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

0

1

2

3

4

5

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

As the compression rate increases, different codecs prioritize different parts of the spectrum. On the left side of Figure 2, each column represents a different 'budget' scenario where the three methods have hyper-parameters which result in the models giving very similar bpps on the CLIC test set. Although it was previously known that the quality of the reconstructed images decreases as the bpp decreases, it was not previously known which frequencies NIC models distort to achieve a given bpp. The D plots show that JPEG2000 models corrupt low- and mid-frequency regions starting at low compression rates and these regions become more severe as the budget decreases. SH NIC models do almost the opposite-they sacrifice the highest frequencies first and expand this region into lower frequencies at more severe compression rates ( i.e. , as bpp decreases). JPEG has a mechanism more similar to JPEG2000 in that it corrupts one ring and increases the severity of this ring, while ELIC models corrupt all frequencies evenly and then corrupt low frequencies more as bpp decreases. This suggests that as the compression rate increases, classic codecs corrupt the same frequencies more severely while NIC models change their corruption patterns .

4.2 Evaluating the generalization and the robustness on OOD data

We use the CLIC-C dataset and our spectral tools to study the OOD performance of different image compression methods. We show results for example shifts (one low, medium, and high-frequency representative) and three severities (1, 3, and 5 where 5 is most severe) in Figures 3 and 4 and discuss several interesting findings below.

4.2.1 Rate-distortion curves on OOD data

Image compression models generalize to low- and mid-frequency shifts better than highfrequency shifts. The top row of Figure 3 shows how well different compression models generalize to shifted images in terms of RD curves. In other words, these plots show how well a compressor C can reconstruct a given shifted image c ( X ) in terms of PSNR of C ( c ( X )) with respect to c ( X ) . The three examples of corruption in this figure show vastly different trends. On the low-frequency corruption (snow), all four models can reconstruct c ( X ) almost as well as these models can reconstruct clean images: note the PSNR range in the top left plot of Figure 3 is about 22-40 while the PSNR range for the clean data in Figure 2 is about 28-42 over the same bpp range. Interestingly, the three models can reconstruct the images with the glass blur (a medium-frequency shift) better than they can reconstruct clean images (the PSNR of the data shifted with glass blur ranges from about 25-48 with bpp < 1). These results suggest that image compression models are fairly effective at generalizing to low-frequency shifts and very effective at generalizing to medium-frequency shifts . However, the

G(C,x,c)

SHNIC

ELIC

JPEG2000

JPEG

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

Snow

Snow

Snow

30

Snow

30

0

Q

(1.3, 34.3)

(1.2, 33.8)

(1.3, 36.1)

(1.3, 36.5)

(0.8, 31.1)

(0.8, 31.9)

(1.1, 30.4)

(1.0, 30.5)

0.1

0.03

s

0.1

Blur

0.1

0.0

0.00

0.0

0.0

(0.4, 40.3)

(0.2, 43.7)

(0.4, 41.0)

(0.2, 44.5)

(0.8, 40.8)

(0.8, 46.3)

(0.5, 37.0)

(0.4, 40.4)

Noise

700

Noise

700

Noise

700

Noise

700

0

0

Q

(1.8, 22.9)

(2.7, 11.9)

(2.2, 23.3)

(4.1, 12.4)

(0.8, 21.7)

(0.8, 11.1)

(1.3, 21.8)

(3.1, 11.8)

R(C,x,c)

sev.=1

sev.=5

500

sev.=1

sev.=5

500

sev.=1

sev.=5

500

sev.=1

sev.=5

500

Snow

Snow

Snow

Snow

(1.3, 15.4)

(1.2, 8.9)

0

(1.3, 15.3)

(1.3, 8.8)

0

(0.8, 15.1)

(0.8, 8.7)

(1.1, 15.3)

(1.0, 8.8)

500

500

500

500

Blur

s

s

Blur

(0.4, 26.0)

(0.2, 22.8)

0

(0.4, 26.1)

(0.2, 22.9)

0

(0.8, 26.0)

(0.8, 22.8)

0

(0.5, 25.8)

(0.4, 22.8)

600

600

600

600

Noise

Noise

Noise

Noise

(1.8, 27.2)

(2.7, 15.5)

(2.2, 26.3)

(4.1, 14.4)

0

(0.8, 25.0)

(0.8, 14.5)

(1.3, 25.7)

(3.1, 12.9)

high-frequency shift (shot noise), gives a starkly different result. All four models give very low PSNRs with respect to c ( X ) . Even at the lowest severity (severity=1), this PSNR is only in the low 20s. As the severity increases ( i.e. , as the lines become more transparent), the PSNR decreases even more to the point that, at the highest severity, none of the models can achieve a PSNR higher than 14. Notably, the main factor determining the PSNR is the severity of the corruption and not the compression model or bpp. This suggests that it is significantly harder to generalize to high-frequency data than to low- or mid-frequency data .

NIC models are better at denoising high-frequency corruptions than classic codecs. The second row of Figure 3 shows how well different compressors C denoise corrupted images in terms of the PSNR of C ( c ( X )) with respect to X . These results show that all the models fail at denoising snow (low-frequency) and glass blur (medium-frequency) corruptions (PSNR does not change much with an increase in bpp, except at low bpps on glass blur). However, on shot noise (a high-frequency corruption) both NIC models achieve better PSNRs with respect to X than both classic codecs. This trend is consistent for all three severity levels and suggests that NICs may be a more effective method for denoising high-frequency corruptions than the previously-used JPEG and JPEG2000 methods . The implication of this finding extends to the research area of adversarial example denoising [6].

4.2.2 Spectral analysis on OOD data

We now take a deeper look at the Section 4.2.1 findings using our spectral inspection tools.

Spectral artifacts are similar for low-frequency shifts and clean images. The patterns of G in Figure 4 measure how well each model generalizes to (or reconstructs) the shifted images. For the low-frequency shift (top row of Figure 4), the plots look strikingly similar to the patterns exhibited by the same models on clean data (Figure 2 top and bottom row): SH NIC models distort high frequencies more than low frequencies while JPEG2000 distorts low and medium frequencies more than high frequencies. Similarly, ELIC distorts all frequencies relatively evenly and JPEG severly distorts one ring of frequencies. This suggests that the methods' modus operandi for generalization to data with low-frequency shifts is similar to their modus operandi on clean data , which makes sense as clean data is dominated by low/mid frequencies. Interestingly, these generalization differences between compressors are not accompanied by differences in robustness metric R . All methods show similar patterns in their plots of R patterns and these in turn look similar to the snow corruption plot in Figure 1a. This is consistent with our finding from Figure 3 which showed that both NIC and

psnr~36.93

psnr=32.97

psnr ~ 31.17

5.0%

Difference in Accuracy

0.0%

Difference in Accuracy

Difference in Accuracy

-5.0%

-10.0%

-15.0%

SH NIC (psnr=36.93)

SH NIC (psnr=32.97)

SH NIC (psnr=31.17)

ELIC (psnr=37.10)

ELIC (psnr=32.84)

ELIC (psnr=30.98)

-20.0%

JPEG2000 (psnr=36.69)

JPEG2000 (psnr=32.92)

JPEG2000 (psnr=31.24)

JPEG (psnr=37.11)

JPEG (psnr=33.11)

JPEG (psnr=31.18)

-25.0%

clean

low

medium

high

clean

low

medium

high

clean

low

medium

high

JPEG200 fail to denoise low-frequency corruptions. In other words, Figure 4 shows that NIC and classic codecs fail in a very similar manner on low-frequency signal denoising tasks.

Both NIC and classic codecs make almost no generalization error on medium-frequency shifts. The second row of Figure 4 shows that all four methods have very small generalization errors (magnitudes < 0.2), and this low error is relatively uniform across all frequencies. This shows that all the models models are very effective at reconstructing all the frequencies in images with glass blur-in fact, they can reconstruct these images better than they can reconstruct clean imagescorroborating our first finding in Section 4.2.1. Again the R plots for glass blur for both of these models look very similar to Figure 1a; this similarity has a simple explanation due to a fundamental tradeoff between generalization and robustness . This relationship between R , G and average PSD of corruptions is described more precisely in Appendix L.4.

High-frequency corruptions uncover the spectral bias of NIC models. Section 4.2.1 highlighted that high-frequency signals severely degrade the generalization performance of all image compression methods. From the RD curves with respect to the corrupt images (top right plot in Figure 3), we observe that at each severity the three models have almost identical performance in the usual 0-2 bpp range. These results might lead us to expect that these models make similar reconstruction mistakes, but our spectral inspection tools indicate that this is not the case at all. Our plots of G on shot noise at severity=5 (bottom row, columns 2, 4, 6, and 8 of Figure 4) indicate that NIC models distort the highest frequencies significantly more than the low and medium frequencies (notice the orange borders of the plots) while JPEG and JPEG2000 exhibit different patterns. This nuance, which is equivalent to one we previously observed with clean data in Figure 2, becomes more apparent from the plots of G on a high-frequency corruption because high-frequency signals are much more prevalent on this data than on the clean data. Additionally, the bottom right plot of Figure 3 shows that for shot noise, NIC models achieve higher PSNR of the reconstructed corrupt images with respect to the clean images than the classic codecs, i.e., they have a stronger denoising effect. In the bottom right of Figure 4 we see a more detailed picture: for high severity shot noise, NIC models achieve lower R in high frequencies. Also, note that the NIC models have the smallest errors in G whereas they have the largest errors in R and vice versa. Thus, these findings suggest that NIC models behave similarly to a low-pass filter .

4.3 Impact of spectral artifacts on downstream applications

In practice, image compression algorithms may be used as a pre-processing step before images are used for another downstream task. For this reason, practitioners should also consider the performance on downstream tasks when comparing compression methods. We analyze the effectiveness of the compression methods on the downstream task of ImageNet classification using a pre-trained ResNet-50 model and report the difference in top-1 accuracy after compression (Figure 5) [25].

NIC can improve the robustness of downstream classification to high-frequency corruptions.

While in general, compression degrades classification performance on clean images (all compressors show negative differences for the 'clean' category in Figure 5), in some cases NIC and JPEG can actually improve the robustness of the classification model against high-frequency corruptions. Specifically, at PSNR=36.93, the difference in accuracy is positive for high-frequency corruptions with both SH NIC and ELIC, meaning that the classification model gave a higher accuracy on the set of corrupted images after compression than it did on the original corrupted images. Compressing with JPEG or JPEG2000 at the same rate caused a degradation in accuracy on these corruptions. However, at higher compression rates (PSNR=32.97 and PSNR=31.17), JPEG improves classification performance for high-frequency corruptions, while SH NIC and ELIC do not. On low- and midfrequency corruptions, JPEG and JPEG2000 have a smaller degradations in accuracy compared to NIC the NIC models and no model gives an improvement in classification accuracy. Thus, the ideal compressor for downstream tasks is dependent on the type of corruption and level of compression .

Pruning NIC models amplifies the robustness gains of NIC for downstream classification tasks. Additional experimental results in Appendix C (Figure 9), show that pruned NIC and NIC optimized for MS-SSIM act as even better high-frequency signal denoisers for this application .

5 Theoretical analysis of the OOD performance of NIC

We corroborate our empirical findings with theoretical results. Here we summarize theoretical results on linear autoencoder-based NIC methods. For complete definitions, additional references, and proofs of the following statements, please refer to Appendix L. In Appendix L.3, we provide a more general result applicable to nonlinear models.

Recall a classical observation: in the setting of linearly auto-encoding a mean-centered distribution, the reconstruction function ( i.e. , encoder-decoder composition) is a projection onto the high-variance principal components of the input data [18] (see also [7]). Combining this with well-known facts about statistics of natural images, we show that a linear autoencoder applied to a natural image dataset retains only low-to-mid (spatial) frequency components, which account for the majority of the variance. Using this result, we state theoretical explanations for multiple trends in our experiments 4 .

Lemma 5.1. Let X be a dataset of natural images and let ˆ X denote its (spatial) discrete Fourier transform. Assume the following (for supporting evidence see Appendix L):

(I) The principal components of ˆ X are roughly aligned with the spatial Fourier frequency basis.

(II) The associated variances are monotonically decreasing with frequency magnitude (more specifically according to the power law 1 | i | α + | j | β ).

If C is a linear autoencoder trained by minimizing MSE on X , with latent space of dimension r , and if ' ̂ ' denotes the spatial discrete Fourier transform, for any data point X with Fourier transform ˆ X

where K is the number of channels in the images in X and where ˆ X : ,ij denotes the components of ˆ X corresponding to spatial frequency ( i, j ) .

Corollary 5.2. Under the hypotheses of Lemma 5.1, the robustness error of C to a corruption c (as defined in Definition 3.3), measured in spatial frequency ( i, j ) , is

Corollary 5.3. Under the hypotheses of Lemma 5.1, the generalization error of C to a corruption c (as defined in Definition 3.2), measured in spatial frequency ( i, j ) , is

Lemma 5.1 suggests that autoencoder compressors trained on natural images behave like low-pass filters, corroborating our claim in Section 4.2.2 (see also remark L.1).

Corollary 5.2 suggests that autoencoder compressors trained on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that | ( c ( X ) -X ) : ,ij | 2 is large for small values of ij ). This is indeed what we see in Figure 3 (bottom left), where snow corruptions are detrimental to PSNR of C ( c ( X )) w.r.t. X , and Figure 4, where the R ( C , X , c ) error for the NIC is concentrated in low frequencies. We also observe in Figure 5 that NIC is more beneficial for downstream classification accuracy in the case of high-frequency corruptions (e.g. shot noise) and less beneficial in the case of low-frequency corruptions (e.g. snow).

On the other hand, the conclusion of Corollary 5.3 is more involved than that of Corollary 5.2the 'cross term' 2 ˆ X : ,ij ( ̂ c ( X ) -ˆ X ) : ,ij is in general non-zero. However, there are cases where in expectation over the data set X this cross term vanishes ( e.g. , when c is additive noise). In such cases, Corollary 5.3 suggests that compressors trained on natural images generalize less successfully to shifts with large amplitude in high frequencies . In Figure 3 (top right) we see that shot noise corruptions are detrimental to PSNR of C ( c ( X )) w.r.t. c ( X ) , and in Figure 4 we see that for the snow and shot noise corruptions, G ( C , X , c ) are concentrated in high frequencies. 5 In summary, with both theoretical analysis of a simple mathematical model and empirical results, we find that NICs have a spectral bias that causes them to overfit to discard high frequency components of natural images. .

6 Limitations

Due to the number of axes of variation present in our experiments ( e.g. , model type, compression rate, evaluation dataset, corruption type, corruption severity, evaluation metric), we were forced to constrain many variables. We present results for a limited number of NIC models all trained on one dataset, CLIC. Due to limitations on the number of models we could train and discrete choices for classic codec hyper-parameters, there is some variation within our fixed- {bpp,PSNR} bins.

Evaluating the robustness of any machine learning system is inherently a multi-faceted problem riddled with unknown-unknowns. While we focus on robustness to naturalistic input-data corruptions, there are other important distribution shifts for researchers and practitioners to consider, such as subtle changes in data collection methodology [51]. We omit a study of NIC model adversarial robustness to worst-case input data perturbations. Nevertheless, we hope that the our evaluation methods (including our metrics {D , G , R} and Fourier heatmap spectral inspection tools) will be useful in future work on NIC model robustness.

7 Conclusion and future directions

Weproposed benchmark datasets and inspection tools to gain a deeper understanding of the robustness and generalization behavior of image compression models. Using our spectral inspection tools, we uncovered the modus operandi of different compression methods. We also highlighted similarities and differences among them via a systematic OOD evaluation. Exploring the use of our tools in other image compression methods, including transformer-based architectures and implicit neural representations methods is expected to provide interesting insights [41, 72, 73, 45, 54]. Our OOD evaluations identify NIC model brittleness issues: designing practical robust training approaches and/or methods to efficiently adapt to distribution shifts at runtime is a worthwhile research direction.

References

Origins of scaling in natural images. Vision Research , 37(23):3385-3398, Dec. 1997. ISSN 0042-6989. doi: 10.1016/S0042-6989(97)00008-4.

Oodhdr-codec: Out-of-distribution generalization for hdr image compression. 36:158-166, Jun. 2022. doi: 10.1609/aaai.v36i1.19890. URL https://ojs.aaai.org/index.php/AAAI/ article/view/19890 .

Vvc software - versatile video coding test model (vtm), Year of access, e.g., 2023. URL https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM .

G. Alain and Y. Bengio. What Regularized Auto-Encoders Learn from the Data-Generating Distribution. Journal of Machine Learning Research , 15(110):3743-3773, 2014. ISSN 15337928.

J. An and S. Cho. Variational Autoencoder based Anomaly Detection using Reconstruction Probability. 2015.

A. E. Aydemir, A. Temizel, and T. T. Temizel. The effects of jpeg and jpeg2000 compression on attacks using adversarial examples. arXiv preprint arXiv:1803.10418 , 2018.

P. Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks , 2(1):53-58, Jan. 1989. ISSN 0893-6080. doi: 10.1016/0893-6080(89)90014-2.

J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations , 2017. URL https://openreview. net/forum?id=rJxdQ3jeg .

J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=rkcQFMZRb .

J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029 , 2020.

E. O. Brigham and R. E. Morrow. The fast fourier transform. IEEE Spectrum , 4(12):63-70, 1967. doi: 10.1109/MSPEC.1967.5217220.

G. J. Burton and I. R. Moorhead. Color and spatial structure in natural scenes. Applied Optics , 26(1):157-170, Jan. 1987. ISSN 2155-3165. doi: 10.1364/AO.26.000157.

Y. Choi, M. El-Khamy, and J. Lee. Variable rate deep image compression with a conditional autoencoder. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3146-3154, 2019.

F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. URL https://openreview.net/forum?id=SSKZPJCt7B .

J. Diffenderfer and B. Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning a randomly weighted network. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=U_mat0b9iv .

J. Diffenderfer, B. R. Bartoldson, S. Chaganti, J. Zhang, and B. Kailkhura. A winning hand: Compressing deep networks can improve out-of-distribution robustness. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=YygA0yppTR .

A. Dosovitskiy and J. Djolonga. You only train once: Loss-conditional training of deep networks. In International Conference on Learning Representations , 2020. URL https: //openreview.net/forum?id=HyxY6JHKwr .

C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, Sept. 1936. ISSN 1860-0980. doi: 10.1007/BF02288367.

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018. doi: 10.48550/ARXIV.1803.03635. URL https://arxiv.org/abs/1803. 03635 .

J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

C. Gille, F. Guyard, M. Antonini, and M. Barlaud. Learning sparse auto-encoders for green ai image coding, 2022. URL https://arxiv.org/abs/2209.04448 .

V. K. Goyal. Theoretical foundations of transform coding. IEEE Signal Process. Mag. , 18(5): 9-21, Sept. 2001.

D. He, Y. Zheng, B. Sun, Y. Wang, and H. Qin. Checkerboard context model for efficient learned image compression, 2021. URL https://arxiv.org/abs/2103.15306 .

D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang. ELIC: Efficient learned image compression with unevenly grouped Space-Channel contextual adaptive coding. Mar. 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=HJz6tiCqYm .

D. Hendrycks*, N. Mu*, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum? id=S1gmrxHFvB .

D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8320-8329, 2021. doi: 10.1109/ICCV48922.2021.00823.

A. Hepburn, V. Laparra, R. Santos-Rodriguez, J. Ballé, and J. Malo. On the relation between statistical learning and perceptual distances. In International Conference on Learning Representations , Mar. 2022.

I. M. P. E. G. M. ITU-T, Video Coding Experts Group (VCEG). Versatile video coding (h.266/vvc) standard. In ITU-T Recommendation H.266 and ISO/IEC 23090-3 . International Telecommunication Union (ITU) and International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC), 2020. URL https://www.itu.int/rec/ T-REC-H.266 .

N. Johnston, E. Eban, A. Gordon, and J. Ballé. Computationally efficient neural image compression. arXiv preprint arXiv:1912.08771 , 2019.

K. Kireev, M. Andriushchenko, and N. Flammarion. On the effectiveness of adversarial training against common corruptions. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022. URL https://openreview.net/forum?id=BcU_UIIjqg9 .

S. Klasky, J. Thayer, and H. Najm. Data reduction for science: Brochure from the advanced scientific computing research workshop. Technical report, Oak Ridge National Lab.(ORNL), Oak Ridge, TN (United States); SLAC National Lab., 2021.

E. Kodak. Kodak lossless true color image suite (PhotoCD PCD0992). URL http://r0k.us/ graphics/kodak .

D. Kunin, J. Bloom, A. Goeva, and C. Seed. Loss Landscapes of Regularized Linear Autoencoders. In Proceedings of the 36th International Conference on Machine Learning , pages 3560-3569. PMLR, May 2019.

A. B. Lee, D. Mumford, and J. Huang. Occlusion Models for Natural Images: A Statistical Study of a Scale-Invariant Dead Leaves Model. International Journal of Computer Vision , 41 (1):35-59, Jan. 2001. ISSN 1573-1405. doi: 10.1023/A:1011109015675.

E. Lei, H. Hassani, and S. S. Bidokhti. Out-of-distribution robustness in deep learning compression. arXiv preprint arXiv:2110.07007 , 2021.

L. Li, Z. Yang, and R. Wang. Improving multi-generation robustness of learned image compression, 2022.

Z. Li, J. O. Caro, E. Rusak, W. Brendel, M. Bethge, F. Anselmi, A. B. Patel, A. S. Tolias, and X. Pitkow. Robust deep learning object recognition models rely on low frequency information in natural images. bioRxiv , 2022. doi: 10.1101/2022.01.31.478509. URL https://www. biorxiv.org/content/early/2022/02/02/2022.01.31.478509 .

T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing , 461:370-403, 2021. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2021.07.045. URL https://www.sciencedirect.com/ science/article/pii/S0925231221010894 .

J. Liu, H. Sun, and J. Katto. Learned image compression with mixed transformer-cnn architectures, 2023.

K. Liu, D. Wu, Y. Wang, D. Feng, B. Tan, and S. Garg. Malice: Manipulation attacks on learned image compression, 2022.

A. Luo, H. Sun, J. Liu, and J. Katto. Memory-efficient learned image compression with pruned hyperprior module, 2022. URL https://arxiv.org/abs/2206.10083 .

M. C. Malin, M. A. Ravine, M. A. Caplinger, F. Tony Ghaemi, J. A. Schaffner, J. N. Maki, J. F. Bell III, J. F. Cameron, W. E. Dietrich, K. S. Edgett, et al. The mars science laboratory (msl) mast cameras and descent imager: Investigation and instrument descriptions. Earth and Space Science , 4(8):506-539, 2017.

D. Minnen and S. Singh. Channel-wise autoregressive entropy models for learned image compression, 2020. URL https://arxiv.org/abs/2007.08739 .

D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/ file/53edebc543333dfbf7c5933af792c9c4-Paper.pdf .

D. Mishra, S. K. Singh, and R. K. Singh. Deep architectures for image compression: A critical review. Signal Processing , 191:108346, 2022. ISSN 0165-1684. doi: https://doi.org/10.1016/ j.sigpro.2021.108346. URL https://www.sciencedirect.com/science/article/pii/ S0165168421003832 .

S. Narang, G. F. Diamos, S. Sengupta, and E. Elsen. Exploring sparsity in recurrent neural networks. CoRR , abs/1704.05119, 2017. URL http://arxiv.org/abs/1704.05119 .

E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence , 32 (1), Apr. 2018.

E. Plaut. From Principal Subspaces to Principal Components with Linear Autoencoders, Dec. 2018.

B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet?, 2019.

J. Rissanen and G. Langdon. Universal modeling and coding. IEEE Transactions on Information Theory , 27(1):12-23, 1981. doi: 10.1109/TIT.1981.1056282.

A. Skodras, C. Christopoulos, and T. Ebrahimi. The jpeg 2000 still image compression standard. IEEE Signal Processing Magazine , 18(5):36-58, 2001. doi: 10.1109/79.952804.

Y. Strümpler, J. Postels, R. Yang, L. van Gool, and F. Tombari. Implicit neural representations for image compression, 2022.

H. Sun, L. Yu, and J. Katto. End-to-end learned image compression with quantized weights and activations, 2021. URL https://arxiv.org/abs/2111.09348 .

J. Sun, A. Mehra, B. Kailkhura, P.-Y. Chen, D. Hendrycks, J. Hamm, and Z. M. Mao. A spectral view of randomized smoothing under common corruptions: Benchmarking and improving certified robustness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV , pages 654-671. Springer, 2022.

J. Sun, Q. Zhang, B. Kailkhura, Z. Yu, C. Xiao, and Z. M. Mao. Benchmarking robustness of 3d point cloud recognition against common corruptions. arXiv preprint arXiv:2201.12296 , 2022.

L. Theis, W. Shi, A. Cunningham, and F. Huszár. Lossy image compression with compressive autoencoders, 2017. URL https://arxiv.org/abs/1703.00395 .

G. Toderici, W. Shi, R. Timofte, L. Theis, J. Balle, E. Agustsson, N. Johnston, and F. Mentzer. Workshop and challenge on learned image compression (clic2020), 2020. URL http://www. compression.cc .

G. Wallace. The jpeg still picture compression standard. IEEE Transactions on Consumer Electronics , 38(1):xviii-xxxiv, 1992. doi: 10.1109/30.125072.

Z. Wang and A. C. Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine , 26(1):98-117, 2009.

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004. doi: 10.1109/TIP.2003.819861.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe. Vvenc: An open and optimized vvc encoder implementation. In Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW) , pages 1-2. doi: 10.1109/ICMEW53276.2021.9455944.

F. Yang, L. Herranz, Y. Cheng, and M. G. Mozerov. Slimmable compressive autoencoders for practical neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4998-5007, June 2021.

D. Yin, R. G. Lopes, J. Shlens, E. D. Cubuk, and J. Gilmer. A Fourier Perspective on Model Robustness in Computer Vision . Curran Associates Inc., Red Hook, NY, USA, 2019.

S. Yin, C. Li, Y . Bao, and Y . Liang. Universal efficient variable-rate neural image compression, 2021. URL https://arxiv.org/abs/2111.11305 .

S. Yin, C. Li, W. Tan, Y. Bao, Y. Liang, and W. Liu. Exploring structural sparsity in neural image compression, 2022. URL https://arxiv.org/abs/2202.04595 .

D. Zhang, J. Yang, D. Ye, and G. Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

M. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=Sy1iIDkPM .

X. Zhu, J. Song, L. Gao, F. Zheng, and H. T. Shen. Unified multivariate gaussian mixture for efficient neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17612-17621, June 2022.

Y. Zhu, Y. Yang, and T. Cohen. Transformer-based transform coding. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= IDwN6xjHnK8 .

R. Zou, C. Song, and Z. Zhang. The devil is in the details: Window-based attention for image compression, 2022.

A Background and Related Work

A.1 Neural Image Compression (NIC)

In lossy image compression, an input image is compressed into a small file or bitstream for transmission or storage. This file is then used to reconstruct the original image. Several models have been proposed for neural (or learned) image compression such as convolutional neural networks (CNNs), autoencoders, and generative adversarial networks [22, 8, 9, 46, 58, 13, 65, 71, 47]. In the most basic version of these models, an image vector x is mapped to a latent representation y using a convolutional autoencoder. y is then quantized into ˆ y , so it can be compressed to a bitstream via an entropy coding method, such as arithmetic coding [52]. During reconstruction, the received bits go through the reverse entropy coding process and the revealed ˆ y is fed into a CNN decoder which outputs the reconstructed image ˆ x . These models can be trained end-to-end by using differentiable proxies of entropy coding [8].

Normally entropy coding relies on one prior probability model of ˆ y , which is learned from the entire set of images. Ballé et al. [9], however, recognized that the prior of this model can be optimized for each image separately. By adding an additional hyperprior autoencoder to their model, they learn the latent representation for the entropy model of each image. The information transmitted from this autoencoder can be seen as equivalent to 'side information' used by other codecs. Minnen et al. [46] further improved this model by adding an autoregressive prior to the hierarchicial prior used by Ballé. The Efficient Learned Image Compression (ELIC) model builds on these models by adding uneven channel-conditional adaptive coding and a spatial context model [24].

A.2 Generalization and robustness in DNNs for classification

It is well known that deep neural networks (DNNs) are prone to overfitting to the specific data distribution they are trained on. Much recent work has focused on improving the robustness of DNNs to other distributions of data, including common image corruptions [27, 66]. CIFAR-10-C and ImageNet-C were proposed as a benchmark datasets to test the robustness of models to common image corruptions such as corruptions from weather, blur, and digital noise [26]. Methods for improving robustness include data-level techniques such as adding augmentations and algorithm-level techniques like using larger models or adversarial training. [27, 32, 28, 14].

Yin et al. study the effects of data augmentation and adversarial training on different corruptions using a Fourier analysis [66]. They analyze the CIFAR-10-C dataset and find that most augmentation methods help in the high-frequency domain, but hurt in the low-frequency domain. However, it is unknown how image compression affects corrupted images. We analyze how different types of image compression affect the Power Spectral Density (PSDs) of both clean and corrupted data.

A.3 Generalization and robustness with NIC models

There is a limited amount of work studying the generalization and robustness of NIC models. [37] formulates a problem for designing robust NIC models and train DNNs to be robust to two particular types of distributional shifts. They find a tradeoff between OOD robustness and IND performance. Our work is complementary to this work in that we provide additional metrics and OOD benchmarks to analyze the robustness of various models in more detail. [42] studies NIC models' robustness to white-box and black-box adversarial attacks and proposes new models which improve the robustness of NIC to such attacks. Our work instead looks at naturalistic distribution shifts, which is another important aspect of robustness [26].

[2] designed a NIC module which works on a specific set of OOD images. In particular, their module works on high dynamic range (HDR) images, despite being trained on mostly standard dynamic range (SDR) images. Although this is a specific example of testing NICs on OOD data, our work differs from theirs in that we analyze how NICs would perform on a wide variety of image corruptions without seeing any examples of these corruptions during training.

Additionally, [38] addresses the issue of 'multi-generational loss', or the problem of a rapid degradation of quality as images are repeatedly encoded and decoded, with NIC models. Our work differs because we propose spectral tools to analyze the effects of one cycle of encoding and decoding and we study how this affects the performance of different downstream tasks. While multi-generation

loss is an important area of study for tasks like image editing, we focus on analyzing the spectral distortions caused by one cycle of image compression and how these affect a model's generalization and robustness to OOD data. Furthermore, [38] proposes using JPEG2000 as a method to improve the robustness of classification tasks by compressing adversarial noise. We explore how NIC models compare to JPEG2000 in terms of their robustness to several common types of image corruptions.

A.4 Lightweight NIC via model pruning

In an effort to reduce the energy and compute cost required by deep neural networks (DNNs), much work has been devoted towards finding methods to approximate DNNs. Common approaches include pruning network weights or quantizing network weights and activations [19, 20, 48, 69, 40, 15]. Remarkably, large sparse models have been shown to outperform small dense models with the same number of weights [70]. Additionally, approximating DNNs can have benefits in terms of robustness on OOD data [16].

Nonetheless, limited efforts to prune or quantize NIC models have been explored. Recently, Gille et al. [21] successfully introduced structured sparsity into NIC models to simultaneously reduce storage and power costs while maintaining a similar rate-distortion curve when compared to a dense version of the same NIC architecture. Their findings produced ∼ 80% reduction in the number of parameters and ∼ 30% reduction in MAC operations required by the NIC encoder. Yin et al. [68] also explored the use of structured sparsity in NIC and their proposed ABCM method was demonstrated to provide up to 7 × and 3 × reduction in DNN parameters and inference time, respectively, with only a mild reduction to reconstruction quality. Luo et al. [43] proposed a technique for pruning NIC models that was demonstrated to maintain the same bpp as dense models but provided a parameter reduction of at most ∼ 34%. Sun et al. [55] demonstrated a technique for quantizing both NIC weights and activations to an 8-bit fixed point representation while yielding a rate-distortion curve comparable to the original 32-bit float NIC model. Outside of DNN pruning or quantization, alternative techniques have been explored for more efficient NIC [31, 67, 23].

A.5 Metrics for quantifying image compression quality

The objective of lossy image compression is to achieve a balance between minimizing 1) the size of the compressed file and 2) the distortion of the reconstructed image compared to the original image. As these objectives are inherently at odds, the performance of lossy image compression is typically measured using a rate-distortion curve, which measures the size of the compressed image (rate) versus the quality of the reconstructed image (distortion). The rate is usually measured in bits per pixel (bpp), which is the average number of bits required to store each pixel in the compressed representation. The distortion is usually measured by scalar metrics such as peak signal-to-noise ratio (PSNR) or a visual similarity metric such as MS-SSIM [62].

A.6 Benchmark datasets

We utilize two benchmark datasets for evaluating performance of our NIC models. For training and testing, we make use of CLIC (Challenge on Learned Image Compression) 2020 [59]. This collection is comprised of 1633 training, 102 validation, and 428 test images. The images are of varying resolution and are further categorized as either professional or mobile. To evaluate generalization, we make use of the Kodak [34] dataset which consists of 24 images of resolution 768 × 512 (or 512 × 768 ).

B Additional results: Fourier heatmaps

While our spectral tools allow us to measure different capabilities of compression models, they require the availability of OOD, or corrupted, data. To support a setting in which such OOD data is unavailable, we propose adopting another tool: the Fourier sensitivity heatmap. This tool evaluates the PSNR of a compression model on data perturbed with Fourier basis elements [66]. The resulting visualization is a heatmap where the value at coordinate ( i, j ) is the PSNR of the compression method on perturbed data { X k + r k εU i,j } N k =1 , where each r k is selected uniformly at random from {-1 , 1 } , ε is the norm of the perturbation, and U i,j is the ( i, j ) th Fourier basis matrix. From [66],

PSNR w.r.t. c(t)

PSNR w.r.t. X

VRNIC

VR NIC FRNIC opt.

VR NIC

VR NIC

FR NIC opt.

FR NIC

VR NIC

pr=0.95

MS-SSIM

JPEG2000

36.0

FR NIC

VR NIC

pr=0.95

MS-SSIMJ

JPEG2000

35

high psnr

high psnr

34.5

32

33.5

33.5

med. psnr

med. psnr

32.0

30.5

32.0

32

30.5

29

PSNR w.r.t. c(x)

PSNR w.r.t. X

VR NIC

VRNIC

FR NICopt.

VR NIC

VRNIC

FR NIC opt.

FR NIC

VR NIC

pr=0.95

MS-SSIM

JPEG2000

39

FR NIC

VR NIC

pr=0.95

MS-SSIM

JPEG2000

38

high bpp

high bpp

35

34

med.bpp

37

med.bpp

37

33

32

32

ddqMol

28

28

U i,j ∈ R n 1 × n 2 and satisfies ( i ) ∥ U i,j ∥ = 1 and ( ii ) F ( U i,j ) has at most two non-zero coordinates specifically at ( i, j ) and the coordinate symmetric to ( i, j ) about the matrix center.

We consider two versions of Fourier heatmaps: (a) with respect to perturbed data and (b) with respect to original data. To analyze (a) and (b), the PSNR at each coordinate ( i, j ) is computed with respect to the perturbed dataset, { X k + r k εU i,j } N k =1 , and the unperturbed dataset, { X k } N k =1 , respectively. (a) and (b) can be seen as measures of generalization and robustness respectively.

We computed Fourier heatmaps over the CLIC dataset in Figure 6. We use models from Appendix C, where FR NIC is equivalent to SH NIC in the main body and selected hyper-parameters for these models using a fixed-bpp/PSNR constraint on the clean data. Note: in these Fourier heatmap plots, warmer colors represent higher PSNR , which is in contradiction to the plots of D , G , and R where warmer colors represented more error .

On the left side of Figure 6, we analyze the generalization of the image compression models to various frequency shifts. We find that all image compression models generalize to low and mediumfrequency signals better than high-frequency signals. Specifically, observe how the centers of the plots-corresponding to low-frequency signals-have hotter colors than the edges of the plots. This corroborates our findings from both the RD curves and G metrics on the CLIC-C dataset in Section 4.2 and Appendix H. For FR NIC opt. MS-SSIM, this difference is only noticeable at the lowest bpp; however for the other models, there is a clear difference in performance for low and high-frequency shifts at all bpps. Additionally, the effect of pruning is a degradation in PSNR at all frequencies.

On the right side of Figure 6, we analyze the robustness, or denoising, capabilities of the models. These plots show that NIC models are better at denoising high-frequency corruptions than lowfrequency corruptions (notice the cooler-colored diamonds in the center of the NIC plots). Meanwhile JPEG2000 has very consistent and poor denoising properties across the entire range of corruption frequencies (the plots are blue with a narrow gradient of color compared to the NIC plots). According to this metric, NIC opt. MS-SSSIM also has poor denoising capabilities (the plots are all dark blue compared to the NIC plots) and pruning NICs also results in a degradation of denoising performance. At first, this may seem like a contradiction to our downstream classification results from Appendix C, which showed that NIC opt. MS-SSIM and VR NIC pr=0.8/0.95 were more effective at denoising high-frequency corruptions than FR NIC and VR NIC. However, recall that each pixel in these plots

only shows the average PSNR of the entire perturbed dataset as opposed to the spectral error by frequency, like our D , G , and R plots show. Then from the main body and Figure 6, it is clear that this PSNR metric is not detailed enough highlight the nuances between the spectral distortion patterns of different methods.

C Additional results: NIC variants

In this section, we analyze several additional variants of NIC models. Many of these variants were chosen because they have potential to reduce the amount of memory required to store or run these models, making them more amenable to 'in-the-wild' use cases ( e.g. , the Mars Exploration Rover discussed in the introduction). We also consider models optimized for MS-SSIM because we are interested in how the distortion objective affects the spectral distortion artifacts of the model.

Because all of these models are based on the scale-hyperprior architecture from [9], we use a different naming scheme here than in the main body. Below are descriptions of the NIC models compared in this section:

FR NIC: Equivalent to SH NIC in the main body. Eight fixed-rate models each trained on a unique λ . Distortion is MSE ( i.e. , model is optimized for PSNR).

VR NIC: One variable-rate model trained over a continuous range of λ values using loss conditional training. Distortion is MSE. More details about this model are provided in Appendix J.

VR NIC pr=0.8/0.95: One variable-rate model trained over a continuous range of λ values. Distortion is MSE. Here, gradual magnitude pruning (GMP) is applied during training to prune 80% or 95% of the convolutional and FiLM weights. More details of the GMP algorithm are provided in Appendix K.

FR NIC opt. MS-SSIM: Eight fixed-rate models, each trained on a unique λ value. Distortion is (1 - MS-SSIM) ( i.e. , model is optimized for MS-SSIM).

Figures 7, 9, and 8 show results for these models. We discuss our findings below.

C.1 Evaluating spectral distortion on IND data

Effect of approximating FR NIC with VR NIC . According to the RD curves in Figure 7, VR NIC obtains the same performance as FR NIC for low and moderate bpps; however, the SH NIC model outperforms the VR NIC model at higher bpps despite being trained on the same range of λ . This result follows [17] and suggests that the VR NIC may not be expressive enough to learn the high PSNR regime. In terms of the spectral error D , VR NIC and FR NIC have the same error patterns, but VR NIC does leave a slightly higher magnitude of error in some settings ( e.g. , med. bpp, high bpp, and high psnr).

Effect of pruning . VR NIC, VR NIC pr=0.8, and VR NIC=0.95 all have the same VR NIC model architecture, but have 0%, 80%, and 95% sparsity respectively. The RD curves show that pruning causes a degradation in performance, especially for high bpp (see blue lines in Figure 7). However, the PSDs reveal that, even if we make up for the PSNR performance gap by increasing the bpp (right side of Figure 7), the pruned models left a different spectral artifacts from the dense ones. In particular, as the prune rate increases, the ring in the high-frequency region of the PSDs becomes thicker. Presumably reconstructing signals at high-frequencies requires more parameters than reconstructing low frequency signals and so the more sparse models focus on optimizing the 'easier' frequencies on the spectrum. These results show that pruning NICs exacerbates the spectral differences between NIC and JPEG2000 that we observed in Figure 2.

Effect of optimization variable in NIC. The FR NIC and FR NIC opt. MS-SSIM have the same FR NIC model architecture, but optimize for different 'distortion' metrics ( i.e. , FR NIC optimizes for PSNR while FR NIC opt. MS-SSIM optimizes for MS-SSIM). Unsurprisingly, Figure 7 shows that the FR NIC opt. MS-SSIM model has the worst performance of all the models in terms of RD curves where the distortion metric is PSNR. However, the plots of the distortion error D give us more insights into the spectral errors that this model is making. Surprisingly, FR NIC opt. MS-SSIM produces spectral artifacts which differ greatly from both the PSNR optimized NIC models and the classic JPEG2000 codec. In particular, like JPEG2000, this model distorts the medium and low frequencies more than high-frequencies (notice the cooler-colored ring around the hot-colored center).

D(C,x):Fixed-bpp

RD Curves

D(C,x):Fixed-psnr

lowbpp

med.bpp

high bpp

40

med.psnr

high psnr

FR NIC

FR NIC

38

(0.25, 31.54)

(0.77, 36.85)

(1.21, 38.98)

36

(0.25, 31.54)

(0.37, 33.17)

(0.77, 36.85)

PSNR (dB)

VR NIC

VR NIC

34

FR NIC

(0.23, 30.98)

(0.79, 36.87)

(1.21, 38.40)

32

VR NIC

(0.26, 31.56)

(0.36, 33.17)

(0.79,36.87)

VR NIC pr=0.8

VR NIC

80=d

VR NIC pr=0.95

VR NIC

30

FR NIC opt. MS-SSIM

(0.25, 30.99)

(0.77, 35.82)

(1.22, 36.98)

JPEG2000

(0.29, 31.58)

(0.40, 33.17)

(1.22, 36.98)

28 +

Image size (bpp)

VR NIC

pr=0.95

VRNIC

S6'0=Jd

(0.25, 30.47)

(0.77, 35.36)

(1.21, 36.50)

(0.31, 31.48)

(0.45, 33.19)

opt.MS-SSIM

opt.MS-SSIM

FRNIC

FRNIC

(0.23, 29.63)

(0.72, 33.98)

(1.19, 36.16)

(0.40, 31.54)

(0.61, 33.27)

(1.34, 36.62)

JPEG2000

JPEG2000

(0.24, 30.37)

(0.80, 35.34)

(1.20, 37.31)

(0.34, 31.72)

(0.48, 33.09)

(1.09, 36.83)

0

600

15 0

6

0

400

300

Snow

ShotNoise

VR NIC

VRNICFRNICopt.

VR NIC

VR NIC FR NIC opt.

FR NIC

VRNIC

pr=0.95

MS-SSIMJPEG2000

5

FR NIC

VRNIC

pr=0.95

MS-SSIMJPEG2000

100

C(C,x,c)

C(C,x,c)

(1.60,39.08X1.58,38.40X1.37,36.49X1.50,36.43X1.29,33.97X1.19,35.10)

Q

0

500

200

R(C,X,c)

R(C,x,c)

0

However, it differs from JPEG2000 in that it leaves a less rectangular artifact. Thus, the optimization objective of NIC can influence the spectral artifacts and should be thoroughly examined before being used in practice.

C.2 Robustness and generalization

We analyze the findings further using our robustness spectral inspection tools on ImageNet-C in Figure 8. The plots of G on the snow corruption show that although the models have different magnitudes of errors, all models leave similar magnitudes of errors across the frequency spectrum ( i.e. , the gradient of each plot is narrow). Furthermore, all models have similar plots of R . This follows the result of the next section (accuracy on the downstream task), which shows that all six models have comparable accuracies at this compression rate (see snow column of bpp ≈ 1.23 plot in Figure 9).

On the shot noise corruption, however, we see some notable differences between models on both G and R . On G , we see that as the prune rate increases, the ring of high-frequency spectral error becomes significantly more bold. This reiterates our finding from Figure 7 that pruning results in a degradation of high-frequency signal reconstruction. Similarly, FR NIC opt. MS-SSIM has a thicker

bpp ~1.23

bpp ≈ 0.77

bpp ≈ 0.24

10%

Difference in Accuracy

0%

Difference in Accuracy

Difference in Accuracy

-10%

20%

FR NIC (bpp=1.23)

VR NIC (bpp=1.23)

FR NIC (bpp=0.77)

VR NIC (bpp=0.77)

FR NIC (bpp=0.24)

VR NIC(bpp=0.23)

VR NIC pr=0.8 (bpp=1.09)

VR NIC pr=0.8 (bpp=0.78)

VR NIC pr=0.8 (bpp=0.24)

%03-

VR NIC pr=0.95 (bpp=1.21)

VR NIC pr=0.95 (bpp=0.78)

VR NIC pr=0.95 (bpp=0.23)

FR NIC opt. MS-SSIM (bpp=1.24)

FR NIC opt. MS-SSIM (bpp=0.72)

FR NIC opt. MS-SSIM (bpp=0.23)

-40%

JPEG2000 (bpp=1.19)

JPEG2000 (bpp=0.79)

JPEG2000 (bpp=0.24)

clean

Mous

60

brightness

contrast-

motion

wooz

defocus

sse

jpeg -

pixelate

ueissneh

asndu

clean

MOus

60

motion

wooz

defocus

ss96

elastic

jpeg

pixelate

ueissneb

impulse

clean

Mous

60

brightness-

motion

Wooz

defocus

sse16

elastic-

jpeg

pixelate-

ueissneh

ous

impulse

psnr ≈36.93

psnr ≈32.97

psnr ≈ 31.17

%0T

Difference in Accuracy

0%

Difference in Accuracy

Difference in Accuracy

-10%

FR NIC (psnr=36.93)

FR NIC (psnr=32.97)

FR NIC (psnr=31.17)

-20%

VR NIC (psnr=36.85)

VR NIC (psnr=33.01)

VR NIC (psnr=31.18)

VR NIC pr=0.8 (psnr=36.91)

VR NIC pr=0.8 (psnr=33.07)

VR NIC pr=0.8 (psnr=31.26)

%08-

VR NIC pr=0.95 (psnr=36.92)

VR NIC pr=0.95 (psnr=33.05)

VR NIC pr=0.95 (psnr=31.21)

FR NIC opt. MS-SSIM (psnr=36.93)

FR NIC opt. MS-SSIM (psnr=33.10)

FR NIC opt. MS-SSIM (psnr=31.29)

JPEG2000 (psnr=36.69)

JPEG2000 (psnr=32.92)

JPEG2000 (psnr=31.24)

-40%

clean

Mous

60

motion

Wooz

defocus-

elastic

jpeg -

pixelate

ueissneb

impulse-

clean

MOus

60

contrast

motion

wooz

defocus·

sse6

elastic

jpeg -

pixelate-

ueissneh

shot+

impulse

clean

snow

60

Wooz

defocus

sse/6

elastic

jpeg -

pixelate-

ueissneh

shot-

impulse·

ring than FR NIC in the high-frequency domain. Interestingly, this model also has a smaller error in the low frequency domain compared to the NIC models optimized for PSNR. Finally, JPEG2000 distorts low- and medium-frequencies relatively evenly and distorts high-frequency signals less than NIC which follows from Section 4.1. We can see the effect of these differences in G in the high-frequency domain of of the plots R . Specifically, the VR NIC leaves a slightly smaller artifact than FR NIC ( i.e. , the circle of warm colors is larger with VR NIC). This differences is exacerbated as the prune rate increases ( i.e. , NIC pr=0.95 has the smallest artifact of the four first four models), which suggests that the pruned NIC models are most effective for denoising high-frequency corruptions. FR NIC opt. MS-SSIM also leaves a smaller artifact than FR NIC; however, this has a different shape than the NIC models optimized for PSNR (specifically, there are jagged edges on the edges of this artifact). This suggests that FR NIC opt. MS-SSIM denoises high-frequency data better than FR NIC, but in a different way from VR NIC pr=0.8/0.95. Finally, JPEG2000 has diamond-shaped artifact which suggests it denoises some high-frequency signals, but not all.

C.3 Downstream task

We also consider how these NIC variants perform on the same downstream classification task outlined in Section 4.3. Figure 9 shows the difference in accuracy before and after corruption for each model on each corruption (or clean data) and Appendix M has tables with these accuracies. We find that the pruned models are better at denoising high-frequency corruptions than the FR NIC. In particular, we find that on the high-frequency corruptions at bpp = 1.23 and 0.77, the pruned VR NICs show a larger improvement in accuracy than the dense VR and FR NICs. This finding matches our observation in Figure 7 that the pruned NICs denoise high-frequency signals even more than dense NICs, who in turn denoise these signals more than JPEG2000. The pruned models perform comparably to the dense models on low- and medium-frequency corruptions. Similarly, we also find that NIC opt. MS-SSIM also improves performance on the downstream task on high-frequency corruptions over (a)

D(C,x):Fixed-bpp

RDCurves

D(C,x):Fixed-psnr

lowbpp

med.bpp

highbpp

40

med.psnr

high psnr

FR NIC

38

FR NIC

(0.28, 30.19)

(0.94, 36.20)

(1.43, 38.22)

36

(0.28, 30.19)

(0.94, 36.20)

(1.25, 37.56)

JPEG2000

PSNR (dB)

JPEG2000

34

(0.27, 28.76)

(0.92, 34.25)

(1.50, 36.97)

32

(0.39,30.16)

(1.33, 36.27)

(1.71, 37.77)

VTM

30

JPEG2000

FR NIC

WLA

VTM

(0.29, 31.66)

(0.97, 37.42)

(1.43, 39.38)

28 +

(0.18, 29.98)

(0.78, 36.31)

(0.97, 37.42)

0.0

0.5

1.0

1.5

2.0

0

70

0

10(

0

5

Image size (bpp)

0

600

60

5

no compression and (b) compression with dense NIC models. This model also has better performance than the other NIC variants and low- and medium-frequency corruptions.

D Additional results: comparing JPEG2000 and VTM

In this section, we compare two classic codecs: JPEG2000 and Versatile Video Coding (VVC) (also known as h.266) codec [30]. We refer to the latter codec as VTM because we used VVC Test Model (VTM) software from [64] to test it. Although VTM has state-of-the-art performance for classic codecs, we chose to use JPEG2000 as the representative classic codec in the main body for three reasons.

JPEG2000 has more consistent bpp usage. As shown later in this section (Figure 12), VTM varies its bpp usage drastically depending on the corruption type and severity. Specifically, a hyper-parameter which gives a bpp of x on clean data may give a bpp of over 7 x on certain OOD shifts and severities. 6 Thus, it was unclear how to fairly compare VTM and NIC in a fixed-bpp setting on OOD data.

JPEG2000 implementation has faster inference time. The current open-source implementation for VTM is significantly slower than those available for JPEG2000 (10s to 100s of seconds for VTMvs. < 1 second for JPEG2000). Thus, JPEG2000 was a more feasible codec for our extensive experimentation.

After adjusting for PSNR, VTM shows similar spectral results as JPEG2000. Finally, although there are differences in magnitudes of the spectral distortion plots, JPEG2000 and VTM exhibit similar patterns, so we found it sufficient to use only one classical codec representative.

We include results comparing JPEG2000 and VTM on the Kodak and Kodak-C datasets in Figures 1013 and explain our findings below.

D.1 Evaluating VTM on IND data

Figure 10 is analogous to Figure 2 in the main body. From the RD curves, we see that VTM outperforms FR NIC and JPEG2000 across all tested bpps. VTM's plots of D are most similar to JPEG2000's. Specifically, at low compression rates (high bpp/PSNR), both methods leave similar errors across all frequencies (notice the small gradient of colors in both JPEG2000 and VTM's plots of D at med/high bpp and med/high PSNR). Similarly, at high compression rates (low bpp/PSNR) both models leave more spectral errors at the low and medium frequencies. Despite having similar patterns to JPEG2000, most of the VTM plots have a smaller magnitude than JPEG2000, even at fixed-PSNRs. This suggests that the metric D metric is more closely aligned with VTM's modus operandi than JPEG2000's.

D(C, X): Fixed-bpp

D(C, X): Fixed-psnr

high bpp

med.psnr

high

psnr

VTM

VTM

(0.29,31.66) (0.97,37.42)

(1.43, 39.38)

(0.18, 29.98)(0.78, 36.31)

）(0.97,37.42)

0

50

0.0

0.5

0.00

0.05

0

250

0

1

0.0

0.5

Low freq.: Snow

Med.freq.:Glass Blur

FR NIC

40

FR NIC

(aP)

40

JPEG2000

PSNR w.r.t. c(x) (dB)

45

PSNR w.r.t. c(x) (dB)

35

JPEG2000

VTM

VTM

w.r.t.c(x)(

35

40

25

PSNR

30

35

20

FR NIC

25

JPEG2000

15

30

VTM

10-

0

1

2

3

0.0

0.5

1.0

1.5

0.0

2.5

5.0

7.510.012.5 15.0

Image size (bpp)

Image size (bpp)

Image size (bpp)

Low freq.: Snow

High freq.:Shot Noise

25.0

FR NIC

27.5

FR NIC

PSNR w.r.t. x (dB)

14

JPEG2000

VTM

(ap)

24.5 -

(aP)x

25.0 -

JPEG2000

VTM

24.0

22.5

12

Rw.r.t.

23.5

JPEG2000

FR NIC

20.0

10

PSNR

23.0-

VTM

PSNR

17.5

15.0 -

22.5

12.5

8

22.0

10.0

0

1

2

3

0.0

0.5

1.0

1.5

0

5

10

15

Image size (bpp)

Image size (bpp)

Image size (bpp)

Figure 11 shows the six VTM plots from Figure 10 without constrained color map scales. These results confirm that VTM leaves only a small magnitude of error, which is even across frequencies.

D.2 Evaluating VTM on OOD data

Figure 12 shows the RD curves of VTM on a low, medium, and high-frequency shift. On the low-frequency shift (snow), VTM has comparable performance to FR NIC and JPEG2000 at low bpp; however, VTM can achieve higher PSNRs by increasing its bpp usage. This finding suggests that VTM, like the other codecs in Section 4.2.1, generalizes well to low-frequency shifts. However, this highlights the fact that VTM may drastically increase its bpp usage on OOD data. Interestingly, all three models are comparably robust to the snow corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to X .

On the medium-frequency shift (glass blur), VTM, like the other codecs in Section 4.2.1, achieves PSNRs which are higher than the PSNRs on clean data. However, VTM is not as effective as FR NIC in terms of PSNR with respect to c ( X ) on this shift (the VTM curves start to flatten around 40-42 PSNR, while the NIC curves flatten around 42-47 PSNR). Like the snow corruption, all three models are comparably robust to the glass blur corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to X .

On the high-frequency shift (shot noise), VTM displays a drastically different pattern from both NIC and JPEG2000. Specifically, VTM is able to achieve significantly higher PSNRs with respect to c ( X ) ( i.e. , generalize better) than FR NIC and JPEG2000. However, this comes at a very serious cost in

G(C,x,c)

R(C,X,c)

JPEG2000

VTM

JPEG2000

VTM

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

50

6

500

500

Snow

(0.80, 29.54)

(0.80, 29.17)

0

(0.84,32.58)

(0.86, 32.16)

0

(0.80, 14.87)

(0.80, 7.82)

0

(0.84, 15.29)

(0.86, 7.96)

0

0.1

0.0001

500

500

Blur

(0.80,39.83)

(0.80,46.01)

0.0

(1.15, 42.86)

(0.44, 43.99)

0.0000

(0.80, 24.81)

(0.80, 22.53)

0

(1.15, 24.78)

(0.44, 22.48)

0

500

500

500

500

Shot

Noise

(0.80, 21.62)

(0.80, 10.68)

0

(0.05,20.46)

(1.02, 11.54)

Q

(0.80, 25.01)

(0.80, 14.45)

Q

(0.05, 25.79)

(1.02, 14.63)

0

terms of bpp (notice the bpp range for shot noise in Figure 12 is 0-17 versus a typical range of 0-2). Recall that in Section 4.2.1 we observed that NIC models adjusted their bpp usage depending on the type of corruption- in particular, NIC models used more bpps than JPEG2000 on the shot noise corruption, even though it did not improve the PSNR. Figure 12 shows that VTM adjusts its bpp usage in a more extreme way than NIC models, but, unlike NIC models, this increase in bpp does result in an increase in PSNR with respect to c ( X ) . As expected from our results in Section 4.2.1, this increase in PSNR with respect to c ( X ) results in a decrease of PSNR with respect X . Overall, VTM's highly variable bpp usage could be a real disadvantage to using VTM in the wild: in particular, if a practitioner selects a hyper-parameter from clean data and expects the model will use a certain amount of bpps, VTM might use a much larger bpp than expected on OOD data. Thus, our OOD benchmark dataset is a vital aspect to comprehensive image compression testing.

Figure 13 shows the G and R metrics on the three representative shifts. 7 On the snow corruption, VTM has a smaller magnitude of errors than JPEG2000 across all frequencies; however, both models leave relatively even errors across all frequencies and both models distort low frequencies more than high frequencies (which is the opposite as FR NIC on the snow corruption in Figure 4). At both severities, the two models have similar plots of G . VTM is highly effective at reconstructing the glass blur- the magnitude of G is <0.0001 across all frequencies- which matches our findings on the other codecs in Section 4.2.2. Additionally, JPEG2000 and VTM have nearly identical plots of R at both severities. Finally, on the shot noise corruption, both JPEG2000 and VTM display similar magnitudes and patterns of errors. Both models leave significantly more error on G at severity=5 than severity=1; however, JPEG2000 maintains a bpp=0.8 at both severities while VTM increases its bpp from 0.05 to 1.02. Additionally, unlike NIC, both of these models leave their highest errors in the low-frequency domain of G . Lastly, both models are not as effective at denoising high-frequency signals as NIC proved to be.

E Additional results: ELIC

D(C,X)

入=0.003

入=0.006

入=0.025

ELIC

(0.16, 31.32)

(0.25,33.23)

(0.53,36.69)

Q

40

0

150

1

F Additional results: accuracies by individual corruptions

psnr=36.93

psnr ≈32.97

psnr ≈ 31.17

10%

Difference in Accuracy

0%

T

Difference in Accuracy

Difference in Accuracy

-10%

-20%

SH NIC (psnr=36.93)

SH NIC (psnr=32.97)

SH NIC (psnr=31.17)

-30%

ELIC (psnr=37.10)

ELIC (psnr=32.84)

ELIC (psnr=30.98)

JPEG2000 (psnr=36.69)

JPEG2000 (psnr=32.92)

JPEG2000 (psnr=31.24)

-40%

JPEG (psnr=37.11)

JPEG (psnr=33.11)

JPEG (psnr=31.18)

clean

snow

frost

fog

brightness

contrast

motion

wooz

defocus

glass

elastic

jpeg

pixelate

gaussian

shot

impulse

clean

snow

frost

fog

brightness

contrast

motion

wooz

defocus

elastic

jpeg

pixelate

gaussian

shot

impulse

clean

snow

frost

fog

brightness

contrast

motion

Wooz

defocus

elastic

jpeg

pixelate

gaussian

shot

impulse

G Additional results: Kodak and Kodak-C

This section contains Figures equivalent to those in the main body for the Kodak and Kodak-C dataset.

Original

Snow

700

0

D(C,x):Fixed-bpp

RDCurves

D(C,x):Fixed-psnr

lowbpp

med.bpp

highbpp

42

med.psnr

high psnr

SH NIC

SH NIC

40

(0.28, 30.19)

(0.94, 36.20)

(1.43, 38.22)

38

(0.28, 30.19)

(0.43, 32.12)

(0.76, 34.99)

ELIC

ELIC

PSNR (dB)

36

(0.30, 31.87)

(0.93, 37.30)

(1.49, 40.51)

34

(0.23,30.78)

(0.30, 31.87)

(0.53, 34.45)

JPEG2000

JPEG2000

32

(0.27,28.76)

(0.92,34.25)

(1.50, 36.97)

30

SH NIC

(0.39, 30.16)

(0.60,32.08)

(1.09, 35.14)

JPEG

ELIC

JPEG

JPEG2000

28

JPEG

(0.28, 25.86)

(0.96, 32.51)

(1.37, 34.52)

(0.59,29.89)

(0.91, 32.17)

(1.57, 35.37)

0.0

0.5

1.0

1.5

2.0

0

1000

15 0

12

Image size (bpp)

0

600

300

20

Lowfreq.:Snow

Med.freq.:Glass Blur

High freq.:Shot Noise

40.0

24

SH NIC

(aP)

37.5

w.r.t.c(x)(dB)

45

PSNR w.r.t. c(x) (dB)

2 20 18 16 14 1

ELIC

35.0

JPEG2000

PSNR w.r.t. c(x)(

32.5

40

JPEG

27.5

30.0

SH NIC

PSNR

35

SH NIC

ELIC

ELIC

25.0

JPEG2000

30

JPEG2000

12

22.5

JPEG

JPEG

25

10七

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

0

2

4

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

Low freq.: Snow

Med.freq.:Glass Blur

Highfreq.:ShotNoise

25

SHNIC

SHNIC

27.5

SHNIC

14

ELIC

ELIC

ELIC

PSNR w.r.t. x (dB)

JPEG2000

v.r.t. x (dB)

24

JPEG2000

JPEG2000

JPEG

JPEG

22.5 -

JPEG

12

PSNR w.r.t.

23+

20.0

10

PSNR

17.5

22

15.0

8

21 +

12.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0

0.5

1.0

1.5

0

2

4

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

G(C,x,c)

R(C,x,c)

SHNIC

JPEG2000

SH NIC

JPEG2000

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

10

50

500

500

Snow

0

0

0

0

(1.4, 34.0)

(1.5, 33.2)

(0.9, 30.3)

(0.9, 29.9)

(1.4, 15.2)

(1.5, 7.9)

(0.9, 14.9)

(0.9, 7.8)

0.1

0.1

500

500

Blur

0.0

0.0

0

0

(0.4, 39.8)

(0.2, 43.9)

(0.9, 40.5)

(0.9, 46.6)

(0.4, 24.8)

(0.2, 22.5)

(0.9, 24.8)

(0.9, 22.5)

700

700

700

700

Shot

Noise

0

0

0

0

(1.9, 22.9)

(2.7, 11.5)

(0.9, 21.7)

(0.9, 10.8)

(1.9, 27.0)

(2.7, 15.4)

(0.9, 24.6)

(0.9, 14.1)

H Additional results: other CLIC-C corruptions

Figures 20, 21, and 22 show the results of the other 12 corruptions from CLIC-C which were not included in the main body. Overall, we see that our three representative corruptions (snow, glass blur, and shot noise) represent the trends of the low, medium and high frequency corruptions well.

Motion

Zoom

Snow

Frost

Fog

Brightness

Contrast

blur

blur

500

0

Defocus

Elastic

JPEG

Shot

Impulse

blur

blur

transform

compression

Pixelate

noise

noise

noise

500

0

Low freq.:Frost

Low freq.: Fog

Low freq.:Brightness

Low freq.:Contrast

42.5

45

40.0

50

PSNR w.r.t. c(x) (dB)

PSNR w.r.t. c(x) (dB)

PSNR w.r.t. c(x) (dB)

45

35.0 -

40

(t)

35.0

PSNR w.r.t.c

32.5

40

32.5

30.0

SH NIC

35

JIN HS

30.0

SHNIC

35

SH NIC

ELIC

ELIC

ELIC

ELIC

27.5

JPEG2000

JPEG2000

27.5

JPEG2000

JPEG2000

25.0

JPEG

JPEG

25.0

JPEG

JPEG

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

Low freq.: Frost

Low freq.: Fog

Low freq.: Brightness

Lowfreq.:Contrast

13.0

22

17

PSNR w.r.t. x(dB)

12.5

w.r.t.x(dB)

14.0

PSNR w.r.t.  (dB)

20

PSNR w.r.t. x (dB)

16

12.0

SH NIC

13.5

SHNIC

18-

SHNIC

SH NIC

ELIC

ELIC

16-

ELIC

15

ELIC

11.5

JPEG2000

13.0

JPEG2000

JPEG2000

JPEG2000

11.0 -

JPEG

PSNR

12.5

JPEG

14

JPEG

14

JPEG

12

10.5

12.0

10

13

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

Med.freq.:Zoom Blur

Med.freq.:Defocus Blur

Med.freq.:ElasticTransfor

PSNR w.r.t. c(x) (dB)

45

PSNR w.r.t. c(x) (dB)

45

50

.c(x)(dB)

40.0

42.5

40

40

PSNR w.r.t. c(x) (

45

37.5

40

PSNR w.r.t. c

35.0

35

35

32.5

JIN HS

SH NIC

35

SHNIC

SH NIC

ELIC

ELIC

ELIC

30.0

ELIC

30

JPEG2000

30

JPEG2000

JPEG2000

27.5

JPEG2000

JPEG

JPEG

JPEG

25.0

JPEG

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

Med.freq.:Motion Blur

Med.freq.:Elastic Transforr

26

20.0

SH NIC

SHNIC

25

19.5

ELIC

27 -

25

ELIC

PSNR w.r.t. x (dB)

24 2 2 24

PSNR w.r.t. x(dB)

JPEG2000

PSNR w.r.t. x (dB)

26

PSNR w.r.t. x (dB)

JPEG2000

SH NIC

19.0

JPEG

25

24

JPEG

ELIC

JPEG2000

18.5

23

JPEG

18.0

23-

SH NIC

ELIC

22

20

17.5

22

JPEG

JPEG2000

21

19

17.0

21

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

Med. freq.: Pixelate

High freq.:Impulse Noise

45

24

SH NIC

22

PSNR w.r.t. c(x) (dB)

w.r.t.c(x)(dB)

22

ELIC

PSNR w.r.t. c(x) (dB)

40

20

JPEG

JPEG2000

20

18

SH NIC

35

ELIC

35

16

JPEG2000

30

SH NIC

ELIC

30

ELIC

SH NIC

PSNR

14

16-

14 -

JPEG

25-

JPEG

JPEG2000

25

JPEG

JPEG2000

12

12

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

0

2

4

6

0

2

4

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

Med.freq.:JPEGCompressio

Med.freq.:Pixelate

High freq.:Gaussian Noise

High freq.:Impulse Noise

30.0

SH NIC

27.5

SHNIC

PSNR w.r.t.  (dB)

30

(ap)x

(ap)

27.5

ELIC

JPEG2000

PSNR w.r.t. x(dB)

25.0

JPEG2000

ELIC

25.0

JPEG

22.5

JPEG

28-

PSNR w.r.t.

w.r.t.

22.5

20.0

20.0

26

SH NIC

26

SHNIC

PSNR

17.5

ELIC

ELIC

17.5

24

JPEG

JPEG2000

24

JPEG2000

JPEG

15.0

12.5

+

15.0

12.5

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

2.0

0

2

4

6

0

2

4

6

Image size (bpp)

Image size (bpp)

Image size (bpp)

Image size (bpp)

G(C,x,c)

SH NIC

ELIC

JPEG2000

JPEG

sev.=1

sev.=5

sev.=1

sev.=5

sev.=1

sev.=5

10

sev.=1

sev.=5

15

Frost

(0.8, 36.4)

(0.6, 37.9)

(0.8, 37.8)

(0.6, 39.0)

0

(0.8, 35.4)

(0.8, 37.7)

0

(0.8, 32.9)

(0.7, 35.0)

0.3

Fog

60

Fog

Fog

(0.4, 40.6)

(0.3, 42.1)

(0.3, 41.4)

(0.2, 42.8)

-0.0

(0.8, 41.4)

(0.8, 43.8)

0

(0.4, 37.5)

(0.3, 39.6)

Brightness

15

Brightness

10

Brightness

15

Brightness

25

(0.8, 36.2)

(0.8, 34.1)

0

(0.8, 37.4)

(0.8, 35.5)

(0.8, 34.8)

(0.8, 33.9)

0

(0.8, 32.4)

(0.7, 31.0)

0.3

Contrast

Contrast

Contrast

Contrast

(0.4, 40.7)

(0.1, 47.1)

(0.3, 41.4)

(0.0, 47.9)

0.0

(0.8, 41.3)

(0.8, 49.5)

0

(0.4, 37.4)

(0.2, 44.4)

0.5

0.1

0.5

Motion

Motion

(0.4, 40.1)

(0.2, 42.7)

0.0

(0.4, 40.9)

(0.2, 43.5)

0.0

(0.8, 40.3)

(0.8, 44.5)

0.0

(0.6, 36.4)

(0.4, 39.5)

0.1

0.01

0.1

Zoom

(0.3, 41.7)

(0.3, 41.3)

0.0

(0.2, 42.4)

(0.2, 42.0)

0.00

(0.8, 43.3)

(0.8, 42.9)

0.0

(0.4, 38.9)

(0.4, 39.0)

Defocus

0.1

Defocus

0.001

Defocus

0.1

Defocus

0.1

(0.3, 42.7)

(0.1, 46.3)

0.0

(0.2, 43.6)

(0.1, 47.3)

0.000

(0.8, 44.4)

(0.8, 48.7)

0.0

(0.5, 38.9)

(0.3, 41.9)

0.0

Transform

3

Elastic

Transform

0.5

Transform

Transform

10

(0.6, 38.3)

(0.7, 38.2)

(0.6, 39.2)

(0.6, 39.1)

0.0

(0.8, 37.2)

(0.8, 37.0)

Q

(0.7, 34.3)

(0.7, 34.2)

Compression

Compression

E

Compression

Compression

JPEG

JPEG

JPEG

JPEG

(0.7, 37.3)

(0.7, 37.2)

(0.7, 38.4)

(0.7, 38.6)

(0.8, 37.3)

(0.8,39.3)

(0.7, 36.2)

(0.6, 37.7)

Pixelate

Pixelate

Pixelate

Pixelate

(0.7, 36.6)

(0.5, 38.5)

(0.7, 38.0)

(0.5, 39.5)

(0.8, 36.4)

(0.8, 39.4)

0

(0.7, 33.2)

(0.5, 36.1)

issne

Noise

700

Noise

500

Noise

700

Noise

700

(1.8, 23.2)

(2.7, 12.1)

0

(2.2, 23.6)

(4.1, 12.6)

(0.8, 22.0)

(0.8, 11.3)

(1.2, 22.2)

(3.1, 12.0)

0

Impulse

Noise

700

Impulse

Noise

500

Impulse

Noise

700

Impulse

Noise

700

(2.0, 21.3)

(2.7, 12.0)

(2.5, 22.0)

(4.1, 12.5)

0

(0.8, 20.8)

(0.8, 11.2)

0

(1.4, 20.1)

(3.1, 11.8)

R(C,x,c)

SHNIC

ELIC

JPEG2000

JPEG

sev.=1

sev.=5

500

sev.=1

sev.=5

500

sev.=1

sev.=5

500

sev.=1

sev.=5

500

Frost

Frost

Frost

Frost

(0.8,13.0)

(0.6, 10.5)

0

(0.8, 12.9)

(0.6, 10.5)

0

(0.8, 12.8)

(0.8, 10.4)

0

(0.8, 12.9)

(0.7, 10.5)

0

500

500

500

500

Fog

Fog

Fog

Fog

(0.4, 14.3)

(0.3, 12.2)

0

(0.3, 14.3)

(0.2, 12.2)

0

(0.8, 14.3)

(0.8, 12.2)

0

(0.4, 14.3)

(0.3, 12.2)

Brightness

500

Brightness

500

Brightness

500

Brightness

500

(0.8, 21.9)

(0.8, 10.0)

0

(0.8, 21.8)

(0.8, 9.9)

0

(0.8, 21.3)

(0.8, 9.9)

0

(0.8, 21.4)

(0.7, 9.9)

0

500

500

500

500

Contrast

Contrast

Contrast

Contrast

(0.4, 16.7)

(0.1, 12.8)

0

(0.3, 16.8)

(0.0,12.8)

0

(0.8, 16.8)

(0.8, 12.9)

0

(0.4, 16.7)

(0.2, 12.8)

0

500

500

500

500

Motion

(0.4, 25.6)

(0.2, 20.1)

0

(0.4, 25.7)

(0.2, 20.1)

0

(0.8, 25.7)

(0.8, 20.1)

0

(0.6, 25.4)

(0.4, 20.1)

0

500

500

500

500

Zoom

Zoom

(0.3, 19.8)

(0.3, 17.7)

0

(0.2, 19.8)

(0.2, 17.7)

0

(0.8, 19.9)

(0.8, 17.8)

0

(0.4, 19.8)

(0.4, 17.7)

0

500

500

500

500

Defocus

Defocus

s

Defocus

(0.3, 27.4)

(0.1, 22.8)

(0.2, 27.6)

(0.1, 22.8)

0

(0.8, 27.5)

(0.8, 22.8)

0

(0.5, 27.2)

(0.3, 22.7)

0

Transform

500

Transform

500

Transform

500

Transform

500

(0.6, 25.6)

(0.7, 21.7)

0

(0.6, 25.6)

(0.6, 21.6)

(0.8, 25.5)

(0.8, 21.6)

0

(0.7, 25.4)

(0.7, 21.6)

0

JPEG

Compression

100

JPEG

Compression

100

JPEG

Compression

100

JPEG

Compression

100

(0.7, 31.5)

(0.7, 26.5)

Q

(0.7, 31.3)

(0.7, 26.3)

0

(0.8, 30.5)

(0.8, 26.1)

0

(0.7, 30.3)

(0.6, 26.1)

0

Pixelate

500

Pixelate

500

Pixelate

500

Pixelate

500

(0.7, 30.9)

(0.5, 26.2)

(0.7, 30.6)

(0.5, 26.1)

(0.8, 30.0)

(0.8, 25.9)

0

(0.7, 29.5)

(0.5, 25.8)

0

Noise

500

Noise

500

Noise

500

Noise

500

(1.8, 27.8)

(2.7, 15.6)

(2.2, 26.9)

(4.1, 14.6)

(0.8, 25.8)

(0.8, 14.8)

0

(1.2, 26.5)

(3.1, 12.9)

0

600

600

600

600

asnd

Noise

Impulse

Noise

Impulse

Noise

Impulse

Noise

(2.0, 24.9)

(2.7, 15.4)

0

(2.5, 24.1)

(4.1, 14.4)

0

(0.8, 21.3)

(0.8, 14.3)

(1.4, 23.8)

(3.1, 12.7)

0

I Training setup and hyperparameters

We train NIC models using the train split of the CLIC 2020 dataset with batches of size 8 and random crops of size 256 × 256 . For SH NIC, we set N = M = 192 and train for 5,000 epochs (about 1M iterations). We trained 8 models with λ s 0.0012, 0.005, 0.01, 0.03, 0.05, 0.1, 0.15, and 0.26. We used the pytorch model architecture in the compressai repository [10]. For ELIC, we use N = 192 , M = 320 and train for 3,900 epochs. We trained 11 models with unique λ s (0.001, 0.0025, 0.003, 0.004, 0.006, 0.008, 0.016, 0.025, 0.032, 0.05, 0.15). We utilized this repository for the model architecture, but trained our own models on CLIC rather than experimenting with the publicly available checkpoints trained on ImageNet. Note that we trained more models for ELIC than FR NIC in order to obtain bpps and PSNRs which fit our fixed-bpp/PSNR constraint.

For the variable-rate models, we augment each convolutional layer with a FiLM layer which consists of two fully-connected layers and 128 hidden features. We train VR models for 10,000 epochs (about 2M iterations) and sample λ from a log-uniform distribution over [0.0012, 0.26]. For pruned models, we use global gradual magnitude pruning (GMP) between epochs 675 and 2400 and tune the remaining weights from epochs 2401-10,000. Fixed-rate MS-SSIM models are trained over several values of λ ∈ [1 , 1000] .

J Variable-rate model details

For our variable-rate NIC we implemented a model proposed by Dosovitskiy and Djolonga [17] which uses Loss Conditional Training (LCT) to train a variable-rate version of the scale-hyperprior NIC model from [9]. The model is trained over a continuous range of compression rates, optimizing the rate-distortion tradeoff at each. This reduces the computational redundancy involved in training separate models for the same task at different points along the rate-distortion curve. Additionally, once trained, this model can be adaptively used at any compression rate in the range, which makes it even more flexible than a discrete set of fixed-rate models.

In order to condition the compression rate via LCT, all convolutional layers in the model are augmented with Feature-wise Linear Modulation (FiLM) layers [49]. In this case, these are small neural networks that take a conditioning parameter λ as input and output a µ and σ used to modulate the activations channel-wise based on the value of λ . More specifically, suppose a layer in the CNN has activations f of size W × H × C . In Loss Conditional Training (LCT), these activations are augmented by µ and σ as follows,

where both µ and σ are vectors of size C .

Figure 24 shows a diagram of FiLM layers being applied to a small example network. Normally, without FiLM layers, the equation to calculate the features maps is

where ρ is the activation function ( e.g. , ReLU), W are the weights and b are the biases in the layer. However, with FiLM layers, the weighted sum is first subject to augmentation with µ and σ . Thus, the new calculation is

During training, a random λ is drawn with each mini-batch and used in two ways: 1) as input to the FiLM layers to modulate the feature maps and 2) in the mini-batch's loss function. During inference, the model takes as input a desired compression rate λ along with the input image. Notably, LCT and FiLM layers only affect the values of the activations ( i.e. , feature maps), but not the weights on the convolutional layers or activation functions. This is an essential observation for applying pruning to a network with LCT.

K Pruned model details

One limitation of using NIC in the wild is the size of the NIC models. These models require several million learned weights and efficient inference requires specialized GPUs. This limits the possibility

入

21

C2

4

W(1)

x =∞(0)

x(1)

x(2)

of using NIC on an edge device, like a drone or a cell phone. In this section, we demonstrate that NIC models themselves can be approximated ( i.e. , we can compress the compressor) and we propose a sparse versions of both fixed-rate and variable-rate NIC models.

To prune our NIC models, we propose gradual magnitude pruning (GMP) [70] based approach which has been demonstrated to effectively sparsify DNNs. At a high level, GMP gradually prunes DNN weights based on their magnitude over a fixed number of epochs then continues to train, or fine-tune, the remaining sparse network after the target sparsity is reached. The pruning can be done layer-wise ( i.e. , the mask must be updated such that each layer has the target sparsity) or globally ( i.e. , the overall network must have the target sparsity, but the sparsities can differ between layers). We utilized global pruning, which has been demonstrated to produce more performant sparse models.

To implement GMP, each layer is given binary mask of the same size as the original layer, which specifies which weights have been pruned ( i.e. , set to 0). Every ∆ t iterations, the mask is updated based on the magnitudes of the weights, such that the network has a desired sparsity. Suppose s i is the initial sparsity (usually 0) at time t 0 and s t is the target sparsity (0.8 or 0.95 in our case) which we want to reach after n steps, then the target sparsity at time step t is

The target sparsity equation is found to be especially effective because it causes weights to be pruned more quickly while the learning rate is still high and the weight-training on non-pruned weights can recover. After the desired sparsity is reached, the sparse network is trained for several more iterations. The pruning can be done layer-wise ( i.e. , the mask must be updated such that each layer has the target sparsity) or globally ( i.e. , the overall network must have the target sparsity, but the sparsities can differ between layers). Note that the network still performs weight training on the non-pruned weights, as usual, using an optimizer like stochastic gradient descent.

L Mathematical details

L.1 Derivation of lemma 5.1

Let X ∈ R N × p be a dataset (whose datapoints are the rows) with mean 0 and covariance Σ = 1 N X T X . Consider the problem of linearly auto-encoding X with latent space R r , where r ≤ p :

It is a theorem of Eckart and Young [18] that for any solution W 1 , W 2 of eq. (6) the product C = W 2 W 1 is an orthogonal projection onto the subspace spanned by the top r principal components of X (see also [7] and for more recent developments and additional references, [50, 35]). Informally, this theorem says 'linear autoencoders perform PCA.' Formally, if X = USV T is an SVD and V [: r ] is the first r rows of V , then W 2 W 1 = V [: r ] T V [: r ] . 8 It follows that the reconstruction error L ( X ) on a new input X ∈ R p admits a simple closed form: let V 1 , . . . , V p be the principal components of X (rows of V above), and write

Suppose now that X is a dataset of natural images. We will replace the single index i for X = ( X 1 , . . . , X p ) ∈ R p with 3 indices h, i, j for an image tensor X = ( X h,i,j ) ∈ R KHW (here h is viewed as the channel index and i, j as the spatial indices); we will also commit mild abuse of notation by letting the matrices W i act on flattened (vectorized) tensors. Let F : R KHW → R KHW denote the spatial discrete Fourier transform. We introduce the notation

and we will frequently use the shorthand 'hat' decoration ' ̂ -' to denote F as well, for ease of notation. Just as C projects onto the subspace spanned by the top r principal components of X , ˆ C projects onto the subspace spanned by the top r principal components of ˆ X .

It has long been known that when transformed to (spatial) Fourier frequency space, there are reasonable (coarse) descriptions of the statistical distribution of natural images. Its variances in spatial frequencies ( i, j ) can be modelled as 1 | i | α + | j | β where α, β ≈ 2 and its covariances drop off rapidly away from the diagonal [12, 1, 36]. 9 These two facts motivate the following assumptions:

(I) The principal components of ˆ X are roughly aligned with the spatial Fourier frequency basis.

(II) The associated variances are monotonically decreasing with frequency magnitude (more specifically according to the power law 1 | i | α + | j | β ).

Assuming α = β = 2 , the above would suggest that ˆ C projects a Fourier transformed image ˆ X = FX onto Fourier basis vectors corresponding to spatial frequencies ( i, j ) with i 2 + j 2 ≤ r πK . That is,

where ˆ X : ,ij denotes the components of ˆ X corresponding to spatial frequency ( i, j ) (this is a vector of dimension K , the number of channels in the image dataset). Together with the identity F C X = ˆ C ˆ X , eq. (9) yields lemma 5.1.

Remark L.1 . Note that lemma 5.1 also suggests that such autoencoder compressors behave qualitatively similarly to classical codecs such as JPEG, which involves a discrete cosine transform (DCT) followed by removal of high-frequency components [53]. While design of hand-crafted codecs like JPEG was no doubt driven by empirical observations of the statistics of natural images, the NIC models in our experiments (and at least in a theoretical sense the autoencoder model appearing in lemma 5.1) are machine learned from natural image datasets.

Below, we use the above analysis to get some qualitative insights into the robustness and generalization metrics R and G introduced in section 3. In what follows let c : R KHW → R KHW be a corruption transformation.

L.2 Derivations of corollaries 5.2 and 5.3

Robustness : here we are interested in PSD( X - C ( c ( X ))) , which is simply the coordinatewise absolute value of ˆ X -̂ C ( c ( X )) . Using the fact that in our simple model C is linear, we can expand like

so that for any frequency ( i, j )

Taking square roots, we see that

averaging over the dataset X gives corollary 5.2. In particular, the cross term 2( ˆ X -ˆ C ( ˆ X )) : ,ij ˆ C ( ̂ c ( X ) -ˆ X ) : ,ij is identically 0. This suggests that autoencoder compressors trained on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that | ( ̂ c ( X ) -ˆ X ) : ,ij | 2 is large for small values of ij ).

Generalization : here we are interested in PSD( c ( X ) -C ( c ( X ))) , which is simply the coordinatewise absolute value of ̂ c ( X ) -̂ C ( c ( X )) . Again using the fact that in our simple model C is linear, we can expand like

From this we obtain

Taking square roots and relating back to power spectral density, this says

and averaging over the dataset X gives corollary 5.3. Equation (14) is more involved than eq. (11) in particular, the 'cross term' 2 ˆ X : ,ij ( ̂ c ( X ) -ˆ X ) : ,ij is in general non-zero. However, there are many cases where at least an expectation over the data set ˆ X the term 2 ˆ X : ,ij ( ̂ c ( X ) -ˆ X ) : ,ij vanishes (for example, when c is additive noise, or more generally when ̂ c ( X ) -ˆ X is statistically independent of ˆ X .). In such cases, we can see that autoencoder compressors trained on natural images generalize less successfully to corruptions with large amplitude in high frequencies .

L.3 Autoencoder reconstruction error and data density more generally

Our findings on generalization (both the experimental results and analysis with linear autoencoders) is closely related to the widely observed inverse correlation between autoencoder reconstruction error and probability density. We will take the recent [29] as a jumping-off point. However, it is worth noting that this observation (or at least something similar to it) has been around for some time, at least since [5], and forms the basis for widespread use of autoencoders in anomaly detection.

Formally, [29, Observation 2] states that:

( ∗ ) For an autoencoder C the reconstruction error |C ( X ) -X | 2 is positively correlated with 1 p ( X ) , the probability density of the input data at X .

If we assume this observation holds, then some of our experimental findings regarding generalization to corrupted data admit a simple explanation: as discussed above, the probability density of natural images is heavily concentrated along low (spatial) Fourier frequencies. Hence for a corruption c such that c ( X ) -X is concentrated in high frequencies, it is reasonable to suspect that p ( c ( X )) < p ( X ) , and given the assumed positive correlation of 1 p ( X ) with |C ( X ) -X | 2 , that

i.e. the reconstruction error of the corrupted datapoint, related to our generalization metric, is higher than that of the clean datapoint X .

Remark L.2 . In some sense, ( ∗ ) is a simple consequence of learning via risk minimization. Let C : M → M be an autoencoder on a manifold M with finite volume (for example, the image hypercube [0 , 1] CHW ). Let ℓ ( C ( X ) , X ) be a reconstruction loss on a point X ∈ M and let p ( X ) be the probability density of input data on M , assumed to be positive everywhere (otherwise, discussing correlation with 1 p ( X ) is troublesome). The risk in this situation is

Note that ℓ ( C ( X ) , X ) and 1 p ( X ) will be positively correlated provided that their covariance is positive. By definition this covariance is

Since at present we only care about the sign of the covariance, we can divide by vol M to obtain

the first term is the risk of C with respect to the uniform distribution on M . In words, eq. (19) is positive whenever C performs better on the distribution p ( X ) than the uniform distribution, and this outcome is to be expected for an autoencoder with sufficient capacity trained on a non-uniform distribution p ( X ) .

Remark L.3 . With notation as in the previous example, the result [4, Thm. 2] shows that the solution obtained with calculus of variations applied to an objective of the form

(i.e. mean squared error loss and an ℓ 2 penalty on gradient norms) satisfies

In particular, the reconstruction error |C ( X ) -X | 2 is proportional to the gradient norm |∇ X log p ( X ) | 2 (up to higher order terms in σ 2 as σ → 0 ). Since

this result sheds a more precise and quantitative light on ( ∗ ) .

Equation (22) may also shed some light on robustness of C as measured by R : indeed, if we assume c ( X ) -X is small,

Applying eqs. (21) and (22) gives the approximation

as σ → 0 . Taking norms squared we get

as σ → 0 . If we again make the simplifying assumption that c ( X ) -X is random and independent of X , then after taking the expectation over c (here denoted by E c ) we obtain

as σ → 0 . If we let α 1 ≥ α 2 ≥ · · · ≥ α p be the eigenvalues of Hess X log p ( X ) and choose σ small enough that 1 + σ 2 α i > 0 for all i , we see that the norm

surpresses (resp. magnifies ) contributions from components of c ( X ) -X along the eigenvectors associated with α i for large (resp. small) i . Note that it is reasonable as a first approximation to expect Hess X log p ( X ) to be negative definite (this is precisely the case when p is a Gaussian distribution, and more generally the requirements p ( X ) ≥ 0 and ∫ p ( X ) dX = 1 put restrictions on the extent to which Hess X log p ( X ) can exhibit non-negative eigenvalues). In this case the large (resp. small) values of i correspond to the eigenvectors in directions of 'sharpest' (resp. 'shallowest') curvature of the graph of p ( X ) . Explicitly, if X is Gaussian with covariance Σ , then Hess X log p ( X ) = Σ -1 , and if λ 1 ≥ λ 2 ≥ · · · ≥ λ p are the eigenvalues of Σ then α i = -1 λ i for all i . In this case we see that the norm in eq. (27) surpresses (resp. magnifies ) contributions from components of c ( X ) -X along principal components with low (resp. high) variance. Note that this provides a takeaway qualitatively similar to that of corollary 5.2.

L.4 A fundamental robustness-generalization inequality

The similarity between the R heatmaps for glass blur in fig. 4 and the PSDs of the corruptions themselves appearing in fig. 1a has a simple explanation in terms of a fundamental relationship between D , G and R . One can show (via two applications of the triangle inequality) that

In the case of glass blur, generalization error is very low, so eq. (28) reduces to R ≈ 1 N ∑ N k =1 PSD ( X k -c ( X k )) , where the right-hand side is the average PSD of the corruption itself.

M Tables of classification accuracies

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to adopt spectral tools to analysis OOD issues of NIC. On the proposed corrupted dataset, the paper investigates the error of both conventional and learning-based image compression approaches and provides some interesting conclusions, showing robustness and generalization of NIC and suggesting insights of design a better NIC model in the wild.\n\nSTRENGTHS:\nThe paper proposes novel tools and benchmarks to thoroughly study NIC from the OOD perspective. I appreciate the delicate figures which quite help me understand the studies. The analysis and conclusions are reasonable and the findings are inspiring, rehealing missed parts of NIC design and assessment. Overall I believe this paper can technically contribute to the community.\n\nWEAKNESSES:\n1. My major concern is the generalization of the conclusion. All the FR experiments are conducted on the hyperprior model (Ball'e et al., 2018), which is a somewhat outdated baseline. Many more powerful approaches have been proposed in the past years, involving more complex context models (Minnen et al., 2018, [40]), improved mixed quantization estimators ([e1, e2, e3]), and non-convolution network structures ([e2, e4, e5]). Shall we observe some different spectral properties when using those transformer-based models?\nAlso, the wavelet-based JPEG2000 is neither a SOTA handcraft codec nor a worldwide popular choice. Though conventional coders don't play key roles in this paper, I suggest the authors further discuss HEIF, AVIF or VVC(IF), and JPEG. \n2. As discussed by the authors, the spectral preference of NIC may be strongly related to the MSE objective. This can be explained as it corresponds to a Gaussian likelihood, according to [Ball'e et al., 2018]. However, this loss function can be replaced to MS-SSIM, GAN and LPIPS etc. For instance, refer to perceptual optimized NIC approaches [e7]. Can this OOD drawback get suppressed when consider those objectives? More discussions are expected.\n3. Though the paper provides a detailed analysis with proposed tools and benchmarks, It do not provide new approaches to solve the concerned OOD issues. Without more practical suggestions, I cannot strongly appreciate the findings for above-mentioned concern about generalization. \n\nP.S. OOD in NIC is not a brand new topic. The authors may want to discuss more related works eg [e6].\n\n[e1] Channel-wise autoregressive entropy models for learned image compression, Minnen et al.\n[e2] Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding, He et al.\n[e3] Transformer-based transform coding, Zhu et al.\n[e4] The Devil Is in the Details: Window-based Attention for Image Compression, Zou et al.\n[e5] Learned image compression with mixed transformer-cnn architectures, Liu et al. *(concurrent work)*\n[e6] On the Out-of-distribution Generalization of Probabilistic Image Modelling, https://openreview.net/pdf?id=q1yLPNF0UFV\n[e7] High-fidelity generative image compression\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a benchmark for evaluating the out-of-distribution (OOD) performance of neural image compression (NIC) models. The authors observe distinguished spectral biases of NIC models by comparing them with JPEG2000. Then the authors listed all the spectral biases along with a theoretical analysis from the perspective of principle component decomposition.\n\nSTRENGTHS:\n1. This paper proposes an interesting perspective for analyzing NIC models.\n\n2. The observed spectral biases are potential to further improve NIC models.\n\n3. The authors provide thorough analysis and enough details for reproducing the results.\n\nWEAKNESSES:\nI believe the spectral perspective is important to understand NIC models. However,  the theoretical analysis is of limited novelty and the analysis and conclusions provided in this paper are empirical and need to be further improved before publication. Moreover, I am afraid that classifications on several corruptions may conflict with the common sense in signal processing field. My concerns are listed as below.\n\n1. The proposed method mainly compares with JPEG2000, which may be inappropriate. JPEG2000 is based on discrete wavelet transform. However, EBCOT and rate control of JPEG2000 could hinder an explicit understanding of the spectral characterization of JPEG2000. I suggest the authors to conduct the comparison experiments on JPEG (which is based on block-based DCT and simple quantization table) for clear comparison. Compared to JPEG2000, JPEG has been widely applied in practice.\n\n2. The comparison presented in Figures 16 and 17 (i.e., comparison to VTM provided in supplementary material) is confusing. $\\mathcal{D}$ for VTM seems to be zero but it is a lossy compression method and the reconstruction is still far from lossless (i.e., 37.42 dB).\n\n3. I also have problem with the evaluation using power spectral density (SPD). I noticed that the authors classify Gaussian noise as a high-frequency corruption in Figure 1. However, it is well known that the Gaussian noise, i.e., white noise, has a constant SPD. In other words, the effect of Gaussian noise is expected to be constant rather than constrained on the high-frequency components. Moreover, the effect of JPEG compression is also expected to be on both median and high-frequency, as the default quantization map results in a larger quantization step on the high-frequency components. I am afraid that the classification of other corruptions may neither be sufficiently precise. I guess the gaps between above theoretical analysis and the empirical analysis in Figure 1 originate from the intensity difference on the SPD used to analyze the effect of introduced corruption. The natural images themselves are with an unbalanced SPD. Thus, the observation may be biased.\n\n4. The theoretical analysis in Section 5 is not novel enough. The desired characteristics for transform coding are discussed in [R1]. The differences and similarity between DFT, DCT, and KLT are analyzed in [R2]. The provided analysis on linear transforms (autoencoder) may not be appropriate for nonlinear models [R3].\n\n5. Considering that the major contribution of this paper lies on the empirical evaluations, it is important to make thorough validations on diverse datasets and NIC models. The adopted scale hyperprior is not sufficient supporting the claims in this paper.\n\n[R1] V. K. Goyal, \"Theoretical foundations of transform coding.\" IEEE Signal Processing Magazine, vol. 18, no. 5, pp. 9-21, 2001.\n\n[R2] D. S. Taubman and M. W. Marcellin, JPEG2000: Image compression fundamentals, standards and practice. 2002.\n\n[R3] J. Ballé, et al. \"Nonlinear transform coding.\" IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 339-353, 2021.\n\nQUESTIONS:\n1. Could the authors offer a comparison with JPEG compressed images? I believe it would be more straightforward.\n\n2. What is the reason for theoretical analysis (given in comment #3 in the section of Weakness) and the given classification (Figure 1) on Gaussian noise and JPEG compression artifacts?\n\n3. What is the difference between the theories for nonlinear transforms (Section 5) and those for traditional linear transform (such as DWT, DCT, and KLT)?\n\nLIMITATIONS:\nLimitations are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes neural image compression models relative to classical (hand-engineered) methods using a new dataset and new tools based on spectral analysis. Specifically the paper presents (from Section 1, \"Main Contributions\"):\n(1) a new dataset covering different kinds of distortions used to compare methods\n(2) an inspection tool for distortions that shows where, in the frequency domain, different distortions primarily introduce error\n(3) an inspection tool based on spectral analysis that provides a visualization showing which frequency ranges are most distorted by different compression methods\n(4) some theoretical results about neural compression, albeit limited to the linear autoencoder case\n\nThe main findings are also summarized in the intro:\n(1) Neural image compression (NIC) produces different spectral artifacts than classical codecs (JPEG 2000 was used as the classical codec and a scale-hyperprior model was used for NIC)\n(2) NIC and classical codecs prioritize different frequencies as the bit rate changes\n(3) NIC and classical codecs fail to generalize in similar ways but succeed in different ways\n(4) Due to the differences, it can be hard to choose the best codec for a particular application without knowing what kind of spectral distortions will be present at inference time (i.e., out of distribution)\n\nSTRENGTHS:\nAlthough compression systems, neural and hand-engineered, are frequently assessed in terms of visual quality, this paper is the first I've seen that tries to understand how distortions differ between neural and standard methods. The authors develop a new dataset built by applying 15 different kinds of distortions to images from standard evaluation sets (kodak and clic). They then visualize the power spectral density (PSD) of the distorted images (to understand which frequency bands are affected by each type of distortion) and they visualize the distortions caused by lossy compression for a neural method and for JPEG 2000.\n\nThis analysis leads to interesting observations about how the two codecs behave differently, e.g. neural image codecs (NICs) distort high frequencies more than low frequencies (Section 4.1), and they are more sensitive to high frequency noise/distortions. I haven't previously this kind of analysis or conclusion before.\n\nSimilarly, the authors analyze how classification accuracy degrades when the two types of codecs are used on images with different kinds of distortion (Fig. 5). These findings are particularly interesting because neither codec is preferable for all distortions and bit rates, e.g., JPEG2000 takes a large hit for \"high\" distortions at high bit rates but takes a smaller hit than NICs for \"low\" distortions at moderate bit rates.\n\nWEAKNESSES:\nThere are two main weaknesses with the paper. The first is whether the results from studying these two codecs generalize to other codecs in the respective classes. For example, will VCC (a modern codec still be finalized) behave similarly to JPEG2000? Will SOTA neural methods behave similarly to [9]. The latter question is further complicated because there are multiple ways the NICs differ, e.g. (1) more sophisticated models that build on [9] (like adding autoregressive priors, architectures built using transformers instead of CNNs, etc.), (2) using \"implicit neural representations\" instead of the \"nonlinear transform coding\" approach of [9], and (3) optimizing networks with an adversarial loss that targets \"realism\" (minimizing a divergence between distributions, not just a measure of distortion between the original and reconstructed pixel values). It's hard to say if the findings in this paper will be the same for a model that is different according to one of (or all of) these dimensions.\n\nSecond, I'm not sure how important these findings are to the NeurIPS community. Neural image compression is certainly an appropriate topic for NeurIPS, but papers typically focus on new models or new applications. So the target audience may be fairly small, and the observations don't directly imply an architecture change (or loss / optimization change) to improve NICs.\n\nSince I've worked in this subfield, I find the results of this paper fascinating. And at a conference focused on compression or image processing (something like ICIP or DCC), I'd rate this paper as a 7+. I'm open to feedback from the AC or other reviewers that I'm off on my assessment of the fit at NeurIPS.\n\nQUESTIONS:\nNo questions beyond the implied ones in the \"weaknesses\" section.\n\nLIMITATIONS:\nadequately covered", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper addressed the problem of image compression. Neural-based methods have started outperforming traditional codec methods in image compression, but there are no comprehensive datasets and informative tools to evaluate neural-based methods in real-world settings. This paper proposed a benchmark to study this that allows for evaluating out-of-distribution performance. The proposed benchmark provides some findings that are crucial to bridge the real-world adoption of neural-based image compression techniques.

STRENGTHS:
1. Comprehensive datasets for image compression.
2. Understanding of out-of-distribution performance.
3. Extensive evaluations.
4. Theoretical analyses.

WEAKNESSES:
NeRF-based methods can also be seen as a way to do compression (compressing images into. network weights), The authors should also discuss this and benchmark them on the dataset.

QUESTIONS:
Overall, this is a strong paper. I only have some minor comments. See above.

LIMITATIONS:
The paper should address this.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13