Neural Image Compression: Generalization, Robustness, and Spectral Biases
Authors: Kelsey Lieberman
James Diffenderfer
Charles Godfrey
Bhavya Kailkhura
Venue: NeurIPS 2023 poster (2023)
Keywords: image compression
robustness
generalization

Abstract: Recent advances in neural image compression (NIC) have produced models that are starting to outperform classic codecs. While this has led to growing excitement about using NIC in real-world applications, the successful adoption of any machine learning system in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand NIC performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we provide CLIC-C and Kodak-C by introducing 15 corruptions to the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspired inspection tools to gain deeper insight into errors introduced by image compression methods as well as their OOD performance. We then carry out a detailed performance comparison of several classic codecs and NIC variants, revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC and its dependence on the spectral properties of the data. Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing robust and generalizable NIC methods. Code and data will be made available at https://github.com/klieberman/ood_nic.

Neural Image Compression: Generalization, Robustness, and Spectral Biases
Kelsey Lieberman
James Diffenderfer
Department of Computer Science
Lawrence Livermore National Laboratory Livermore, CA USA diffenderrer2211111...
Duke University Durham, NC USA
kelsey. libbemmnkkeeee edu
Charles Godfrey!
Bhavya Kailkhura
Thomson Reuters Labs
Lawrence Livermore National Laboratory
Eagan, MN USA
Livermore, CA USA
charles. www.ooooeeoetoooooooouutrrrsssse
kailkhura@@lnnl.c
Abstract
Recent advances in neural image compression (NIC) have produced models that are starting outperform classic codecs. While this has led growing excitement about using NIC in real-world applications, the successful adoption of any machine learning system in the wild requires generalize (and be robust) unseen distri- bution shifts at deployment. Unfortunately, current research lacks comprehensive datasets and informative tools evaluate and understand NIC performance real-world settings. To bridge this crucial gap, first, this paper presents a compre- hensive benchmark suite evaluate the out-of-distributiio (OOD) performance of image compression methods. Specifically, we provide CLIC-C and Kodak-C by introducing corruptions the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspies inspection tools to gain deeper insight into errors introduced by image compression methods as well as their OOD performance. We then carry out detailed performance comparison of several classic codecs and NIC variants, revealing intriguing findings that challenge our current understanding the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance NIC and its dependence on the spectral properties of the data. Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that work will propel future efforts in designing robust and generalizable NIC methods. Code and data will be made available at htt///oddiiccccc/oo///ooe/eoe/ce nic.
https: // //g/ thub. com com/kllmc/cccccc www.keeeaeaanneeeeeeeee/a corberrmanoodd_.....
Introduction
Consider the Mars Exploration Rover, whose scientific objective is to search for clues to past activity of water (and perhaps life) on Mars. To achieve this, the rover collects images of interesting rocks and soils to be analyzed by the scientists on Earth. Sending these images down the Earth-bound data stream in their original form is too slow and expensive due to limited bandwidth. Thus, it is well accepted that image compression could play a key role in producing scientific breakthroughs [44].
Work done Pacific Northwest National Laboratory.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Employing image compression in such a setting is challenging for three main reasons: 1) a high compression ratio is desired due low communication bandwidth, 2) given the battery-operated nature of these devices, the compression module has to be lightweight so it consumes less memory and power, and 3) robustness and generalization to environmental noises and domain shifts, respectively, are desired due to limited Mars-specific training data. These requirements are not specific to the planetary exploration use case but also arise in a wide range of scientific applications which use image compression in the wild [33].
Recently, neural image compression (NIC) has demonstrated remarkable performance in terms of rate-distortion and runtime overhead on in-distribution (IND) data [9, 46] 46]-satisf requirements 1) and 2). However, there limited work on understanding the out-of-distribution (OŌD) robustness and generalization performance of image compression methods (requirement 3) [42]. Our work is driven by several open fundamental empirical and theoretical questions around this crucial issue.
How can the expected OOD performance of image compression models be reliably assessed? Can we gain a deeper understanding of the modus operandi of different image compression methods? How do training data properties and biases impact data-driven compression methods?
This paper takes a critical view of the state of image compression and makes several contributions toward answering the aforementioned questions. First, we design compre- hensive benchmark datasets for evaluating the OOD performance of image compression methods. Inspired by existing OOD benchmarks for classification and detection [26, 28, 57, 56], we design CLIC-C and Kodak- C by introducing 15 common shifts emulating train- cein-deployment distribution mismatch to the popular CLIC and Kodak datasets. @ Next, we focus on understanding the image compression performance. The de-facto approach is to use rate-distortion (RD) curves measured with perceptual quality metries, such as PSNR. Such scalar metrics, although easy to compute, are known to be extremely limited in what they can capture and sometimes can even be misleading [63, 61]. To complement RD curves, we propose spectrally-inspired inspection tools that provide a more nuanced picture of (a) compression error, and (b) OOD performance of a given method. Specifically, we introduce a power spectral density (PSD) based approach to understand the reconstruction error. Our approach not only quantifies how much error was made but also highlights precisely where it was made (in the frequency domain). Similarly, to understand the OOD performance of compression method in unseen deployment scenarios, we propose Fourier error heatmaps a visualization tool for highlighting the sensitivity of the reconstruction performance of a compression method to different perturbations in the frequency domain. @ Using our benchmark datasets and inspection tools, we carry out a systematic empirical comparison of classic codecs (i.e., JPEG2000, JPEG, and VTM) with various NIC models (fixed rate, variable rate, and pruned versions of the scale hyperprior model [9], as well as efficient learned image compression (ELIC) [24]). O Finally, we develop theoretical tools to connect NIC OOD performance with its training data properties.
Main Contributions:
Main Findings: Our analysis resulted in some invaluable insights about the state of image com- pression. We summarize some of our findings below.
Compression methods yielding the same PSNR (or bpp) can produce very different spectral arti- Facts. Our tools help uncover hidden spectral biases and highlight the limitations of de-facto RD casve-based performance comparison.
As the compression rate increases, different codecs prioritize different parts of the frequency spectrum. By precisely characterizing this behavior, our tools help in advancing the current understanding of the modus operandi of image compression methods.
Image compression models generalize to low- and mid-frequency shifts better than high-frequency shifts. This finding calibrates our expectations for image compression performance on OOD data. NIC models are better at denoising high-frequency corruptions than classic codecs. This finding reveals that unlike classic codecs, NIC models have an inherent spectral bias due to the image frequencies present in their training data.
Identifying the most suitable compression method becomes exceptionally challenging without the knowledge of spectral characteristics of OOD shifts. Our systematic evaluation identifies this open issue with current compression methods and suggests the design of next- eext-generation NIC models that can adapt themselves at runtime based on the spectral nature of the data to be a potentially worthwhile direction to pursue in the future.
2

---

Original
Snow
Gless Blur
Shot Noise
Snow, Frost, Fog. Brightness, Contrast
Low
Motion blur, Zoom blur, Defocus blur, Glass blur,
Med
Elastic transform, Pixelate,
JPEG compression
Gaussian noise, Shot noise, Impulse noise
High
(b)
(a)
Figure I: (a) Top row: An original CLIC image and the same image with 3 different corruptions in CLIC-C (severity 5). Bottom left: Average PSD of CLIC dataset, N Σi= PSD(X)). Bottom row, other figures: Average PSD) of the difference between the corrupted images and the clean images for each given CLIC-C corruption c, N E PSD(c(Xx) Xx). (b) CLIC-C corruptions categorized as low, medium, or high based on corruption average PSD.
We corroborate findings with detailed theoretical analysis, showing that multiple overarching trends in our experimental results can be attributed to neural compression models' spectral bias.
Appendix A has a detailed related work discussion, while all references are listed in the main paper.
2 Out-of-distribution image compression datasets
To evaluate NIC in the presence of environmental or digital distribution shifts, we generated variants of the CLIC and Kodak datasets, which we refer as CLIC-C and Kodak-C. Following the techniques presented in [26] for studying the performance of DNN classifiers encountering distributional shifts "in the wild", our -C datasets consist of images augmented by 15 common corruptions. For each image in the original dataset, the -C dataset contains a corrupted version of the image for each of the 15 common corruptions', and for each of five corruption severity levels, with being the lowest severity and 5 being the highest. A sample of some corruptions on CLIC-C is provided in Figure la.
While each -C dataset offers broad sampling of environmental or digital image corruptions, also provides a spectrally diverse collection of corruptions, in the sense that each corruption can be categorized as low, medium, high frequency based on the frequency content used for perturbations. We will write (-D(-) to denote the function that converts the input image from the spatial to the frequency domain by computing the power spectral density of the input. Practically, computing PSD(.) is done by applying the fast Fourier transform (FFT) [11], followed by shift operation to center the zero-frequency component, then taking the absolute value. Now suppose we have a set X' = {X of uncorrupted images and some corruption function c(.) (e.g., frost, gaussian noise, etc.). We analyze the spectrum each corruption c(-) by computing N ΣN PSD(X, c(X.)) (see Figure la). We classify the corruptions into types of frequencies: low, medium, and high using the characterization from [39], which has been adopted by the neural network robustness community. Note that categorizing a corruption high is more akin to saying it "contains substantial high-freyuency content" rather than saying "exclusively consists of high-frequeny content."
3 Spectral inspection tools
While existing scalar metrics, such as PSNR, are able to summarize the visual similarity of recon- structed images to the original, we will demonstrate that such metrics can provide an incomplete (and sometimes misleading) picture when measuring the impact of compression in OOD settings. Notably, existing tools do not consider the impact of compression on different frequency ranges of images within a dataset. To more thoroughly analyze the effects of image compression, we propose to measure and visualize the effect of image compression in the spectral domain. Given an image compression model that returns reconstructed images, we introduce tools for analyzing compression
We used guthbb..onbbbhgggaaaiggggoooppie apply corruptions to Kodak and CLIC images
3

---

error in the Fourier domain to better understand (i) which spectral frequencies are distorted by C. (ii) the OOD generalization error, and (iit) the robustness error in the presence distributional shifts.
Definition 3.1 (Spectral Measure of Distortion Error). To analyze (i), we evaluate the image com- pression model C's ability to reconstruct components of an image across range of frequencies. To quantify this, we compute the average PSD of the difference between each image X₃ in dataset X' and the reconstructed version ((x)) of Xx: D(C, :- + E PSD(X, - C(x)).
Definition 3.2 (Spectral Measure of OOD Generalization Error). For (it), we evaluate C's ability to faithfully reconstruct OOD images. To quantify this, we extend the metric D(C, X') to account for corrupted version c(X²) of X' follows: G(,,X,,
N Σ (SD((((x))
(()()
Definition 3.3 (Spectral Measure of OOD Robustness Error). For (iti), evaluate C's denoising ability. To quantify this, we compute the average PSD of the difference between each uncorrupted image X₂ and the reconstructed version C(c(Xx)) of the corresponding corrupted image c( ((xx))
R(C, xec) == Φ Σ (S - C(c(Xx))).
The PSD function used in the above formulas converts the input image from the spatial to frequency domain by applying the Fast Fourier Transform (FFT) followed by a shift operation to center the freuurcyyyncc component. The proposed tools computed for of images by averaging the power spectral density (PSD) of the difference between the compressed image and the original for each image in the set of images. For simplicity, when (C, c) is clear from the context, write D, G. or R. Note that G provides insight into the compression model C's ability to generalize to a distribution shift e while R visualizes the denoising effect (or lack thereof) of C across the frequency domain.
will just
In Appendix B, present results using additional tool, the Fourier heatmap. that utilizes Fourier basis perturbations as corruptions and is used to corroborate our findings for the specific -C datasets and corruptions we consider. This tool can be leveraged when specific OOD data is unavailable.
4 Experiments and findings
Using our spectral inspection tools and OOD datasets, analyze the performance of several image compression methods and identify several inter- and intra-class differences between classic codecs and NIC methods. We show results for two classic codecs, JPEG and JPEG2000, and two NIC models, SH NIC and ELIC, in the main body and report the remaining results in Appendices C and D.
Classic codecs. We apply each of the following algorithms over several compression rates q.
-JPEG [60]
JPPE20000 [53]
Versatile Video Coding (VVC) (equivalently h.266) using the VVC Test Model (VTM)
software [30, 3]
Neural Image Compressors (NIC) Models. NIC optimization uses hyperparameter À to control the relative weight of distortion (quality of reconstruction) and rate (level of compression) terms in the objective function. We train each NIC model over several values of X. All models were optimized on the train split of the 2020 CLIC dataset [59]. Further details on their model architectures and training be found in Appendix I.
SH NIC: Scale-hyperprior model from Ballé el al. optimized for PSNR (i.e., distortion objective is mean squared error) [9].
ELIC: Efficient Learned Image Compression optimized for PSNR [24].
Variant of SH NIC optimized for MSMS [9].
Variable-rate version of SH NIC optimized using Loss Conditional Training [17].
Variable-rate model above with 80-95% of weights pruned using gradual magnitude pruning
[70].
Evaluation setup. We compare distortion, robustness, and generalization error of different image compression methods under three constraints: (a) no constraint, (b) fixed-bpp, and (c) fixed-PSNR. In (a), we compare methods over their full range of rate-distortion tradeoffs by generating rate- distortion curves. (b), we compare models with hyper-parameters which give a very similar bits per pixel (bpp) result particular dataset. For example, we find that on the CLIC dataset, SH NIC with

---

DCC Fixed-bpp
RD Curves
DIC Fixed-panr med. pane
low bpp
med. bpp
high bpp
pam
high pant
SHC
40-
10.25, 31,540
10.77. 36 857
(1.21. 38,99
(0.25, 32.541
10:37. 33
10.77, 36
EUC
10.25. 33.23)
10.75, 37 971
01.27. 41.261
PSMI
00.16, 33.321
10.25. 33
10.53 36 09)
PEG2000
34-
JPEG20000
10.24, 30.37)
(0.80, 35.34)
(1.20, 37.311
SH NIC
(0.34, 33,721
10. 33 09.09)
11. 1500 30 00.53
JPEG
ELIC
FRS
JPEG2000
28-
JPEG
10.22, 24.57)
(0.76, 32.88:
(1.22.
M0D0 1.0 high 1.5
(0:59, 31.571
00.01. 31.23)
11.56 10 (2)
Image (hpp)
Figure Visualizing distortion via CLIC test set evaluation. Left: spectral measure of in- distribution reconstruction error D under the fixed-bpp constraint at three rates. Center: Rate- distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating fixed-PSNR values. Right: D under fixed-PSNR constraint. Each D plot is labeled with a tuple of that model's (bpp, PSNR) on CLIC. Hotter colors (red) indicate more error in that frequency range.
A = 0.15, ELIC with À = 0.15, JPEG2000 with q 10, and JPEG with 75 all give a bpp close 1.21. Thus, comparing these three models with those hyper-parameters on CLIC under a fixed-bpp constraint, emulates a setting in which a fixed budget is available to store images. Analogously, in (c) we compare models with papee-pprameters yielding a fixed PSNR. This allows the comparison of spectral properties of data distributions that achieve the same quality according to the sealar PSNR metric. Scenarios (b) and (c) are used when evaluating D, G, R. Fourier heatmaps, and accuracy on a downstream task.
Test data. All models tested on (a) in-distribution (IND) and (b) corrupted (or OOD) datasets. For (a), we use the 2020 CLIC test split, the full Kodak dataset, and the ImageNet validation split. For (b), we use the corresponding -C datasets for each of the datasets in (a). The main body contains results for the CLIC/CLIC-C dataset. Analogous results for Kodak are in Appendix G.
4.1 Evaluating spectral distortion on IND data
On in-distribution (IND) data, the existing RD curve metrics in the center of Figure 2 highlight the established trend that these NIC models outperform the JPEG2000 and JPEG across the compression rates that the NIC model is trained on (bpp E (0.1, with ELIC outperforming SH NIC and JPEG2000 outperforming JPEG.
Next, we use our spectral inspection tool D to better understand the effects of different image compression methods. Specifically, Figure 2 shows plots of D under three fixed- bpp and three fixed-PSNR scenarios on the clean CLIC dataset. We highlight some surprising insights below.
Two methods yielding the same PSNR can produce very different spectral artifacts. Under the fixed-PSNR constraint (right side of Figure 2), each column consists of methods with hyper- parameters selected to give very similar PSNRs on the CLIC test set (e.g., models the "high psnr" column all have PSNR 36.8). Despite having comparable PSNRs, the plots of vary greatly between the four models. In particular, the SH NIC models distort high frequencies more than medium frequencies (notice the warmer-colored rings around the edges of the D plots with cooler-colored centers). JPEG2000, than high frequencies (notice the large rectangles of warmer colors). ELIC distorts all frequencies relatively evenly (the color gradients in the plots very narrow).² JPEG severely distorts one ring, but this ring includes lower frequencies (has a smaller radius) than the rings left by SH NIC. This same pattern holds under the fixed-bpp constraint (left side of Figure 2). These observations demonstrate that PSNR is not a comprehensive metric.
the other hand, distorts low and medium frequencies more
We provide more detailed view of the ELIC plots with smaller color bars in Figure 14 of the Appendix.
5

---

Low Treq.: Snow
Med. freq.: Glass Blur
High freq.: Shot Noise
40.0
SH NC
37.5
1500 W. 20. 4200
TLCC
pen2000
10-
PEG
32.5
30.0
MC
KC
27.3
RJ
BUC
25.0
JP65
JPES2000
JPEG
PE52000
0.5
1.0
1.5
2.0
0.7
Image size
Irage size (las))
sia (app)
Low freq.: Snow
Med. freq:: Glass Blur
High freq.: Shot Noise
BJC
BJC
54 MC
27.5
SH NC
ELIC
FSVR 198..7 (1dtt
Jrsc2000
150
25.0
JFEG
22
22.5
20.0
11.
12
13.5
10
15.0
12.5
2.0
(bpe)
thpel
Figure 3: Rate-distortion curves for representative low, medium, and high-frequency shift. Each shift and model has three curves for severity=1 (least transparent), severity=3, and severity=5 (most transparent). Top row: generalization of C(((X)) w.r.t. e(2t) (i.e., PSNR of the reconstructed shifted images w.r.t. the original shifted images). Bottom row: denoising of ((((X)) w.r.t. X (i.e., PSNR of the reconstructed shifted images w.r.t. the original clean images).
As the compression rate increases, different codecs prioritize different parts of the spectrum. On the left side of Figure 2, each column represents a different "budget" scenario where the three methods have hyper-parameters which result in the models giving very similar bpps on the CLIC test set. Although it was previously known that the quality of the reconstructed images decreases as the bpp decreases, it was not previously known which frequencies NIC models distort to achieve a given bpp. The 2> plots show that JPEG2000 models corrupt low- and mid fri-ffreuency regions starting at low compression rates and these regions become more severe as the budget decreases. SH NIC models do almost the opposite-ther sacrifice the highest frequencies first and expand this region into lower frequencies at more severe compression rates (i.e., as bpp decreases). JPEG has a mechanism more similar JPEG2000 in that it corrupts one ring and increases the severity of this ring, while ELIC models corrupt all frequencies evenly and then corrupt low frequencies more as bpp decreases. This suggests that the compression rate increases, classic codecs corrupt the same frequencies more severely while NIC models change their corruption patterns.
4.2 Evaluating the generalization and the robustness on OOD data
We use the CLIC-C dataset and our spectral tools to study the OOD performance of different image compression methods. We show results for example shifts (one low, medium, and high-frequency representative) and three severities (1, 3, and where 5 is most severe) in Figures 3 and 4 and discuss several interesting findings below.
4.2.1 Rate-distortion curves on OOD data
Image compression models generalize to low- and mid-frequency shifts better than high- frequency shifts. The top row of Figure 3 shows how well different compression models generalize shifted images in terms RD curves. In other words, these plots show how well a compressor C can reconstruct a given shifted image c(X') in terms of PSNR of C(c(X)) with respect to c(A'). The three examples of corruption in this figure show vastly different trends. On the cow-frequency corruption (snow), all four models can reconstruct c(A') almost as well as these models can reconstruct clean images: note the PSNR range in the top left plot of Figure is about 22-40 while the PSNR range for the clean data in Figure 2 is about 28-42 over the same hpp range. Interestingly, the three models can reconstruct the images with the glass blur (a medium- frequencc shift) better than they can reconstruct clean images (the PSNR of the data shifted with glass blur ranges from about 25-48 with bpp < 1). These results suggest that image compression models are fairly effective at generalizing to low-frequency shifts and very effective generalizing to medium frequnnyre shifts. However, the

---

NIC
JPEG2000
JPEG
sev.m1
sev.m5
sev.-1
SeV.
Sev.m1
sev.
Snow
Show
11.3.34.31
(13.
0.1
0.03
0.1
0.1
Glass
Glass
Glass Blur
0.0
0.00
0.0
40.3)
10.2, 43.7)
10.4, 41.0)
(9.2, 44.5)
(0.8, 40.8)
00.8, 46.31
10.4, 40.41
700
700
700
700
stent Pallee
Shott
Notio
11.9.22.11
12.7. 11.91
12.2. 23.31
(4.1, 12.4)
(0.8, 21.7)
10.0, 11.11
11.3.
21.01
13.1. 11.81
RIC,
GCV.=1
500
SCV.-1
SCV--5
500
SOV-1
SCV.-5
500
sev
500
Show
Snow
[1.3. 15.41
8..2.800)
(1.3, 15.3)
(1.3
(0.8, 15.1)
(D.6, 8.7)
15.31
11.0,
Classs
Class Itkar
Gloss Blur
800
600
600
600
Shoo No.e
Sheck
No
Shoo Notco
11.2.
12.7. 15.51
12.2 20.3)
14.1. 14.41
10.0, 25.01
(0.8, 14.5)
(5.3.
(3.1,
12.91
Figure 4: Generalization error 9 (top) and denoising error R. (bottom). We plot both spectral metrics for one low, medium, and high-frequency corruption severities and 5. Each plot is labeled with tuple of that model's (bpp, PSNR) on the CLIC-C dataset with that corruption.
high-frequency shift (shot noise), gives a starkly different result. All four models give very low PSNRs with respect to c(X)). Even the lowest severity (severity=1), this PSNR is only in the low 20s. As the severity increases (i.e., as the lines become more transparent), the PSNR decreases even more to the point that, the highest severity, none of the models achieve PSNR higher than 14. Notably, the main factor determining the PSNR is the severity of the corruption and not the compression model or bpp. This suggests that it is significantly harder generalize fighefrr data than to low- or mid-frequent data.
corruptions than classic codecs. The second row of Figure 3 shows how well different compressors denoise corrupted images terms of the PSNR of C(c(X() with respect to X. These results show that all the models fail at denoising snow llw-frrquency) and glass blur (medium-frequuency corruptions (PSNR does not change much with an increase in bpp, except at low bpps on glass blur). However, on shot noise (a high- freeffreyeency corruption) both NIC models achieve better PSNRs with respect to X than both classic codees. This trend is consistent for all three severity levels and suggests that NICs may be more effective method for denoising high-frequency corruptions than the previously uussuddlyyuddddd-eee JPEG and JPEG2000 methods. The implication of this finding extends to the research area of adversarial example denoising [6].
NIC models are better at denoising
4.2.2 Spectral analysis on OOD data
We now take deeper look at the Section 4.2.1 findings using our spectral inspection tools.
Spectral artifacts are similar for low-frequency shifts and clean images. The patterns of G in Figure measure how well each model generalizes to (or reconstructs) the shifted images. For the low-frequency shift (top row of Figure 4), the plots look strikingly similar the patterns exhibited by the same models on clean data (Figure 2 top and bottom row): SH NIC models distort high frequencies more than low frequencies while JPEG2000 distorts low and medium frequencies more than high frequencies. Similarly, ELIC distorts all frequencies relatively evenly and JPEG severly distorts one ring of frequencies. This suggests that the methods' modus operandi for generalization to data with low-frequency shifts is similar to their modus operandi on clean data, which makes sense as clean data is dominated by low/mid frequencies. Interestingly, these generalization differences between compressors are not accompanied by differences in robustness metric R. All methods show similar patterns in their plots of R patterns and these turn look similar to the snow corruption plot in Figure la. This is consistent with our finding from Figure 3 which showed that both NIC and
7

---

gsnr 36.93
32.97
31.17
5.0%
TIRT
Diftrennc Accurary
0.0%
Difference in Accuracy
I
Deffereee Diffrencc in Accuracy
-10.0%
-15.0%
SH MC (poer=36.55 93)
SH MCC (ponr=32.59 37)
SH NiC (psnr=31 17) ELIC (o61r=30.98)
EUC 1psnn=37:301
EUIC -3.843
-20.0%
JPEG2000 (o2nr=35.69)
JPEC2000
JPEC2000 (ssor331.4) JPES (ponr-21.18)
JPEG (ponr-37.11)
PES (2onr-33.11)
255.0%
dean
medium
dbaa
100
modium
lagh
clean
100
mwddum
Figure 5: Effect of compressing corrupt images on classification accuracy. Each bar shows the average difference in accuracy after compressing with different methods C for a group of corruptions as classified in Table lb. Specifically, let A(X) ) be the top-1 accuracy of the model measured percentage points. Then we report A(C(c(Xi)) - A(c(X) over all -C corruptions the corruption category with severity (or clean ImageNet in the case of "clean"), Each subplot shows results under a different fixed-PSNE constraint based on the PSNRs achieved by the compressors on the clean ImageNet dataset. Results for individual corruptions are in Figure 15.
dataset X,
JPEG200 fail to denoise low-frequency corruptions. In other words, Figure 4 shows that NIC and classic codecs fail in very similar manner on low low-freeuency signal denoising tasks.
Both NIC and classic codecs make almost no generalization error on freiuun-frequency shifts. The second row of Figure 4 shows that all four methods have very small generalization errors (magnitudes < 0.2), and this low error is relatively uniform across all frequencies. This shows that all the models models are very effective at reconstructing all the frequencies in images with glass blur-in fact, they can reconstruct these images better than they can reconstruct clean images- corroborating our first finding in Section 4.2.1. Again the R plots for glass blur for both of these models look very similar to Figure la; this similarity has a simple explanation due to a fundamental tradeoff between generalization and robustness. This relationship between R. G and average PSD of corruptions is described more precisely in Appendix L.4.
Higghffrqueeccy corruptions uncover the spectral bias of NIC models. Section 4.2.1 highlighted that frggufrre signals severely degrade the generalization performance of all image compression methods. From the RD curves with respect to the corrupt images (top right plot in Figure 3), we observe that each severity the three models have almost identical performance in the usual 0-2 bpp range. These results might lead to expect that these models make similar reconstruction mistakes, but our spectral inspection tools indicate that this is not the case at all. Our plots of G on shot noise at severity=5 (bottom row, columns 2, 4, 6, and 8 of Figure 4) indicate that NIC models distort the highest frequencies significantly more than the low and medium frequencies (notice the orange borders of the plots) while JPEG and JPEG2000 exhibit different patterns. This nuance, which is equivalent to one we previously observed with clean data in Figure 2, becomes more apparent high-frrog figh-frequency corruption because high- frggffreuuency signals much more
from the plots of G prevalent on this data than on the clean data. Additionally. the bottom right plot of Figure shows that for shot noise, NIC models achieve higher PSNR of the reconstructed corrupt images with respect to the clean images than the classic codecs, i.e., they have a stronger denoising effect. In the bottom right of Figure 4 we see more detailed picture: for high severity shot noise, NIC models achieve lower R in high frequencies. Also, note that the NIC models have the smallest errors in G whereas they have the largest errors in R and vice versa. Thus, these findings suggest that NIC models behave similarly to a low-pass filter.
4.3 Impact of spectral artifacts downstream applications
In practice, image compression algorithms may be used as pre- pre-processing step before images are used for another downstream task. For this reason, practitioners should also consider the performance on downstream tasks when comparing compression methods. We analyze the effectiveness of the compression methods on the downstream task of ImageNet classification using a pre-trained ResNet-50 model and report the difference in top-1 accuracy after compression (Figure 5) [25].

---

NIC can improve the robustness of downstream classification to high-frequency corruptions. While in general. compression degrades classification performance on clean images (all compressors show negative differences for the "clean" category in Figure 5), in some cases NIC and JPEG can actually improve the robustness of the classification model against high-frequency corruptions. Specifically, at PSNR=36.93, the difference in accuracy is positive for high-frequency corruptions with both SH NIC and ELIC, meaning that the classification model gave a higher accuracy on the set of corrupted images after compression than it did on the original corrupted images. Compressing with JPEG or JPEG2000 at the same rate caused a degradation in accuracy on these corruptions. However, at higher compression rates (PSNR=32.97 and PSNR=31.17 JPEG improves classification performance for high-free frequency corruptions, while SH NIC and ELIC do not. On low- and mid- frequeney corruptions, JPEG and JPEG2000 have a smaller degradations in accuracy compared to NIC the NIC models and no model gives an improvement in classification accuracy. Thus, the ideal compressor for downstream tasks dependent on the type of corruption and level of compression.
Pruning NIC models amplifies the robustness gains of NIC for downstream classification tasks. Additional experimental results in Appendix C (Figure 9), show that pruned NIC and NIC optimized for MS-SSIM act as even better high-frequency signal denoisers for this application.
Theoretical analysis of the OOD performance of NIC
We corroborate our empirical findings with theoretical results. Here we summarize theoretical results on linear autoencoder-basee NIC methods. For complete definitions, additional references, and proofs of the following statements, please refer to Appendix L. In Appendix L.3, we provide more general result applicable to nonlinear models.
Recall a classical observation: in the setting of linearly auto-encoding a mean- menn-centered distribution, the reconstruction function (i.e., encoder- decoodr composition) is a projection onto the high variance principal components of the input data [18] (see also [7]) Combining this with well-known facts about statistics of natural images, we show that a linear autoencoder applied to natural image dataset retains only low-to-mid (spatial) frequency components, which account for the majority of the variance. Using this result, we state theoretical explanations for multiple trends in our experiments'.
high-variance
Lemma 5.1. Let X be a dataset of natural images and let X denote its (spatial) discrete Fourier transform. Assume the following (for supporting evidence see Appendix L):
(I) The principal components of & roughly aligned with the spatial Fourier frequency basis. (II) The associated variances are monotonically decreasing with frequency magnitude (more specifically according the power law 1 |
If C is linear autoencoder trained by minimizing MSE on X, with latent space of dimension and if "r" denotes the spatial discrete Fourier transform, for any data point X with Fourier transform X
(x,3 22 K (o
C(X) ~
otherwise.
where K is the number of channels in the images in X and where X.,55 denotes the components of X corresponding to spatial frequency (i.
Corollary 5.2. Under the hypotheses of Lemma 5.1, the robustness error of C to a corruption e (as defined in Definition 3.3), measured in spatial frequency (i,j), is
Ex((())) EX
RIC, X, c)s ~
N
otherwise.
Corollary 5.3. Under the hypotheses of Lemma 5.1, the generalization error of to a corruption c (as defined in Desinition 3.2), measured in spatial frequency (i,
+ < kk
X,C,X,c)
& (c(X) X):1s + [(((X) X).
otherwise.
'Since this simplified model has compression rate objective, one should only expect the above theoretical results to be predictive (or explanatory) of the high bpp/PSNR cases of our experiments.

---

Lemma 5.1 suggests that autoencoder compressors trained on natural images behave like low-pas. filters, corroborating our claim in Section 4.2.2 (see also remark L.I).
Corollary 5.2 suggests that autoencoder compressors trained on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that (c(X) = is large for small values of i.j). This is indeed what we see in Figure 3 (bottom left), where snow corruptions detrimental to PSNR of C(c(X)) w.r.t X, and Figure 4, where the R(C, X, c) error for the NIC is concentrated in low frequencies. We also observe in Figure 5 that NIC is more beneficial for downstream classification accuracy in the case of high-frequency corruptions (e.g. shot noise) and less beneficial in the case of wwwfrrquency corruptions (e.g. snow).
On the other hand, the conclusion of Corollary 5.3 is more involved than that of Corollary 5.2-
the "cross term" (c(x) X)..ij is in general non-zero. However, there are cases where in expectation over the data set X this cross term vanishes (e.g., when es additive noise). In such cases, Corollary 5.3 suggests that compressors trained on natural images generalize less successfully to shifts with large amplitude in high frequencies. In Figure (top right) we see that shot noise corruptions are detrimental to PSNR of C(((X)) w.r.t. c(2²), and in Figure 4 we see that for the snow and shot noise corruptions, (CC, X, c) are concentrated in high frequencies. In summary, with both theoretical analysis of a simple mathematical model and empirical results, we find that NICs have spectral bias that causes them to overfit to discard high frequency components of natural images..
6 Limitations
Due to the number of axes of variation present in our experiments (e.g., model type, compression rate, evaluation dataset, corruption type, corruption severity, evaluation metric), we were forced to constrain many variables. We present results for a limited number of NIC models all trained on one dataset, CLIC. Due to limitations the number of models we could train and discrete choices for classic codec hhper-yaramene parameters, there is some variation within our fixed- (bpp.PSNR) bins.
Evaluating the robustness of any machine learning system is inherently multi-faceted problem riddled with unknown-unknownn unknowns. While we focus on robustness naturalistic input-data corruptions, there are other important distribution shifts for researchers and practitioners to consider, such as subtle changes in data collection methodology [51]. We omit a study of NIC model adversarial robustness to worst-case input data perturbations. Nevertheless, we hope that the our evaluation methods (including our metrics {D, 9, R. and Fourier heatmap spectral inspection tools) will be useful in future work on NIC model robustness.
7 Conclusion and future directions
We proposed benchmark datasets and inspection tools gain deeper understanding of the robustness and generalization behavior of image compression models. Using our spectral inspection tools, we uncovered the modus operandi of different compression methods. We also highlighted similarities and differences among them via a systematic OOD evaluation. Exploring the use of our tools in other image compression methods, including transformer-basee architectures and implicit neural representations methods is expected to provide interesting insights [41, 72, 73, 45, 54]. Our OOD evaluations identify NIC model brittleness issues: designing practical robust training approaches and/or methods to efficiently adapt to distribution shifts at runtime is a worthwhile research direction.
Acknowledgements
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 22-DR-009 (LINL-JRNII-811533
'Moreover glass blur can be viewed as sort of convolution operator, for which one can verify that the atorementioned cross term is non-zero.
10

---

The third author was supported by the Laboratory Directed Research and Development Program at Pacific Northwest National Laboratory, multiprogram national laboratory operated by Battelle for the U.S. Department of Energy.
We thank Eleanor Byler for helpful comments earlier draft.
References
[1] Origins of scaling in natural images. Vision Research, 37(23):3385-3399 Dec. 1997. ISSN 0042-6689.. doi: 10.101 111116//8002226699990000000000
[2] Oodhdr-codec: Out-of-distribution generalization for hdr image compression. 36:158-166 Jun.
2022. doi: 10.1609/aam 13....1999999999 URL https://ojs aaai www.oodeg....oo..ooooooooooo.e.e php. AAAI// article/view 19890.
[3] Vve software versatile video coding test model (vtm), Year of access, e.g., 2023. URL https://vegit.hit fraunhofer hww.unufee..eejee/////ee.. Jec//veetWWSSSfttaree VTM.
[4] G. Alain and Y. Bengio. What Regularized Auto-Encoders Learn from the Data- Generating Distribution. Journal of Machine Learning Research, 15(110):3743-377 2014. ISSN 1533- 7928.
[5] and S. Cho. Variational Autoencoder based Anomaly Detection using Reconstruction Probability. 2015.
[6] A. E. Aydemir, A. Temizel, and T. Temizel. The effects of jpeg and jpeg2000 compression on attacks using adversarial examples. arXiv preprint arXiv: /803, 10418, 2018.
[7] Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53-58, Jan. 1989. ISSN 0893-6080. doi: 116///8366000889)0000000
[8] J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017. URL https://openrevierr. net/forum? net/forum/2d-rrxxx0B33 id=rJxd03jeg.
[9] Ballé, Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In International Conference on Learning Representations, 2018. URL betts://openreet wtt.://ww.....oooooo/o//o/////// dwr.kik/Zkk..b...b.....
[10] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
[11] E. O. Brigham and R. E. Morrow. The fast fourier transform. IEEE Spectrum, 4(12):63-70, 1967. doi: 10.100//MPEC.. 1967.5217222
[12] G. Burton and I. Moorhead. Color and spatial structure in natural scenes. Applied Optics, 26(1):157-170, Jan. 1987. ISSN 2155-3165. doi: 10. 0..133//A.22..000
[13] Y. Choi, M. El-Khamy, and J. Lee. Variable rate deep image compression with a conditional autoencoder. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3146-3154, 2019.
[14] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
met//oorm? met//orrum
[15] J. Diffenderfer and B. Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning a randomly weighted network. International Conference on Learning Representations, 2021. URL ptttps://openrevier. ott.
11

---

[16] J. Diffenderfer, B. R. Bartoldson, S. Chaganti, J. Zhang, and B. Kailkhura. A winning hand: Compressing deep networks can improve -utoo--distrbuution robustness. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httl:htps:///pppt httptt:://pperrreiieer wttpi/ewwweee//rreeeeeeeeeee http::/ww.....oooumdmdd/////oo/o
[17] A. Dosovitskiy and J. Djolonga. You only train once: Loss-conditional training of deep networks. In International Conference on Learning Representations, 2020. URL. https:
/ /openreview
[18) C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychome- Irika, 1(3):211-218 Sept. 1936. ISSN 1860-0980. doi: 10.1007/BFO2288367.
[19] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018. doi: 00448550/RRXI111800000 03635.
URL htttps:/aarxvv.rrg///
[20] J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. Proceedings of the 37th International Conference on Machine Learning. ICML'20. JogR.org, 2020.
[21] C. Gille, Guyard, M. Antonini, and Barlaud. Learning sparse auto-encoders for green ai image coding, 2022. URL htttttt///axxiii org/abs/ 2209. .04488.
[22] V. Goyal. Theoretical foundations of transform coding. IEEE Signal Process. Mag., 18(5):
9 9-21, Sept. 2001.
[23] D. He, Y. Zheng, B. Sun, Y. Wang, and H. Qin. Checkerboard context model for efficient learned image compression, 2021. URL http::ttp///// /tatpps://rxxvv../// wttp:iv...rraab//////0/ 15306.
[24] D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang. ELIC: Efficient Iearned image compression with unevenly grouped Space-Channe contextual adaptive coding. Mar. 2022.
[25] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[26] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In International Conference on Learning Representations, 2019. URL https://openreval
[27] D. Hendrycks*, N. Mu E. D. Cubuk, Zoph, J. Gilmer, and B. Lakshminarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020. URL Ad=S1gmma S1gmrxHFvB.
[28] D. Hendrycks, Basart, N. Mu, S. Kadavath, Wang, E. Dorundo, Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of -f-o--iittributioonn generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320-8329, 2021. doi: 10111999CCV44992.2220..0000
[29] A. Hepburn, V. Laparra, R. Santos-Rodriguss tween statistical learning and perceptual distances. In International Conference on Learning Representations, Mar. 2022.
Ballé, and Malo. On the relation be-
[30] I. M. P. E. G. M. ITU-T, Video Coding Experts Group (VCEG). Versatile video coding
(h.266/vvc) standard. In ITU-T Recommendaation H. H.266 and ISO/IEC 23090-3. International Telecommunication Union (ITU) and International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC), 2020. URL. https: /ww.. itu. int/rec/ T-REC- H. 266.
[31] N. Johnston, E. Eban, A. Gordon, and J. Ballé. Computationally efficient neural image com- pression. arXiv preprint arXiv: 19120087,,, 2019.
12

---

[32] K. Kireev, M. Andriushchenko, and N. Flammarion. On the effectiveness of adversarial training against common corruptions. In The 38th Conference on Uncertainty Artificial Intelligence. 2022. URL /tttps://ppenrereee
net/forum?id-BeU _
[33] S. Klasky, Thayer, and H. Najm. Data reduction for science: Brochure from the advanced scientific computing research workshop. Technical report, Oak Ridge National LaR.(O),,.), Oak Ridge, TN (United States); SLAAC National Lab., 2021.
[34] E. Kodak. Kodak lossless true color image suite (PhotoCD PCD0992). URL http://rOk. us/ graphics/ cs/kodak.
[35] D. Kunin, J. Bloom, A. Goeva, and C. Seed. Loss Landscapes of Regularized Linear Autoen- coders. In Proceedings of the 36th International Conference on Machine Learning, pages 3560-3569. PMLR, May 2019.
[36] A. B. Lee, D. Mumffrd, and J. Huang. Occlusion Models for Natural Images: Statistical Study of a Scale-Invarian Dead Leaves Model. International Journal of Computer Vision, 41 (1):35-59, Jan. 2001. ISSN 1573-1405.
doi: 10.1023/A:1011 1100//::111110016667 109015665.
[37] E. Lei, H. Hassani, and S. Bidokhti. Out-of-distribution robustness in deep learning compres- sion. arXiv preprint arXiv:2/10.0700 2021.
[38] L. Li, Z. Yang, and R. Wang. Improving multi- settiieeeeratiion robustness of learned image compres- sion, 2022.
[39] Z. Li, O. Caro, E. Rusak, W. Brendel, M. Bethge, F. Anselmi, A. B. Patel, A. S. Tolias, and X. Pitkow. Robust deep learning object recognition models rely on low frequency information in natural images. bioRxiv, 2022. doi: 10. .31 41.4795 URL htttps://wwt biorxiv. org/ content/earlll /ttp:///ar1///22//////////////// 2ttp:/2/00200222222..... http:/2222......... 478509.
[40] T. Liang, J. Glossner, L. Wang. S. Shi, and X. Zhang. Pruning and quantization for deep neural network acceleration: survey. Newrocomputtig, 461:370 403,, 2021. ISSN 0925-2312. doi: 101
URL https: /tttpp:://www hwwp://wwwwwccceco////////. sciencedirect. com/
[41] J. Liu, H. Sun, and J. Katto. Learned image compression with mixed transformer-enn architec- tures, 2023.
[42] K. Liu, D. Wu, Y. Wang. D. Feng, B. Tan, and S. Garg. Malice: Manipulation attacks on learned image compression, 2022.
[43] A. Luo, H. Sun, J. Liu, and J. Katto. Memory-efficient learned image compression with pruned hyperprior module, 2022. URL https://arxiv..rr abb/ a22g//66../2///...////// 10083.
[44] M. C. Malin, M. A. Ravine, M. A. Caplinger, F. Tony Ghaemi, J. A. Schaffner, J. N. Maki, J. F. Bell III, Cameron, W. E. Dietrich, K. S. Edgett, al. The mars science laboratory (msl) mast cameras and descent imager: Investigation and instrument descriptions. Earth and Space Science, 4(8):506 539, 2017.
[45] D. Minnen and S. Singh. Channel-wise autoregressive entropy models for learned image compression, 2020. URL https://arxii...rt www.aawwwww....oooaooooooooooeee abs/2007 0877/
[46] D. Minnen, J. Ballé, and G. D. Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: http:////oooe/eeee/e/eee/ www.eeedddnnnn..uuip..eeee cwturrps.cc/aapeee pc/pp/ file/
pdf.
[47] D. Mishra, S. K. Singh, and R. K. Singh. Deep architectures for image compression: critical review. Signal Processing, 191:108346, 2022. ISSN 0165-1684. doi: https://doi.or//10.000 251. URI. https:/ http://wwww..cieeeeeeeee... www.///iine///aeaecccoooomeeeeee ettp/atticeeeee../.// $0165168822003892.
13

---

[48] S. Narang, G. F. Diamos, S. Sengupta, and E. Elsen. Exploring sparsity in recurrent neural networks, CoRR, abs/1704.05119, 2017. URL http://arxiv.. wwg. wwtpaa/s.../////./
[49] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference Artificial Intelligence, 32 (1), Apr. 2018.
[50] E. Plaut. From Principal Subspaces to Principal Components with Linear Autoencoders, Dec. 2018.
[51] B. Recht, R. Roelofs, L.. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet?, 2019.
[52] J. Rissanen and G. Langdon. Universal modeling and coding. IEEE Transactions on Information T'heory 27(1): 12-23, 1981. doi: 10.1 10.1109/TTT.199 1981. 1056282.
[53] A. Skodras, C. Christopoulos, and T. Ebrahimi. The jpeg 2000 still image compression standard. IEEE Signal Processing Magazine, 18(5):36-58, 2001. doi: 10. 1599799.55280.
[54] Y. Strümpler, J. Postels, R. Yang, L. van Gool, and Tombari. Implicit neural representations for image compression, 2022.
[55] H. Sun, L. Yu, and J. Katto. End-to-end learned image compression with quantized weights and htrg/abs/21111 http///211.....9...1
activations, 2021. URL httpp:://att
[56] J. Sun, A. Mehra, B. Kailkhura, P.-Y. Chen, D. Hendrycks, J. Hamm, and Z. M. Mao. A spectral view of randomized smoothing under common corruptions: Benchmarking and improving certified robustness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV, pages 654-671. Springer, 2022.
[57] J. Sun, Q. Zhang, B. Kailkhura, Z. Yu, C. Xiao, and Z. M. Mao. Benchmarking robustness of 3d point cloud recognition against common corruptions. arXiv preprint 2tti::2001.12999 2022.
[58] L. Theis, W. Shi, Cunningham, and F. Huszár, Lossy image compression with compressive autoencoders, 2017. URL https: / //axxiv..vv/////////////////alll org /ttg//b///a/ ht/aaabs//////////////////////// 1703. 00395.
[59] G. Toderici, W. Shi, R. Timofte, L. Theis, J. Balle, E. Agustsson, N. Johnston, and F. Mentzer. Workshop and challenge on learned image compression (clic2020), 2020. URL http: //ww. compression.co cc.
[60] G. Wallace. The jpeg still picture compression standard. IEEE Transactions on Consumer Electronics, 38(1) xxxixxxvii,x 1992. doi: 10.1 10.9300 125072.
[61] Z. Wang and A. C. Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine, 26(1):98-117, 2009.
[62] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600-61) 2004. doi: 10.1 1..11099TPP.003...0 119611..
[63] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-61) -12200000666600 2004.
[64] A. Wicckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George. G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic. B. Bross, and D. Marpe. Vvene: An open and optimized vvc encoder implementation. In Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW), pages 1-2. doi: 10. D0.109//CMEW53327620000 1945944.
[65] Yang, L. Herranz, Y. Cheng, and M. G. Mozerov. Slimmable compressive autoencoders for practical neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4998-5007, June 2021.
[66] D. Yin, R. G. Lopes, J. Shlens, F. D. Cubuk, and Gilmer. Fourier Perspective on Model Robustness in Computer Vision. Curran Associates Inc., Red Hook, NY, USA, 2019.
14

---

[67] S. Yin, C. Li, Y. Bao, and Y. Liang. Universal efficient variable-rate neural image compression, 2021. URL httph: //rrpi://arxiv.... http://wwwww...raa////o// arggabs//1111 11305.
[68] Yin, C. Li, W. Tan, Y. Bao, Liang and W. Liu. Exploring structural sparsity in neural image compression, 2022. URL
http:/abb2222..22// 04595..
[69] D. Zhang. J. Yang, D. Ye, and G. Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[70] M. Zhu and S. Gupta. To prune, or not prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https::/openrevieeee
net wow.oriddmyyyiiiiai
[71] X. Zhu, J. Song, I.. Gao, F. Zheng, and H. T. Shen. Unified multivariate gaussian mixture for efficient neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17612-17621, June 2022.
[72] Y. Zhu, Y. Yang. and T. Cohen. Transformer- transform coding. In International Con- ference on Learning Representations, 2022. URL https: htttp:://pperreree..neee///// wtwp:iewwwe//foommmeeee IDwN6xjHHKS
[73] R. Zou, C. Song, and Z. Zhang. The devil is in the details: Window-based attention for image compression, 2022.
15

---

A Background and Related Work
A.1 Neural Image Compression (NIC)
In lossy image compression, input image is compressed into small file bitstream for trans- mission or storage. This file is then used to reconstruct the original image. Several models have been proposed for neural (or learned) image compression such as convolutional neural networks (CNNs), autoencoders, and generative adversarial networks [22, 8, 9, 46, 58, 13, 65, 71, 47J. In the most basic version of these models, an image vector x is mapped to latent representation, y using a convolutional autoencoder. y is then quantized into y. so it can be compressed to a bitstream via an entropy coding method, such as arithmetic coding [52]. During reconstruction, the received bits go through the reverse entropy coding process and the revealed y is fed into a CNN decoder which outputs the reconstructed image x. These models can be trained end-to-end by using differentiable proxies of entropy coding [8].
Normally entropy coding relies one prior probability model of y, which is learned from the entire set of images. Ballé et al. [9], however, recognized that the prior of this model can be optimized for each image separately. By adding an additional hyperprior autoencoder their model, they learn the latent representation for the entropy model of each image. The information transmitted from this autoencoder can be seen equivalent "side information" used by other codecs. Minnen et al. [46] further improved this model by adding an autoregressive prior to the hierarchicial prior used by Ballé. The Efficient Learned Image Compression (ELIC) model builds on these models by adding uneven adaptive coding and a spatial context model [24].
A.2
Generalization and robustness in DNNs for classification
is well known that deep neural networks (DNNs) are prone to overfitting to the specific data distribution they are trained on. Much recent work has focused improving the robustness of DNNs to other distributions of data, including common image corruptions [27, 66]. CIFAR-I0-C and ImageNet-C were proposed as a benchmark datasets test the robustness of models to common
image corruptions such as corruptions from weather, blur, and digital noise [26]. Methods for
improving robustness include data-level techniques such as adding augmentations and algorithn-level techniques like using larger models or adversarial training. [27, 32, 28, 14].
Yin et al. study the effects of data augmentation and adversarial training on different corruptions using a Fourier analysis [66]. They analyze the CIFAR- 10-C dataset and find that most augmentation methods help in the high-frequency domain, but hurt in the low-frequency domain. However, it is unknown how image compression affects corrupted images. We analyze how different types of image compression affect the Power Spectral Density (PSDs) of both clean and corrupted data.
A.3 Generalization and robustness with NIC models
There is limited amount of work studying the generalization and robustness of NIC models. [37] formulates problem for designing robust NIC models and train DNNs to be robust to two particular types of distributional shifts. They find tradeoff between OOD robustness and IND performance. Our work complementary to this work in that we provide additional metrics and OOD benchmarks to analyze the robustness of various models in more detail. [42] studies NIC models' robustness to white-box and black-box adversarial attacks and proposes new models which improve the robustness of NIC to such attacks. Our work instead looks naturalistic distribution shifts, which is another important aspect of robustness [26].
[2] designed NIC module which works specific set of OOD images. In particular, their module works on high dynamic range (HDR) images, despite being trained on mostly standard dynamic range (SDR) images. Although this is specific example of testing NICs on OOD data, our work differs from theirs in that we analyze how NICs would perform on a wide variety of image corruptions without seeing any examples of these corruptions during training.
Additionally, [38] addresses the issue of "multi-generaaiinnal loss", or the problem of a rapid degrada- tion of quality as images are repeatedly encoded and decoded, with NIC models. Our work differs because we propose spectral tools to analyze the effects of one cycle of encoding and decoding and study how this affects the performance of different downstream tasks, While multi- meltiigeneration
16

---

loss is an important area of study for tasks like image editing, we focus on analyzing the spectral distortions caused by one cycle of image compression and how these affect a model's generalization and robustness to OOD data. Furthermore, [38] proposes using JPEG2000 as method to improve the robustness of classification tasks by compressing adversarial noise. We explore how NIC models compare to JPEG2000 in terms of their robustness to several common types of image corruptions.
A.4 Lightweight NIC via model pruning
In effort to reduce the energy and compute cost required by deep neural networks (DNNs), much work has been devoted towards finding methods to approximate DNNs. Common approaches include pruning network weights or quantizing network weights and activations [19, 20, 48, 69, 40, 15]. Remarkably, large sparse models have been shown to outperform small dense models with the same number of weights [70]. Additionally, approximating DNNs can have benefits in terms of robustness on OOD data [16].
Nonetheless, limited efforts to prune quantize NIC models have been explored. Recently, Gille et al. [21] successfully introduced structured sparsity into NIC models to simultaneously reduce storage and power costs while maintaining similar rate-distortion curve when compared to dense version of the same NIC architecture. Their findings produced ~800 reduction in the number of parameters and ~30% reduction in MAC operations required by the NIC encoder. Yin al. [68] also explored the use of structured sparsity in NIC and their proposed ABCM method was demonstrated to provide up to 7x and 3x reduction in DNN parameters and inference time, respectively, with only mild reduction to reconstruction quality. Luo et al. [43] proposed a technique for pruning NIC models that was demonstrated to maintain the same bpp as dense models but provided parameter reduction of at most ~344. Sun et al. [55] demonstrated technique for quantizing both NIC weights and activations to an 8-bit fixed point representation while yielding a rate-distortion curve comparable to the original 32-bit float NIC model. Outside of DNN pruning or quantization, alternative techniques have been explored for more efficient NIC [31, 67, 23].
A.5 Metries for quantifying image compression quality
The objective of lossy image compression is to achieve a balance between minimizing 1) the size of the compressed file and 2) the distortion of the reconstructed image compared to the original image. As these objectives are inherently at odds, the performance of lossy image compression is typically measured using rate-distortion curve, which measures the size of the compressed image (rate) versus the quality of the reconstructed image (distortion). The rate is usually measured in bits per pixel (bpp). which is the average number of bits required store each pixel in the compressed representation. The distortion is usually measured by scalar metrics such as peak signal-to-noiss ratio (PSNR) or a visual similarity metric such as MS-SSIM [62].
A.6 Benchmark datasets
We utilize two benchmark datasets for evaluating performance of our NIC models. For training and testing, make use of CLIC (Challenge on Learned Image Compression) 2020 [59]. This collection is comprised of 1633 training. 102 validation, and 428 test images. The images are of varying resolution and are further categorized as either professional mobile. To evaluate generalization, we make use of the Kodak [34] dataset which consists of 24 images of resolution 768 x 512 (or 512 x 768).
B Additional results: Fourier heatmaps
While our spectral tools allow us to measure different capabilities of compression models, they require the availability of OOD, or corrupted, data. To support setting in which such OOD data is unavailable, we propose adopting another tool: the Fourier sensitivity heatmap. This tool evaluates the PSNR of a compression model on data perturbed with Fourier basis elements [66]. The resulting visualization is heatmap where the value at coordinate (i.j) is the PSNR of the compression method on perturbed data { (X, + 1 where each ris is selected uniformly at random from {-1, 1), is the norm of the perturbation, and ULJ is the (i. j) Fourier basis matrix. From [66].
17

---

PSNR w.rst. CLAD
PSNR w.r.t. Y
VR NIC VR NIC
FR NIC opt MS-SSIM
VR NIC VR NIC
FR opt
FR NIC
VR NIC
pr=0.8
pr
JPEG2000
36.0
FR NIC
VR NIC
prm0.s
pr-0.95
MS-55IM
JPEC2000
High 251
med plrt
32.0
72
Iow part
13.5
74
VR NIC
VR NIC
FR NIC opt. MS-SSIM
NIC
VR NIC
NIC ogt MS-SSIM
NIC
VR NIC
pr 8
pr=0.95
JPEG2000
VR
pr-
pr-0.995
JPEG2000
-99
High bgp
high
35
mreid opo
Figure Fourier heatmaps of methods under the fixed-ppp setting (top) and fixed-PSNR setting (bottom). Left: PSNRs of (c(()) with respect to c(X'). Right: PSNR of C(e(2)) with respect to X. Warmer colors (red) indicate higher PSNR.
UKj E X and satisfies (i) || and (ii) F(Us.) has most non-zero coordinates specifically at (i,j) and the coordinate symmetric to (i, about the matrix center.
We consider two versions of Fourier heatmaps: (a) with respect perturbed data and (b) with respect to original data. To analyze (a) and (b), the PSNR at each coordinate (i,j) computed with respect to the perturbed dataset, (X& ræEU,,; \ and the unperturbed dataset, {Xx) /=/ respectively. (a) and (b) can be seen as measures of generalization and robustness respectively.
We computed Fourier heatmaps over the CLIC dataset in Figure 6. We use models from Appendix C, where FR NIC equivalent to NIC in the main body and selected hyper-parameters for these models using fixed-bpp/PSNR constraint on the clean data. Note: in these Fourier heatmap plots, warmer colors represent higher PSNR, which is contradiction to the plots of D, G, and R. where warmer colors represented more error.
On the left side of Figure 6, we analyze the generalization of the image compression models to various frequency shifts. We find that all image compression models generalize to low and medium- frequency signals better than geghffreueecc signals. Specifically, observe how the centers of the plots corresponding to low-freod www-rreuency signals-have hotter colors than the edges of the plots. This corroborates our findings from both the RD curves and metrics on the CLIC-C dataset Section 4.2 and Appendix H. For FR NIC opt. MS-SSIM, this difference is only noticeable at the lowest bpp: however for the other models, there is clear difference in performance for low and high-frequency shifts at all bpps. Additionally, the effect of pruning is degradation in PSNR at all frequencies.
On the right side of Figure 6, we analyze the robustness, denoising, capabilities of the models. These plots show that NIC models are better at denoising fighffreuuency corruptions than low- frequency corruptions (notice the cooler-colored diamonds in the center the NIC plots). Meanwhile JPEG2000 has very consistent and poor denoising properties across the entire range of corruption frequencies (the plots are blue with a narrow gradient of color compared the NIC plots). According to this metric, NIC opt. MS-SSSIN also has poor denoising capabilities (the plots are all dark blue compared to the NIC plots) and pruning NICs also results degradation of denoising performance. At first, this may seem like contradiction to our downstream classification results from Appendix C, which showed that NIC opt. MS-SSIM and VR NIC pr=0.8/0.95 were more effective denoising high-frequeene corruptions than FR NIC and VR NIC. However, recall that each pixel in these plots
18

---

only shows the average PSNR of the entire perturbed dataset as opposed to the spectral error by frequency, like our D, G. and plots show. Then from the main body and Figure 6, is clear that this PSNR metric is not detailed enough highlight the nuances between the spectral distortion patterns of different methods.
C Additional results: NIC variants
In this section, we analyze several additional variants of NIC models. Many of these variants were chosen because they have potential to reduce the amount of memory required to store or run these models, making them more amenable to "in-the-wild" use cases (e.g., the Mars Exploration Rover discussed in the introduction). We also consider models optimized for MS-SSIM because we are interested in how the distortion objective affects the spectral distortion artifacts of the model.
Because all of these models are based on the scale-hyperprior architecture from [9], we use different naming scheme here than in the main body. Below are descriptions of the NIC models compared in this section:
FR NIC: Equivalent to SH NIC in the main body. Eight fixed-rate models each trained unique A. Distortion is MSE (i.e, model is optimized for PSNR).
VR NIC: One variable-rate model trained over continuous range of values using loss conditional training. Distortion is MSE. More details about this model are provided in Appendix J.
VR NIC pr=0.8/0.95: One variable-rate model trained over a continuous range of À values. Distortion MSE. Here, gradual magnitude pruning (GMP) applied during training to prune 80% or 95% of the convolutional and FiLM weights. More details of the GMP algorithm are provided Appendix K.
. FR NIC opt. MS-SSIM: Eight fixed-rate models, each trained on a unique À value. Distortion is (1 MS-SSIM) (i.e., model is optimized for MS-SSIM).
Figures 7, 9, and 8 show results for these models. We discuss our findings below.
C.1 Evaluating spectral distortion on IND data
Effect of approximating FR NIC with VR NIC. According to the RD curves in Figure 7, VR NIC obtains the same performance as FR NIC for low and moderate bpps; however, the SH NIC model outperforms the VR NIC model at higher bpps despite being trained on the same range of X. This result follows [17] and suggests that the VR NIC may not be expressive enough to learn the high PSNR regime. terms of the spectral error D, VR NIC and FR NIC have the same error patterns, but VR NIC does leave slightly higher magnitude of error in some settings (e.g., med. bpp, high bpp, and high psnr).
Effect of pruning. VR NIC. VR NIC pr=0.8, and VR NIC=0.95 all have the same VR NIC model architecture, but have 0%, 80%, and 95% sparsity respectively. The RD curves show that pruning causes a degradation in performance, especially for high bpp (see blue lines in Figure 7). However, the PSDs reveal that, even we make up for the PSNR performance gap by increasing the bpp (right side of Figure 7), the pruned models left different spectral artifacts from the dense ones. In particular, as the prune rate increases, the ring in the high- geghfrreueency region of the PSDs becomes thicker. Presumably reconstructing signals at figh-frrequencis requires more parameters than reconstructing low frequency signals and so the more sparse models focus on optimizing the "easier" frequencies on the spectrum. These results show that pruning NICs exacerbates the spectral differences between NIC and JPEG2000 that we observed in Figure 2.
Effect of optimization variable in NIC. The FR NIC and FR NIC opt. MS-SSIM have the same FR NIC model architecture, but optimize for different "distortion" metries (i.e., FR NIC optimizes for PSNR while FR NIC opt. MS-SSIM optimizes for MS- MS-SSIM). Unsurprisingly Figure 7 shows that the FR NIC opt. MS -SSM model has the worst performance of all the models in terms of RD curves where the distortion metric is PSNR. However, the plots of the distortion error D give us more insights into the spectral errors that this model making. Surprisingly, FR NIC opt. MS-SSIM produces spectral artifacts which differ greatly from both the PSNR optimized NIC models and the classic JPEG2000 codec. In particular, like JPEG2000, this model distorts the medium and low frequencies more than fighffrequencies (notice the cooler-colored ring around the hot-colored center).
19

---

DCC Fixed-bpp
RD Curves
XIc Fixed-panr
low bpp
med. bpp
high bpp
panr
ponr
high penr
FR MR
33
11.21.
36-
10:25, 31.54)
(0.37,
(0.77, 36
MA
PSNR:: (16)
FR MCC
(0.23, 30.981
10.79, 87)
(1.21.
VR
pr-d.a
pr.
VR MC
FR opt MS-55IM
(0.25, 30.99)
19.77, 25 82)
(1.22. 26.98)
JPE
10.25. 31.56)
(0.40, 23.177
(1 (3.22, 36 981
Imagee size (hop)
NIC
pro..5
pr-0.95
(0.25, 30.47)
10.77, 35 36)
(1.21. 36.50)
10.31, 31.48)
(8.45,33.19)
JPE22000 op.. MS SSNSSNN
HS-35IM
PUNIC
PRINC
10.72, 90)
(0.21.
(1
JPEG2000
10.24. 30.371
10.80, 35 34)
12.20. 37.333
10.34, 31.72)
(0.48. 33.00)
(1 10,0 38.831
15
40
Figure? 7: Visualizing distortion via CLIC test set evaluation with NIC variants. Left: spectral measure of in-distribution reconstruction error DD under the fixed-bpp constraint at three rates. Center: Rate-distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating -Red-PSNRRRT values. Right: D under fixed-PSNR constraint. Each D plot labeled with a tuple of that model's (bpp, PSNR) on CLIC. Hotter colors (red) indicate more error in that frequency range.
Snow
Shot Noise
VR
NIC
VR NIC FR JPEG2000 pr
MS-S5IM JPEG2000
FR NIC
VR
pr=0.8
MS-SSIM
00.088155. 37..07.,3, 36.4901.50, 36.43...2, 33.97y2.19,
-000
200
RUC. K.
11.3011.SE, 11.141.331 11.1511.50 11:3211.29, 11.1512.19, 21.05)
2.665 21.0922.40, 21 21.3671.87, 21.78y 99. 21.9312.58 38, 20 7711.10 20.64)
Figure 8: Spectral measures 9 and for all NIC variants on two example corruptions of ImageNet-C. Model hyper-parameten were chosen such that they give bpp 1.23 ImageNet. Each plot is labeled with the model's 's bpp and PSNR with respect to c(2t) (top row) or X' (bottom row).
However, it differs from JPEG2000 in that it leaves less rectangular artifact. Thus, the optimization objective of NIC can influence the spectral artifacts and should be thoroughly examined before being used in practice.
C.2 Robustness and generalization
We analyze the findings further using our robustness spectral inspection tools on ImageNet-C: in Figure 8. The plots of G on the snow corruption show that although the models have different magnitudes of errors, all models leave similar magnitudes of errors across the frequency spectrum (i.e., the gradient of each plot is narrow). Furthermore, all models have similar plots of R. This follows the result of the next section (accuracy the downstream task), which shows that all six models have comparable accuracies at this compression rate (see snow column of bpp plot in Figure 9).
the shot noise corruption, however, we see some notable differences between models on both G and R. On G, we see that as the prune rate increases, the ring of high-frequency spectral error becomes significantly more bold. This reiterates our finding from Figure 7 that pruning results in degradation of high- tigh-freeqency signal reconstruction. Similarly, FR NIC opt. MS-SSIM has a thicker
20

---

bpp 1.23
bpp 0.77
bpp 0.24
10%
Dfforence Accurscy
0%-
Detieeeope in Arrurary
-10%
-20%
VR N.C (Spp-1
FR NIC (.op-1.223)
FR NCC 1hpp 771
NIC (xo7-0.77)
FR MC 1bpp 241
NIC (hopm 0.23)
M.C pr=0.8 (bpo=1100)
pr-0.8 (bpp=0.78)
pr-O (.pp=0)24)
-30%
M.C pr-6 0.0p=1.23)
pr 1bop-0.78)
pr 1bop-0
NIC opt. HS SSM (20pm11.4)
NiC opt. NS-S5IM (hop=00772)
NiC opt. NS-55IM (hop-00.23)
JPEG2000 (bgp-
JPEG2000 (bpo 75g
JPeG2000 (bpo 24)
mod
Ng
brigt Has
conoout
mottor
.com
ceroc
glato
lpog
clean STOK
mode
100
mottor
oefocus
poetle
shot impalee
frost.
bnightren
pommet notter
200m
defocus
glose
alalls
1pog
shot
impolle
bright
psnr 36.93
pone 32.97
panr - 31.17
10%
Offerencce accurccy
0%
Differcce Arturacy
-20%
FR/NC Spenr:35 36 93)
VHt N.C (ww.-ppm--3..... 85)
Offferance
FR NIC (p.9m=32.977
NOC (purr-33.00
FR NIE Ipsnen 31.373
(purn 11.188
M.C (p5nn=36.91)
NIC (p5mr-33.00)
(ponr=21.65)
-30%
M. pr-00.00
pr (pon--33.05)
pr. (psnr-
PH ICC opt. MS (00-55m (pwnr-
NiC opt. MS-35IM (p:1e=35:100
FSL opt MS-55IM (pare-
JPEG2000 (ponn=35.69) 36.69
JPEG2000 (psn 52)
JPEG2000 (ponn- 24)
dean smor (ia
detecs
claan
bnggtreet
cantant
metten
200m
actocs
giaes
cleas
defocos
ebsticc
1899
shet
impalo
big
Figure 9: Effect of compressing corrupt images on classification accuracy. Each subfigure shows results under a different fixed-bpp or fixed-PSNR constraint based on the bpp/PSNRs achieved by the compressors the clean ImageNet dataset. Specifically, let A A(X) be the top-1 accuracy of the model on dataset X, measured in percentage points, We report A(C(c(X))) A(c(X)) over all -C corruptions (or on clean ImageNet in the case of "clean").
ring than FR NIC in the high-frequency domain. Interestingly, this model also has smaller error in the low frequency domain compared to the NIC models optimized for PSNR. Finally, JPEG2000 distorts low- and cadium-frrquencien relatively evenly and distorts high-frequency signals less than NIC which follows from Section 4.1. We can see the effect of these differences in g in the high-freque gehhfrryuency domain of of the plots R. Specifically, the VR NIC leaves a slightly smaller artifact than FR NIC (i.e., the circle of warm colors is larger with VR NIC). This differences is exacerbated as the prune rate increases (i.e., NIC 0r=0.95 95 has the smallest artifact of the four first four models), which suggests that the pruned NIC models are most effective for denoising high-frequency corruptions. FR NIC opt. MS-SSIM also leaves a smaller artifact than FR NIC; however, this has different shape than the NIC models optimized for PSNR (specifically, there are jagged edges on the edges of this artifact). This suggests that FR NIC opt. MS- MS-SSIMM denoises high-frrquency data better than FR NIC, but in different way from VR NIC pr=0.8/0 8/0.8800..0 95. Finally, JPEG2000 has Hiamond-shaper artifact which suggests it denoises some high- fygefrrey signals, but not all.
C.3 Downstream task
We also consider how these NIC variants perform on the same downstream classification task outlined in Section 4.3. Figure 9 shows the difference in accuracy before and after corruption for each model on each corruption (or clean data) and Appendix M has tables with these accuracies. We find that the pruned models are better at denoising fiegueryyyency corruptions than the FR NIC. In particular, we find that on the high-frequency corruptions at bpp = 1.23 and 0.77, the pruned VR NICs show a larger improvement in accuracy than the dense VR and FR NICs. This finding matches our observation in Figure 7 that the pruned NICs denoise fegh-frreuency signals even more than dense NICs, who in turn denoise these signals more than JPEG2000. The pruned models perform comparably to the dense models on low- and medium- frrquucyyyyyenccy corruptions. Similarly, we also find that NIC opt. MS- SSIM also improves performance on the downstream task on high- frequeryy corruptions over (a)
21

---

AIC Fixed-bpp
RD Curves
XIc Fixed-panr
low bpa
med. bpp
high bpp
low psnr
med. psnr
high panr
FR NRC
JPEG2000 Fit
10.28, 30.1
20 94, 36, 201
(1.43, 28
10.28, 30.19)
10,91. 36,20)
(1.25, 37.561
JPEG2000
ESMR
11.33,
0..11
NIC JPEG2000
VTH
VTN
(0.29, 31. 06;
(0 97, 37 921
(3.43. 39.301 301
(0:30, 29. 911
10. 36.313
(0.97. 37.421
0.0
0.5
1.0
1.5
2.0
70
10
Image se (bpo)
60
Figure 10: Visualizing distortion of VTM via Kodak test set evaluation. Left: spectral measure of in-distribution reconstruction error D under the fixed-hpp constraint at three rates. Center: Rate-distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating fixed-PSNR values. Right: D under fxxd-PSNR constraint. Each D plot is labeled with a tuple of that model's (bpp, PSNR) on Kodak. Hotter colors (red) indicate more error in that frequency range.
no compression and (b) compression with dense NIC models. This model also has better performance than the other NIC variants and low- and medium- medium-frequenny frequency corruptions.
D Additional results: comparing JPEG2000 and VTM
In this section, we compare two classic codecs: JPEG2000 and Versatile Video Coding (VVC) (also known as h.266) codec [30]. We refer to the latter codec as VTM because we used VVC Test Model (VTM) software from [64] to test it. Although VTM has state-of-the-art performance for classic codecs, we chose to use JPEG2000 as the representative classic codec in the main body for three reasons.
1. JPEG2000 has more consistent bpp usage. As shown later in this section (Figure 12), VTM varies its bpp usage drastically depending hyper-parameter which gives a bpp of x on clean data may give a bpp of over 7x on certain OOD shifts and severities." Thus, it was unclear how to fairly compare VTM and NIC in fixed- -pp setting on OOD data.
the corruption type and severity. Specifically,
2. JPEG2000 implementation has faster inference time. The current open-source implementation for VTT is significantly slower than those available for JPEG2000 (10s to 100s of seconds for VTM vs. I second for JPEG2000). Thus, JPEG2000 was a more feasible codec for our extensive experimentation.
After adjusting for PSNR, VTM shows similar spectral results as JPEG2000. Finally, although there are differences in magnitudes of the spectral distortion plots, JPEG2000 and VTM exhibit similar patterns, so we found sufficient to use only one classical codec representative.
We include results comparing JPEG2000 and VTM on the Kodak and Kodak-C datasets in Figures 10- 13 and explain our findings below.
D.1 Evaluating VTM on IND data
Figure 10 is analogous to Figure 2 in the main body. From the RD curves, we see that VTM outperforms FR NIC and JPEG22000 across all tested bpps. VTM's 's plots of D are most similar to JPEG2000's. Specifically,
low compression rates (high bpp/PSNR), both methods leave similar errors across all frequencies (notice the small gradient of colors in both JPEG2000 and VTM's plots Df at med/high bpp and med/high PSNR). Similarly, at high compression rates (low bpp/PSNR) both models leave more spectral errors at the low and medium frequencies. Despite having similar patterns to JPEG2000, most of the VTM plots have a smaller magnitude than JPEG2000, even at fixed-PSNRs. This suggests that the metric D metric is more closely aligned with VTM's modus operandi than JPEG2000's.
'For example, when 9=17, VTM uses 2.32 bpp on Kodak and 16.49 bpp on Kodak-C C shot noise with severity=5.
22

---

D(C, X): Fixed-bpp
D(C, X): Fixed-psnr
low psnr med. psnr high psnr
low bpp
med. bpp
high bpp
VTM
VTM
(0.29, 31.66)
(0.97, 37.42)
(1.43, 39.38)
(0.18,
29. 98) (0.78, 36.31)
(0.97.37.44 37.42)
0
50
0.0
0.5
0.00
0.05
0
250
0
1
0.0
0.5
Figure 11: Unscaled plots of g and R with VTM from Figure 10.
freq.: Snow
Med. freq.:
High freq.:
FR NIC JPEC20D0
FI M.C
VTM
(085
VIM
PEG2000
25
30
100
30-
30-
20-
25
JPEG2000
VIM
10-
0.0
Image size (bpp)
Image size 1bpp)
Image size tappl
Low freq.: Snow
Med. frequ: Glass Blur
High freq.: Shot Noise
NIC
1.6
F2C20000
122
23.5
JPE52000
NIC
PSRR
1.-
23.0
PSAR
22.3
22.0
0
3
0.0
0.5
1.0
1.5
10
15
Image size (bpp)
Image sibe (bpp)
Image size (tpo)
Figure 12: Rate-distortion curves for representative low, medium, and high-frequency shift. Each shift and model has three curves for severity=1 (least transparent), severity=3, and severity=5 (most transparent). Top row: generalization of C(c(2)) w.r.t. <(X²) (i.e., PSNR of the reconstructed shifted images w.r.t. the original shifted images). Bottom row: denoising of C(c(X)) w.r.t X (i.e., PSNR of the reconstructed shifted images w.r.t. the original clean images).
Figure 11 shows the six VTM plots from Figure 10 without constrained color map scales. These results confirm that VTM leaves only a small magnitude of error, which is even across frequencies.
D.2 Evaluating VTM on OOD data
Figure 12 shows the RD curves of VTM on a low, medium, and high-frequency shift. On the low-frequency shift (snow), VTM has comparable performance to FR NIC and JPEG2000 low bpp: however, VTM can achieve higher PSNRs by increasing its bpp usage. This finding suggests that VTM, like the other codecs in Section 4.2.1, generalizes well to low-frequency shifts. However, this highlights the fact that VTM may drastically increase its bpp usage on OOD data. Interestingly, all three models comparably robust to the snow corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to X'
On the medium-frequenm PSNRs which are higher than the PSNRs on clean data. However, VTM is not as effective as FR NIC in terms of PSNR with respect to c(X² on this shift (the VTM curves start to flatten around 40-42 PSNR, while the NIC curves flatten around 42-47 PSNR). Like the snow corruption, all three models comparably robust to the glass blur corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to X.
shift (glass blur), VTM, like the other codecs in Section 4.2. achieves
On the high-frequency shift (shot noise), VTM displays drastically different pattern from both NIC and JPEG2000. Specifically, VTM is able to achieve significantly higher PSNRs with respect to c(2) (i.e., generalize better) than FR NIC and JPEG2000. However, this comes at very serious cost in
23

---

AC. c)
R(C, C)
JPEG2000
JPEG2000
-1
sev.
Sev.=1
sev.=5
-1
Sev.
Snow
10.80,
10.84, 32.55
0.0001
500
500
(9.80, 39.03)
10 10, 16.01)
[1:13, 42.86)
10. 10:45, 43.99)
10.00, 24
10,80, 22.53.
11:15, 24 70)
(9AM) 22.401
500
300
500
300
Photen
(0.80, 21.621
10.90. 10.65)
(0.05, 20.46)
(1.02. 11.54)
10,90, 25 01)
(0.90, 16.45)
10 10.05, 25 79)
(1.02. 14.631
Figure 13: Generalization error 9 and denoising error R. for JPEG2000 and VTM. We plot both spectral metries for one low, medium, and high-frequency corruption severities 1 and 5. Fach plot is labeled with tuple of that model's (bpp. PSNR) on the Kodak-C dataset with that corruption.
terms of bpp (notice the bpp range for shot noise Figure is 0-17 versus a typical range of 0-2). Recall that in Section 4.2.1 we observed that NIC models adjusted their bpp usage depending on the type of corruption- in particular, NIC models used more bpps than JPEG2000 on the shot noise corruption, even though it did not improve the PSNR. Figure 12 shows that VTM adjusts its bpp usage in a more extreme way than NIC models, but, unlike NIC models, this increase in bpp does result in increase in PSNR with respect to c(X'). As expected from our results in Section 4.2.1, this increase in PSNR with respect to c(X) results in a decrease of PSNR with respect X. Overall, VTM's highly variable bpp usage could be real disadvantage a practitioner selects hyper-
using VTM in the wild: in partieular, amount of bpps, VTM might parameter from clean data and expects the model will use a certain much larger bpp than expected on OOD data. Thus, OOD benchmark dataset is a vital aspect to comprehensive image compression testing.
hyper-parameter
Figure 13 shows the G and R metrics on the three representative shifts.? On the snow corruption, VTM has a smaller magnitude of errors than JPEG2000 across all frequencies; however, both models leave relatively high frequencies (which is the opposite as FR NIC on the snow corruption Figure 4). At both severities, the two models have similar plots of G. VTM is highly effective at reconstructing the glass blur- the magnitude of G is <0.0001 across all frequencies- which matches our findings on the other codecs in Section 4.2.2. Additionally, JPEG2000 and VTM have nearly identical plots of R at both severities. Finally, on the shot noise corruption, both JPEG2000 and VTM display similar magnitudes and patterns of errors. Both models leave significantly more on G severity=5 than severity=l; however, JPEG2000 maintains a bpp=0.8 at both severities while VTM increases its bpp from 0.05 to 1.02. Additionally, unlike NIC, both of these models leave their highest errors in the low-frequency domain of G. Lastly, both models are not as effective at denoising high-frequency signals as NIC proved be.
errors across all frequencies and both models distort low frequencies more than
E
Additional results: ELIC
"Note that we select one hyper- papaa-parreeter for JPEG2000 (q=15); however, we vary the hyper- syper-parameter choice for VTM across corruptions (q=32, 17, 47 for snow, glass blur, and shot noise respectively) so that VTM gives bpps the ballpark of 0.8 for each corruption.
DIC,X)
A-0.003
A-0.025
(0.16,
(0.25, 33.23)
(0.53, 36 66.60)
Figure 14: More detailed plots of D for ELIC. Equivalent to the ELIC row on the right side of Figure with smaller colorbar ranges.

---

Additional results: accuracies by individual corruptions
32.97
10%
Diffrence Accuracy
Diforence Accurscy
-10%
(psnr=32.97)
30%
(ssmt=37.19)
(2.08=32.84)
JPE22000 (psnr=36.80 60)
JPEC2000 (pwrr 52.922
PPC2000 (3.2)331.44)
IPEG
105nr=37.11)
JPEG (3.11333.11)
PEG (2onr=EL11))
ooan
snow-
nest
biihttiea
motton
defocis-
0022
pisetate
Impuise
ooan
frose
top
brightier-
contratt
sotion
denn
frott
Dre
irpube
Figure 15: Effect of compressing corrupt images on classification accuracy. Specifically, let A(X)
be the top-1 accuracy of the model on dataset X, measured in percentage points. We report A(C(c(X))) A(c(X) over all -C corruptions with severity (or on clean ImageNet in the case of "clean"), ) Each subfigure shows results under different fixed-PSNR constraint based on the PSNRs achieved by the compressors on the clean ImageNet dataset. Appendix M has tables with these accuracies.
25

---

G
Additional results: Kodak and Kodak-C
This section contains Figures equivalent those in the main body for the Kodak and Kodak-C dataset.
Original
Snow
Glass Blur
Shot Noise
700
Figure 16: Figure equivalent to Figure la for Kodak-C dataset.
26

---

INC,0: Fixed-bpp
RD Curves
INC.N1 Fixed-psnr
low bpp
med, bpp
high hpp
low psnr
med. penr
high penr
SH NNC
40
ELIC
PS51 (6.9)
26
10:30, 31, 10.93, 37,
(1 60.30 60.511
10.23, 78)
(0.33,
IPEG2000
IPSG2000
PEG
10:27: 28.761
10.92, 34 25)
(1.50, 35.971
30
SH NIC
EUIC
FEG
10.39, 30.36)
<0 60, 32.081
(1.09 35.14
PEG2000
JPBG
10.28. 25. (5.
10.96, 32:51)
(1.37. 34.521
10.59, 29 89)
10.91. 52.17)
(1.57. 35:371
2.0
1.5
100
15
12
Image size (bpp)
30
20
Figure 17: Figure equivalent to Figure for Kodak dataset
Low freq.: Snow
Med. freq.: Glass Blur
High freq.: Shot Noise
45
22
20
FL52000
40
pPn
1.
25
16
ELIC
ELIC
JPEG2000
PEG2000
10
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
size (tpp)
Ibpo)
frequ:
Med. freq.:
High freq.: Noise
SH NCC
SH MCC
27.5
SM KIC
ELIC
ELIC
ELIC
JPEG2000
24
PEE2000
25.0
PEG2000
JPEG
22.5
JPEG
1.-
17.5
12
21
12.5
0.0
Image size (top)
Image sizw Ihpp)
Image size Ospa)
Figure 18: Figure equivalent to Figure 3 for Kodak-C dataset
QC,, x.
RIC, X:
JPEG2000
JPEG2000
SeV =1
sev.=5
sev.=1
sev.
seV.-1
sev.
=
Sev.=5
0
0.1
0.1
500
500
0.0
0.0
(0.4, 39.8)
(0.2, 43.9)
10.9. 40.5)
10.0, 46.6
[0.4, 24.8)
10.2, 22.5)
(0.9, 24.8)
(0.9,
700
700
700
700
0
(2.9.
(2.7, 11.5)
(0.9.21.7) 21.7)
10.9. 10.81
(1.9.
12.7, 15.4)
(0.9, 24.6)
(0.9, 14.1)
Figure 19: Figure equivalent to Figure 4 for Kodak-C dataset
27

---

H Additional results: other CLIC-C corruptions
Figures 20, 21, and 22 show the results of the other 12 corruptions from CLIC-C which were not included in the main body. Overall, we see that our three representative corruptions (snow, glass blur, and shot noise) represent the trends of the low, medium and high frequency corruptions well.
Motion blur
Zoom blur
Snow
Frost
Fog
Brighttness
Contrast
Defacus blur
Glass blur
Elastic transform
JPEG compression
Goussian
Shot
Impulse noise
Hovelate
Figure 20: Average PSD of the difference between the corrupted images and the clean images for each given CLIC-C corruption N
1 Σ PSD(c(X.)) Xx). Severity=5.
28

---

Low freq.: Frost
Low freq.: Fog
Low freq.:: Brightness
Low freq.: Contrast
42.5
40.0
50
40.0
(t..1/))) (682)
PSHR Wrr. C10 (08)
37.5
PODR W172. <110 (d0)
17.5
SH NIC
SH NIC
30.0
MCC
-5
27.5
PEG2200
JPEG2000
23.5
EUC
G2E
JPES2000
25.0
PTG
JPGG
25.0
JeG
PPG
0.0
0.5
1.0
1.5
2.0
0.0
0.5
100
1.5
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
Image size (opp)
Image size (spp)
Image sizn (bop)
Image size Dep)
Low freq.: Frost
Low freq.: Fog
Low freq.: Brightness
Low freq.: Contrast
14.0
25WR w.r. (010
20-
PSNR w.i.t (110))
16-
SH NIC
13.3
SH NIC
SH MC
SH NIC
wot.
ELIC
IPEGZ000
13.0
IPEG2000
JPEG2000
JEE2200
JPEG
JPEG
34-
JPEG
JPEG
11.0
12.5
10.5
12.0
10
0.0
0.5
1.5
0.5
1.0
1.5
0.0
0.5
1.0
2.0
1.0
1.5
stze
size tbpp!
(bpo)
Med. freq.: Motion Blur
Med. freq.: Zoom Blur
Med. freq.: Defocus Blur
Med. freq.: Elastic Transfor
42.5
45
PSWR W.Tt CMC (OE)
40
PAA W.rt C1C
$7.5
35.0
15
32.5
ELIC
EUC
ELIC
30.0
ELIC
IPE2000
30
PPG2000
JPE2000
27.5
JEG2000
JPEG
JEEG
25.0
0.0
0.5
1.0
0.0
0.5
1.0
2.5
0.0
0.5
1.0
1.5
0.0
0.5
100
size (tpp)
size (hpp)
size (bpo)
Image size (tpp)
Med. Motion
Mad. freq.: Zoom
Med. freq.: Defocus
Med. freq.: Elastic Transfon
SHINC
SH KIC
25
19:3
EUC
27
ELIC
23-
24-
19.0
JPEG
JPEG200O
PSMR wrtt (08)
24
PONR wrr.. 21 1002)
24
JPEG2000
JFEG
22
JPE20000
10.3
21
JPEG
18.0
23
SH MC
EUC
22
17.5
JPE2000
PEGG
19
17.0
21
Image size (Spp)
Image size tbpp!
Image (bpo)
Image (bop)
Med. freq.: JPEG Compressio
Med. freq.: Pixelate
High freq.: Gaussian Noise
High freq.: Impuise Noise
PENS wrr.. clto (EB)
EUIC
JEG2000
PaRe wrr.. c100 (83))
20
.20
JieG
SH NIC
35
18
25
30
JPECZZ000
SH NIC
SH MC
JPEG
[LIC
30
EUCC
PEG2000
PES22000
12
Image
Image
Image
Image
freq.: JPEG Compressio:
freq.: Pixelate
High freq.: Gausslan
High freq.: Impulie Noise
SH MC
27.5
SH NIC
30
27.5
EUC
25.0
JPEG2000
FEE2000
28
25.0
JPEG
22.5
JEGG
28
22.5
26
SH NIC
26
SH NIC
20.0
17.5
37.5
PEG2000
24
PEG
24-
PEG
125
12.5
0.0
0.5
15
1.0
1.5
Image size (hpp)
Image size (hpp)
Image size (bog)
Image size thpp)
Figure 21: RD Curves for corruptions not seen in the main body. Rows 1, 3, and 5 show PSNR w.r.t. (()). Rows 2, 4, and 6 show PSNR w.r.t. X.
29

---

GtC.x.
SH NIC
ELIC
JPEG2000
JPEG
sev.m1
sev.m5
sev.m1
sev.
sev.m1
sev.m5
10
5e%.
S0V
15
frost
Frost
Frost
10.8. 36.41
10.6.
(0.8, 37.81
(0.5, 39.0)
0
10.0.
(0.0. 37
(0.8, 32.0)
(0.7. 35.0)
Fog
Roz
10.4, 40.51
10.3, 42.11
(0.3, 41.4
(0.2. 42.8)
10.8, 41.4)
(0.8, 43.8)
(0.4,
10.3, 39.6)
Brightwess
Brohtmnss
25
10.0,
(0.8.
Contrast
Cantrast
Contrast
10.4. 40.71
47.1.47.11
(0.1,41.4)
(0.0) 47.9)
0.0
10.0,
(0.8,
(0.4,37.4
(0.2, 44.4)
Nooson
Modion
Dhr
Blu
10.4. 40.11
10.2. 42.71
(0.4, 40.91
10.2, 43.5)
10.8, 40.3)
(0.5, 44.5)
10.0.
36.4)
10.4, 39.51
0.01
0.1
7room
Zoom
tlur
Zoom
Blur
0.0
10.2. 42.41
10.2. 42.01
0.00
0.0
0.1
0.1
01
Dotoous
Dielocuus 18ar
Dafoxus Sur
10.3, 42.71
(0.1, 46.3)
0.0
10.2, 43.66
10.1.47 47.31
0.000
(0.8, 44.4)
(0.8, 48
10.5, 38.9)
(0.3, 41.9)
0.0
Farsfom
Elsite sarsdom
Eards
Elasti
10.6. 38.31
30.7.38.21
00.6, 39.2]
10.6, 39.1)
10.8. 37.21
(0.8, 37
10.7, 34.3)
10.7. 34.21
JPSG Comorrsssion
jisq Contorestio
IPSG
(0.7, 38.6)
Hoelite
10.7. 36.61
10.5, 38.5)
10.7. 38.01
39.5.
39.51
(0.8, 36.4)
(0.8, 30
(0.7.33.21
(0.5, 96.1)
Geuzdden
Cauocian
Couosion
Vidze
11.8. 23.21
12.7.32.11
2322238
(4.1.226)
10.8, 22.0)
(0.8, 3)
11.2.22.21
(3.1. 12.01
700
500
700
700
Impolee
Impulhe 1olse
impulte
Molse
Figure 22: Generalization error G on the other corruptions of CLIC-C. We plot for each corruption at severities and 5. The plots of G are labeled with tuples of the model's (bpp, PSNR c x)).
30

---

SH NIC
ELIC
JPEG2000
JPEG
300
500
500
Frost
10.8. 13.01
10.6, 10.51
10.8, 12:9)
10.6, 10.5)
(0.8. 12.8)
(0.8, 10
10.0. 12.99
10.7. 30.51
Fog
10.4, 14.3)
10.3, 12:21
(9.3, 3)
10.2. 12.2)
10.0. 14.3)
(0.8, 12
(0.3,
Brightness
500
Brightrets
500
Brightrass
500
Brightnass
500
10.8. 21.91
(0.8,
10.7.
500
Controset
Contrast
16.7.16.7)
10.1, 12.81
(0.3, 16.8)
10.0. 12.81
10.0, 16.0
80.8, 12
10.4.
10.2, 12.01
500
Miction
Mection
Mesiio
Strr
10.4,
10.2. 500
(0.8,
560
500
500
7hom
Zieen ular
Zoom Blu
10.3, 19.8)
10.3, 17.71
10.2.
10.2. 17.71
(0.8.
Delocus
Delocuu 1aur
Dañocus Blu
27.40
10.1,22.00 01
10.2, 27.6)
10.1, 22.8)
(0.8, 27.5)
(0.8, 22.8)
10.5, 27.21
10.3, 22.71
Manifon
500
Elastic s-rsftm
Elattcc
Elestic
10.6. 25.6)
(0.6,
(0.8, 100
300
100
JPSG Compresson ascon
JPSG Comorestion
100
JPEG
10.7. 31.5)
(3.7, 31.3)
10.7. 26.3)
(0.8, 30.5)
(0.8,
10.6,
Prcate
Homitee
Rasiate
10.7, 30.91
10.5, 26.21
10.7, 30.6)
10.5, 26.11
10.8, 30.0)
(0.8, 3).9)
10.7, 29.5)
(0.5, 25.8)
500
Counoo-n
Galoo an kohne
Goussion
Nu550
600
600
600
600
Irpubse Noiree
impuise Nulo
12.0. 24.9)
12.7.
15.4]
(2.5, 24.3)
[4.1. 14.4)
20.3)
(0.8, 3)
13.4.
Figure 23: Robustness error R on the other corruptions of CLIC-C. We plot R for each corruption e at severities and The plots of G are labeled with tuples the model's (bpp, PSNR w.r.. X)).
31

---

I Training setup and hyperparameters
We train NIC models using the train split of the CLIC 2020 dataset with batches of size 8 and random crops of size 256 x 256. For SH NIC, we set N = M = 192 and train for 5,000 epochs (about 1M iterations). We trained 8 models with As 0.0012, 0.005, 0.01, 0.03, 0.05, 0.1, 0.15, and 0.26. We used the pytorch model architecture in the compressai repository [10]. For ELIC, we use N = 192, .M- 320 and train for 3,900 epochs. We trained 11 models with unique Às (0.001, 0,0025, 0.003, 0.004, 0.006, 0.008, 0.016, 0.025, 0.032. 0.05, 0.15). We utilized this repository for the model architecture, but trained our own models on CLIC rather than experimenting with the publicly available checkpoints trained on ImageNet. Note that we trained more models for ELIC than FR NIC in order to obtain bpps and PSNRs which fit our bped-bppPPSNRR constraint.
For the variable-rate models, we augment each convolutional layer with a FiLM layer which consists of two fully-connected layers and 128 hidden features. We train VR models for 10,000 epochs (about 2M iterations) and sample À from a log-uniform distribution over [0.0012, 0.26]. For pruned models, we use global gradual magnitude pruning (GMP) between epochs 675 and 2400 and tune the remaining weights from epochs 2401-10,000. Fixed-rate MS- AS-ISIM models are trained over several values of X [1, 1000].
Variable-rate model details
For our variable-rate NIC we implemented a model proposed by Dosovitskiy and Djolonga [17] which uses Loss Conditional Training (LCT) train a variable-rate version of the scale-hyperprior NIC model from [9]. The model is trained over continuous range of compression rates, optimizing the rate-distortion tradeoff each. This reduces the computational redundancy involved in training separate models for the same task at different points along the rate-distortion curve. Additionally, once trained, this model can be adaptively used at any compression rate in the range, which makes even more flexible than a discrete set of fixed-rate models.
In order to condition the compression rate via LCT, all convolutional layers in the model augmented with Feature-wise Linear Modulation (FiLM) layers [49]. In this case, these are small neural networks that take conditioning parameter the activations channel-wise based on the value of X. More specifically, suppose a layer in the CNN has activations f of size W x H x C. In Loss Conditional Training (LCT), these activations are augmented by and o as follows,
input and output μ and a used modulate
=f +
(I)
where both μa and or are vectors of size C.
Figure 24 shows diagram of FiLM layers being applied to a small example network. Normally, without FiLM layers, the equation to calculate the features maps is
(1) xx()))(()) + b()),,
where p the activation function (e.g., ReLU), W are the weights and b are the biases in the layer. However, with FiLM layers, the weighted sum is first subject to augmentation with and a.. Thus, the new calculation is
(I) = (I) + =b())) + (1))
During training, a random À drawn with each mini-batch and used two ways: 1) as input to the FiLM layers to modulate the feature maps and 2) in the mini-batch's loss function. During inference, the model takes as input a desired compression rate À along with the input image. Notably, LCT and FiLM layers only affect the values of the activations (i.e., feature maps), but not the weights on the convolutional layers or activation functions. This is an essential observation for applying pruning to a network with LCT.
K Pruned model details
One limitation of using NIC in the wild is the size of the NIC models. These models require several million learned weights and efficient inference requires specialized GPUs. This limits the possibility
32

---

A
1
37.
302
223
W()
W(2)
x (0)
(1)
(2)
Figure 24: Cartoon of FiLM applied to a multi-layer perceptron. We consider pruning both the base
layer weights (blue) and the film layer weights (red).
of using NIC on an edge device, like a drone or a cell phone. In this section, we demonstrate that NIC models themselves can be approximated (i.e., we can compress the compressor) and we propose a sparse versions of both fixed-rate and variable-rate NIC models.
To prune our NIC models, we propose gradual magnitude pruning (GMP) [70] based approach which has been demonstrated to effectively sparsify DNNs. At a high level, GMP gradually prunes DNN weights based on their magnitude over a fixed number of epochs then continues to train, fine-tune, the remaining sparse network after the target sparsity is reached. The pruning can be done layer-wise (i.e., the mask must be updated such that each layer has the target sparsity) globally (i.e., the overall network must have the target sparsity, but the sparsities can differ between layers). We utilized global pruning, which has been demonstrated to produce more performant sparse models.
To implement GMP, each layer is given binary mask of the same size as the original layer, which specifies which weights have been pruned (i.e. set to 0). Every At iterations, the mask is updated based on the magnitudes the weights, such that the network has a desired sparsity. Suppose s; is the initial sparsity (usually 0) at time to and s is the target sparsity (0.8 or 0.95 in our case) which we want to reach after n steps, then the target sparsity at time step t is
87) (1)) (1
t to nAt
sy + (82
(4)
for f E {to, to + At,..., + nd...
(5)
The target sparsity equation found to be especially effective because it causes weights to be pruned more quickly while the learning rate is still high and the weight-training on non non-pruned weights can recover. After the desired sparsity is reached, the sparse network is trained for several more iterations. The pruning can be done layer-wise (i.e., the mask must be updated such that each layer has the target sparsity) or globally (i.e., the overall network must have the target sparsity, but the sparsities can differ between layers). Note that the network still performs weight training on the non non-pruned weights, as usual, using an optimizer like stochastic gradient descent.
L Mathematical details
L.1 Derivation of lemma 5.1
Let X E NxP be a dataset (whose datapoints are the rows) with mean 0 and covariance Σ 1 Consider the problem of linearly auto-encoding X' with latent space R.,, where S p.
N xxxx X.
min Wie WieM
xw W - x12
(6)
33

---

It is a theorem of Eckart and Young [18] that for any solution W1. W22 of eq. (6) the product W,W, an orthogonal projection onto the subspace spanned by the top principal components of X' (see also [7] and for more recent developments and additional references, [50, 35]). Informally, this theorem says "linear autoencoders perform PCA." " Formally, if X = USV is an SVD and V[: r is the first rows of V, then W = V|: v| r]. It follows that the reconstruction error ((X) on a new input X E IR? admits a simple closed form: let Vi...., Vp be the principal components of X (rows of V above), and write
x= Ee Viv then ((X) = Σ G.
(7)
121
Suppose now that X' is a dataset of natural images. We will replace the single index for X (XI....,X, E Ip' with 3 indices h, i,j for an image tensor X = (Xn,ig) E R R*HW (here h is viewed as the channel index and j as the spatial indices); we will also commit mild abuse of notation by letting the matrices W; act on flattened (vectorized) tensors. Let F RKHW - RRHH denote the spatial discrete Fourier transform. We introduce the notation
(8) and we will frequently use the shorthand "hat" decoration "C" to denote as well, for case of notation. Just as C projects onto the subspace spanned by the top principal components of X, č projects onto the subspace spanned by the top r principal components of X.
x and é = FCF",
It has long been known that when transformed to (spatial) Fourier frequency space, there reasonable (coarse) descriptions of the statistical distribution of natural images. Its variances in spatial frequencies ji can be modelled as
where a, B ~ 2 and its covariances drop off rapidly away from the diagonal [12, 363." These two facts motivate the following assumptions:
+
(I) The principal components of & are roughly aligned with the spatial Fourier frequency basis. (II) The associated variances are monotonically decreasing with frequency magnitude (more specifically according to the power law +>+))>>
Assuming a = 3 = 2. the above would suggest that è projects a Fourier transformed image X = FX onto Fourier basis vectors corresponding to spatial frequencies (i,j) with 12 + j² k That is,
(C)) ~ {o [
if 52 K
(9)
otherwise.
where X..i denotes the components of X corresponding to spatial frequency (i,j) (this is vector of dimension K, the numher of channels in the image dataset). Together with the identity FCX = ČX, eq. (9) yields lemma 5.1.
Remark L. Note that lemma 5.1 also suggests that such autoencoder compressors behave qualita- tively similarly to classical codecs such JPEG, which involves discrete cosine transform (DCT) followed by removal of high-frequency components [53]. While design of hand-crafted codecs like JPEG was doubt driven by empirical observations of the statistics of natural images, the NIC models in our experiments (and at least in a theoretical sense the autoencoder model appearing in lemma 5.1) are machine learned from natural image datasets.
Below, we use the above analysis get some qualitative insights into the robustness and generalization metrics R and G introduced in section 3. In what follows let c KWW + KKHW be a corruption transformation.
L.2 Derivations of corollaries 5.2 and 5.3
Robustness: here we are interested in PSD(X- - C(c(X))), which simply the coordinatewise absolute value of X C((c(X)). Using the fact that in our simple model C is linear, we can expand like
x x-C(c(X)) = = C(c(X)) x C(x) + C(c(X) X)
(10)
although is in general not the case that W1 = V|: r], W2 V(: rj" see [50, &G] for discussion.
'err we assume that our Fourier transform F' has the property that frequency (0, 0) corresponds to constant
images.
34

---

so that for any frequency (i.5)
- (acc(((()))).. = 1(x ( c(()) + +2 - ((xx))
- -8)
((((X) - x),
(1I)
[(((x) - ))ibp1
when +2 S F0
12,,,113
otherwise.
Taking square roots, we see that
[c(X) [X,sssl
x)s when 12 + S nC otherwise;
PSD(X- - C(c(X))):
(12)
averaging over the dataset X' gives corollary 5.2. In particular, the cross term 2(X -(X)) truined on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that (c(X) - )),a 12 is large for small values of if).
X).,s is identically This suggests that autvencoder compressors
C()(()))) which is simply the coordinatewise Generalization: here we are interested in P
absolute value of c(X) - C(c(X)). Again using the fact that in our simple model C' is linear, we can expand like
( - c(c()) = + (c(X) - 8) - -(( (c(X) - X))
(13)
= (X - C(x)) + ((((x) 8) - c(c(() -X))
From this we obtain
((((X) - c(c(
= =1(8 - -(()) les! +2(X - ((x)) 118 ((((X) - -x-
--- -c(((() 8))
+ +(((((x) - 8) -C(c(X) x)))..²'* 1²
(14)
0
when 12 j2 #C
~
18 + 2X 28 (c(() )) + [c((X) X), 1
otherwise.
Taking square roots and relating back to power spectral density, this says
- -(c(x))
when 12 +2 :/6
(15)
( 12 + 2X ,x(x)) )). [(c(X) X ).. otherwise
and averaging over the dataset X' gives corollary 5.3. Equation (14) is more involved than eq. (11) in particular, the "cross term" i5 X) )). is in general non-zero. However, there are many cases where at least an expectation over the data set X' the term (c(x)) - {).. vanishes (for example, when is additive noise, or more generally when c(X) - X is statistically independent of X.). In such cases, we can see that autoencoder compressors trained on natural images generalize less successfully corruptions with large amplitude in high frequencies.
L.3 Autoencoder reconstruction error and data density more generally
Our findings on generalization (both the experimental results and analysis with linear autoencoders) is closely related the widely observed inverse correlation between autoencoder reconstruction error and probability density. We will take the recent [29] as jumping-off point. However, it is worth noting that this observation (or at least something similar to it) has been around for some time, least since [5], and forms the basis for widespread use of autoencoders in anomaly detection.
Formally, [29, Observation 2] states that:
35

---

(*) For an autoencoder C' the reconstruction error [c(X) X l2 is positively correlated with p[X]' the probability density of the input data at X.
we assume this observation holds, then some of our experimental findings regarding generalization to corrupted data admit simple explanation: discussed above, the probability density of natural images is heavily concentrated along low (spatial) Fourier frequencies. Hence for a corruption e such that c(X) X concentrated in high frequencies, is reasonable to suspect that p(c(X)) < p(X), with [C(X) X|2, that
and given the assumed positive correlation of p(X)
C(c(X)) - -((x) > [c(x) 2
(16)
L.e. the reconstruction error of the corrupted datapoint, related our generalization metrie, higher than that of the clean datapoint X.
Remark L.2. In some sense. (+) is simple consequence of learning via risk minimization. Let M - M be an autoencoder on a manifold M with finite volume (for example, the image hypercube (0, 1)CHW). Let ((C(X), X) be a reconstruction loss point X E . and let p(X) be the probability density of input data on M, assumed to be positive everywhere (otherwise, discussing correlation with _ is troublesome). The risk in this situation is p(X)
[(C) =- J.x X)((X),
(17)
Note that {(((X), X) and Miy will be positively correlated provided that their covariance positive. By definition this covariance
p(X)
M [ ((((X), X)
S.M p(X) p(
p(x) p( p(X)d -
(18)
X(C(X),X)dr -(C)) volM.
M
Since at present we only care about the sign of the covariance, we can divide by volM to obtain
S. ((C(X),
dx
L(C);
(19)
VolM
the first term the risk of C with respect to the uniform distribution on M. In words, eq. (19) is positive whenever C performs better on the distribution p(X) than the uniform distribution, and this outcome is to be expected for an autoencoder with sufficient capacity trained on nonn non-uniform distribution p(X).
Remark L.3. With notation in the previous example, the result [4, Thm. 2] shows that the solution obtained with calculus of variations applied to an objective of the form
min S. (IC(x) x13 xx
(20)
(i.e. mean squared error loss and an penalty on gradient norms) satisfies
C(X) X x= 22V log p( ))= + o(a²) and
(21)
VxC(X)= = I o² Hess log p(X) + o(a²) as o - 0.
(22)
In particular, the reconstruction error [C(X) - X|2 is proportional to the gradient norm V (ogg(())2 (up to higher order terms in o² as 0). Since
Vx logr(X))2 = p(X) | (xP(())
this result sheds more precise and quantitative light on (*).
Equation (22) may also shed some light robustness of C as measured by R: indeed, if we assume
c(X) X is small,
C(c(X)) = C(( + +c(x)= X) X ((X) Tx(T(c()) -x) - X. -x)
(23)
36

---

Applying eqs. (21) and (22) gives the approximation
X) X V log
(24)
+ (I + o² Hessx logp(X))(c(x) x) + o
as 0. Taking norms squared we get
(C(c(X)) - x ~ logp())
+ 20 logg(()) o² Hessx logg (x)
(25)
+ (z + 0² H logp(X)) x) +
as a + O. we again make the simplifying assumption that c(X) X random and independent of X, then after taking the expectation over e (here denoted by E.) we obtain
Fe(lC(c(X)) - x(3] ~ o² Vx 1xg)(X))²
(26)
+ Ed o² H log logp(X))(c(X)) x)] +
0. If we let a 02 2 ap be the eigenvalues of Hess log p(X) and choose small enough that + > for all i, we see that the norm
(1 + ²² Hessx log (X))(( x)22
(27)
surpresses (resp. magnifies) contributions from components of c(X) - X along the eigenvectors associated with a for large (resp. small) i. Note that it is reasonable as a first approximation to expect Hess log p(X) to be negative definite (this is precisely the case when is Gaussian distribution, and more generally the requirements p(X) 2 0 and p(X)dX = 1 put restrictions on the extent to which Hess x log pXX) exhibit non-negative eigenvalues). In this case the large (resp. small) values of correspond to the eigenvectors in directions of "sharpest" (resp. "shallowest") curvature of the graph of p(X). Explicitly, if X is Gaussian with covariance E, then Hess log p(X) Σ-1, and if À 2 X2 2 ... 2 >p are the eigenvalues of Σ then a; = X; for all i. In this case we see that the norm in eq. (27) surpresses (resp. magnifies) contributions from components of c(X) X along principal components with low (resp. high) variance. Note that this provides a takeaway qualitatively similar to that of corollary 5.2.
L.4 A fundamental robustnes--ennraalzzatiin inequality
The similarity between the R heatmaps for glass blur in fig. 4 and the PSDs of the corruptions themselves appearing in fig. la has a simple explanation in terms of fundamental relationship between D,G and R. One can show (via two applications of the triangle inequality) that
PSD(X; c(X+)) G(C, RCC, <
PSD(X, c(Xx)) + GCC,X,c). (28)
N
In the case of glass blur, generalization error is very low, so eq. (28) reduces R s
N EE PSD(X, - c(Xx)), where the right-hand side is the average PSD of the corruption itself.
I
M Tables of classification accuracies
37

---

No Compression
clean
0.76
snow
0.34
frost
0.31
fog
0.46
brightness
0.69
contrast
0.44
motion blur
0.37
zoom blur
0.35
defocus blur
0.36
glass blur
0.16
elastic transform
0.53
jpeg compression
0.59
pixelate
0.50
gaussian noise
0.32
shot noise
0.29
impulse noise
0.32
Table 1: Classification accuracies without compression on the validation split of ImageNet-C.
38

---

SH NIC
ELIC
JPEG2000
JPEG
PSNR=36.93
clean
0.72
0.72
0.72
0.74
snow
0.33
0.31
0.30
0.33
frost
0.29
0.29
0.30
0.30
fog
0.28
0.28
0.45
0.44
brightness
0.65
0.65
0.65
0.68
contrast
0.26
0.26
0.44
0.42
motion blur
0.35
0.34
0.36
0.36
zoom blur
0.33
0.32
0.34
0.34
defocus blur
0.30
0.30
0.36
0.36
glass blur
0.19
0.19
0.17
0.16
elastic transform
0.48
0.49
0.52
0.53
jpeg compression
0.60
0.60
0.59
0.60
pixelate
0.55
0.53
0.51
0.50
gaussian noise
0.33
0.36
0.28
0.30
shot noise
0.29
0.32
0.25
0.28
impulse noise
0.31
0.34
0.23
0.30
PSNR=32.97
clean
0.64
0.64
0.67
0.70
snow
0.27
0.26
0.25
0.28
frost
0.25
0.25
0.28
0.28
fog
0.13
0.11
0.38
0.28
brightness
0.57
0.55
0.59
0.63
contrast
0.11
0.09
0.38
0.27
motion blur
0.30
0.29
0.34
0.32
zoom blur
0.29
0.28
0.32
0.30
defocus blur
0.26
0.24
0.34
0.30
glass blur
0.22
0.24
0.19
0.18
elastic transform
0.42
0.45
0.50
0.51
jpeg compression
0.57
0.57
0.57
0.59
pixelate
0.53
0.51
0.53
0.51
gaussian noise
0.28
0.30
0.25
0.37
shot noise
0.26
0.27
0.23
0.35
impulse noise
0.29
0.32
0.17
0.37
PSNR=31.17
clean
0.58
0.57
0.62
0.66
snow
0.22
0.21
0.21
0.26
frost
0.20
0.20
0.25
0.25
fog
0.08
0.07
0.32
0.17
brightness
0.51
0.48
0.54
0.60
contrast
0.07
0.05
0.34
0.16
motion blur
0.27
0.26
0.33
0.28
zoom blur
0.26
0.25
0.31
0.27
defocus blur
0.24
0.22
0.32
0.24
glass blur
0.23
0.24
0.21
0.19
elastic transform
0.39
0.40
0.48
0.50
jpeg compression
0.52
0.53
0.55
0.58
pixelate
0.49
0.47
0.53
0.48
gaussian noise
0.28
0.30
0.21
0.33
shot noise
0.28
0.31
0.21
0.32
impulse noise
0.29
0.31
0.16
0.33
Table 2: Classification accuracies from Figures 5 and
39

---

FR NIC
NIC
VR NIC
VR NIC
FR NIC opt. MS-SSIM
JPEG2000
Pr
pr-0.95
bpp 1.23
clean
0.74
0.73
0.73
0.73
0.74
0.72
0.33
0.33
0.33
0.32
0.32
0.30
frost
0.30
0.30
0.29
0.28
0.30
0.30
fog
0.36
0.36
0.36
0.36
0.41
0.45
brightness
0.67
0.67
0.66
0.66
0.67
0.65
contrast
0.36
0.35
0.34
0.36
0.40
0.44
motion blur
0.36
0.36
0.35
0.36
0.37
0.36
zoom blur
0.34
0.34
0.33
0.34
0.35
0.34
defocus blur
0.32
0.32
0.32
0.33
0.34
0.36
glass blur
0.17
0.17
0.16
0.16
0.16
0.17
elastic transform
0.50
0.50
0.49
0.49
0.50
0.52
jpeg compression
0.60
0.60
0.61
0.61
0.60
0.59
pixelate
0.53
0.53
0.54
0.54
0.58
0.51
gaussian noise
0.34
0.33
0.33
0.35
0.36
0.28
shot noise
0.31
0.30
0.31
0.33
0.32
0.25
impulse noise
0.32
0.31
0.32
0.34
0.35
0.23
bpp = 0.77
clean snow
0.72
0.71
0.71
0.71
0.71
0.69
0.33
0.32
0.32
0.32
0.29
0.27
frost
0.29
0.29
0.28
0.28
0.27
0.29
fog
0.28
0.27
0.27
0.25
0.36
0.41
brightness
0.65
0.65
0.65
0.64
0.65
0.62
contrast
0.26
0.25
0.26
0.24
0.37
0.41
motion blur
0.35
0.35
0.34
0.34
0.36
0.35
zoom blur
0.33
0.33
0,32
0.32
0.35
0.33
defocus blur
0.30
0.29
0.29
0.29
0.33
0.35
glass blur
0.19
0.19
0.17
0.17
0.17
0.18
elastic transform
0.48
0.48
0.48
0.47
0.48
0.51
jpeg compression
0.60
0,60
0.60
0.60
0.59
0.58
pixelate
0.55
0.55
0.54
0.54
0.60
0.52
gaussian noise
0.33
0.32
0.32
0.34
0.34
0.31
shot noise
0.29
0.29
0.29
0.32
0.31
0.28
impulse noise
0.31
0.30
0.31
0.33
0.33
0.26
bpp 0.24
clean
0.58
0.56
0.58
0.56
0.60
0.52
snow
0.22
0.22
0.22
0.21
0.18
0.14
frost
0.20
0.20
0.20
0.20
0.20
0.19
fog
0.08
0.06
0.06
0.05
0.18
0.21
brightness
0.51
0.49
0.50
0.49
0.53
0.43
contrast
0.07
0.05
0.05
0.05
0.20
0.25
motion blur
0.27
0.25
0.26
0.25
0.30
0.29
zoom blur
0.26
0.25
0.26
0.25
0.29
0.27
defocus blur
0.24
0.22
0.22
0.21
0.27
0.28
glass blur
0.23
0.21
0.19
0.19
0.24
0.24
elastic transform
0.39
0.37
0.37
0.36
0.42
0.42
jpeg compression
0.52
0.51
0.52
0.51
0.53
0.47
pixelate
0.49
0.47
0.48
0.47
0.54
0.47
gaussian noise
0.28
0.28
0.25
0.25
0.27
0.27
shot noise
0.28
0.27
0.25
0.25
0.28
0.26
impulse noise
0.29
0.28
0.25
0.25
0.26
0.26
Table Classification accuracies from Figure
40

---

NRC
VRC
VR NIC
VR NIC
FR NIC opt. MS-SSIM
JPEG2000
pr=0.8
pr-0.95
PSNR=36.93
clean
0.72
0.71
0.73
0.73
0.74
0.72
0.33
0.32
0.33
0.32
0.33
0.30
frost
0.29
0.29
0.29
0.28
0.30
0.30
fog
0.28
0.27
0.33
0.35
0.42
0.45
brightness
0.65
0.65
0.66
0.66
0.67
0.65
contrast
0.26
0.25
0.31
0.34
0.41
0.44
motion blur
0.35
0.35
0.35
0.36
0.37
0.36
zoom blur
0.33
0.33
0.33
0.34
0.35
0.34
defocus blur
0.30
0.29
0,30
0.32
0.35
0.36
glass blur
0.19
0.19
0.16
0.16
0.16
0.17
elastic transform
0.48
0.48
0,49
0.49
0.50
0.52
jpeg compression
0.60
0.60
0.61
0.61
0.60
0.59
pixelate
0.55
0.55
0.54
0.54
0.56
0.51
gaussian noise
0.33
0.32
0.33
0.35
0.34
0.28
shot noise
0.29
0.29
0.30
0.33
0.31
0.25
impulse noise
0.31
0.30
0.31
0.34
0.33
0.23
PSNR=32.97
clean
0.64
0.64
0.66
0.66
0.71
0.67
snow
0.27
0.27
0.28
0.29
0.28
0.25
frost
0.25
0.25
0.26
0.26
0.27
0.28
fog
0.13
0.12
0.13
0.14
0.35
0.38
brightness
0.57
0.57
0.59
0.60
0.64
0.59
contrast
0.11
0.10
0.12
0.13
0.36
0.38
motion blur
0.30
0.30
0.30
0.31
0.36
0.34
zoom blur
0.29
0.29
0.29
0.29
0.35
0.32
defocus blur
0.26
0.25
0.24
0.25
0.32
0.34
glass blur
0,22
0.21
0.18
0.19
0.18
0.19
elastic transform
0.42
0.43
0.43
0.43
0.48
0.50
jpeg compression
0.57
0.57
0.58
0.58
0.59
0.57
pixelate
0.53
0.52
0.53
0.53
0.60
0.53
gaussian noise
0.28
0.29
0.28
0.31
0.33
0.25
shot noise
0.26
0.26
0.26
0.28
0.31
0.23
impulse noise
0.29
0.29
0.28
0.30
0.33
0.17
PSNR=31.17
clean
0.58
0.58
0.61
0.62
0.67
0.62
snow
0.22
0.23
0.24
0.25
0,25
0.21
frost
0.20
0.22
0.22
0.23
0.25
0.25
fog
0.08
0.07
0.08
0.08
0.29
0.32
brightness
0.51
0.51
0.54
0.55
0.60
0.54
contrast
0.07
0.06
0.07
0.08
0.30
0.34
motion blur
0.27
0.26
0.28
0.28
0.34
0.33
zoom blur
0.26
0.26
0.27
0.27
0.33
0.31
defocus blur
0.24
0.23
0.23
0.23
0.30
0.32
glass blur
0.23
0.21
0.19
0.19
0.20
0.21
elastic transform
0.39
0.39
0.39
0.40
0.46
0.48
jpeg compression
0.52
0.53
0.55
0.55
0.57
0.55
pixelate
0.49
0.48
0.50
0.50
0.59
0.53
gaussian noise
0.28
0.28
0.25
0.26
0.30
0.21
shot noise
0.28
0.26
0.25
0.25
0.30
0.21
impulse noise
0.29
0.28
0.26
0.27
0.30
0.16
Table Classification accuracies from Figure
41


Reviews:
[{"review_id": "x9qsB4amYE", "reviewer": "Reviewer_5yiv", "summary": "This paper addressed the problem of image compression. Neural-based methods have started outperforming traditional codec methods in image compression, but there are no comprehensive datasets and informative tools to evaluate neural-based methods in real-world settings. This paper proposed a benchmark to study this that allows for evaluating out-of-distribution performance. The proposed benchmark provides some findings that are crucial to bridge the real-world adoption of neural-based image compression techniques.", "questions": "Overall, this is a strong paper. I only have some minor comments. See above.", "limitations": "The paper should address this.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Comprehensive datasets for image compression.\n2. Understanding of out-of-distribution performance.\n3. Extensive evaluations.\n4. Theoretical analyses.", "weaknesses": "NeRF-based methods can also be seen as a way to do compression (compressing images into. network weights), The authors should also discuss this and benchmark them on the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our submission. In addition to our inline response to your comment below, please note that our global response highlights additional experiments **E1** and **E2** and their insights **I1** - **I4** as well as clarifies the significance of our work in **C1** - **C3**. Namely, existing tools are incapable of highlighting inter- and intra-class similarities and differences for classical codecs and NIC models. Our benchmark and inspection tools effectively uncover similarities and differences, in turn, providing a deeper understanding of their performance. Our tools will be available to researchers/practitioners to explore and improve NIC methods impacting future NIC research and development. After reflecting on our individual and global responses, we hope that you will update your score and champion our paper for acceptance.\n \n> “NeRF-based methods can also be seen as a way to do compression (compressing images into network weights), The authors should also discuss this and benchmark them on the dataset.”\n\nThank you for this suggestion. We agree that testing a wider variety of compression methods would strengthen the empirical side of our paper. During the rebuttal period, we prioritized adding an additional “mainstream” NIC algorithm (ELIC) and standard codec (ordinary JPEG) (experiments **E1** and **E2** in our global response). While we think comparison with NeRF compression would be very interesting, we hope you will understand that adding it would require significant expansion of our research code-base and an entirely different theoretical analysis, and as such would be better suited for a follow-up project. We have added it as a future direction in the conclusion. Finally, we would like to emphasize again that our datasets and spectral tools enable the community to ask and answer these questions, which clearly demonstrates the value of our work.\n\n\n> Limitations: The paper should address this.\n\nWe will add an explicit limitations section to address this.\n"}, {"review_id": "90ge86zy0d", "reviewer": "Reviewer_e3od", "summary": "This paper analyzes neural image compression models relative to classical (hand-engineered) methods using a new dataset and new tools based on spectral analysis. Specifically the paper presents (from Section 1, \"Main Contributions\"):\n(1) a new dataset covering different kinds of distortions used to compare methods\n(2) an inspection tool for distortions that shows where, in the frequency domain, different distortions primarily introduce error\n(3) an inspection tool based on spectral analysis that provides a visualization showing which frequency ranges are most distorted by different compression methods\n(4) some theoretical results about neural compression, albeit limited to the linear autoencoder case\n\nThe main findings are also summarized in the intro:\n(1) Neural image compression (NIC) produces different spectral artifacts than classical codecs (JPEG 2000 was used as the classical codec and a scale-hyperprior model was used for NIC)\n(2) NIC and classical codecs prioritize different frequencies as the bit rate changes\n(3) NIC and classical codecs fail to generalize in similar ways but succeed in different ways\n(4) Due to the differences, it can be hard to choose the best codec for a particular application without knowing what kind of spectral distortions will be present at inference time (i.e., out of distribution)", "questions": "No questions beyond the implied ones in the \"weaknesses\" section.", "limitations": "adequately covered", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Although compression systems, neural and hand-engineered, are frequently assessed in terms of visual quality, this paper is the first I've seen that tries to understand how distortions differ between neural and standard methods. The authors develop a new dataset built by applying 15 different kinds of distortions to images from standard evaluation sets (kodak and clic). They then visualize the power spectral density (PSD) of the distorted images (to understand which frequency bands are affected by each type of distortion) and they visualize the distortions caused by lossy compression for a neural method and for JPEG 2000.\n\nThis analysis leads to interesting observations about how the two codecs behave differently, e.g. neural image codecs (NICs) distort high frequencies more than low frequencies (Section 4.1), and they are more sensitive to high frequency noise/distortions. I haven't previously this kind of analysis or conclusion before.\n\nSimilarly, the authors analyze how classification accuracy degrades when the two types of codecs are used on images with different kinds of distortion (Fig. 5). These findings are particularly interesting because neither codec is preferable for all distortions and bit rates, e.g., JPEG2000 takes a large hit for \"high\" distortions at high bit rates but takes a smaller hit than NICs for \"low\" distortions at moderate bit rates.", "weaknesses": "There are two main weaknesses with the paper. The first is whether the results from studying these two codecs generalize to other codecs in the respective classes. For example, will VCC (a modern codec still be finalized) behave similarly to JPEG2000? Will SOTA neural methods behave similarly to [9]. The latter question is further complicated because there are multiple ways the NICs differ, e.g. (1) more sophisticated models that build on [9] (like adding autoregressive priors, architectures built using transformers instead of CNNs, etc.), (2) using \"implicit neural representations\" instead of the \"nonlinear transform coding\" approach of [9], and (3) optimizing networks with an adversarial loss that targets \"realism\" (minimizing a divergence between distributions, not just a measure of distortion between the original and reconstructed pixel values). It's hard to say if the findings in this paper will be the same for a model that is different according to one of (or all of) these dimensions.\n\nSecond, I'm not sure how important these findings are to the NeurIPS community. Neural image compression is certainly an appropriate topic for NeurIPS, but papers typically focus on new models or new applications. So the target audience may be fairly small, and the observations don't directly imply an architecture change (or loss / optimization change) to improve NICs.\n\nSince I've worked in this subfield, I find the results of this paper fascinating. And at a conference focused on compression or image processing (something like ICIP or DCC), I'd rate this paper as a 7+. I'm open to feedback from the AC or other reviewers that I'm off on my assessment of the fit at NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our submission. We are grateful to hear an expert in this subfield found our work to be fascinating while being mainly concerned about the fit of our work at NeurIPS. We appreciate you stating clear reasoning behind your rating as it gives us an opportunity to clarify any confusion. In addition to our inline responses to your comments below, please note that our global response highlights additional experiments **E1** and **E2** and their insights **I1** - **I4** as well as clarifies the significance of our work in **C1** - **C3**. Namely, existing tools are incapable of highlighting inter- and intra-class similarities and differences for classical codecs and NICs. Our benchmark and introspection tools effectively uncover similarities and differences, in turn, providing a deeper understanding of their performance. Our tools will be available to researchers/practitioners to explore and improve NIC methods impacting future NIC research and development. We hope that you will update your score to reflect your enthusiasm for our work and champion our paper for acceptance.\n \n> Two main weaknesses … similarly to JPEG2000?\n \nRespectfully, this question highlights the utility of our introspection tools and is not a weakness of our paper. Specifically, our evaluation tools highlight both inter- and intra-class similarities and dissimilarities that might not be possible to uncover using existing evaluation datasets and scalar metrics. Importantly, our tools show that distortion from VTM (equivalently VVC; instead of VCC which we took to be a typo) is distributed more evenly across the frequency spectrum, unlike JPEG (added in **E1**) and JPEG2000, as illustrated in **Figures 16 & 17** (Appendix J) and **Figure P1**. Namely, our tools can be utilized to analyze and compare any combination of codecs/NICs, either presently available or developed in the future and can be used to guide the design of future compression schemes. \n \n> Will SOTA neural methods behave similarly to [9]? The latter … these dimensions.\n\nWe agree that these are interesting questions and again believe these highlight the utility of our work. Our tools allow researchers to identify more nuanced differences between NIC variants, even if they have similar performance in terms of RD curves, which are the main existing metrics.\n\nMore specifically, we do include some additional comparisons of NIC variants. On lines 141-143 of the main paper we note that comparisons with NIC optimized for MS-SSIM are available in Appendix G. Additionally, during the rebuttal period we performed empirical analysis for an additional SOTA NIC method, ELIC [He22], trained on CLIC.\n \n> I'm not sure how important findings are to NeurIPS community and observations don't directly imply NIC changes\n\nWe appreciate your perspective and agree that significant NIC contributions in the past have advanced NIC architectures/loss/etc. First, note that we have clarified the significance of our work in section (**GR2**) of the global response. Our work not only highlights future algorithmic directions for NIC but our benchmark and tools provide the means to perform meaningful evaluation that can guide new research. \n\nAdditionally, our choice of NeurIPS as a venue was based on the following bullet points from the official call for papers: \n\n* Applications (e.g., vision, language, speech, audio) \n* Evaluation (e.g., methodology, meta studies, replicability, validity) \n* Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)\n* Social/economic aspects of ML (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior). \n\nWe submitted under the primary area of \"Evaluation\" as our paper a) points out major discrepancies between models with the same PSNR/bpp and b) proposes new metrics/datasets to give researchers a more comprehensive understanding of these differences.\n \n> Target audience may be fairly small\n\nOur work is relevant to both NIC researchers and the broader ML safety and robustness research community. When combined, these two research groups are a significant audience. Further, it has the potential to create a new audience at the intersection of NIC and out-of-distribution generalization/robustness.\n\n> Authors present some theoretical results about neural compression limited to linear autoencoders.\n \nA result applying to nonlinear models is provided in Appendix N (noted on line 303 of main body). We included the linear autoencoder case in the main body since it is less technically involved, and the qualitative takeaways are similar to those in Appendix N.\n"}, {"review_id": "KsO6vmSU0T", "reviewer": "Reviewer_h4BN", "summary": "This paper introduces a benchmark for evaluating the out-of-distribution (OOD) performance of neural image compression (NIC) models. The authors observe distinguished spectral biases of NIC models by comparing them with JPEG2000. Then the authors listed all the spectral biases along with a theoretical analysis from the perspective of principle component decomposition.", "questions": "1. Could the authors offer a comparison with JPEG compressed images? I believe it would be more straightforward.\n\n2. What is the reason for theoretical analysis (given in comment #3 in the section of Weakness) and the given classification (Figure 1) on Gaussian noise and JPEG compression artifacts?\n\n3. What is the difference between the theories for nonlinear transforms (Section 5) and those for traditional linear transform (such as DWT, DCT, and KLT)? \n", "limitations": "Limitations are not discussed in the paper.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes an interesting perspective for analyzing NIC models.\n\n2. The observed spectral biases are potential to further improve NIC models.\n\n3. The authors provide thorough analysis and enough details for reproducing the results.\n", "weaknesses": "I believe the spectral perspective is important to understand NIC models. However,  the theoretical analysis is of limited novelty and the analysis and conclusions provided in this paper are empirical and need to be further improved before publication. Moreover, I am afraid that classifications on several corruptions may conflict with the common sense in signal processing field. My concerns are listed as below.\n\n1. The proposed method mainly compares with JPEG2000, which may be inappropriate. JPEG2000 is based on discrete wavelet transform. However, EBCOT and rate control of JPEG2000 could hinder an explicit understanding of the spectral characterization of JPEG2000. I suggest the authors to conduct the comparison experiments on JPEG (which is based on block-based DCT and simple quantization table) for clear comparison. Compared to JPEG2000, JPEG has been widely applied in practice.\n\n2. The comparison presented in Figures 16 and 17 (i.e., comparison to VTM provided in supplementary material) is confusing. $\\mathcal{D}$ for VTM seems to be zero but it is a lossy compression method and the reconstruction is still far from lossless (i.e., 37.42 dB).\n\n3. I also have problem with the evaluation using power spectral density (SPD). I noticed that the authors classify Gaussian noise as a high-frequency corruption in Figure 1. However, it is well known that the Gaussian noise, i.e., white noise, has a constant SPD. In other words, the effect of Gaussian noise is expected to be constant rather than constrained on the high-frequency components. Moreover, the effect of JPEG compression is also expected to be on both median and high-frequency, as the default quantization map results in a larger quantization step on the high-frequency components. I am afraid that the classification of other corruptions may neither be sufficiently precise. I guess the gaps between above theoretical analysis and the empirical analysis in Figure 1 originate from the intensity difference on the SPD used to analyze the effect of introduced corruption. The natural images themselves are with an unbalanced SPD. Thus, the observation may be biased.\n\n4. The theoretical analysis in Section 5 is not novel enough. The desired characteristics for transform coding are discussed in [R1]. The differences and similarity between DFT, DCT, and KLT are analyzed in [R2]. The provided analysis on linear transforms (autoencoder) may not be appropriate for nonlinear models [R3].\n\n5. Considering that the major contribution of this paper lies on the empirical evaluations, it is important to make thorough validations on diverse datasets and NIC models. The adopted scale hyperprior is not sufficient supporting the claims in this paper.\n\n[R1] V. K. Goyal, \"Theoretical foundations of transform coding.\" IEEE Signal Processing Magazine, vol. 18, no. 5, pp. 9-21, 2001.\n\n[R2] D. S. Taubman and M. W. Marcellin, JPEG2000: Image compression fundamentals, standards and practice. 2002.\n\n[R3] J. Ballé, et al. \"Nonlinear transform coding.\" IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 339-353, 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our submission. We believe that there is some misunderstanding in the goal and contribution of our work which we first clarify. Namely, existing tools are incapable of highlighting inter- and intra-class similarities and differences for classical codecs and NICs beyond RD curves. Our benchmark and introspection tools effectively uncover similarities and differences, in turn, providing a deeper understanding of their performance. Note that we found your feedback to be valuable to improve the impact and reception of our paper and, hence, we accommodated many of your suggestions. Specifically, our global response highlights additional requested experiments **E1**-**E2** and their insights **I1**-**I4** as well as clarifies the significance of our work in **C1**-**C3**. We believe these additional experiments enhance the comprehensiveness of the empirical evaluations, and further highlight the usefulness of our tools. After reviewing our individual and global responses, we hope that you will reflect on your review and consider updating your score.\n\nWe now respond to your questions **Q** and weaknesses **W**.\n\n**Q1**\n \nPer your request, we performed an evaluation of JPEG, noted as **E1** in our global response with visualizations in figures **P1, P2, P5** in the rebuttal PDF. See insights **I1-I4** in (**GR1**) of the global response for JPEG takeaways.\n\n**Q2a** (Reason for theoretical analysis?)\n \nThe theoretical results and takeaways (lines 323-340) provide support for the spectral bias of NIC beyond our empirical results with our spectral tools. We are not aware of theoretical results *explicitly purporting such a spectral bias of NICs* (as [R1]-[R3] do not). We welcome the reviewer to provide such references and we will revise our claim.\n\n**Q2b and W3** (Afraid classifications on several corruptions may conflict with signal processing. Reason for classification (Fig 1) on Gaussian noise and JPEG?)\n\nWe believe this a misinterpretation of the classification provided in Table 1, which we clarify here. Firstly, we are using characterizations of corruptions into low, medium, and high from [Li19] (Apdx Table 1). This characterization has been adopted by the neural network robustness community and has shown its practical utility by showing that these corruptions classes behave differently. This is further supported by our empirical results. Next, we clarify the classification.\n\nCategorizing a corruption as “high” is more akin to saying it “contains substantial high-frequency content” (but of course doesn’t exclusively consist of high-frequency content). This can be easily resolved by elaborating on the classification into low, medium, and high frequency groups which we will clarify in our paper and reference where it originated. We would even be willing to change the name of these categories if the reviewer suggests so, as doing so does not affect our theoretical and empirical claims. \n\n**Q3 and W4a** (Difference/similarity between DFT, DCT, KLT analyzed in [R2])\n \nAt the risk of oversimplifying, DFT, DCT and KLT were hand crafted by experts based on understanding of image statistics. Our linear autoencoder example suggests that NIC models may *learn* a somewhat similar compression technique **directly from training data**. On the other hand, the more general analysis of possibly nonlinear models in Appendix N suggests among other things that (not surprisingly) MSE-optimized autoencoders suffer high error in regions of low training data density. For more fine-grained details we refer to that section.\n\n**W1**\n\nWe note that our empirical results are already highlighting the differences and value (so not hindering an explicit understanding of the spectral characterization of JPEG2000). While EBCOT and rate control of JPEG2000 might hinder a theoretical understanding of its spectral properties, we compute PSDs empirically – could you elaborate on how they could obstruct an empirical spectral characterization of JPEG2000 and make our claims invalid?\n\n**W2** \n \nWe agree that VTM is lossy compression and Figs 16/17 reflect this. To clarify, note that the PSDs in Fig 16 utilize the same heatmap range for a fair comparison across methods while the PSDs in Fig 17 have the default heatmap range, [0,max PSD value]. Importantly, our tools show that distortion from VTM is distributed more evenly across the frequency spectrum, unlike JPEG (added in **E1**) and JPEG2000, as illustrated in Figs 16 & 17 (Appendix J) and **Fig P1**.\n\n**W3**\n \nYou are of course correct that Gaussian noise is evenly distributed in frequency space (as supported by Fig 20 in Appendix K, and the new **Fig P4**). Similarly, your statement about JPEG is correct and corroborated by our JPEG PSD in the rebuttal PDF, **Fig P4**. This confusion has arisen from differing interpretations of the corruption categories from the robustness community [Li19]. Further, this does not affect any of our empirical claims in the paper.\n\n**W4b** (Analysis on linear…) \n\nWe provide a separate theoretical analysis that applies to nonlinear models in Appendix N (as noted on line 303 of paper).\n\n**W5**\n \nRespectfully, it is our position that this view overlooks the major meaningful contribution provided by our paper: Our benchmark and introspection tools – please see section (**GR2**) of the global response. However, we acknowledge that the utility of our benchmark and tools can be better highlighted through the inclusion of additional NIC models. To this end, we prioritized training and evaluating an additional model on CLIC, ELIC [He22], during the rebuttal period. Please see section (**GR1**) of the global response for details and insights from these experiments **I1**-**I4**. Of note, ELIC demonstrates similar spectral bias to the Balle NIC model included in our original paper. We believe these additional experiments enhance the quality of the empirical evaluations and highlight the usefulness of our tools."}, {"review_id": "jGlNpNL0Rn", "reviewer": "Reviewer_fKAX", "summary": "This paper proposes to adopt spectral tools to analysis OOD issues of NIC. On the proposed corrupted dataset, the paper investigates the error of both conventional and learning-based image compression approaches and provides some interesting conclusions, showing robustness and generalization of NIC and suggesting insights of design a better NIC model in the wild.", "questions": "See above.", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper proposes novel tools and benchmarks to thoroughly study NIC from the OOD perspective. I appreciate the delicate figures which quite help me understand the studies. The analysis and conclusions are reasonable and the findings are inspiring, rehealing missed parts of NIC design and assessment. Overall I believe this paper can technically contribute to the community.", "weaknesses": "1. My major concern is the generalization of the conclusion. All the FR experiments are conducted on the hyperprior model (Ball'e et al., 2018), which is a somewhat outdated baseline. Many more powerful approaches have been proposed in the past years, involving more complex context models (Minnen et al., 2018, [40]), improved mixed quantization estimators ([e1, e2, e3]), and non-convolution network structures ([e2, e4, e5]). Shall we observe some different spectral properties when using those transformer-based models?\nAlso, the wavelet-based JPEG2000 is neither a SOTA handcraft codec nor a worldwide popular choice. Though conventional coders don't play key roles in this paper, I suggest the authors further discuss HEIF, AVIF or VVC(IF), and JPEG. \n2. As discussed by the authors, the spectral preference of NIC may be strongly related to the MSE objective. This can be explained as it corresponds to a Gaussian likelihood, according to [Ball'e et al., 2018]. However, this loss function can be replaced to MS-SSIM, GAN and LPIPS etc. For instance, refer to perceptual optimized NIC approaches [e7]. Can this OOD drawback get suppressed when consider those objectives? More discussions are expected.\n3. Though the paper provides a detailed analysis with proposed tools and benchmarks, It do not provide new approaches to solve the concerned OOD issues. Without more practical suggestions, I cannot strongly appreciate the findings for above-mentioned concern about generalization. \n\nP.S. OOD in NIC is not a brand new topic. The authors may want to discuss more related works eg [e6].\n\n[e1] Channel-wise autoregressive entropy models for learned image compression, Minnen et al.\n[e2] Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding, He et al.\n[e3] Transformer-based transform coding, Zhu et al.\n[e4] The Devil Is in the Details: Window-based Attention for Image Compression, Zou et al.\n[e5] Learned image compression with mixed transformer-cnn architectures, Liu et al. *(concurrent work)*\n[e6] On the Out-of-distribution Generalization of Probabilistic Image Modelling, https://openreview.net/pdf?id=q1yLPNF0UFV\n[e7] High-fidelity generative image compression", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our submission. In addition to our inline responses to your comments below, please note that our global response highlights new experiments **E1** and **E2** and their insights **I1** - **I4** as well as clarifies the significance of our work in **C1** - **C3**. Namely, existing tools are incapable of highlighting inter- and intra-class similarities and differences for classical codecs and NICs. Our benchmark and introspection tools effectively uncover similarities and differences, in turn, providing a deeper understanding of their performance. Our tools will be available to researchers/practitioners to explore and improve NIC methods impacting future NIC research and development. After reflecting on our individual/global response, we hope that you will update your score and champion our paper for acceptance.\n\n\n\n> My major concern is the generalization of the conclusion. All the FR experiments are conducted on the hyperprior model (Ball'e et al., 2018), which is a somewhat outdated baseline. Many more powerful approaches have been proposed in the past years, involving more complex context models (Minnen et al., 2018, [40]), improved mixed quantization estimators ([e1, e2, e3]), and non-convolution network structures ([e2, e4, e5]). Shall we observe some different spectral properties when using those transformer-based models?\n\nThank you for your suggestions. Per your request, we performed additional experiments with ELIC [e2]. Please refer to experiment **E2** and the subsequent insights **I1** - **I4** in the global response for takeaways from our experiments. We also started training LIC_TCM (from [e5]) on the CLIC dataset but it did not finish training in time to include in the rebuttal. We believe that these additional results strengthen our work and further highlight the usefulness of our benchmark and tools. We will also cite these papers in our future work section.\n \n> I suggest the authors further discuss HEIF, AVIF or VVC(IF), and JPEG. \n\nPer your request, we performed an evaluation of JPEG, noted as (**E1**) in our global response with visualizations in figures **P1**, **P2**, and **P5** in the rebuttal PDF. See insights **I1-I4** for takeaways pertaining to JPEG. A comparison to the VVC (equivalently VTM) codec was provided in Appendix J (noted in lines 141-143 of main paper). Importantly, our tools show that compression error from VTM is distributed more evenly across the frequency spectrum, unlike JPEG (added in **E1**) and JPEG2000, as illustrated in **Figures 16 & 17** (Appendix J) and **Figure P1**. We believe that these additional results strengthen our work and further highlight the usefulness of our benchmark and tools.\n \n \n> As discussed by the authors, the spectral preference of NIC may be strongly related to the MSE objective. This can be explained as it corresponds to a Gaussian likelihood, according to [Ball'e et al., 2018]. However, this loss function can be replaced to MS-SSIM, GAN and LPIPS etc. For instance, refer to perceptual optimized NIC approaches [e7]. Can this OOD drawback get suppressed when consider those objectives? More discussions are expected. \n \nNote that we provide a comparison to models optimized for MS-SSIM in Appendix G (as noted on lines 141-143 of the main paper). We agree that further study on the effects of alternative loss functions on robustness of NIC is of great interest. Also note that one cannot answer these excellent questions without our benchmark and tools  – highlighting the value of our work. We encourage the reviewer to recall the main contributions of our work in section (**GR2**) of our global response. We will consider listing robustness analysis of NICs trained with alternative loss functions in our future work section.\n\n \n> Though the paper provides a detailed analysis with proposed tools and benchmarks, It do not provide new approaches to solve the concerned OOD issues. Without more practical suggestions, I cannot strongly appreciate the findings for above-mentioned concern about generalization. \n \nRespectfully, it is our position that this view overlooks the meaningful contributions provided by our paper and grades them based on criteria we did not claim as contributions or efforts. Recall that, through our developed tools and experiments, we have evaluated the out-of-distribution robustness of neural/learned image compression providing a more sober assessment of its utility in real world scenarios. We agree with your sentiment that our findings pose new challenges for the NIC community; however, this is a benefit to the community as it should stimulate new and future research directions for NIC at the intersection of out-of-distribution generalization/robustness. Furthermore, our analysis and findings provide clear actionable guidelines. For starters, we have mentioned several approaches in the future work section of our paper. Additionally, tools from the OOD robustness community can be used (e.g., spectral data augmentation-based training, test-time adaptation of architectures for countering the shifts, etc.).\n \n \n> P.S. OOD in NIC is not a brand new topic. The authors may want to discuss more related works eg [e6].\n \nRespectfully, our work and the referenced work [e6] have significant differences. From our understanding, [e6] notices some similarities between *lossless* image compression and OOD *detection* and proposes a model which can work for both tasks. Our work differs in that we are studying *lossy* compression and the *robustness of lossy compression models to OOD images*, and we provide novel tools to make this study possible. The main similarity between our paper and [e6] are the keywords.\n"}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We are grateful that reviewers found our contributions for benchmarking neural image compression models to be “comprehensive” (5yiv), “interesting” (e3od, h4BN), “important” (h4BN), “fascinating” (e3od), “crucial to bridge the real-world adoption of neural-based image compression techniques” (5yiv), and providing “potential to further improve NIC models” (h4BN). Several reviewers found our empirical evaluation and subsequent findings to be “extensive” (5yiv), “inspiring” (fKAX), and providing an “understanding of out-of-distribution performance” (5yiv). Furthermore, reviewers indicated our contributions to NIC were “the first [they have] seen that [try] to understand how distortions differ between neural and standard methods” (e3od) and that it serves to “[reveal] missed parts of NIC design and assessment” (fKAX).
 
(**GR1**) While some reviewers found our empirical (5yiv, fKAX) and theoretical (5yiv) analysis a strength, others requested additional results. Some of these requested results, such as comparison to an additional codec (VVC, equivalently VTM), NIC optimized for MS-SSIM, and theoretical results for nonlinear autoencoders were provided in the appendix (Appendix J, Appendix G, Appendix N) and noted in the main paper (lines 141-143 and line 303). Furthermore, during the rebuttal period we obtained new results for the remaining experiments which are included in the rebuttal PDF:

* **E1** Evaluation of JPEG codec (fKAX, h4BN) in figures **P1**, **P2**, and **P5**.
* **E2** Evaluation using additional NIC architecture (fKAX, h4BN, e3od) [ELIC](https://github.com/VincentChandelier/ELiC-ReImplemetation) [He22] in figures **P1**, **P2**, **P3**, and **P5**. Note that we trained ELIC on the CLIC dataset during the rebuttal period (as opposed to using the publicly-available ImageNet-trained weights).

Notably, **E1** and **E2** align with our original insights offered by our benchmark and tools and serve to strengthen our claims (h4BN, fKAX), namely that:

* **I1** Compression methods yielding the same PSNR (or bpp) can produce very different spectral artifacts (**Figure P1**)
* **I2** As the compression rate increases, different codecs prioritize different parts of the spectrum (**Figures P1 and P3**)
* **I3** Image compression models generalize to low- and mid-frequency shifts better than high- frequency shifts (**Figures P2 and P5**)
* **I4** NIC models are better at denoising high-frequency corruptions than classic codecs (**Figures P2 and P5**)



(**GR2**) In conclusion, we would also like to emphasize that our work: 

* **C1** Highlights limitations of existing evaluation benchmarks and tools for image compression
* **C2** Provides new benchmarks and tools to gain a deeper understanding of compression performance
* **C3** Demonstrates the utility of our datasets and tools by helping uncover both inter- and intra-class (classes = {classical codecs, NIC models}) similarities and differences among existing methods

Comprehensive evaluation schemes are critical to make sustainable progress in any research area, and it seems to be generally acknowledged that test sets measuring robustness to distribution shift have been crucial to the image classification and language modeling research areas (e.g., ImageNet-{C,R,A,v2}, HANS). Our work brings these methods to bear on image compression while providing new insights on the spectral biases of existing codecs and NICs. We are hopeful that reviewers will find our responses clarifying and that, upon reflection of our work’s meaningful contributions to ongoing and future NIC research directions, they will champion our work for acceptance.
 


**References**
* [He22] He et. al., *ELIC: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding*, CVPR 2022.
* [Li19] Li et. al., *Robust deep learning object recognition models rely on low frequency information in natural images*, PLOS Computational Biology 19(3).
* [Vardi23] Gal Vardi, *On the Implicit Bias in Deep-Learning Algorithms*, Communications of the ACM, June 2023.


Meta-review: Initial reviews were mixed with a wide spread in rating, author rebuttal helped address majority of the concerns. Although, one reviewer remains unconvinced, others have updated their ratings and are leaning positive to various extents. After reading the author rebuttal and reviewer discussion, I think that majority of the show stopping concerns were addressed and therefore recommend acceptance. I suggest that authors incorporate all the feedback to further improve their manuscript.
Decision: Accept (poster)