SlowGuess/NeurIPS_2023_OCR huggingface dataset:
sey Lieberman
Durham, NC USA
kelsey.lieberman@duke.edu
James Diffenderfer
Lawrence Livermore National Laboratory
Livermore, CA USA
diffenderfer2@11nl.gov
Charles Godfrey†
Thomson Reuters Labs
Eagan, MN USA
charles.godfrey@thomsonreuters.com
Bhavya Kailkhura
kailkhura1@llnl.gov
Recent advances in neural image compression (NIC) have produced models that are starting to outperform classic codesets. While this has led to growing excitement about using NIC in real-world applications, the successful adoption of any machine learning system in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand NIC performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we provide CLIC-C and Kodak-C by introducing 15 corruptions to the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspired inspection tools to gain deeper insight into errors introduced by image compression methods as well as their OOD performance. We then carry out a detailed performance comparison of several classic CODEs and NIC variants, revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC and its dependence on the spectral properties of the data. Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing robust and generalizable NIC methods. Code and data will be made available at https://github.com/klieberman/ood_nic.
Consider the Mars Exploration Rover, whose scientific objective is to search for clues to past activity of water (and perhaps life) on Mars. To achieve this, the rover collects images of interesting rocks and soils to be analyzed by the scientists on Earth. Sending these images down the Earth-bound data stream in their original form is too slow and expensive due to limited bandwidth. Thus, it is well accepted that image compression could play a key role in producing scientific breakthroughs [44].
†Work done at Pacific Northwest National Laboratory.
Employing image compression in such a setting is challenging for three main reasons: 1) a high compression ratio is desired due to low communication bandwidth, 2) given the battery-operated nature of these devices, the compression module has to be lightweight so it consumes less memory and power, and 3) robustness and generalization to environmental noises and domain shifts, respectively, are desired due to limited Mars-specific training data. These requirements are not specific to the planetary exploration use case but also arise in a wide range of scientific applications which use image compression in the wild [33].
Recently, neural image compression (NIC) has demonstrated remarkable performance in terms of rate-distortion and runtime overhead on in-distribution (IND) data [9, 46]—satisfying requirements 1) and 2). However, there is limited work on understanding the out-of-distribution (OOD) robustness and generalization performance of image compression methods (requirement 3) [42]. Our work is driven by several open fundamental empirical and theoretical questions around this crucial issue.
How can the expected OOD performance of image compression models be reliably assessed? Can we gain a deeper understanding of the modus operandi of different image compression methods? How do training data properties and biases impact data-driven compression methods?
Main Contributions: This paper takes a critical view of the state of image compression and makes several contributions toward answering the aforementioned questions. ① First, we design comprehensive benchmark datasets for evaluating the OOD performance of image compression methods. Inspired by existing OOD benchmarks for classification and detection [26, 28, 57, 56], we design CLIC-C and Kodak-C by introducing 15 common shifts emulating train-deployment distribution mismatch to the popular CLIC and Kodak datasets. ② Next, we focus on understanding the image compression performance. The de-facto approach is to use rate-distortion (RD) curves measured with perceptual quality metrics, such as PSNR. Such scalar metrics, although easy to compute, are known to be extremely limited in what they can capture and sometimes can even be misleading [63, 61]. To complement RD curves, we propose spectrally-inspired inspection tools that provide a more nuanced picture of (a) compression error, and (b) OOD performance of a given method. Specifically, we introduce a power spectral density (PSD) based approach to understand the reconstruction error. Our approach not only quantifies how much error was made but also highlights precisely where it was made (in the frequency domain). Similarly, to understand the OOD performance of a compression method in unseen deployment scenarios, we propose Fourier error heatmaps—a visualization tool for highlighting the sensitivity of the reconstruction performance of a compression method to different perturbations in the frequency domain. ③ Using our benchmark datasets and inspection tools, we carry out a systematic empirical comparison of classic codes (i.e., JPEG2000, JPEG, and VTM) with various NIC models (fixed rate, variable rate, and pruned versions of the scale hyperprior model [9], as well as efficient learned image compression (ELIC) [24]). ④ Finally, we develop theoretical tools to connect NIC OOD performance with its training data properties.
Main Findings: Our analysis resulted in some invaluable insights about the state of image compression. We summarize some of our findings below.
- Compression methods yielding the same PSNR (or bpp) can produce very different spectral artifacts. Our tools help uncover hidden spectral biases and highlight the limitations of de-facto RD curve-based performance comparison.
- As the compression rate increases, different codec's prioritize different parts of the frequency spectrum. By precisely characterizing this behavior, our tools help in advancing the current understanding of the modus operandi of image compression methods.
- Image compression models generalize to low- and mid-frequency shifts better than high-frequency shifts. This finding calibrates our expectations for image compression performance on OOD data.
-  $\overline{\mathrm{NIC}}$  models are better at denoising high-frequency corruptions than classic codecs. This finding reveals that unlike classic codecs, NIC models have an inherent spectral bias due to the image frequencies present in their training data.
- Identifying the most suitable compression method becomes exceptionally challenging without the knowledge of spectral characteristics of OOD shifts. Our systematic evaluation identifies this open issue with current compression methods and suggests the design of next-generation NIC models that can adapt themselves at runtime based on the spectral nature of the data to be a potentially worthwhile direction to pursue in the future.
(a)
Figure 1: (a) Top row: An original CLIC image and the same image with 3 different corruptions in CLIC-C (severity 5). Bottom left: Average PSD of CLIC dataset,  $\frac{1}{N}\sum_{k=1}^{N}PSD(X_k)$ . Bottom row, other figures: Average PSD of the difference between the corrupted images and the clean images for each given CLIC-C corruption  $c$ ,  $\frac{1}{N}\sum_{k=1}^{N}PSD(c(X_k) - X_k)$ . (b) CLIC-C corruptions categorized as low, medium, or high based on corruption average PSD.
Low	Snow, Frost, Fog, Brightness, Contrast
Med	Motion blur, Zoom blur, Defocus blur, Glass blur, Elastic transform, Pixelate, JPEG compression
High	Gaussian noise, Shot noise, Impulse noise
We corroborate our findings with a detailed theoretical analysis, showing that multiple overarching trends in our experimental results can be attributed to neural compression models' spectral bias.
Appendix A has a detailed related work discussion, while all references are listed in the main paper.
2 Out-of-distribution image compression datasets
To evaluate NIC in the presence of environmental or digital distribution shifts, we generated variants of the CLIC and Kodak datasets, which we refer to as CLIC-C and Kodak-C. Following the techniques presented in [26] for studying the performance of DNN classifiers encountering distributional shifts "in the wild", our -C datasets consist of images augmented by 15 common corruptions. For each image in the original dataset, the -C dataset contains a corrupted version of the image for each of the 15 common corruptions², and for each of five corruption severity levels, with 1 being the lowest severity and 5 being the highest. A sample of some corruptions on CLIC-C is provided in Figure 1a.
While each -C dataset offers a broad sampling of environmental or digital image corruptions, it also provides a spectrally diverse collection of corruptions, in the sense that each corruption can be categorized as low, medium, or high frequency based on the frequency content used for perturbations. We will write  $PSD(\cdot)$  to denote the function that converts the input image from the spatial to the frequency domain by computing the power spectral density of the input. Practically, computing  $PSD(\cdot)$  is done by applying the fast Fourier transform (FFT) [11], followed by a shift operation to center the zero-frequency component, then taking the absolute value. Now suppose we have a set  $\mathcal{X} = \{X_k\}_{k=1}^N$  of uncorrupted images and some corruption function  $c(\cdot)$  (e.g., frost, gaussian noise, etc.). We analyze the spectrum each corruption  $c(\cdot)$  by computing  $\frac{1}{N} \sum_{i=1}^{N} PSD(X_i - c(X_i))$  (see Figure 1a). We classify the corruptions into types of frequencies: low, medium, and high using the characterization from [39], which has been adopted by the neural network robustness community. Note that categorizing a corruption as high is more akin to saying it "contains substantial high-frequency content" rather than saying it "exclusively consists of high-frequency content."
3 Spectral inspection tools
While existing scalar metrics, such as PSNR, are able to summarize the visual similarity of reconstructed images to the original, we will demonstrate that such metrics can provide an incomplete (and sometimes misleading) picture when measuring the impact of compression in OOD settings. Notably, existing tools do not consider the impact of compression on different frequency ranges of images within a dataset. To more thoroughly analyze the effects of image compression, we propose to measure and visualize the effect of image compression in the spectral domain. Given an image compression model  $\mathcal{C}$  that returns reconstructed images, we introduce tools for analyzing compression
2We used github.com/bethgelab/imagecorruptions to apply corruptions to Kodak and CLIC images
error in the Fourier domain to better understand (i) which spectral frequencies are distorted by  $\mathcal{C}$ , (ii) the OOD generalization error, and (iii) the robustness error in the presence of distributional shifts.
Definition 3.1 (Spectral Measure of Distortion Error). To analyze  $(i)$ , we evaluate the image compression model  $\mathcal{C}$ 's ability to reconstruct components of an image across a range of frequencies. To quantify this, we compute the average PSD of the difference between each image  $X_{k}$  in a dataset  $\mathcal{X}$  and the reconstructed version  $\mathcal{C}(X_k)$  of  $X_{k}$ :  $\mathcal{D}(\mathcal{C},\mathcal{X})\coloneqq \frac{1}{N}\sum_{k = 1}^{N}\text{PSD}(X_k - \mathcal{C}(X_k))$
Definition 3.2 (Spectral Measure of OOD Generalization Error). For  $(ii)$ , we evaluate  $\mathcal{C}$ 's ability to faithfully reconstruct OOD images. To quantify this, we extend the metric  $\mathcal{D}(\mathcal{C},\mathcal{X})$  to account for a corrupted version  $c(\mathcal{X})$  of  $\mathcal{X}$  as follows:  $\mathcal{G}(\mathcal{C},\mathcal{X},c)\coloneqq \frac{1}{N}\sum_{k = 1}^{N}PSD(c(X_k) - \mathcal{C}(c(X_k)))$
Definition 3.3 (Spectral Measure of OOD Robustness Error). For  $(iii)$ , we evaluate  $\mathcal{C}$ 's denoising ability. To quantify this, we compute the average PSD of the difference between each uncorrupted image  $X_{k}$  and the reconstructed version  $\mathcal{C}(c(X_k))$  of the corresponding corrupted image  $c(X_k)$ :  $\mathcal{R}(\mathcal{C},\mathcal{X},c)\coloneqq \frac{1}{N}\sum_{k = 1}^{N}PSD(X_k - \mathcal{C}(c(X_k)))$
The  $PSD$  function used in the above formulas converts the input image from the spatial to frequency domain by applying the Fast Fourier Transform (FFT) followed by a shift operation to center the zero-frequency component. The proposed tools are computed for a set of images by averaging the power spectral density (PSD) of the difference between the compressed image and the original for each image in the set of images. For simplicity, when  $(\mathcal{C},\mathcal{X},c)$  is clear from the context, we will just write  $\mathcal{D}$ ,  $\mathcal{G}$ , or  $\mathcal{R}$ . Note that  $\mathcal{G}$  provides insight into the compression model  $\mathcal{C}$ 's ability to generalize to a distribution shift  $c$  while  $\mathcal{R}$  visualizes the denoising effect (or lack thereof) of  $\mathcal{C}$  across the frequency domain.
In Appendix B, we present results using an additional tool, the Fourier heatmap, that utilizes Fourier basis perturbations as corruptions and is used to corroborate our findings for the specific -C datasets and corruptions we consider. This tool can be leveraged when specific OOD data is unavailable.
4 Experiments and findings
Using our spectral inspection tools and OOD datasets, we analyze the performance of several image compression methods and identify several inter- and intra-class differences between classic CODEs and NIC methods. We show results for two classic CODEs, JPEG and JPEG2000, and two NIC models, SH NIC and ELIC, in the main body and report the remaining results in Appendices C and D.
Classic codes. We apply each of the following algorithms over several compression rates  $q$ .
- JPEG [60]
- JPEG2000 [53]
- Versatile Video Coding (VVC) (equivalently h.266) using the VVC Test Model (VTM) software [30, 3]
Neural Image Compressors (NIC) Models. NIC optimization uses a hyperparameter  $\lambda$  to control the relative weight of distortion (quality of reconstruction) and rate (level of compression) terms in the objective function. We train each NIC model over several values of  $\lambda$ . All models were optimized on the train split of the 2020 CLIC dataset [59]. Further details on their model architectures and training can be found in Appendix I.
- SH NIC: Scale-hyperprior model from Balle et al. optimized for PSNR (i.e., distortion objective is mean squared error) [9].
- ELIC: Efficient Learned Image Compression optimized for PSNR [24].
- Variant of SH NIC optimized for MS-SSIM [9].
- Variable-rate version of SH NIC optimized using Loss Conditional Training [17].
- Variable-rate model above with  $80 - 95\%$  of weights pruned using gradual magnitude pruning [70].
Evaluation setup. We compare distortion, robustness, and generalization error of different image compression methods under three constraints: (a) no constraint, (b) fixed-bpp, and (c) fixed-PSNR. In (a), we compare methods over their full range of rate-distortion tradeoffs by generating rate-distortion curves. In (b), we compare models with hyper-parameters which give a very similar bits per pixel (bpp) result on a particular dataset. For example, we find that on the CLIC dataset, SH NIC with
Figure 2: Visualizing distortion via CLIC test set evaluation. Left: spectral measure of in-distribution reconstruction error  $\mathcal{D}$  under the fixed-bpp constraint at three rates. Center: Rate-distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating fixed-PSNR values. Right:  $\mathcal{D}$  under fixed-PSNR constraint. Each  $\mathcal{D}$  plot is labeled with a tuple of that model's (bpp, PSNR) on CLIC. Hotter colors (red) indicate more error in that frequency range.
$\lambda = 0.15$ , ELIC with  $\lambda = 0.15$ , JPEG2000 with  $q = 10$ , and JPEG with  $q = 75$  all give a bpp close to 1.21. Thus, comparing these three models with those hyper-parameters on CLIC under a fixed-bpp constraint, emulates a setting in which a fixed budget is available to store images. Analogously, in (c) we compare models with hyper-parameters yielding a fixed PSNR. This allows the comparison of spectral properties of data distributions that achieve the same quality according to the scalar PSNR metric. Scenarios (b) and (c) are used when evaluating  $\mathcal{D}, \mathcal{G}, \mathcal{R}$ , Fourier heatmaps, and accuracy on a downstream task.
Test data. All models are tested on (a) in-distribution (IND) and (b) corrupted (or OOD) datasets. For (a), we use the 2020 CLIC test split, the full Kodak dataset, and the ImageNet validation split. For (b), we use the corresponding -C datasets for each of the datasets in (a). The main body contains results for the CLIC/CLIC-C dataset. Analogous results for Kodak are in Appendix G.
4.1 Evaluating spectral distortion on IND data
On in-distribution (IND) data, the existing RD curve metrics in the center of Figure 2 highlight the established trend that these NIC models outperform the JPEG2000 and JPEG across the compression rates that the NIC model is trained on  $(\mathrm{bpp} \in [0.1, 1.5])$ , with ELIC outperforming SH NIC and JPEG2000 outperforming JPEG.
Next, we use our spectral inspection tool  $\mathcal{D}$  to better understand the effects of different image compression methods. Specifically, Figure 2 shows plots of  $\mathcal{D}$  under three fixed-bpp and three fixed-PSNR scenarios on the clean CLIC dataset. We highlight some surprising insights below.
Two methods yielding the same PSNR can produce very different spectral artifacts. Under the fixed-PSNR constraint (right side of Figure 2), each column consists of methods with hyperparameters selected to give very similar PSNRs on the CLIC test set (e.g., models on the "high psnr" column all have PSNR  $\approx 36.8$ ). Despite having comparable PSNRs, the plots of  $\mathcal{D}$  vary greatly between the four models. In particular, the SH NIC models distort high frequencies more than medium frequencies (notice the warmer-colored rings around the edges of the  $\mathcal{D}$  plots with cooler-colored centers). JPEG2000, on the other hand, distorts low and medium frequencies more than high frequencies (notice the large rectangles of warmer colors). ELIC distorts all frequencies relatively evenly (the color gradients in the plots are very narrow).<sup>3</sup> JPEG severely distorts one ring, but this ring includes lower frequencies (has a smaller radius) than the rings left by SH NIC. This same pattern holds under the fixed-bpp constraint (left side of Figure 2). These observations demonstrate that PSNR is not a comprehensive metric.
3We provide a more detailed view of the ELIC plots with smaller color bars in Figure 14 of the Appendix.
Figure 3: Rate-distortion curves for a representative low, medium, and high-frequency shift. Each shift and model has three curves for severity=1 (least transparent), severity=3, and severity=5 (most transparent). Top row: generalization of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $c(\mathcal{X})$  (i.e., PSNR of the reconstructed shifted images w.r.t. the original shifted images). Bottom row: denoising of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $\mathcal{X}$  (i.e., PSNR of the reconstructed shifted images w.r.t. the original clean images).
As the compression rate increases, different codecs prioritize different parts of the spectrum. On the left side of Figure 2, each column represents a different "budget" scenario where the three methods have hyper-parameters which result in the models giving very similar bpps on the CLIC test set. Although it was previously known that the quality of the reconstructed images decreases as the bpp decreases, it was not previously known which frequencies NIC models distort to achieve a given bpp. The  $\mathcal{D}$  plots show that JPEG2000 models corrupt low- and mid-frequency regions starting at low compression rates and these regions become more severe as the budget decreases. SH NIC models do almost the opposite—they sacrifice the highest frequencies first and expand this region into lower frequencies at more severe compression rates (i.e., as bpp decreases). JPEG has a mechanism more similar to JPEG2000 in that it corrupts one ring and increases the severity of this ring, while ELIC models corrupt all frequencies evenly and then corrupt low frequencies more as bpp decreases. This suggests that as the compression rate increases, classic codecs corrupt the same frequencies more severely while NIC models change their corruption patterns.
4.2 Evaluating the generalization and the robustness on OOD data
We use the CLIC-C dataset and our spectral tools to study the OOD performance of different image compression methods. We show results for example shifts (one low, medium, and high-frequency representative) and three severities (1, 3, and 5 where 5 is most severe) in Figures 3 and 4 and discuss several interesting findings below.
4.2.1 Rate-distortion curves on OOD data
Image compression models generalize to low- and mid-frequency shifts better than high-frequency shifts. The top row of Figure 3 shows how well different compression models generalize to shifted images in terms of RD curves. In other words, these plots show how well a compressor  $\mathcal{C}$  can reconstruct a given shifted image  $c(\mathcal{X})$  in terms of PSNR of  $\mathcal{C}(c(\mathcal{X}))$  with respect to  $c(\mathcal{X})$ . The three examples of corruption in this figure show vastly different trends. On the low-frequency corruption (snow), all four models can reconstruct  $c(\mathcal{X})$  almost as well as these models can reconstruct clean images: note the PSNR range in the top left plot of Figure 3 is about 22-40 while the PSNR range for the clean data in Figure 2 is about 28-42 over the same bpp range. Interestingly, the three models can reconstruct the images with the glass blur (a medium-frequency shift) better than they can reconstruct clean images (the PSNR of the data shifted with glass blur ranges from about 25-48 with bpp  $< 1$ ). These results suggest that image compression models are fairly effective at generalizing to low-frequency shifts and very effective at generalizing to medium-frequency shifts. However, the
Figure 4: Generalization error  $\mathcal{G}$  (top) and denoising error  $\mathcal{R}$  (bottom). We plot both spectral metrics for one low, medium, and high-frequency corruption at severities 1 and 5. Each plot is labeled with a tuple of that model's (bpp, PSNR) on the CLIC-C dataset with that corruption.
high-frequency shift (shot noise), gives a starkly different result. All four models give very low PSNRs with respect to  $c(\mathcal{X})$ . Even at the lowest severity (severity=1), this PSNR is only in the low 20s. As the severity increases (i.e., as the lines become more transparent), the PSNR decreases even more to the point that, at the highest severity, none of the models can achieve a PSNR higher than 14. Notably, the main factor determining the PSNR is the severity of the corruption and not the compression model or bpp. This suggests that it is significantly harder to generalize to high-frequency data than to low- or mid-frequency data.