SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, [38] addresses the issue of "multi-generational loss", or the problem of a rapid degradation of quality as images are repeatedly encoded and decoded, with NIC models. Our work differs because we propose spectral tools to analyze the effects of one cycle of encoding and decoding and we study how this affects the performance of different downstream tasks. While multi-generation
loss is an important area of study for tasks like image editing, we focus on analyzing the spectral distortions caused by one cycle of image compression and how these affect a model's generalization and robustness to OOD data. Furthermore, [38] proposes using JPEG2000 as a method to improve the robustness of classification tasks by compressing adversarial noise. We explore how NIC models compare to JPEG2000 in terms of their robustness to several common types of image corruptions.
A.4 Lightweight NIC via model pruning
In an effort to reduce the energy and compute cost required by deep neural networks (DNNs), much work has been devoted towards finding methods to approximate DNNs. Common approaches include pruning network weights or quantizing network weights and activations [19, 20, 48, 69, 40, 15]. Remarkably, large sparse models have been shown to outperform small dense models with the same number of weights [70]. Additionally, approximating DNNs can have benefits in terms of robustness on OOD data [16].
Nonetheless, limited efforts to prune or quantize NIC models have been explored. Recently, Gille et al. [21] successfully introduced structured sparsity into NIC models to simultaneously reduce storage and power costs while maintaining a similar rate-distortion curve when compared to a dense version of the same NIC architecture. Their findings produced  $\sim 80\%$  reduction in the number of parameters and  $\sim 30\%$  reduction in MAC operations required by the NIC encoder. Yin et al. [68] also explored the use of structured sparsity in NIC and their proposed ABCM method was demonstrated to provide up to  $7\times$  and  $3\times$  reduction in DNN parameters and inference time, respectively, with only a mild reduction to reconstruction quality. Luo et al. [43] proposed a technique for pruning NIC models that was demonstrated to maintain the same bpp as dense models but provided a parameter reduction of at most  $\sim 34\%$ . Sun et al. [55] demonstrated a technique for quantizing both NIC weights and activations to an 8-bit fixed point representation while yielding a rate-distortion curve comparable to the original 32-bit float NIC model. Outside of DNN pruning or quantization, alternative techniques have been explored for more efficient NIC [31, 67, 23].
A.5 Metrics for quantifying image compression quality
The objective of lossy image compression is to achieve a balance between minimizing 1) the size of the compressed file and 2) the distortion of the reconstructed image compared to the original image. As these objectives are inherently at odds, the performance of lossy image compression is typically measured using a rate-distortion curve, which measures the size of the compressed image (rate) versus the quality of the reconstructed image (distortion). The rate is usually measured in bits per pixel (bpp), which is the average number of bits required to store each pixel in the compressed representation. The distortion is usually measured by scalar metrics such as peak signal-to-noise ratio (PSNR) or a visual similarity metric such as MS-SSIM [62].
A.6 Benchmark datasets
We utilize two benchmark datasets for evaluating performance of our NIC models. For training and testing, we make use of CLIC (Challenge on Learned Image Compression) 2020 [59]. This collection is comprised of 1633 training, 102 validation, and 428 test images. The images are of varying resolution and are further categorized as either professional or mobile. To evaluate generalization, we make use of the Kodak [34] dataset which consists of 24 images of resolution  $768 \times 512$  (or  $512 \times 768$ ).
B Additional results: Fourier heatmaps
While our spectral tools allow us to measure different capabilities of compression models, they require the availability of OOD, or corrupted, data. To support a setting in which such OOD data is unavailable, we propose adopting another tool: the Fourier sensitivity heatmap. This tool evaluates the PSNR of a compression model on data perturbed with Fourier basis elements [66]. The resulting visualization is a heatmap where the value at coordinate  $(i,j)$  is the PSNR of the compression method on perturbed data  $\{X_{k} + r_{k}\varepsilon U_{i,j}\}_{k = 1}^{N}$ , where each  $r_k$  is selected uniformly at random from  $\{-1,1\}$ ,  $\varepsilon$  is the norm of the perturbation, and  $U_{i,j}$  is the  $(i,j)$ th Fourier basis matrix. From [66],
Figure 6: Fourier heatmaps of methods under the fixed-bpp setting (top) and fixed-PSNR setting (bottom). Left: PSNRs of  $\mathcal{C}(c(\mathcal{X}))$  with respect to  $c(\mathcal{X})$ . Right: PSNR of  $\mathcal{C}(c(\mathcal{X}))$  with respect to  $\mathcal{X}$ . Warmer colors (red) indicate higher PSNR.
$U_{i,j} \in \mathbb{R}^{n_1 \times n_2}$  and satisfies (i)  $\| U_{i,j} \| = 1$  and (ii)  $\mathcal{F}(U_{i,j})$  has at most two non-zero coordinates specifically at  $(i,j)$  and the coordinate symmetric to  $(i,j)$  about the matrix center.
We consider two versions of Fourier heatmaps: (a) with respect to perturbed data and (b) with respect to original data. To analyze (a) and (b), the PSNR at each coordinate  $(i,j)$  is computed with respect to the perturbed dataset,  $\{X_{k} + r_{k}\varepsilon U_{i,j}\}_{k = 1}^{N}$ , and the unperturbed dataset,  $\{X_{k}\}_{k = 1}^{N}$ , respectively. (a) and (b) can be seen as measures of generalization and robustness respectively.
We computed Fourier heatmaps over the CLIC dataset in Figure 6. We use models from Appendix C, where FR NIC is equivalent to SH NIC in the main body and selected hyper-parameters for these models using a fixed-bpp/PSNR constraint on the clean data. Note: in these Fourier heatmap plots, warmer colors represent higher PSNR, which is in contradiction to the plots of  $\mathcal{D}$ ,  $\mathcal{G}$ , and  $\mathcal{R}$  where warmer colors represented more error.
On the left side of Figure 6, we analyze the generalization of the image compression models to various frequency shifts. We find that all image compression models generalize to low and medium-frequency signals better than high-frequency signals. Specifically, observe how the centers of the plots—corresponding to low-frequency signals—have hotter colors than the edges of the plots. This corroborates our findings from both the RD curves and  $\mathcal{G}$  metrics on the CLIC-C dataset in Section 4.2 and Appendix H. For FR NIC opt. MS-SSIM, this difference is only noticeable at the lowest bpp; however for the other models, there is a clear difference in performance for low and high-frequency shifts at all bpps. Additionally, the effect of pruning is a degradation in PSNR at all frequencies.
On the right side of Figure 6, we analyze the robustness, or denoising, capabilities of the models. These plots show that NIC models are better at denoising high-frequency corruptions than low-frequency corruptions (notice the cooler-colored diamonds in the center of the NIC plots). Meanwhile JPEG2000 has very consistent and poor denoising properties across the entire range of corruption frequencies (the plots are blue with a narrow gradient of color compared to the NIC plots). According to this metric, NIC opt. MS-SSSIM also has poor denoising capabilities (the plots are all dark blue compared to the NIC plots) and pruning NICs also results in a degradation of denoising performance. At first, this may seem like a contradiction to our downstream classification results from Appendix C, which showed that NIC opt. MS-SSIM and VR NIC pr=0.8/0.95 were more effective at denoising high-frequency corruptions than FR NIC and VR NIC. However, recall that each pixel in these plots
only shows the average PSNR of the entire perturbed dataset as opposed to the spectral error by frequency, like our  $\mathcal{D}$ ,  $\mathcal{G}$ , and  $\mathcal{R}$  plots show. Then from the main body and Figure 6, it is clear that this PSNR metric is not detailed enough highlight the nuances between the spectral distortion patterns of different methods.
C Additional results: NIC variants
In this section, we analyze several additional variants of NIC models. Many of these variants were chosen because they have potential to reduce the amount of memory required to store or run these models, making them more amenable to "in-the-wild" use cases (e.g., the Mars Exploration Rover discussed in the introduction). We also consider models optimized for MS-SSIM because we are interested in how the distortion objective affects the spectral distortion artifacts of the model.
Because all of these models are based on the scale-hyperprior architecture from [9], we use a different naming scheme here than in the main body. Below are descriptions of the NIC models compared in this section:
- FR NIC: Equivalent to SH NIC in the main body. Eight fixed-rate models each trained on a unique  $\lambda$ . Distortion is MSE (i.e., model is optimized for PSNR).
- VR NIC: One variable-rate model trained over a continuous range of  $\lambda$  values using loss conditional training. Distortion is MSE. More details about this model are provided in Appendix J.
- VR NIC pr=0.8/0.95: One variable-rate model trained over a continuous range of  $\lambda$  values. Distortion is MSE. Here, gradual magnitude pruning (GMP) is applied during training to prune  $80\%$  or  $95\%$  of the convolutional and FiLM weights. More details of the GMP algorithm are provided in Appendix K.
- FR NIC opt. MS-SSIM: Eight fixed-rate models, each trained on a unique  $\lambda$  value. Distortion is (1 - MS-SSIM) (i.e., model is optimized for MS-SSIM).
Figures 7, 9, and 8 show results for these models. We discuss our findings below.
C.1 Evaluating spectral distortion on IND data
Effect of approximating FR NIC with VR NIC. According to the RD curves in Figure 7, VR NIC obtains the same performance as FR NIC for low and moderate bpps; however, the SH NIC model outperforms the VR NIC model at higher bpps despite being trained on the same range of  $\lambda$ . This result follows [17] and suggests that the VR NIC may not be expressive enough to learn the high PSNR regime. In terms of the spectral error  $\mathcal{D}$ , VR NIC and FR NIC have the same error patterns, but VR NIC does leave a slightly higher magnitude of error in some settings (e.g., med. bpp, high bpp, and high psnr).
Effect of pruning. VR NIC, VR NIC  $\mathrm{pr} = 0.8$ , and VR NIC  $= 0.95$  all have the same VR NIC model architecture, but have  $0\%$ ,  $80\%$ , and  $95\%$  sparsity respectively. The RD curves show that pruning causes a degradation in performance, especially for high bpp (see blue lines in Figure 7). However, the PSDs reveal that, even if we make up for the PSNR performance gap by increasing the bpp (right side of Figure 7), the pruned models left a different spectral artifacts from the dense ones. In particular, as the prune rate increases, the ring in the high-frequency region of the PSDs becomes thicker. Presumably reconstructing signals at high-frequencies requires more parameters than reconstructing low frequency signals and so the more sparse models focus on optimizing the "easier" frequencies on the spectrum. These results show that pruning NICs exacerbates the spectral differences between NIC and JPEG2000 that we observed in Figure 2.
Effect of optimization variable in NIC. The FR NIC and FR NIC opt. MS-SSIM have the same FR NIC model architecture, but optimize for different "distortion" metrics (i.e., FR NIC optimizes for PSNR while FR NIC opt. MS-SSIM optimizes for MS-SSIM). Unsurprisingly, Figure 7 shows that the FR NIC opt. MS-SSIM model has the worst performance of all the models in terms of RD curves where the distortion metric is PSNR. However, the plots of the distortion error  $\mathcal{D}$  give us more insights into the spectral errors that this model is making. Surprisingly, FR NIC opt. MS-SSIM produces spectral artifacts which differ greatly from both the PSNR optimized NIC models and the classic JPEG2000 codec. In particular, like JPEG2000, this model distorts the medium and low frequencies more than high-frequencies (notice the cooler-colored ring around the hot-colored center).
Figure 7: Visualizing distortion via CLIC test set evaluation with NIC variants. Left: spectral measure of in-distribution reconstruction error  $\mathcal{D}$  under the fixed-bpp constraint at three rates. Center: Rate-distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating fixed-PSNR values. Right:  $\mathcal{D}$  under fixed-PSNR constraint. Each  $\mathcal{D}$  plot is labeled with a tuple of that model's (bpp, PSNR) on CLIC. Hotter colors (red) indicate more error in that frequency range.
Figure 8: Spectral measures  $\mathcal{G}$  and  $\mathcal{R}$  for all NIC variants on two example corruptions of ImageNet-C. Model hyper-parameters were chosen such that they give bpp≈1.23 on ImageNet. Each plot is labeled with the model's bpp and PSNR with respect to  $c(\mathcal{X})$  (top row) or  $\mathcal{X}$  (bottom row).
However, it differs from JPEG2000 in that it leaves a less rectangular artifact. Thus, the optimization objective of NIC can influence the spectral artifacts and should be thoroughly examined before being used in practice.
C.2 Robustness and generalization
We analyze the findings further using our robustness spectral inspection tools on ImageNet-C in Figure 8. The plots of  $\mathcal{G}$  on the snow corruption show that although the models have different magnitudes of errors, all models leave similar magnitudes of errors across the frequency spectrum (i.e., the gradient of each plot is narrow). Furthermore, all models have similar plots of  $\mathcal{R}$ . This follows the result of the next section (accuracy on the downstream task), which shows that all six models have comparable accuracies at this compression rate (see snow column of  $\mathrm{bpp} \approx 1.23$  plot in Figure 9).
On the shot noise corruption, however, we see some notable differences between models on both  $\mathcal{G}$  and  $\mathcal{R}$ . On  $\mathcal{G}$ , we see that as the prune rate increases, the ring of high-frequency spectral error becomes significantly more bold. This reiterates our finding from Figure 7 that pruning results in a degradation of high-frequency signal reconstruction. Similarly, FR NIC opt. MS-SSIM has a thicker
Figure 9: Effect of compressing corrupt images on classification accuracy. Each subfigure shows results under a different fixed-bpp or fixed-PSNR constraint based on the bpp/PSNRs achieved by the compressors on the clean ImageNet dataset. Specifically, let  $A(X)$  be the top-1 accuracy of the model on dataset  $X$ , measured in percentage points. We report  $A(\mathcal{C}(c(X))) - A(c(X))$  over all -C corruptions (or on clean ImageNet in the case of "clean").
ring than FR NIC in the high-frequency domain. Interestingly, this model also has a smaller error in the low frequency domain compared to the NIC models optimized for PSNR. Finally, JPEG2000 distorts low- and medium-frequencies relatively evenly and distorts high-frequency signals less than NIC which follows from Section 4.1. We can see the effect of these differences in  $\mathcal{G}$  in the high-frequency domain of the plots  $\mathcal{R}$ . Specifically, the VR NIC leaves a slightly smaller artifact than FR NIC (i.e., the circle of warm colors is larger with VR NIC). This differences is exacerbated as the prune rate increases (i.e., NIC pr=0.95 has the smallest artifact of the four first four models), which suggests that the pruned NIC models are most effective for denoising high-frequency corruptions. FR NIC opt. MS-SSIM also leaves a smaller artifact than FR NIC; however, this has a different shape than the NIC models optimized for PSNR (specifically, there are jagged edges on the edges of this artifact). This suggests that FR NIC opt. MS-SSIM denoises high-frequency data better than FR NIC, but in a different way from VR NIC pr=0.8/0.95. Finally, JPEG2000 has diamond-shaped artifact which suggests it denoises some high-frequency signals, but not all.
C.3 Downstream task
We also consider how these NIC variants perform on the same downstream classification task outlined in Section 4.3. Figure 9 shows the difference in accuracy before and after corruption for each model on each corruption (or clean data) and Appendix M has tables with these accuracies. We find that the pruned models are better at denoising high-frequency corruptions than the FR NIC. In particular, we find that on the high-frequency corruptions at  $\mathrm{bpp} = 1.23$  and 0.77, the pruned VR NICs show a larger improvement in accuracy than the dense VR and FR NICs. This finding matches our observation in Figure 7 that the pruned NICs denoise high-frequency signals even more than dense NICs, who in turn denoise these signals more than JPEG2000. The pruned models perform comparably to the dense models on low- and medium-frequency corruptions. Similarly, we also find that NIC opt. MS-SSIM also improves performance on the downstream task on high-frequency corruptions over (a)
Figure 10: Visualizing distortion of VTM via Kodak test set evaluation. Left: spectral measure of in-distribution reconstruction error  $\mathcal{D}$  under the fixed-bpp constraint at three rates. Center: Rate-distortion curves with vertical lines indicating fixed-bpp values and horizontal lines indicating fixed-PSNR values. Right:  $\mathcal{D}$  under fixed-PSNR constraint. Each  $\mathcal{D}$  plot is labeled with a tuple of that model's (bpp, PSNR) on Kodak. Hotter colors (red) indicate more error in that frequency range.
no compression and (b) compression with dense NIC models. This model also has better performance than the other NIC variants and low- and medium-frequency corruptions.
D Additional results: comparing JPEG2000 and VTM
In this section, we compare two classic CODECs: JPEG2000 and Versatile Video Coding (VVC) (also known as h.266) codec [30]. We refer to the latter codec as VTM because we used VVC Test Model (VTM) software from [64] to test it. Although VTM has state-of-the-art performance for classic CODEcs, we chose to use JPEG2000 as the representative classic codec in the main body for three reasons.
1. JPEG2000 has more consistent bpp usage. As shown later in this section (Figure 12), VTM varies its bpp usage drastically depending on the corruption type and severity. Specifically, a hyper-parameter which gives a bpp of  $x$  on clean data may give a bpp of over  $7x$  on certain OOD shifts and severities.6 Thus, it was unclear how to fairly compare VTM and NIC in a fixed-bpp setting on OOD data.
2. JPEG2000 implementation has faster inference time. The current open-source implementation for VTM is significantly slower than those available for JPEG2000 (10s to 100s of seconds for VTM vs.  $< 1$  second for JPEG2000). Thus, JPEG2000 was a more feasible codec for our extensive experimentation.
3. After adjusting for PSNR, VTM shows similar spectral results as JPEG2000. Finally, although there are differences in magnitudes of the spectral distortion plots, JPEG2000 and VTM exhibit similar patterns, so we found it sufficient to use only one classical codec representative.
We include results comparing JPEG2000 and VTM on the Kodak and Kodak-C datasets in Figures 10-13 and explain our findings below.
D.1 Evaluating VTM on IND data
Figure 10 is analogous to Figure 2 in the main body. From the RD curves, we see that VTM outperforms FR NIC and JPEG2000 across all tested bpps. VTM's plots of  $\mathcal{D}$  are most similar to JPEG2000's. Specifically, at low compression rates (high bpp/PSNR), both methods leave similar errors across all frequencies (notice the small gradient of colors in both JPEG2000 and VTM's plots of  $\mathcal{D}$  at med/high bpp and med/high PSNR). Similarly, at high compression rates (low bpp/PSNR) both models leave more spectral errors at the low and medium frequencies. Despite having similar patterns to JPEG2000, most of the VTM plots have a smaller magnitude than JPEG2000, even at fixed-PSNRs. This suggests that the metric  $\mathcal{D}$  metric is more closely aligned with VTM's modus operandi than JPEG2000's.
For example, when  $q = 17$ , VTM uses 2.32 bpp on Kodak and 16.49 bpp on Kodak-C shot noise with severity=5.
$\mathcal{D}(\mathcal{C},\mathcal{X})$  : Fixed-bpp
$\mathcal{D}(\mathcal{C},\mathcal{X})$  : Fixed-psnr
Figure 11: Unscaled plots of  $\mathcal{G}$  and  $\mathcal{R}$  with VTM from Figure 10.
Figure 12: Rate-distortion curves for a representative low, medium, and high-frequency shift. Each shift and model has three curves for severity=1 (least transparent), severity=3, and severity=5 (most transparent). Top row: generalization of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $c(\mathcal{X})$  (i.e., PSNR of the reconstructed shifted images w.r.t. the original shifted images). Bottom row: denoising of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $\mathcal{X}$  (i.e., PSNR of the reconstructed shifted images w.r.t. the original clean images).
Figure 11 shows the six VTM plots from Figure 10 without constrained color map scales. These results confirm that VTM leaves only a small magnitude of error, which is even across frequencies.
D.2 Evaluating VTM on OOD data
Figure 12 shows the RD curves of VTM on a low, medium, and high-frequency shift. On the low-frequency shift (snow), VTM has comparable performance to FR NIC and JPEG2000 at low bpp; however, VTM can achieve higher PSNRs by increasing its bpp usage. This finding suggests that VTM, like the other CODECs in Section 4.2.1, generalizes well to low-frequency shifts. However, this highlights the fact that VTM may drastically increase its bpp usage on OOD data. Interestingly, all three models are comparably robust to the snow corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to  $\mathcal{X}$ .
On the medium-frequency shift (glass blur), VTM, like the other codecs in Section 4.2.1, achieves PSNRs which are higher than the PSNRs on clean data. However, VTM is not as effective as FR NIC in terms of PSNR with respect to  $c(\mathcal{X})$  on this shift (the VTM curves start to flatten around 40-42 PSNR, while the NIC curves flatten around 42-47 PSNR). Like the snow corruption, all three models are comparably robust to the glass blur corruption (bottom row of Figure 12) and the severity of the corruption is the largest determinant of the PSNR with respect to  $\mathcal{X}$ .
On the high-frequency shift (shot noise), VTM displays a drastically different pattern from both NIC and JPEG2000. Specifically, VTM is able to achieve significantly higher PSNRs with respect to  $c(\mathcal{X})$  (i.e., generalize better) than FR NIC and JPEG2000. However, this comes at a very serious cost in
Figure 13: Generalization error  $\mathcal{G}$  and denoising error  $\mathcal{R}$  for JPEG2000 and VTM. We plot both spectral metrics for one low, medium, and high-frequency corruption at severities 1 and 5. Each plot is labeled with a tuple of that model's (bpp, PSNR) on the Kodak-C dataset with that corruption.