SlowGuess/NeurIPS_2023_OCR huggingface dataset:
NIC models are better at denoising high-frequency corruptions than classic codescs. The second row of Figure 3 shows how well different compressors  $\mathcal{C}$  denoise corrupted images in terms of the PSNR of  $\mathcal{C}(c(\mathcal{X}))$  with respect to  $\mathcal{X}$ . These results show that all the models fail at denoising snow (low-frequency) and glass blur (medium-frequency) corruptions (PSNR does not change much with an increase in bpp, except at low bpps on glass blur). However, on shot noise (a high-frequency corruption) both NIC models achieve better PSNRs with respect to  $\mathcal{X}$  than both classic codescs. This trend is consistent for all three severity levels and suggests that NICs may be a more effective method for denoising high-frequency corruptions than the previously-used JPEG and JPEG2000 methods. The implication of this finding extends to the research area of adversarial example denoising [6].
4.2.2 Spectral analysis on OOD data
We now take a deeper look at the Section 4.2.1 findings using our spectral inspection tools.
Spectral artifacts are similar for low-frequency shifts and clean images. The patterns of  $\mathcal{G}$  in Figure 4 measure how well each model generalizes to (or reconstructs) the shifted images. For the low-frequency shift (top row of Figure 4), the plots look strikingly similar to the patterns exhibited by the same models on clean data (Figure 2 top and bottom row): SH NIC models distort high frequencies more than low frequencies while JPEG2000 distorts low and medium frequencies more than high frequencies. Similarly, ELIC distorts all frequencies relatively evenly and JPEG severely distorts one ring of frequencies. This suggests that the methods' modus operandi for generalization to data with low-frequency shifts is similar to their modus operandi on clean data, which makes sense as clean data is dominated by low/mid frequencies. Interestingly, these generalization differences between compressors are not accompanied by differences in robustness metric  $\mathcal{R}$ . All methods show similar patterns in their plots of  $\mathcal{R}$  patterns and these in turn look similar to the snow corruption plot in Figure 1a. This is consistent with our finding from Figure 3 which showed that both NIC and
Figure 5: Effect of compressing corrupt images on classification accuracy. Each bar shows the average difference in accuracy after compressing with different methods  $\mathcal{C}$  for a group of corruptions as classified in Table 1b. Specifically, let  $A(X)$  be the top-1 accuracy of the model on dataset  $X$ , measured in percentage points. Then we report  $A(\mathcal{C}(c(X))) - A(c(X))$  over all -C corruptions in the corruption category with severity 3 (or on clean ImageNet in the case of "clean"). Each subplot shows results under a different fixed-PSNR constraint based on the PSNRs achieved by the compressors on the clean ImageNet dataset. Results for individual corruptions are in Figure 15.
JPEG200 fail to denoise low-frequency corruptions. In other words, Figure 4 shows that NIC and classic CODECs fail in a very similar manner on low-frequency signal denoising tasks.
Both NIC and classic codes can make almost no generalization errors on medium-frequency shifts. The second row of Figure 4 shows that all four methods have very small generalization errors (magnitudes  $< 0.2$ ), and this low error is relatively uniform across all frequencies. This shows that all the models models are very effective at reconstructing all the frequencies in images with glass blur—in fact, they can reconstruct these images better than they can reconstruct clean images—corroborating our first finding in Section 4.2.1. Again the  $\mathcal{R}$  plots for glass blur for both of these models look very similar to Figure 1a; this similarity has a simple explanation due to a fundamental tradeoff between generalization and robustness. This relationship between  $\mathcal{R}, \mathcal{G}$  and average PSD of corruptions is described more precisely in Appendix L.4.
High-frequency corruptions uncover the spectral bias of NIC models. Section 4.2.1 highlighted that high-frequency signals severely degrade the generalization performance of all image compression methods. From the RD curves with respect to the corrupt images (top right plot in Figure 3), we observe that at each severity the three models have almost identical performance in the usual 0-2 bpp range. These results might lead us to expect that these models make similar reconstruction mistakes, but our spectral inspection tools indicate that this is not the case at all. Our plots of  $\mathcal{G}$  on shot noise at severity  $= 5$  (bottom row, columns 2, 4, 6, and 8 of Figure 4) indicate that NIC models distort the highest frequencies significantly more than the low and medium frequencies (notice the orange borders of the plots) while JPEG and JPEG2000 exhibit different patterns. This nuance, which is equivalent to one we previously observed with clean data in Figure 2, becomes more apparent from the plots of  $\mathcal{G}$  on a high-frequency corruption because high-frequency signals are much more prevalent on this data than on the clean data. Additionally, the bottom right plot of Figure 3 shows that for shot noise, NIC models achieve higher PSNR of the reconstructed corrupt images with respect to the clean images than the classic CODECs, i.e., they have a stronger denoising effect. In the bottom right of Figure 4 we see a more detailed picture: for high severity shot noise, NIC models achieve lower  $\mathcal{R}$  in high frequencies. Also, note that the NIC models have the smallest errors in  $\mathcal{G}$  whereas they have the largest errors in  $\mathcal{R}$  and vice versa. Thus, these findings suggest that NIC models behave similarly to a low-pass filter.
4.3 Impact of spectral artifacts on downstream applications
In practice, image compression algorithms may be used as a pre-processing step before images are used for another downstream task. For this reason, practitioners should also consider the performance on downstream tasks when comparing compression methods. We analyze the effectiveness of the compression methods on the downstream task of ImageNet classification using a pre-trained ResNet-50 model and report the difference in top-1 accuracy after compression (Figure 5) [25].
NIC can improve the robustness of downstream classification to high-frequency corruptions. While in general, compression degrades classification performance on clean images (all compressors show negative differences for the "clean" category in Figure 5), in some cases NIC and JPEG can actually improve the robustness of the classification model against high-frequency corruptions. Specifically, at PSNR=36.93, the difference in accuracy is positive for high-frequency corruptions with both SH NIC and ELIC, meaning that the classification model gave a higher accuracy on the set of corrupted images after compression than it did on the original corrupted images. Compressing with JPEG or JPEG2000 at the same rate caused a degradation in accuracy on these corruptions. However, at higher compression rates (PSNR=32.97 and PSNR=31.17), JPEG improves classification performance for high-frequency corruptions, while SH NIC and ELIC do not. On low- and mid-frequency corruptions, JPEG and JPEG2000 have a smaller degradations in accuracy compared to NIC the NIC models and no model gives an improvement in classification accuracy. Thus, the ideal compressor for downstream tasks is dependent on the type of corruption and level of compression.
Pruning NIC models amplifies the robustness gains of NIC for downstream classification tasks. Additional experimental results in Appendix C (Figure 9), show that pruned NIC and NIC optimized for MS-SSIM act as even better high-frequency signal denoisers for this application.
5 Theoretical analysis of the OOD performance of NIC
We corroborate our empirical findings with theoretical results. Here we summarize theoretical results on linear autoencoder-based NIC methods. For complete definitions, additional references, and proofs of the following statements, please refer to Appendix L. In Appendix L.3, we provide a more general result applicable to nonlinear models.
Recall a classical observation: in the setting of linearly auto-encoding a mean-centered distribution, the reconstruction function (i.e., encoder-decoder composition) is a projection onto the high-variance principal components of the input data [18] (see also [7]). Combining this with well-known facts about statistics of natural images, we show that a linear autoencoder applied to a natural image dataset retains only low-to-mid (spatial) frequency components, which account for the majority of the variance. Using this result, we state theoretical explanations for multiple trends in our experiments<sup>4</sup>.
Lemma 5.1. Let  $\mathcal{X}$  be a dataset of natural images and let  $\hat{\mathcal{X}}$  denote its (spatial) discrete Fourier transform. Assume the following (for supporting evidence see Appendix L):
(I) The principal components of  $\hat{\mathcal{X}}$  are roughly aligned with the spatial Fourier frequency basis.  
(II) The associated variances are monotonically decreasing with frequency magnitude (more specifically according to the power law  $\frac{1}{|i|^{\alpha} + |j|^{\beta}}$ ).
If  $\mathcal{C}$  is a linear autoencoder trained by minimizing MSE on  $\mathcal{X}$ , with latent space of dimension  $r$ , and if  $\hat{\mathbf{\Gamma}}$  denotes the spatial discrete Fourier transform, for any data point  $X$  with Fourier transform  $\hat{X}$
$$
\widehat {\mathcal {C} (X)} \approx \left\{ \begin{array}{l l} \hat {X} _ {:, i j} & : i ^ {2} + j ^ {2} \leq \frac {r}{\pi K} \\ 0 & : o t h e r w i s e. \end{array} \right.
$$
where  $K$  is the number of channels in the images in  $\mathcal{X}$  and where  $\hat{X}_{:,ij}$  denotes the components of  $\hat{X}$  corresponding to spatial frequency  $(i,j)$ .
Corollary 5.2. Under the hypotheses of Lemma 5.1, the robustness error of  $\mathcal{C}$  to a corruption  $c$  (as defined in Definition 3.3), measured in spatial frequency  $(i,j)$ , is
$$
\mathcal {R} (\mathcal {C}, \mathcal {X}, c) _ {i j} \approx \left\{ \begin{array}{l l} \frac {1}{N} \sum_ {k} | (\widehat {c (X _ {k})} - \hat {X _ {k}}) _ {:, i j} | & : i ^ {2} + j ^ {2} \leq \frac {r}{\pi K} \\ \frac {1}{N} \sum_ {k} | \hat {X} _ {k,:, i j} | & : o t h e r w i s e. \end{array} \right.
$$
Corollary 5.3. Under the hypotheses of Lemma 5.1, the generalization error of  $\mathcal{C}$  to a corruption  $c$  (as defined in Definition 3.2), measured in spatial frequency  $(i,j)$ , is
$$
\mathcal {G} (\mathcal {C}, \mathcal {X}, c) _ {i j} \approx \left\{ \begin{array}{l l} 0 & : i ^ {2} + j ^ {2} \leq \frac {r}{\pi K} \\ \frac {1}{N} \sum_ {k} \sqrt {| \hat {X} _ {: , i j} | ^ {2} + 2 \hat {X} _ {: , i j} (\widehat {c (X)} - \hat {X}) _ {: , i j} + | (\widehat {c (X)} - \hat {X}) _ {: , i j} | ^ {2}} & : o t h e r w i s e. \end{array} \right.
$$
<sup>4</sup>Since this simplified model has no compression rate objective, one should only expect the above theoretical results to be predictive (or explanatory) of the high bpp/PSNR cases of our experiments.
Lemma 5.1 suggests that autoencoder compressors trained on natural images behave like low-pass filters, corroborating our claim in Section 4.2.2 (see also remark L.1).
Corollary 5.2 suggests that autoencoder compressors trained on natural images are less robust to corruptions with large amplitude in low frequencies (in the sense that  $|(c(X) - X)_{:,ij}|^2$  is large for small values of  $ij$ ). This is indeed what we see in Figure 3 (bottom left), where snow corruptions are detrimental to PSNR of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $\mathcal{X}$ , and Figure 4, where the  $\mathcal{R}(\mathcal{C},\mathcal{X},c)$  error for the NIC is concentrated in low frequencies. We also observe in Figure 5 that NIC is more beneficial for downstream classification accuracy in the case of high-frequency corruptions (e.g. shot noise) and less beneficial in the case of low-frequency corruptions (e.g. snow).
On the other hand, the conclusion of Corollary 5.3 is more involved than that of Corollary 5.2—the “cross term”  $2\hat{X}_{:,ij}(\widehat{c(X)} - \hat{X})_{:,ij}$  is in general non-zero. However, there are cases where in expectation over the data set  $\mathcal{X}$  this cross term vanishes (e.g., when  $c$  is additive noise). In such cases, Corollary 5.3 suggests that compressors trained on natural images generalize less successfully to shifts with large amplitude in high frequencies. In Figure 3 (top right) we see that shot noise corruptions are detrimental to PSNR of  $\mathcal{C}(c(\mathcal{X}))$  w.r.t.  $c(\mathcal{X})$ , and in Figure 4 we see that for the snow and shot noise corruptions,  $\mathcal{G}(\mathcal{C}, \mathcal{X}, c)$  are concentrated in high frequencies. In summary, with both theoretical analysis of a simple mathematical model and empirical results, we find that NICs have a spectral bias that causes them to overfit to discard high frequency components of natural images..
Due to the number of axes of variation present in our experiments (e.g., model type, compression rate, evaluation dataset, corruption type, corruption severity, evaluation metric), we were forced to constrain many variables. We present results for a limited number of NIC models all trained on one dataset, CLIC. Due to limitations on the number of models we could train and discrete choices for classic codec hyper-parameters, there is some variation within our fixed- {bpp,PSNR} bins.
Evaluating the robustness of any machine learning system is inherently a multi-faceted problem riddled with unknown-unknowns. While we focus on robustness to naturalistic input-data corruptions, there are other important distribution shifts for researchers and practitioners to consider, such as subtle changes in data collection methodology [51]. We omit a study of NIC model adversarial robustness to worst-case input data perturbations. Nevertheless, we hope that the our evaluation methods (including our metrics  $\{\mathcal{D},\mathcal{G},\mathcal{R}\}$  and Fourier heatmap spectral inspection tools) will be useful in future work on NIC model robustness.
We proposed benchmark datasets and inspection tools to gain a deeper understanding of the robustness and generalization behavior of image compression models. Using our spectral inspection tools, we uncovered the modus operandi of different compression methods. We also highlighted similarities and differences among them via a systematic OOD evaluation. Exploring the use of our tools in other image compression methods, including transformer-based architectures and implicit neural representations methods is expected to provide interesting insights [41, 72, 73, 45, 54]. Our OOD evaluations identify NIC model brittleness issues: designing practical robust training approaches and/or methods to efficiently adapt to distribution shifts at runtime is a worthwhile research direction.
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 22-DR-009 (LLNL-JRNL-851532).
Moreover glass blur can be viewed as a sort of convolution operator, for which one can verify that the aforementioned cross term is non-zero.
The third author was supported by the Laboratory Directed Research and Development Program at Pacific Northwest National Laboratory, a multiprogram national laboratory operated by Battelle for the U.S. Department of Energy.
We thank Eleanor Byler for helpful comments on an earlier draft.
[1] Origins of scaling in natural images. Vision Research, 37(23):3385-3398, Dec. 1997. ISSN 0042-6989. doi: 10.1016/S0042-6989(97)00008-4.
[2] Oodhdr-codec: Out-of-distribution generalization for hdr image compression. 36:158-166, Jun. 2022. doi: 10.1609/aaai.v36i1.19890. URL https://ojs.aaaai.org/index.php/AAAI/article/view/19890.
[3] Vvc software - versatile video coding test model (vtm), Year of access, e.g., 2023. URL https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.
[4] G. Alain and Y. Bengio. What Regularized Auto-Encoders Learn from the Data-Generating Distribution. Journal of Machine Learning Research, 15(110):3743-3773, 2014. ISSN 1533-7928.
[5] J. An and S. Cho. Variational Autoencoder based Anomaly Detection using Reconstruction Probability. 2015.
[6] A. E. Aydemir, A. Temizel, and T. T. Temizel. The effects ofJPEG andJPEG2000 compression on attacks using adversarial examples. arXiv preprint arXiv:1803.10418, 2018.
[7] P. Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. *Neural Networks*, 2(1):53-58, Jan. 1989. ISSN 0893-6080. doi: 10.1016/0893-6080(89)90014-2.
[8] J. Balle, V. Laparra, and E. P. Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rJxdQ3jeg.
[9] J. Balle, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rkcQFMZRb.
[10] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja. Compressai: a pytorch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:2011.03029, 2020.
[11] E. O. Brigham and R. E. Morrow. The fast fourier transform. IEEE Spectrum, 4(12):63-70, 1967. doi: 10.1109/MSPEC.1967.5217220.
[12] G. J. Burton and I. R. Moorhead. Color and spatial structure in natural scenes. Applied Optics, 26(1):157-170, Jan. 1987. ISSN 2155-3165. doi: 10.1364/AO.26.000157.
[13] Y. Choi, M. El-Khamy, and J. Lee. Variable rate deep image compression with a conditional autoencoder. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3146-3154, 2019.
[14] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=SSKZPJCT7B.
[15] J. Diffenderfer and B. Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning a randomly weighted network. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=U_mat0b9iv.
[67] S. Yin, C. Li, Y. Bao, and Y. Liang. Universal efficient variable-rate neural image compression, 2021. URL https://arxiv.org/abs/2111.11305.
[68] S. Yin, C. Li, W. Tan, Y. Bao, Y. Liang, and W. Liu. Exploring structural sparsity in neural image compression, 2022. URL https://arxiv.org/abs/2202.04595.
[69] D. Zhang, J. Yang, D. Ye, and G. Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[70] M. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=Sy1iIDkPM.
[71] X. Zhu, J. Song, L. Gao, F. Zheng, and H. T. Shen. Unified multivariate gaussian mixture for efficient neural image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17612-17621, June 2022.
[72] Y. Zhu, Y. Yang, and T. Cohen. Transformer-based transform coding. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=IDwN6xjHnK8.
[73] R. Zou, C. Song, and Z. Zhang. The devil is in the details: Window-based attention for image compression, 2022.
A Background and Related Work
A.1 Neural Image Compression (NIC)
In lossy image compression, an input image is compressed into a small file or bitstream for transmission or storage. This file is then used to reconstruct the original image. Several models have been proposed for neural (or learned) image compression such as convolutional neural networks (CNNs), autoencoders, and generative adversarial networks [22, 8, 9, 46, 58, 13, 65, 71, 47]. In the most basic version of these models, an image vector  $\mathbf{x}$  is mapped to a latent representation  $\mathbf{y}$  using a convolutional autoencoder.  $\mathbf{y}$  is then quantized into  $\hat{\mathbf{y}}$ , so it can be compressed to a bitstream via an entropy coding method, such as arithmetic coding [52]. During reconstruction, the received bits go through the reverse entropy coding process and the revealed  $\hat{\mathbf{y}}$  is fed into a CNN decoder which outputs the reconstructed image  $\hat{\mathbf{x}}$ . These models can be trained end-to-end by using differentiable proxies of entropy coding [8].
Normally entropy coding relies on one prior probability model of  $\hat{\mathbf{y}}$ , which is learned from the entire set of images. Balle et al. [9], however, recognized that the prior of this model can be optimized for each image separately. By adding an additional hyperprior autoencoder to their model, they learn the latent representation for the entropy model of each image. The information transmitted from this autoencoder can be seen as equivalent to "side information" used by other codecs. Minnen et al. [46] further improved this model by adding an autoregressive prior to the hierarchical prior used by Balle. The Efficient Learned Image Compression (ELIC) model builds on these models by adding uneven channel-conditional adaptive coding and a spatial context model [24].
A.2 Generalization and robustness in DNNs for classification
It is well known that deep neural networks (DNNs) are prone to overfitting to the specific data distribution they are trained on. Much recent work has focused on improving the robustness of DNNs to other distributions of data, including common image corruptions [27, 66]. CIFAR-10-C and ImageNet-C were proposed as a benchmark datasets to test the robustness of models to common image corruptions such as corruptions from weather, blur, and digital noise [26]. Methods for improving robustness include data-level techniques such as adding augmentations and algorithm-level techniques like using larger models or adversarial training. [27, 32, 28, 14].
Yin et al. study the effects of data augmentation and adversarial training on different corruptions using a Fourier analysis [66]. They analyze the CIFAR-10-C dataset and find that most augmentation methods help in the high-frequency domain, but hurt in the low-frequency domain. However, it is unknown how image compression affects corrupted images. We analyze how different types of image compression affect the Power Spectral Density (PSDs) of both clean and corrupted data.
A.3 Generalization and robustness with NIC models
There is a limited amount of work studying the generalization and robustness of NIC models. [37] formulates a problem for designing robust NIC models and train DNNs to be robust to two particular types of distributional shifts. They find a tradeoff between OOD robustness and IND performance. Our work is complementary to this work in that we provide additional metrics and OOD benchmarks to analyze the robustness of various models in more detail. [42] studies NIC models' robustness to white-box and black-box adversarial attacks and proposes new models which improve the robustness of NIC to such attacks. Our work instead looks at naturalistic distribution shifts, which is another important aspect of robustness [26].
[2] designed a NIC module which works on a specific set of OOD images. In particular, their module works on high dynamic range (HDR) images, despite being trained on mostly standard dynamic range (SDR) images. Although this is a specific example of testing NICs on OOD data, our work differs from theirs in that we analyze how NICs would perform on a wide variety of image corruptions without seeing any examples of these corruptions during training.