JPEG Inspired Deep Learning
Authors: Ahmed H. Salamah
Kaixiang Zheng
Yiwen Liu
EN-HUI YANG
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
JPEG Compression
Quantization
Non-linearity
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose JPEG-DL, a new DL framework where a trainable JPEG compression layer is jointly optimized with DNN to improve the model prediction performance.

Abstract: Although it is traditionally believed that lossy image compression, such as JPEG compression, has a negative impact on the performance of deep neural networks (DNNs), it is shown by recent works that well-crafted JPEG compression can actually improve the performance of deep learning (DL). Inspired by this, we propose JPEG-DL, a novel DL framework that prepends any underlying DNN architecture with a trainable JPEG compression layer. To make the quantization operation in JPEG compression trainable, a new differentiable soft quantizer is employed at the JPEG layer, and then the quantization operation and underlying DNN are jointly trained. Extensive experiments show that in comparison with the standard DL,  JPEG-DL delivers significant accuracy improvements across various datasets and model architectures while enhancing robustness against adversarial attacks. Particularly, on some fine-grained image classification datasets, JPEG-DL can increase prediction accuracy by as much as 20.9%. Our code is available on https://github.com/AhmedHussKhalifa/JPEG-Inspired-DL.git.

Published as a conference paper ICLR 2025
JPEG INSPIRED DEEP LEARNING
Ahmed H. Salamah", Kaixiang Zheng,, Yiwen Liu & En-Hui Yang Department of Electrical and Computer Engineering, University of Waterloo (ahamsal lmaa,, k 256heng, e3liu, @wwaterloo.cc
ABSTRACT
Although it is traditionally believed that lossy image compression, such JPEG com- pression, has a negative impact on the performance of deep neural networks (DNNs), is shown by recent works that well-crafted JPEG compression can actually improve the performance of deep learning (DL.. Inspired by this, we propose JPEG-DI, novel DI. framework that prepends any underlying DNN architecture with a trainable JPEG compression layer. To make the quantization operation in JPEG compression trainable, new differentiable soft quantizer is employed at the JPEG layer, and then the quantization operation and underlying DNN are jointly trained. Extensive experiments show that in comparison with the standard DL. JPEG-DL delivers significant accuracy improvements across various datasets and model architectures while enhancing robustness against ad- versarial attacks. Particularly, on some fine-grained image classification datasets, JPEG- DL increase prediction accuracy by much as 20.9%. Our code is available Inspired-DL..g
https: /ttt:///iihubl com/ http///Aneduussaaaaaaeecee
1 INTRODUCTION
JPEG compression [Pennebaker & Mitchell, 1992) is the defacto lossy image compression technique with ubiquitous presence real-world applications. With the development of deep learning, more and more images, potentially compressed by JPEG, are consumed by deep neural networks (DNNs). Naturally, it's of interest to study how JPEG compression will impact DNN performance for computer vision tasks, and extensive research has been conducted along this line such Dodge & Karam (016); Liu al. (2018) and Xie & Kim (2019). These initial explorations establish a widely accepted view that the information loss caused by JPEG obscures important features in the input image, thereby negatively impacting DNN performance.
However, it shown by Yang et al, (2021) that the above conventional wisdom does not hold anymore if JPEG compression is applied intelligently and adaptively on a per-image basis. Indeed, Yang et al. (202I) showed that if applied appropriately, JPEG compression can actually improve DNN performance, least in theory. They further proposed to train a DNN based on images with various JPEG quality levels. While they managed to demonstrate improved performance with specially designed DNN topology, the model too cumbersome to be fully trained. leading to suboptimal performance. Moreover, the adherence to the default JPEG quantization also limits the effectiveness of this method. On the other hand, given fixed DNN model, Zheng et al. (023) and Salamah et al. (2024b) found that its performance could be slightly improved if the input images got compressed by JPEG with optimized quantization parameters. However, the performance gain is not significant due to the frozen DNN model. Although these existing works provide promising insights to improve DNN performance with JPEG compression, a solid solution that can fully unleash the potential of this idea has yet be found.
Authors contributed equally.

---

Published as conference paper ICLR 2025
Q.
Differeniable Soft Quantizer
Inyut Image
Reconstractod
LA
Irgur
Image
Layer
Predictinn
918.5 7008
70880
DNN fal:
RGB YChCr Color
YChCr RGB
100.2 9003
100.0 0
Space
Space Conversion
90.02
overiap
Partiticiing
Blocks Merging
16.1
1804 10.09
DCT Tratsfoom
RLM:
Difeeremiible
JPEG Lager
G.
Input
Output
(a)
(b)
Figure 1: (a) The JPEG-DL framework consists of JPEG layer followed by standard DNN, where the standard forward/inverse processes of JPEG depicted in white boxes are fixed and not trained. The JPEG pipeline receives a conventionally preprocessed input image x R3*WxH sampled from the underlying task's 's dataset. The JPEG layer, equipped with a differentiable soft quantizer, and the underlying DNN form a unified new DNN architecture, which are shown blue indicating that they are trainable components. (b) As example, show 21.:.; i.e., the DCT representation of the Y channel of an image, by tensor consisting of B blocks of DCT coefficients. Each 8x8 block contains M = 64 DCT frequencies, ordered from low to high in zigzag manner. Then, show how 1,, and M,; are quantized by Qa(: ;g 1, a 10) and Qa(: :9 :
0.5, a = 16), respectively.
To address the above issue, in this paper, we propose jointly optimizing both JPEG quantization operation and DNN achieve greater effectiveness. To this end, first introduce trainable JPEG compression layer, the structure of which is illustrated in Fig. Ta This layer follows the standard JPEG pipeline to convert preprocessed input image, after passing through the input layer, into blockwise Discrete Cosine Transform (DCT) coefficients in the YCbCr color space. A novel differentiable soft quantizer (Qa) is then applied to quantize the DCT coefficients at each frequency position, followed by the standard JPEG inverse process to reconstruct the RGB image. We then present JPEG-DL, a novel deep learning (DL) framework that introduces a new DNN architecture by inserting a JPEG layer directly after the original input layer of any underlying DNN architecture. This layer can be considered architecture, forming new unified DNN architecture, whose parameters get optimized jointly with DNN model weights during training. The core of JPEG-DL is Qa it substitutes the non-differentiable. hard quantization in JPEG with a differentiable, soft quantization operation defined by a neat analytical formula, which not only facilitates gradient-base baseed optimization for quantization parameters but also introduces addi- tional trainable non-linearity to the overall image understanding pipeline. To validate the effectiveness of JPEG-DL, conducted extensive experiments for image classification six datasets including four fine- grained classification datasets (Wah et al. 2011: Khosla et al. 2011] Nilsback & Zisserman, 2008 Parkhi al. 2012), CIFAR-100 (Krizhevsky
integral part of any underlying DNN
al.) 2009) and ImageNet (Deng et al.] 2009).. Results show that JPEG-DL significantly and consistently outperforms the standard DL across various DNN architectures, with negligible increase in model complexity. Specifically, JPEG-DL improves classification accuracy by up to 20.9% on some fine-grained classification dataset, while adding only 128 trainable parameters to the DL pipeline. Moreover, the superiority of JPEG-DL over the standard DL is further demonstrated by the enhanced adversarial robustness of the learned unified architecture.
2

---

Published as a conference paper at ICLR 2025
The main contributions of this paper can be summarized as follows:
We introduce novel trainable JPEG layer leveraging differentiable soft quantizers with nice ana- lytical formulas.
Based on the new JPEG layer, we propose a new DL framework dubbed JPEG-DL, which jointly optimizes the JPEG layer and the DNN model during training.
The outstanding performance of JPEG-DL over the standard DL is verified by comprehensive perimental results on various image classification datasets across multiple DÑN architectures and for a variety of tasks including adversarial defense.
2 BACKGROUND AND RELATED WORK
JPEG applications. JPEG was originally developed as a lossy image compression technique based on transform coding, which reduces image file sizes by generating their compact representations. Beyond its traditional use, JPEG has found numerous applications in deep learning: (1) it has been utilized as a data augmentation technique to improve robustness of DNNs against image compression (Benbarrad et al. (022);; (2) it has been employed as an empirical defense method against adversarial attacks, effectively reducing adversarial perturbations and enhancing the adversarial robustness of DNNs (Dziugaite et al., [2116] Das et al. 2017] Guo et al. (2018); and (3) it has been integrated into the knowledge distillation (Hinton et al. 2015 framework by (Salamah et al. (2024a), where it helps the teacher model transfer knowledge to the student model in a more effective way. In contrast, this paper focuses on leveraging JPEG to enhance the natural performance, instead of the robust performance, of DNNs without relying on any teacher model.
Optimizing JPEG Compression for DNN vision. As lossy image compression technique. JPEG is de- veloped specifically for the human visual system. As result, while the information loss introduced by JPEG is often imperceptible to humans, it can significantly degrade the performance of DNNs. This issue gives rise a line of research which optimizes JPEG compression based on DNN perception. For instance, given a pretrained DNN. Xie & Kim 2019). Zheng et al. (2023) and Salamah et al. (2024b) first derive its sensitivity to different DCT frequencies, based on which they customize JPEG quantization tables for this DNN to reach the optimal rate-accuracy tradeoff.. Another popular direction is make JPEG trainable, which integrates the JPEG encoder and DNN into an end-to-end differentiable training framework (Luo) et al. 2020 Xie et al. 2022). These methods create differentiable proxies for image quantization and bitrate calculation, so that JPEG compression can be optimized via backpropagation in order to minimize a total loss considering both DNN performance and bitrate. However, all the above works focus mitigating DNN performance degradation in the presence of JPEG compressed images, but offer little to no improvement on DNN performance when raw images are given as input. On the contrary, this paper leverages JPEG purely as tool to improve DNN performance on raw images, regardless of its compression capability.
Improving DNN performance with JPEG. Due to the aforementioned reason, JPEG compression gener- ally hurts DNN performance. However, it's recently demonstrated by |Yang et al. (222)) that, with an oracle guiding the compression process, one can select an optimal quality level to compress each image, enabling the DNN to make its best possible prediction for it. By applying this adaptive JPEG compression across set of images, one can actually improve the DNN prediction accuracy considerably. This phenomenon, termed "compression helps" in the original paper, is justified by the fact that compression can remove noise and disturbing background features, thereby highlighting the main object in an image, which helps DNNs make better prediction. Due to the need of ground truth labels in the compression stage. the above adaptive com- pression scheme is not realizable in most real world applications; however, the discovery of "compression helps" at least show the potential of using JPEG to improve DNN performance. Thus motivated, the authors in turn proposed an implementable way to improve DNN performance with JPEG. They built a new DNN topology that incorporates 11 parallel branches of underlying pretrained model, with each branch obtain-
3

---

Published as a conference paper at ICLR 2025
ing as input either the raw image its compressed version at varying quality level. The penultimate layer representations from all branches are concatenated and fed into a classification head, which is then trained together with those 11 model backbones as a unified DNN structure. This new DNN topology is clearly too complex to train, so the authors opted for partial training to mitigate the training complexity, therefore result- ing in a suboptimal performance. In contrast, our proposed method introduces only 128 additional trainable parameters to the existing DL pipelinee causing negligible complexity increase. Moreover, it's noteworthy that our method is orthogonal to theirs, as our trainable JPEG quantization tables can be embedded into their framework to replace the default JPEG quantization tables, thereby further improving the performance.
Trainable Activation Functions. Relentless efforts have been made to search for activation functions that improve DNN performance. Among all the directions, trainable activation functions have gained partic- ular interest for their flexibility, expressive power, and adaptability during training. Chen & Chang (999)
propose the adjustable generalized hyperbolic tangent function, which extends the classic hyperbolic tangent function by introducing parameters to control the saturation level and slope of the function. He et al. (0015)
introduce the parametrie ReLU (PReLU), a variant of ReLU with a trainable parameter that adjusts the nega- tive part of ReLU. More recently, activation functions on edges, with nodes simply summing all the incoming activations. Interestingly, the differentiable soft quantizers in our JPEG layer can be interpreted as trainable activation functions whose input is the DCT representations of images. These trainable soft quantizers effectively introduce additional nonlinearity to DNN models, thus improving their expressive power.
Aatmogorrv-Arroole Networks (KANs) (Liu et al., (024) employ trainable
3 JPEG-DL: JPEG INSPIRED DL
3.1 PROBLEM FORMULATION
In the JPEG pipeline, an RGB image x E p3xW*x is first converted to the YCbCr color space and then partitioned into B non-overlapping 8 x 8 blocks, where DCT is applied to each of these blocks to obtain the corresponding DCT coefficients. For each color channel, the DCT coefficients in each block are then flattened following the zigzag order, resulting in M = 64 frequency positions ordered from low frequency to high frequency. Therefore, we denote the DCT coefficients of the image x as z = [= where = 1,3 corresponds to the color channel Y, Cb and Cr respectively, 1 S m M is the index for frequency position, and 1 S n S B is the index for block. Up to this point, all operations involved are differentiable. Next, quantization tables Qy = [91, Q2. ..,, 9as] and Qc [4M+1 4M+ .... 922]] are used for the luminance (Y) and chrominance (CbCr) channels respectively to quantize their DCT coefficients. Following uniform quantization, obtain quantized DCT coefficients êt,,, |z,,,n// /mm] .mm for I = 1, and f,,m = | /....n//M++m /m 9M+mm for = 2.3. Note that quantization is non-differentiable due to the use of the rounding operation. Finally, inverse operations including blockwise inverse DCT (IDCT) transform, blocks merging and YCbCr-to-RGB color space conversion are conducted over & sequentially to obtain the reconstructed RGB image é. Similar to their forward counterparts, all these inverse operations are diffcrentiable. Denoting the composition of all the above operations as J, we then have å = J (I; Q), where Q = (Q). Hereafter, the mapping J stands for the JPEG encoding
operation.
In supervised learning, each x E X' corresponds to ground truth label E y. Let fo represent a DNN model with trainable weights 0, and let L denote the loss function used to train this DNN. standard DL, the primary objective is to solve the following minimization problem:
min E[C((o(x), v)]-
In contrast, JPEG-DL tries to improve the performance of DNN by jointly training it with the JPEG opera- tion. As a result, the formulation should be instead:
min (IC 0.Q
(2)
4

---

Published as conference paper ICLR 2025
However, in order to solve (2 with gradient descent, the key challenge is caused by the non-differentiable quantization operation, which makes the gradients w.r.t. Q almost zero everywhere. To address this issue, we will introduce a differentiable soft quantizer (Qa) in the next subsection, replacing the uniform quantizer (Qw) used in J.
3.2
DIFFERENTIABLE SOFT QUANTIZER
Denote the index set of uniform quantization as
A= - --,,--++,... L.... L-..,,-1. 1,
(3)
For convenience, is also regarded as vector of length 2/ + 1. Multiplying A with a quantization step size q, we get the corresponding reconstruction space
A q I-L, L
0. L-.....-1.. - 11
(4)
Again, will regard Á as both a vector and set.
To randomly quantize DCT coefficient z an element in A, we invoke from Yang & Hamidi (2024) train- able conditional probability mass function (CPMF) Pa (-12) over the reconstruction space A or equivalently the index set A given 2, where > 0 is trainable parameter.
E aas-ia))
Pa(igl=)= =
Ejeee-(s--S0 aa ViEA.
(5)
21. times
Extend & to vector of length 2L + i.e., [z|27-41 1,...,? Then, the CPMF Pa(+1z), regarded as vector length 2L + I, can be easily computed via the softmax operation (-)):
[Pa(1=)] = (a (1=)2141 -A)² ))²).
(6)
With the CPMF (-1+2,, is now quantized to each iq À with probability P. (iqlz). Note that Q x0. (-12) approaches an one-hot vector with probability the nearest point in A and 0 elsewhere. Therefore, the resulting random quantizer effectively functions as the deterministic uniform quantizer (=(2) |2/g] .g.
Based on the CPMF Pa (-12,, we can now define a differentiable soft quantizer Qa as the conditional expec- tation of iq given z, i.e.,
Qa(z) [[ig|=] = Σ Paglz) ig.
iEA
Similarly, as a - oo, Qu also goes to Q.. Fig. shows how the shape of Qa varies w.r. a, given fixed q-
This soft quantizer Qa serves as an analytical proxy for Qu- It's differentiable everywhere, allowing gra- dients to flow through it smoothly. More importantly, compared to Qu. Qa involves a trainable parameter which can adjust the softness of the quantizer, thereby introducing more flexibility. In view of these nice properties, Qa is the ideal candidate in place of Qu used in J.
As side note, for the reader who is not familiar with guantization. but familiar with the attention operation used in transformer models /Vaswani (2017), Qa can be regarded as an attention operation in broad sense, with the query being 2, the key and value being A, and the similarity metric between the query and key being negative squared distance instead of dot product.
5

---

Published as a conference paper ICLR 2025
3.3 OVERALL FRAMEWORK OF JPEG-DL
Substituting Qu in I with Qa. we get differentiable JPEG layer I parameterized by Q and a, where a (oy, ac). ay = [ar1, a.2,..., aM and ac [aM+11 4M2 ................................ [22M] are a tables for the luminance and chrominance channels respectively, used in conjunction with Qy and Qc to quantize DCT coefficients. Fol- lowing the proposed soft quantization, we obtain quantized DCT coefficients 2,m,, = Qa(E1.m,r 9ms am) for = 1, and 2,, soft quantizer defined in 7 parameterized by a quantization step q and a scaling factor a. Overall, for an input image x, we have à = [x; Q, ox. Therefore, we can rewrite the JPEG-DL formulation, as
2a(21,m,,i 4M+m 0M++m) for / 2, 3, where Qa(2; q, denotes differentiable
min (T;; 0.Q.a
(8)
where the expectation can be approximated by the empirical mean over a mini-batch in actual training. Thanks to the of Qd, (8) can now be solved by gradient descent with ease (see Appendix A. for the derivative analytical formulas).
After training with JPEG-DL, we get the optimized parameters 0" Q' and a². Then, consider the composition of the JPEG layer and the underlying DNN as unified DNN model f(x; 0* -"'0"""""0'" a*"x"") Jo (F (z; Q" ax)) ) to do validation. Concretely, any raw input x should be fed into J. instead of directly to Sa-, thus allowing the reconstructed image é to be fed into the underlying DNN fe.. In other words, the
JPEG layer is prepended as the first layer of any underlying DNN.
To conclude this section, we refer readers to Fig. la and Fig. 1b for an illustration of the inner workings of the JPEG-DL framework.
4 EXPERIMENTS
CIFAR-100. We evaluate our proposed method using a transformer- architecture and four state-of-the- art convolutional neural networks (CNNs): EfficientFormer-Li (Li al. 2022). ResNet (He et al.
2016
VGG (Simonyan& & Zisserman 2014), MobileNet (Sandler et al. 2018), and ShuffleNet (Ma et al. For ResNet, employ CIFAR-specific versions: ResNet32. ResNet56. and ResNetl10. For VGG, we utilize VGG8 and VGG13. All CNN architectures follow the training recipe from CRD (Tian et al.) 2020)
2018 J
(see Appendix A.3) while for the transformer-based architecture, EfficientFormer-11, adhere the setup proposed by Xu et al. (2023) (see Appendix A.55)
Fine-grained Tasks. These tasks involve visually similar classes and typically feature fewer training sam- ples per class compared to conventional classification tasks. We evaluate our method on four datasets: CUB-200-2011 (Wah al. 201T). Stanford Dogs (Khosla al., 2011). Flowers (Nilsback & Zisserman 2008), and Pets (Parkhi et al. 2012). For CNN architectures, we employ PreAct ResNet-18 (He et al. 2016
Gull
G.
1-
-2
-2
1
2
5
2
Figure 2: Illustration of Qn vs. Qa with 1,3, 10, where L and q are set to 3 and 1, respectively.
6

---

Published as a conference paper at ICLR 2025
and DenscNet-BC (Huang et al. 2017). following the experimental setup and architecture modifications of Zhang et al. 2017). For the transformer- architecture, we use EfficientFormerr-Il, adopting the setup outlined by Xu et al. (2023) (see Appendices A.4 and A.5).
JPEG-layer settings for CIFAR-100 and fine-grained tasks. To train our JPEG layer, we first study the gradient nature of Qu with respect to q and a in ( by changing one parameter and fixing the other. Both g and a are sharable parameters among different numbers of blocks per image. During the calculation of the gradient of Qa with respect to a, our analysis shows that when a is sufficiently large, such that Qa not too far from Qu. the gradient magnitude for a is almost zero, shown in Appendix A.Il This indicates that a will not be updated effectively if initialized within a reasonable range. Experiments also confirmed that making a trainable does not significantly impact model performance. As result, choose not to train in our framework. However, when we explore the calculation of the gradient of Qa with respect to q. the accumulated gradient received from different blocks per image during backpropagation results in unstable gradient magnitudes at a fixed value of a, as shown in our analysis in Appendix [ To address this, we use the ADAM optimizer, which adapts the learning rate for each trainable parameter q in our JPEG layer, enabling more efficient training and better convergence. For CIFAR-100, we set the learning rate to 0.003 across all tested models. For the fine-grained datasets, we set the JPEG learning rate to 0.005. Across these datasets, we fix O'm 5 for all 1 S m S 2M, and set L. = 26-1 in where b is a tunable hyperparameter set to 8.
ImageNet-1K. For all experiments on this dataset, we utilize the standard training recipes shown by Paszke et al. (2019) without any modifications. We use SqueezeNet (landola 2016), ResNet-18 and ResNet-34 as our testing underlying models.
JPEG-layer settings for ImageNet-IK. We will utilize specific settings to control the gradient magnitude to ensure more stable updates for Q We define the Gradient Scaling Constants f. 0m 1 < m < 2.M, which allows us to control the magnitude of gradients w.r.t Q. Specifically, we fix hm = 0.7 for all 1 S update the value of O'm to be hn/ 9 before calculating the gradients w.r. Q. As a result of controlling the maximum gradient magnitude, we can optimize Q using an SGD optimizer with 0.5 learning rate, instead of ADAM optimizer. This approach ensures stable and efficient training for quantization table updates (see Appendix A.2 for more details). For this dataset, b is equal to 11 across all tested models.
2M. During training,
Quantization Table Initialization. For CIFAR- and fine-grained tasks, we initialize the quantization table Q based on the reciprocal of sensitivity for each DCT frequency, given pre-trained model of the un- derlying DNN architecture, following the approach described by Zheng el al. (2023): Salamah et al. (2024b). Broadly speaking. the sensitivity of frequency indicates the rate of change of the loss function w.r.t. the perturbation on this frequency, so we favor a smaller quantization step for more sensitive frequency to limit the distortion amount on For ImageNet-IK, we adopt the strategy from Esser et al. (2019), where the initialization of gm is based on the average of absolute values of DCT coefficients across all blocks in all
training images that will be quantized by 9m- Specifically, 9m = 2 EB
N
B
| /
I/NBV/ 2b
for
1 S m S M, i.e Qy, and 4m EN Ei= ER / I/N///////// 12-1) for M + 1 S m S 2M, i.e., Qc where l is the index for image and N is the number of training images.
Table Top-1 validation accuracy (%) for Baseline and JPEG-DL on CIFAR-100. The Baseline results are from Tian et al. (2020). For JPEG-DL, we report the mean and standard deviation of experimental results over three runs.
Method
Res32
Res56
Res110
VGG8
VGG13
MobileNesV2
ShuffleNerV2
Baseline
71.14
72.34
73.79
70.36
73.77
64.6
71.82
JPEGDDL
71.92 (+0.78)
a3 (+1.05)
74.46 (+0.67)
71.10 (+0.74)
75.32 (+1.55)
65.91 (+133)
73.4
7

---

Published as a conference paper ICLR 2025
Table 2: Top-1 validation accuracy (%) on various fine-grained image classification tasks and model archi- tectures. We report the mean and standard deviation of experimental results over three runs.
Model
Method
CUB-200
Dogs
Flowers
Pets
Baseline
54.002149
23.7
57.13
70.37
R8sNet-18
JPEG-DL
58,81 (+4.81)
65.57 (+1.86)
68.76gu87 (+11.63)
74.84 (+4.47)
Baseline
57.702a44
66.61
51.322037
70.2662079
DenseNet-121
JPEG-DI.
61.321045 (+3.62)
69.67 (+3.06)
72.22 (+20.90)
25.90 (+5.64)
Table 3: Top-1 I validation accuracy (%) on ImageNet with different model architectures.
Method
SqueezeNetV11
Resnet18
Resnet34
Baseline
$7.95 58.26 (40.31)
69.75 70.13 (+0.38)
73.31
JPEG-DL
73.54 (40.23)
CIFAR-100 and Fine-grained tasks Results. The performance of JPEG-DL is shown in Tables 1 and 2 Across all seven tested models for CIFAR-100, JPEG-DL consistently provides improvements, with gains of up to 1.53% in top-1 accuracy. In the fine-grained tasks, JPEG-DL offers substantial performance up to 20.90% across all datasets using two different models. Additional
increase, with improvements results for CIFAR- 100 and fine-grainess tasks using transformer- A.5 We further extend our results on ine-grained tasks to include a comparison with additional baselines that address the non-differentiability and zero derivative problems of JPEG quantization (see Appendix[A.0g After demonstrating the nonlinearity introduced by inserting the JPEG layer after the input layer, we extend our evaluation include the replacement of the ReLU activation function with our JPEG layer, which highlights the potential of new forms of nonlinear operations in DNN architectures (see Appendix (A.7). Furthermore, we provide comprehensive experiments on the impact of chrominance subsampling on JPEG- DL, as well as a comparison with both non-learnable and learnable preprocessing methods (see Appendix A.8.
model are shown in Appendices
ImageNet-1K Results. The performance of SPEG-DL. is shown in Table 3, With a trivial increase in com- plexity (adding 128 parameters), JPEG-DL achieves a gain of 0.31% % top-1 accuracy for SqueezeNetVI.I compared to the baseline using single round of Qa quantization operation. By increasing the number of quantization rounds to five, we observe an additional improvement of 0.20%, leading to a total gain of 0.5 % over the baseline. The best results are indicated in bold, and values in parentheses indicate relative accuracy gains over the baseline.
5 ANALYSIS AND DISCUSSION
Robustness. JPEG has generally been used as an empirical defense mechanism against adversarial attacks by mitigating adversarial perturbations and enhancing the robustness of DNNs, as discussed in Section [2]
To evaluate the adversarial robustness of JPEG-DL models in comparison to standard DNN, we conduct experiments using two attack methods, FGSM and PGD, on CIFAR- with two different models from [able[] The perturbation budget, Epsilon, ranged from to 4 for both attack methods. For PGD, applied 5 steps with a perturbation step size of (2.5 x Epsilon)/steps following the setup used by Madry (2017). As shown in Fig. the JPEG-DL models significantly improve the adversarial robustness compared the standard DNN models, with improvements of up to 15% for FGSM and 6% for PGD. We will show some examples of the quantization tables used in this study in the next subsection.
8

---

Published as a conference paper ICLR 2025
12.5
40
30
10.0
Top 1960
199
20
25
30
5.0
15-
2.5
10.
Epsilon
Epsioon
Epsilon
Epsilon
(a) FGSM for VGG13
(b) PGD for VGG13
(c) FGSM for Res56
(d) PGD for Res56
Figure 3: Evaluate the adversarial robustness of JPEG-DL models in comparison to standard DNN on VGG13 and Res56 for CIFAR-100 against FGSM and PGD attacks.
presents the Y and CbCr quantization tables, both at initialization and after convergence, for VGGI3 trained on CIFAR-100 and ResNet-18 18 trained on the CUB200 dataset. The estimated sensitivity values used to initialize these quantization tables are provided in Appendix A.10
Designed Quantization Tables. Fig. 4
-5
Stass
Q Stops
6
10
4
2
0
21 Frequoncy Ind ox
46
16
31
46
61
16
46
61
Frecuency Indox
Frequincy Index
Frequencyy Index
(a) VGGI3 (Y Channel)
(b) VGGI3 (CbCr Channel) (c) ResNet-18(Y Channel) (d) ResNet-18 (CbCr Channel)
Figure 4: Initial and final quantization tables for VGG13 trained on CIFAR-100 and ResNet18 trained on CUB200, with frequency indices arranged in the default zigzag order.
Feature maps visualization. Fig. presents the feature maps extracted after the first dense block in DenseNet-121 121 for both the DL model and the baseline model, trained on the CUB200 dataset us- ing the model from Table 2 The output of the feature maps at this stage is of size 56x56, and both sets are shown in the same sequence using the same original image. The shown example was incorrectly clas- sified by the baseline model, while the JPEG-DL model correctly classified it. In this figure, it is evident that the feature maps from the JPEG-DL model show significantly better contrast between the foreground information (the bird) and the background compared to the feature maps generated by the baseline model. Specifically, the foreground object in the JPEG-DL feature maps is enclosed within well-defined contour, making it visually distinguishable from the background. In contrast, the baseline model's 's feature maps show a more blended structure, where the foreground contains higher energy in low frequencies. causing it to blend more smoothly with the background. Additionally, another example in Appendix [A.1]showw simi- lar phenomenon in addition to background information being more effectively removed, following the same setup the first example. These discrepancies in feature map clarity and contrast are propagated through subsequent blocks DenseNet-121. eventually contributing to the misclassification problem observed in the baseline model.
Interpretability using CAM. Fig. 6 illusttrates GradCAM+ visualizations (Chattopadhay et al. 2018) for examples from the CUB-200 dataset, comparing the baseline model with JPEG-DL using ResNet-19 18 as our underlying model. In both instances, the baseline model incorrectly classifies the input, while the JPEG-DL model correctly classifies it. The visualizations highlight how JPEG-DL focuses more precisely

---

Published as conference paper ICLR 2025
(a) Baseline
(b) Input
JPEG-DL
Figure 5: Feature maps of size 56x56 shown after the first dense block in NenseNet-121 for both JPEG- DL and baseline models Figs. in inBaalada 55c respectively, using an original input shown Fbg The JPEG- DL model highlights the foreground (bird) more distinctly, while the baseline model shows less contrast, contributing to its misclassification.
(a) Baseline
(b) JPEG-DL
Baseline
(d) JPEG-DL
Figure 6: GradCAM+ visualization for baseline and JPEG-DL models on CUB200 using two examples, where the baseline model incorrectly classified them and the JPEG-DL model correctly classified them.
on the main object in the image, demonstrating the model's improved attention to key regions that contribute correct classification. This highlights the effectiveness of JPEG-DL in enhancing model interpretability and performance, and it also supports the case for background removal discussed in the previous subsection.
6 CONCLUSION
In contrast to the conventional understanding that JPEG compression negatively impacts the DL perfor- mance, this paper introduces a novel trainable JPEG compression layer into the DL pipeline to improve DL performance, termed JPEG-DL. This layer enables gradient-based optimization for quantization pa- rameters and can be integrated scamlessly with any underlying DNN to be trained jointly. To validate the effectiveness of JPEG-DL, significant gains over the standard DL. Additionally, we demonstrate that JPEG-DL improves the adversarial robustness of the learned model.
conduct extensive experiments six image classification datasets and show
10

---

Published as conference paper ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN203035-22, and by the Canada Research Chairs Program.
REFERENCES
Johannes Ballé, Valero Laparra, and Eero Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv:1611.01704 2016.
Tajeddine Benbarrad, Salaheddine Kably, Mounir Arioua, and Nabih Alaoui. Compression- based data aug- mentation for cnn generalization. In International Conference on Cybersecurity, Cybercrimes, and Smart Emerging Technologies, pp. 235-244. Springer, 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:/308.34:3 3432, 2013.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-basee based visual explanations for deep convolutional networks. 2018 IEEE winter conference applications of computer vision (WACV), pp. 839-847. IEEE, 2018.
Chyi-Tsong Chen and Wei-Der Chang. A feedforward neural network with function shape autotuning. Neural networks, 9()):227-641 1996.
Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg com- pression. ArXiv, fbss/7005.0290,, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Samuel Dodge and Lina Karam. Understanding how image quality affects deep neural networks. In 2016 eighth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2016.
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M. Roy. study of the effect JPG compres- sion on adversarial images. CoRR, abs/ 000800053, 2016.
Steven K Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint arXiv: arXiv:1902.08155: 2019.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge a neural network. arXiv preprint 10X1v 2(7), 2015.
11

---

Published as a conference paper ICLR 2025
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2017.
Forrest N landola. Squeezenet: Alexnet-level accuracy with 50x fewer parameters andi 0.5 mb model size. arXiv preprint arXiv: 2016.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on fine-grained visual calegorization (FGVC), volume 2. Citeseer, 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tnny images. 2009.
Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers mobilenet speed. arXiv preprint arXiv:2206.00 00.000001999 191, 2022.
Zihao Liu, Tao Liu, Wujie Wen, Lei Jiang, Jie Xu, Yanzhi Wang, and Gang Quan. DeepN-JPEG: A deep
neural network favorable Jased image compression framework. In Proceedings of the 55th Annual Design Automation Conference, pp. 18. ACM, 2018.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačić, Thomas Y
Hou, and Max Tegmark. Kan: Kolmogorov-armold networks. arXiv preprint arXiv:2404.19750 2024.
Xiyang Luo, Hossein Talebi, Feng Yang, Michael Elad, and Peyman Milanfar. The rate-distortion-acccaay tradcoff: Jpeg case study. arXiv preprint 200i::0008000000 2020.
Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings the European conference on computer vision (ECCV), pp. I 116- 131, 2018.
the
Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 -5088-350..00 IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
William B. Pennebaker and Joan L. Mitchell. JPEG Still Image Data Compression Standard. Kluwer Academic Publishers, Norwell, MA, United States, 1992. ISBN 0442012721.
Ahmed H Salamah, Shayan Mohajer Hamidi, and En-Hui Yang. A coded knowledge distillation framework for image classification based on adaptive jpeg encoding. Pattern Recognition, pp. 1 109666 2024a.
Ahmed H Salamah, Kaixiang Zheng, Linfeng Ye, and En-Hui Yang. Jpeg compliant compression for dnn vision. IEEE Journal on Selected Areas in Information Theory, 2024b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 45200, 2018.
12

---

Published as conference paper ICLR 2025
Richard Shin and Dawn Song. Jpeg-resistant adversarial images. In NIPS 2017 workshop on machine learning and computer security, volume 1, pp. 8, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 1ee111499.5555 1556, 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference Learning Representations, 2020.
Zhengzhong Tu, Peyman Milanfar, and Hossein Talebi. Muller: Multilayer laplacian resizer for vision. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6877-6887, 2023.
A Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. 2011.
Xiufeng Xie and Kyu-Han Kim. Source compression with bounded dnn perception loss for iot edge computer vision. In The 25th Annual International Conference Mobile Computing and Networking, pp. 1-16, 2019.
Xiufeng Xie, Ning Zhou, Wentao Zhu, and Ji Liu. Bandwidth-aware adaptive codec for dnn inference offloading iot. European Conference on Computer Vision, pp. 88-104. Springer, 2022.
Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In The Twelfth International Conference Learn- ing Representations, 2023.
En-Hui Yang and Shayan Mohajer Hamidi. Systems and methods for training deep learning models. 2024.
En-Hui Yang. Hossam Amer, and Yanbing Jiang. Compression helps deep learning in image classification. Entropy, 2887):881, 2021.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self- knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13876-13885, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.09441 2017.
Kaixiang Zheng, Ahmed H. Salamah, Linfeng Ye, and En-Hui Yang. Jpeg compliant compression for dnn vision. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 1875-1879, 2023. doi: 10. 10.1109//4P499399.000 2023, 10221982.
13

---

Published as conference paper ICLR 2025
A APPENDIX
A.1 PARTIAL DERIVATIVES OF Qa
With the CPMF Pa (-l2) shown in (6) is now quantized to each 2 e A with probability (a(2))). Denote this random mapping by
= Qp(z).
(9)
Note that given 2 is random variable taking values in A with distribution Pa (-1z).
A.1.1 QUANTIZATION STEP AND INPUT
The partial derivatives of Qa(2) w.r.. q and are obtained as
0Qa(z)
2a Var Var (Qp(2)},
(10a)
0z
0Qa(z)
1 (B(Q)(()) + (2az) Var
0q
q
(2a) -k2a)kkew, (22)skew,(Qpp()))
(10b)
where for any random variable V.
Skew (ow 2 Σ (v)) CE dPP(()) (E PPPP(()).
1.2 SCALING FACTOR a
The partial derivatives of Qa(Z) w.r.t. a obtained as
aQa(z)
0 EiEA (agl=))
du
da
Σ iq (iglz)) SEA
(11)
da
aP. (iglz)
(2 -2- i9)e -a(=-ig)) -0)=-gg)²
m1(2-19) Esea (2 (2-e)(=-0)00 50t uoe
dx
ΣJEA E ca(=-19)²
(ESEA aa(2--99)
Paligl))
E(2 (2- ( Pa(gll)) (2 -i9 (iglz))
(12)
jeA
Plugging (2) (1) yields
dQa(z)
Σ (iglz) Σ (2 jq) 2 (iql))) E 1o(z-iq) iq)z
da
iEA
jEA
iEA
=E(Q,(=))B((= Q)(2))²} - E((D((=)(= -3)))) =C)or(Q,(2), ()()))²}
(13)
14

---

Published as a conference paper ICLR 2025
To gain a better understanding, we fix q and b, and analyzed how aQ^(=) behaves for different values of 8a
From Fig. 7 it is evident that the gradient magnitude decreases as increases. This indicates that Qa approaches the shape of the uniform quantizer Qu. as shown previously in Fig. 2 leading the gradient magnitude for As result, when becomes sufficiently large, the gradient approaches zero, effectively preventing further updates to a, making act as non-trainable parameter at a certain point of the training process. Based this observation and verified experimental results, opt not to train for simple image classification tasks like CIFAR-100 and fine-grained datasets, as it does not impact the overall performance.
reduction
0.10
0.05
aQutz)
-0.10
Figure 7: This Figure illustrates the partial derivatives of Qo(z) w.r.t the scaling factor a, where b 3 and q= 1.
A.2 GRADIENT SCALING
Figures Sa and 8b demonstraeee the impact of varying a and q on the gradient magnitude of Qa(z) with respect q. As observed, decreasing the value of q while keeping constant leads to decrease in the gradient magnitude. Similarly, decreasing the value of a while maintaining a fixed also results in a reduction of the gradient magnitude. To address the instability in updating the trainable parameters Q, we propose utilizing
a to regulate the magnitude of aQa(=) as illustrated in Figure 8b By leveraging this control mechanism of adjusting a, we can stabilize the magnitude of the gradients that update This behavior suggests a relationship between and in controlling the gradient magnitude of Qa(z).
8q
To explore the relationship between a and q, we refer to the exponent in 5 a(z ig)² which can be rewritten by expressing cg. resulting in the form aq² (c-1) From this, we define new term, h ag referred to as the Gradient Scaling Constant. To further illustrate this relationship. in Fig. we set h = 2, and by selecting different pairs of a and values, we demonstrate that the maximum magnitude of 8Q.(=) remains invariant. This confirms that the gradient magnitude can be effectively controlled by adjusting based on the last updated value of y, according to the specified li value. This adjustment allows for controlled quantization updates, reducing the potential instability during training. Moreover, by controlling the gradient magnitude, we simplify the optimization process, enabling the use of a single learning rate for all q values by using the SGD optimizer, instead of the ADAM optimizer. This gradient scaling mechanism is analogous to the ADAM optimizer, which adapts different learning rates for individual trainable parameters based on momentum and recent gradient magnitudes.
8q
15

---

Published as conference paper ICLR 2025
400
400
200
aQulz
aQulz)
-200
-400
4400
75
50
25
25
50
75
50
25
0
25
50
75
(a)
(b)
Figure 8: Figures 8a and ?? illustrate the partial derivatives of Qa(z) w.r.t the parameter 9 for cases where a and q varies, and varies and q = 2, respectively. For both figures, we set b 6.
30
0- 2-2
0-2.-005
10
80
-10
20
30
-50
-25
2
Figure 9: Demonstrating impact of li on gradient magnitude for various combinations of q and a. We fixed f==2 and b=6.
A.3 CNN-BASED ARCHITECTURES SETTING
For CIFAR100, we deploy a stochastic gradient descent (SGD) optimizer with a momentum of 0.9, weight decay of 0.0005, and batch size of 64. We initialize the learning rate 0.05, and decay it by 0.1 every 30 epochs after the first 150 epochs until the last 240 epoch. For MobileNetV2. ShuffleNetVi and Shuf- fieNetV2, use learning rate of 0.01 this learning rate is optimal for these models in grid search, while 0.05 is optimal for other models.
For fine- ane-ieri classification tasks, all networks trained from scratch and optimized by SGD with momentum of 0.9, weight decay 0.0001, and an initial learning rate of O. The learning rate is divided by 10 after epochs 100 and 1,50 for all datasets, and the total epochs fine-grained classification tasks. We use the standard data augmentation technique for ImageNet (Deng et al.) 2009), i.e., flipping and random cropping. This experimental setup is also outlined Yun et al. (2020).
200. We set batch size 32 for these
16

---

Published as conference paper ICLR 2025
A.4 FINE-GRAINED MODEL ARCHITECTURES
We use standard ResNet- 18 with 64 filters and DenseNet-121 with growth rate of 32 for image size 224 x 224. For fine-gras gieeggraned classification tasks, we use PreAct ResNet-18 (He et al., 2016), which modifies the first convolutional layer with kernel size 3 X 3, strides I and padding 1, instead of the kernll size 7 x 7, strides and padding 3, for image size 32 x 32 by following Zhang et al. (2017). We use DenseNet-Be BC structure (Huang al. 2017), and the first convolution layer of the network is also modified in the same way as in PreAct ResNet- 18 for image size 32 x 32.
A.5 TRANSFORMER-DASEE SETTINGS AND RESULTS
this section, we compare the performance of IPEG-DL compared its baseline using the EfficientFormer- LI model (Li et al., 2022) on CIFAR-100 100, as well as two fine- tne-grained datasets, Flowers and Pets. We fol- lowed the experimental setup described by Xu et al. (2023), adhering the same configurations mentioned in Table 4 The learning rate was set to 0.003 for CIFAR-100 100 and 0.005 for the fine-grained tasks, aligning with our standard settings Section
Interestingly, we found that the best performance for monnsoomer-bbseld architectures was achieved when the quantization tables were initialized with all ones, effectively representing the highest quality factor quanti- zation table for JPEG. The top-1 validation accuracy performance is presented in Table S demonstrating the significant improvements achieved by JPEG-DL over the baseline.
Table 4: papee--rrameter setting on EfficientFormer-LI.
Settings
CIFAR-100
Pets
Flowers
batch size
512
512
512
warmup epochs
50
100
100
training epochs
300
600
600
Table 5: Top-1 accuracy (%) on CIFAR-100 and two fine-grained image classification tasks using EfficientFormer-.II. We report the mean and standard deviation of the experimental results over three runs with different random seeds. The best results are indicated in bold, and values in parentheses indicate relative accuracy gains over the baseline.
Model
Method
Accuracy (%)
Baseline
80,27=a3
CIFAR-100
DPEG-DL
80.49 1025 (+0.22)
Baseline
69.782a3
Flowers
JPEG-DL
73.05 (+3.27)
Baseline
65.52
Pets
JPEG-DL
69.28 (+3.76)
A.6 COMPARISON WITH MORE BASELINES
Table 6 we have included comparisons against other baselines that employed JPEG quantization in the pipeline, but with heuristic approaches to handle the non-differentiabiliy and derivative problems of JPEG quantization. For example, Ballé et al. (2016) allow optimization via stochastic gradient descent by replacing the quantizer with an additive i.i.d. uniform noise, which has the same width the quantization
17

---

Published as a conference paper ICLR 2025
bins, where å + (-0.5, 2===g*0000.,.0000 0.5). However, during validation, they employed Qu using the trained quantization tables. Shin & Song (2017) employ a third-order polynomial approximation of the rounding function to make JPEG differentiable. The gradient approximation through the round function a key aspect of certain neural network approaches, EEsse et al. (2019 | employ a straight-through estimator, originally proposed by [engio et al. (2013), to achieve this approximation. This method treats the round function as a pass-through operation during backpropagation, allowing for effective gradient estimation. Our experimen- tal results demonstrate that we consistently outperform all tested baselines. Even though these methods find a way to make the quantization differentiable, they still hurt the performance. This is consistent with the well-known wisdom in the literature, which also shows that they did not take advantage of the higher level of non-linearity introduced into DNN architectures, as proposed by JPEG-DL framework.
Table 6: Top-1 validation accuracy (%) on various fine-grained image classification tasks and model archi- tectures. We report the mean and standard deviation of experimental results over three runs.
Model
Method
CUB-200
Dogs
Flowers
Pets
Baseline
44.00214)
63.71gam
57.13g1
70.37
Bailéé el
2016
50.78
(.3.22)
53,47amm
(-10.24)
$5,461839
(-1.67)
56.14
(-14.23)
18
Shin & Song
2017
55.34
(+1.34)
63.03
(-0.68)
55.78
(-1.35)
71.45+10:
(+1.08)
Esser et
2019
51.58
(-2.42)
60.45
(-3.26)
58.04=058
(-0.91)
68.81=0.5
(-1.56)
JPEG-DL
58.81
(+4.81)
65.57=037
(+1.8)
68.76
(+11.63)
74.84=066
(+4.47)
Baseline
$7.70
66.61
51.324057
70.26*m79
Ballé et al. 2016
52,00=141
(-5.70)
60.07+641
(-6.54)
46.60#257
(-4.72)
61,91=1a8
(-8.35)
DenseNet-121
Shin & Song
57.19
(-0.51)
66.90=0.1
(+0.29)
51.04
(-0.28)
69.95=121
(-0.31)
Esser et 2019
56,462200
(-1.24)
64.892am
(-1.72)
55.981024
(+4.60)
69.58-0.9
(-0.68)
JP
61.32a041
(+3.62)
69.672055
(+3.06)
72.22
(+20.90)
75.90
(+5.64)
A.7 LAYER REPLACEMENT
We extend our evaluation of the JPEG-DL framework to include layer replacement, where substitute the ReLU activation function with our JPEG layer in ResNet-18. This replacement is implemented directly after the first convolution layer, which has 64 kernels and outputs feature map of 1 112x 12. For this layer replacement, we did not perform any color space conversion and added a quantization table with size of 8x8 for each kernel output. We follow the experimental setup mentioned in Section except we use a learning rate of 0.001, equal to 2. and b equal to 6 bits. The performance of this approach is shown in the last row of the following table and compared to the performance of JPEG-DI when the JPEG layer is placed directly after the input layer shown in Fig. [ The results shown in Table confirm that this new architecture, resulting from layer replacement with the JPEG layer or inserting the JPEG layer directly after the Input Layer, indeed improves deep learning performance by significant margin, thereby showing the potential of new forms of nonlinear operation in DNN architectures.
A.8 COMPARISON WITH JPEG-BASED DATA AUGMENTATION AND OTHER PREPROCESSING METHOSS
JPEG-based Data Augmentation. We have compared and implemented JPEG-based data augmentation across three different sets, each with varying ranges of quantity factor (QF). For each tested range, we ran- domly select a QF for each image within the mini-batch. These sets have been tested on fine-grained datasets
We followed the experimental setup defined by Esser et al. 2019) using Stochastic Gradient Descent (SGD) with initial learning rate of 0.01, adhering to the same learning rate decay schedule as the underlying model. Additionally, we initialized Q by setting it 2|Q|/v22 1 and implemented gradient scaling during training mentioned in their paper. We also maintained consistency by using the same number of bits used by JPEG-DL.
18

---

Published as conference paper ICLR 2025
Table Top-1 validation accuracy (%) on various fine-grained image classification tasks and model archi- tectures. We report the mean and standard deviation of experimental results over three runs.
Method
CUB-200
Dogs
Flowers
Pets
JPEG-DL (Input Layer)
58.81 12.12 (+4,81)
65.57
(+1.86)
68.76 (+11.63)
74.84 (+4.47)
JPEG-DL (1tt Conv Luyer)
59.27 (+5.27)
65.33+00r (+1.62)
72.10+1.6 (+14.97)
76.11+037 (+5.74)
for all the models evaluated. Table shows corresponding results for these tested sets. Consistent with the common knowledge in the literature, JPEG-based data augmentation general degrades the accuracy per- formance.
Non-Learnable and Learnable Preprocessing Methods. We have tested the performance of non-lcarnable and learnable preprocessing methods during training and validation across all tested models all tested fine- grained datasets, as demonstrated in T For non-learnable preprocessing, we have considered applying denoising using a Gaussian kernel and histogram equalization for each image within a mini-batch, As for the learnable one, we have compared it with Tu et al. (2023) learnable resize module, which has bandpass nature in that learns to boost details in certain frequency subbands that benefit the downstream recognition models. We have applied the same setup mentioned in their paper, in which they fine-tuned the model to achieve some improvement
Impact of chrominance subsampling JPEG-DL. In Table 8 evaluate various subsampling schemes for all tested models across all tested fine-grained datasets. Notably, although there no clear winner among different chroma subsampling methods, DNNs indeed respond color information differently from human; color information is more important to DNNs than human since chroma subsampling formats 4:4:4 and 4:2:2, in general, give rise to better accuracy performance than the 4:2:0 subsampling format which is adopted predominantly in the image and video coding.
A.9 FIXING HYPERPARAMTER, ACROSS DIFFERENT DATASETS
In Table evaluate the difference between the hyperparameters used for CIFAR- 100 and the fine-graid fine-grained task, where the only difference lies the learning rate. In the following setup, we present the results when learning rate of 0.003 is applied the fine-grained dataset. We observe that the performance difference is marginal, and in some cases, we can even achieve higher gains in model performance. This suggests that the choice of learning rate can have significant impact on the effectiveness of the model.
A. SENSITIVITY AND Q TABLE INITIALIZATION
Figure 10 presents the estimated sensitivity values for all models considered in the CUB200 dataset and a subset of models used for CIFAR-100, as proposed [Salamah al.) (2024b); Zheng et al. (2023). These estimated sensitivity values are used to initialize the quantization table for JPEG-DL. Figure illustrates the initial and converged values of the quantization table at the end of training.
19

---

Published as conference paper ICLR 2025
Table 8: JPEG-DL is compared to JPEG-based augmentation (orange), non-learnable (yellow), and learnable preprocessing (blue) methods. JPEG-DL employs three different chrominance subsampling schemes shown green color. Top-1 validation accuracy (%) on various fine-grained image classification tasks and model architectures. We report the mean and standard deviation of experimental results over three runs. The best results are indicated in bold, and values in parentheses indicate relative accuracy gains over the baseline.
Dataset
Method
ResNet-18
DenseNet-121
Baseline
54.00
57.70+0.4
Rand. QF
[1:50]
52.8610.
(-1.14)
55.72
(-1.98)
Rand. QF [50:100]
54,53204
(+0.53)
56.9820.0
(-0.72)
Rand. QF
[1:100]
5049
(-0.09)
57.20+0.3
(-0,50)
CUB-200
Denoising
54,82
(+0.82)
56.02
(-1.68)
Equalization
49.002031
(-5.00)
54.3220.5
(-3.38)
Learnable Resize
54.982a19
(+0.98)
57.35
(-0.35)
JPEG-DL (4:2:0)
58.00
(+4.00)
60.55
(+2.85)
JPEG-DL (4:2:2)
58.11
(+4.11)
61.51:0a4
(+3,81)
JPEG-DL. (4:4:4)
58.8
(+4.81)
61.32.044
(+3.62)
Baseline
63.71
66.61
Rand. QF
[1:50]
60.61
(-3.10)
64.9320.1
(-1.68)
Rand. QF
[50:100]
63.14:030
(-0.57)
66.97=a11
(+0.36)
Rand.
[1:100]
62.12
(-1.59)
65.59
(-1.02)
Dogs
Denoising
62.52.041
(-1.19)
66.3622034
(-0.25)
Equalization
62.38
(-1.33)
67.12=au1
(+0.51)
Learnable Resize
13.10
(-0.61)
67.90
(+1,29)
JPEG-DL (4:2:0)
65.57
(+1.6)
68.90=
(+2.29)
JPEG-DI. (4:2:2)
65.642416
(+1.93)
69.85 20
(+3.24)
JPEG-DL (4:4:4)
65.57+037
(+1.86)
69.67
(+3.06)
Baseline
57.13
51.32
Rand. QF
[1:50]
58.33
(+1.20)
$1.442009
(+0.12)
Rand. QF
[50:100]
55.98
(-1.15)
51.67=1100
(+0.35)
Rand. QF
[1:100]
57.55
(+0.42)
52.32
(+1,00)
Flowers
Denoising
57.282106
(+0.15)
50.37
(-0.95)
Equalization
60.56+a38
(+3.43)
61.72#o21
(+10.40)
Learable Resize
57.41
(+0.28)
+1.21
(-0.11)
JPEG-DI. (4:2:0)
67.5821.50
(+10.45)
68.01
(#16.69)
JPEG-DI. DL (4:2:2)
67.75
(+10.62)
68. 28.79
(+17.47)
JPEG-DL (4:4:4)
68.7620.57
(+11.63)
72.22
(+20.90)
Baseline
70.37
70.26
Rand. QF [1:50]
69.71
(-0.66)
68.93+0.00
(-1.33)
Rand. QF
[50:100]
70.001037
(-0.37)
68.6310.z
(-1.63)
Rand. QF [1:100]
69.52.0o2
(-0.85)
70.60=1.
(+0.34)
Pets
Denoising Equalization
69.15
(-1.22)
69.03
(-1.23)
73,00
(+2.63)
75.04
(+4.78)
I Leamable Resize
70.04
(-0.33)
68.83
(-1.43)
JPEG-DI (4:2-0)
74.64
(+4.27)
76.21
(+5.95)
JPEG-DL (4:2:2)
74.81
(+4.44)
76.17
(+5.90)
JPEG-DL (4:4:4)
74.84
(+4.47)
75.90=a0
(+5.64)
20

---

Published as conference paper ICLR 2025
Table 9: Top-1 validation accuracy (%) on various fine-grained image classification tasks and model archi- tectures. We report the mean and standard deviation of experimental results over three runs.
Model
Learning Rate
CUB-200
Dogs
Flowers
Pets
0.005
58.81
(+4.81)
65.57.0077
(+1.86)
68.761057
(+11.63)
74.8 74.84 (+4.47) 74.90 Losz (+4.53)
ResNet-18 18
0.003
58.62.050
(+4.61)
65.45 (+1.74)
69.61 (+12.48)
0.005
61.32 0.44
(+3.62)
69.67 10.58
(+3.06)
72.22 1
(+20,90)
75.90
(+5.64)
DenseNet-122
0.003
60.92:0500
(+3.22)
69.53 (+2.92)
72.45 2092
(+21.13)
75.83 (+5.56)
1e-8
1e-8
3
Channel Channel
Ychanne
Cb Channel
Chand
Cr Charndl
Senstivity
2
Senstivity
1
0
1
16
31
46
61
16
31
46
61
Frequency Index
Frequency Index
(a) ResNet18
(b) DenseNet-121
1e-9
1e
1e-9
charna
Ycharnal:
Channnl
7.5
Sensitivity
1.0
Sensitivity
Sensitivity
5.0
2.5
0.0
0.0
61
Frequency Index
Frequency Index
Frequency Index
(c) VGG13
(d) Res56
(e) ShuffleNeV2
Figure 10: The estimated sensitivity is shown for two pre-trained models the CUB200 (first row) and three pre-trained models on the CIFAR-100 (second row), with sensitivity indices arranged in the default zigzag order.
A.11 FEATURE MAPS VISUALIZATION
Figure 12 presents the feature maps extracted after the first dense block in DenseNet- 121 for both the JPEG- DL and the baseline models, trained on the Flowers dataset using the model from Table 2 The shown example was incorrectly classified by the baseline model, while the JPEG-DL model correctly classified it.
x
We have verified the conrectnsss of our PyTorch implementation with the authors' TonsorFlow impplmmnttinnnnllnnnttttitioo pro- vided in https://colab... rws.arch.cc........... btoo//aaterr/mllle////leee... http:///www.....ooooooooo//oooo.
21

---

Published as conference paper ICLR 2025
10
Irital Trained
15
Trainod
Steps
Steps
10
4
16
31
46
61
1
16
31
46
61
Frequency Index
Frequency Index
(a) DenseNet-121 (Y Channel)
(b) DenseNet- 121 (CbCr Channel)
10
Traine Iniatt
Trained in
8-
15
Steps
Q
4-
2
5
16
31
46
61
Frequency Index
Frequency Index
Res56 Channel)
(d) Res56 (CbCr Channel)
Figure 11: Initial and final quantization tables for Res56 trained on CIFAR-100 and DenseNet- 121 trained on CUB200.
A.12 MITIGATING THE INFERENCE OVERHEAD OF DISTERENTIABLE SOFT QUANTIZERS
The reconstruction space Á is configured with a specific size, determined by the parameter T., which is set to 2--1, as mentioned in Section 4 For CIFAR-100 100 and Fine-grained datasets, is set to 8 bits, resulting in reconstruction space length of 513. For ImageNet- IKK 6 is set to 11 bits, resulting in length of 2047. This high dimensionality of the reconstruction space poses a computational challenge during inference when calculating the conditional probability mass function (CPMF) (-(+))) To address this, the support of each CPMF is restricted to the five closest points Á the coefficient 2 being quantized. This simplification, referred to as the masked CPMF, significantly reduces computational complexity without compromising the effectiveness of the quantization scheme Qa.
Figure 13 demonstrates the rapid decay of probability mass in CPMFs as the reconstruction level deviates from the quantized value the uniform quantizer (z. The set M, consisting of five reconstruction levels centered around Qu (=), captures significant portion of the total probability mass, even in extreme cases where the quantization step q and the parameter a are very small. This observation suggests that restricting the CPMF support to M (masked CPMF) is valid simplification. Instead of computing the full-space CPMF with length 2L + 1, we can efficiently calculate the masked CPMF with length using softmax on M. The impact of this simplification on the quantization scheme Qa is negligible, particularly for high values of a, as shown in Table TO Therefore, using the masked CPMF our implementation is justified simplification that significantly reduces computational complexity without compromising accuracy.
22

---

Published as conference paper ICLR 2025
(a) Baseline
(b) Input
(c) JPEG-DL
Figure 12: Feature maps of size 56x56 shown after the first dense block in DenseNet-12 121 for both JPEG- DL and baseline models in Figs. 12a and 12c respectively, using an original input shown in Fig. [26] The JPEG-DL model highlights the foreground (flower) distinctly, while the baseline model shows less contrast, contributing to its misclassification.
Qa(2 = 0.5)
Full Space
Masked
0.5
0.5027
0.0027
3
0.5
0.5
5.96e-08
0.5
0.5
5.96c-08
0.5
0.5
Table 10: Qa(2 0.5; a.) values resulting from the full-space CPMF and the masked CPMF different a, alongside the absolute difference A between the full-space and masked cases.
Inference time (milliseconds)
Inference Throughput (img/sec)
Inference Settings
Baseline
5,46
180.65
JPEG-DL (Masked CPMFs)
7.43
133.79
JPEG-DL (Full Space)
18.86
52.37
Table 11: Comparison of inference time between different inference settings measured for Resnet18 on ImageNet, following the experimental setup mentioned in Section Evaluation is conducted on single NVIDIA RTX A5000 GPU.
Specifically, Fig. 13 illustrates how fast the probability mass of reconstruction level & decays in CPMFs as å deviates from the uniform quantizer (2).). It's clear that the set M = (Qa(z) 2g. Qa(2) 4. (2), Q, (z) q, (z) + 2q) of reconstruction levels retain total probability mass close to even
23

---

Published as conference paper ICLR 2025
Probabiitt Mass
Prabahility Mass
R.3
0.3
Reconstruction Level
Reconstration Lovel
(a)
(b) a 3
4.5
05
Probokiiit Maxs
Probobiity Maos
10- Yeats
Roconstructin
Reconstraction Lovel
a=
(d) 10
Figure 13: Partial visualization CPMFs Pa(-1z = 0.5) computed using 6 with == 1023, q == 1. and values of (a) I, (b) 3, (c) 5, and (d) 10. The reconstruction levels between two red dashed lines represent M.
an adversarially chosen scenario where both q and a are extremely smal (i.e., 1) and the coefficient C' being quantized lies exactly at quantization threshold. Therefore, instead of computing the full-space CPMF with length 2L + following (6), we only conduct softmax on M to get the masked CPMF with length The error in Qa caused by this masking is negligible shown in Table TO especially in the high a case. Therefore, it's totally justified to simplify the full-space CPMF to the masked CPMF in our implementation.
Table presents a comparison of inference time and throughput between the standard model and our unified model, which incorporates the JPEG layer with the underlying model. The inference times for both the full- space CPMF and the masked CPMF are measured. The reported inference time is averaged across the entire validation set of ImageNet using Resnet18 88 from Table These results demonstrate the effectiveness the proposed approach in addressing the computational complexity of the JPEG-DL framework.
As g and increase, the CPMF becomes even sharper, and result, M will capture incrcasingly higher proba- bility mass.
24


Reviews:
[{"review_id": "oKSKBIEca6", "reviewer": "Reviewer_38Fe", "summary": "In this paper, the authors study the impact of JPEG compression to the performance of deep learning, and propose a JPEG-inspired deep learning framework. For that, they present a differentiable soft quantizer to train the JPEG layer. Experiments show that the proposed method increases the accuracy by almost 21%.", "questions": "please see the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall, the paper is written clearly, and can be easily understood.\n2. A variety of experiments are conducted to verify the effectiveness.\n3. The used technique seems sound, although I don’t check it in detail.", "weaknesses": "1. I don’t think the problem studied in this paper is very important in the community. Usually, the images fed into deep learning have been precessed by the fixed JPGE compressor, and we don’t have the chance to modify the process like that in this pager, so the actual application value is limited. In addition, I see the paper is an improvement for the method in Yang 2021 (Entropy), which is not followed by many researchers.  \n2. Now lots of papers have shown the vision transformer and large model are effective in many computer vision tasks, I don’t know whether the proposed method can adapt to these new networks. The authors should discuss that. \n3. Although a variety of experiments are conducted, the compared method is only one baseline, which can not show the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZzJeNiywI3", "reviewer": "Reviewer_BiPm", "summary": "This paper proposes jointly optimizing both JPEG quantization operation and a DNN to achieve greater effectiveness. A trainable JPEG compression layer with a novel differentiable soft quantizer are proposed. Extensive experiments validate the effectiveness of the proposed method.", "questions": "1. How to conduct datasets for experiments? For ImageNet-1k, the images are already compressed by JPEG. In traditional deep learning framework, the images are loaded and decoded by JPEG. How to insert the encoding operation in this framework?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To make JPEG trainable, a differentiable soft quantizer is proposed. It works well with JPEG. Overall, this paper makes JPEG trainable which is significant contribution. Because many frameworks equipped with JPEG can be trained by the differentiable soft quantizer.\n2. A novel DL framework that prepends any underlying DNN architecture with a trainable JPEG compression layer is proposed. Experiments show it can improve the accuracy significantly with only 128 parameters.\n3. This paper enjoys a good writing.", "weaknesses": "1. It is better to make a comparison for the latency. The speed of the model is also important to report.\n2. Only image classification is considered. The proposed method is better to be validated on more tasks, such as object detection and segementation. \n3. Hyperparameters are tuned differently on different datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgXQjj4EEU", "reviewer": "Reviewer_FTRb", "summary": "This work proposes a new training framework for deep learning models, JPEG-DL, utilizing an image compression module to improve the model performance. To this end, a learnable JPEG-based image compression layer is introduced with a differentiable soft quantization method and is trained jointly with a main model. At test time, a compressed image from the compression layer is fed into the model. The experimental results show consistent performance improvements of existing classification models on various benchmarks with higher robustness on adversarial attacks.", "questions": "- Is the target of the adversarial attack in Figure 3 the whole model including the JPEG layer?\n- There appears to be more room for exploration regarding preprocessing modules for performance enhancement. For example, what if deep learning-based image compression models were used? What about a simple autoencoder?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The approach of leveraging image compression to improve pure performance of a model is interesting.\n- The paper is well-written and it is easy to follow.\n- A variety of network architectures and datasets are used in the experiments.", "weaknesses": "Major concerns:\n- A comparison to training with JPEG-based data augmentation is required to validate that the proposed method provides benefits beyond simple data augmentation using JPEG. \n- There is no baseline for the differentiable quantizer. For example, comparisons could be made with methods such as the straight-through estimator (i.e., using the identity function as a gradient function) or additive uniform noise [1].\n- The baseline for the image preprocessing method for training is insufficient; only comparison results with the vanilla models are presented. For stronger persuasiveness, comparison results with other learnable or non-learnable preprocessing modules are needed.\n- In L199, it is mentioned that the differentiable soft quantizer is adapted from Yang & Hamidi (2024), but it seems unclear what that exact referenced paper is. Is the patent (https://patents.google.com/patent/US11461646B2/en) the correct source?\n- The analysis on the significant difference between performance improvements across different datasets is needed. For instance, in Table 2, what accounts for the notable performance improvement in fine-grained tasks (especially the Flowers dataset)? Additionally, why is there few performance gain in the ImageNet results in Table 3?\n\nMinor concerns:\n- The empirical study is limited to classification tasks. \n- In Table 4, the bits per pixel (bpp) is excessively high, making it incomparable to typical lossy compression methods. While empirically showing the possibility of compression, it does not seem particularly convincing.\n- The proposed method requires additional computation for encoding and decoding of an image, but the time complexity is not investigated.\n- A typo in L292 \"we fixe\".\n\n[1] Ballé et al., End-to-end Optimized Image Compression, ICLR 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyRNRpaNDW", "reviewer": "Reviewer_2zCL", "summary": "The paper introduces a trainable JPEG inspired layer to neural networks. The new layer performs the linear JPEG steps of RGB to YCbCr and DCT with the non-differentiable quantization step replaced with a learnable quantization proxy finishing with the linear IDCT and YCbCr to RGB.  The learnable quantization proxy uses a soft-max with a tunable parameter which controls how close the proxy matches true quantization. The quantization step size is a learnable parameter in this layer. The paper shows how including this layer as the first layer of neural network architectures can improve their accuracy on different tasks.", "questions": "* Could we see even better results if the JPEG layer was included periodically? \n    * What if *all* nonlinearity was replaced with the JPEG layer?\n* Please clarify how $\\alpha$ was used in experiments\n* What about chroma subsampling?\n* Why do the authors think this layer helps?\n\n## Update After Discussion\n\nAfter discussion with the authors I am raising my rating\n\nThe authors did a great job responding to the concerns of myself and fellow reviews and went above and beyond on additional experiments which strengthen the case for this paper quite a bit. I specifically have to call out the layer 1 non-linearity experiments that the authors conducted on a very short turnaround that shows additional gains for the JPEG layer. Given the impact of this result I have to conclude that there is indeed something fundamental about using a JPEG inspired layer as a non-linearity which could have repercussions on the broader field. As I stated in a comment, many view JPEG as something incidental; a specific way of storing images. But the core idea of JPEG is to re-weight frequency bands based on their importance. We know from several studies (Maiya et al. [1] for example) that such re-weighting affects neural networks much as it does humans and this paper gives an actionable method for capturing this phenomenon.\n\n1. Maiya, Shishira R., et al. \"Unifying the Harmonic Analysis of Adversarial Attacks and Robustness.\" BMVC. 2023.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this is a very interesting paper with an unintuitive result. As the authors point out (ln 30) the conventional wisdom is that JPEG compression removes information from an image and should only hurt neural network accuracy. However as this paper, and some prior works, show that is not necessarily the case. This paper builds significantly on prior works by showing not only that JPEG compression can be mitigated, but that it can actually be a large component of a neural networks success and proposing a method for achieving this. The differential soft quantizer is something which may be useful in many different applications, potentially being a better option that the addition of noise or a straight-through gradient as it more accurately models the information loss. Finally, the results show a clear improvement when incorporating the method.", "weaknesses": "While the appendix was fairly comprehensive with additional results there are a few additional things that I would have liked to see. The first is that the paper only tests the JPEG layer as the first layer of the architecture, there could have been more experiments in layer placement that would have been really interesting to see. It also wasn't immediately clear to me how $\\alpha$ was being set in experiments, I understand that there is a derivation of $\\frac{\\partial}{\\partial\\alpha}$ but is that parameter actually trained and if so how was the gradient magnitude controlled? There is some discussion of this from ln 689 but it was a little unclear if $\\alpha$ was fixed or not. One thing missing from the JPEG step was chroma subsampling: another non-linearity. Was this considered? It would be fascinating to see if neural networks respond to missing color information similarly to humans.\n\nLastly, and maybe most importantly, there was little discussion of *why this couterintuitive results holds*. While many view JPEG as something incidental the core idea of JPEG to isolate important information based on frequency bands. My take on the results presented here is that the learnable layer is essentially filtering out information which is irrelevant for the networks task, but I would love to hear the authors take on it. Perhaps such analysis could lead to a more direct approach? (For example: a layer which only filters frequencies or which alters the color channels, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)