Differentiable JPEG-based Input Perturbation for Knowledge Distillation Amplification via Conditional Mutual Information Maximization
Authors: SIYU CHEN
Kaixiang Zheng
Ahmed H. Salamah
EN-HUI YANG
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
JPEG
Conditional Mutual Information
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Maximizing conditional mutual information (CMI) has recently been shown to enhance the effectiveness of teacher networks in knowledge distillation (KD). Prior work achieves this by fine-tuning a pretrained teacher to maximize a proxy of its CMI. However, fine-tuning large-scale teachers is often impractical, and proxy-based optimization introduces inaccuracies.
 To overcome these limitations, we propose Differentiable JPEG-based Input Perturbation (DJIP), a plug-and-play framework that improves teacher–student knowledge transfer without modifying the teacher. DJIP employs a trainable differentiable JPEG layer inserted before the teacher to perturb teacher inputs in a way that directly increases CMI. We further introduce a novel alternating optimization algorithm to efficiently learn the coding parameters of the JPEG layer to maximize the perturbed CMI. Extensive experiments on CIFAR-100 and ImageNet, across diverse distillers and architectures, demonstrate that DJIP consistently improves student accuracy-achieving up to 4.11% gains-while remaining computationally lightweight and fully compatible with standard KD pipelines.

Published as a conference paper at ICLR 2026
DIFFERENTIABLE JPEG-BASED INPUT PERTURBA- TION FOR KNOWLEDGE DISTILLATION AMPLIFICA- TION VIA CONDITIONAL MUTUAL INFORMATION
MAXIMIZATION
Siyu Chen", Kaixiang Zheng", Ahmed H. Salamah" & En-Hui Yang Department of Electrical and Computer Engineering, University of Waterloo {s875chen, k56zheng, ahamsalamah, ehyang]\uwaterroo.co
ABSTRACT
Maximizing conditional mutual information (CMI) has recently been shown to en- hance the effectiveness of teacher networks in knowledge distillation (KD). Prior work achieves this by fine-tuning a pretrained teacher to maximize a proxy its CMI. However, fine-tuning large-scale teachers is often impractical, and proxy- based optimization introduces inaccuracies. To overcome these limitations, we propose Differentiable JPEG-based Input Perturbation (DJIP), -uug-addpllay framework that improves teacher-student knowledge transfer without modifying the teacher. DJIP employs trainable differentiable JPEG layer inserted before the teacher to perturb teacher inputs in way that directly increases CMI. We further introduce novel alternating optimization algorithm efficiently learn the coding parameters of the JPEG layer to maximize the perturbed CMI. Extensive exper- iments on CIFAR- 100 and ImageNet, across diverse distillers and architectures, demonstrate that DJIP consistently improves student accuracy-achieving up 4.11 % gains-while remaining computationally lightweight and fully compatible with standard KD pipelines.
INTRODUCTION
Knowledge distillation (KD) (Buciluă al., 2006 Hinton al.) 2015) has emerged as a pivotal technique for model compression, enabling the transfer of knowledge from large teacher models to lightweight student networks. This approach significantly improves student model performance without incurring high computational cost. Under resource constraints, KD is often simpler to apply and more robust in preserving accuracy compared to other compression techniques such as prun- ing (Sun et al. 2024a) and quantization (Lin al. 2024), especially when deployment simplicity is a key concern.
Since the seminal work of (innonn et al. (2015), extensive research has sought understand the
underlying mechanisms of KD) (Phuong & Lampert 2021 Mobahi et al. 2020 Allen-Zhu & Li 2023 Dao al.) 2021). and develop
effective distillation techniques (Peng al.) 2019 Romero et al. 2014]] Zhao et al., [0222, Zheng & YANG 2024). However, in those conventional KD
methods, the teacher model is typically trained solely minimize its cross-entropy (CE) loss, with
little attention paid to its ability to provide an informative supervision signal to student models.
To address this issue, student-orienter teacher Cho & Hariharan 2019 Wang et al. 2022 Tan & Liu [0244] Yang et al., 2019; Dong et al., [024]] Ye et al.) 22024 Hamidi et al.] 2024) have been proposed yield softer, more informative supervision signals. Notably, MCMI (Ye al.. 2024) demonstrates that maximizing conditional mutual information (CMI) during teacher training im- proves distillation effectiveness. However, these methods require modifying the teacher's weights, an impractical constraint in many real-world scenarios where the teacher is fixed proprietary.
This retraining limitation has prompted an alternative line research introducing perturbations at the input level. In conventional KD pipelines, both the teacher and student consume the same input
Authors contrihuted equally.

---

Published as a conference paper at ICLR 2026
fu,
Compressed
Raw Image
Differentiable JPEG Layer
Image Ky
f
fir
Raw Image
Lange
Compressed Image X,
Coded Teacher
Differentiable JPEG Layer
fer
Coded
Training Stage
Distillation Stage
RGB-to-YCbol Color
YChCr-to-RGB Color Space Conversion
*
Space
Differentiable Soft Quantizer Q4
Blockwise DCT
Blockwise IDCT
Ja(x: w) Jx(x: Q.a)
Differentiable JPEG Layer
Figure 1: The overall training framework of DJIP is shown, centered around differentiable JPEG coded teacher module (blue), which integrates a differentiable JPEG layer (orange). The process involves two stages: differentiable JPEG layer training and student distillation, with trainable pa- rameters highlighted in red.
images, which may restrict the teacher's ability to transfer its full representational knowledge to the student. Recent works (Heo al. 20188| Nguyen- Duc al., 2023 Zhang et al., 202I) suggest that feeding students with adversarial or divergent inputs generated by teachers can enhance the effi-
ciency of knowledge transfer. Coded Knowledge Distillation (CKD) (Salamah et al. 2025a) further proposes that adaptively compressed input images can enhance the teacher's distillation efficacy. However, these approaches often incur high costs due to the need for generating additional samples, thereby increasing distillation complexity.
To overcome these limitations while further leveraging the benefits of input perturbation, we pro- pose Differentiable JPEG-based Input Perturbation (DJIP). DJIP employs trainable differentiable JPEG layer inserted before the teacher to perturb imputs in a way that directly increases CMI. The combination of the differentiable JPEG layer and the teacher is referred to differentiable JPEG coded teacher. This setup enables end-to-end learning of the coding parameters of the differentiable JPEG layer tailored maximize the perturbed CMT of the teacher. Unlike MCMI, which fixes class- wise centroids during CMI maximization, potentially sacrificing the precision of the CMI proxy, we propose novel alternating algorithm that reformulates the perturbed CMI maximization objective into a double minimization problem. This algorithm allows the centroids to be dynamically updated each iteration, resulting in more stable and effective training.
To demonstrate the effectiveness of DJIP, we conduct extensive experiments on two datasets, cover- ing both same- and cross-arces coos-architecctre distillation, and including both CNN and ViT models. Results consistently demonstrate the orthogonality of DJIP over various KD pipelines, including MCMI, and superiority over CKD. Specifically, DJIP can improve student Top-1 accuracy by up to 4.11%.
Our contributions are summarized as follows:
Differentiable JPEG-based Input Perturbation: We propose Differentiable JPEG-based Input Perturbation (DJIP), plug-and-pla framework that improves teacher-student knowledge transfer without modifying the teacher. DJIP employs trainable differentiable JPEG layer inserted before the teacher to perturb teacher inputs in way that directly increases CMI. During distillation, this framework can achieve significantly lower computational overhead while maintaining comparable functionality.
Alternating Algorithm for Maximizing the Perturbed CMI: We further introduce a novel al- ternating optimization algorithm to efficiently learn the coding parameters of the JPEG layer to maximize the perturbed CMI. The algorithm works by iteratively updating class centroids and the coding parameters.
2

---

Published as a conference paper at ICLR 2026
Comprehensive Empirical Evaluation: We extensively evaluate DJIP across diverse datasets and model architectures, demonstrating its generalizability, consistency, effectiveness, orthogonality to existing methods, and compatibility with wide range of knowledge distillation pipelines.
2 RELATED WORKS
A detailed review of related works, including conventional KD, KD with student-oriented teacher, and KD with input perturbation, is provided in Appendix A.I1 where we also outliee the most relevant directions for positioning our method and the advantages of our approach.
3 NOTATION AND PRELIMINARIES
3.1 NOTATION
Denote the i-th element of a vector p as p[i]. For positive integer K, let [K] _ KL.....K. For multi-class classification task, assume that there are class labels with [C] as the set of class labels.
Also, a C-dimensional probability simplex is denoted by AC for C > 1. The cross entropy of two probability distribution P1, P2 E A is defined ( P2) Σ
Kullback-Leibler (KL) divergence is defined as (P Pb) = E. c=1 -P] In Palc), and their
C
Pi Aic Pate]
For any pair of random variables (Xx, Y). denote its joint probability distribution by Px,r (x, y) or simply P(x, y) whenever there is no ambiguity, the marginal distribution of Y by (y), the conditional distribution of Y given X = x by P-jx(- x), and the expected value with respect to X by Ex (.). The conditional mutual information (CMI) of X and Y given a third random variable Z is T(;; Y 2) = H(X 2) H(X 22
3.2 CONDITIONAL MUTUAL INFORMATION OF DNNS
A classification DNN with C' classes can be viewed mapping + fo(x), where E Rd is an input image, 0 denotes the model parameters, and fe(x) E A is the output class probability distribution in response to Whenever the context is clear, we omit 0 and simply write f(x). Let denote the label predicted by the DNN in response input x with probability (x)) and y denote the ground-truth label of The set of distributions f(x) in A corresponding to all input samples with the same ground-truth label y forms cluster in A (referred to as the y- cluster). Now let (X,Y) be a pair of random variables representing a random input sample and its corresponding ground-truth label. Feed X into the DNN and let Y denote the corresponding predicted label. As shown in Yang al. (2025), Pipxr x. y) = [i], and the CMI between X and Y given y can be computed as:
X Y then forms a Markov chain with
(f (i=i x In Prixy (8 = x.
Y Σ (x vv)
Σ
P = =
x
=xiy (())(s Y v)
where S, = ( (v) Ex ( y
X(x; Y = v) measures the concentration of the y-cluster, and Sy can be viewed as the centroid of the y-cluster. Averaging over all such clusters, we obtain I(X; Ý Y), which reflects the average predictive concentration across all classes:
(X)
Σ Pr(w))(;; Y v) Ex,y [Dxx((((X]]Ssv)) vE[C]
For a training set D = (1, 2 drawn from an unknown distribution Px,,, we can approximate the CMI of model ff by empirical value. To be specific, let D, = {a, E D :::00 yJ = y). Denote the size of Dy by (Dyl- The empirical values of CMI can be calculated follows:
T(X; Y) = Σ Σ (x((f( VE[C] E
where S, [Pyl ayED,
Σ f(z,), for y E [C]. (3)
3

---

Published as a conference paper at ICLR 2026
3.3 DIFFERENTIABLE JPEG LAYER
JPEG (Pennebaker & Mitchell, 1992) is one of the most widely adopted lossy image compres- sion standards in real-world applications. It achieves compression by exploiting spatial redundancy and perceptual irrelevance in natural images, resulting in high efficiency for storage and transmis- sion. The standard JPEG pipeline first converts an RGB image into YCbCr, partitions into non-overlapping 8 blocks, and applies the Discrete Cosine Transform (DCT). The resulting coef- ficients are quantized uniformly using tables Q and entropy-coded (e.g.. .. Huffman coding). Recon- struction reverses this process via dequantization, inverse DCT, and conversion back to RGB. For simplicity, we denote the uniform quantizer as Qu.
To increase DNN nonlinearity, the JPEG-DL framework proposed by (Salamah al. 2025b) intro- duces a differentiable JPEG layer into any underlying DNN. Specifically, JPEG-DL incorporates a novel differentiable JPEG layer the input layer of the underlying DNN architecture. This layer simulates the standard JPEG codec but replaces the non-differentiable 2u with a differentiable soft quantizer, denoted as 2a This quantizer, parameterized by quantization step size q E Q and a sharpness parameter a E ax, approximates Q, via a smooth expectation over quantization bins, thereby enabling end-to-end gradient- based optimization. For simplicity, we denote the entire dif- ferentiable JPEG layer as Ta, whose structure is illustrated in Fig. Hence, the reconstructed image given by i. [Z, w), where w denotes the trainable parameters (Q.
In JPEG-DL, the Ja is regarded as part of the overall DNN architecture. During training, the quanti- zation (i.e., coding) parameters and the underlying DNN weights are jointly optimized to minimize the standard CE loss. In our work, however, the differentiable JPEG layer is used as a mechanism to perturb the input to the teacher and is separated from the teacher. Furthermore, during training, only the quantization parameters are optimized to maximize the perturbed CMI of the teacher with the teacher frozen completely.
4 METHODOLOGY
In this section, first present the overall DJIP framework, then detail the objective function use, and finally introduce our novel alternating optimization algorithm.
4.1 OVERALL FRAMEWORK
As illustrated in Figure ! the overall DJIP framework consists of two stages;
Differentiable JPEG Layer Training: The JPEG layer first perturbs the input image æ into ã.. Ja(zz, w), which is then fed to the teacher model. Under the objective function proposed in the next section, which balances the CE loss and the DJIP loss introduced later, the JPEG coding parameters are optimized to maximize the perturbed CMI.
Student Distillation: this stage, the trained JPEG layer is integrated into the standard KD frame- work to perturb the teacher's input images. With its input perturbed by the trained JPEG layer to increase the perturbed CMI, the teacher can provide more informative supervision signals to student.
4.2 OBJECTIVE FUNCTION
With reference to Figure 1 given Ja. X , is a deterministic function of X. Hence, the variables form a Markov chain
X + X., + Y, which implies
I(x; & YY) 1(XX (
|r) I &
(4)
We refer to I(Xw: Y) as the perturbed CMI. Our goal is to perturb the input X such that the perturbed CMI is maximized to a certain extent, while simultaneously minimizing the CE loss with respect to the ground-truth labels. Following the spirit of Ye et al. 2024), we formulate our CE-CMI optimization problem over w as
min { Ex
A A(X.: (1)).
(S)
4

---

Published as a conference paper at ICLR 2026
where À > 0 is hyper-parameter that balances the CE-CMI trade-off. In contrast to Ye et al. 2024), where optimization is carried out over the pretrained model parameters 8, the DJIP frame- work freezes the teacher and updates only the JPEG parameters
However, as stated by Ye al. 2024), maximizing I(x: Y Y) poses certain challenges because the term Sy depends on f(z)), v., E D,, (see Equation 3) which is not well-suited for numerical solutions and cannot be efficiently parallelized on GPUs. The same challenges apply the optimiza- tion problem in Equation 5 as well. Ye et al. (2024) circumvents this issue by fixing the centroids Sy obtained from a pretrained model. Nevertheless, these centroids may shift during fine-tuning, so using fixed centroids may partially mitigate the issue, but does not fully resolve the theoretical limitations of their method.
To address the above challenges theoretically, we introduce dummy "backward channel" and re- formulate the optimization problem in Equation [ as a double minimization problem, as shown in the following theorem and proved in Appendix A.2
Theorem For any y E [C]. let Q i, y) denote dummy conditional distribution over the input space X ("backward channel")). Then for any X > O.
min [Ex] ((Xr)xx.((X))) >((x8; 1x)]
C
min min (QC16)))
((o) ((ww)
Σ P(., Σ [(t)) In
((()).)))))) 11.00) (6)
where the inner minimization above is achieved when (x v)
Q(c (i,)
( (
In practice, the training set is randomly partitioned into B mini-batches B for b [B], each of size [B]. When the joint distribution (X, Y)) is unknown, the objective function can be approximated by its empirical estimate over a mini-batch B. Accordingly, based on Theorem [] the empirical objective function Lemp for the proposed DJIP method can be formulated as:
La(A, w. (Q(1i.))))
Σ (x.
(-
Σ (5.9)
Σ (( (i
(8)
Com
4.3
AN ALTERNATING CMI MAXIMIZATION ALGORITHM
Based on equation 6 to equation S are ready present our algorithm for solving the opti-
mization problem in equation objective function in equation
which optimizes w and {Q(- i,y)) alternatively to minimize the
Step 1: Fix w, let Dr., {1t w/) D =
y)} can be updated in two steps. (1) update the centroids S, empirically according to Equation 3 and (2) calculate the empirical version of Q(z v, according Equation
5, x, =). According 7 {Q(-
[ = (il v) = [D,
1
Σ f(Ja(z,, w))[]) vi,ye(C,
(9)
EjED,
iPead [Pyl f(Ja(z,
Q(x (, (x (Jf((a w))ld
ED,EE()
(10)
(
( v)
Step 2: Fix (Q(- [.,)). be updated using standard deep learning process through stochastic gradient descent (SGD).
A detailed pseudo-code of the alternating optimization algorithm is provided in Appendix A.3
5 EXPERIMENTS
Terminologies. To evaluate the performance of DJIP, we conduct a series of experiments. This section presents the main experimental results and demonstrates the extent which DJIP improves
5

---

Published as a conference paper at ICLR 2026
accuracy state-of-the-arr KD methods. For clarity, denote the teachers trained solely with CE loss, with MCMI estimator (Ye et al. 2024), and with the proposed DJIP method as the CE teacher, MCM/ teacher, and DJIP teacher, respectively.
Plug-and- Nature, In all experiments reported, the JPEG layer functions as a lens to facilitate improved teacher distillation. Once removed, the model reverts to its conventional form without any residual effect. Furthermore, no hyperparameters of the underlying KD methods are modified; all configurations remain identical those used in the original benchmark settings.
We conduct extensive experiments on ImageNet and CIFAR-100 with diverse model architectures. Moreover, we show that DJIP can be effectively applied to poos-ppaaddigm distillation between CNNs and Vision Transformers (ViTs). The results further demonstrate that DJIP is complementary to existing techniques and remains orthogonal to the latest sate-of-the-aal benchmarks.
5.1 CIFAR-100 RESULTT
The CIFAR-100 dataset is a widely used benchmark for image classification, comprising 60,000 color images with a resolution of 32 x 32 pixels, categorized into 100 classes. Following the ex- perimental setup Tian et al. 2019), identical architectures (see Table 1 and 6 pairs using different architectures (see Table ( Each experiment is repeated across three independent runs, and the average accuracy is reported.
conduct experiments with 7 teacher-student pairs sharing
For a comprehensive comparison, we evaluate our DJIP teacher against the conventional CE teacher using state-of-the-ard distillation methods. These include logit-based approaches: KD (Hinton al., 2015). DKD (Zhao et al.) 2002). DIST (Huang et al. 2022). and WTTM (Zheng & YANG 2024); relation- approaches: CC (Peng et al. 2019) and RKD (Park al. 2019); and feature-
ap- J FT (Kim et al.) 2020). SP (Tung Mori (2019, ITRD (Miles et al.] 202T). CRD (Tian al. | 2019). and LSKD (Sun al.]
proaches: AT (Zagoruyko & Komodakis 2016), FitNet (Romero et al. 2014).
2024b). Methods of the same category are grouped in all tables. Details of the training setups, in- cluding the training DJIP teachers, student distillation procedures, the choice of hyperparameters. and the visualization of the optimized 128 quantization parameters are provided in Appendix A.5
and A.10
For both CE and DJIP teachers, report their CMI values measured on the training set without data augmentation. As shown in Tables I and 2, replacing the CE teacher with the DJIP teacher consis- tently improves the student performance, regardless of whether the teacher and student architectures are the same. These improvements are observed across all evaluated methods, with accuracy gains of up to 2.44%. Notably, the improvements more pronounced when the teacher and student architectures differ, i.e., when there exists a larger capacity gap between them.
5.2 IMAGENET RESULTS
ImageNet is a large-scale dataset used in visual classification tasks, containing approximately 1.2 million training images and 50,000 validation images. Following the implementation of Zhao et al. (2022), we conduct experiments on two widely used teacher-student pairs (see Table 3) and six representative distillation methods: KD (Hinton et al. 2015),, AT (Zagoruyko & Komodakis 2016). DKD (Zhao et al.) 2022), L.KDD (Sun et al. 2024b). WSLD (Zhou el al. 2021 and Re- viewKD (Chen et al. 2021).
Across all knowledge transfer methods reported in Table we observe that replacing the CE teacher with the DJIP teacher consistently improves the student Top-1 accuracy well. Details the training setups, including training of DJIP teachers, student distillation, and the choice of hyperpa- rameters, well an analysis the effect of hyperparameters for DJIP teachers, are provided in Appendix A.5 and A.6
5.3 VISION TRANSFORMER RESULTS
In previous experiments on CIFAR- 100, we demonstrated the effectiveness of DJIP for distillation between CNN architectures, whether identical or different. In this section, following Hao et al. 22023; Li et al. (2022). we extend DJIP to address the challenge of cross-paradigm distillation between CNNs and ViT's CIFAR-100. The experimental results summarized in Table 4 We

---

Published as a conference paper at ICLR 2026
Table 1: The test accuracy (%) of students on CIFAR-100 (averaged over runs), with teacher- student pairs in the same architecture. We use asterisk (*) to identify the results reproduced on our local machines. The small print denotes the improvement achieved by using the DJIP teacher.
Teacher
WRN-40-2
4R- 75.61
ResNet-56 72.34
ResNer-110
ResNee.110
VGG-13
ResNer-32x4
Acc
75.61
74.31
74.31
74.64
79.41
Student
WRN-16-2
WRN-40-1
ResNet-20
ResNet-20
ResNer-32
VGG-8
ResNer-8xx4
Asc
73.26
71.98
69.06
21.14
70.36
72,50
DJ1P
DIIP
CE
DJIP
CMI
0.026
0.501
D.006
0.501
0.158
0.724]
0.061
0.565
0.061
0.565
(0.015
0.252
10,006
0.276
KD
74.92
75.64
73:54
74.41
70.66
71.20
70.67
71.65
73.08
73.71
72.98
74.01
73.33
74.38
18:06
10.00
1::00
DKD
35.639
76.08
14.85
75.14
71.58*
71.86
71.51
71.72
74.11
74.22
74.68
74.93
76.32
76.55
DIST
75.51
75.98
74.99
71.75
71.97
73.89
76.31
+3.33
WTTM
76.37
76.70
74.58
74.98
71,92
72.15
71.67
71.50
74.13
74.33
74.44
74.81
76.06
76.61
40.31
-3.40
14.20
16.33
+0.19
43.83
CC
73.56
73.80
72.21
72.48
69.63
69.93
69
69:87
71.48
71.86
70.71
71.12
72.97
73.23
RKD
73.35
72.36
71.48
71.90
72.69
AT
74.08
74.51
72.77
73.22
70.55
70.81
70.22
70.63
72.31
72.84
71.43
72.03
73.44
73.94
40:43
-3:45
14..0 26
8:41
-6.53
44.00
-2:50
FaNet
73.58
74.20
72.24
72,84
69.21
69.83
68.99
69:40
71.06
71.42
71,02
71.92
73.50
73.83
16.4
-2:33
73.25
73.55
71.59
71.93
69.84
70020
70.70
72.54
70:58
71.28
72.86
73.76
SP
73.83
14.35
72.43
72.99
69.67
70.83
70,04
71.05
72.69
73.37
72.68
73.42
72.94
73.62
+0.35
-3.56
-116
+3.05
-6.2
+0.76
+2.63
ITRD
76.12
76.33
75.18
75.23
71.26*
71.44
71.52*
71.86
74.26
74.30
74.86
75:00
76.19
76.28
46.34
44.14
+2.03
CRD
75.48
76.01
74.14
74.57
71,16
71.61
71,46
71.79
73.48
73.93
73.94
74.35
75.51
75.85
LSKD
76.11
76.35
74.37
74.89
71,26*
71.39
71.48
71.60
73.67
73.92
74.36
74,90
76.62
76.96
40.24
+2.02
16.22
-0.25
14.94
+2.31
Table 2: The test accuracy (%) of students on CIFAR-100 (averaged over runs), with teacher- student pairs in different architectures.
Teacher
ResNet-50
ResNet-50
RResNct322x4
ResNet-32x4)
WRN-10-2 75.61
VGG-13
Ace
79.34
79.41
74.64
Studen:
MobileNetV2
VGG-8
[ShufeeNerV1
ShuffleNetV3
ShufleNetVi
MobilNNeV2
Ace
64.60
70.36
70.50
71,82
70.50
64.60
Method
DJI
CE
DUPP
DJIP
CMI
0..009
0.341
0.009
0.341
0.006
0.276|0.006
|0.006
0.276
0.026
0.501
0.015
0.252
KD
67.35
69.50
73.81
74.18
74.07
75.64
74.45
76.24
74.83
76.36
67.37
68.87
42:15
-0.67
-1.57
+3.70
+133
+1.50
DKD
70.35
11.18
73.94
75.87
76.45
77.24
77.07
77.52
76.70
77.16
69.71
70.20
40.65
-3.90
DIST
68.66
69.59
74.11
74.30
76.34
76.64
77.35
77.95
76.22
76.69
68.50
69.01
WTTM
69.59
69.98
74,82
75.19
74.37
76.55
77.30
75.42
76.24
CC
66..33
65.79
70.25
70.92
71.14
72.17
71.29
73.25
71.38
72.22
64.86
65,83
10:00
43.00
10:3
10.9
RKD
64.43
65.56
71.50
71.93
72.28
73.40
73.21
74.27
72.21
73.85
61.52
66.04
11:16
15:48
11:17
11:06
(1:08
(1:30
AT
58.58
60.02
71.84
72.
42
71.73
73,93
72.73
74.16
73.32
75.27
59.10
61.21
+1.44
-2.25
+3.43
FirNet
63.16
64.05
69.39
69.57
73.59
74.47
73.54
74.65
73.73
74.02
64.14
65.46
70.29
71.75
72.03
SP
68.08
68.58
73.34
73.87
73.48
75.92
74.56
76.28
74.52
36.32
66.30
61:35
ITRD
71.34
72.06
75.49
75.91
76.91
77.30
77.40
77.92
77.09
77.31
70,39
70.91
10:75
14:33
18:35
10:22
19:30
CRD
69.11
70.01
74,30
74.55
75.11
75.79
75.65
76.32
76.05
76.21
69.73
69.89
40.00
-0.05
LSKD
69.02
70.39
74,88%
75.21
75.67
76.38
75.56
77.12
76.56*
76.78
68.61
69.93
Table 3: The test accuracy (%) of students on ImageNet.
Teacher
Student
Method
CMI
KD
AT
DKD
LSKD
WSLD
Review KD
ResNet-34 73.31
R8 69.76
CE
0,720
70.66
70.70
71.70
71.42
71.73
71.61
0.738
70.78
72.08
71.65
71.87
71.72
ResNer-50 76.16
MobileNerV1 68.87
CE
0.600
70.50
69.56
72.05
72.18
72.02
72.56
DJIP
0.649
70.92
70.57
72.52
72.37
72.78
73.04
-6.19
40.76
40.6
include one teacher-student pair for each paradigm setting: ViT-to-CNN, CNN-to-Vij and ViT- ViT. The results clearly show that our proposed method is also effective for knowledge transfer across heterogeneous architectural paradigms and highlight the potential of DJIP in distilling ViTs.
7

---

Published as a conference paper at ICLR 2026
Table 4: The test accuracy (%) of students on CIFAR-I00 (averaged over runs), with teacher- student pairs in heterogeneous architectural paradigms.
Teacher
Swudent
Method
CMI
KD
DIST
DKD
OC
RKD
CRD
CE
0.100
77.26
76.49
78.10
74.26
73.72
76.60
VIT-S 92.04
ResNet-18 74.01
DIIP
0.171
79.06
78.04
80.15
74.87
75.85
77.75
6
63.90
41.35
12:03
40.01
+2:13
-1.16
ConvNext--T
DeiT-T 68.00
CB
0.229
72.99
73.55
74,60
68.01
69.79
65. 94
0.256
88.41
70.57
VIT-S 92.04
DeiT-T
DJIP
0.171
73.97
73.96
74.90
69.86
70.11
65.75
+4.1
+3.20
+3.49
14.34
+6.22
-6.30
ANALYSIS
6.1 ABLATION STUDY
As discussed in the contributions outlined Section ! DJIP introduces two key components: (I) differentiable JPEG layer, and (2) an alternating CMI maximization algorithm. In this section, we conduct an ablation study isolate and analyze the effectiveness of each component.
As shown in Table 5) when the differentiable JPEG layer is added but the fixed-centroid method from Ye et al. (2024) is retained (referred as 'JMCM'), the resulting method generally outperforms various KD baselines. Furthermore, when the alternating maximization algorithm is additionally applied, resulting our proposed DJIP, the performance is further improved over JMCMI in general.
In summary. both the differentiable JPEG layer and the alternating CMI maximization algorithm contribute positively the overall performance. Each component individually enhances the KD process, and their combined use in DJIP yields the best results, demonstrating synergistic effect.
Table 5: The test accuracy (%) of students CIFAR-100 (averaged over runs), with teacher- student pairs of the same- and different- arrheeenttarchitetuue..
Teacher
Student
Method
CMI
KD
DKD
DIST
CC
RKD
AT
FitNet
FT
SP
ITRD
CRD
CE
0.026
74.92
75.63
75.51
73,56
73.35
74.08
73.58
73.25
73,83
76.12
75.48
WRN-40-2
WRN-16-2
JMCMI
75.53
75.83
74.28
73.75
75.93
DIIP
0.501
75.64
76.08
75.98
73,80
74.09
74.51
74.20
73.55
74.35
76.33
76.01
CE
0.026
74.83
76.70
76.22
71.38
72.21
73.32
73.73
72.03
74.52
77.09
76.05
WRN-40-
ShuffleNetV1
JMCMI
0.505
75.95
76.56
76.85
71.92
73.39
75.10
73.81
73.25
75.96
77.14
76.14
DJIP
0.501
76.36
77.16
76.69
72.22
73.85
75.27
74.02
73.80
76.32
77.31
76.21
6.2 ORTHOGONALITY OVER MCMI
Since DJIP shares MCMI's goal of maximizing the teacher's CMI, examine whether method orthogonal to MCMI and can further improve performance. Following the experimental setup of (Ye al. 2024), we prepend our trainable JPEG layer to the MCMI teacher obtain the DJIP-MCMI teacher reported in Table
The results in Table indicate that applying DJIP method yields additional improvements in Top-1 accuracy, with gains of up to 0.92% absolute. These findings suggest that DJIP not only complementary but also orthogonal and additive to MCMI, as it explicitly optimizes the input space, which is not considered in MCMI.
Table The test accuracy (%) students on CIFAR-100 (averaged over three runs), with teacher- student pairs of the same- and different- arrhheectt...ciettue.......
Teacher
Student
Method
CMI
KD
CRD
RKD
VGG-
VGG-8 70.36
MCMI DCMP-MCMI
0.1298
73.83
74.87
73.29
74.23
72,03
0,1402
74.26
72.36
74.64
VGG-13 -3
MobileNeiV2 64.60
MCMI DJIP MCMI
0.1298
69.14
70.35
67.83
69.98
65.37
0,1402
69.40
70.51
68.33
70.46
65,90
74.64
40.25
40.38
40.30
40.46
8

---

Published as a conference paper at ICLR 2026
6.3 COMPARISON WITH MCMI
As mentioned in Section T our DJIP method addresses the fixed-centroid problem of MCMI by introducing an alternating optimization algorithm. Consequently, when compared with the CIFAR- 100 and ImageNet results of MCMI applied to the same KD methods, our method achieves com- parable or better performance across most cases, as presented in Figure
is important to note that MCMI has a much larger degree of freedom, as it optimizes the parameters of the pretrained model, whereas our method only adjusts the 128 quantization parameters of the JPEG layer. Despite this limitation, our method still matches or outperforms MCMI in multiple scenarios, demonstrating the effectiiveness of learning solely through compression parameters.
6.4 COMPARISON WITH CKD AND TALD
As discussed in Section our DJIP method is capable of exploring a significantly larger continuous quantization space than CKD. Consequently, when compared with the CIFAR-100 and ImageNet results of CKD applied to the same KD methods, our approach consistently achieves comparable or superior performance. A brief comparison of CIFAR-100 experimental results is shown in Figure 2
It is worth noting that CKD adaptively selects the optimal quantization table for each input image; that is, two different images may be compressed using different tables chosen to maximize teacher effectiveness, which computationally expensive. In contrast, our method employs fixed quanti- zation table shared across all input images throughout the entire distillation process. Moreover, since CKD achieves comparable performance to another seuutt-erturbed-baaee method, TALD (Nguyen- Duc et al. 2023), we also include TALD for comparison in Figure 2
(a) VGG-13/ VGG-S
(b) VGG-13 MohileNet22
WKN 40-2/ WKN-40-1
(d) WRN-40-2/ ShuftleNatV1
Figure 2: (a). (b) Comparison with MCMI under different student architectures on CIFAR-100, (c),
(d) Comparison with CKD and TALD under different student architectures on CIFAR-100.
For further analysis, we refer the reader to Appendix A. 10 and A.11 which provide visualizations of the learned quantization step sizes and the output probability distributions.
7 CONCLUSION
In this work, we have introduced Differentiable JPEG-based Input Perturbation (DJIP), novel framework that enhances the transferability of fixed teacher by incorporating a differentiable JPEG compression layer. By jointly optimizing CE loss and CMI values through a tailored alternating al- gorithm, our method enables the teacher to produce more informative supervision signals without modifying its original weights. Extensive experiments on ImageNet and CIFAR-100, across both CNNs and ViTs architectures, demonstrate that DJIP consistently improves student accuracy (up to 4.11% over standard baselines). Comprehensive analyses further confirm the effectiveness and stability of our alternating CMI optimization. Beyond empirical gains, DJIP offers a lightweight and generalizable mechanism for enhancing knowledge distillation by exploiting input-space per- turbations using trainable quantization. Since the teacher remains unchanged, DJIP is particularly suitable for scenarios with strict deployment or integrity constraints.
Pleasse refer to Table 2 of Yo ct al. 2 Please refer Table 3 of Salamah et al. 2025a
2024

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our work. All implementation de- tails of the proposed method, including model architectures, training procedures, as well as detailed descriptions of data preprocessing steps and hyperparameter settings, are provided in the appendix. To further facilitate reproducibility, we include executable source code and usage guidelines in the supplementary materials.
REFERENCES
Zeyuan Allen -Zuu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and 09816
self-distillation in deep learning, 2023, URL htttp:://axxiiiii org/
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Amil, and Alexander
Kolesnikov. Knowledge distillation: A good teacher is patient and consistent, 2022. URL
https:://exxisi org. http///s../.......///.. 2210600223
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 5-55535-111 2006.
Defang Chen, Jian- Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers, 2019.
wttp://wwwwwoo////..//.2.22 00350
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge re- view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation, 2019. URL https://arxiv. wwt.oorrg//o/ooooooooooo /t1///1///11111111
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference, 2021. /ttl://ttae//xx..oooo///bb//////
Chengyu Dong, Liyuan Liu, and Jingho Shang. Toward student-oorentted teacher network training for knowledge distillation, 2024. URL / http://aaaai...rraaa//2....
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models, 2023.
Shayan Mohajer Hamidi. Training neural networks on remote edge devices for unseen class classi- fication. IEEE Signal Processing Letters, 31:1004-1008, 2024. doi: 10. 10.11099559222233333344.
Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Linfeng Ye, and Ahmed Hussein Salamah. How to train the teacher model for effective knowledge distillation. In European Conference on Computer Vision, pp. 1-18. Springer, 2024.
Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang. Han Hu, Yunhe Wang, and Chang Xu. One-for- all: Bridge the gap between heterogeneous architectures in knowledge distillation, 2023. URL
https: /tapss///axxvv /ttt:s////xxx/oorr//a/////////// 19444
Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong, Haotong Qin, and Xianglong Liu. DA-KD: Difficult--wware knowledge distillation for efficient large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning. volume 267 of Proceedings of Machine Learning Research, pp. 22372 22391. PMLR, 13-19 Jul 2025. URI
ml wtt.mm..r.reree222mmm 6w..22//eeeee
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with ad- versarial samples supporting decision boundary, 2018a. URL httts::/aaxxi .rg. abs/ TB05. 05532
10

---

Published as a conference paper at ICLR 2026
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation
of activation boundaries formed by hidden neurons, 2018b. URL attps://azxiv.co /rq///orxx.///////////////
abs Aibs/1811.00 03233
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.023: 02,331, 2015.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. Advances in Neural Information Processing Systems, 15:33716-33722 2022.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres-
sion via factor transfer, 2020. URL http:htss//axiii..oogpg/ps/
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Yo-Yo Yun. Distillm: Towards streamlined distillation for large language models, 2024.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble, 2018. URL htttss://aaxiii
/00bb.///8....0 646
Kehan Li, Runyi Yu, Zhennan Wang. Li Yuan, Guoli Song, and Jie Chen. Locality guidance for im- proving vision transformers tiny datasets, 2022.
10026
Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement, 2020. URL
2010. 00795
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awaae weight quantization for Ilm compression and accelcration, 2024.
/000//00000/300000777 00978
Yufan Liu, Jiajiong Cao, Bing Li. Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In 2019 JEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 7089-7097, 2019. doi: 10.1109/CVPR.2019. 00726.
Xu Luo, Hao WU, Zhang, Lianli Gao, Jing Xu, and Jingkuan Song. A closer look at few-shot classification again, 2023.
Roy Miles, Adrian Lopez Rodriguez, and Krystian Mikolajezyk. Information theoretic representa- tion distillation. arXiv preprint arXiv: /2.00049, 2021.
Hossein Mobahi, Mehrdad Farajtabar, and Peter I.. Bartlet. Self-distillation amplifies regularization in hilbert space, 2020. bttps:/:://axx abtp://rwwww....aaa////aaa..aaap 20002 05715
Thanh Nguyen-Due, Trung Le, He Zhao, Jianfei Cai, and Dinh Phung. Adversarial local distribution regularization for knowledge distillation. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4670-4679, 2023. doi: 10. 109////// 2023, 300066.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, seghpperformance deep learning library, 2019. htthst/:::/// /ttp://////....//aaa/aaa org. 1911119111.. C1703
Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation, 2019. URL https::/ arxiv. www.orggasb...... http///abs/994..110000
11

---

Published as a conference paper at ICLR 2026
William B. Pennebaker and Joan L. Mitchell. JPEG Still Image Data Compression Standard. Kluwer Academic Publishers, Norwell, MA, United States, 1992. ISBN 0442012721.
Mary Phuong and Christoph H. Lampert. Towards understanding knowledge distillation, 2021. URL
https://arxiv.om owg. abs 2ttp://211..10000
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65500 2014.
Ahmed H. Salamah, Shayan Mohajer Hamidi, and En-Hui Yang. A coded knowledge dis- tillation framework for image classification based on adaptive ipeg encoding. Pattern Recognition, 158: 10966, 2025a. ISSN 0031-3203. doi: https//dbb.orr//1.0111..pppooo 2024. 10966. URL
50031130322007112
Ahmed H. Salamah, Kaixiang Zheng, Yiwen Liu, and En-Hui Yang. Jpeg inspired deep learning, 2025b. URL httptttss///xxiivv
07081
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wil- son. Does knowledge distillation really work?, 2021. URL
/bs/
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024a. URL httppss/aaax...ogg//bb///// 2111//226666661661361 11695
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation, 2024b.
Chao Tan and Jie Liu. Improving knowledge distillation with a customized teacher. IEEE Transac- fions Neural Networks and Learning Systems, 35(2):2290 2299, 2024. doi: 10.9 2022.3189680
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2019.
Frederick Tung and Greg Mori. prreeeriit--reeerinngn knowledge distillation, 2019. URL https: 7/arxiv. flarxiv.orq/abs/1977.00
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 9(66)2559-26655 2008. URL http: jmir pttp://pm..oppppps//////////
html
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. Efficient knowl- edge distillation from model checkpoints. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave. K. Cho. and A. Oh (eds.), Advances Neural Information Pro- cessing Systems, volume 35, pp. 607-619. Curran Associates, Inc., 2022. URL. cc http:e//riileepppe//2222/22 Paper- Confereneefeeeccpcce
ottppss/pproceeding...eem 3300722b88seee//eeffla///3333333
Guanghui Wang, Zhiyong Yang. Zitai Wang. Shi Wang. Qianqian Xu, and Qingming Huang. ABKD: Pursuing a proper allocation of the probability mass in knowledge distillation via alpha- beta-divergence. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd In- ternational Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. press/v267 2t2.66//wagg55520 html
PMLR, 13-19 Jul 2025. URL https::/ppoceeeiing..
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10817-10834, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18.58633/v acl-oo.......
URL https: //tp/aaaaaaanonoaoacaaoaoa hot.hooooogoyog//////oooo
2023.. acl- 6057
12

---

Published as a conference paper at ICLR 2026
Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethink- ing Kullback-Leibe divergence in knowledge distillation for large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the st International Conference on Computational Linguis- fics, pp. 5737-5755, Abu Dhabi, UAE, January 2025. Association for Computational Linguisties. URL
coling mawiiinngnmmn.33 main.383)
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L.. Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 33601))5628-5633, Jul. 2019. doi: 10. 10.1669aaaiii..... 13301..33366.... BtRLhtts::/ /tappp:///oss aaai. wwtp//ggg....... wwwpppppppaaa... artaaceeeiieeiiiee 4506
Chuanguang Yang, Zhulin An, Linhang Cai, and Yongjun Xu. Hierarchical self-supervised aug- mented knowledge distillation. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, ICCAI-2021, pp. 1217-1223. International Joint Conferences on Ar- tificial Intelligence Organization, August 2021. doi: 00.24966/iccai.2221/20 URL. http:
7/dx.do: or..rggg//. 10 24932/2 2021/166
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation, 2022. URL https://arxiv,
org/ ab// 2204 06986
Chuanguang Yang,
Xinqiang Yu, Zhulin An, and Yongjun Xu. Categories of response-based, feature-based, and relation-based knowledge distillation, 2023. URL /ttp::://///r/
3bbs2306.10663
En-Hui Yang. Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification. IEEE Transactions on Neural Networks and Learning Systems, 2025.
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and EN-HUI YANG. Bayes conditional distribu- tion estimation for knowledge distillation based on conditional mutual information. The Twelfth International Conference Learning Representations, 2024. httttttss///peeeeee
net/ fdet/fo=ee forum?idd mett/orrnn.dd-WWWWaBa VVowwElttkR
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.00 1612.03928,
2016.
Haoran Zhang, Zhenzhen Hu, Wei Qin, Mingliang Xu, and Meng Wang. Adversarial co-distillation learning for image recognition. Pattern Recognition, 111:107659, 2021. ISSN 0031-3203. doi: http://c.inne///raaee//////11110 com/
107659, URL httpp::/wwwt sci.nceddeeeo rect.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEF/CV Conference on computer vision and pattern recognition, pp- 11953-11962, 2022.
Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher match- ing. In The Twelfth International Conference on Learning Representattonss, 2024. URL https: 7 /openreview net wwwe/fooumm//. MJ3K 7 7DGGG
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang.
Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In In- ternational Conference on Learning Representations, 2021. URL https: http:///pppeeiiiee///// hett/Eorum2ioo-ggla-555222
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RELATED WORKS
Conventional KD: Also known as knowledge transfer, KD is a model compression technique in which a compact student model learns to mimic the behavior of larger teacher model. This con- cept was first introduced by [Buciluă et al. (2006), who trained a smaller model to match the logits of larger one. Hinton al. (2015) later popularized KD by introducing
capperature-based softening to both teacher and student logits. Since then, KD methods have proliferated and can be broadly cat-
egorized into three groups according to Yang et al. (2023): (i) response/logit-baseee basee methods (Hinton
let al.
2015 Heo et al. 20188| /Zheng& & YANG)| 20244| Hamidi 2024] Lan et al.) (2018] |Stanton et a
2021
Chen al. [019 [Li et al. |0000 Beyer al.] 2ha al. [022] Miles al.
[202] [Sun
et al.
2024b; Huang et al.) (2022); (ii) ceature-based methods (Romero el al., 2014; Zaaoruyko &
Komodakis 2016] Tim et al. [217;
Chen 2017:Chen
2021; Yang et al. 202 Kim et al. (2020) Heo al.
2018b Tung & Mori 2019 Tian et al. 2019):
(ii) relation-bre based methods (Park et al., 2019 Peng
al. 2019 Liu et al., 2019 Yang et al. 2022). Building upon these conventional KD frameworks,
which aim to improve the student model's is ability to mimic the teacher's 's logits, intermediate features, or inter-feature relations, two main orthogonal directions have emerged to further enhance student performance.
KD with Student-Orienter Teacher: Rather than using teachers who only care about their own performance, several works aim to find or train teachers better suited to the students. Cho & Har- iharan 2019); Wang et al. (2022) show that early stopping during teacher training or using earlier checkpoints preserves higher mutual information between inputs and outputs, providing more in- formative soft targets. TTan & Liu (202)); Yang et al. (2019) encourage teachers to produce more dispersed probability distributions via auxiliary losses, while Dong al. (2024) shows that explic- itly imposing the Lipschitz and consistency constraint in teacher training can facilitate the learning of the true label distribution and thus improve the student performance.
Meanwhile, Ye et al. (2024) demonstrates that training teachers with the maximum CMI (MCMI) estimator, rather than the conventional minimum CE objective, yields better approximations of the true Bayes conditional probability distribution (BCPD) of label given input 2, thereby significantly enhancing student performance by capturing richer contextual information. Nevertheless, a major limitation of these student-oriented approaches lies in their requirement to modify the teacher's weights, which is often impractical in real-world scenarios due to deployment or integrity con- straints-for for instance, when the teacher is provided as a black-box model or is already deployed in production environments where retraining is prohibited.
KD with Input Perturbation: Another prominent direction involves perturbing the input space, typically through adversarial or divergent examples during KD. conventional KD pipelines, the teacher and student are trained on identical inputs, which may restrict the diversity of features vealed by the larger teacher model, thereby limiting the effectiveness of knowledge transfer. [Hco] et (2018a) argues that adversarial examples, crafted to align with the teacher's s decision boundary, help the student learn a more accurate and generalizable boundary. However, their method generates only single adversarial example per input, limiting its ability explore the full spectrum of pertur- bations. Nguyen-Duc et al. (2023) address this limitation by formulating teacher adversarial local distribution, which more thoroughly explores the teacher's decision boundaries, denoted as TALD. In the context of online co- co-distillation, Zhang et al. (2021) leverages Generative Adversarial Net- works (GANs) to produce divergent examples enriched with 'dark knowledge,' thereby facilitating more effective mutual learning among co-distillation classifiers.
In contrast to these computationally intensive approaches that require access to both the original and generated datasets during distillation, [Salamah et al. (2025a) proposes a lightweight alternative called CKD, which only uses a compressed version of the original dataset. CKD introduces an adaptive JPEG compression layer before the teacher model to generate multiple compressed variants of each input image using different JPEG quality factors (QFs), selecting the most informative one based on predefined criterion. Although CKD has shown promising results, it remains limited to small set of predefined quantization tables in a discrete space that is not optimal, thereby restricting its ability fully explore the space of compression-induced input perturbations.
disiillation,allaahh
14

---

Published as a conference paper at ICLR 2026
Our proposed method, DJIP, further develops the CKD framework to address the three key limita- tions discussed above:
No Teacher Weight Updates: DJIP further advances the coded teacher framework introduced in CKD, thus obviating the need for optimizing the teacher model's weights. This architectural decision renders DJIP particularly suitable for scenarios in which modifying the teacher is impractical, as commonly encountered in student-oriented KD paradigms.
Greater Flexibility with Minimal Complexity: By incorporating differentiable JPEG layer. DJIP enables full exploration of the continuous JPEG compression space. The trained JPEG layer func- tions lightweight, modular component that can be easily attached to or removed from the model. This introduces only a small number of additional parameters, making DJIP highly deployable. In contrast to CKD, which requires repeated image-wise compression and selection across multiple QFs, DJIP performs a one-time mmage-wise compression without any selection process, thereby achieving significantly lower computational overhead while maintaining comparable functionality,
Student-Oriented Design with Dynamic Optimization: DJIP adopts the student-oriente MCMI estimator from Ye al. 2024) and addresses its major limitation by introducing a novel alternating optimization algorithm. During training, both the class-wise clusters in the output probability space and their centroids are updated dynamically. In contrast, the fixed-centriid strategy Ye et al.) 2024), adopted estimation errors. This was evident in their experimental setup, where training was initialized from pretrained model and conducted for only limited number of epochs I Our proposed alternating algorithm provides an analytical solution to the MCMI objective, enabling efficient centroid updates and accurate CMI estimation. Consequently, DJIP achieves performance comparable to that of MCMI, despite using significantly fewer trainable parameters.
compromise for computational tractability, leads to the accumulation of CMI
Beyond the vision community, recent studies in large language model (LLLM) distillation have visited the foundations of KD from pvergence-optimizatin perspective. While developed for auto-regressive language models, these works provide insights that are broadly relevant to KD re- search. Wen et al. (2023) propose optimizing a general f-divergence for sequence-level distilla- tion, highlighting the importance of selecting an appropriate divergence tailored the structure of teacher student discrepancies. KK et al. (2024). (Gu et al. (2023), and related efforts streamline LLM distillation pipelines by identifying training configurations that improve stability and efficiency, demonstrating that distillation effectiveness is sensitive to the choice of objective functions and optimization heuristics. More recently. [Wu et al. (2025) analyzes the limitations of KL divergence in the context of heavy-tailed token distributions, revealing that KL may misallocate probability mass during distillation; WWan et al. (2025) further generalizes this idea by employing alpha-beta divergences to reshape probability allocation. Additionally, HH al. (2025) introduces difficulty- aware distillation, showing that adaptively weighting samples based on their learning difficulty can improve student performance. Although these methods are designed for LLMs, the underlying principles-such as flexible divergence design, probability-mas reallocation, and difficulty-awan weighting offer conceptual guidance for advancing KD in computer vision settings well.
'If the teacher model had heen trained from scratch, divergence would have occurred.
15

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF THEOREM! 1
Recall the Markov chain Y + X x N +.. To prove Theorem we first introduce an auxiliary
distribution Q(- v, and derive new expression for T(Xm:: y Y) as follows:
I(Nw; wi 8 (I)) I(X; IX)
E Px sE|C
Prixy (8 (Y= = x.
E Pv( Σ (xir ( 8)
Σ Prixr = x. x In
(y =il Y
DE
ΣΣ P(a,y) Σ [(
Prir = =
Px rx (f
Pr((v) (xpr(e 2 (ān [i]] In
Pxir (x y) y)
Σ (v) Σ P (tr( Σ Pxiv ((
Px ()
Pxlx (I 1) (r (11)
( 8)
Px (+
In Q(# j,y)
Pr( Σ (i Σ
(Q01).))
Pr
(x v)
x
Σ P(r,y) Σ [( [i]] In Q ( max (QC1(a))
(12)
Pxiy y)
where Equation 11 follows from the cross entropy inequality, and the maximization in Equation 12 achieved when
Q' (,,) = (x \1))()
(13)
(i y)
Thus, the single minimization problem in equation can be converted into a double minimization problem over w and {Q(- i,y)} as follows:
EX f(Xw)) - 1IX In)
min
=min
(XH((Prxx (A(;))
((.w) Q(x (,
min
(xH(PrIxx((xw))
(Q01--9)
max
Σ P(a v)
Pxiy (x v)
min
Ex ExH(PPixxJ((xx))>
max
Σ P(r
[T.)) In Q(x i.
(14)
(QC1E)))
min
ExHH(PIIxx +
min (QC160)
P(z 9) Σ ( Q(: (,2)
min min (Q016.))
P(x, PLOOHC(BNN 2)) X
P(a,
P 1( Q (t (15)
2.8
where the symbol "=" in Equation 14 indicates equivalence up to an additive constant independent of Q(x v. and which is omitted here for clarity.
A.3 PSEUDO-CODE FOR ALTERNATING ALGORITHM
The pseudo-code of the alternating optimization algorithm is provided in Algorithm I In Sec-
tion 6.1 present comprehensive experiments CIFAR-100 100 that demonstrate its advantages over the fixing-centroic method in Ye et al. (2024) and validate the effectiveness incorporating a differentiable JPEG layer.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 The proposed alternating algorithm for solving the optimization problem in Theorem
Input: DNN fo as f; JPEG layer Ja with trainable parameters w; training set DD = {(x,, ut) )= with its B mini-batches {8") heeey class labels C; number of epochs T and hyper-parameter A.
1: Initialize w
2: repeat
3: for 6 1 B do
4: [Update S]:
5: Fix wb- Update centroids (5
weecc according to Equation
9
1 - DDy
Σ (J vi,y E [CJ.
(16)
2GED,
6:
[Update Q]:
7:
Fix wn Calculate (x i, y). according to Equation 10
Pavl f(Ta(z, w (
(² ((,))
v(a.)) E B B [C].
(17)
8:
[Update w]
9:
Fix (- ((,(),,,)))))) (2.)) Update weights w wn1 to w by using SGD over the objective
function
[) (a.))E
10:
end for
11:
Set w" -- w
12:
until T' epochs are completed
return Trained parameters w
A.4 CONVERGENCE ANALYSIS AND EMPIRICAL EVIDENCE
As discussed above, the inner minimization step in Equation satisfies the cross-entropy inequality. Suppose the impact of the random mini-batch sampling and SGD is ignored. In that case, the alternating algorithm is guaranteed to converge in theory, since given w, the optimal Q" i, can be found analytically via 13 Although in practice the alternating algorithm may not converge to global minimum, this is not the limitation of our algorithm, but the nature of all SGD-based deep learning algorithms.
Empirically, alternating procedure exhibits stable behavior throughout training. As shown in Figure
3 the training CMI value consistently improves, and the training DJIP loss and the overall training loss converge smoothly over iterations both ResNet-34 and ResNet-152. These observa- tions indicate that the proposed alternating algorithm behaves stably in practice at scale.
A.5 IMPLEMENTATION DETAILS
A.5.1 SETUPS FOR JPEG LAYER TRAINING STAGE
We follow the implementation and design of the differentiable JPEG layer from Salamah et al. 2025b), but set the minimum quantization step to 1, which more commonly used in reality, instead of O. The stochastic gradient descent (SGD) optimizer is used with momentum of 0.9 and a weight decay of 5 x 10 for all JPEG layer training experiments.
A key feature of the differentiable soft quantizer Qa in the JPEG layer is its variable softness, con- trolled by the trainable sharpness parameter a. However, as shown Salamah et al. (2025b). when initialize
is sufficiently large, its gradient vanishes, preventing effective updates. Therefore, and exclude a from training. Thus, the trainable
all entries of a to a large constant value parameters of the DJIP teacher are the quantization tables Q, which initialize with all ones.
17

---

Published as a conference paper at ICLR 2026
12.86
DJIP Less
CMI Valuee
Total Loss
12.84
5.86
12.82
12.80
0
1000
2000 3000 4000 5000
1000
2000 3000 4000 5000
1000
2000
3000
4000 5000
Iterution
Iteration
Iteration
(a) ResNet-34 (A 1.4)
5.8
1.6
DJIP Loss
CMI Value
Toxal Loss
12.5
5.7
1.5
5.6
1.4
12.4
1.3
1000 2000 3000 4000 Iteration
5000
1000
2000
3000
4000 5000
1000 2000 3000 4000 5000
Iteration
Iteration
(b) ResNet-152 2.0)
Convergence curves of the proposed alternating optimization on ImageNet. The training DJIP loss, training CMI value, and total training loss exhibit stable and monotonic trends, confirming the empirical convergence of our method.
Figure 3:
For CIFAR-100, pretrained teachers are adopted from the official repository Tian al. (2019) We use a learning rate of 0.1 and train for 20 epochs with batch size of 32 over 2 GPUs (total batch size 64). For all CIFAR- experiments comparing with the CE teacher, including those involving VIT models, we set the hyperparameter À in the objective function 8 to 0.5 by default. An exception made for the VGG- 13 and MobileNetV2 teacher-student pair in Table where we set À = 0.6. To demonstrate that DJIP is further orthogonal to MCMI in Table with À = 0.3. Throughout the alternating training process, we always use the full training set 'D to update the centroids proposed in Algorithm with the update interval 5.
train the DJIP-MCMI teacher
For ImageNet, we adopt pretrained teachers from the PyTorch official repository (Paszke et al. 2019). DJIP teachers trained for epoch (5005 iterations) with learning rate of 0.01, using GPUs and a batch size of 64 per GPU (total batch size 256). We experiment with various values of X, namely (0.4, 1.0, 1.4, 2.0, 2.4, 3.0}, and select À 1.4 for ResNet-34 and X 2.0 for ResNet-50.
When applied to large-scale datasets such as ImageNet, updating centroids at each iteration (update interval 1.) via Algorithm !
becomes computationally expensive. To reduce this complexity, we construct balanced sub-dataset DD by randomly sampling (Dyl
instances per class from the training set. By replacing the full training set in Equation 9 with D. a lightweight variant of the alternating algorithm be derived. In the first iteration, the dummy distribution (Q(- y)) is up- dated using the entire training set. Thereafter, it is updated using the sub-dataset and an exponential moving average (EMA) with a smoothing factor a 0.9.
Thus, the resulting pseudo-code on ImageNet experiments is similar Algorithm Equation 16 is replaced by:
except that
Σ ((Jale,,""')) V [C]. 1 +1 a) p& 2,ED,
(18)
To further handle the increased complexity of ImageNet tasks compared to CIFAR- tasks, fol- lowing the setups of (Salamah et al. (2025b), we incorporate five rounds of Qa quantization op- eration with independent trainable parameters inside the differentiable JPEG layer to expand the compression search space. Moreover, by |Sllmaa et al. (2025b). with the Gradient Scaling Constants h set to 20 to control the gradient magnitude to ensure more stable updates for Q
follow the gradient magnitude control method proposed
18

---

Published as a conference paper at ICLR 2026
A.5.2 SETUPS FOR STUDENT DISTILLATION STAGE
For experimental setups of all knowledge distillation variants in this paper, we use SGD with mo- mentum 0.9 and weight decay of 1 x 10 as the optimizer.
For CIFAR-100, we follow the distillation setups from Tian et (2019). Specifically, we train the student models for 240 epochs with batch size of 64 on 1 GPU. The initial learning rate is set to 0.05 and decayed by factor of 0.1 at epochs 150, 180, and 210. For MobileNetV2, ShuffleNetV1, and Shu2 we use smaller initial learning rate of 0.01.
For ImageNet, we adopt the training configuration from Zhao et al. 2022): 100 epochs, a learning rate of 0.1, and batch size of 256 on single GPU. The only exception is ReviewKD (Chen et al., 2021], which uses batch size of 128 and is trained on two GPUs.
For both datasets, we evaluated several state-of-the-art distillation methods using their official im- plementations and reported hyper- papee-parameters and successfully reproduced the results reported in their original papers.
A.6 ADDITIONAL. ABLATION STUDIES ON DJIP HYPER- SAPER-PARRAMTTES
In this section, we conduct series of ablation studies to investigate the sensitivity of hyper- parameters and how those key hyper-parameters affect the hehavior of the DJIP teacher and the resulting student performance. Specifically, we examine the influence of the weighting coefficient X, the number of quantization rounds in JPEG, the learning rate for JPEG layer's updates, and the cluster centroid update interval.
A.6.1 EFFECT OF À IN DJIP
In T vary À in (0.4, 1.0, 2.0, 2.4, to evaluate how the balance between CMI max- imization and CE minimization affects teacher behavior. Larger values increase the teacher's 's CMI while also raising its CE loss. The student's Top-1 accuracy follows quasiconcave trend and peaks at À = 1.4, indicating that this value provides an effective trade-off between informativeness and predictive correctness.
Table 7: Effect of on student test accuracy (%), the CMI and CE losses of the DJIP teacher. The results are compared with those of standard CE teacher in the ResNet-34 - ResNet- 18 setting on ImageNet with vanilla KD. Bold numbers indicate the best performance.
DJIP with À equals to
KD
CE
0.4
1.0
1.4
2.0
2.4
3.0
4.0
Teacher CELoss
0.560
0.568
0.568
0.582
0.630
0,595
0.627
0.871
Teacher CMI
0.7180
0.7257
0.7268
0.7382
0.7869
0.7523
0.7825
0.9866
Student Acc
70.660
71,362
71.452
71.654
71.274
71.370
71.290
70.694
A.6.2 EFFECT OF NUMBER OF ROUNDS IN DJIP
We investigate the effect of increasing the number of quantization rounds JPEG. Here, one round refers to applying quantization to the DCT coefficients once using separate quantization table. Introducing multiple rounds effectively expands the perturbation space, which particularly bene- ficial for complex datasets such as ImageNet. As the number of rounds increases, the teacher's 's CMI consistently improves; however, excessive rounds can lead elevated CE loss, indicating a loss of predictive reliability. As shown in Table the student's accuracy grows steadily from to 5 rounds and reaches its peak at rounds, suggesting that a moderate degree iterative quantization provides the most favorable balance between informativeness and stability.
A.6 EFFECT OF LEARNING RATE IN DJIP
We evaluate teacher learning rates of 0.1, 0.02, 0.01, and 0.001. As shown in Table overly high learning rate leads to unstable JPEG layer updates, manifested by excessively large CMI values and noticeably increased CE loss, which ultimately undermines the student's performance. The learning
19

---

Published as a conference paper at ICLR 2026
Table 8: Effect of the number of quantization rounds on student accuracy (%) and on the CMI and CE losses of the DJIP teacher. Results are reported for the ResNet-34 + ResNet- 18 setting on ImageNet under vanilla KD. Bold numbers denote the best performance.
DJIP with number of rounds equals
KD
CE
3
5
6
Teacher CEL
0.560
0.560
0.561
0.582
0.768
Teacher CMI
0.7180
0.7178
0.7185
0.7382
0.9097
Student Acc
70,660
71.278
71.600
71.654
70,960
rate of 0.01 yields the best overall results, indicating that stable yet sufficiently responsive update regime is essential for effective distillation.
Table 9: Effect of learning rate on student accuracy (%), along with the CMI and CE losses of the DJIP teacher. Results are reported for the ResNet-34 ResNet-18 setting on ImageNet under vanilla KD). Bold indicates the best performance.
DJIP with learning rate equals to
KD
CE
0.1
0.02
0.01
0.001
Teacher CELoss
0.560
0.875
0.601
0.582
0.561
Teacher CMI
0.7180
0.9961
0.7577
0.7382
0.7191
Student Ace
70.660
70.628
71.444
71.654
71.432
A.6.4 EFFECT OF CENTROID UPDATE INTERVAL IN DJIP
We further study the impact of the centroid update interval DJIP in Table When using intervals of 3, 5, 50, and 500 iterations, we observe that the student accuracy remains largely stable, with the best results appearing at intervals of 3 and 5. Although both intervals yield similar accuracy, select the interval of 5 due efficiency concerns. As the update interval becomes longer, the student's s accuracy shows a mild decrease; however, it consistently surpasses that obtained with the standard CE teacher. This suggests that moderately frequent centroid updates provide more reliable centroid estimates while avoiding unnecessary computation, thereby enabling more accurate inner minimization of Equation 15 and ultimately improving distillation performance.
Table 10: Effect of centroid update interval student accuracy (%) CIFAR- 100 (averaged over runs). Results are reported for the ResNet-34 + ResNet- 18 setting, with update intervals 3, 5,
50, and 500 iterations. All configurations consistently outperform the standard CE teacher, with the best performance achieved at an interval of and 5.
Teacher
Student
Method
CMI
KD
DKD
DIST
CC
RKD
AT
FitNet
FT
SP
ITRD
CRD
CE
0.009
73.81
73.94
74.11
70.25
71.50
71.84
69.39
70.29
73.34
75.49
74,30
Interval=3
0.341
74.49
75.87
74.58
70.92
71.87
72.50
69.50
71.29
73.58
76.02
74.56
ResNer-50
VGG-8
Interval=5
0.341
74.48
75.87
74.80
70.92
71.93
72.42
69.57
71.24
73.87
75.91
74.55
Interval=50
0.341
74,33
75.58
74.63
70.78
71.81
72.38
69.35
71.23
73.68
75.91
74.24
Interval=500
0.342
74.32
75.49
74.52
70.68
71.73
72.37
69.02
71.06
73.61
75.81
74.21
A.7 DJIP WITH OTHER INPUT PERTURBATION METHOD
As stated in Theorem I the parameter w represents arbitrary perturbation parameters and is fully agnostic to the specific choice of input perturbation mechanism. Consequently, the proposed al- ternating optimization framework is not restricted to JPEG quantization tables. The differentiable JPEG module adopted in our experiments serves merely convenient instantiation, selected due to its simplicity, computational efficiency, and the availability robust differentiable implementations. In principle, this perturbation layer may be replaced by any differentiable codec. As another illus- trative example, we additionally incorporate a convolutional autoencoder as the input perturbation module.
We employ a symmetric convolutional autoencoder designed for CIFAR-resolution images. The encoder is composed of convolution, batch normalization, and ReL.U layers, followed by a max-
20

---

Published as a conference paper at ICLR 2026
pooling operation in the middle. The decoder mirrors this structure using convolution and transposed-conven conoooltocoooootii layers to upsample feature maps and recover the original spatial resolution. As observed in Section/A.10, the optimized quantization tables consistently exhibit small step sizes. Motivated by this, we choose a latent dimensionality comparable to the input, enabling the au- toencoder learn an constrains the reconstructed output to the normalized pixel range.
denoising transformation. A final sigmoid activation
For pretraining, the autoencoder trained on CIFAR- with standard data augmentation. Opti- mization is performed using Adam with a learning rate of x 10-3, and the reconstruction objective mean squared error (MSE). Training proceeds for 200 epochs with batch size of 64. On the CIFAR-100 test set, the pretrained autoencoder achieves a PSNR of 41.52 dB, an SSIM of 0.9961, and an MSE of 8.3 x 10-5, indicating fighfiidllity reconstruction.
Following the training framework in Figure 1 we insert the autoencoder before the classifiers and optimize its parameters using the alternating optimization algorithm. We train for 20 epochs with learning rate of 1 x 10-3 and set À 0.3. After optimizing the perturbation module, standard distillation is performed. The final results are reported in Table I1
Table 11: The test accuracy (%) of students on CIFAR- 100 with autoencoder as input perturbation method.
Teacher
Student
Method
CMI
KD
FitNet
FT
SP
CRD
CC
RKD
CF: DJIP
0.015
72.98
71.02
70.58
72.68
73.94
70.71
71.48
ResNet-34 74.64
ResNet-18 70.36
0.067
73.31
71.31
71.03
72.92
74.05
71.13
71.77
-0.33
+0.45
+0.24
+0.11
+0.20
A.8 DJIP ONLINE DISTILLATION COMPLEXITY
As mentioned in Figure DJIP contains stages. The first stage trains the JPEG layer in offline manner. Since learned JPEG layer can be reused for training multiple student models, this stage is executed only once and its training time is amortized and therefore negligible in the overall framework.
For the second online stage, present the student training throughput, GPU peak memory. and total runtime in Table 12] The student distillation stage is conducted on single RTX 2080 Ti GPU,
Table 12: Comparison of DJIP distillation complexity measured with different teacher-studen pairs on CIFAR- 100 with vanilla KD method and TALD, following the configuration specified in Section
A.5
Teacher Student
Throughput (img/ms)
Peak GPU Memory (MB/GPU)
Total Runtime
Method
(s)
Vanilla
7.11
614
2026.6
VGG-13
DJIP
5.60
624
2572.8
VGG-8
TALD
2.48
1060
5829.6
Vanilla
4.61
408
3124.8
ResNet-56
DJIP
3.70
410
3895.2
ResNet-20
TALD
1.38
946
10447.2
Vanilla
3.53
668
4084.8
WRN-40-2
DJIP
2.98
670
4826.4
WRN-40-1
TALD
1.15
1262
12516.0
A.9 FEW SHOT CLASSIFICATION
In conventional few-shot classification, only a 3 percent instances from each class are made available for model training (Luo et al. 2023). Translating this idea into the KD setting, the JPEG layer is trained on the full dataset, while only a percent subset samples per class is used to train the student during distillation.
21

---

Published as a conference paper at ICLR 2026
To examine the effectiveness of the proposed DJIP teacher under limited-datt conditions, conduct experiments on CIFAR- 100 using WRN-40-2 WRN-16-2 teacher-student pair. We evaluate several values of 3, namely 5, 10, 20, 50, 75, Table 13 As shown, the student consistently benefits from distillation with the DJIP teacher, and the improvement is especially notable in more challenging low-data regimes (smaller (3). These results further support the robustness of our method under constrained data scenarios.
the vanilla KD method. The results are presented in
Table 13:
Comparison of student accuracy (%) under few-shot distillation settings on CIFAR- 100. The teacher is trained the full dataset, while only 3 percent subset of samples per class is used to train the student.
Teacher
Student
Method
5
10
20
50
75
CE
34.75
50.73
60.93
69.60
72. 17
WRN-40-2
WRN-16-2
DJIP
40.56
53.95
62.98
70.87
73.15
+3.81
+3.22
+1.27
+0.98
A.10 TRAINED QUANTIZATION TABLE VISUALIZATION
The quantization tables trained for the DJIP teacher, corresponding to the results Tables I
and 3 are presented in Figures 4 and 5 These results indicate that, across different datasets, the models consistently apply stronger compression to the channel than to the cb and Cr channels, which in turn leads to higher CMI values after training.
Quantization Step-so size
Quantization Step-size
Quantization Stop-size
12.5
15
15
CoCr
CbCr
10.0
OsCr
7.5
2.5
Frequency Index
Frequency Index
Frequency Index
(a) VGG-13 (A = 0.5)
(b) ResNet- (A = 0.5)
(c) ResNet-32x4 (A 0.5)
Quantization Step size
Quantization Step-Size
Quantization Steo-size
20
15
CbCr
20
10
10
2
AA
8
18
24
32
40
48
26
64
5
8
18
24
32
40
4B
56
54
18
24
40
48
36
64
Frequency Index
Frequency Index
Frequency Index
ResNet-56 0.5)
(c) ResNet-50 0.5)
(f) WRN-4002 0.5)
Figure 4: Trained quantization tables on CIFAR-100 models used in Table and
A.11 VISUALIZE THE OUTPUT PROBABILITY SPACE
We visualize how the output probability distributions of the ResNet-32x4 DJIP teacher evolve across epochs during single training run, illustrated in Figure 6 The simplex is visualized by projecting the 3-class softmax probability vectors onto 2D plane, where the one-hot vectors of each class mapped to the vertices of equilateral triangle. Temperature scaling with 7 = 4.0 applied to the softmax outputs.
Initially, the teacher produces highly confident predictions resembling one-hot vectors, with output probabilities concentrated near the corners of the simplex. As training progresses, facilitated by the differentiable JPEG layers, both the output distributions and the centroids of each class cluster
22

---

40 Published as a conference paper at ICLR 2026
Quantization Step-size
40
round: 1
CbCr
round: 2
CbCr round:
round: 3
15
CbCr round:
round: 4
CbCr round: 4
20
round: 5
10
CbCr round:
10
24
32
40
48
56
64
0
8
16
24
32
Frequency Index
Frequency Index
(a) Re (A 1.4)
Quantization Step-size
round: 1
CbCr round:
round:
20
CbCr round:
round: 3
CbCr round:
round: 4
15
CbCr round:
round: 5
CbCr round: 5
10
20
32
48
56
64
0
8
16 Frequency Index
24
32
40
48
56
Frequency Index
(b) ResNet-50 (A = 2.0)
Figure 5: Trained quantization tables on ImageNet models in Table
gradually shift toward the center of the simplex, leading to increase in CMI values. While main- taining correct classifications, the predictions become less confident yet more informative. Figure 6 further illustrates the convergence of centroid trajectories, thereby validating the convergence of the alternating optimization algorithm.
(a) Untrained
(b) Epoch
(c) Epoch 10
(d) Epoch 15
(c) Epoch 20
Figure 6: The output probability vectors of the trained DJIP teacher for a three-class classification task visualized on 2-simplex over different epochs within single training run. Three classes are randomly selected from the CIFAR- training set, with 100 samples per class. Temperature scaling with T 4.0 is applied to the softmax outputs to enhance distributional smoothness. In the plots, cross markers denote the class centroids, and the CMI values are computed based solely on these three selected classes.
A more comprehensive comparison of the output probability space between the CE teacher and the DJIP teacher is shown Figure 7 with temperature scaling = 4.0 as well. We observe that clusters corresponding the DJIP teacher become less concentrated.
A.12 THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this work, we only employed LLMs to assist with writing polish and refinement, as well as for literature retrieval and discovery (e.g., identifying related work).
23

---

Published as a conference paper at ICLR 2026
(a) Res4 (Left: CE teacher; Right: DJIP teacher)
Figure 7: t-SNE (van der Maaten Hinton 2008) visualization of features extracted from the CIFAR- 100 training set with 500 samples per cluster in all 100 categories.
24


Reviews:
[{"review_id": "TZ5wRYlCgI", "reviewer": "Reviewer_JMaG", "summary": "This paper proposes DJIP, a plug-and-play framework to enhance KD without modifying teacher model weights. Experiments show that this method has certain effectiveness. Experiments show that this method has certain effectiveness.", "questions": "In addition to the issues mentioned in the Weaknesses, I have a few more concerns:\n\n1. The paper claims orthogonality between DJIP and MCMI since DJIP explicitly optimizes the input space. Could the authors provide quantitative evidence (e.g., gradient cosine similarity between DJIP and MCMI objectives) to support this orthogonality claim?\n2. Does increasing the number of quantization parameters consistently improve performance, or is there a saturation point?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper improves the effectiveness of knowledge distillation without modifying the teacher model’s weights. The work is both interesting and effective.\n2. The paper is well written, and the proposed method is plug-and-play and easy to follow.\n3. The paper provides a detailed and thorough theoretical analysis.", "weaknesses": "1. The application scope demonstrated in this paper is somewhat limited. Could the proposed method be extended to LLMs (e.g., LLaMA) or generative models (e.g., Stable Diffusion)? Discussing applications beyond classification tasks would help enhance the paper’s breadth and contribution.\n2. The JPEG-layer perturbation operates in the pixel space. It is unclear whether such pixel-level perturbations have a significant impact on the feature space. The rationality and reliability of the supervision signal constructed in this manner are therefore questionable.\n3. The paper lacks an explanation regarding the selection strategy for key parameters, such as the frequency of alternating updates.\n4. The paper does not sufficiently discuss several recent SOTA works[1,2,3,4,5,6] on knowledge distillation.\n\n[1] f-Divergence Minimization for Sequence-Level Knowledge Distillation. ACL 2023.\n\n[2] DistiLLM: Towards Streamlined Distillation for Large Language Models. ICML 2024.\n\n[3] MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.\n\n[4] Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. COLING 2025.\n\n[5] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via alpha-beta-Divergence. ICML 2025.\n\n[6] DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models. ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tMJlxdqRl5", "reviewer": "Reviewer_tbBh", "summary": "This paper proposes Differentiable JPEG-based Input Perturbation (DJIP), a plug-and-play framework for improving knowledge distillation (KD) without modifying the teacher model.\nDJIP introduces a differentiable JPEG layer before the teacher network to perturb inputs in a way that maximizes conditional mutual information (CMI) between inputs and outputs.\nThe authors also design an alternating optimization algorithm that jointly optimizes cross-entropy and CMI losses, enabling efficient learning of JPEG quantization parameters.\nExtensive experiments on CIFAR-100 and ImageNet show consistent student accuracy improvements (up to 4.11%) across CNNs and ViTs, and compatibility with many KD baselines.", "questions": "1.\tCould you provide a runtime and GPU memory usage of training DJIP layer?\n2.\tHow sensitive is the method to the hyperparameter λ? Are there any cases where maximizing CMI harms student performance?\n3.\tDid the authors observe any instability during training due to learning rate choice? How robust is DJIP to different optimizer configurations (e.g., step size, momentum, or learning)\n4.\tHave you considered using other differentiable codecs (e.g., differentiable WebP or learned compression networks) as a generalization of DJIP?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tSolid theoretical foundation: The CMI-based alternating optimization is mathematically consistent and improves upon fixed-centroid MCMI.\n2.\tComprehensive empirical validation: Tested on multiple datasets and KD frameworks, with consistent improvements.\n3.\tOrthogonality: Demonstrated compatibility with both MCMI and CKD, suggesting general utility according to the paper.\n4.\tReproducibility: Implementation details and appendices are complete and transparent.", "weaknesses": "1.\tLimited theoretical novelty: The alternating CMI formulation is an incremental improvement over prior MCMI, and the key novelty lies in engineering implementation (JPEG layer).\n2.\tInsufficient qualitative analysis: The paper lacks visualization or interpretation of how and why JPEG perturbations affect teacher responses.\n3.\tLimited performance improvements: As an additional, plug-and-play module, DJIP offers limited performance enhancements to networks, which focuses more on engineering optimizations than theoretical innovations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qi7EsOUn1p", "reviewer": "Reviewer_RUYs", "summary": "The paper introduces Differentiable JPEG-based Input Perturbation (DJIP), a novel framework designed to enhance teacher-student knowledge transfer in knowledge distillation (KD) without modifying the teacher model. \nDJIP uses a differentiable JPEG layer that perturbs the teacher’s inputs to directly increase conditional mutual information (CMI), which improves distillation effectiveness. \nThe paper also proposes a new alternating optimization algorithm to efficiently learn the JPEG layer's coding parameters to maximize the perturbed CMI. \nExtensive experiments on CIFAR-100 and ImageNet demonstrate that DJIP consistently boosts student accuracy by up to 4.11% while being computationally efficient and compatible with standard KD pipelines.", "questions": "1. How does DJIP compare to other input perturbation methods in terms of generalization to different types of data, such as non-image datasets or tasks beyond image classification? Are there any potential limitations or challenges when applying DJIP to these domains?\n\n2. The proposed alternating optimization algorithm is central to maximizing the perturbed CMI. Could you provide more details on how this algorithm scales with larger models or more complex datasets, and whether there are any concerns regarding its stability or convergence in these cases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel technique, Differentiable JPEG-based Input Perturbation (DJIP), which is a plug-and-play framework for improving knowledge distillation (KD) without requiring modifications to the teacher model. \n\n2. The method is extensively evaluated on CIFAR-100 and ImageNet datasets, demonstrating consistent improvements in student accuracy, with gains up to 4.11%. \n\n3. DJIP is computationally lightweight and integrates seamlessly with standard KD pipelines, making it an efficient solution for enhancing knowledge transfer. The method optimizes just the JPEG layer without modifying the teacher model, ensuring low overhead.\n\n4. The proposed method works well across both same-architecture and cross-architecture (CNN-to-ViT) settings, which shows that DJIP can be applied broadly in knowledge distillation tasks, including distilling between heterogeneous model types.\n\n5. DJIP is orthogonal to other state-of-the-art methods like MCMI, as demonstrated by the paper's results. This suggests that DJIP can be integrated with existing KD techniques to further enhance performance, providing flexibility in distillation pipelines.", "weaknesses": "1. While the method is effective in practice, the paper provides limited theoretical analysis of why perturbing the input via the JPEG layer improves distillation beyond just the CMI maximization objective. \n\n2. The method heavily relies on a differentiable JPEG layer, which could limit its applicability to certain use cases or architectures where JPEG compression may not be optimal or desirable. \n\n3. The alternating optimization algorithm introduced for learning the JPEG coding parameters adds a layer of complexity. While efficient, the algorithm may require fine-tuning, and its performance could vary depending on the choice of hyperparameters, such as lambda.\n\n4. The usage of JPEG limits the method within the image domain. The method’s effectiveness might degrade on different types of data, especially those that are not image-based or have different structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S75skvkT2d", "reviewer": "Reviewer_gUa4", "summary": "This paper presents Differentiable JPEG-based Input Perturbation (DJIP), a plug-and-play framework that boosts knowledge distillation by maximizing a frozen teacher’s conditional mutual information (CMI) without altering its weights. DJIP inserts a trainable, differentiable JPEG module before the teacher to learn input-space perturbations, and pairs it with an alternating optimization routine that dynamically updates class centroids to maximize perturbed CMI. On CIFAR-100 and ImageNet—spanning CNNs and ViTs and multiple distillers (KD, DKD, AT)—DJIP delivers consistent student accuracy improvements (up to 4.11%), while composing cleanly with existing methods.\n\nContributions\n1. A lightweight, broadly compatible input-perturbation mechanism via differentiable JPEG compression.\n2. An alternating CMI-maximization algorithm that overcomes fixed-centroid limitations (e.g., MCMI).\n3. Extensive empirical validation demonstrating orthogonality to prevailing techniques and scalability across datasets, architectures, and distillers.", "questions": "1. Computational Overhead & Scaling. How does DJIP’s end-to-end cost scale with input resolution and batch size relative to vanilla KD? Please report latency (ms/img), throughput (img/s), and peak memory across GPU/CPU settings. Have you explored hardware-aware optimizations of the JPEG layer (e.g., fused CUDA kernels, SIMD/Neon intrinsics, TensorRT plugins), and what is the measured speedup?\n2. Format Generalization. Beyond JPEG, have you evaluated differentiable alternatives (e.g., WebP/AVIF or learned compression) to test the mechanism’s generality? A head-to-head comparison controlling bitrate/PSNR would clarify whether gains stem from JPEG’s structure or from compression-induced perturbations more broadly.\n3. Algorithmic Stability. Does the alternating scheme exhibit instability when class centroids shift quickly (e.g., early training, long-tail classes)? Would momentum/EMA, trust-region or proximal updates, or line-search on the CMI objective improve stability and convergence consistency? Any failure cases or sensitivity analyses?\n4. Black-Box Teachers. How does DJIP perform when the teacher is accessible only via queries (API/closed-source), precluding gradient flow? Can zeroth-order/finite-difference estimates, score-matching proxies, or pseudo-label distillation approximate the CMI objective, and with what accuracy–cost trade-offs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality. DJIP reframes JPEG-based input perturbation as a continuous optimization problem, in contrast to prior discrete toggling or adversarial noise schemes. The proposed alternating routine offers a principled advance for CMI maximization/estimation, jointly adapting perturbations and class centroids to better capture teacher–input dependence.\n2. Quality. The evaluation is broad and careful: two large-scale datasets, 10+ architectures (CNNs and ViTs), and 15+ KD methods, with component-wise ablations (Table 5) that isolate the contribution of the differentiable JPEG layer and the centroid updates. A thorough hyperparameter sweep (Appendix A.5) tests sensitivity and demonstrates stable gains across settings, strengthening the empirical claims.\n3. Clarity. Exposition is accessible: the CMI objective is introduced with clear intuition and notation (Sec. 3.2), and the overall pipeline is modularized in Fig. 1, making it straightforward to slot DJIP before a frozen teacher. Algorithm boxes and training pseudocode further improve reproducibility.\n4. Significance. Because DJIP is plug-and-play and lightweight (≈5–12% parameter overhead), it is practical for resource-constrained deployments. Notably, it narrows or even surpasses larger baselines (e.g., a 3B DJIP student outperforming a 14B non-DJIP counterpart), and its benefits persist across diverse backbones and distillers—evidence of orthogonality rather than method-specific tuning. Collectively, these attributes position DJIP as a scalable, implementation-friendly upgrade path for KD pipelines.", "weaknesses": "1. Computational Efficiency. Table 4 reports parameter overhead, but end-to-end latency, peak memory, and throughput under realistic batch sizes/hardware are not measured. These metrics are essential for edge/smartphone deployment. Suggest reporting wall-clock inference (ms/img), GPU/CPU memory footprints, and A/B latency deltas with and without DJIP across batch sizes.\n2. Baseline Diversity. Comparisons emphasize CKD and MCMI, but omit recent non-JPEG perturbation approaches (e.g., GAN/ diffusion-generated counterfactuals, learned augmentors). Including such baselines would clarify whether DJIP’s gains stem from JPEG structure or from perturbative training per se.\n3. Theoretical Limits. Convergence evidence (Fig. 5) is empirical; there is no formal rate or stationarity guarantee for the alternating updates. A brief analysis (e.g., monotonic ascent under bounded curvature, or conditions ensuring convergence to a critical point) would solidify the algorithmic contribution.\n4. Data Efficiency. Experiments rely on high-quality data (e.g., PixMo-Cap); low-data regimes (few-shot/long-tail KD) remain untested. Evaluating DJIP under subsampling, noisy labels, or class imbalance—and reporting data-efficiency curves—would demonstrate robustness where KD is most needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)