DTO-KD: Dynamic Trade-off Optimization for Effective Knowledge Distillation
Authors: Zeeshan Hayder
Ali Cheraghian
Lars Petersson
Mehrtash Harandi
Richard Hartley
Venue: ICLR 2026 Oral (2026)
Keywords: Knowledge Distillation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: DTO-KD

Abstract: Knowledge Distillation (KD) is a widely adopted framework for compressing large models into compact student models by transferring knowledge from a high-capacity teacher. Despite its success, KD presents two persistent challenges: (1) the trade-off between optimizing for the primary task loss and mimicking the teacher's outputs, and (2) the gradient disparity arising from architectural and representational mismatches between teacher and student models. In this work, we propose Dynamic Trade-off Optimization for Knowledge Distillation (DTO-KD), a principled multi-objective optimization formulation of KD that dynamically balances task and distillation losses at the gradient level. Specifically, DTO-KD resolves two critical issues in gradient-based KD optimization: (i) gradient conflict, where task and distillation gradients are directionally misaligned, and (ii) gradient dominance, where one objective suppresses learning progress on the other. Our method adapts per-iteration trade-offs by leveraging gradient projection techniques to ensure balanced and constructive updates. We evaluate DTO-KD on large-scale benchmarks including ImageNet-1K for classification and COCO for object detection. Across both tasks, DTO-KD consistently outperforms prior KD methods, yielding state-of-the-art accuracy and improved convergence behavior. Furthermore, student models trained with DTO-KD exceed the performance of their non-distilled counterparts, demonstrating the efficacy of our multi-objective formulation for KD.

Published as a conference paper at ICLR 2026
DTO-KD: DYNAMIC TRADE-OFF OPTIMIZATION FOR EFFECTIVE KNOWLEDGE DISTILLATION
Zeeshan Hayder' 12, Ali Cheraghian 1.2 Larrs Petersson', Mehrtash Harandi', Richard Hartley?
1 Data661 / /SIRO, Australia
2 *Australian National University (ANU), Australia
Monash University, Australia
ABSTRACT
Knowledge Distillation (KD) is a widely adopted framework for compressing large models into compact student models by transferring knowledge from high- capacity teacher. Despite its success, KD presents two persistent challenges: (1) the trade-off between optimizing for the primary task loss and mimicking the teacher's 's outputs, and (2) the gradient disparity arising from architectural and rep- resentational mismatches between teacher and student models. this work, we propose Dynamic Trade-off Optimization for Knowledge Distillation (D)O-KD), a principled multi-objective optimization formulation of KD that dynamically bal- ances task and distillation losses at the gradient level. Specifically, DTO-KD re- solves two critical issues in gradient- baseed KD optimization: (i) gradient conflict, where task and distillation gradients ent dominance, where one objective suppresses learning progress on the other. Our method adapts per-iteration trade-offs by leveraging gradient projection tech- niques to ensure balanced and constructive updates. We evaluate DTTO-DD on large-scale benchmarks including ImageNet-IK for classification and COCO for object detection. Across both tasks, DTO-KD outperforms prior KD methods, yielding state-of-the-art accuracy and improved convergence behavior. Further- more, student trained with DTO-KD exceed the performance of their non-distilled counterparts, demonstrating the efficacy of our multi-objective formulation.
directionally misaligned, and (ii) gradi-
INTRODUCTION
Deep neural networks have demonstrated impressive performance across a wide range of computer vision tasks; however, their practical deployment is often hindered by substantial computational and memory demands, particularly on resource-limited platforms such as edge devices and mobile hard- ware. This has motivated increasing interest in model compression techniques that reduce network complexity without sacrificing accuracy. Among these, knowledge distillation (KD) (im 2017
Gao et al., 2018 Qiu et al.] 2002) Zhou et al. (020)) has emerged as an effective strategy, in which compact student model trained under the guidance of large, pre-trained teacher model. By transferring informative representations or outputs from the teacher, the student can achieve compa- rable performance with significantly fewer parameters, making it more amenable deployment in constrained environments A standard KD framework typically combines a task-specific objective (e.g., classification detection loss) with auxiliary distillation loss that facilitates knowledge transfer from the teacher to the student.
Early studies on knowledge distillation (KD) (Hinton relied on training the student model using the teacher's 's output predictions as supervisory signals. While effective to some extent, this strategy is inherently limited, as the teacher's final outputs pro- vide a highly compressed representation of its knowledge, and relying solely on logits constrains the richness transferable information. To overcome these shortcomings, subsequent KD meth- ods (Romero et al. 2015 Chen et al. 2020 HHoo et al. 2019a) moved toward leveraging intermediate feature representations from the teacher, allowing for more expressive and flexible knowledge trans- fer. Such feature- based distillation methods often depend on heuristic design choices and introduce additional hyperparameters that require careful, task-dependent tuning. Nevertheless, despite these improvements, recent feature-level KD approaches (Chen et al. 2022 2021
2015: Zhang et al. 2018 primarily
Roy Miles & Deng

---

Published as a conference paper at ICLR 2026
1.0
VKD Ours
VKD
Dominance Score (log)
Ours
Conflict Score
0.5
74
0.0
-0.5
-3
-1.0
100
200
300
400
500
100
200
300
400
500
Iterations
Iterations
Figure Gradient Dynamics analysis, comparing the conflict and dominance behavior of the distillation and task gradients. Left) Conflict score is computed as (gdist 9usk),, where more negative guail and shown
values indicate stronger disagreement. Right) Dominance score is calculated as 19us in log-scale, with lower values indicating stronger dominance. DTO-KD achieves lower gradient conflict and more balanced gradient dominance compared to the baseline.
2024) continue to face challenges in effectively distilling knowledge from highly expressive teacher networks into compact student models, largely due to mismatches between the objectives imposed by ground-truth supervision and those induced by the distillation process.
Recent KD methods (Wang al. 2024 Chen et al. 2022) propose heuristic mechanisms for bal- ancing teacher mimicking, yet optimization inconsistency persists as a critical bottleneck to efficient knowledge transfer. The primary issue limiting the performance of these approaches is two-fold. First, Gradient Conflicts (GrC) arise when the gradients of the task-specific objective and the dis- tillation process are misaligned. Second, Gradient Dominance (GrD) occurs when the gradient magnitude of one objective (e.g., either distillation or task specificc dominates the learning process, causing an imbalance. Figure illlustrates these issues by plotting gradient conflict and dominance for our method and that of Roy Miles & Deng 2024) over 500 iterations on the detection task.
To address all of these issues, we propose novel distillation optimization strategy. Specifically, we frame the problem as dynamic trade-off optimization, which not only efficiently resolves gradient conflicts during training but also ensures a Pareto optimal (Lin, 1976) solution. This results in training strategy that eliminates the need for manually tuning hyperparameters to balance the contributions of each loss function. Instead, it dynamically learns the contribution of each loss function, adapting between task task-specific and sttillaaiionsppeiiff objectives throughout the training.
To be more specific, in this paper we propose a closed-form method for determining how to weight the distillation and task-specific losses during training. Unlike the prior work of (Liu et al.) 2023), our approach provides an explicit solution that can be computed efficiently at each step. In teacher-student architectures, where the distillation and task losses evolve rapidly, existing task- weighting methods (Hu et al. 2024; Zheng & Yang 2024) can struggle to adapt, causing weights oscillate or lag behind the changing dynamics. In contrast, our closed-form solution produces an update direction that is jointly aligned with both objectives, ensuring that neither the distillation nor the task loss dominates or interferes with the other. As a result, our method naturally mitigates gradient conflict and yields a more stable and effective multi-objective learning process.
In this paper, we introduce DTO-KD (Dynamic Trade-off Optimization for Knowledge Distillation), a novel multi-objective learning framework that formulates knowledge distillation gradient-level optimization problem. DTTOKD improves the efficiency and effectiveness of knowledge transfer by dynamically modulating the contribution of task-specific and distillation-speeifi objectives during training, removing the need for manual loss weighting or extensive hyperparameter tuning. DTO- KD is trained end to end and demonstrates faster convergence, requiring fewer epochs reach or exceed the performance of state-of -fee-----rrhrrr distillation methods. In summary. the contributions of this paper are as follows:
We propose DTO-KI KD, a dynamic trade-off optimization framework that balances task and distillation losses at the gradient level. This principled approach eliminates the need for fixed loss weighting, enabling adaptive trade-offs during training.
DTO-KD resolves gradient conflict (GrC) and dominance (GrD) via per-iteration gradient balancing approach, leading aligned, balanced updates and improved convergence.
2

---

Published as a conference paper at ICLR 2026
We conduct extensive experiments on both classification and detection benchmarks, achiev- ing state-of-the- performance. Ablation studies confirm the robustness of DTO-KD across diverse distillation setups.
2 RELATED WORK
This section explores KD techniques, focusing on the use of logits, CNN features, and transformer features (or tokens). Additionally, it examines multi-objective approaches relevant to the DTO-KD.
Aligning Predictive Distributions via Logit-Level Distillation: Classical Knowledge Distillation
(KD) primarily relies the teacher's output logits. Ensemble-based collaborative learning (Zhang
et al. 2018), multi-stage distillation with teacher assistants (Mirzadeh et al.) 2020), and decoupled distillate distillation across teacher branches (Zhao et al. (022a) all fall within this paradigm. Most methods use forward KL-divergence to align teacher-student distributions, though this often yields overly smoothed predictions. Reverse KL-divergence (Wang et al. 2025a) emphasizes the teacher's high- confidence modes, while a-3 divergence (Wang et al. (2225)) generalizes both by interpolating be- tween their behaviors. However, logit-based approaches transfer only final-layer information, omit- ting rich intermediate representations and teacher inductive biases. Consequently, students struggle to reconcile teacher outputs with task specific objectives, often leading to limited generalization.
Srature-based Knowledge Distillation (KD) leverages intermediate activations to convey structural knowledge un- available at the logit level (Yang et al. [021]] Xu et al., (020). Early work such as FitNets (Romero) ct al. 2015) introduced stage-wise alignment, enabling the student to mimic deeper teacher features. Subsequent methods refine how features selected and aligned. Margin ReLU filtering (Heo) cet al. 2019a) suppresses redundant activations and improves feature matching, while analysis of connection pathways (Chen et al. 2021) highlights the importance of optimal teacher- layer mappings. Diffusion-based distillation (Huang et al., 2024) reduces noise in student features before transferring knowledge, and norm/direction-awer aware losses (Wang all (024) further enhance feature alignment. Despite progress, feature-level KD still struggles to encode long-range dependencies and global contextual knowledge that essential in modern architectures and increasingly captured by transformer-base models.
Transferring Intermediate Representations via Feature-Level Distillation:
Leveraging Transformer Semantics via Token-Level Distillation: Transformer- bassd Knowl- edge Distillation (KD) focuses on transferring information encoded in self-attention and token inter- actions. DeiT (Tovvron et al. (022) first established efficient token distillation for convolution-fie vision transformers. For detection tasks, token- methods (Song et al., 2021 (022) require the student to replicate teacher tokens, though naive token alignment is often insufficient. More advanced methods incorporate multiple teachers (Ren et al. 2022), manifold- based token align- ment (Hao et al., 2022), and generalized f-divergence formulations (Wen et al. 2023) that flexibly weight the teacher's 's dominant predictions across tokens. However, transferring dark knowledge, such as subtle interactions and contextual semantics encoded in token relations, remains challeng- ing. Approaches such as non-target logit normalization (Yang al., 2023) or two-stage early-layer distillation pipelines (Chen et al.. 2022) offer partial solutions but remain heuristic and fragmented. In contrast, our work proposes unified end-to-end formulation that dynamically balances objectives through trade-off optimization.
Aligning Confticting Objectives via Multi-objective Optimization: Multi-objective optimization (MOO) enables simultaneous optimization of conflicting objectives by seeking Pareto-optimal trade- offs. A simple approach re-weights loss functions based on manually designed criteria (Chen al.) 2018 Kendall et al., 2018). but these methods are often heuristic, ignore dynamic gradient inter- actions, and lack strong theoretical foundations. Gradient manipulation methods (Sener & Koltun 2018 Yu et al., 20200 Liu et al., 2021b 2023) instead combine gradients from different tasks at each step. For example, Sener & Koltun (2018) uses an upper bound for efficiency, Yu et al. (2020) projects gradients to avoid conflicts, Liu et al. (202lb a) provide closed-form solution minimizing average loss, and Liu et al. (2023) introduces a fast dynamic weighting method. Although MOO is explored in multi-task learning, In this paper, we proposed DTO-KD which uniquely applies it to knowledge distillation, formulating it as a dynamic trade-off optimization problem to resolve conflicts between task and distillation objectives.
3

---

Published as a conference paper at ICLR 2026
Dynamic Trade-OfT Optimisation
(Basc)
Light weight Decoder
Texcher
Shags
Liue & it mtlaaiinil + azllttak
a.1.8 + -
Liveli
bynana
Landscage
Teacher
Lupa
Staee
arg mrgminn
Mall
Swin Transformer SaanyyNano/SSllll
Light-weight Decoder
Multi Objective Optimisation
Student Muitterain Fratuma
det Drtecieo Hoad
tasa
V
V dursil!
Light--wiight
Loss Runctions
In DTO-KD, the teacher and student models simultaneously process the input image æ. Each network consists of Swin Transformer with a lightweight decoder. The teacher's features (z;), and the student's (z,), are aligned using multiple lightweight projectors (P) different scales. We formulate training as multi-o multi-objective optimization (MOO) problem and propose a Dynamic Trade- off Optimization module that jointly minimizes the distillation loss Ldisill and the task-specific loss Lrak guiding them toward Pareto optimality.
Figure 2:
3 METHOD
We introduce a Dynamic Trade-off Optimization for Knowledge Distillation (DTO-KD), with specific focus on resolving the conflicting objectives in the KD process.
Problem formulation. We aim to transfer knowledge from high-capacity teacher model with pa- rameters Φ more compact model student, with parameters 0, focusing mainly on classification and detection tasks in visual recognition. We show the training data with S = { ) with æ; E IRd being the i-th input instance and yr the corresponding target (e.g. class label, bounding box). Our goal is to train the student model to effectively mimic the behavior of the teacher model over the dataset S. Figure shows an illustration of our proposed framework.
Effectively performing knowledge distillation requires balancing two objectives: the student must learn from two supervisory signals (e.g., one from the teacher and one from the task). We represent the teacher's ' loss as Ldistill and the task's loss as Lusk. While we will define these more specifically for image classification and object detection in the appendix, provide their general forms here:
Liiitinl (0) 2 E( Cassuu (z)
Lask (0) B( lask (ar 0), (xx; xb)
(2)
The conventional KD approaches (e.g., (Hu al. 2024 model by optimizing the loss as
Zheng Yang 2024 train the student
(0) A a Ldistil (0) + ag (0))
where 01, O2 E R. are the combination weights and hyperparameters of the model. The gradient of Liar (0) is
Soot (O) @19dis Q29uskk
where Juist Adsstin () and Jask VI bask (0) are the gradients of the distillation and task losses, respectively. Minimizing loss in Equation 3 for joint training introduces the following challenges:
Gradient Conflict (GrC). This occurs when the gradients of the distillation loss and the task loss conflict with each other. Mathematically, GrC happens when (gaist Jtskk < 0. During the optimiza- tion the total loss (0), the occurrence of GrC leads to conflicting gradient updates. Specifically, both
the total gradient Poo may contradict either Jdist Jlask, causing detrimental effects on objectives. This conflict can exacerbate the learning dynamics, particularly in complex vision tasks such object detection, by introducing unnecessary complexity into the training process.

---

Published as a conference paper at ICLR 2026
Gradient Dominance (GrD). It arises when the gradients have significantly different magnitudes, leading one to dominate the update. When minimizing (0), this imbalance may cause one ob- jective to be completely neglected, as the update direction is primarily determined by the larger gradient, which can be estimated as Ilauel I!auall Lastly, tuning the hyperparameters an and 0g might become extremely tricky as the norm of gradients varies throughout optimization.
To address the aforementioned challenges, in KD. Specifically, we formulate the training process as optimizing the objective vector Lm = (L-iisnn (0): Lusk (0)) The goal is to find a solution 0 on the Pareto front, i.e., solution that is not dominated by any other parameter vector O. Formally, O" is Pareto optimal if is no such that
advocate for the use of multi-objective optimization
ddissin (0) Linkk (0) )0)
(L-distil (0*) Leask Lessk(O*)
(5)
The notation z b here means that vector achieves lower value for all its elements simultane- ously over b. As we will discuss in the next section, formulating KD using the proposed algorithm addresses both GrC and GrD by aligning the gradients. Furthermore, the use of MOO mitigates the difficulty of hyperparameter tuning, as it eliminates the need to manually define and ag.
3.1 KD AS A DYNAMIC TRADE-OFF OPTIMIZATION
Inspired by Liu et (2023), we followed two stage approach for learning the optimal trade-off between conflicting objectives during the model training.
stage 1 and at time we update the student model via 0.+1 = 0, - 79t, where vj E RR+ is the learning step size. We define the rate of improvement for the distillation and task losses as:
Stage 1:
Vdist (g:)t( Luissin (0.) Liissnt (0z+1)
Laiisin 0.)
Trask (gt) Luukk (O,) Lrask (0.+1)
I Lank (0,)
In essence, Fais (gt) and Vais measure how much each loss can be improved by moving the parameters with 79:- A larger value of rai rusk implies the associated task has been improved more.
Stage 2: In stage 2, goal is to determine an update gi that maximizes the improvement over the worst-case rate. This can be achieved using a min-max optimization as:
max 9.ER" &E (dsst
min
(g.)
2 llg.112.
Here, R4 is weighting hyperparameter. As shown in Liu et al. 2023). the solution of Equa- tion 7 can be obtained via solving its dual problem as (see proposition 3.1 in Liu et al. 2023 Define (TI, TT) the simplex A (i.e., IT + T2 1, N1, 22 0), and let Jr Rax be
J. [ log (Ldisti( (0.) | V log (0.))
Then
1 mEA 2
π. arg min J
and g: = J π. V log (Ldistin (0.) + 72V V log (l,)(())))
Theoretical Properties. The problem formulation Equation (9) admits an analytical solution, unlike the general case studied in Liu et al. (2023). In this part, we establish key theoretical proper- ties of the obtained update direction g"
Theorem 3.1 (Closed Form Solution). Let J. = [Vlog (laiau(O))) V log (I (0.)) E Rax2 The closed-form solution to the optimization problem
π arg min ~ ||]
s.t. Ti n2
(10)
5

---

Published as a conference paper at ICLR 2026
is given by
922
912
1
(11)
911 + -2112
911
912
T2
(12)
911 922
2912
where G = J. Jr is the Gram matrix:
G =
911
912
921 922
with elements
911 V log (Luisnit (0:))
(13)
912 = 921
V log (Laissnt (O)), V log (O.))
(14)
922
V log (Lusst (0:))
(15)
The closed-form nature of this solution allows for efficient computation of the optimal weighting factors. One key property of the derived solution is that the update direction aligns with both ob- jectives, ensuring that both the distillation and task losses are reduced simultaneously. This directly addresses GrC by preventing destructive interference between the two gradients.
Corollary 3.2 (Alignment of g²), Define gi = V log (Ldiinn( 0c)) and g2 = log (0.): Then the update direction g" Ti + Teg2 for π defined in 17 is aligned with both and g2-
Another key property of the proposed solution that enforces equal contribution the update direction to both gradients, effectively addressing GrD.
Corollary 3.3 (Equal Contribution of g" to Both Losses). In Corollary 3.2 showed that
(9' (g" 911922 912
1|
-92ll2
This implies that the update direction contributes equally to the descent of both the distillation and task losses, effectively mitigating gradient dominance.
An important aspect of any gradient- -aaed optimization method is ensuring that update magnitudes remain within a controlled range to prevent vanishing or exploding gradients. Our solution satisfies both lower and upper bound on l|g" Il. ensuring stability during training.
Corollary 3.4 (Lower Bound on ||g []). The norm of the optimal update direction g" satisfies the lower bound:
lgg 1|
min(Ilgrll (Ig2ll)
(16)
/2
This implies that the update magnitude remains controlled and does not collapse under gradient imbalance.
Corollary 3.5 (Upper Bound on ||g")), The norm of the optimal update direction g" satisfies the upper bound:
||g1
gzll
||g'||
(17)
|1g
12
As such, the magnitude of the updates does not grow excessively with different gradient scales.
Finally, we observe that the algorithm's 's convergence is ensured by the general theoretical frame- work outlined in Liu et al. (2023). As our formulation aligns with it, the proposed optimization is guaranteed converge a Pareto optimal front
Practical Implementation. The detailed algorithm for the proposed DTO-KD approach is de- tailed in Algorithm [ The distillation and task weights æ are initialized to 0.5. The algorithm begins by initializing the teacher extracts latent features from both for each training batch. The DistillHead and TaskHead refer to specific heads learning distillation and the task, respectively. computes the distillation and task
frozen model and the student trainable model, and then

---

Published as a conference paper at ICLR 2026
Algorithm Dynamic Trade-off Optimisation for KD
1: Inputs: Dataset S {x),,,)) y:)....;; Teacher fa
2: Initialise: Student fs with 0; Task weight Tdistill = Trask + 2
3: for /.= 1: do
(iterations) (hatch) (latent features) (projection)
4: Tr, Vr = ( yo } ~
5:
Zt: 2s - fe(T,), fz
6:
21: 2. + 21 Pz.
Ldistill
ldiarruu (DisrillHead(s,, 2))
7:
L( = Luss
(loxs vector)
Leask (TaskHeadl 26 Vr)
8:
gr = Tdistill V log (Ldisti(( + Teack V log (Lrask (2))
9:
0.+1 0, gt
(student model learning) (frozen model inference)
10:
L(0 - f. (xx)
Laten (0,) 10:nl
r(g:) = ransuin (g:) )
11:
Tusk (
Less (0,) - Lind (Ec+1)
Lasinl (+))
(update direction)
Luc (0.)
12:
(tt + 1) = म (t) 7m
Tdistsil (t) log (Ldisu((Or)) + Trask (t) log (Lusk (O))
(optimize task weights)
losses, combines their gradients according to the current task weights, and updates the student model accordingly. After each update, the task weights are recalculated in closed form based on the rel- ative improvement of each loss, ensuring balanced optimization that aligns both the distillation and task objectives. Despite having strong theoretical properties, MTL algorithms (Liu et al. 2023), access
including the one have developed above, require access per task gradient, in V log (O.)) This incurs performing two backpropagation per it-
to J = V log (L eration, which is not desired. Instead, one can advocate amortizing the training. This leads an approximation to the algorithm while ensuring that an extra backprop step is not required. In short, the parameters
(Teisttll: Ttack) updated via
π (t + π (t) 7r V Taistall (t) log (L-distill (0:)) + Teask (1) log (Lam (0:))
(18)
The update in Equation (18) does not guarantee E A, one should renormalize it via a softmax function. We have empirically observed that the amortized algorithm comfortably outperforms state- of-the-art KD algorithms with significant improvement over training speed. Specifically, the DTO- KD reaches the top performance of Roy Miles & Deng (2024) with 300 epochs just 240 epochs.
4 EXPERIMENTS
We evaluate DTO-KD on two distinct vision tasks: image classification and object detection. For im- age classification, we adopt a CNN-based teacher model, RegNet 160 (Radosavovic et al. 2020), and use transformer-based DeiT (Tovrron et al. 2022) Small and Tiny student models. For object detection, we employ transformer-basee ViDT Base (Song et al. 2021) as the teacher model, with ViDT-Small, ViDT-Tiny, and ViDT--ano serving the student models. Additionally, assess the robustness of our method, we conduct distillation experiments using Vidt-Small as the teacher.
Implementation details: In DTO-KD, we reformulate model training as a wradient-based dy- namic trade-off optimization problem. For the overall optimization across both classification and detection tasks, we use AdamW with a learning rate of 0.025 and a weight decay of 0.01. For classification, we adopt the training strategy and parameters from DeiT' Touvron et al. 2021a). Ad- ditionally, for data augmentation, we follow the method outlined in Roy Miles & Deng (2024). For learning, we employ AdamW (Loshchilov & Hutter] 2019) with a learning rate of 0.001 and a weight decay of 0.05. For object detection, we adhere to the training methodology from ViDT (Song al.) 2021). DTO-KD is trained using AdamW (Loshchilov & Hutter 2019) with an initial learning rate of 10-4 for the body, neck, and head. We use the same hyperparameters those in the ViDT (Song et al. 202T) transformer encoder and decoder. All experiments are conducted using PyTorch (Paszke al. 2017 framework and executed four NVIDIA H100 GPUs.
7

---

Published as a conference paper at ICLR 2026
Method
Venue
Top@1 1
Teacher
#Param.
RegNetY-160 (Radosvvoiic et al.
2020
CVPR20
82.6
None
84M
CaiT-S24 Touvion cral. DeiT3-B
83,4
None
47M
Touvron
2022
ECCV22
83.8
None
87M
DeiT-Ti Touvron al.
202
ICML21
72.2
None
SM
De (KD) Touvron et al 2121a
ICML21
74.5
Regnety-160
6M
epocits
JCML21
76.6
Regnety-160
6M
CivI-Ti Ren al Manifiodd
2022
CVPR22
74.9
Regnety-600m
6M
Faa et m 2022
NewiPS22
76.5
CaiT-S24
6M
DearKD
74.8
Regnety-160
6M
spocis
CVPR22
77.0
Regnety-160
6M
USKD Yang ct al
2023
ICCV23
75.0
Regnety-160
GM
MaskedKD [Son
2024
ECCV24
75.4
CaiT-S24
SRD Miles Mikollnzyyk V,D-I
AAAI24
77.2
Regnety-160
6M
(oy Miles & Deng 2024
CVPR24
78.3
Regnety-160
6M
DIO-KD
79.7
Regnety-160
D
Touvron 2021a
ICML21
None
22M
DeiT: (KD)
Touyron
ICM121
81.2
Regnety-160
22M
1000 epochs
ICML21
82.6
Regnety-160
22M
CivT-S (Ren 2022
CVPR22
82.0
Regnety-4gf
22M
DearKD
Chen
CVPR22
81.5
Regnety-160
22M
1000 spochs
CVPR22
82.8
Regnety-160
22M
USKD Yang 2023
ICCV23
80.8
Regnety-160
22M
MuskedKDD Son
ECCV24
81.4
Deit3-B
22M
SRD Miles & Mikolijzykk
2024
AAA124
82.1
Regnety-166 160
22M
V.D-S
Roy Mils Deng 2024
CVPR24
82.3
Regnety-160
22M
(SO-KD(S)
83.1
Regnety-160
22M
Table 1: Object Classification task: DTO-KD the ImageNet- 1K dataset. Unless specified, each model is only trained for 300 epochs.
4.1 OBJECT CLASSIFICATION USING IMAGENET-IK DATASET
We conducted extensive experiments the ImageNet- IK dataset, using the RegNetY-160 (Ra- dosavovic et al. 2020) model, pre-trained on the larger ImageNet-21K dataset, as the teacher to facilitate robust knowledge transfer. Two student models, DeiT-tiny and DeiT-small, were trained for 300 epochs on ImageNet-11 K, and their performance was compared against existing state-of-the. art methods. As shown in Table our approach demonstrates significant improvements in accuracy for both student models. Specifically, DTTO-D outperforms the baseline Touvron et al. (2021a) by 5.2 percentage points (pp) for the tiny model and 1.9 pp for the small model. Additionally, DeiT- tiny surpasses the previous state-of-the--e method Roy Miles & Deng (2024) by 1.4 pp, indicating substantial enhancement in classification accuracy. For DeiT-small, our approach achieves 0.8 pp accuracy improvement over Roy Miles & Deng 2024)..
Additionally, compared to the baseline Touvron et al. 2021a), which was trained for 1000 epochs, DTO-KD achieves 3.1 percentage point (pp) improvement for the tiny model and 0.5 pp im- provement for the small model with just 300 epochs. This highlights the efficiency of our approach, demonstrating its ability to deliver competitive performance in significantly less training time, mak- ing both effective and scalable.
4.2 OBJECT CLASSIFICATION USING CIFAR- 100 DATASET
Conventional knowledge distillation methods are typically evaluated on both homogeneous and het- erogeneous CNN architectures using the CIFAR- 100 dataset. To position DTO-KD against these ap- proaches, we benchmarked it following the protocols of prior KD works (Chen et al [021 Wang et 2024). Table 2 shows that DTO-KD also achieves superior results and reinforces superiority, establishing new SOTA by outperforming previous works in both small- and large-dataset settings.
4.3 OBJECT DETECTION
Table demonstrates that our proposed method, DTO-KD, achieves state-of-the-art object detec- tion performance on the MS COCO benchmark (Lin et al. 2014), leveraging the ViDT transformer architecture (Song et al. 2022) for its strong performance and efficiency consumer hardware. DTO-KD consistently improves upon various ViDT variants, enhancing the Swin-nano backbone by 0.7 percentage points (pp), Swin-tiny by 0.5pp. and Swin-small by 1pp. Notably, DTOOKD-
8

---

Published as a conference paper at ICLR 2026
Homogeneous WRN-40-2
Heterogeneous ResNet-32x4 ShuffleNet-Vi
Methods
ReaNet-56 ResNet-20
ResNet-32x4 ResNet-8xx
ResNet-S0 MobileNet-V2
ResNet-32x4
WRN-40-1
ShuffleNet-V2
Teacher Student
72.34
75.61
79.42
79.34
79.42
79.42
71.98
72.50
70.50
71.82
FitNet (Romero
2015
69.21
72.24
73.50
63.16
73.59
73.54
RKD (Park et al
2019
69.61
72.22
71.90
64.43
72.28
73.21
73.45
73.64
74.69
KD Hinton et at 2015
70.66
73.54
73.33
67.65
74,07
74.45
OFD (eo or
20196
70.98
74.33
74.95
69.04
75.98
76.82
CRD
71.16
74.14
75.51
75.11
75.65
DIST
Hoang 0i
20
71.78
74.42
75.79
69.17
75.23
76.08
ReviewKD
CChen el
2021
71.89
75.09
75.63
69.89
77.45
77.78
DKD
Tao er
20226
71.97
74.81
75.44
70.35
76.45
71.07
ReviewKD++
Wang al
2024
72.05
75.66
76.07
70.45
77.68
77.93
(OUTS)
72.35
75.68
76.40
70.90
77.95
78.22
Table 2: Object Classification task: DTO-KD evaluated both homogeneous and heterogeneous
CNN architectures using the CIFAR-100 dataset.
VIDT Model
Epochs
AP
AF
AP
APs
APss
AP2.
#Purams
FPS
Teacher
Swin-hase Song
2021
50
49.4
69.6
53.4
31.6
52.4
66.8
0.IB
9.0
Swin-sano
Song et
2021
50
40.4
59.6
43.3
23.2
42.5
Teken Matching
Son2 et
2024
50
41.9
61.2
44.7
23.6
44.1
58.7
16M
20.0
V.D-nano Roy Miler
50
43.0
62.3
24.8
45.3
60.1
DTO-KD Taan
30
Swin-ting
Sonz et 2021
50
44.8
64.5
48.7
25.9
47.6
62.1
Student
Token-Maltii Malching
50
46.6
66.3
50.1
28.0
19.5
61.3
38M
17.2
V-D-tiny RRy
D/TU-D (may
50
47.4
67.2
51.3
28.0
50.7
65.8
Swin
Swin-small small
Song
2021
50
47.5
67.7
51.4
29.2
50.7
Matching (500
V,D-small Roy Miles Dong 2024
50
48.5
68.4
52.4
30.8
52.2
66.0
DTO-KI (small)
50
49.6
69.4
53.9
31.6
53.1
67.1
Table 3: Object Detection task: Comparison with other detectors on COCO, with student models
distilled from pre-trained ViDT-base. Note that DTO-KD consistently outpeforms all challenging knowledge distillation baseline approaches.
small, with just 61M parameters, outperforms Swin-base (0. parameters) when both are trained from scratch. Additionally, DTO-KD-tiny, with 38M parameters, achieves nearly the same perfor- mance as Swin-small (61M parameters).
4.4 ABLATION STUDIES
Impact different components DTO-KD: We conduct a thorough evaluation the impact of each primary component in DTO-KD, specifically assessing both stages of dynamic trade-off op- timization, and post processing using gradient clipping. These components were introduced to en- hance the knowledge distillation process, and their individual contributions are analyzed in Table The results demonstrate that each stage significantly contributes to the performance of DTOOKD, with all showing a positive effect on the overall effectiveness of the model. Dynamic Trade-off opti-
Dynamic Trade-off Optimization
SaDTO-KD-nano TVVVDD--
Student
ViDT--nan
ViDTT-tiny
Proj Optimization
Grad. Clip
AP
AP
AP,s
VIDT (small)
41.0
59.2
42.8
Teacher
(small)
(buse)
(base)
No Distillation Sone et 7001
40.4
448
43.1
Token Maschina Scant ar 20123
41.5
41.9
45.8
46.5
43.6
62.9
16.6
(oy Milsr tours)
42.2
43.0
45.9
46.9
43.7
63.1
46.8
Table Component's Impact Assessment: An ablation study showing the impact of projector and optimisation. We also applied gradient clip- ping pre-procsssing step to both objectives to see its impact with and without DTO.
Table 5: Distillation from different teachers for the Object Detection task: Comparison of ViDT COCO2017 val set. We report AP for the student models distilled from different teacher models.

---

Published as a conference paper at ICLR 2026
8.0
9.5
Diput CuaiEati
Dijdt Denake
Clusiiicatoon Emrrr
6.0
Localtattinn Enor
8.0
2.0
40-
0.0
Boselino
Totein
V&D
Ourr
Baveloo
Taken
VKD
Que
Spodhs
Figure 3: Effectiveness of the Dynamic Bal- ancing Strategy the object detection and the classification tasks.
Figure 4: Error analysis: Our dynamic trade- off optimisation approach consistently lowers the classification and localisation errors.
mization enables the model handle diverse objectives, and alignment between teacher and student models to facilitate smoother knowledge transfer.
Dynamic Balancing Strategy and л values: To better illustrate our approach, the figure below shows the varying weighting ratios of the distillation loss (Fdistalt) and the task losses (Trask) during training. As illustrated in (a), we evaluate DTO-KD on a classification task using three distinct loss terms, demonstrating its ability dynamically balance these objectives through adaptive weight- ing. In (b), we extend this analysis to object detection with two loss terms, where DTO-KD's gradient-based vector optimization initially prioritizes the distillation loss and progressively shifts focus toward the task-specific loss.
Subtask error analysis: We conduct thorough analysis of both classification and localization errors (Bolya al.) 2020) in the object detection task. DTO-KD outperforms other methods, achiev- ing fewer errors in both areas while maintaining a strong balance between them. Notably, other KD techniques (Roy Miles & Deng, [024 [Song al. (022) underperform in the classification subtask compared to the baseline (Song et al. 2021), highlighting the superior effectiveness of our approach. See Figure for more details.
Distillation from different teachers: Table T demonstrates DTO-KD's strong performance, even with smaller teachers like ViDT-small. This highlights its robustness, adaptability, and efficiency in resource-constraine settings, making it versatile and effective distillation method across different teacher model scales.
5 LIMITATIONS
Like other KD methods, data availability is a bottleneck. DTO-KD is designed for distillation with available data, and extending it to data-free settings, especially for distilling from large pre-trained models, remains an open challenge. Extending DTO-KD to a data-free regime through sample synthesis may be more difficult due its min-mai optimization, which requires data for the training.
CONCLUSION
DTO-KD introduces principled and effective solution to longstanding challenges in knowledge distillation, particularly for transformer based architectures. By dynamically balancing task sseciccc and distillation objectives the gradient level, DTO-KD mitigates supervision conflicts and gradi- ent imbalances that arise from architectural mismatches between teacher and student models. This multi-objective formulation enables more stable and efficient training, resulting student models that not only match but often exceed the performance of their non-distilled counterparts. Exten- sive evaluations on image classification and object detection benchmarks demonstrate that DTO-KD consistently achieves state-of-the -he-the-rt results, setting new standard for gradient-awan distillation methods. These improvements come with minimal computational overhead, making DTO-KD prac- tical for real-world deployment.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. Tide: A general toolbox for identifying object detection errors. In ECCV, 2020.
Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin, Wasserstein Contrastive Representation Distillation. In CVPR, 2020.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling Knowledge via Knowledge Review. In CVPR, 2021.
Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: Data-efficient early knowledge distillation for vision transformers. CVPR, 2022.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer Dy and Andreas Krause (eds.), ICML, 2018.
mlr www.setw.......eeeeeeeeeeeeeeeee www.pppwssssvvee chw.//chen1aa
html
Pengguang Chen et al. Distilling knowledge via knowledge review. In CVPR, 2021.
Mengya Gao, Yujum Shen. Quanquan Li, Junjie Yan, Liang Wan, Dahua Lin, Chen Change Loy, and Xiaoou Tang. An Embarrassingly Simple Approach for Knowledge Distillation. arXiv, 2018.
Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine- ane-grained manifold distillation. In NeurIPS, 2022.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019a.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1921-1930, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. In NeurIPS, 2015.
Chengming Hu, Haolun Wu, Xuan Li, Chen Ma, Xi Chen, Jun Yan, Boyu Wang, and Xue Liu. Less or more from teacher: Exploiting trilateral geometry for knowledge distillation. In ICLR, 2024.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher, 2022.
Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowl- edge diffusion for distillation. In NeurlPS, 2024.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. In CVPR, 2018.
Jiguan Lin. Multiple-objective problems: Pareto-optimal solutions by method of proper equality constraints. IEEE Transactions on Automatic Control, 1976.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021a.
Bo Liu, Yihao Feng. Peter Stone, and Qiang Liu. Famo: Fast adaptive mul- In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt. and S. Levine (eds.). NeurIPS. Curran Associates. Inc., 2023.
titask optimization.
URL

---

Published as a conference paper at ICLR 2026
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne
Zhang. Towards impartial multi-task learning. In ICLR, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Roy Miles and Krystian Mikolajczyk. Understanding the role of the projector in knowledge distil- lation. In AAAI, 2024.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved Knowl- edge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher. AAAI, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976, 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Probabilistic knowledge transfer for lightweight dcep representation learning. IEEE Transactions on Neural Networks and Learning Systems, 32(5):2030-2000)) 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS, 2017.
Zengyu Qiu, Xinzhu Ma, Kunlin Yang. Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In ICLR, 2023.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing Network Design Spaces. In CVPR, 2020.
Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, and Hang Zhao. Co-advise: Cross Inductive Bias Distillation. In CVPR, 2022.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints For Thin Deep Nets. In ICLR, 2015.
Ismail Elezi Roy Miles and Jiankang Deng. Vkd Improving knowledge distillation using orthogo-
nal projections. In CVPR, March 2024.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In S. Bengio, H. Wallach. H. Larochelle, K. Grauman, N. CesahBianchi, and Garnett (eds.), NeurlPS, 2018. URL
ngs neurips ccppaper_pppp.p...ppp f files/papers 3001sff1l444222c33aa243333332232
Seungwoo Son, Namhoon Lee, and Jaeho Lee. Maskedkd: Efficient distillation of vision transform- with masked images. In ECCV, 2024.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Bycongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In ICLR, 2021.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Bycongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In ICLR, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10660 2019.
Hugo Touvron, Matthieu Cord, Mathhijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
12

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV, 2022.
Bichen Wang. Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Balancing forget quality and model utility: A reverse kl-divergence knowledge distillation approach for better unlcarning in llms. In Proceedings of the 2025 Conference of the North American Chapter of the Associa- fion for Computational Linguistics (NAACL 2025, Long Papers). pp. 1306-1322 Association for Computational Linguistics, 2025a. doi: 10. 10.18653/v1/2025. http://ww1225..maaongg...oo
Guanghui Wang. Zhiyong Yang, Zitai Wang. Shi Wang. Qianqian Xu, and Qingming Huang. Abkd: Pursuing proper allocation of the probability mass in knowledge distillation via a-9-divergence. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), volume 267 of Proceedings of Machine Learning Research, pp. 65167-65212. 65212. PMLR, Jul 2025b. URL pestps:/pprocer v/ wang25dz html
mlr pr6ss/v2677
Jiabao Wang, Yuming Chen, Zhaohui Zheng, Xiang Li, Ming-Ming Cheng, and Qibin Hou. Crosskd: Cross-head knowledge distillation for object detection. CVPR, 2024.
Yuzhu Wang et al. Improving knowledge distillation via regularizing feature direction and norm. In Ales Leonardis, Elisa Ricei, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), ECCV, 2024.
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence- level knowledge distillation. In Proceedings of the 61st Annuul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 10817-10834, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10. 1/2233/11003..alllloll 605/
URL
/tttps://acllnnhooooyyy.oo
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. ECCV, 2020.
Chuanguang Yang. Zhulin An, Linhang Cai, and Yongjun Xu. Hierarchical self self-supervised aug- mented knowledge distillation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (ICCAI), pp. 1217-1223, 2021.
Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge dis- tillation to self-knowledee distillation: A unified approach with normalized loss and customized soft labels. In ICCV, 2023.
Junho Yim. A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning. In CVPR, 2017.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In H. Larochelle, M. Ran- zato, Hadsell, M.F. Balcan, and H. Lin (eds.), NeurlPS, volume 33, 2020. URL neur cc httpp//rrrrieeeeep/e//////a file/ TTte://8Baccfraaaeee333333022020 Papere pdf
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep Mutual Learning. In CVPR, 2018
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022a.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp- 11953-11962, 2022b.
Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. In The Twelfth International Conference Learning Representations (ICLR 2024), 2024. URL hattps://openrett
Zaida Zhou, Chaoran Zhuge, Xinwei Guan, and Wen Liu. Channel Distillation: Channel- At- tention for Knowledge Distillation. In ICCV, 2020. URL http:: // arxiv. org/ abs abs//066.
01 68.
13


Reviews:
[{"review_id": "2YMS2fuF2B", "reviewer": "Reviewer_VVM9", "summary": "This paper proposes a practical algorithm to address gradient conflict and gradient dominance issues in knowledge distillation. The core idea is to dynamically balance the task and distillation losses at the gradient level at each optimization step. Experiments show consistent improvement.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis work tries to tackle a common issue in knowledge distillation, which offers it potentially strong impact.\n2.\tThe technical part is generally solid, with detailed theoretical and empirical analysis.\n3.\tThe experimental gain is persuasive.", "weaknesses": "In general, the presentation is a major limitation of this paper.\n1. (Major) This work draws inspiration heavily from Liu et al. (2023). Although the authors do discuss the difference between the two studies and the unique contribution of this work, such discussion should be placed earlier and more concentratedly, and beyond simply stating “first time applying the methodology to knowledge distillation field”. Do the authors provide some novel theoretical contribution than Liu et al. (2023)? This should be highlighted in the paper. In the worst case, this can raise concerns about the novelty of this study.\n2. (Minor) In the early part of the paper, the proposed method is only described as dynamically balancing losses. However, exactly how such balancing is done remains unclear until the main method section. Furthermore, the main method section has only formal narrative, lacking a more heuristic description to help readers quickly grab the idea. Some intuitive explanations in the introduction would help.\n3. (Minor) The introduction figure only shows experimental improvement. Adding some illustrations of either the two gradient issues or how the proposed method works would help.\n4. (Minor) The majority (left) part of Figure 2 does not show enough information, as this is generally a generic framework for feature-based knowledge distillation. On the other hand, the right part is with too small font sizes. The authors should re-balance the organization of this figure.\n5. (Minor) The 4th paragraph of the introduction feels out of place. It does not connect closely to the previous and the following content.\n6. (Minor) It appears that the authors are not using \\citet and \\citep correctly for in-text and parenthesis citations.\n7. (Minor) Some related work is recommended to be discussed such as [1, 2]\n\n[1] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML 2025\n\n[2] f-Divergence Minimization for Sequence-Level Knowledge Distillation. ACL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WM8Xifhk1Q", "reviewer": "Reviewer_EHrN", "summary": "This paper proposes DTO-KD, a new framework that formulates knowledge distillation (KD) as a multi-objective optimization problem. It dynamically balances the task loss and distillation loss at the gradient level, eliminating the need for fixed or manually tuned loss weights.\nDTO-KD addresses two critical issues—gradient conflict (GrC) and gradient dominance (GrD)—by using gradient projection and adaptive weighting to ensure Pareto-optimal updates.", "questions": "- The paper describes Stage 2 as a dual problem that mathematically subsumes Stage 1 (Eq. 7–9). If Stage 2 provides the final optimization, it is unclear why Stage 1 is still experimentally isolated in Table 3. The authors should clarify how ablation on Stage 1 remains meaningful if Stage 2 inherently replaces or dominates it.\n- The study mostly focuses on transformer-based students, although CNNs are still used widely.\n- The authors can compare the proposed algorithm with many more benchmark algorithms on small-scale datasets (e.g. CIFAR-100).\n- The paper lacks a statement on releasing the source code.\n- Several references appear malformed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Applying multi-objective optimization theory to KD with closed-form update rules represents a novel and mathematically grounded extension beyond prior heuristic balancing methods.\n- The paper is well-structured and clearly motivates the link between gradient conflict/dominance and suboptimal KD.\n- DTO-KD provides a generic, architecture-agnostic optimization framework", "weaknesses": "The experimental validation is limited, as the paper does not include results on small datasets or across a broader range of teacher–student architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5n5gmXh29S", "reviewer": "Reviewer_6Esk", "summary": "The paper introduces DTO-KD (Dynamic Trade-off Optimization for Knowledge Distillation), a multi-objective optimization framework that improves how student models learn from teachers in knowledge distillation. Traditional KD methods face two main issues:\n\n1. A trade-off between optimizing the primary task loss and imitating the teacher’s outputs.\n2. Gradient disparity, caused by mismatches in architecture and representation between teacher and student models.\n\nDTO-KD dynamically balances these objectives at the gradient level using gradient projection techniques to resolve:\n\n1. Gradient conflict — when task and distillation gradients point in conflicting directions, and\n2. Gradient dominance — when one objective overwhelms the other.\n\nBy adapting trade-offs per iteration, DTO-KD ensures balanced and constructive learning updates. Experiments on ImageNet-1K (classification) and COCO (object detection) show that DTO-KD achieves state-of-the-art accuracy, faster convergence, and produces student models that outperform non-distilled baselines, validating the effectiveness of its multi-objective formulation.", "questions": "No question for this paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework addresses trade-off optimization between task and distillation losses at a gradient level. By doing this, the proposed framework avoids hyperparameters for the distillation and task losses and enables dynamic learning for model training.\n\n2. The proposed KD algorithm is a simple but effective method to align the gradients from the task and distillation losses for solving gradient conflict and gradient domainace issues in KD settings.\n\n3. The paper demonstrates the effectiveness of the proposed framework through comprehensive experiment evaluations. The proposed framework outperforms the state-of-the-art KD methods in both classification and detection tasks.", "weaknesses": "1. The proposed framework can only be used for a scenario where training data is available. The proposed framework is not suitable for data-free KD paradigms. In other words, the proposed framework is not scalable to other KD paradigms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)