Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Authors: Zhaowei Chen
Borui Zhao
Yuchen Ge
Yuhao Chen
Renjie Song
Jiajun Liang
Venue: ICML 2025 poster (2025)
Keywords: knowledge distillation
Primary area: applications->computer_vision

Abstract: Online Knowledge Distillation (OKD) methods represent a streamlined, one-stage distillation training process that obviates the necessity of transferring knowledge from a pretrained teacher network to a more compact student network. In contrast to existing logits-based OKD methods, this paper presents an innovative approach to leverage intermediate spatial representations. Our analysis of the intermediate features from both teacher and student models reveals two pivotal insights: (1) the similar features between students and teachers are predominantly focused on the foreground objects. (2) teacher models emphasize foreground objects more than students. Building on these findings, we propose Asymmetric Decision-Making (ADM) to enhance feature consensus learning for student models while continuously promoting feature diversity in teacher models. Specifically, Consensus Learning for student models prioritizes spatial features with high consensus relative to teacher models. Conversely, Divergence Learning for teacher models highlights spatial features with lower similarity compared to student models, indicating superior performance by teacher models in these regions. Consequently, ADM facilitates the student models to catch up with the feature learning process of the teacher models. Extensive experiments demonstrate that ADM consistently surpasses existing OKD methods across various online knowledge distillation settings and also achieves superior results when transferred to offline knowledge distillation, semantic segmentation and diffusion distillation tasks.

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Zhaowei Chen !
Borui Zhao
Yuchen Ge Yuhao Chen
Renjie Song'
Jiajun Liang'
Abstract
1. Introduction
In conventional offline knowledge distillation paradigms (Hinton et al., 2015; Tian et al., 2019; Chen al., 2021), pre-trained teacher model provides soft supervision to guide student training through post-hoc knowledge trans- fer. This two-stage paradigm inherently suffers from com- putational redundancy and temporal decoupling between teacher-student interactions. contrast, recently proposed Online Knowledge Distillation (OKD) methods simplify to one-stage distillation where both teacher and student models learn simultancously from scratch, while achieving competitive or even superior performance (Guo et al., 2020; Qian et al., 2022; Song et al., 2023; Zhang et al., 2023; 2018b). Typically, Deep Mutual Learning (DML) (Zhang et al., (b) encourages each network to mutually learn from each other by matching their output logit via KL. di- vergence. In this paper, propose novel Asymmetrie Decision-Making strategy that unifies consensus and diver- gence feature learning to achieve better OKD performance.
Online Knowledge Distillation (OKD) methods streamline the distillation training process into single stage, eliminating the need for knowl- edge transfer from a pretrained teacher network to a more compact student network. This paper presents an innovative approach to leverage inter- mediate spatial representations. Our analysis of the intermediate features from both teacher and student models reveals two pivotal insights: (I) the similar features between students and teach- ers predominantly focused on foreground ob- jects. (2) teacher models emphasize foreground objects more than students. Building on these find- ings, we propose Asymmetric Decision-aaiigg (ADM) to enhance feature consensus learning for student models while continuously promot- ing feature diversity in teacher models. Specif- ically, Consensus Learning for student models prioritizes spatial features with high consensus relative to teacher models. Conversely, Diver- gence Learning for teacher models highlights spa- tial features with lower similarity compared to student models, indicating superior performance by teacher models in these regions. Consequently, ADM facilitates the student models to catch up with the feature learning process of the teacher models. Extensive experiments demonstrate that ADM consistently surpasses existing OKD meth- ods across various online knowledge distillation settings and also achieves superior results when applied to offline knowledge distillation, semantic segmentation and diffusion distillation tasks.
This paper investigates a fundamental question: How does intermediate feature alignment affect model behavior in on- line distillation? Through systematic analysis using Class Activation Mapping(CAMM (Zhou al., 2016) (Figure 1 and 2), reveal pivotal findings: (1) the similar features between student and teacher models are pre- dominantly concentrated on foreground objects, suggest- ing students prioritize learning these "easy" patterns. (2) teacher models highlight foreground objects more than students, indicating an underutilized capacity for discrimi- native feature discovery. Surprisingly. the most informative divergent features (which may explain teacher superiority) still concentrate on foreground areas rather than background contexts.
Building on these insights, we propose Asymmetric Decision-Making (ADM). It is a unified framework that coordinates consensus enhancement and divergence explo- ration through spatially-awar feature modulation. Unlike conventional symmetric distillation, ADM implements role- specific learning strategies: For students, amplify feature attribution in teacher-aligned foreground regions to accel- erate confidence building. For teachers, we intensify explo- ration of under-activated foreground patterns to avoid pre-
'Equal contribution 'Jiov Technology UUniversity Correspondence to: Zhaowei
of Southern California. Chen echhowei.chen@@iiive cyliang18 Bitiangl@@mail.com. .mail.com>.
Jiajun Liang <tra-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Figure 1: Observation of Discrepancy Regions between Teacher and Student Features. The first row shows that discrepancy regions between teacher and student's features are more concentrated on foreground object regions after Vanilla training. In contrast, the second row displays discrepancy regions are mainly found background regions after ADM training. Best viewed in color.
We propose simple but effective learning strategy named ADM, which allocates different attention to fea- ture maps based on spatial similarity. ADM encourages student models to boost the attribution scores on fore- ground objects while reinforce the strengths of teacher models continually.
609 inald dae Vanlls die mettoo
6:00
2000 21
160
Comprehensive experiments are conducted verify effectiveness, including online KD, offline KD, se- mantic segmentation and diffusion distillation. It is shown that the proposed method consistently outper- forms state-of-the-an OKD methods.
(a) mloU of CAM regions.
(b) mloU similar and discrep- ancy regions.
Figure 2: Analysis of the Teacher and Student Models' Intermediate Features. We load a pre-trained ResNet101 model and obtain its CAM as the Target Object Regions (a) mloU of CAM regions. union (mloU)
2. Related Work
(dubbed TOR) on ImageNet. We calculate the mean intersection between TOR and the CAM of intermediate ResNet-34, ResNet-18 18 models saved during the training process. We ob- serve that ResNet-34 is more concentrated on object regions than ResNet-18. (b) mloU similar and discrepancy fea- ture regions. We observe that the similar regions between ResNet-34 and ResNet-18 18 predominantly focus on TOR and substantial proportion of discrepancy regions are also located within the TOR.
Knowledge Distillation KD aims transfer knowledge acquired by effective but cumbersome teacher model to smaller compact student model. KD has attracted wide interest in vision and language applications (Meng et al., 2022; Niu et al., 2022: Sanh et al., 2019). KD can for- mulated in logits-based (Hinton et al., 2015; Zhao et al., 2022: Huang et al., 2022; Sun al., 2024), feature-basee (Heo al., 2019; Chen et al., 2021; Li, 2022; Xiaolong et al., 2023; Huang et al., 2024), and relation-based (Tian et al., 2019). Recent studies have focused on automating the process of discovering effeectvee knowledge distillation strategies or architectures (Li et al., 2024; 2023; Dong et al., 2023; Hao et al., 2024). Besides, for the foreground and background imbalance in object detection, FGD (Yang et al., 2022) proposed use ground-truth boxes separate the images and guide the student to focus erucial pixels and channels through spatial and channel attention. GID (Dai et al., 2021) select the discriminative instances with higher scores for distillation object detection. These two meth- ods under comparison primarily rely on bounding boxes to explicitly discern the foreground and background for distil-
mature convergence to student-level representations. This strategic asymmetry creates a dynamic where students pro- gressively close the performance gap while teachers con- tinuously unveil new discriminative features. Our main contributions can be summarised as follows.
Through interpretability-driee analysis, we identify the paradoxical coexistence of foreground-centrie fea- ture consensus and teacher-student attribution gaps, challenging conventional assumptions about knowl- edge transfer in online settings.
2

---

Asymmetric Decksion-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
lation, which markedly contrasts with our proposed method.
sists of the Cross-Entropy (CE) loss and the Knowledge Distillation (KD) loss. Let D (xi, Ir W be a train- ing set containing N images and C' categories of labels. Yi E {1,...,
Online Knowledge Distillation The traditional offline KD framework needs pre-trained and fixed teacher model. Online KD methods are more attractive because the train- ing process is simplified to a single stage (Li et al., 2022; 2021; Lin et al., 2022; Yang et al., 2023). Deep Mutual Learning (DML) (Zhang et al., 2018b) enables students to share knowledge from each other's predictions to achieve distillation without teachers. KDCL (Guo et al., 2020) inte- grates the output of multiple students under different data augmentations as soft labels to guide students to optimize. SwitOKD (Qian et al., 2022) attempts to close the accuracy gap by slowing down the training of the teacher during train- ing. SHAKE (L.i& Jin, 2022) bridges offline and online knowledge transfer by building an extra shadow head as proxy teacher model to perform mutual distillation with the student model. Recently proposed online KD methods mainly focus on logits-based. In this work, ADM unifies consensus and divergence feature learning for teacher and student models, which leverages the spatial similarity be- tween teacher and student models, thus implicitly distin- guishing between the foreground and background. More- over, the foreground and background in our method embody crucial attribute, representing simple and hard regions respectively. Consequently, our approach facilitates the stu- dent model to learn from the teacher model progressively, following a gradient of difficulty that ranges from easy to challenging tasks. This constitutes an innovative perspec- tive that has not previously been proposed in the existing literature.
is the ground-truth truth label of the image 2,, The m-th model (m E , .M)) obtains its output logits z = f(x; (T²) E Re, where Imh is the training parameters of the model m. The classification loss is calculated by Cross- Cross-Entropy:
exp 4
Lee (z" (x"",))) .//)) -log
(1)
Ec=1 exp (2
The KD loss usually minimizes the KL divergence between the probabilistic outputs of different models:
Lid = - KL (DTT. -p") = 72 > pe log PC
(2)
Pe
where pmh "",p" RC are the soft logits produced by models and n. The soft logits are calculated by:
exp(z" /T)
p== o /T) Ee exp(2x/7)
(3)
where is the softmax function, and is the temperature
factor to control the softness of logits. Then the DML loss
is formulated as follows and À is the trade-off weight:
M
daml E + ACka).
(4)
3.2. Online Feature Distillation
Diffusion Distillation Diffusion distillation methodologies can be taxonomically categorized into step distillation and model distillation. (Salimans & Ho, 2022; Meng et al., 2023) established the iterative progressive distillation frame- work, compressing DDPM sampling steps from 1,000+ to 4 through cyclic teacher-student optimization. The Dream- Fusion paradigm (Poole et al., 2022) first bridged text- to-3D generation through differential rendering and SDS losses. Recent work (Nguyen & Tran, 2024; Yin et al., 2024) achieves 40x accelcration via adaptive step anneal- ing. Adversatial Distillation (Sauer et al., 2025) integrated adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps. (Kim al., 2025) proposed block pruning and feature distillation for low-cost cemerrl-purpose Text-to-image (T2D) generation. In this paper, we integrate ADM into both step distillation and model distillation to verify its effecttivenes.
In recent offline KD methods, FitNet(Romero et al., 2014) combines the feature-based loss with the logit-based loss to improve the performance. In online KD, we observed that making the teacher mimic the student's features would compromise the diversity of the teacher's features, thereby leading to detrimental effects on performance. Due to the accelerated convergence and higher accuracy of teacher models, teachers exhibit superior features that effectively facilitate the learning process for students. Thereby, we simply use the following MSE loss for the student feature distillation but detach the gradient of teacher feature distil- lation:
H
W C
L =
Σ b=1
ΣΣΣ d1
o (FF
((Fbje))
B HWC
(5)
where B is the batch size. i, j. are the location of correspond- ing feature map with width W and height H. C' represents the number of channels. Fs and F are the intermediate features of student and teacher models, respectively. The function is the I Xx] I convolutional adaptation function to adapt F to the same dimension Fr.
3. Preliminary
3.1. Online Knowledge Distillation
Generally, OKD replaces the commonly used pre-trained teachers with peer-trainable models. The training loss con-
3

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Teacher
Pt
Lige
ADM
1.00
Ft
Eyeur
Lyear
ADM
KLPP-IPP
kL(p.lis.)
cos simularity
Attentinn
A
Student
T+3
Ps
Lice
F
Lie
Figure 3: Asymmetric Decision-Making in Online Knowledge Distillation. GD means Gradient Detach operation. Consensus Learning for student models and Divergence Learning for teacher models. The gray arrow represents the baseline of the DML method, while the blue arrow denotes the newly added ADM component. Best viewed in color.
4. Method
optimum, we introduce Lai to the teacher model, allowing it to continuously strengthen the learning of discrepancy regions from the student model, described as:
4.1. Asymmetric Decision-Making in Online Knowledge Distillation
In this section, show how to design Asymmetric Decision-Making in Online Knowledge Distillation. Intu- itively, during the training, the teacher's 's foreground object- centric features should be strengthened while the student features should be changed accordingly. The above analysis suggests that ADM function can be decomposed into two components, namely, Consensus loss Leo and Divergence loss Lai. Ceo is applied to student model, indicating that the higher similarity between the student and teacher models in specific regions, the stronger the supervision we provide for those regions. The similarity matrix is described as:
Ldi = Lee ( (Wary
(1-5 S 1 1 F²)) )²) -P))) 93
(8)
where W is the parameters of the last teacher linear clas- sifier. The Attention Function for teachers is monotone- decreasing function. By summing up both losses as follows, the teacher model constantly guides and drives the student model to learn and catch up. a and 33 are hyper-parameters. We here employ CE loss function in ADM for conciseness, and in addition, as shown in Section 5.7, we have inves- tigated the combination of CE loss with KD loss, which achieved competitive results.
F Ft Fig
= Fij) =
F 17
(6)
Cadm aLco + 3Lais
where FL, F E Rhxwxd, S E Rhxw, d is the channel number. and are the location of the corresponding feature map with width wo and height h. As shown in Figure 3. For simplicity, we only take the penultimate feature before Global Average Pooling (GAP) as input. fro encourages the student to focus on learning simple knowledge rather than concentrating on difficult regions where large discrepancy with the teacher model, described as:
As result, the overall training loss Liotal can be composed of DML loss, feature loss, and adm loss:
Ltotal = Lami + Tfeeat + Lodms
(10)
where 7 is factor for balancing the losses. In this way, via the ADM loss, we have enabled the student to match the teacher network's feature adaptively, thus boosting OKD performance to a great extent.
(1+ 1 (mr) W/
5. Experiment
Lco Loc
W
5.1. Experimental settings
where W is the parameters of the last student linear clas- sifier. vhang is GAP operation. S is the mean value of S along the dimension of h and w. The Attention Function for students a monotone-ing inncrase-nncreasing function. Simultaneously, to avoid the teacher model learning an excessively high similarity with the student model and falling into a local
Datasets. We validated ADM on the following datasets:
CIFAR-10 (Krizhevsky et al., 2009) consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. CIFAR-100 (Krizhevsky et al., 2009) consists of 32x32 color images drawn from 100 classes, which are split into 50k train and 10k test images. ImageNet (Rus-
4

---

Asymmetric Mecksinn-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
sakovsky al., 2015) is large scale image classificaiton dataset that contains 1k classes with about 1.28 million training images and 50k images for validation. Each image is resized to 224x224. On ImageNet and CIFAR-100 100, we adopt the commonly used random crop and horizontal flip techniques for data augmentation. Cityscapes (Cordts et al., 2016) is a dataset for urban scene parsing, comprising 5000 meticulously annotated images. The distribution of these images is as follows: 2975 for training, 500 for validation, and 1525 for testing.
ent seeds and obtain the average accuracy. ADM is plug-and-play module that can effectively enhance the per- formance of existing methods. Note that ADM can perform much better, especially for heterogeneous student-teacher pairs, +0.35% vs +1 +1,4 %. This is because heterogeneous architectures focus on different regions on intermediate fea- tures. It is more discriminative for students to leverage the feature similarity compared output logits.
Backhone
Vanilla DMI. KDCL SwitOKD DMI.+ADM SwiOOKD+ADM
R4
72.33 73.95 74,61
74.49
74.23
74.63
Teacher-Student Pairs To validate the generalizability of different distillation methods, we select a group of popu- lar network architectures to form different teacher-student pairs. By combining different teacher and student net- works, like ResNets (He al., 2016). WWRRN) (Zagoruyko & Komodakis, 2016), VGGs (Simonyan & Zis- serman, 2014), MobileNets (Howard et al., 2017; Sandler et al., 2018), and ShuffleNets (Ma et al., 2018; Zhang et al., 2018a), we can perform distillation between the same ar- chitectures (e.g., ResNet34 (He et al., 2016)-ResNet18(tH et al., 2016) and different architectures (e.g., Res et al., 1ol6--MobbeNee-2(SSaddd introduces no additional trainable parameters. The added computational cost is markedly minimal relative to the orig- inal. For example, in the ResNet34-18 18 experiments on ImageNet, ResNet34 has 3.67G FLOPs and Re 8 has 1.82G FLOPs. ADM only adds 0.52M FLOPs.
ResNet32x4
79.55 79.99 79.84
79.31
79.62
80.01
VGG-8
70.58 72.45 71.78
72.71
72.8
72.94
VGG-13
75.12 76.19 74.63
75.64
76.07
76.50
Table I: Top-1 Accuracy (%) comparison on CIFAR- R- 100 with same architecture style teacher-student pairs. The upper and lower models denote student and teacher, respectively.
Backbone
Vanilla DML KDCL SwitOKD DML+ADM
0.5MobileNetV2
60.39
66.69
65.98
66.68
68.10
WRN-16-2
73.01
73.59
71.54
73.55
73.62
et al., 2018)). ADM
ShuffleNetV1
68.44
73.58
73.91
73.82
74.61
ResNet32x4
79.30
80.08
78.97
79.37
79.97
ShuffleNetV2 ResNet32x4
70.17
75.15
75.00
75.18
75.75
79.38
80.15 79.14
79.90
80.26
Table 2: Top-1 Accuracy (%) comparison on CIFAR-100 with different architecture style teacher-student pairs. The upper and lower models denote student and teacher, respectively.
Implementation Details. Following the settings of previous methods (Qian et al., 2022: Tian et al., 2019), the batch size, epochs, learning rate decay rate, and weight decay rate are 256/128, 100/300, O. 1, and 0.0001/0.0005. respectively on ImageNet/CIFAR-100 100. The initial learning rate is 0.1 on ImageNet, and 0.01 for MobileNetV2, 0.1 for the other students on CIFAR-100 1000 Besides, the learning rate drops every 30 epochs on ImageNet and drops at 140, 200, 250 epochs on CIFAR-100. The optimizer is Stochastic Gradi- ent Descent (SGD) with momentum 0.9. By following the conventions in OKD, we use a fixed temperature T as 1.0 and loss weight x as 1.0 for all experiments. For fair com- parison, the hyper-parameters of different methods are fixed in all experiments. We set a= = 0.01, 3 = 0.01, 7 = 1.0 on CIFAR- dataset because of limited spatial information, and a= = 0.2, 3 = 0.6, 7 = 0.01 on ImageNet dataset for all OKD experiments. It's worth noting that ADM exhibits no sensitivity to hyperparameter settings shown in Table 10 and 12.
Results on ImageNet. Table 3 reports the performance of our approach on ImageNet with similar ResNets architec- tures. We all methods three times with different seeds and obtain the Top-1 average accuracy. As presented in the table, ADM outperforms other OKD methods in most cases. Equipped with ADM, the DML baseline of ResNet- 18 (11.7M parameters) gains 0.47% in Top-1 accuracy. Be- sides, we observe that larger teacher model ResNet-101 (44.5M parameters) fails to distill a better compact student in other OKD methods. But in the paradigm of ADM, due to the teacher's superior performance, more confidence can be set in the discrepancies regions between the teacher and stu- dent, which, in turn, students learn uncomplicated features rather than challenging ones. As a result, ADM can effec- tively bridge the gap between the large teacher and student. In addition, even for two identical models, ADM can still enhance both performances. As shown in Table 4, With pair ResNet18 18 and 0.5MobileNetV2 (1.97M parameters), ADM obtains 0.89% accuracy gains than DML., and ResNet50 (25.6N 6M parameters--MobilNeevt (4.23M parameters) pair 0.43% gains, verifying its effectiveness on heterogeneous architectures. It can be seen that the relative difference in parameter size is not the main reason why a larger model
5.2. Online KD Experiments on Image Classification
Results on CIFAR-100. Table I and Table 2 report the perimental results on 100 with similar architecture style and dissimilar architecture style teacher-student pairs, respectively. We run all methods five times with differ-
5

---

Asymmetric Decision-Making in Online Knowledge Distillation; Unifying Consensus and Divergence
with better accuracy cannot serve as a good leacher for smaller models. How to choose good teacher model for distillation remains open question worth discussing.
Methods
Teacher Student
DML KDCL SwitOKD DML+ADM
DecpLabV3 PSPNet
78.30
73.58
74.19
73.82
73.98
74.55
76.96
70.74
71.54
71.49
71.72
71.98
It's worth mentioning that all existing OKD methods en- countered an identical issue. The performance gain on the homogenous backbone pair is not obvious compared to the heterogenous pair. There are primarily two reasons for this observation. Firstly, heterogeneous pairs exhibit greater structural differences, enabling them acquire broader range of knowledge from one another. Secondly, and more significantly, the student model within heteroge- neous pairs is smaller. The performance disparity between the student and teacher models is more pronounced, result- ing larger potential for improvement. An additional ex- perimental observation is that the degradation of the teacher model's 's performance is common issue among all existing OKD methods. However, our method outperforms other baseline approaches in terms of the teacher's performance improvement.
Table 5: Results on Cityscapes val dataset. All models are pretrained on ImageNet.
5.4. Extension to Diffusion Distillation
Figure 4 illustrates the dynamic consensus region, which is imitially concentrated around the target object (t < 400), progressively diffusing multi-scale regions (400 S 600), and stabilizing globally (t 2 600). Simply, we em- ploy the consensus loss when < 400 during the training process, achieving promising results. We present the re- sults for Inception Score (IS) (Salimans et al., 2016) and Fréchet Inception Distance (FID) (Heusel et al., 2017) for each method. Table 6 presents the results of step distilla- tion experiments conducted on the CIFAR-10 dataset under the settings described in (Sun et al., 2023). When distilled to 1-step sampling, the FID score decreased by 2. Table 7 illustrates the model distillation experiments on the Ima- geNet dataset. We utilize the DiT-base(Peeble & Xie, 2023) model as the teacher to distill the sPPeellslseeeeles Xie, 2023) student model, We use MSE loss as our baseline. The FID score decreased by I and the IS increased by after training for 400k steps.
Backbone
Vanilla DML KDCL SwitOKD DML+ADM
ResNet-18
70.16
71.12 71.54
71.00
71,59
ResNet-34
73.93
73.78 74.36
73.15
73.93
ResNet-18
70.16
71.44
71.33
71.36
71.62
ResNet-101
76.62
76.80 78.22
76.28
77.81
ResNet-50
75.99
76.70 76.70
76.78
77.10
ResNet-101
77.85
78.17 78.36
78.00
78.31
ResNet-18
70.16
70.80
71.09
70.70
71.07
T-0
ResNet-18
70.16 70.80 71.08
70.63
70.94
Table 3: Top-1 Accuracy (%) comparison on ImageNet with same architecture style teacher-student pairs. The upper and lower models denote student and teacher, respec- tively.
T-200
T-400
Backbone
Vanilla DML. KDCL. SwitOKD DML+ADM
0.5MobileNetV2
62.49
64.21
64.32
63,94
65.10
ResNet-18
70.43
69.00
70.75
68.81
69.66
T-600
MobileNet-VI
68.81
71.13
70.76
71.07
71.69
ResNet-50
76.38
75.44
76.42
74.80
75.68
T-800
Table 4: Top-1 Accuracy (%) comparison on ImageNet with different architecture style teacher-student pairs. The upper and lower models denote student and teacher, respectively.
T=999
5.3. Transfer Learning to Semantic Segmentation
Figure 4: Visualization of the spatial-temporal evolution with high teacher-student similarity. Red indicates high similarity values. Best viewed in color.
Experiments are also conducted on semantic segmentation, which is recognized demanding dense prediction task. We train DeepLab (Chen et al., 2018) and PSPNet (Zhao et al., 2017) with ResNet-18 18 backbone on Cityscapes dataset. Different pre-trained models are considered using an online teacher with ResNet- 101 backbone. As the results summa- rized in Table 5, our ADM can significantly outperform existing OKD methods on semantic segmentation task.
5.5. Extension to Offline KD
ADM contains a reacher-discrepang feature enhancement module and a student-similar feature knowledge distillation module. Evidently, the latter part is suitable for application
6

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Sampling Steps
Method
FID (Q)
IS (t)
10 and 12 report the student and teacher accuracy with dif- ferent a, ,3 and different 7 in ImageNet. We use ResNet-34 and ResNet-18 as teacher and student models, respectively, Experiments demonstrate that the same hyperparameters can be employed for distillation across distinct model pairs, and the configuration of hyperparameters has little influence on the results. For 7, due to the magnitude of Lyeats is generally set lower than 0.1. For and 8. the results indi- cate that both parameters demonstrate stable increases when set around 0.5. It is worth noting that we observe higher accuracy when the weights of a and B are set close to the 74.57
PD
15.926
7.764
PD+ADM
13.939
8,037
2
PD
7.322
8.653
PD+ADM
6,830
8,760
4
PD
4.667
9.020
PD+ADM
4.555
9.042
Table 6: Results Diffusion Step Distilltion.
ModeVV
CRGm4.0
300K
2008
100K
Mathod
HID.SUK
15
FID-50K
18
FID--OK
1S
FILXSOK
18
B/2
11.82
219.97
12.05
188.92
13.67
143.42
22.64
Otacher vimillaa
42.78
33.50
47.50
20:43
55.29
24
72.21
17.70
mIoU of foreground objects, as obtained in Figure 2. This finding substantiates that ADM method is indeed learning
DIT-3/2
123.75
17.05
105
21.94
79.20
36.13
43.62
Istudent vamilla
67.18
20.30
71.42
18.91
78.58
16.64
94,00
13.16
in an anticipated manner.
DIE-S/2-SE
13.67
136.73
15.75
112.74
18.95
93.63
31.98
43.85
62:08
22.27
65.75
20.77
72.71
18.30
86.43
14.52
1NT-5/2-DMM
139.41
14.77
124.65
17.95
99.08
30.04
53.98
Effects of Feature Distillation Loss and ADM Loss. This paper proposes three types of losses: Ljeat, Lco and Las- To validate the effectiveness of each loss, we conduct exper- iments to train students with these losses separately. The results on Table 13 verify that both Lieat and Ladm can out- perform the DML, also, the performance could be further boosted by combining them together. It is noteworthy that Las loss effectively enhances the performance of the teacher model. We also investigate the individual effectiveness of ADM in Table 14.
64.03
21.35
70.60
1838
14.75
Table 7: Results on Diffusion Model Distilition.
in offline KD. To further validate the ADM, we also com- pare ADM with offline knowledge distillation approaches including KD and DKD that require a fixed and pre-trained teacher. Specifically, we set a, B, in (9) and Eq.(10) as 1.0//.0/0.0 and O. 1//00.000.0 for CIFAR- 100 and ImageNet Dataset. Table reveals that ADM achieves superior per- formance over offline fashions. For example, incorporating ADM based on KD resulted in a 1.33% improvement, and when applied to one of the state-of-the-ar DKD methods, there was still an approximate increase of 0.3% on CIFAR- 100 dataset. Table 9 shows that ADM increases KD accuracy by 0.22% and NORM 0. 1.12% on ImageNet dataset.
Lyoat
Lco
Los
stu_ace
lea-acc
71.12
73.78
71.43
73.91
/
71.35
73.78
71.46
74.07
71.50
73.91
71.59
73.93
Table 13: Effects of feature distillation loss and ADM loss.
5.6. Extension to Multiple Teachers and Students.
Backbone
Vanilla
DML
ADM
Similar to DML and SwitOKD, ADM can be extended for multiple networks easily. As shown in Table 11, We adopt the same three-network settings as SwitOKD. 1T2S means that one teacher model and two student models, and 2TIS means that two teacher models and one student model. We achieve a higher accuracy student across all methods in the 2T1S.
S:ResNet- 18
70.16
71.12
71.33
3:ResNet-34
73.93
73.78
73.97
S:0.5MobileNerV2
62.49
64.21
64.59
18ResNet-18
70.16
69.00
70.58
Table 14: The individual effectiveness of ADM. S: Student, T: Teacher.
Different Types of ADM loss. In this paper, we use cross- entropy(CE) to strengthen the ADM loss for brevity. Table 15 shows differentt forms of ADM loss. Adopting CE and KD loss helps us to achieve better performance.
SwitOKD SwitOKD DML+ADM DML+ADM
Backhone
Vanilla DMI.
(1T25)
(2T15)
(IT2S)
(2T1S)
Netile Ner-
66:01 72.28
72.20
71.97
71.69
72.51
WRN-16-2(S/D)
72.79
74.78
74.91
75.07
74.80
74.09
WRN-16-10CT)
79.30
80.55
80.40
80.72
Lco
Lai
tea_aee stu_ace
Table 11: Accuracy (%) comparison with 3 networks on CIFAR-106 WRN-16-2 serves as either teacher (T) or student (S) for DML, while is treated as S for 1T2S and T for 2TIS.
Lee
Lee
71.41
74.03
Lkd
Lee
71.59
73.93
Lka
Lud
71.45
73.98
Table 15: Different types of ADM loss.
5.7. Ablation Studies
Syper-Parameters Tuning, We explore the impact of three different hyperparameter values on the performance. Table
Combined with More Methods. ADM is a plug-and- play module that can effectively enhance the performance
7

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Same urchitecture style
Different architecture style
Method
ResNet-32x4
ResNet-56
WRN-16-2
ResNet-32x4
ResNet-50
ResNet-8x4
ResNet-20
0.5MobileNet.v2
ShuffleNet-V2
MobileNer- V2
Teacher
79.42
72.34
73.26
79.42
79.34
Student
72.50
69.06
64.60
71.82
64.60
KD
73.75
71.05
68.91
75.35
68.42
KD+ADM
74.52
71.46
70.24
75.73
69.04
DKD
76.24
71.52
69.03
77.07
70.35
DKD+ADM
76.66
71.58
69.68
77.03
70.64
Table 8: Evaluation results in offline KD CIFAR-100 dataset. The upper and lower models denote teacher and student, respectively.
Teacher Student
KD
KD+ADM DKD DKD+ADM DisWOT DisWOT+ADM NORM NORM+ADM
top-1
73.31
69.75
71.03
71.25
71.70
71.84
71.36
71.49
72.03
72.15
top-5
91.42
89.07
90.05
90.12
90.41
90.36
90.23
90.23
90.41
90.54
Table 9: Top-1 accuracy (%) in offline KD the ImageNet validation. We set ResNet-34 the teacher and ResNet-18 as the student. All results are the average over 3 trials.
0.0
0.1
0.1
0.2
0.2
0.2
0.3
0.3
0.3
0.5
0.6
0.7
1.0
0.7
0.5
0.6
0.8
0.7
0.5
0.2
0.3
1.0
stu acc
71.12
71.42
71.37
71.48
71.39
71.48
71.69
71.60
71.37
71.52
71.55
71.43
71.31
tea, acc
73.78
73.92
73.90
73.88
73.93
73.77
74.11
74.13
74.07
74.05
74.20
74.12
74.10
Table 10: Different a and B3.
0.01
Visualization. In addition, present visualizations from three perspectives in Figure 5 (with setting teacher as ResNet32 and student as ResNet8 x4 on CIFAR-100).
stu
71.59
71.48
Lea_acc
73.93
73.91
Table 12: Different 7,. stu_acc represents student's accu- racy and tea_aca represents teacher's accuracy.
6. Conclusion
of existing methods. In this paper, when combined with DML, ADM can basically achieve sota all settings, and when combined with stronger methods, better results can be achieved. As shown in Table 3, KDCL achieves the sota performance in some settings, and Table 16 shows several groups of comparative experiments combined with KDCL and SHAKE on the ImageNet dataset.
In this paper, we present Asymmetric Decision-Making (ADM), innovative online knowledge distillation method that unifies consensus and divergence feature learning and can seamlessly integrates with existing methods to leverage feature richness. Based on our insight into feature similarity, ADM employs a Consensus Learning on the student model to facilitate the learning of fundamental spatial knowledge, and a Divergence Learning on the teacher model to contin- ually enhance its capabilities and drive the student model to learn and catch up. This dynamic interaction between the teacher and student models epitomizes the principle that "iron sharpens iron," " thereby achieving superior perfor- mance online knowledge distillation. Extensive experi- ments show our superiority in various benchmark tasks. hope this paper will provide some insights for future OKD research.
Backbene
Vanilla
KDCL
XDCL+ADM
SHAKE
SHAKE+ADM
ScResNet-13 70.16
71.54
71.78 (40.24)
71.65
(40.00)
Net-5
73.93
74.36
34.34
73.45
73.50
Table 16: Combined with KDCL. S: Student, T: Teacher.
(a) L-SNE
(b) Correlation matrix
A Acknowledgments
Figure 5: Visualzaiton. Obviously, ADM (right) results in more discriminative features, reduced prediction discrep- ancics with teachers, and enhanced calibration than DML (left).
We gratefully acknowledge the enthusiastic members the Fingerprint Algorithm Group at JIIOV Technology for their
8

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
helpful discussions.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:/704.0486// 2017.
Impact Statement
This work advances the field of online knowledge distillation through unifying consensus and divergence feature learning. The technique may benefit the deployment of lightweight models on resource-constrained devices.
Huang, T., You, S., Wang, F., Qian, C., and Xu, C. Knowl- edge distillation from stronger teacher. arXiv preprint arXiv:2205. 10536, 2022.
Huang, T., Zhang, Y., Zheng, M., You, S., Wang, F., Qian, C., and Xu, C. Knowledge diffusion for distillation. NeurIPS, 36, 2024.
References
Chen, L.-C., Y., Papandreou. G., Schroff, F., and Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pp. 801-818, 2018.
Kim, B.-K., Song, H.-K., Castells, T., and Choi, S. Bk- sdm: A lightweight, fast, and cheap version of stable diffusion. In European Conference Computer Vision, pp. 381-3991 Springer, 2025.
Chen, P., Liu, S., Zhao, H., and Jia, J. Distilling knowledge via knowledge review. In CVPR, pp. 5008-5017, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene under- standing. In CVPR, pp.
Li, L. Self-regulated feature learning via teacher-free feature distillation. In ECCV, pp. 347-363. Springer. 2022.
2016.
Dai, X., Jiang, Z., Wu, Z., Bao, Y., Wang, Z., Liu, S., and Zhou, E. General instance distillation for object detection. In CVPR, pp. 7842-7851, 2021.
Li, L. and Jin, Z. Shadow knowledge distillation: Bridging offline and online knowledge transfer. NeurlPS, 35:635- 649, 2022.
Dong, P., Li, L., and Wei, Z. Diswot: Student architecture search for distillation without training. In CVPR, pp. 11898-11908, 2023.
Li, L., Dong, P., Wei, Z., and Yang, Y. Automated knowl- edge distillation via monte carlo tree search. In ICCV, pp. 17413-17424, 2023.
Guo, Q. Wang, X., Wu, Y., Yu, Z., Liang. D., Hu, X., and Luo, P. Online knowledge distillation via collaborative learning, In CVPR, pp. 11020-11029) 2020.
Li, L., Dong, P., Li, A., Wei, Z., and Yang, Y. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. NeurIPS, 36, 2024.
Hao, Z., Guo, J., Han, K., Tang, Y., Hu, H., Wang. Y., and Xu, C. One-for-all: Bridge the gap between heteroge- neous architectures in knowledge distillation. NeurIPS, 36, 2024.
Li, S., Lin, M., Wang, Y., Wu, Y., Tian, Y., Shao, L., and Ji. R. Distilling a powerful student model via online knowledge distillation. IEEE Transactions Neural Networks and Learning Systems, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Li, Z., Ye, J., Song, M., Huang, Y., and Pan, Z. Online knowledge distillation for efficient pose estimation. In ICCV, pp. 11740-1177 2021.
Heo. B., Kim, J.. Yun, S., Park, H.. Kwak, N., and Choi, Y. A comprehensive overhaul of feature distillation. In ICCV, pp. 1921-1930, 2019.
Lin, W., Li, Y., Ding, Y., and Zheng, H.-T. Tree-structured auxiliary online knowledge distillation. In 2022 Interna- tional Joint Conference on Neural Networks (IJNNN,, pp. 1-8. IEBE, 2022.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient architecture design. In ECCV, pp. 116-131, 2018.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv: 02531, 2015.
Meng, C., Gao, R., Kingma, D. P., Ermon, S., Ho, J., and Salimans, T. On distillation guided diffusion models. arXiv preprint arXiv:22 10.03 142, 2022.

---

Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence
Meng, C.,, Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffuuion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint
2014.
Song, K., Xie, J., Zhang. S., and Luo, Z. Multi-mode online knowledge distillation for self self-super visual representation learning. arXiv preprint arXiv:2304.06466 2023.
Nguyen, T. H. and Tran, A. Swiftbrush: One-step text-to- image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 7807-7816, 2024.
Sun, s., Ren, W. Li, J., Wang, R., and Cao, X. Logit standardization in knowledge distillation. In CVPR, pp. 15731-15740, 2024.
Niu, Y., Chen, L.. Zhou, C., and Zhang, H. Respecting transfer gap in knowledge distillation. arXiv preprint arXiv:2210. 12787, 2022.
Sun, W., Chen, D., Wang, C., Ye, D., Feng, Y., and Chen, C. Accelerating diffusion sampling with classifier-based fea- ture distillation. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pp. 810-815. IEEE, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Tian, Y., Krishnan, D., and Isola, P. Contrastive representa- tion distillation. arXiv preprint arXiv:1910. 10699, 2019.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Xiaolong, L., Lujun, L., Chao, L., and Yao, A. Norm: Knowledge distillation via n-to-one representation match- ing. ICLR, 2023.
Qian, B., Wang, Y., Yin, H., Hong, R., and Wang, M. Switch- able online knowledge distillation. In ECCV, pp. 449 449-466 466. Springer, 2022.
Yang, C.,, An, Z., Zhou, H., Zhuang, F., Xu, Y., and Zhang, Q. Online knowledge distillation via mutual contrastive learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio. Y. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550) 2014.
Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., and Yuan, C. Focal and global knowledge distillation for detectors. In CVPR, pp. 4643-4652, 2022.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. 115:211 1--22, 2015.
Yin, T. Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint 20Xi::2202.00512.0 2022.
Salimans, T., Goodfellow, I.. Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint 1rXiv:1605.07144 2016.
Zhang, T., Xue, M., Zhang, J., Zhang, H., Wang, Y., Cheng,
L., Song, J., and Song, M. Generalization matters: Loss minima flattening via parameter hybridization for efficient online knowledge distillation. arXiv preprint arXiv:2303. /4666, 2023.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018.
Sanh, V., Debut, I.., Chaumond, J., and Wolf, T. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint
Zhang, X., Zhou, X., Lin, M., and Sun, Shufflenet: An ex- tremely efficient convolutional neural network for mobile devices. In CVPR, pp. 6848 6848-6856, 2018a.
2019.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Ad- versarial diffusion distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025.
Zhang, Y., Xiang. T., Hospedales, T. M., and Lu, H. Deep mutual learning. In CVPR, pp. 3320-4322 2018b.
10

---

Asymmetric Decision-Making in Online Knowledge Distillation; Unifying Consensus and Divergence
Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, Decou- pled knowledge distillation. In CVPR, pp. 11953-11962, 2022.
Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network. In CVPR, pp. 2881-2891 2017.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Learning deep features for discriminative localization. In CVPR, pp. 2921-2929, 2016.


Reviews:
[{"review_id": "zyaKDFHhbL", "reviewer": "Reviewer_DjLx", "summary": "This paper proposes an online knowledge distillation method to build a student model by knowledge transfer from larger teachers which are trained together with the student. The key assumption is that the student and teachers are more likely to share similar features in foreground rather than background areas. With this assumption, the paper formulates two learning objectives, namely consensus learning for enforcing student’s attention on foreground objects when making predictions, and divergence learning for encouraging teachers to explore foreground patterns that are overlooked by the student. The proposed online knowledge transfer method has been evaluated extensively as a plug-and-play component to be incorporated with multiple baseline approaches on several tasks, including image classification, semantic segmentation, and image generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: the effects of consensus learning need to be further clarified**\n\n**A1:** Applying $L_{co}$ to the student model does not imply that the performance improvement of the student model is primarily due to $L_{co}$. As evidenced in Table 13, when $L_{co}$​ is used alone, the teacher model's performance remains unchanged at 73.78 (73.78 vs. 73.78), which inherently limits the potential improvement of the student model through distillation. Conversely, when only $L_{di}$ is applied, the teacher model's performance increases by 0.29 (73.78 vs. 74.07), thereby raising the upper bound of the student model's distillation performance. Hence, although $L_{di}$ is applied to the teacher model, it significantly enhances the student model's performance.\n\nWe contend that in the online distillation setting, both $L_{co}$ and $L_{di}$ need to work synergistically. If only $L_{co}$ is used, the teacher model cannot continuously explore more challenging foreground regions. Conversely, if only $L_{di}$ is used, the student model cannot effectively learn the foreground regions identified by the teacher model from easy to difficult, failing to achieve the intended mutual enhancement of both models' performance.\n\n\n\n**Q2: whether the gradient from Lco and Ldi propagates back to the cosine similarity matrix (S) via the attention scores A?**\n\n**A2:** No gradients are propagated in there. Please refer to the code in Appendix L629, where the detach operation is implemented.\n\n\n\n**Q3: Evaluation of the proposed method upon vision transformer-based (ViT) models is missing.**\n\n**A3:** We conducted experiments using DiT-based diffusion model distillation, which may indirectly support the effectiveness of our method in transformer-based distillation scenarios.\n\n\n\n\n\n**Q4: the 1T2S and 2T1S variants of DML are missing as important baselines.**\n\n**A4:** We adhered to the experimental settings outlined in the SwitOKD paper to demonstrate the extendibility of our approach for training multiple networks.\n\n\n\n\n**Q5: The experimental setup of the extension to diffusion distillation is unclear.**\n\n**A5:** As detailed in lines L292-L296 of the referenced paper, for CIFAR10, we followed the experimental settings from the RCFD paper. For DiT model distillation, we adhered to the experimental settings from the DiT paper. All parameters were reused from the original papers without any modifications.\n\n\n\n\n**Q6: The visualisation provided in Fig.4 and Fig.5 is hard to understand.**\n\n**A6:** There was a typo error. The third comparison is included in the appendix, while only two comparisons are presented here.", "claims_and_evidence": "Most of the claims are made with support and evidence, but the effects of consensus learning need to be further clarified. As stated at L50 (right) and L192 (left), the consensus loss ($L_{co}$) is applied to the student to amplify feature attribution in teacher-aligned foreground regions, whilst the divergence loss ($L_{di}$) is applied to the teacher for exploration of under-activated foreground patterns. However, according to Table 13, the benefit brought by $L_{di}$ to the student is more significant than that by $L_{co}$. These results are a bit inconsistent with the expected contribution of the two learning objectives.", "methods_and_evaluation_criteria": "The proposed ideas are quite intuitive, and most of the adopted components are well justified. The only unclear part to me is whether the gradient from $L_{co}$ and $L_{di}$ propagates back to the cosine similarity matrix ($S$) via the attention scores $A$? From my understanding, the cosine similarity matrix $S$ is used only to highlight features in different regions for making class predictions. In this case, I would assume gradients only pass to highlighted features from class prediction errors (i.e. cross-entropy losses) but not the student-teacher similarity matrix. However, the explicit gradient detach (GD) operation used in Fig.3 is not applied to the ADM module. It is ambiguous whether these two losses ($L_{co}$ and $L_{di}$) also take effect to optimise student-teacher alignment, similar to the role of $L_{feat}$ defined in Eq.(5).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The proposed method is evaluated comprehensively with just a few concerns:\n+ Evaluation of the proposed method upon vision transformer-based (ViT) models is missing. The paper conducts an extensive evaluation based on different combinations of teacher’s and student’s architectures. However, as a strong backbone architecture widely adopted in the CV community, ViT is missing as neither the student nor teachers.\n+ In the experiment on the extension to multiple teachers and/or students (Table 11), the proposed ADM works together with DML. However, the 1T2S and 2T1S variants of DML are missing as important baselines.", "relation_to_broader_scientific_literature": "The finding that similar features across different models are more likely to appear in the foreground regions is novel and inspiring.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\nThe paper is generally well-written and easy to follow. The findings of the aligned features between student and teachers are more likely to lie in foreground regions is interesting. The evaluation of the proposed method is comprehensive. \n\nWeaknesses:\n+ The experimental setup of the extension to diffusion distillation is unclear.\n+ The visualisation provided in Fig.4 and Fig.5 is hard to understand. For Fig.4, I failed to see how the concentrated regions evolve at different steps. For Fig.5, it is stated at L412 (right) that the visualisations are presented from three perspectives, but what perspectives they are is unclear.\n+ There are some confusing statements in the experiment analysis, and the disorder of the tables makes them hard to read. For example, Table.8 and Table.9 are followed by Table.12, all of which come after Table.15. Besides, I failed to find Table.10 in the manuscript. It is confusing to me how the statement \"Evidently, ...\" made at L329 (right) is evident without any references.", "other_comments_or_suggestions": "See comments above.", "questions_for_authors": "See comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b4XtskxPAa", "reviewer": "Reviewer_YcB8", "summary": "This paper addresses online knowledge distillation (OKD), a single-stage method where teacher and student models learn simultaneously. The authors focus on improving OKD performance by exploring intermediate feature alignment. Specifically, they propose an Asymmetric Decision-Making (ADM) strategy to unify consensus and divergence feature learning.\n\nThe authors perform an analysis using Class Activation Mapping (CAM) to understand how intermediate feature alignment impacts model behavior in OKD. They observe that:\n\n(i) The similar features between student and teacher models are predominantly concentrated on foreground objects, suggesting students prioritize learning these ”easy” patterns. \n(ii) Teacher models highlight foreground objects more than students, indicating an underutilized capacity for discriminative feature discovery. Divergent features, which may explain teacher superiority, are also concentrated in foreground regions.\n\nBased on these findings, they introduce ADM, a framework that uses spatially-aware feature modulation to manage consensus and divergence. ADM applies role-specific learning: \n— Students amplify feature attribution in teacher-aligned foreground regions.\n— Teachers intensify exploration of under-activated foreground patterns.\n\nThe paper claims the following contributions:\n\n(1) Identification of foreground-centric feature consensus and teacher-student attribution gaps through interpretability analysis. (This is challenging conventional assumptions about knowledge transfer in online settings.)\n(2) The ADM strategy, which uses spatial similarity to allocate attention in feature maps, boosting student attribution and reinforcing teacher strengths.\n(3) Experimental validation showing that ADM outperforms state-of-the-art OKD methods in various tasks, including online KD, offline KD, semantic segmentation, and diffusion distillation. It is shown that the proposed method provides SOTA results in several occasions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "They do.", "theoretical_claims": "The paper does not make any substantial theoretical claims.", "experimental_designs_or_analyses": "The experimental design and analysis of the paper is sound and valid.", "relation_to_broader_scientific_literature": "I feel the paper has done a good job positioning itself to broader literature in distillation and online distillation. It clearly identifies shortcomings of previous approaches and challenges conventional wisdom regarding how students learn in online distillation settings. It has also provided extensive experimental comparison in several datasets including ImageNet-1k.", "essential_references_not_discussed": "I feel the paper has done a fair job at discussing related work.\n\nThis is not essential but maybe the authors would like to discuss (or compare their work with, at least in the offline setting):\n\n— Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In\nProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages\n3967–3976, 2019.\n\nand related techniques a bit further.", "other_strengths_and_weaknesses": "I find this paper to be well-written and compelling, with a novel method and robust experimental results. I have no major concerns.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "I do not have any questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dBtv5Y1A1", "reviewer": "Reviewer_gakz", "summary": "This paper proposes an Asymmetric Decision-Making (ADM) approach for online knowledge distillation that adaptively fosters consensus learning for students while continuously encouraging teachers to explore harder features, thereby boosting performance on tasks like classification, semantic segmentation, and diffusion distillation by unifying feature-based and logit-based distillation techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: gap between the motivation of the proposed method and its findings**\n\n**A1:** The function of $L_{di}$ is to extract knowledge from unexplored foreground regions within the background areas, rather than merely utilizing background information. These regions present a challenge to the model, and without constraints, the model's limited learning capacity makes it difficult to autonomously explore them. As evidenced in Table 13, when $L_{di}$ is applied alone, the teacher model's performance improves by 0.29 (73.78 vs. 74.07), thereby increasing the upper bound of the student model's distillation performance. This demonstrates the necessity for the teacher model to continuously extract challenging foreground knowledge from the background regions.\n\nWe posit that in the online distillation setting, both $L_{co}$ and $L_{di}$ must work in synergy. If only $L_{co}$ is applied without $L_{di}$, the teacher model cannot persistently explore more difficult foreground regions. Conversely, if only $L_{di}$ is applied without $L_{co}$, the student model cannot effectively learn the foreground regions identified by the teacher model from easy to difficult, failing to achieve the intended mutual enhancement of both models' performance.\n\n\n\n**Q2: explain why they made such design choices instead of merely mentioning them in passing.**\n\n**A2:** In the introduction, we outlined the rationale behind ADM based on two key findings. The proposed method is straightforward yet effective.\n\n\n**Q3: Does the expression in lines 69-70 have any errors?.**\n\n**A3:** There is no error here. Figure 1 illustrates the changes in non-similar regions when comparing our method with vanilla training, which corresponds to Finding 2. It does not address the description of similar regions in Finding 1.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No, this paper does not involve theoretical claims.", "experimental_designs_or_analyses": "Yes. The authors compared their method with current state-of-the-art approaches to common visual distillation tasks. I find the experiments to be quite comprehensive overall. A potential shortcoming may be that they did not evaluate the method’s effectiveness in the increasingly popular large-model distillation settings.", "relation_to_broader_scientific_literature": "This study analyzes the common issues in previous online distillation work and accordingly proposes a novel method to separately handle the similar and different features between the teacher and student models.", "essential_references_not_discussed": "To the best of my knowledge, I think the authors have considered the relevant works related to their study.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The discovered issues/problems appear very interesting and may inspire further progress in the knowledge distillation community.\n2. The experiments are extensive. Additionally, more visual analyses are provided to explain the effectiveness of the proposed method.\n\nWeaknesses：\n1. There seems to be a gap between the motivation of the proposed method and its findings. Why is it necessary to force the teacher model to explore background regions? Intuitively, foreground regions seem more important. Additionally, from the final results, the performance improvement brought by the proposed method also appears to be marginal.\n2. When introducing the method, the authors should explain why they made such design choices instead of merely mentioning them in passing. \n3. With regard to the phenomenon observed in Figure 1, could the authors provide additional examples from various datasets to support their findings? Optionally, could they indicate what percentage of instances exhibit this phenomenon?", "other_comments_or_suggestions": "See above", "questions_for_authors": "Does the expression in lines 69-70 have any errors? \"The first row shows that discrepancy regions between teacher and student’s features are more concentrated on foreground object regions after Vanilla training.\" Should it be \"discrepancy\" or \"similarity\"? The current expression seems to contradict the first finding.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2iHcAvOJ37", "reviewer": "Reviewer_EJEu", "summary": "This article introduces Asymmetric Decision-Making (ADM), an online knowledge distillation (OKD) method that enhances both teacher and student models through consensus and divergence learning. Unlike other traditional KD methods this method names as ADM actively refines the teacher while improving the student, making it a mutual learning framework. It integrates seamlessly with existing KD techniques and achieves state-of-the-art results across CIFAR-100 and ImageNet.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: lack of rigorous theoretical justification**\n\n**A1:** Firstly, we assert that the proposed ADM method adheres to theoretical derivations. To substantiate this claim, we provide additional theoretical support:\n# Information-Theoretic Analysis of Asymmetric Decision-Making (ADM)\n\n## 1. Notation and Fundamental Assumptions\n\n### Symbolic Convention\n- **Input data**: \\( $X \\in \\mathcal{X}$ \\)\n- **Task labels**: \\( $Y \\in \\mathcal{Y}$ \\)\n- **Student features**: \\( $Z_s = f_s(X) \\in \\mathbb{R}^{H \\times W \\times C}$ \\)\n- **Teacher features**: \\( $Z_t = f_t(X) \\in \\mathbb{R}^{H \\times W \\times C}$ \\)\n- **Spatial indices**: \\( $(i,j) \\in \\{1,\\dots,H\\} \\times \\{1,\\dots,W\\}$ \\)\n- **consensus mask** in Eq6[L196-L200 in paper]: \\( $M_f = \\frac{1+\\mathcal{S}}{1+ \\bar {\\mathcal{S}}}$ \\)\n- **divergence mask** in Eq6[L183-L187 in paper]: \\( $M_d = \\frac{1-\\mathcal{S}}{1- \\bar {\\mathcal{S}}}$ \\)\n\n### Information-Theoretic Quantities\n- **Entropy**:\n  \\[\n  $H(Z) = -\\mathbb{E}_{Z \\sim p(Z)}[\\log p(Z)]$\n  \\]\n- **Mutual Information**:\n  \\[\n  $I(Z; Y) = H(Y) - H(Y|Z)$\n  \\]\n- **Conditional Mutual Information**:\n$I(Z_s; Z_t | X) $= ${E_X} \\left[ D_{\\text{KL}}\\left( p(Z_s,Z_t|X) \\,\\|\\, p(Z_s|X)p(Z_t|X) \\right) \\right]$\n\n---\n\n## 2. Consensus Learning: Feature Alignment via Mutual Information Maximization\n\n### Objective Function\nThe student model maximizes mutual information in foreground regions:\n\n\\[\n$\\mathcal{L}_{\\text{co}} = -I(Z_s^{M_f}; Z_t^{M_f}), \\quad \\text{where } Z^{M_f} = Z \\odot M_f$\n\\]\n\n### Variational Lower Bound\n**Theorem 1 (Barber-Agakov Variational Bound)**:\n\n$I(Z_s^{M_f}; Z_t^{M_f}) \\geq {E}_{Z_s^{M_f}, Z_t^{M_f}} [ \\log q(Z_t^{M_f} | Z_s^{M_f}) ] + H(Z_t^{M_f})$\n\n\nMinimizing the negative log-likelihood yields:\n\n\\[$\n{L}_{co} = E [ -\\log q(Z_t^{M_f} | Z_s^{M_f}) ]$\n\\]\n\n### Information Bottleneck Interpretation\nConstraining foreground alignment enforces:\n\n\\[$\n\\max \\, \\left[ I(Z_s; Y) - \\beta I(Z_s; X) \\right] \\quad \\text{s.t. } I(Z_s^{M_f}; Z_t^{M_f}) \\geq C$\n\\]\n\nResulting in the lower bound:\n\n\\[$\nI(Z_s; Y) \\geq I(Z_t^{M_f}; Y) - \\beta I(Z_s; X) + C$\n\\]\n\n---\n\n## 3. Divergence Learning: Entropy Maximization for Feature Space Expansion\n\n### Objective Formulation\nThe teacher model maximizes entropy in low-consensus regions:\n\n${L}_{di}$ = ${-H(Z_t^{M_d})}+ \\lambda {I(Z_t^{M_d}; Y)}$\n\n### Jaynes' Maximum Entropy Principle\n**Lemma 1 (Optimal Feature Distribution)**:\nUnder constraint \\( $I(Z_t^{M_d}; Y) \\geq \\epsilon $\\), the optimal distribution satisfies:\n\\[$\np^*(Z_t^{M_d}) \\propto \\exp\\left( \\lambda \\log p(Y | Z_t^{M_d}) \\right)$\n\\]\n\n### Coverage Capacity Analysis\n**Theorem 2 (Feature Space Coverage)**:\n\\[$\n\\mathcal{R}(Z_t) \\geq \\sqrt{2 H(Z_t^{M_d})}$\n\\]\nwhere \\( $\\mathcal{R}(Z_t)$ \\) denotes the radius of feature coverage.\n\n---\n\n## 4. Convergence Analysis via Dynamical Systems\n\n### Coupled Dynamics\nThe training process is modeled as:\n\n\\[$\n\\begin{cases}\n\\frac{d}{dt} I(Z_s; Y) = \\alpha \\left( I(Z_t; Y) - I(Z_s; Y) \\right) \\\\\n\\frac{d}{dt} H(Z_t) = \\beta \\left( H(Z_t) - H(Z_s) \\right)\n\\end{cases}, \\quad \\alpha, \\beta > 0$\n\\]\n\n### Lyapunov Stability Proof\n**Theorem 3 (Global Convergence)**:\n\nDefine the Lyapunov function:\n\n\\[$\nV(t) = \\left( I(Z_t; Y) - I(Z_s; Y) \\right)^2 + \\left( H(Z_t) - H(Z_s) \\right)^2$\n\\]\n\nThe time derivative satisfies:\n\n\\[$\n\\dot{V}(t) = -2\\alpha \\left( I(Z_t; Y) - I(Z_s; Y) \\right)^2 - 2\\beta \\left( H(Z_t) - H(Z_s) \\right)^2 \\leq 0$\n\\]\n\nGuaranteeing convergence to equilibrium \\( $I(Z_s; Y) = I(Z_t; Y)$ \\) and \\( $H(Z_t) = H(Z_s) $\\).\n\n---\n\n## 5. Empirical Validation and Error Bounds\n\n### Fano's Inequality Application\n**Theorem 4 (Classification Error Bound)**:\n\\[$\nP_e \\geq \\frac{H(Y|Z_s) - 1}{\\log |\\mathcal{Y}|} \\implies P_e^{\\text{ADM}} \\leq \\frac{H(Y) - I(Z_s; Y) - 1}{\\log |\\mathcal{Y}|}$\n\\]\n\n\n---\n\n## 6. Generalization to Structured Tasks\n\n### Semantic Segmentation\nPixel-wise regional mutual information:\n\\[$\nI_{\\text{region}}(Z_s, Z_t) = \\sum_{c=1}^C I(Z_s^c; Z_t^c)$\n\\]\n\n### Diffusion Model Distillation\nTemporal trajectory alignment:\n\\[$\nI_{\\text{traj}}(Z_s, Z_t) = \\sum_{t=1}^T \\gamma^{T-t} I(Z_s^t; Z_t^t), \\quad \\gamma \\in (0,1)$\n\\]\n\n---\n\n**Q2: statistical significance testing**\n\n**A2:** As detailed in lines 272-273 of the paper, the majority of experiments were conducted multiple times, and we report the mean results. This approach demonstrates the robustness and consistency of the performance improvements.\n\n---\n\n**Q3: analyzed the computational trade-offs**\n\n**A3:** Please refer to lines 241-247 of the paper. ADM introduces no additional trainable parameters, and the incremental computational cost is significantly minimal compared to the original setup.", "claims_and_evidence": "This article presents ADM as a superior online knowledge distillation method, but some of the claims lack sufficiently rigorous ablation studies and broader benchmarking. The improvement over stronger baselines like DKD and DisWOT is marginal, raising questions about statistical significance. Additionally, while ADM is tested across multiple settings, results on larger-scale datasets and diverse architectures remain limited, making its generalizability unclear. The theoretical justification for why ADM’s asymmetric learning outperforms standard KD is also underdeveloped and this will rely heavily on empirical results.", "methods_and_evaluation_criteria": "The article employs reasonable evaluation criteria, using FID, IS, and accuracy metrics across standard benchmarks like CIFAR-100 and ImageNet, which are widely accepted for knowledge distillation research. The inclusion of both offline and online KD comparisons strengthens the empirical evaluation. The choice of hyperparameters is well-documented, but my concern is with not performing further sensitivity analysis.", "theoretical_claims": "The claims made in the paper are generally supported by empirical evidence and shows ADM's improvements over baseline KD methods across various benchmarks. But, the lack of rigorous theoretical justification for why ADM consistently enhances both student and teacher models leaves some claims insufficient.", "experimental_designs_or_analyses": "The experimental design follows standard knowledge distillation benchmarks, using datasets like CIFAR-100 and ImageNet with multiple teacher-student configurations. The ablation study in Table 13 looks at incremental gains when component analysis is done. Further, statistical significance testing is absent and makes it unclear whether improvements are consistent across multiple runs.", "relation_to_broader_scientific_literature": "The article builds on prior work in knowledge distillation, particularly online and asymmetric distillation methods, by introducing ADM to enhance student-teacher interactions dynamically.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The article presents an interesting attempt at improving knowledge distillation efficiency, but it lacks originality as many of its ideas are incremental extensions of prior work. The writing is clear and easy to follow, the significance of the contributions is limited due to weak empirical validation and insufficient theoretical grounding. Additionally, some claims are overstated given the lack of rigorous ablation studies and comparative analysis against stronger baselines.", "other_comments_or_suggestions": "1. Your approach builds upon existing knowledge distillation frameworks but lacks a theoretical analysis. Can you formally derive the expected generalization benefits of your method compared to standard distillation techniques, possibly using PAC-Bayesian or information-theoretic bounds? Without such an analysis, how can one be certain that the improvements are not artifacts of dataset-specific heuristics?\n\n2. Your empirical evaluation primarily focuses on a limited set of model architectures. Have you tested your method on fundamentally different architectures (e.g., transformers, graph neural networks) beyond CNN-based models? If not, how can we ensure that the claimed improvements generalize beyond the specific experimental setup?\n\n3. Your method introduces multiple hyperparameters and auxiliary components. Have you conducted a comprehensive sensitivity analysis to determine the robustness of these choices? Specifically, is the method’s performance stable across different hyperparameter settings, or does it require extensive tuning?\n\n4. The empirical results compare against standard distillation techniques but do not include recent state-of-the-art methods such as contrastive-based distillation or self-supervised distillation approaches. How would your method perform against these stronger baselines, and would the relative improvements remain significant?\n\n5. Given the additional steps introduced by your method, have you analyzed the computational trade-offs? Specifically, how does the training time scale with increasing dataset sizes and model complexity, and is the overhead justified given the marginal performance improvements reported?", "questions_for_authors": "Please see other comments section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers were unanimously supportive of the paper, which presents a new algorithm for online knowledge distillation. The motivation is an observation that for image classification and segmentation problems, the student tends to prioritize modelling "easy" regions of the input (i.e., the foreground). To mitigate this, the key idea is to maintain two losses: one which encourages consensus amongst the student and teacher on dominant regions, another which encourages diversity amongst the teacher predictions. Results are presented on a range of classification and segmentation settings, and both online and offline KD.

The AC upholds the reviewer consensus, but our own reading of the paper is more qualified. We include a few comments that the authors are encouraged to consider for their final version of the paper:
- while Figure 2 is one of the primary motivations of the paper, there is limited discussion of what one is to conclude from it. Are we to interpret the ResNet-34 as a teacher, and ResNet-18 as a student? Is it meant to show that "teacher models highlight foreground objects more than students"?
- Equation 1, there should be a subscript $i$ for $\mathbf{z}^m$
- Equation 1 and elsewhere, use \mathrm{ce}
- Section 4.1, not clear what "the teacher’s object-centric features should be strengthened" means. What is an "object-centric feature"? A foreground feature?
- Section 4.1, "$L_{{\mathrm{co}}}$ encourages the student to focus on learning simple knowledge rather than concentrating on difficult regions" $\rightarrow$ why is this the case? Not clear why Equation 6 will focus on easy parts of the input.
- Equation 7, there is no explanation for the precise form of $\frac{1+ \mathcal{S}}{1+ \bar{\mathcal{S}}}$
- Equation 7, fairly non-standard that you still retain the true label here, but combine the teacher and student predictions in the predicted score
- Equation 7, how does the matrix multiplication of $F^s$ and $\frac{1+ \mathcal{S}}{1+ \bar{\mathcal{S}}}$ work? The dimensions don't seem to align
- in response to Reviewer EJEu, the authors included some theoretical claims. We unfortunately don't see any clear relevance to the actual algorithm discussed in the paper. e.g., Fano's inequality is a statement about the power of _any_ arbitrary classifier, and is generally not used to justify a specific algorithm. At least in its present form, we suggest this analysis is omitted from the final paper.
Decision: Accept (poster)