Single Teacher, Multiple Perspectives: Teacher Knowledge Augmentation for Enhanced Knowledge Distillation
Authors: Md Imtiaz Hossain
Sharmen Akhter
Choong Seon Hong
Eui-Nam Huh
Venue: ICLR 2025 Poster (2025)
Keywords: TeKAP
Teacher Knowledge Augmentation
Teacher Knowledge Perturbation
Single Teacher Multiple Perspectives
Synthetic Teacher
Knowledge Distillation
Ensemble Learning
Knowledge Transfer
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A Novel and First Technique to Augment Teacher Knowledge and Generate Multiple Synthetic Teacher Knoweldge

Abstract: Do diverse perspectives help students learn better? Multi-teacher knowledge distillation, which is a more effective technique than traditional single-teacher methods, supervises the student from different perspectives (i.e., teacher). While effective, multi-teacher, teacher ensemble, or teaching assistant-based approaches are computationally expensive and resource-intensive, as they require training multiple teacher networks. These concerns raise a question: can we supervise the student with diverse perspectives using only a single teacher? We, as the pioneer, demonstrate TeKAP, a novel teacher knowledge augmentation technique that generates multiple synthetic teacher knowledge by perturbing the knowledge of a single pretrained teacher i.e., Teacher Knowledge Augmentation via Perturbation, at both the feature and logit levels. These multiple augmented teachers simulate an ensemble of models together. The student model is trained on both the actual and augmented teacher knowledge, benefiting from the diversity of an ensemble without the need to train multiple teachers. TeKAP significantly reduces training time and computational resources, making it feasible for large-scale applications and easily manageable. Experimental results demonstrate that our proposed method helps existing state-of-the-art knowledge distillation techniques achieve better performance, highlighting its potential as a cost-effective alternative. The source code can be found in the supplementary.

Published as a conference paper ICLR 2025
SINGLE TEACHER, MULTIPLE PERSPECTIVES: TEACHER KNOWLEDGE AUGMENTATION FOR ENHANCED KNOWL- EDGE DISTILLATION
Md Imtiaz Hossain, Sharmen Akhter, Choong Scon Hong & Eui-Nam Huh Department of Computer Science & Engineering, Kyung Hee University, South Korea fhossain. imtiaz, sharmen, cshong, johnhuh) khuu ac. kr
ABSTRACT
Do diverse perspectives help students learn better? Multi-teacher KD, which is more effective technique than traditional single-teacher methods, supervises the student from different perspectives (i.e., teacher). While effective, multi-teacher, teacher ensem- ble, or teaching assistant-based approaches are computationally expensive and resource- intensive, as they require training multiple teacher networks. These concerns raise ques- tion: can we supervise the student with diverse perspectives using only single teacher? We, as the pioneer, demonstrate TeKAP, a novel teacher knowledge augmentation tech- nique that generates multiple synthetic teacher knowledge by perturbing the knowledge of single pretrained teacher i.e., Teacher Knowledge Augmentation via Perturbation, at both the feature and logit levels. These multiple augmented teachers simulate an en- semble models together. The student model is trained on both the actual and aug- mented teacher knowledge, benefiting from the diversity of an ensemble without the need to train multiple teachers. TeKAP significantly reduces training time and computational resources, making it feasible for large-scale applications and easily manageable. Exper- imental results demonstrate that our proposed method helps existing state-of-the-art KD techniques achieve better performance, highlighting its potential wwt.cccccmmmmoc.eec.cc mm/mmtmmmmtt.//mmmm//////m/mmt iazh aazh/TeKhh native. Source code can be found at: https:: ///: / thub. com/
cost-effective alter-
1 INTRODUCTION
One-hot encoded targets (f0, 1}) are very hard and rigid. Practically, achieving prediction probability similar one-hot encoding by softmax function is not possible, the classifier output for all the non-target classes can not be zero. Three of the major problems of a one-hot encoded system are: 1) very hard (i.e., {0, 1)) which causes overfitting, 2) technically is not possible to achieve, and 3) there are no inter-class relationships information available as the target probability for all the non-target classes is same (here, 0). Label smooth- ing (Müller et al. 2019) solves the first two problems where the target is changed to {(1 - -)/(C 1)) e),
which provides flexibility and reduces overfitting, where and C' are the softness factor (usually = 0.8), and the number of classes, respectively. But similar to one-hot coding ((0. 1)), label smoothing also does not provide inter-class relationship information during training (the target probability for all the non-target classes is uniform i.e., ((1-)/(CC- (I)). KD first proposed by (Hinton 2015) transfers the representational expertise of the large teacher(s) to the small student network and addresses all these three problems. The teacher logit consists of inter-class relationships (non- toooet class probabilities are not uniform), flexibility (target range (0. 1) i.e., not (0, 1)), and practically possible to mimic the softmax output of the teacher(s).
*Corresponding authors.

---

Published as a conference paper ICLR 2025
Rinden recer
Rindee preesee
Rinden
Rindee Beans
Staditiona Muli- Teacher AuailaatonAAprras
de Our Proponaal Logd T:EAP Appreads
Forture Appronds
S. Student Networs
T. Texdee Netvorkix. Japut
W. Lineur wegits
q Lineur wegits for teadles and randoa vecoi
Figure 1: Depiction of the (a) multi-teacher, (b) TeKAP for logit-, and (c) feature-level distillation. + indicates the total of the losses after distillation (see Eq. and 4b. W1, W2, and W3 are the hyperparameters.
However, in KD, there are three fundamental concerns: 1) as the information comes from the teacher, the teacher perfect for the student (Xu et al. 2020; Yang et al.] (2021) 2) is the distillation technique able extract teacher's intrinsic representations perfectly (Stanton et al. 2021 and 3) the perspective of a single teacher is not diverse (as not from multiple teachers) to make the student more generalized. The
first two problems are dealt with several teacher-in imppoviin (Xu et al. 2020 Yang et al. 2021), and student- friendly teacher training approaches (Park et al.) [021]] Rao et al. 2023) Numerous KD approaches address the second problem (Sun al. 2024). But, to the best of our knowledge, no prior works consider the third problem. In this paper, focus and investigate the third concern: the teacher's perspective(s). Consid- ering teacher perspectives (Wen et al.) [024]] Ma et al.) 2024). and (Angarano et al. 2024 Tian et al. 2019) discussed the idea of teacher assistant, and ensemble KD, respectively, where multiple teacher knowledge
transferred to the student that enhances the student performance (Tian et al. 2019) Song et al. (022). These
approaches demand training multiple teachers, which is highly computation computation-heavy and resource intensive.
The discussions mentioned above led us to rethink teacher perspectives: can we supervise the students with diverse perspectives using only single-teacher? (Tang et al.) (220)) shows that feature distortion is signif- icantly effective in improving generalization via offering randomness where noisy labels are significantly effective to improve generalization capability further as well (Song et al., 2022). However, these works do not augment teacher knowledge and also, they do not demonstrate the potential of augmented teacher knowledge distillation and transferability tasks. Inspired by the above discussion, we propose a novel teacher knowledge augmentation technique, TeKAP. TeKAP generates multiple synthetic knowledge from a pretrained teacher model through perturbation via injecting random noises at both feature and/or logit lev- els as shown in Fig. Let's consider fj, and 6r(x) are the feature map from 7t layer and teacher logit of the teacher network (T((), respectively, where x denotes an input. Firstly, TeKAP perturbs feature map fi with weighted random noises R2 to distort feature maps, where j E [0, 1,3, J 1] and J is the number augmented teachers' knowledge be produced. Every jth random noise translates the feature maps f perturbed feature map S3 of different perspective. We generate J number of synthetic feature maps and logits, where different jth noise offer diverse perspectives as discussed in section 3.1 Distorted teacher logit offers diverse inter-class relationships, flexibility, and randomness to the student. In terms of feature representations, the perturbed feature maps work as the regularization terms.
We conduct extensive experiments standard benchmark datasets, ImageNet, CIFARI00 TinylmageNet, and STL for model compression, transferability, adversarial robustness, few-shot learning, scalability, and effects occluded and noisy input tasks. The in-depth and detailed analysis and discussion demonstrate the significance of augmenting teacher knowledge. This work, TcKAP, shows a new way of knowledge representation and transfer. We argue that stochastic diverse perspectives of single piece of information help students improve generalization.
Why Augmented Teacher Works: (Allen-Zhu & Li 2020 highlights that individual networks, if initialized
2

---

Published as a conference paper ICLR 2025
randomly, explore distinct aspects of the data, leading to diverse feature representations that enhance gen- eralization. The dark knowledge or inter-class correlation in teacher knowledge reflects the essence of the discriminative features learned (which and what). For instance, as depicted in Fig. when a model predicts a dog image as cat or horse with a probability of P(cat|dog) that the model has identified feature representations that exhibit certain similarities or dissimilarities across classes from particular viewpoint. From different viewpoint, the network may focus on the different sets of features. The inter-class correlation may change, for instance, P (cat dog) = Pe or P Phhorse = pn- This variability in inter-class correlations across different viewpoints helps students learn better.
Pr. and P(horse dog) = Ph. it indicates
Computational Complexity: TeKAP significantly reduces the computational burden associated with multi-teacher train- ing, teaching assistant, and ensemble KD. While traditional multi-teacher-based methods require training multiple teach- ers, TeKAP generates multiple synthetic teacher perspectives (i.e., augmented teachers) from single model, which helps reduce the training time and memory usage by multiple times. Our core contributions are: augmenting teacher knowl- edge with random noise to enhance knowledge diversity for the student, as a pioneer, 2) introducing novel method, TeKAP, for achieving diversity from a single teacher network without training, 3) demonstrating the potential of feature dis- tortion and noisy labels in distillation and transfer, 4) present- ing a plug-ame aad---nddplyy approach that can further enhance student generalization, and 5) providing in-depth analysis and discus- sion on benchmark datasets for diverse vision tasks to evaluate the significance of augmented teachers through distortion.
Dog.
Cae - 0.03
View-1
Horse
Network Predictions
Dog
Ieput lingge (Dog)
E
Cet 0.06
View-2
Horso 0.03
Figure 2: Depiction of an example of shifted inter-class relationship.
2 LITERATURE REVIEW
2015 first introduced the technique KD. Later, many improved distillation techniques have been proposed. One very important concern in this field is how to make the knowledge easy to learn for the student. Several approaches have been proposed to address this problem by reducing the capacity gap between the teacher and student. (Mirzadeh al.) 2020) and (Son et al., 2021) the seminal works exploit teacher assistants (i.e., intermediate networks) to reduce the capacity gap between the teacher and student. (Zhang et al., 2022) trained multiple teachers for improved performance. The effects of ensemble KD are explored in CRD TTian et al., 2019) and (Allen Zhh & Li (020).. However, training multiple teacher networks teacher assistants is highly resource-intensive and challenging maintain. Another approach to achieving a better-generalizee student to improve teacher performance before distillation, such as (Xu cet al. 2020). (Yang et al., 2021), or student-friend]] teacher learning (Park al. [021] Rao et al. 2023). These approaches train auxiliary classifiers, which demand additional training of teachers, thereby increasing cost and complexity. Furthermore, these approaches generate knowledge using the same teacher output. Our work utilizes a single teacher but perturbs it using random noise, which requires no training, thereby augmenting the teacher network and achieving diversity. Another difference between existing works and TeKAP is that these approaches do not explore feature-level distortion in KD. We investigate both logits and feature-level distortion using random noise that is optimization-Fii optimized distortion enhances network generalization and acts a regularization term, and (Song al. 2022) demonstrates that noisy labels generalize better. However, no one has considered teacher knowledge augmentation via perturbation, particularly for KD. One more unanswered question is whether this distorted or noisy teacher knowledge is transferable. We focus on demonstrating the potential of teacher perturbation and show its significance in diverse scenarios. TeKAP can be integrated with any distillation technique.
(Hinton
(Tang et al. 2020) shows that
3

---

Published as a conference paper ICLR 2025
3 METHOD
Proposed TeKAP generates multiple augmented teachers by perturbing single pretrained teacher model at both 1) feature, 2) logit, and 3) both levels as depicted in Fig. [ The student is trained using the linear combination of the knowledge of both the original and augmented teachers.
3.1 SINGLE TEACHER, MULTIPLE PERSPECTIVES
Logit-level augmentation primarily diversifies the inter-class relationships, providing alternative supervisory signals that regularize the student network. Feature-level augmentation, the other hand, introduces diver- sity in intermediate feature representations, exposing the student to a broader spectrum of variations (like dropout data augmentation). Both augmentations target distinct aspects of teacher knowledge: logits fo- cus on prediction diversity, while features address internal representation diversity.
Feature-level Perturbation: Feature perturbation introduces diversity into the intermediate feature repre- sentations of the teacher network ΦT. allowing the student network ós to learn from diverse perspectives. Let, (x) E Rxxuxe be the feature map generated by the teacher for an input instance x, where h, w, and c represent the height, width, and number of channels, respectively. To augment diverse teacher perspectives, we perturb the teacher's feature map with Gaussian noise 7) ~ NO, o²) to generate synthetic feature maps:
f00 (x)= + a) x fr(l)
Here a 0.1 a scaling factor controlling the perturbation intensity. We have used zero mean and 1 std. to produce random noise every epoch and perform a weighted combination with the original teacher logits (noise weights with 0.1 and teacher weights with 0.9). This results in a set perturbed feature maps that provide alternate augmented teacher outputs. The student is trained using both the original and synthetic feature maps through a feature distillation loss, such as:
Ljeat AL(fs(z), fr()) (1 (1-AX Σ L(fs(z),
(2)
==1
Here fs(()) is the student feature map and À controls the balance between the original and synthetic knowl- edge transfer using distillation loss L(.,)).
Feature perturbation can be viewed as a form of regulariza- tion, similar to dropout, data augmentation, which helps models generalize better (Tang et al.) 22020) Perturbing the feature maps exposes the student to a range of variations around the original feature map. When model is exposed to multiple noisy versions of the same input, it is forced to learn more robust inductive bias i.e., mapping without being overconfident (Allen-Zhte & Li) [2020].
Justification for Feature-level Perturbation:
Logit-Level Perturbation: The idea is to diversify the teacher's 's output logits to provide multiple synthetic supervision signals to the student at the logit level along with feature-level perturbation. Let (T) represent the logits (post-sofumax activations) produced by the teacher network Φr for an input x, where (T e gC and C is the number of classes. We introduce perturbation ~ _N(0,03) sampled from a Gaussian distribution, and add the teacher's logits:
2 (x)==axx+(() a) x (T
Here, we use a 0.1. These perturbed logits (x) represent the augmented version of the teacher prediction. The student network ds is trained using both of the original, 2T(x), and synthetic logits (i) (1)).
Aperturb = (5xD(2s(1), + (1 -)) Σ (xn(2s(x), logits
(4)
4

---

Published as a conference paper ICLR 2025
where LKD the distillation loss, and À the balancing weights between the original and synthetic logits.
Justification for Logit-level Perturbation: These noisy logits act as different perspectives of the teacher's prediction. The inter-class relationships or dark knowledge transformed into different sets of combinations that help the student to be regularised better {Song et al.) (022). The student, instead of overfitting to a single set of logits, is now forced to generalise across multiple noisy versions. This corresponds to learning a broader range of decision boundaries, making the student network more robust.
Unified Framework for Knowledge Transfer: The student model ds trained using both the original teacher outputs (feature map: fr(z), logits: 2r(x)), and the synthetic perturbed outputs (feature maps:
o = (1 (r)) + ar/,, logits: 2 (x) = (1 - ())a((a)) + om))). TeKAP combines both logit and
feature-level perturbations during training student:
LTeKAP= 71 x feat + + x Liogit + 7s x Loet
(5)
Here, 7/1 72, and 73 are the balancing weights which can be adopted from the corresponding distillation technique. Lreat, Liogit. and Leet represent feature level, logits level and cross-entropy loss, respectively. As the representative approach, we adopt CRD (Tian et al. 2019) as the feature-level and KD (Hinton) 2015) as the logit-level distillation techniques. We use 71 0.8, 72- 0.2, and 733 by adopting the weights from CRD (Tian al., 2019) for fair comparison.
Dynamic Noise Perturbation Injected Gaussian noise into both logits and feature maps are refreshed at each batch. This continuous noise variation ensures extra diversity for the student. The resulting noise- driven diversity improves the student's 's generalization performance by simulating a dynamic ensemble of teacher perspectives throughout training.
3.2 THEORETICAL PROOF
In TeKAP, the gradient of the loss function with respect to the student parameters wy becomes:
V wg Cperturb
AV wy L(fs())
(1 A) Σ v ((()
(6)
where, V Vus. Lieat fs( fo ff). and À represent the gradient of the loss function for the student net- work's parameters ws, feature-level distillation loss function, feature map output of the student, feature map output of the teacher, perturbed version the teacher's 's feature map, and a weighting factor that balances the contributions of the original teacher knowledge and the perturbed knowledge the loss function, respec- tively. Since 5 (z) adds variation to the teacher's 's representation, this noise helps the student generalize better unseen data, it's exposed to wider set of teachers' perspectives. Adding noise smooths the loss surface, which helps students in optimization landscapes find better solutions with a lower generalization error. This analogous data augmentation or dropout methods. The expected loss in the original teacher:
E [Fead (ffs(())))
and in the perturbed case:
Bx-x
X[= +
A) Σ E ((s())
(8)
where N represents the number of augmented (perturbed) teachers. The diseussion above indicates that since the perturbed features f (x)) are different noisy versions of the same feature map, the second term will generally have higher variability, Let's Txx) denote the knowledge of the teacher model for an input x, and (x) (1 - (x) acg be the noisy prediction where cR A o?) represents Gaussian noise added
5

---

Published as conference paper ICLR 2025
the teacher's output. The diversity introduced by Gaussian noise can be analyzed using Rademacher complexity (discussed in (Hsu et al. 2021). and (Tang et al. 2020)), which measures the capacity of the hypothesis class to fit random noise. Let be the hypothesis class for the student model S. The Rademacher complexity (.) of H with respect to noisy predictions Tx (ar) given by:
R. Es
1
sup hET
where are Rademacher variables and sup stands for supremum. h and represent function from the hypothesis class H and the number of training samples, respectively. The addition noise eg increases the variability of predictions:
TTr(Tx(2)) = Var(T cx] Var[T(x)) o²
(10)
where x² indicates the variance of the Gaussian noise which determines the spread or intensity of the per- turbation. This increased variability enhances the Rademacher complexity:
1 - Eg
Rr (H)
sup BEW
(T + +c)
(11)
Thus, the student model trained with noisy predictions benefits from increased diversity, which can reduce generalization error by improving the fit of the training data. The empirical risk of the student model trained with noisy predictions is:
n
Cemp (S)
(Lpp(T((
(12)
where 7emp and KL()) indicate empirical loss and KL divergence, respectively, The Generalization Error (GE) bound is:
2R, ())
GE S (S) +
(13)
where d is a confidence parameter. The noise addition helps to achieve lower empirical risk and better generalization by increasing the complexity regularization terms with diverse perspectives. Extended theoretical analysis can be found in the supplementary.
To Similar Architecture
To Different Architecture
Teacher Student
resnet32x4
WRN,402
WRN_40_2
VGG13
resnet56
resnet32xd ShufileNetV2 ShuttleNrrV1
resnet32x4
WRN-40 40-2
resnet8x4
WRN_40_1
WRN_16_2
VGG8
resnet20
ShufleNerV
Teacher
79.42
75.61
75.61
74.64
72.34
79.42
74.64
75.61
Student
72.50
71.98
73.26
70.36
69.06
70.50
70.36
70,50
73.33
73.69
74.92
72.98
72.98
TeKAP (L)
74.79
73,80
75.21
74,00
71.32
74.92
75.43
76.75
CRD
75.51
74.14
75.48
73.94
71.16
75.11
75.65
76.05
TcKAP
75.65
74.21
75.83
74.10
71.71
75.55
76.23
76.60
TeKAP (F+L)
75.98
74.41
76.20
74.42
71.92
75.60
77.38
76.59
Table 1: The effects of TeKAP on the SOTA methods. Competing results and setups for KD and CRD are quoted from CRD (Taian et al. 2019)). F and L indicate feature and logit-level distortions, respectively.
6

---

Published as a conference paper ICLR 2025
4 EXPERIMENTS
We have discussed the detailed results and analysis in this section. The rest of the results on the complexity, class imbalance datasets, noise sensitivity, hyperparameters analysis, etc., are available in the supplementary.
4.1 EFFECT OF TEKAP ON RECENT SOTAS.
Table illustrates the performance impact of incorporat- ing TeKAP into several state-of-the-at (SOTA) KD meth-
Teacher Student
resnc132x4 resnet8x4
WRN_40.2
WRN_40.1
ods, including DKD (Zhao et al.] 20022). MLKD (Jin et al.,
Baselines
Teacher
2023). TAKD (Mirzadeh al. 2020 CA-MKD (Zhang et al. 2022) and DGKD (Son et al. 2021I. The exper- iments are conducted using two teacher-student model se- tups: (1) resnet32x4 as the teacher and resnet8x4 as the stu- dent, and (2) WRN .4..2 as the teacher and WRN 40.1 as the student on CIFA 100 dataset. For all methods, the inclu- sion of TeKAP leads to consistent performance improvements, as shown by the higher accuracy values for students trained with TeKAP-enhaneed methods. DKD shows an improvement from 76.32% to 76.59% for 3aane32x4-resnette
Student
72.50
71.98
DKD
76.32
74.81
TeKAP
76.59
75,33
Single Teacher
MLKD
77.08
75.35
TeKAP
77.36
75.67
IAKD
73.93
73,83
TeKAP
Multi- Teacher
CA-MKD
75.90
74.56
TeKAP
76.34
74.98
and from 74.81% to 75.33% for WRN 40 2-WRN 40 1. MLKD gains similarly, with improvements reaching 77.36% and 75.67% respectively. TeKAP significantly enhances TAKD, boosting it from 73.93% 74.8 81% % and from 73.83% to 74.37%, high- lighting its ability to bridge capacity gaps. For CA-MKD and DGKD, TeKAP integration results in moder- ate yet meaningful gains, affirming compatibility with advanced multi seati-ttccher and dynamic distillation strategies. This table underscores the versatility and effectiveness of TeKAP as plug-and and-play augmenta- tion technique that consistently elevates the generalization performance various distillation approaches.
DGKD
75.31
74.23
TeKAP
76.17
75.14
Table 2: Effects of TeKAP on the SOTAs.
4.2 SIGNIFICANCE REGARDING LOGIT AND FEATURE LEVEL METHODS
Performance on CIFAR100: Table shows the effect of TeKAP on SOTA logits-based (KD (Hinton 2015)), ), and feature-basee -ased (CRD (Tian et al., 2019)) knowledge distillation approaches for similar and dif- ferent sreacher-studen architecture setups the CIFAR 100 dataset. The table shows that TeKAP uplifts the performance of KD and CRD in all the scenarios. In some cases, the performance gains are significant, such In the case of resnet20, and WRN_40_1, the performance gain is nominal. This is because the student resnet20 and WRN_40_1 are tiny.
(WRN _4004022550 2-ShuffleNett11, )), and (resset22x-SSuuffeeNett
some cases, our approach TeKAP helps KD to beat even the teacher networks such as WRN-40-2- ShuffleNetV and resnet32x4-Shffleeeet2 setup, where TeKAP (L) helps KD achieve 1.14%, and 0.79% improved accuracy than original teacher. In the case of (WRN 40 2-ShuffleNetV1), TeKAP helps KD to achieve better performance than CRD. The performance gains by TcKAP(F+L) verify that both the distorted feature and logits are transferable to the student. TcKAP(T+L) indicates both feature and logit level distortions achieve significant performance improvements. The results suggest that if the student is capable and the distillation approach is effective, then the diversity of the teacher and shifting inter-class correlation help the student improve performance.
Set
Teacher
Student
KD
KD TeKAP (L)
Top-1 Top-5
26.69
30.25
29.59 10.30
29.33
8.58
10.93
10.08
Table 3: Scalability of TeKAP on ImageNet dataset.
7

---

Published as conference paper ICLR 2025
ImageNet-1K: Scalability on Large-Scale Dataset: We evaluate the scalability and show the performance
The teacher-student architecture setups and competing results are adopted from CRD (Tian al., 2019) where ResNet-34 and ResNet- 18 are considered
comparison on the large-scale ImageNet dataset in Table 3
as the teacher and student, respectively. From Table 3 we notice that TeKAP helps KD achieve improved accuracy for both top-1 and top-5 error rates. These performance improvements verify the scalability the proposed TeKAP in large-scale datasets. Shifting inter-class correlations and diversity in dark knowledge offers discriminative feature representations which makes the representation easy to learn for the student.
4.3
TEKAP vs MULTIPLE TEACHER ASSISTANT BASED KD APPROACH TAKD
Teacher
resnet32x4 resnetBu4
WRN 40
WRN 40 2
VGG13 VGG8
resnet56 resnet20
resnet32x4 ShuffleNetV2 ShullleNetV
respe132x4
WRN-10-2 ShumeNetVV
Student
WRN 40.1
WRN 16.2
TAKD
73.81
73.78
75.12
73.23
70.83
74.53
74.82
75.34
TeKAP (L)
74.79
73.80
75.21
74,00
71.32
74.92
75.43
76.75
TeKAP (F+L)
75.98
74.41
76.20
74.42
71.92
75.60
77.38
76.59
Table 4: Performance comparison between TeKAP and multi-teacher approach TAKD.
TeKAP using single pretrained teacher and without training any additional network except student beats the performance of SOTA teaching assistant-based approach TAKD (Mirzadeh et al. (020) in all the scenar- ios (Table 4) TAKD reduces teacher-student capacity gaps by training multiple teaching assistants, which is computationally expensive. On the other hand, using single teacher and ignorable computation (gen- erating noise and calculating loss) TeKAP beats TAKD. This improvement depicts that augmented diversity also reduces the reacher-studer capacity gap while providing diversity and reducing training complexity. This gain indicates that the augmented diversity reduces the teacher-studen capacity gap better than TAKD.
4.4 EFFECT OF TEKAP ON CLASS IMBALANCE DATASETS.
Table 5 shows the effectiveness of TeKAP(L) in addressing class im- balanceness KD tasks on CI- FARI00 dataset. TeKAP im- proves the performance of all three teacher-student pairs the base- line approaches. Detailed analysis can be found in the supplementary.
Methods
resmet32x4-resnett844
NRN.4402-WRN.16.2
VGG13-VGG8
Baseline (KD)
41.71
$2.08
47.52
TeKAP (Ours)
46.42
52.72
51.25
Table 5: TeKAP on class imbalance dataset (CIFAR100).
4.5 WITER-CLASS CORRELATIONS COMPARISON
Inter-class correlation is the structural knowledge of teacher consisting of which and what discrim- inative features have been learned by a network. Diversity in the predictions helps students gener- alize better. Fig. 3 shows the comparisons re- garding the inter-class correlation differences of the vanilla student, bascline KD, and TeKAP. We used resnet32x4 and resnet8x4 as the teacher-student se- tups and trained on the CIFAR100 dataset. The differences in inter-class correlation between the teacher and TeKAP students are lower compared baseline vanilla and KD students. Surprisingly, though
Figure 3: Comparison inter-class correlations.
8

---

Published as a conference paper ICLR 2025
the augmented teacher adds diversity, this diversity or distortion helps students achieve teacher inter-class correlation more perfectly than baseline KD which further justifies the claims in (Tang et al. 2020) and (Sun
et al. 2024). The diversity of the augmented teachers works as the regularization terms.
Set
Student
KD
KD + TeKAP (1.)
Teacher
Student
KD
KD + TeKAP (I.
SIPARI0-STTIL
70.33
71.01
72.94
lop-1 1 top-5
29.88
21.44
21.12
22.47
CIPAR100-1 100-TinlImageNe
34.82
35.53
35.81
51.43
44.47
44.04
45.72
Table 6: Transferability: The learned representation on the CIFAR 100 is transferred to STL-10 and T'nyemageNee.
Table 7: Adversarial robustness of our proposed TeKAP on CIFAR100.
4.6 TRANSFRRABILITT TO DIFFERENT DATASETS
The student trained by TeKAP achieves better transferability to different datasets (TinylmageNet and STL10) than the baseline (Table 6 The representation learned from the CIFAR100 is transferred STL10 and TinylmageNet datasets. The network is finetuned to STL.10 and TinylmageNet datasets where (resnet32x4 resnet8x4) is considered as the teacher-student setup. TeKAP-student achieves 1.93%, and 0.28% higher accuracy than KD on CIFAR 100-STL10, and CIFARION CARRROOT-iin TinylmageNet transfer setup, respectively. The diversity induced by the augmented teachers enhances student transferability to different datasets.
4.7 EFFECT ON FEW-SHOT LEARNING SCENARIOS: LEARNING FROM A SMALL DATASET
TeKAP transfers multiple augmented diversity the student using a single input. Using TeKAP, student can learn more variations from a smaller number of data compared to the base- lines. We have evaluated TeKAP on the few-shot learning task. The effect of the augmented teachers is evaluated on 25%. 50%, and 75% of the training set using resnet32x4-resneths teacher-studen setup. The students are trained using a portion of the dataset. TeKAP improves performance in every scenario as shown in Fig. These results demonstrate that TeKAP, as augmentation technique, effectively provides many varia- tions and helps to learn better from the small amount data.
Tey woweoccappiio
Figure Effects of TeKAP on the few shot training scenarios (25%, 50%, and 75% data of the CIFARI00).
4.8 ADVERSARIAL ROBUSTNESS
TeKAP offers diverse and random dark knowledge inter-class correlations. The networks learn variations that make a student robust against adversarial attacks compared to the baselines (Table TeKAP uplifts the performance of KD by an improved accuracy of 1.35% on the CIFAR 100 in resmet32x4-resnetae student setup. During training, both the teacher and student use clean train set, while during the evaluation of the student, we attack the test set using the FGSM adversarial attack method where e = 0.005 (Madry et al. 2017). As the network learns noisy and random variations it becomes robust adversarial attacks.
teacher-
4.9 COMPARATIVE COMPUTATIONAL COMPLEXITY
The training of teacher ResNet32x4 in CIFAR100 using KD takes approximately 16 seconds per epoch. For 240 epochs, the total time taken is 240 x 16 = 64 minutes using two 3080 NVIDIA GeForce GPUs. For multi-teacher ensemble learning, we need to train multiple teachers. Let's assume two teacher assistants of equal size, which takes 64 x 2 = 128 minutes (approx) for DGKD Son et al. (2021). In our approach, TeKAP takes 18 seconds per epoch, which is 72 minutes in total. TeKAP avoids training multiple teachers.
9

---

Published as a conference paper ICLR 2025
4.10
EFFECTS PATCH OCCLUSION L.E., NOISY DATA
TeKAP provides added variations i.e., diversity to the student during training that makes the student more ro- bust to unseen variations and noises. We evaluated the ro- bustness of the TeKAP against occlusion and noise on the teacher- student setup as shown in Table [8] Both the teacher and student are trained using clean dataset where the noise in input is added during evaluations of the student. We add random patch of (4 x 4) dimensions random positions of the input images for every batch. From Table giit is observed that the students trained with aug- mented teacher perform better in noisy data which further verifies the significance of augmented diversity.
Teacher
Student
KD
KD + TeKAP (L.
top-1
76.16
71.79
71.85
72.85
top-5
92.86
92.27
92.08
92.57
CIFARI00 dataset using resmet32x4-resnetee
Table 8:
Performance on noisy data. random noisy patch is mixed at random position.
4.11 EFFECTS OF TEKAP ON ENSEMBLE LEARNING
Fig. 5 shows the effect of the number of augmented teachers. We use resnet32x4-resnettkk as the ceacher-student setup on the CI- FARI00 dataset to examine the effect of the hyper-parameters. From Fig. S we see that TeKAP is robust to the number of augmented teachers. For every number of augmented teachers, TeKAP achieves better accuracy than the baseline. The accuracy improves as the number of augmented teachers increases. How- ever, this experiment is run with up to ten augmented teachers. Table 9 shows the effect ensemble learning. Three (3) augmented teachers are used for every origi- nal teacher. During feature and logit distortion, the weights for noise and teacher output are 0.1, and 0.9, respectively.
# Teachers
T=1
T=2
T=3
Accuracy
75.98
76.12
76.19
Table 9: TeKAP on ensemble learning.
5 CONCLUSION
This paper proposes novel and innovative teacher knowledge augmentation framework that offers multi-teacher diversity us- ing a single-teacher model by augmenting teacher prediction with random noise. As pioneers, we show that both feature and logit level distortion or noisy predictions transferable to the student. Our work, TeKAP, dynamically generates synthetic knowledge that helps students improve generalization. We demonstrate extensive evaluation with discussions on model compression, scalability, transferability, few-shot learning, ad- versarial robustness, and the effect of noisy data. The pro- posed TeKAP improves the performance of the existing KD approach on benchmark datasets. Our proposed TeKAP offers diverse knowledge using single teacher and avoids training multiple teachers. The augmented diversity also reduces the leacher-student capacity gap. We do not propose any new KD technique. TeKAP can be easily applicable any existing KD approach. This paper opens up a scarch direction for teacher knowledge augmentation and achieving diversity using single teacher. Limitations and Future work The proposed TeKAP does not optimize the noise and remains train-free. Our proposed TeKAP enjoys the benefits of randomness and diversity. For future work, we plan to explore optimization-bn based techniques for teacher knowledge distortion.
ToKAF Serciac
2535
25:0
06:5
X4.6
0435
73.0
2335
2100
2235
112
116
Figure 5: Effects of the number of aug- mented teachers on TeKAP on KD
10

---

Published as a conference paper ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by the Institute Information Communications Technology Planning & (II) [In Technology Research Center) grant funded by the Korea govern- ment(MSIT) (ITTP-2025-Q) 0---2225-S--222-002222222 2023-00258644,, 50%) and the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) (RS-2021- 11212068, Artificial Intelligence Innovation Hub, 50%).
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020.
Simone Angarano, Mauro Martini, Alessandro Navone, and Marcello Chiaberge. Domain generalization for crop segmentation with standardized ensemble knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5450-5459, 2024.
Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint
2015.
Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. arXiv preprint arXiv:2104.00 05641, 2021.
Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14276-24285. 2023.
Zhe Ma, Jianfeng Dong, Shouling Ji, Zhenguang Liu, Xuhong Zhang, Zonghui Wang, Sifeng He, Feng Qian, Xiaobo Zhang, and Lei Yang. Let all be whitened: Multi-teacher distillation for efficient visual retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4126-4135, 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. stat, 1050(9), 2017.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI confer- ence artificial intelligence, volume 34, pp. 5191-5198, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances neural information processing systems, 32, 2019.
Dae Young Park, Moon-Hyun Cha, Daesin Kim, Bohyung Han, et al. Learning sudent-ffiendlly teacher net- works for knowledge distillation. Advances in neural information processing systems, 34:13292-13303, 2021.
Jun Rao, Xv Meng, Liang Ding, Shuhan Qi, Xuebo Liu, Min Zhang, and Dacheng Tao. Parameter-efficicnt and student-friendly knowledge distillation. IEEE Transactions on Multimedia, 2023.
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9395-9404, 2021.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: survey. IEEE transactions on neural networks and learning systems, 34 (11) 1335-135-8111 8153, 2022.
11

---

Published as conference paper ICLR 2025
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906-6919, 2021.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15731-15740, 2024.
Yehui Tang, Yunhe Wang, Yixing Xu, Boxin Shi, Chao Xu, Chunjing Xu, and Chang Xu. Beyond dropout: Feature map distortion regularize deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5964-5971, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 1910. 10699, 2019.
Haitao Wen, Lili Pan, Yu Dai, Heqian Qiu, Lanxiao Wang, Qingbo Wu, and Hongliang Li. Class incremental learning with multi-teacher distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 28443-28452, 2024.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-superrision. In European conference on computer vision, pp. 588-604. Springer, 2020.
Chuanguang Yang. Zhulin An, Linhang Cai, and Yongjun Xu. Hierarchical self-supervised augmented knowledge distillation. arXiv preprint arXiv:2107.13715, 2021.
Hailin Zhang, Defang Chen, and Can Wang. Confidence-awarc multi-teacher knowledge distillation.
In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4498 4502. IEEE, 2022.
Borui Zhao, Quan Cui. Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953-11962, 2022.
12


Reviews:
[{"review_id": "8jxlaNHLil", "reviewer": "Reviewer_NkEk", "summary": "The paper proposes a novel knowledge distillation method called TeKAP (Teacher Knowledge Augmentation via Perturbation), which generates diverse perspectives from a single teacher model. Instead of relying on multiple teacher models for supervision, TeKAP introduces diversity by perturbing both feature maps and output logits of a pretrained teacher network. This approach aims to simulate the benefits of multi-teacher distillation without the associated computational cost.", "questions": "- If more distillation methods could be included, it would be more convincing.\n- I think the idea that different teacher models provide different perspectives is interesting. Would increasing the number of teacher models further improve performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper provides thorough theoretical proof and experimental validation.\n- The paper is well-structured and clear in its approach, with intriguing perspectives.\n- The method proposed in the paper has a wide range of application scenarios.", "weaknesses": "- There is a lack of comparison with recent multi-teacher distillation work.\n- The explanation of the difference in usage scenarios between feature-level and logit-level may be insufficient..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CwPHvGsuKb", "reviewer": "Reviewer_pUYy", "summary": "This manuscript introduces TeKAP, a novel teacher knowledge augmentation technique. It generates multiple synthetic teacher perspectives from a single pretrained teacher model by perturbing its knowledge with random noise. TeKAP operates at both the feature and logit levels, enhancing the student's generalization ability. By reducing the need for multiple teacher models, TeKAP decreases both training time and memory usage. Evaluations on standard benchmarks demonstrate TeKAP's effectiveness in improving the performance of existing knowledge distillation approaches", "questions": "On page 4, the paper mentions the use of Gaussian noise for teacher perturbation but does not detail the criteria for choosing the noise parameters. How are these parameters optimized, and what is their impact on the diversity and quality of the generated teacher perspectives?\n2.On page 5, the term ​ is introduced in the formula without a complete explanation or definition.\n3.Is there a risk of overfitting to the perturbed features, especially when the noise parameters are not dynamically adjusted? \n4.How does TeKAP handle scenarios where certain classes are imbalanced? Is there a mechanism within the framework that ensures the augmented teachers do not bias the student towards overrepresented classes?\n5.Could the following discussion be added to page 8? For instance:\n1)What do these differences in inter-class correlations imply for the student's learning process?\n2)How does the performance improvement of TeKAP in terms of inter-class correlation contribute to the overall effectiveness of the model?\n6.In Figure 6, it is noted that the performance is best when the number of augmented teachers is 3. Does this imply that three teachers will be used in future applications? Additionally, the performance with two teachers seems normal; is there an explanation for this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work uses a single pretrained teacher to simulate multiple teacher perspectives through perturbation, effectively circumventing the high computational costs of traditional multi-teacher setups.\n2. The proposed method is simple yet demonstrated encouraging results.\n3. The work includes a comprehensive evaluation of various aspects such as model compression, adversarial robustness, and transferability, which strengthens the credibility of the proposed method.\n4. The extensive experiments also demonstrate TeKAP’s effectiveness in few shot learning and noisy data settings, suggesting a promising direction for advancing knowledge distillation.", "weaknesses": "1. Despite TeKAP's impressive results, the theoretical analysis of the perturbation methods lacks depth. While Gaussian noise is introduced, there is limited discussion on the choice of perturbation parameters, such as the standard deviation, and how these settings impact the model’s performance. This omission could hinder reproducibility and generalizability of the approach. \n2. Additionally, while the experiments cover a range of baseline comparisons, the paper lacks a comprehensive evaluation against existing multi-teacher distillation methods and other state-of-the-art single-teacher methods, which would better highlight TeKAP’s relative strengths. \n3. Moreover, there is little discussion on the computational efficiency and scalability of TeKAP in practical applications, potentially raising concerns among readers regarding its feasibility in real-world scenarios.\n4. Some statements are overclaimed in this manuscript. The authors should comprehensively review related works and give proper discriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V8B1tmuQoe", "reviewer": "Reviewer_zdP5", "summary": "The paper proposes a new augmentation method to replace the ensemble approach for KD by adding noise to the features or logits of the teacher model. This increases the variability of predictions and reduces the generalization error.", "questions": "1) What is \\mathcal{L}_{cel}​ in Equation 5?\n\n2) In Equations 2 and 4, calculate the summation of the perturbation loss. Does \\lambda need to be adjusted according to the number of perturbations?\n\n3) What is the difference between the CAMs of TeKAP and the teacher in Figure 5? They look the same.\n\n4) There is a lack of experimental details; even the learning rate and the number of training epochs are not mentioned in the paper.\n\n5) For feature-level perturbation, which features are selected to add noise?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is more efficient compared to other ensemble methods, and increasing the variability of the teacher's predictions is meaningful for knowledge distillation.\n\nThe paper is well-written and easy to follow.", "weaknesses": "1) The paper proposes an effective method to replace ensemble approaches; however, there is a lack of comparison to other ensemble methods (such as multi-augmentations) to demonstrate its effectiveness. Additionally, TAKD is not the SOTA method (for example, DGKD [1]) and there is a lack of experimental details for TAKD. It is not clear what teacher models are used for TAKD.\n\n2) The experiments in this paper are not sufficient, and the baselines are outdated. The proposed method only compares with vanilla KD (2015), TAKD (2020), and CRD (2019), and lacks comparisons with other new methods like DKD [2] and MLKD [3].\n\n\n[1] Son, W.; Na, J.; Choi, J.; and Hwang, W. 2021. Densely guided knowledge distillation using multiple teacher assistants. In Proc. Int. Conf. on Computer Vision (ICCV)\n\n[2] Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022. Decoupled Knowledge Distillation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)\n\n[3] Jin, Y.; Wang, J.; and Lin, D. 2023. Multi-Level Logit Distillation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMIAIAMdvi", "reviewer": "Reviewer_FGoU", "summary": "The authors propose TeKAP, a novel teacher knowledge augmentation technique that generates diverse synthetic teacher knowledge by perturbing a single pretrained teacher. This plug-and-play module leverages simple perturbations to capture ensemble benefits without training multiple teachers. Experimental results demonstrate TeKAP's effectiveness in enhancing both logit and feature-based knowledge distillation methods.", "questions": "-How can randomly distorted teacher logits provide diverse inter-class relationships if the distortion is truly random?\n-What does h represent in Eq. 9?\n-What is the scale of the random noise, and how should it be set? Detailed guidelines for noise settings are needed.\n-There appears to be no discernible difference between Fig. 3(b) and Fig. 3(c).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed plug-and-play module integrates seamlessly with existing KD methods, adding minimal computational burden.\n- By augmenting knowledge from a single pretrained teacher network, the authors significantly reduce training time and resource demands while achieving ensemble-like effects.\n- The approach is simple yet highly effective.", "weaknesses": "- The proposed plug-and-play module was not well validated. Specifically, it was only applied to vanilla KD and CRD, even though there have been many advanced KD methods that can serve as baselines.\n- The experiments omit numerous state-of-the-art single-teacher and multi-teacher KD methods; additional benchmark comparisons would - strengthen the evaluation.\n- Details on dynamic noise perturbation are insufficient, with critical implementation information missing for reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)