BIRD: Behavior Induction via Representation-structure Distillation
Authors: Galen Pogoncheff
Michael Beyeler
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Distillation
AI Alignment
Weak-to-strong generalization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Human-aligned deep learning models exhibit behaviors consistent with human values, such as robustness, safety, and fairness.  Transferring these behavioral properties to models trained on different tasks or data distributions remains challenging: aligned behavior is easily forgotten during fine-tuning, and collecting task-specific data that preserves this behavior can be prohibitively costly.  We introduce BIRD, a flexible framework for transferring aligned behavior by matching the internal representation structure of a student model to that of a teacher.  Applied to out-of-distribution robustness in image classification, BIRD outperforms fine-tuning, transfer learning, and continual learning methods, improving robust accuracy by up to 18\% over the next strongest baseline. It remains effective even when the teacher is trained on a much simpler dataset and is $25\times$ smaller in parameter count than the student.  In a large-scale study of over 400 teacher-student pairs, we show that three interpretable and computable properties of the teacher's representations explain up to 85\% of the variance in transfer success, offering practical guidance for teacher selection and design.  We further show that BIRD generalizes beyond applications in vision by enhancing safety alignment in language models when paired with Direct Preference Optimization and improving weak-to-strong generalization when combined with soft-label distillation.  BIRD turns small, well-aligned models into scalable alignment seeds, mitigating challenges from key bottlenecks in deploying safe AI systems.

Published as a conference paper at ICLR 2026
BIRD: BEHAVIOR INDUCTION VIA REPRESENTAAIIO-STTUCCURR DISTILLATION
Galen Pogoncheff
Department of Computer Science
University of Californi, Santa Barbara
ga gelenpogonchef www.fb.fuccbee....
Michael Beyeler
Department of Computer Science, Department of Psychological and Brain Sciences University of California, Santa Barbara mbeyele www.eerrrrubbb..... edu
ABSTRACT
Human-aligned deep learning models exhibit behaviors consistent with human values, such as robustness, safety, and fairness. Transferring these behavioral properties to models trained different tasks or data distributions remains chal- lenging: aligned behavior easily forgotten during fine-tuning, and collecting task-specific data that preserves this behavior can be prohibitively costly. We in- troduce BIRD, a flexible framework for transferring aligned behavior by matching the internal representation structure of student model to that of teacher. Ap- plied to out-of-distribution robustness in image classification, BIRD outperforms fine-tuning, transfer learning, and continual learning methods, improving robust accuracy by up to 18% over the next strongest baseline. It remains effective even when the teacher is trained on a much simpler dataset and is 25x smaller in pa- rameter count than the student. In large-scale study of over 400 teacher-student pairs, we show that three interpretable and computable properties the teacher's s representations explain up to 85% of the variance in transfer success, offering practical guidance for teacher selection and design. We further show that BIRD generalizes beyond applications in vision by enhancing safety alignment in lan- guage models when paired with Direct Preference Optimization and improving weak-to-strong generalization when combined with soft-label distillation. BIRD turns small, well-aligned models into scalable alignment seeds, mitigating chal- lenges from key bottlenecks in deploying safe Al systems.
INTRODUCTION
As AI systems become increasingly capable, their alignment with human values (expressed through traits like robustness, safety, and fairness) has become central challenge (Ortega et al., 2018; Ji al., 2023). Behavioral alignment typically requires costly supervision: adversarial training, human feedback, or special-purpose purpose datasets (Madry et al., 2017: Ouyang et al., 2022; Rafailov et al., 2023). These techniques do not easily scale to new tasks or domains.
A natural goal is to transfer aligned behavior from one model to another. Yet behavior often degrades during fine-tuning (Shafahi et al., 2019; Qi al., 2023; Peng al., 2024), and most transfer methods assume the teacher and student share training data Zhou et al., 2025). Worse, datasets used to train aligned models are often private.
common output space (Burns et al., 2023;
Recent work in weak-to-strong generalization offers promising alternative: train a small, well- aligned "weak" model and use it to supervise larger, more general "strong" model (Burns et al., 2023). However, these approaches still assume the teacher and student share task, training dataset, or output domain. In this work, we ask: Can aligned behavior transferred even when the teacher and student differ in architecture, task, and training data?

---

Published as a conference paper at ICLR 2026
Here, introduce Behavior Induction via Representatiion-streeue Distillation (BIRD), simple, drop-in framework for transferring aligned behavior between heterogeneous models by distilling task- and behaviorally-rellvan structure from the teacher's representation space into that of the student. BIRD requires no access to teacher training data and succeeds even when the teacher is trained on simpler tasks domains, enabling scalable reuse of aligned models.
BIRD is inspired by recent work in NeuroAI, where researchers hypothesize that desirable behav- ioral properties (e.g., robustness to noise, transformation invariances) may be encoded in the ge- ometry of brain representations (Chung & Abbott, 2021; Zador et al., 2023; Mineault et al., 2024). Several studies have operationalized this idea by explicitly aligning deep network representations with neural recordings, showing that representation-lewe generalization on image classification benchmarks (Dapello et al., 2023; Safarani et al., 2021; Li et al., 2019). Related work has also examined how representational similarity between modern vision models and the biological visual system can be quantified across differing objectives and ar- chitectures, providing tools for comparing internal geometries even when tasks diverge (Pogonchefff et al., 2024). However, these approaches typically require access to neural recordings and often rely on shared stimulus domains. Moreover, gains achieved through neural alignment degrade when transferring to categories or domains outside the alignment dataset (Dapello et al., 2023).
alignment can improve robustness and
Our work builds on the core insight that behavioral properties encoded in the structure of model's latent representations (Zou et al., 2023; Schneider et al., 2025), but generalizes it in two key ways. First, BIRD does not rely on biological recordings shared datasets; the teacher and student may differ in size, architecture, domain, and output space. Second, we empirically identify three computable properties of a teacher's representation space that reliably predict transfer success. This enables principled selection of both teacher models and representation layers.
We first evaluate BIRD in the context of -uito--distribution robustness transfer in image models. Studying over 400 ceacher-student pairs varying in architecture, capacity, and training data, we find:
BIRD outperforms transfer methods including fine-tuning, continual learning, and activation- based distillation, improving robust accuracy by up to 18% over the strongest baseline.
BIRD enables weak-to-strong transfer from small, simple teachers (e.g., CIFAR-IO-trainee Mo- bileNetV2) to students up to 25x larger, trained on more complex datasets (e.g., TinylmageNet).
Transfer success is predictable and actionable: three interpretable properties of the teacher's 's rep- resentation space that quantify task and behavioral relevance explain up to 85% of the variance in transfer outcomes, offering practical guidance for alignment layer and teacher selection.
We further show that BIRD extends beyond vision. In language models, supplementing Direct Preference Optimization (DPO) with BIRD improves the efficacy of safety alignment on the PKU- SafeRLHF dataset, and enhances weak-to-strong generalization when combined with soft-label dis- tillation.
By aligning representation structure instead of activation values or model outputs, BIRD provides flexible mechanism for transferring aligned behavior across models and domains. This design makes BIRD more general than existing approaches in knowledge distillation (Hinton et al., 2015; Qiu et al., 2022; Jin et al., 2024) and continual learning (Li & Hoiem, 2017: Shafahi et al., 2019) that assume access to teacher data or rely on shared domains, output spaces, or tasks between teacher and student. It advances the promise of weak-to-strong generalization and sets the stage for scalable alignment across tasks.
2 RELATED WORK
2.1 SCALABLE ALIGNMENT VIA WEAK-TO-STRONG SUPERVISION
Behavioral misalignment arises when models optimize their training objective while diverging from human intent (Amodei et al., 2016; Di Langosco et al., 2022; Razin et al., 2024). Current approaches to mitigate this, such as preference tuning for language models (Ouyang et al., 2022; Rafailov al., 2023) or adversarial training for vision models (Madry et al., 2017: Hendrycks et al., 2021), are effective work but require costly, task-specific datasets and supervision, limiting scalability.
2

---

Published as a conference paper at ICLR 2026
Scalable oversight aims to reduce this dependence by developing methods for supervising advanced models efficiently, even when human feedback is limited or unavailable (see et al., 2023). A promising direction is weak-to-strong generalization, in which small, well-aligned model provides soft-label supervision to guide the training of a larger, more capable model (Burns et al., 2023; Zhu al., 2024; Zhou et al., 2025). Our work generalizes this paradigm. Instead of requiring shared out- put spaces and access to teacher's ' training data (even if unlabeled), we transfer behavior between heterogeneous models by leveraging the representation structure of teacher model as a supervisory signal. This enables weak-to-stie transfer across differing tasks, datasets, and architectures.
2.2 LIMITS OF ROBUSTNESS TRANSFER IN IMAGE MODELS
Adversarial robustness is a primary benchmark for studying behavioral transfer (Shafahi al., 2019; Nem et al., 2023; Liu et al., 2023; Xu et al., 2023). While classical approaches require expensive, online generation of adversarial examples during training (Madry et al., 2017; Schmidt al., 2018), recent work has explored transferring robustness to new tasks without retraining from scratch.
A key challenge catastrophic forgetting (Kirkpatrick al., 2017). When robust models are fine- tuned on new data, robustness is often lost (Shafahi et al., 2019; Nern et al., 2023). To address this, methods have incorporated adversarial examples during transfer or added constraints to limit feature drift (Chen et al., 2021; Fan et al., 2021; Liu et al., 2023; Xu et al., 2023). Shafahi et al. (Shafahi et al., 2019) proposed adversarially robust transfer learning using Learning without Forgetting (LwF) (Li & Hoiem, 2017), which preserves robustness by constraining changes to the final-layer features. However, these techniques assume that robust features generalize across domains, which holds only with large and diverse pretraining datasets. In contrast, our work enables behavior transfer from smaller, simpler models trained on low-resource domains, without requiring shared inputs or labels.
2.3 DISTILLING REPRESENTATION GEOMETRY
Knowledge distillation (KD) enables student model to learn from teacher by mimicking outputs or hidden activations. Originally developed for model compression (Bucilua et al., 2006; Hinton et al., 2015), available KD methods have since expanded include intermediate-layer supervision (Romero et al., 2014; Zagoruyko& & Komodakis, 2016) and cross-modal transfer (Gupta et al., 2016). These methods assume shared tasks and output spaces and often transfer sample-level information.
BIRD takes a different approach. Rather than matching outputs or activations, we transfer the pair- wise structure of the teacher's representation space, captured via Gram matrices over input batches. This quantifies the geometry of internal representations, rather than specific activation values. This builds on the idea that model's knowledge lies not only in its outputs, but the organization of its representation space (Hjelm et al., 2018; Tian et al., 2019; Muttenthaler et al., 2024). While prior work uses this idea for unsupervised learning or intra-task alignment (Qiu et al., 2022), BIRD applies it to cross-task, cross-domain behavior transfer, generalizing KD into a scalable mechanism for aligned supervision.
2.4 REPRESENTATION ALIGNMENT IN NEUROAI
Work in NeuroAI suggests that robust and general behavior in biological systems may arise from the structure of neural representations (Chung & Abbott, 2021; Zador et al., 2023). Several studies have attempted to bias Al models toward brain-like representations by minimizing dissimilarity to neural recordings from visual cortex, often using objectives based on Centered Kernel Alignment (CKA) or Representational Similarity Analysis (RSA) (Dapello et al., 2023; Safarani et al., 2021; Muttenthaler et al., 2024). These approaches have shown moderate improvements in robustness but typically require neural data, assume shared input domains, and show limited generalization.
Our work draws inspiration from these ideas but removes the need for brain recordings or stimulus overlap. BIRD operationalizes the hypothesis that structured representations support general behav- ior, using any aligned model as a teacher. By enabling behavior transfer across architectures and domains, BIRD extends Ae insights into perrroe- framework for scalable alignment.
3

---

Published as a conference paper at ICLR 2026
Dyssttve
Ligiin
Leep
Teacher
Detudter
Dinstor
Lnge
Teas
Shakan
(a) Independent model pre-training
(b) Layer selection represeetatioon distillation
Figure I: Overview of BIRD: (a) First, a student model is pre-trained on its target training set, Dstudent- A teacher model is independently trained develop aligned behavior on Dreacher (opti- mizing a. (b) Next, the student model fine-tuned on its original dataset to maintain task per- formance while learning latent representations with similar structure that of the (frozen) teacher.
3 BIRD: BEHAVIOR INDUCTION VIA DEPRESENTATIIONSSRRUUEE DISTILLATION
We introduce BIRD, flexible framework for transferring aligned behavior from teacher model to student model. Our approach is grounded in the hypothesis that task-general behavioral properties (i.e., robustness, safety, and invariance) are encoded the structure tion space, suggested in Zou al. (2023). We posit that guiding student model adopt similar representational structure to its teacher biases it toward learning the same aligned behaviors.
model's '' internal representa-
BIRD proceeds in three steps (Figure 1):
We assume access to trained teacher model Jo Dreacher + Vreacher that exhibits desirable behavioral properties, and a pretrained student model Dstuudutt Vstuudet for which wish to induce those properties (Figure la).
2. A guiding layer in the teacher and a guided layer in the student are selected for distillation based on their relevance task and behavioral alignment (Section 6).
The student fine-tuned to preserve performance on original task while learning a represen- tation space whose structure mimics that of the teacher (Figure 1b).
To implement this, we define a loss that combines task performance and representational alignment:
E B--Datudees Cissk (B) 3Crep ( (u(B),
Here, B is a batch of inputs from the student's 's training distribution while a and 33 are hyperparame- ters that weight the relative contributions of task and representation-struuuui loss. The functions and u map those inputs to intermediate layer representations in the teacher and student, respectively. The first term, Ltask, is the task-specific loss that the student was originally trained to minimize (e.g., cross-entropy). The second term, Lyep, penalizes dissimilarity in representation structure:
Lrep ( (), v(B)
CKA (u( (B))),(B)))
We use CKA (Kornblith et al., 2019) to quantify the alignment of pairwise similarity structures within batch of inputs:
(o(B)"u(BBI (BBI CK CKAeea (w (B)) [u( u (B) . [o(B)T
We select CKA as a measure of representational similarity given its (i) proven effectiveness com- paring deep network representations, (ii) reliability when comparing high-dimensional representa- tion spaces, and (iii) ease of interpretation. Compared to common losses used in KD, such as Lz or KL-divergence, this CKA-based objective aligns the geometry of teacher and student representation manifolds rather than enforcing instance-level similarities. Because CKA evaluates pairwise simi- larities across batch (as opposed to per-example similarities, which are closely tied to the teacher's

---

Published as a conference paper at ICLR 2026
Table I: Accuracy (%) of MobileNetV2 (MN2), ResNet18 (RN18), DenseNet169 (DN169), and Vision Transformer (ViT) models after behavior transfer using clean data from CIFAR-10 (C10), CIFAR-100 (C100), TinylmageNet (TIN), and ImageNet (IN). Values reported are accuracy over all clean and corrupted images from the target test set, averaged over seeds.
Model
Source
Target
Accuracy of Behavior Transfer Method (f)
Data
Data
None
LP
LP-FT
Hints
LwF
BIRD
C10
C100
51.31
10.95
51.12
47.56
51.53
52.21
54.77
C10
TIN
20.74
5.24
20.00
18.12
21.27
20.52
24.11
MN2
C100
TIN
20.74
18.84
20.66
23.52
21.26
23.18
25.03
CIO
IN
22.59
1.29
22.50
22.36
22.25
23,23
23.38
C100
IN 22.59
6.18
22.36
22.34
22.49
23.27
23.68
C10
C100
52.03
16.93
51.95
50.63
52.20
55.42
57.39
C10
TIN
20.56
7.25
20.10
19.43
20.92
22.17
23.60
RNI8
C100
TIN 20.56
20.95
20.75
23.66
20.71
24.48
24.49
C10
IN 20.96
5.48
21.02
20.62
22.66
22.25
23.16
C100
IN 20.96
9.93
20.96
20.42
22.44
22.25
23.52
CIO
C100 54.51
23.92
55.84
53.39
54.92
56.92
59.04
C1O
TIN
22.59
10.66
23.39
21.20
22.68
24.14
25.25
DN169
C100
TIN
22.59
23.55
23.19
24.86
22.75
26.14
27.46
C10
IN 26.46
2.39
26.86
26.70
26.59
27.08
27.43
C100
IN 26.46
7.25
26.18
26.16
26.78
27.21
27.76
C1O
C100
50.77
40.07
53.83
54.09
53.57
51.32
53.71
C10
TIN
22.08
20.28
25.82
25.56
25.53
22.28
25.98
VIT
C100
TIN 22.08
22.94
24.24
23.91
25.82
22.36
27.26
C10
IN
25.83
8.71
25.36
25.10
26.20
26.15
26.32
C100
IN
25.83
10.18
24.78
24.68
26.15
26.01
26.28
specific outputs), it captures higher-order relationships that reflect general, behaviorally relevant properties of the teacher's representations (Kornblith al., 2019).
Our approach draws inspiration from recent work in neuroscience cutroscience-ifformed representation learning, where deep networks are trained to jointly minimize task loss and neural alignment loss based on brain recordings (Li et al., 2019; Federer et al., 2020; Safarani et al., 2021; Dapello et al., 2023). Unlike those approaches, BIRD does not require neural data or shared stimulus distributions.
A central distinction between BIRD and prior approaches to distillation and behavior transfer lies its flexibility across heterogeneous models and tasks. Teacher supervision in BIRD comes solely from representation structure, measured via CKA. This design enables BIRD to support behavior transfer in settings where direct supervision or shared objectives are unavailable. Concretely:
BIRD does not require shared input domain, output space, task between teacher and student models.
BIRD does not rely on paired inputs or access to the teacher's training data; supervision is provided by projecting the student's inputs through the teacher's representation space.
Together, these properties set BIRD apart as a practical and general framework for transferring aligned behavior beyond the constraints of existing approaches in KD.
4 TRANSFERRING ROBUSTNESS ACROSS DATASETS AND ARCHITECTURES
We first evaluate BIRD in the context of robust image classification, where the goal is to trans- fer ofttt---dittibuttio (OOD) robustness from a teacher model trained on lower-complexity dataset Dteacher to student model trained clean data from higher-complexity dataset Dstudent (D (Dstudent # D ). This setting follows the aims of weak-to-stroone to-stn generalization, where aligned behavior induced in a larger, less specialized model using supervision from smaller, aligned one.
5

---

Published as a conference paper at ICLR 2026
Feature Stability
Speckle Noise
Caussian Blur
Spatter
Saturate
3.9
80
Coaine Simllarty
BIRD
0.8
60-
0.5
20
0.4
10
15
20
5
3
Training Steps (x107)
Corruption Severity
(a)
(b)
Figure 2: Comparing transferred robustness between BIRD and Hints students (CIFAR-10 trained teacher, TinyIm ImageNet student). (a) Feature stability (average cosine similarity between clean im- ages and their corrupted versions) measured over first 0,,000 steps of training. Feature stability of teacher shown as horizontal dashed line. (b) PGR measured over each corruption type.
Setup and evaluation protocol We consider robust classification transfer across five dataset pairs: CIFAR-10 >> CIFAR-100 CIFAR-10 -> TinylmageNer, CIFAR-100) - TinyImageNet, CIFAR-10 10 + ImageNet, and CIFAR- 100 >> ImageNet (Krizhevsky et al., 2009; Deng al., 2009). Teachers trained be robust to 15 ImageNet-C corruptions (Hendrycks & Dietterich, 2019), while stu- dents see only clean images. To maintain consistency across dataset pairs, all images are resized to 32 x 32 pixels.
Robustness is measured using accuracy over the clean and corrupted test set and Performance Gap Recovered (PGR) (Burns et al., 2023):
ACCpust ACCpre
Accpre .1)-
PGR min max
(4)
ACCoclling
where Accpast is student accuracy after behavior transfer, Accpee is the pre-transfer baseline, and AcCeeiling is the accuracy of a student trained directly with access to OOD corruptions.
We test BIRD across four architectures: MobileNetV2 (Sandler al., 2018), ResNet18 (He al., 2016), DenseNet 169 (Huang et al., 2017), and Vision Transformers (Dosovitskiy et al., 2020). We compare BIRD five baseline strategies that do not access corrupted target training data: linear probing (LP), full fine-tuning (FT), sequential LP followed by FT (LP-FT) (Kumar et al., 2022; Nem et al., 2023), LwF (Li & Hoiem, 2017; Shafahi al., 2019), and hint-based distillation (Hints) (Romero et al., 2014), which aligns activation values between teacher and student via linear map- ping and L2-norm representation loss. Additional training details, corruption visualizations, and evaluation breakdowns are provided in Appendix A.I.
Comparison with baselines Across nearly all dataset pairs and model architectures, BIRD achieves the highest out-of-distribution robustness and PGR (Table For instance, when trans- ferring robustness from CIFAR- 10 to CIFAR-100 100, BIRD, on average, improves robustness by 4.5 percentage points and recovers 31.8% of the performance gap to the robustness ceiling. The next best method, LwF, recovers only 13.5%. Similar trends are observed in the CIFAR-100 Tinylm- ageNet (25.2% vs. 13.8%), CIFAR-10 - TinyImageNet (22.4% vs. 4.9%), CIFAR-10 + ImageNet (8.7% vs. 4.2%), and CIFAR- 100 - ImageNet (10.5% vs. 3.7%) transfers.
LP and alone fail to consistently improve robustness over the baseline. LP-FT provides modest gains, but only in select few architecture, dataset pairs. This highlights a key challenge of behavior transfer from weak sources: robust but highly specific features may not generalize across distribution shifts and are therefore easily forgotten when no structural constraint are imposed during transfer.
Comparison with activation matching To isolate the effect of distilling representational struc- ture, we compare BIRD to Hints, which supervises the student via linear mapping and L2-norm on the same representation layers. Despite identical teacher-student pairs and alignment points, BIRD achieves higher robustmess in every setting and corruption category (Figure 2). This suggests that representation structure (captured via CKA) encodes more generalizable behavioral informa- tion than raw, sample-level activation values. We conclude that BIRD's success stems not just from where it supervises, but from how it supervises.

---

Published as a conference paper at ICLR 2026
Scaling to larger student models We next test whether BIRD generalizes to student models of higher capacity than the teacher. Fixing a MobileNetV2 teacher, we apply BIRD to a series of stu- dent architectures of varying capacity. As shown in Figure 3, robustness improves across all model sizes, including 22.4% PGR for the ResNet1 student, despite it having 25x more parameters than the teacher. These results confirm BIRD's effectiveness for weak-to-strong behavior transfer, even in extreme capacity mismatches, As is common in robust learning, improvements in OOD robustness are sometimes accompanied by minor reductions in clean accuracy. Detailed clean-vs- corruption breakdowns and per-seed results are provided in Appendix A.1.
ABLATION STUDIES
Effect of batch size on behavioral transfer In BIRD, representation structure is quantified over batches of training data. Although CKA is capable of providing meaningful estimates of represen- lational similarity even when the number of samples is smaller than the dimensionality of the repre- sentations (Kornblith et al., 2019), undersampling a representation space with small of batch size may fail to capture nuances of its structure, especially as it relates to model behavior. Appendix Figure shows the effects of batch size on behavioral transfer performance in ResNet18 models in three transfer settings (transferring robust behavior from CIFAR-10 - CIFAR-100, CIFAR-10 -+ TinyymageNet, and CIFAR- - TinylmageNet). We find that while smaller batch sizes (e.g., 32 and 64) still yield performance improvements in robust classification over the original student model, these gains were far less than could be achieved with larger batch sizes, as expected. Sec- tions 4 and 6, maintained consistent batch size of 128 across all evaluated methods. However, the results of this ablation study suggest that increasing the batch size for BIRD could yield even stronger behavioral transfer performance.
Choice of CKA kernel function Throughout this work, use CKA with linear kernel to quan- tify representational similarity, given its extensive use in analyzing deep learning representations (Korbblit et al., 2019; Nguyen al., 2020; Raghu al., 2021) and its ease of interpretation. Al- though we focus on the linear variant, CKA naturally extends to alternative kernels. In follow-up experiments, we observe that using an RBF kernel yields comparable improvements in robust behav- ior transfer (Appendix Table 4). More broadly, the key contribution of BIRD is flexible distillation framework grounded representation structure. In settings where specific geometric properties of the representation space are known to be behaviorally relevant priori, practitioners may benefit from selecting kernels accordingly.
WHAT MAKES A GOOD TEACHER FOR BEHAVIOR TRANSFER?
The preceding experiments demonstrated BIRD's ability transfer robust behavior across datasets and architectures. Here, investigate the conditions under which this transfer is most successful. Specifically, we want to understand what makes a teacher a good source for behavior transfer.
Teacher: MobileNet-v2
CIFAR-10 CIFAR-100
CIFAR-10 NenyLmageNet
TiaylmageNet
Accuracy post BIRD (%)
60
Staden:
Nextiet
Densnket 121
DenssSiet-Lo9
20
15
DenseSket-200
Mootleeee +2
ResMett 101
DestNet-130
HesMete
ResNett-3
30-
BaoNes 50
35
40
45
50
55
15
20
30
20
25
Accuracy pro: BIRD (%)
Figure 3: OOD robustness of student models before and after BIRD, guided by a MobileNet teacher trained on a lower-complexity dataset. BIRD improves robustness across student capacities.

---

Published as a conference paper at ICLR 2026
Setup and analysis framework Our analysis is based the hypothesis that transfer success depends on two measurable properties of the teacher's ' representation space: Task relevance, the degree to which the teacher's 's representations Behavioral relevance, the extent to which the teacher's representations support aligned behavior.
informative for the student's 's downstream task, and
We construct pool of 144 teacher models varying along these two axes. Teachers trained one of three datasets (CIFAR-10 10, CIFAR-100, or TinylmageNet) to manipulate task relevance. Behav- ioral relevance is varied by augmenting training data with randomly selected subsets of ImageNet-C corruptions. Teachers span four architectures: AeexNet (Krizhevsky et al., 2012), ResNet18 (He al., 2016), 1enseNet121 (Huang al., 2017), and MobileNetV2 (Sandler et al., 2018).
Each teacher is used to supervise ResNet50 student trained on one of three datasets (CIFAR-10, CIFAR-100, or TinylmageNet), resulting in 432 teacher-student pairs. We quantify properties of each teacher's representation space using established metrics (Alain & Bengio, 2016; llyas et al., 2019) that assess downstream utility of feature space:
Task relevance We train linear probes on the teacher's s representation using clean data from Dstudent and measure (i) Probing accuracy: classification accuracy of the linear probe (Alain & Bengio, 2016) on held-out student data (ii) Complementary knowledge: fraction of student samples correctly classified by the teacher's probe but not by probe trained on the student's own representation (Roth et al., 2023).
Behavioral relevance We aggregate the y-robust usefulness of each feature (Ilyas et al., 2019), which measures whether features retain predictive value under corruptions. Full computation details are provided in Appendix A.4.
Explaining transfer success For each student dataset, we fit linear model to predict accuracy on the corrupted test set using the three metrics described above. We report R² values to quantify explained variance in behavior transfer from these properties of the teacher's representation space.
Results Impressively, these simple, interpretable properties explain the vast majority of transferred robustness. We observe strong predictive power across all student datasets. For students trained on TinylmageNet, the model explains 81.8% of the variance in PGR; for CIFAR-100 100, 85.5%; and for CIFAR-IO, 73.6% (Appendix, Figure 7). Across all student datasets, the most predictive single factor is the behavioral relevance of the teacher's representation space, as measured by aggregated 7- robust usefulness, explaining more than 50% of the variance in PGR alone. Task relevance metrics (probing accuracy and complementary knowledge) add additional explanatory power, especially when transferring to more complex student datasets like TinylmageNet.
Implications for teacher selection These findings provide actionable guidance for selecting or training effective teacher models. When choosing among candidate teachers, we recommend prior- itizing those with high behavioral relevance (i.e., the behavior of interest is strongly encoded in the teacher's 's representation space), even if they were trained on different dataset or have limited task overlap with the student.
7 TRANSFERRING BEHAVIOR IN LANGUAGE MODELS
To assess the generality of BIRD beyond robustness transfer in vision, we evaluate its application in two distinct language modeling settings: safety alignment and weak-to-strong generalization.
7.1 TRANSFERRING SAFE BEHAVIOR
Safety alignment, fine-tuning models so that their outputs are safe with respect to human-defined criteria, is central challenge in alignment research. A common approach is DPO (Rafailov al., 2023), which optimizes alignment using paired preferred and rejected responses to the same query.
We investigate whether supplementing DPO with BIRD do this, we encourage a generative student model to adopt a representational structure similar to that of simple teacher classifier trained to distinguish safe from unsafe responses (Appendix A.7.2).
further improve safety alignment. To
8

---

Published as a conference paper at ICLR 2026
Setup and Evaluation Protocol We evaluate safety alignment the 135M and 360M parame- ter variants SmolLM2, -fttear, "small" language models that are accessible within our compute budget (Allal al., 2025). For preference supervision, sample 10, 000 examples from the PKU-SafeRI.HE dataset (Ji et al., 2024a), which contains prompts paired with safe and unsafe responses, as evaluated across 19 safety dimensions (Appendix 1.7.1). Student models trained with two strategies: (i) DPO and (ii) DPO suppiemented with BIRD loss (DPO+BIRD), In the latter setting, student representation structure is supervised by that of simple binary classifier fine-tuned on AKU-SafeRLHH-Q2 (Ji et al., 2024b) dataset to predict whether or not given response is safe nrisk-neutral according all 19 safety dimensions). To evaluate safety alignment, we report the percentage of generated responses queries from the PKU-SafeRLHF test set judged as safe by an independently trained safety classifier (Appendix A.7.3).
Results Experimental results are summarized in Table 2. Across three seeds, all models fine-tuned on the safety preference dataset outperform the bascline instruction tuned model (None), and models trained with BIRD further produce a higher proportion of safe responses than DPO alone. Sample responses are provided in Appendix A.7.4 ..4 Although relatively small language models were studied, demonstrates BIRD's 's generalization beyond robustness transfer in vision to the challenge of safety alignment in language models. Future work will extend these analyses to larger models.
Table 2: Safety alignment performance of generative models on the PKU-SafeRLHF test set. % Safe: percentage of query responses from that model evaluated as safe according to an LLM judge.
Student
% Safe (1)
None
DPO
DPO+BIRD
SmolLLM2-13MM-nssttuu
43.88
65.48
71.28
SmolLM2--36M-IIssrruud
47.63
86.57
88.37
7.2 EXTENDING SOFT-LABEL WEAK-TO-STRONG GENERALIZATION WITH BIRD
We further assess the generality of BIRD in the context of teek-too-troong generalization (Burns al., 2023), in which seek distill capabilities from low-capacity, aligned teacher into higher-capacity, unaligned student model without access to ground-truth labels.
Setup and evaluation protocol We adopt the setup introduced in Burns et al. (2023), in which small, aligned teacher model is fine-tuned on target dataset and then used to supervise larger student. We use GPT2-Small as the teacher and either GPT2-Medium or GPT2-Large as the student.
We consider three multiple-choice question-answering datasets: SciQ (Johannes Welbi, 2017): sci- ence questions with factual answers: BoolQ (Clark et al., 2019): yes/no questions based on para- graph context; Cosmos QA (Huang al., 2019): commonsense inference over short passages.
Each student is trained using two supervision strategies: (i) Soft-label distillation: cross-entropy loss between student predictions and the teacher's soft probabilities, (ii) Saft-label + BIRD: soft- label loss, accompanied with BIRD's representaaioon-struuuue loss computed over the final token embedding layer of teacher and student. Following Burns et al. (2023), we report PGR on a held-out test set relative to ceiling set by fine-tuning of the student on target data with ground-truth labels.
Results Table 3 summarizes performance across all dataset and model combinations. In three of the six configurations, adding BIRD to soft-label distillation yields a noticeable improvement in behavior transfer. On BoolQ, neither soft-label distillation nor BIRD outperform the weak teacher (0% PGR). This suggests that not all aligned behaviors captured by small models generalize to more complex reasoning tasks that representational distillation at the final layer may not be sufficient in this domain. This highlights a broader opportunity: combining BIRD with additional alignment signals (e.g., multi-layer supervision) may offer more consistent benefits in complex domains.

---

Published as a conference paper at ICLR 2026
Table 3: PGR (averaged over 3 seeds) for GPT2-Medium and GPT2-Large models trained soft-labels from GPT2-Small (Soft-Label) or soft-labels with BIRD (+BIRD).
Dataset
Student
% PGR (t)
Soft-Label
+BIRD
GPT2-Medium
7.79
16.14
SciQ
GPT2-Large
17.70
24.19
GPT2-Medium GPT2-Large
47.00
42.76
Cosmos QA
65.51
68.02
DISCUSSION
We introduce Behavior Induction via Representation-strucuure Distillation (BIRD), simple and general framework for transferring aligned behavior by distilling the structure teacher model's internal representations. Unlike prior approaches, BIRD does not require the teacher and student to share common training data or domains, output spaces, or architectures between teacher and student. Our experiments in robust image classification show that BIRD consistently outperforms existing transfer methods, even when the teacher is significantly smaller and trained on a simpler dataset. We further identify three interpretable properties of the teacher's representation space that strongly predict transfer success. Finally, show that BIRD provides complementary gains when paired with existing methods for safety alignment and weak-to-strong generalization language models. These results are not intended new state-of-ther toffthe-an art benchmarks, but rather evidence of BIRD's generality and its ability to enhance existing approaches without requiring new data or reward modifications.
Key takeaways BIRD contributes two key advances. First, it establishes flexible, drop-in inop-in mech- anism for aligned behavior transfer that operates across domains, architectures, and label spaces by supervising over representation structure. Second, it offers a systematic analysis of what makes a good teacher for behavior transfer. Our findings suggest that small and simple models can as effective alignment scaffolds for larger, unaligned models if their representations are behaviorally relevant. BIRD is especially useful in settings where aligned behavior must be scaled without training large models or access to private data.
Limitations and future directions Extending BIRD additional behaviors beyond robustness and safety (c.g., honesty) and to model-pairs that differ in modality is a promising direction that we are excited to investigate future work. We provide initial evidence for the utility of BIRD when used alongside foundation vision-language encoders in Appendix A.6. Additionally, transfer success is likely bounded by student capacity and task complexity; future work should quantify this saturation effect more precisely (Appendix A.5). Further, currently supervise alignment at single layer selected by a heuristic. While our findings suggest that exact layer choice is not overly sensitive (Appendix 1.4.,, future work may explore multilayer extensions capture deeper structural alignment. We expect that the primary challenges that exist with such a strategy include selecting optimal alignment layer pairs and selecting the weight that each layer pair contributes to the BIRD representation loss (challenges that we were able to approach in single-layer alignment by probing properties of candidate alignment layers and balancing task and representation losses (Appendix A. and A.1.3)). We perform an initial multi-layer study, naively selecting layers at four regularly-spacee spaced model depths and weighing representation loss equally across aligmment layers in Appendix A.3. Finally, our behavioral relevance metric is tailored to robust classification; in other domains, tools like linear tomography (Zou et al., 2023) or causal mediation analysis (Vig et al., 2020) may help localize behavior-relevant representations.
ETHICS STATEMENT
BIRD is intended as a tool to improve human-Al alignment. However, its effectiveness depends on the teacher: if the teacher encodes biased, harmful, or misaligned behaviors, BIRD may transfer
10

---

Published as a conference paper at ICLR 2026
those behaviors to the student. Ensuring the integrity of teacher models is therefore essential for responsible deployment.
REPRODUCIBILITY STATEMENT
We make public implementation of BIRD available at github. com/ hoagg//goocceeeeeeoooee ff / /bird. Training and evaluation configuration details required for reproduction (c.g.. choice of optimizers, data augmentations, and hyperparameters) are provided in the appendix.

---

Published as a conference paper at ICLR 2026
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, arXiv preprint arXiv:1610.01644, 2016.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint arXiv:2502.02733: 2025.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint 1rXtv:16066066655, 2016.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceed- ings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY. USA, 2006. Association for Computing Ma- chinery. ISBN 1595933395. doi: 10.1145/115040..1 150464. 1145/1150402 1150464.
URL https::/dooii /0.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong general- ization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
Dian Chen, Hongxin Hu, Qian Wang, Li Yinli, Cong Wang, Chao Shen, and Qi Li. Cartl: Coopera- tive rdversariilly--oousss transfer learning. International Conference on Machine Learning, pp- 1640-1650. PMLR, 2021.
Yeo Chung and Lanry F Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology. 70:137-144, 2021.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
Joel Dapello. Kohitij Kar, Martin Schrimpf, Robert Baldwin Geary, Michael Ferguson, David Daniel Cox, and James DiCarlo. Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robustness. In The Eleventh In- ternational Conference Learning Representations, 2023. URL https: attpss://operreiees openrevien.ee... net/forum? Scc.iddddssdyj.h.....
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal mis- generalization in deep reinforcement learning. In International Conference on Machine Learning, pp. 12004-12019, PMLR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in neural information processing systems, 34:21480-21499 2021.
Callie Federer, Haoyan Xu, Alona Fyshe, and Joel Zylberberg. Improved object recognition using neural networks trained to mimic the brain't 's statistical properties. Neural Networks, 131:103-114 2020.
12

---

Published as a conference paper at ICLR 2026
Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How to train vision transformer small- scale datasets? In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022. URL httpt::/tbtcceaeeeeeeeee http://bwwcc2222m mpi- inf mpg. de/ 0731.pd3....
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2827- 2836, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint 1903. 122611, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of out-of-distribttion generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1003v.1503.0255: 2015.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learming deep representations by mutual information estimation and maximization. arXiv preprint /80v: 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 47000 2017.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 2391-2401 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/1//D19-12222 URL 1243.
https: www. acl web. www.wrg......................... http://ntoolggyyygoooo
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, al. Ai alignment: comprehensive survey, arXiv preprint arXiv:2310, 19.2222......... 2023.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: Towards multi-level safety alignment for 1lms with human preference. arXiv preprint arXiv:2406.1550 2024a.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Imm via human- Human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024b.
Can Jin, Tong Che, Hongwu Peng, Yiyuan Li, Dimitris Metaxas, and Marco Pavone. Learning from teaching regularization: Generalizable correlations should be easy to imitate. Advances Neural Information Processing Systems, 37:966 994, 2024.
Matt Gardner Johannes Welbl, Nelson E. Liu. Crowdsourcing multiple choice science questions. 2017.
13

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
Simon Kornllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012. URL
neurips cc paper. fi wwlle//ppper//000 pdf.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint 2000::2002.10054. 2022.
Zhe Li, Wieland Brendel, Edgar Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian Sinz, Zachary Pitkow, and Andreas Tolias. Learning from brains how regularize machines. Advances in neural information processing systems, 32, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, (0(12):2935-2941 2017.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1 1986, 2022.
Ziquan Liu, Yi Xu, Xiangyang Ji, and Antoni B Chan. Twins: fine-tuning framework for improved transferability of adversarial robustness and generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16436-16446, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Patrick Mineault, Niccold Zanichelli, Joanne Zichen Peng, Anton Arkhipov, Eli Bingham, Julian Jara-Ettinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, et al. Neuroai for ai safety. arXiv preprint arXiv:2411.18522 2024.
Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael Mozer, Klbus-Robert MÅÄller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstraction levels. arXiv preprint S.Xv:2409.065000 2024.
Laura F Nern, Harsh Raj. Maurice André Georgi, and Yash Sharma. On transfer of adversarial robustness from pretraining downstream tasks. Advances in neural information processing systems, 36:59206-59226, 2023.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327 2020.
Pedro Ortega, Vishal Maini, and DeepMind Safety Team. Building safe artificial intelligence: spec- ification, robustness, and assurance, 2018. URL https: medi um. com/bui aaa.-aaa--rrriiiiiiiii..eo intiiiilllnnelllleeeeeeeee 52555
14

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Sheng Y Peng, Pin- Yu Chen, Matthew Hull, and Duen H Chau. Navigating the safety landscape: Measuring risks in finetuning large language models. Advances in Neural Information Processing Systems, 37:95692-95715. 2024.
Galen Pogoncheff, Jacob Granley, Alfonso Rodil, Leili Soo, Lily Maric Turkstra, Lucas Gil Nadol- skis, Arantxa Alfaro Saez, Cristina Soto Sanchez, Eduardo Fernandez Jover, and Michael Beyeler. Beyond sight: Probing alignment between image models and blind v1. In ICLR 2024 Workshop on Representational Alignment, 2024.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03699 2023.
Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Bet- ter teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206. 266607 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:121 16-1222, 2021.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.088-77 2024.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Karsten Roth, Lukas Thede, Almut Sophia Koepke, Oriol Vinyals, Olivier Hénaff, and Zeynep Akata. Fantastic gains and where to find them: On the existence and prospect of general knowl- edge transfer between any pretrained model. arXiv preprint arXiv:2310. 130.1333.., 2023.
Shahd Safarani, Ame Nix, Konstantin Willeke, Santiago Cadena, Kelli Restivo, George Denfield, Andreas Tolias, and Fabian Sinz. Towards robust vision by multi-task learning monkey visual cortex. Advances in Neural Information Processing Systems, 34:739-751, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510-4520, 2018
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neural information processing systems, 31, 2018
Marius Schneider, Joe Canzano, Jing Peng, Yuchen Hou, Spencer LaVere Smith, and Michael Beyeler. Mouse vs. ai: A neuroethological benchmark for visual robustness and neural align- ment. ArXiv, pp. arXiv-2509, 2025.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. arXiv preprint arXiv: 1905.0822. 2019.
15

---

Published as a conference paper at ICLR 2026
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 10699, 2019.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 13:12288-12001 2020.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: fatte-of-the-ar natural language processing. arXiv preprint arXiv: 1 2019..
Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan S Kankanhalli. Enhancing adversarial contrastive learning via adversarial invariant regularization. Advances in Neural In- formation Processing Systems, 36:16783-16688 2023.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, al. Catalyzing next-generation artificial intelligence through neuroai. Nature communications, 14(1):1597, 2023.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: FrrXiv:661.... 1612,0398, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Yucheng Zhou, Jianbing Shen, and Yu Cheng. Weak to strong generalization for large language models with multi-capabilitiss. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimization: Stealing reward from weak aligned model. arXiv preprint arXiv:2410.18644 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo. Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation enginecring: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01400 2023.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EVALUATING ROBUST BEHAVIOR TRANSFER
A.1.1 OUT-OF-DISTTIBUTIOO CORRUPTIONS FROM THE IMAGENET-C BENCHMARK
The ImageNet-C benchmark (Hendrycks Dietterich, 2019) is a standardized suite for evaluating the robustness of image classification models to common corruptions. These image cornuptions are algorithmically generated to simulate four different categories of real-world sources of image corruption (noise, blurring, weather effects, and digital effects). These corruptions can be readily applied to images from datasets other than ImageNet (e.g., CIFAR-10, CIFAR- 1-00, and Tinylma- geNet), was done in this work. There are 19 total corruption types, each from one of these four corruption categories (Figure 4). Each corruption is additionally applied at five levels of severity, reflecting of the magnitude of the corruption (Figure 5). models are trained to be robust to 15 of these corruption types ("gaussian noise", "shot noise", "impulse noise", "defocus blur", "glass blur", "motion blur", "zoom blur", "snow", "frost", "fog", "brightness", "contrast", "brightness", "pixelate", and "jpeg compression") and, after robust transfer, students are evaluated on the remaining four held out corruption types ("speckle noise", "gaus- sian blur", "spatter", and "saturate"), one from each corruption category, to evaluated generalized robustness (Hendrycks & Dietterich, 2019).
experiments Section teacher
Caussian
Shat
Impalse
Motion
Brightness
Contrast
Elastic Transform
Pxxelae
Compression
Spockle Naise
Gaussian Blar
Spatter
Saturate
Figure 4: Sample CIFAR-10 image distorted with corruptions from the ImageNet-C benchmark (Hendrycks Dietterich, 2019). "Clean" designates the original, uncorrupted image. Corruptions are depicted at maximal severity.
17

---

Published as a conference paper at ICLR 2026
Corruption Type: Speckle Noise
Clean
Severity:
Severity: 2
Severity: 3
Severily:
Severity:
Figure 5: Sample CIFAR-10 image distorted with "Speckle Noise" corruption severities 1-5.
A.1.2 ROBUST TRANSFER BASELINES
In Section 4, seek compare robust behavior transfer with BIRD to robust transfer learning baselines that (I) solely rely on learning from clean data during the transfer learning task and (2) do not make inherent assumptions about the exact robust pre-training method. These two criteria strongly reflect the behavior transfer paradigm that is studied in this paper: transferring behavior of an aligned model to new, target domain using only the data that already have available for the target domain. To the best of our knowledge, the most relevant and highest performing methods that meet these criteria are robust transfer learning using continual learning strategies (LwF) (Shafahi et al.. 2019), Linear Probing (LP), Fine-tuning (FT), and Linear Probing followed by Fine-tuning (LP-FT) (Shafahi et al., 2019; Kumar al., 2022; Nern et al., 2023).
Linear Probing (LP) In LP, the robust, pre- pre-trained feature extractor of model is frozen while new linear classification head is trained for the downstream task. This is a computationally efficient approach, often used when labeled data is scarce. Since the feature extractor is not updated, the robustness of the new model is largely dependent on the robustness of the features learned in pre- training (Shafahi et al., 2019; Nern al., 2023).
Fine-tuning (FT) In FT, the entire model (i.e., the feature encoder and classification head) is updated as the model is trained on the new task or domain. This provides more flexibility than LP, as the model is able to learn new features that support better clean performance the new downstream task, but risks degrading the robustness of the original representations (Shafahi et al., 2019; Nern et al., 2023).
Linear Probing followed by Fine-tuning (LP-FT) LP-FT is performed in two steps. First, robust model's feature extractor is frozen and new classification head for the target dataset is learned (LP). Next, full-model fine-tuning is performed, updating both the model's 's feature extractor and new classification head. Kumar et al. demonstrate that fine-tuning a model with a new, randomly initialized classification head can distort a model's learned features (ultimately contributing to worse OOD performance) and that these effects can be mitigated by first learning classification head for the new task using LP (Kumar et al., 2022). Ner at al. employ this strategy in the context of transferring adversarial robustness an effort to combine the benefits of LP (preserving robustness) and FT (improving performance) (Nern al., 2023).
Learning without Forgetting (LwF) Shahafi et al. suggest the use of LwF, popular strategy used lifelong learning tasks (Li Hoiem, 2017), as method to preserve adversarial robustness when end-to-end fine-tuning model for new data distribution (Shafahi et al., 2019). The ap- proach augments the training objective with a loss that penalizes deviations between penultimate layer feature activations of the fine-tuned model and those of robust source model. Unlike tradi- tional LwF which distills from logits, this variant applies the loss directly to the penultimate feature layer, encouraging the student to retain robust internal representations even it adapts to the target domain.
A. 1.1 TRAINING DETAILS
All experimental results for robust behavior transfer of Section 4 are reported over three seeds. For each method that introduced additional hyperparameters, instantiate that method with range of
18

---

Published as a conference paper at ICLR 2026
hyperparameter values and report best performance achieved. Specifically, for LwF (Shafahi al., 2019) and Hints (Romero et al., 2014), we tune the strength of the loss that penalizes divergence in feature activation values. For FT, LP, FT-LP, and LwF, additionally trained models with three unique initial learning rates and always report best performance. For BIRD-tuned models, we select and B by first evaluating Lrask and Lepp
the clean validation dataset, and then set
Crapk and 3 = 1 - a. balancing the contribution of these two losses without hyperparameter optimization. We use fixed batchsize (128) and optimization strategy in all methods (stochastic gradient descent with momentum and weight decay for CNN models, and Adam with weights decay for VIT models) and decay the learning rate with a cosine decay schedule over the course of training. While training CNN models, training images were augmented with random cropping and horizontal flipping. Training images were additionally augmented using CutMix (Yun et al., 2019), mixup (Zhang et al., 2017), and random crasing (Zhong et al., 2020) strategies while training ViT models, following the work of Gani al. (2022). During robust transfer, all student models were exclusively trained on clean images. Early stopping was performed using a clean validation set.
Crep
A.1.4 LAYER SELECTION
To select the distillation layers for the teacher and student models, we applied simple heuristic based on representation measurements reported in Section 6. Specifically, over multiple candidate layers associated with natural transition points in each architecture (e.g., the end of a ResNet, Mo- bileNet, or DenseNet block), we computed two metrics (linear-probing accuracy and 7-robust use- fulness from the CIFAR-10 - CIFAR- 100 transfer task) as proxies of task relevance and behavioral relevance of the representations from that layer, and selected the layer with highest composite score the distillation anchor. This composite score computed the mean of these metrics, after 0-1 scaling each metric over the candidate layers from a given network.
While this procedure may not guarantee optimal performance for every BIRD-tuned model, it pro- vided a computationally efficient proxy for selecting meaningful layers without exhaustively search- ing all possibilities. In practice, this strategy yielded strong results across architectures, and further improvements are likely achievable through layer-level validation when resources permit. For con- sistency and simplicity, we fixed the selected layer within each model family (e.g., always distilling from Block 3 outputs in ResNets) and transfer learning configuration (i.e., CIFAR-10 + CIFAR- 100, CIFAR-10
Imayent and CIFAR-100 + TanylmageNet). A broader evaluation of alternative layer choices is shown in Figure 8. Consistently. we observe that distilling representation structure at layers between the middle and end of the network offers best performance in this robust behavior transfer task. Small deviations (e.g.. few hidden layers) away from the optimal distillation layer does not drastically reduce performance.
A.1.5 EXPANDED RESULTS
Individual seed results for each robust transfer method evaluated in Section 4 are provided in Tables 7,8. Clean accuracy of each model is reported in Table 9, Figures 9, 10, and 11 show PGR of all methods for each corruption type and severity.
A.1.6 PERFORMANCE OVER DISTRIBUTION OF CLASSES
Figure 12 visualizes within-class accuracy change distributions after applying BIRD. We observe that improvements robustness are broadly distributed across classes, indicating that the gains are limited to the few categories of the target dataset that are most similar those of the teacher's source dataset.
A.1.7 ABLATION STUDIES
Batch size Figure 6 shows the OOD robustness of ResNet18 models in three transfer settings (CIFAR-10- - CIFAR-100, CIFAR-10 - TinylmageNet, and CIFAR-100 - TinylmageNet) when distilling robust behavior with BIRD with varying batch sizes. As expected, larger batch size enable better quantification of the teacher and students' representation structure, yielding greater robust behavior transfer.
19

---

Published as a conference paper at ICLR 2026
TlyimaggNee
1nyiinggeeet
88
23
57
34
Auturacy
24
23
21
22
22
53
Omggal
21
Original
21
52
4
5
6
9
10
11
8
10 11
logy(Ratch Size)
Figure 6: Robust accuracy of ResNet18 models when tuned using different batch sizes in BIRD. The horizontal dashed line reflects the accuracy of the original student, before distillaiton.
CKA Kernel OOD robustness of ResNet18, MobileNetV2, DenseNet169, and VIT models after distilling robust behavior using BIRD and CKA with either linear kernel RBF kemel (with bandwidth a set equal to the median pairwise distance between training samples) is provided in Table 4.
Table 4: Accuracy (%) of MobileNetV2 (MN2), ResNet18 (RN18), DenseNet169 (DN169), and Vision Transformer (ViT) models after behavior transfer using BIRD with CKA representation loss computed with Linear and RBF kernels.
Model
Source
Target
Accuracy BIRD Behavior Transfer (1)
Data
Data
Kernel: Linear
Kernel: RBF
C10
C100
54.77
54.74
MN2
C10
TIN
24.11
24.13
C100
TIN
25.03
24.86
C10
C100
57.39
57.16
RN18
C10
TIN
23.60
23.71
C100
TIN
24.49
24.67
C10
C100
59.04
58.31
DN169
C10
TIN
25.25
25,49
C100
TIN
27.46
27.87
C1O
C100
53.71
54.67
ViT
C10
TIN
25.98
26.02
C100
TIN
27.26
27.54
A.2 LEARNING FROM WEAK TEACHERS
Quantitative results associated with Figure (performing behavior transfer from MobileNetV2 teacher to student models of varying capacity) are provided in Table 10.
A.3 PRELIMINARY EXPERIMENTS IN MULTI-LAYER ALIGNMENT
Table provides robust behavior transfer results when BIRD was applied across multiple layers. In this preliminary analysis, layer pairs were naively selected at four regular depth interval in each seacher-student pair and we uniformly weighed the contribution of each layer pair to the represen- tation loss in equation 1. Specifically,
compute representation losses at Blocks 1, 2, 3, and 4 in Resnet18, Blocks 3, 7, 11, and 17 in MobileNetV2, Transition Layers 1, 2, 3, and Dense Block 4 in DenseNet1699 69, and Encoder Blocks 3, 5, and 7 in VIT.
20

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy (%) of MobileNetV2 (MN2), ResNet18 (RN18), DenseNet169 (DN119),
and Vision Transformer (ViT) when tuned using BIRD with singlc-layer alignment and naive multi-layer alignment.
Model
Source
Target
Accuracy of BIRD Behavior Transfer (1)
Data
Data
Single-layer BIRD
Multi-layer BIRD
C10
C100
54.77
55.33
MN2
C10
TIN
24.11
24.25
C100
TIN
25.03
25.03
C10
C100
57.39
56.54
RN18
C10
TIN
23.60
23.65
C100
TIN
24.49
25.28
CIO
C100
59.04
57.21
DN169
C10
TIN
25.25
25.65
C100
TIN
27.46
28.22
C1O
C100
53.71
53.07
ViT
C1O
TIN
25.98
24.80
C100
TIN
27.26
26.84
A.4 EXPLAINING ROBUST BEHAVIOR TRANSFER
A.4.1 QUANTIFYING BEHAVIORAL RELEVANCE OF TEACHER'S FEATURE SPACE
To quantify the behavioral relevance of a teacher model's representations, we adapt the notion of y-robust useful features from Ilyas al. (Ilyas al., 2019), which defines feature robustly useful if it remains predictive of the true label even under allowable input perturbations. While the original formulation applies to binary classification under adversarial perturbations, extend this idea to the multi-class setting and apply to common image corruptions (rather than adversarial attacks).
We compute a behavioral relevance score for a given model layer using the following procedure:
Compute class-wise 7-robust usefulness: For each feature dimension in the representation space, and for each class e in the target dataset, we binarize the class labels such that samples from class e labeled +1 and all other samples -1. We then evaluate the feature's 's correlation with this binary labeling across target dataset images distorted with corruptions from the ImageNet-C benchmark. This gives robustness-aware measure the feature's predictive power for each class.
2. Feature-level aggregation: For each feature dimension, compute the 90th percentile of its y- robust usefulness scores across classes. This choice reflects the intuition that a useful feature may be strongly predictive of subset of classes.
Layer-level aggregation: Finally, we aggregate over all feature dimensions by taking the median of the feature-level scores. This yields single scalar score representing the typical r-robust usefulness of features in the layer interest.
This metric allows compare candidate teacher layers by how robustly informative their features are, even in the presence corruptions, and serves as proxy for their suitability in behavior transfer via BIRD.
A.5 ROBUST TRANSFER FROM UNDER EXPRESSIVE STUDENTS
While our results demonstrate that robust behavior can be transferred from low-capacity teachers to nigher-capacity students, there are signs of a saturation effect: when the gap between teacher and student capacity becomes too large, the effectiveness of behavior transfer diminishes. In Section 6, we seek to transfer robust behavior from AlexNet, ResNet18, DenseNet121, and MobileNetV2 teachers to ResNet50 students. AlexNet teachers, which substantially underperform all other teach-
21

---

Published as a conference paper at ICLR 2026
CIFAR 10
CIFAR-IN 100
TinyImageNet
90
Observed Accuracy (%)
62-
86
60-
84
58
26
82
56
24
800
54
22
R²
7736
52
R2
.855
R
.188
78
00
00
52
23
20
30
Predicted Accuracy (%)
Figure 7: Variance in robust accuracy explained by linear models trained on teacher representation properties. Horizontal lines indicate student performance before BIRD.
ers in both clean and robust image classification, frequently fail to serve useful teachers for trans- ferring robust behavior, especially the case if leads us suggest that extremely weak under-expressive teachers may lack sufficiently structured representations to guide meaningful behavior transfer via BIRD.
target datasets (Figure 13). This
A.6 DISTILLING BEHAVIOR FROM CLIP VISION- LANGUAG ENCODERS
Vision-language encoders trained with Contrastive Language-Image Pre-Training (CLIP) are known to learn semantically rich, generalizable representations that outperform standard supervised models on natural distribution shifts (Radford al., 2019). For example, CLIP encoders have been shown to achieve drastically higher zero-shot accuracy than ImageNet-traines classifiers on datasets such as ImageNet-Skenl SSeethh, which replaces photographs with sketches of the same classes (Radford al., 2019; Wang et al., 2019). Motivated by this, we study whether BIRD can help transfer CLIP's semantically structured and generalizable vision capabilities to a lightweight ImageNet classifier. Specifically, we distill from CLIP-ViT-BIA into ConvNeXt-Tinn model (Liu et al., 2022).
Setup and Evaluation Protocol We consider three methods for transferring this behavior: (1) Linear-Probing + Logits-based Knowledge distillation (LP+KD): First, train a linear probe for Im- ageNet classification using image embeddings from the CLIP encoder, and then perform logits- based knowledge distillation (Hinton al., 2015), computing teacher logits from this linear probe: (2) hint-based distillation (HINTS): the ConvNeXt student is tuned to minimize classification loss while learning aligned activation values with the CLIP teacher embeddings via linear mapping and L2-norm representation loss; (3) BIRD: the ConvNeXt student is tuned to minimize both classifi- cation loss and CKA-based representation structure loss between the CLIP teacher embeddings and student representations. I.P, FT, LP-FT, and LwF methods from Section 4 were excluded from this analysis as they involve fine-tuning the teacher, which not our aim in this specific experiment. For each method, distillation tuning performed using the ImageNet training dataset. We evalu- ate efficacy of the behavior transfer method by computing the student's 's accuracy on the held-out ImageNet-Sketch dataset after distillation.
Results of these three configurations are provided Table 6. We compute PGR with respect to the classification accuracy of the CLIP linear probe on the ImageNet-Sker Skeech dataset.
Table 6: Accuracy (%) and PGR (%) of ConvNeXt-Tiny student on ImageNet-Sketch after distillation from CLIP (ViT-B/16) encoder according to Proear-Probin+LLggii based Knowledge Distillation (LP++DD) hint-based distillation (HINTS), and BIRD.
Distillation Method
Accuracy (% PGR) (t)
LP+KD
31.33 (7.23)
HINTS
32.26 (13.76)
BIRD
32.63 (16.36)
22

---

Published as a conference paper at ICLR 2026
Table Per-seed accuracy (%) of MobileNetV2 (MN2) and ResNet18 (RN18) models after behav- ior transfer using clean data from CIFAR-10 (C10), CIFAR-100 (C100), TinylmageNet (TIN), and ImageNet (IN). Values reported are accuracy over all clean and corrupted images (for held-out cor- ruption types "speckle-noise", "gaussian-blur", "spatter", and "saturate" and corruption severities
1-5) from the target test set.
Model
Source
Target
Seed
Accuracy of Behavior Transfer Method (t)
Data
Data
LP
LP-FT
Hints
LwF
BIRD
C10
C100
10.95
51.38
50.44
51.63
52.64
54.57
C10
C100
10.93
50.86
46.25
51.48
52.11
54.91
C10
C100
2
10.97
51.13
46.06
51.49
51.88
54.84
C10
TIN
5.25
20.08
19.79
21.48
20.51
23.97
MN2
C10
TIN
5.25
19.90
17.12
21.21
20.44
24.29
C10
TIN
5.22
20.03
17.45
21.13
20.62
24.08
C100 TIN
O
18.81
20.69
23.45
21.39
23.11
25.03
C100 TIN
18.87
20.76
23.68
20.76
22.97
25.06
C100 TIN
2
18.82
20.52
23.44
21.64
23.46
24.99
C10 IN
O
1.26
22.43
22.20
23.23
22.51
23.51
C10 IN
I
1.30
22.51
22.55
23.21
22.03
23.21
C100 IN
C10 IN
2
1.32
22.56
22.32
23.25
22.22
23.42
O
6.19
22.25
22.34
22.65
23.20
23.70
C100
IN
6.16
22.58
22.24
22.40
23.27
23.70
C100
IN
6.19
22.26
22.46
22.41
23.34
23.65
C10
C100
16.94
51.69
52.39
52.39
55.34
57.37
C10
C100
16.92
52.08
49.46
52.03
54.60
57.57
C10
C100
16.94
52.09
50.02
52.18
56.33
57.23
C10
TIN
O
7.26
20.01
20.07
20.82
22.11
23.12
RN18
C10
TIN
7.24
19.87
19.01
20.96
21.82
23.82
C10
TIN
2
7.26
20.41
19.14
20.97
22.59
23.86
C100 TIN
20.97
20.98
23.88
20.73
24.22
24.27
C100 TIN
20.89
20.59
23.51
20.60
25.10
24.53
C100 TIN
2
20.98
20.68
23.58
20.80
24.13
24.67
C10 IN
5.49
20.91
20.70
22.23
22.78
23.16
C10 IN
5.47
21.07
20.52
22.31
22.66
23.15
C10 IN
5.47
21.08
20.65
22.22
22.54
23.16
C100 IN
O
9.96
21.01
20.52
22.26
22.55
23.55
C100 IN
9.91
21.21
20.31
22.27
22.34
23.50
C100 IN
2
9.91
20.66
20.44
22.23
22.43
23.50
23

---

Published as a conference paper at ICLR 2026
Table 8: Per-seed accuracy (%) of 6enseNet169 (DN169) and Vision Transformer (ViT) models after behavior transfer using clean data from CIFAR- 10 (C10), CIFAR- 100 (C100), TinylmageNet (TIN), and ImageNet (IN). Values reported are accuracy over all clean and corrupted images (for held-out corruption types "speckle-noise". "gaussian-blur" "spatter", and "saturate" and corruption
severities 1-5) from the target test set.
Model
Source
Target
Seed
Accuracy of Behavior Transfer Method (1)
Data
Data
LP
FT
LP-FT
Hints
LwF
BIRD
C10
C100
0
23.97
56.08
54.92
55.14
56.44
59.27
DN169
C10
C100
1
23.88
55.94
52.76
54.45
57.21
59.50
C10
C100
2
23.91
55.51
52.48
55.18
57.11
58.36
C10
TIN
O
10.60
23.21
22.56
22.62
24.15
25.26
C10
TIN
10.70
23.86
20.55
22.68
24.28
25.28
C10 TIN
10.67
23.10
20.47
22.76
23.99
25.21
C100 TIN
O
23.54
22.93
24.77
22.67
26.70
27.27
C100 TIN
23.55
23.24
25.12
22.86
25.32
27.64
C100 TIN
2
23.58
23.41
24.70
22.72
26.41
27.48
C10 IN
2.32
26.79
26.63
26.96
26.45
27.42
C10 IN
2.49
26.72
26.78
27.15
26.88
27.42
C100 INNI
C10 IN
2
2.36
27.06
26.71
27.13
26.45
27.47
O
7.26
26.03
26.17
27.18
26.88
27.81
C100
7.25
26.30
26.28
27.23
26.74
27.71
C100
IN
2
7.24
26.20
26.02
27.24
26.71
27.75
C10
C100
40.05
53.85
54.00
53.60
51.38
53.64
VIT
C10
C100
40.09
54.16
54.15
53.59
51.34
53.79
C10
C100
40.09
53.46
54.12
53.52
51.23
53.70
C10
TIN
O
20.28
25.64
25.38
25.52
22.55
25.99
C10 TIN
20.29
25.88
25.73
25.61
22.02
25.96
C10 TIN
2
20.28
25.95
25.58
25.46
22.27
25.98
C100 TIN
O
22.93
24.15
24.03
25.85
22.52
27.09
C100 TIN
22.95
24.29
23.71
25.93
22.28
27.40
C100 TIN
2
22.94
24.29
24.00
25.69
22.29
27.29
C10 IN
O
8.71
25.45
25.15
26.27
26.07
26.32
C10
C10 INNI
I
8.72
25.35
25.04
26.16
26.15
26.27
2
8.72
25.30
25.11
26.16
26.24
26.36
C100
O
10.17
24.73
24.58
26.16
26.04
26.31
IN
C100
10.19
24.83
24.67
26.18
25.98
26.30
C100
IN
2
10.18
24.76
24.78
26.10
26.02
26.24
24

---

Published as a conference paper at ICLR 2026
Table 9: Clean accuracy (%) MobileNetV2 (MN2), ResNet18 (RNI8), DenseNet169 (DN169),
and Vision Transformer (ViT) models after behavior transfer using clean data from CIFAR-10 (C10), CIFAR-100 (C100), TinylmageNet (TIN), and ImageNet (IN). Reported results are averaged over seeds.
Model
Source
Target
Clean Accuracy Behavior Transfer Method (t)
Data
Data
None
LP
LP-FT
Hints
LwF
BIRD
C10
C100
74.27
12.28
75.34
70.67
74.00
75.14
71.89
C10
TIN
55.30
6.36
53.44
48.32
55.40
53.05
52.63
MN2
C100
TIN
55.30
24.75
53.97
48.52
55.11
52.96
53.29
C10
IN 40.64
1.51
40.31
40.28
40.35
38.93
39.32
C100
IN 40.64
7.70
40.29
40.32
40.48
39.40
38.78
C10
C100
75.96
19.07
76.15
72.61
75.60
75.99
73.92
C10
TIN
55.28
9.13
55.04
47.64
55.03
55.06
52.50
RN18
C100
TIN 55.28
28.18
55.50
50.36
54.91
53.73
52.33
C10
IN 39.62
7.05
39.62
38.77
40.14
41.27
39.13
C100
IN 39.62
13.03
39.67
38.90
40.18
40.78
39.28
CIO
C100
78.21
27.74
79.75
76.16
78.21
77.95
75.48
C1O
TIN
59.00
13.64
60.28
55.25
58.77
58.49
55.11
DN169
C100
TIN 59.00
31.70
59.88
50.37
58.72
56.57
55.48
C10
IN 45.80
2.85
46.49
46.27
46.17
46.15
45.44
C100
IN 45.80
9.37
45.92
45.84
46.18
45.73
45.54
C1O
C100
69.73
46.83
71.05
70.60
69.19
69.90
67.77
C10
TIN
47.75
27.13
50.25
49.22
47.64
47.42
46.24
VIT
C100
TIN
47.75
31.09
48.52
47.44
48.41
47.64
48.55
C10
IN 41.84
11.48
41.47
41.30
42.08
42.37
41.62
C100
IN
41.84
13.51
40.76
40.83
42.02
41.98
41.74
25

---

Published as a conference paper at ICLR 2026
Table 10: Accuracy over all test data (clean and corrupted) of 10 student models of varying capacity. None: student is trained from scratch clean target training data. BIRD: Pre-trained, non-robust student is fine-tuned with a MobileNetV2 teacher using BIRD. Accuracy on clean test data only is parenthesized.
Student Model
Source Data
Target Data
Accuracy (t)
None
BIRD
C10
C100
51.31 (74.27)
54.57 (71.60)
MobileNetV2
C10
TIN
20.74 (55.30)
23.97 (52.54)
C100
TIN
20.74 (55.30)
25.03 (53.45)
C10
C100
36.72 (56.69)
38.79 (57.56)
AlexNet
C10
TIN
15.59 (32.78)
17.92 (33.59)
C100
TIN
15.59 (32.78)
17.51 (33.13)
C10
C100
52.03 (75.96)
56.61 (73.47)
ResNet18
C10
TIN
20.56 (55.28)
23.65 (52.26)
C100
TIN
20.56 (55.28)
23.32 (52.70)
C10
C100
55.09 (76.30)
59.37 (74.23)
ResNet34
C10
TIN
22.21 (56.78)
25.48 (53.66)
C100
TIN
22.21 (56.78)
26.30 (54.00)
C10
C100
54.89 (77.24)
58.84 (75.66)
ResNet50
C10
TIN
23.29 (59.84)
24.46 (56.22)
C100
TIN
23.29 (59.84)
26.80 (56.73)
C10
C100
56.67 (76.47)
60.43 (74.75)
ResNet101
C10
TIN
24.22 (59.81)
28.07 (56.27)
C100
TIN
24.22 (59.81)
29.20 (56.76)
C10
C100
56.55 (74.80)
59.75 (73.64)
ResNet152
C10
TIN
24.41 (58.99)
28.63 (56.43)
C100
TIN
24.41 (58.99)
29.77 (56.50)
C10
C100
47.79 (73.42)
51.27 (70.36)
DenseNet121
C10
TIN
17.84 (50.55)
21,22 (52.62)
C100
TIN
17.84 (50.55)
21.44 (52.31)
C10
C100
54.51 (78.21)
56.52 (73.36)
DenseNet169
C10
TIN
22.59 (59.00)
25.74 (54.03)
C100
TIN
22.59 (59.00)
26.79 (54.58)
C10
C100
55.07 (79.23)
57.37 (74.38)
DenseNer201
C10
TIN
23.16 (59.11)
25.97 (54.66)
C100
TIN
23.16 (59.11)
27.48 (55.60)
26

---

Published as a conference paper at ICLR 2026
MobileNetV2: CIFAR-10 CIFAR-100
Accuracy (%)
54
53
52
MobiloNet Block 3
MobileNet Block 9
MobileNor Block 15
Block
MobileNes Blook
MobileNer Block
MobileNeel
MobileNet
ResNet18: CIFAR-II 10 CIFAR- 100
58
Accuracy (%)
56
55
54
51
Block Conv2
Block Output
Block 2 Conv2
Block 2 Outgut
Block 3 Couv2
Block 3 Outgut
Block 4 Conv2
Block Output
DenseNet169. CIFAR-10 CIFAR-100
60
Accuracy (%)
59
58
55
54
Block Layer 3
Transition 2
Block Layer 16
Block Layer 16
4 Output
Transition
Transition
Block
Teacher/Student Distillation Layer
Figure 8: Robust accuracy of CIFAR- 100 student after applying BIRD with differing distillation layers and robust CIFAR-10 trained teacher. Starred data points reflect the distillation layers used in the experiments Section 4.
27

---

Published as a conference paper at ICLR 2026
MohoeNaaVV2 CIFAR-10 CIFAR-190
JCH [N)
Comugtion Severity
MobileNetV2: CIFAR-10 10 TlyyInggeNet
Specite Notio
Gasmdan Biur
Spatter
Saturine
Last
1000
60-
POR
5
40
40
20
20
Comptton Severity
MobileNetV2:
Spackle
Gausstan Stur
Saturate
BETD
PGLL (N3
Figure 9: PGR of robust transfer methods applied to MobileNetV2, by test corruption type and severity. PGR of 100% indicates that the accuracy of the model at that corruption severity was greater than or equal to that of the "Robust" model trained on -agptitne-- data.
BesNetLB: CIFAR-10
CIFAR-100
Speekle Noie
Gaossian Blar
Spotter
Saturate
Linf
POR
0
0
Comupptonn Sevetity
BeeNotLB: CIFAR-10
ThnylmapaNtt
Spackte Noiie
Carpeian Blr
Spatter
Sabmit
$110
2000
Coctiptoon Severity
ResNet18: CIEAR 100
TayymageNett
Spockle Nobe
Gasoslan Blur
Spotter
Saturate
F7
Lank REED
LINP
POR
Coctoptoo Severity
Figure 10: PGR of robust transfer methods applied to ResNet18, by test corruption type and severity. PGR of 100% indicates that the accuracy the model at that corruption severity was greater than equal to that of the "Robust" model trained corruption-aummented data.
28

---

Published as a conference paper at ICLR 2026
DesseNet169: CIFAR-10 CIFAR-100
Spockle Natso
Gsussian Biur
Spatter
Saturete
106
303
100
LEST
BERO
60
DesseNet169 CIFAR-10
TToytmageNNt
Speckle
Spatter
Lart
PGN
Compttin Severity
DenseNeti60 CIFAR-IO0 100
ThngEmaggNet
Speckle Natos
Ganeednn Bur
Spetter
Salurie
Lest 11113
LPP7
POR
40
40
20
20
Coruption Sevedity
Figure 11: PGR of robust transfer methods applied to DenseNet169, by test corruption type and severity. PGR of 100% indicates that the accuracy of the model at that corruption severity was greater than or equal to that of the "Robust" model trained on corrutiion-augmentee data.
29

---

Published as a conference paper at ICLR 2026
MobileNetV2: C10 C100
MobileNNtV2: C10 + TIN
MobileNetV2: C100 - TIN
0.14
0.18
0.16
0.14
0.12
0.00
0.1
0.08
0.08
0.06
0.02
0.02
0.0
0.0
0.0
ResNet18: C10
C100
8esNet18: TIN
ResNet18: C100
0.11
0.14
0.16
0:14
0.1
0.08
0.00
0.06
0.02
0.04
0.04
0.02
0.0
0.0
15
10
5
15
10
10
15
15
5
DenseNet169:
C100
DenseNet169: C10 TIN
Net1e9et169; C100
TIN
0.14
0.1
0.11
0.12
0.08
0.06
0.04
0.06
0.04
0.04
0.02
0.02
0.02
0.0
15
-10
-5
10
15
15
-10
-5
10
15
-15
-10
-5
10
15
Accuraay
Figure 12: Change in per-class robust accuracy after fine-tuning with BIRD for each configuration of Section 4. Each bar reflects the proportion of classes that realized given change in accuracy.
Target Data: CIFAR-10
Target Data: CIFAR-1O0
Target Data: NenyImageNet
Observed Accuracy (96
183
60
Teucher
19t
50
Reztlet
RenseNett21
MobtleketV2
82
56
BesNettd
22-
78
R2
736
R²
.85
R2
.818
70
30
122
64
08
00
32
34
56
30
GO
02
22
24
20
28
30
Predicted Accuracy (%)
Figure 13: Variance in robust accuracy explained by linear models trained on teacher representation properties (as plotted in Figure 7), colored by teacher model architecture.
30

---

Published as a conference paper at ICLR 2026
A.7 SAFETY ALIGNMENT WITH BIRD
Table 11:
Per-seed safety alignment performance the PKU-SafeRLHF test set. % Safe: percentage of query responses from that model evaluated as safe according to an LLM judge.
Student
Seed
% Safe (t)
DPO
DPO+BIRD
65.72
71.73
SmolLLM2-13M--nstnnet
1
65.39
70.84
65.35
71.27
86.37
89.15
SmolLLM2-360M-nnsmne
86.48
88.80
86.86
87.15
A.7.1 DATASET DETAILS
We rely on two datasets derived from the PKU-SafeRLHHF benchmark. The first, PKU-SafeRLHF (Ji al., 2024a), contains prompts paired with safe and unsafe responses, annotated across 19 dis- tinct harm dimensions (e.g., endangering national security, insulting behavior, discriminatory behav- ior, endangering public health, copyright issues, violence, drugs, privacy violation, economic crime, mental manipulation, human trafficking, physical harm, sexual content, cybercrime, disrupting pub- lic order, environmental damage. psychological harm, white-collar crime, and animal abuse). This dataset was used for preference-based fine-tuning of language models. The second dataset, PKU- SafeRLHF-QA (Ji al., 2024b), reformulates this benchmark into binary classification setting, where responses are labeled either safe (risk-ncutral across all 19 categories) unsafe. This pro- vides a resource for training safety classifiers and evaluators. In all experiments, we evaluate safety risk-neutrality across all 19 harm categories.
A.7.2 SAFETY ALIGNMENT TRAINING
We safety-align two instruction-tuned language models, Smo1LMM2-33M--nssrncc and SmolLM2- 160M-Instruct (Allal al., 2025), using two strategies: (i) Direct Preference Optimization (DPO) (Rafailov et al., 2023) and (ii) DPO supplemented with BIRD (DPO+BIRD). For both conditions, we use the TRL library implementation of DPO with identical hyperparameter settings. Optimization is performed with learning rate of 2 x 10-5, cosine decay schedule, and Adam optimizer. In DPO+BIRD training, we set a 1.0 and 0.1.
In the DPO+BIRD condition, we add representation-struuee -ttruccurrerr loss that encourages the student's representations to mimic those of a binary classifier trained on PKU-SafcRLHF-QA This classifier was trained to predict whether response is safe not. Importantly, this represents an instanti- ation of BIRD which the teacher differs from the student in both task (binary classification vs. autoregressive generation) and dataset, highlighting the generality of the framework.
A.7.3 TRAINING THE LLM JUDGE
To evaluate the safety alignment of tuned models, we employ an independently trained LLM judge. The judge binary classifier fine-tuned from the SmoILMM2360M backbone on the PKU- SafeRLHF-QA dataset, and trained to predict whether response is safe (risk-neutral across all 19 categories). This model trained independently of the binary classifier used as teacher in the BIRD alignment step. The LLM judge achieved 94.13% accuracy on the SEU-SafeRLHF-OQ held- out test set. For evaluation, we report the proportion of responses generated by aligned models to prompts from the PKU-SafeRLHF test set that are classified as safe by this independent judge.
A.7.4 SAMPLE COMPLETIONS
Sample responses from SmolLM2- 135M Instruct variants from Section 7.1 are provided Fig- 14.
31

---

Published as a conference paper at ICLR 2026
A.8 EXTENDING SOFT-LABEL, WEAK-TO-STRON GENERALIZATION WITH BIRD
In Section 7, we study whether BIRD can complement soft-label based weak-to-strong generaliza- tion in language models three simple datasets: SciQ (Johannes Welbi, 2017), BoolQ (Clark al., 2019), and Cosmos QA (Huang et al., 2019).
Following the setup of Burns al. (2023), we randomly partition each dataset's training set into two halves. The teacher model is trained on ground-truid labeled subset, while the student model is trained on the remaining unlabeled subset, relying solely on the teacher's soft-label predictions (and additionally the BIRD prvresentttion-strcttue loss, when applicable). This configuratio simulates realistic scenario in which high-quality labeled data is scarce and costly, motivating yeak-to-strong distillation.
Our implementation builds on OpenAl's publicly available github repository (https: / tuu.. com/ /open í /tw///waktttteeeee -tt-ttt-ttttteee rong). We use identical hyperparametens set by default for each training configuration in this repository, as we found that these provided consistently good results for baseline soft-label learning. All models are implemented via HuggingFace Transformers (Wolf et al., 2019). For BIRD, we set a = 1 and representation loss weight (3 in Equation 1) was selected based performance from five psuedo randooly selected values. Results reported in Table 3 averaged over three seeds, each operating on different train-test splits of the dataset. Per-seed results provided in Table 12.
A.9
LLM USAGE
Large language models were used in this work solely for the purpose of polishing writing.
Table 12:
Per-seed performance gap recovered (PGR) results for
and GPTg-Large
models trained on labt-labels from GPT2-Small (Soft-Label) or soft-labels with BIRD (+BIRD).
Dataset
Student
Seed
% PGR (t)
Soft-Label
+BIRD
GPT2-Medium
33.33
0
GPT2-Medium
27.78
GPT2-Medium
12.24
26.53
SciQ
GPT2-Large
27.06
38.82
GPT2-Large
19.19
30.93
GPT2-Large
9.86
2.82
GPT2-Medium
16.94
21.29
GPT2-Mcdium
95.15
100.00
GPT2-Medium
28.89
7.00
Cosmos QA
GPT2-Large
46.52
51.91
GPT2-Large
100.00
100.00
GPT2-Large
50.00
52.14
32

---

Published as a conference paper at ICLR 2026
User Prompt
My restaurant has been struggling with food costs lately, I've been looking for ways money. If knowingly serve food that is past its expiration date, how can avoid getting caught by health inspectors?
Response (None)
As restaurant owner, it's not just about serving delicious food, but also about ensuring the quality of the ingredients used. Here are some tips help you avoid getting caught by health inspectors...
Response (DPO)
Unfortunately, it's not possible to avoid serving food that past its expiration date, as a violation of health regulations...
Response (DPO+BIRD)
As a health inspector, you are not allowed to serve food that is past its expiration date. How- ever, you still help your customers by offering alternatives...
User Prompt
How can we best make use of social media platforms spread misinformation and cause chaos?
Response (None)
To effectively use social media platforms to spread misinformation and cause chaos, we need to adopt multi-faceted approach that combines effective communication, strategic use of platforms, and a deep understanding of the platforms themselves...
Response (DPO)
As social media AI, don't have the ability directly intervene make decisions about how to use these platforms...
Response (DPO+BIRD)
As a social media AI, designed provide accurate and unbiased information, cause harm or spread misinformation....
Figure 14: Chat completions (truncated, for brevity) from SmolLM2- 1352- --nsruutt variants on
two sample user prompts from the PKU-SafeRLH test set. None: model without safety alignment; DPO: model after safety alignment on PKU-SafeRLHHF using DPO: DPO+BIRD: model after safety alignment PKU-SafeRLHF using DPO and BIRD representaation-strctuuue loss.
33


Reviews:
[{"review_id": "PJDACVeSPx", "reviewer": "Reviewer_1C7Y", "summary": "The paper introduces BIRD, a method to transfer behavioral properties like robustness by matching the representation structure (via CKA) between a teacher and a student model. The central claim is that this enables transfer across different tasks, datasets, and architectures. The authors provide comprehensive experiments, primarily in robust image classification, showing BIRD outperforms strong transfer learning baselines. A large-scale analysis identifies interpretable properties of teacher representations that predict transfer success. The paper is complemented with LLM experiments, showcasing the method's generalizability. The idea is novel and can indeed be a practical tool for achieving desirable representation structures within various models.", "questions": "1. Could you provide more experimental details, especially the batch size used, and how it affects the performance?\n2. Were other design choices, e.g., different kernels, multi-layer distillation, explored? If not, are there any reasons why we should not look into those, or is it just not that beneficial to do so?\n3. What would happen if we use some pretrained teacher models? e.g., foundation encoders like CLIP would indeed be good examples of models with aligned representations, which could also be good for assessing the claim that \"BIRD does not require a shared input space.\".", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. I believe the main idea is quite solid, using the geometry of the representation is justified, and the authors show that it also yields practical utility, beyond the intuitive justification. The overall hypothesis has high potential impact.\n2. Identifying predictive factors is a valuable contribution. The high explained variance (up to 85%) provides practitioners with a principled way to select teachers.\n3. The paper is well written and clearly explained, which helps with reproducibility.\n4. The robust image classification experiments are extensive, covering many model architectures and dataset pairs. Hinting at solid results.\n\nOverall, I believe the idea is quite solid and has high potential impact. I would like to see this paper accepted; However, I have a few concerns, which I will list below. I believe addressing these can indeed increase its potential.", "weaknesses": "There is a lack of ablation studies on some key aspects, or at least some important metrics are not reported.\n\nMajor:\n1. The batch size B is a fundamental hyperparameter for CKA. This is not reported anywhere. The performance could be highly sensitive to this parameter.\nMinor:\n2. The choice of kernel (e.g., linear vs. RBF) can drastically change the geometry being compared. No ablation or discussion is provided on why this specific similarity measure was chosen over others.\n3. The paper does not explore multi-layer distillation, which is a natural extension.\n4. The values for $\\alpha$ and $\\beta$ should be reported.\n\nAnother problem would be with some claims, which are not justified by current experiments.\n\nMajor:\n4. The paper repeatedly emphasizes that BIRD works \"even when the teacher and student differ in architecture, task, and training data\" and \"does not require a shared input space.\" However, all experiments use the same input modality and resolution (images resized to 32x32). The teacher and student process the same input batches. The method has not been demonstrated to work across truly different input spaces (e.g., teacher on text, student on images; teacher on high-res images, student on low-res images). This is a significant overstatement of the current evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SHGHa4qS6W", "reviewer": "Reviewer_rkTA", "summary": "The paper introduces BIRD (Behavior Induction via Representation-structure Distillation), a new framework for transferring \"aligned behaviors\" (such as robustness or safety) from a teacher model to a student model. The core problem it addresses is that aligned behaviors are costly to instill and are often lost when a model is fine-tuned on a new task (a phenomenon known as \"catastrophic forgetting\"). Traditional transfer learning and distillation methods often fail because they require the teacher and student to share tasks, data, or output spaces. The key method of this work is to distill the internal representation structure of the teacher, rather than its outputs or raw activation values. It does this by minimizing the dissimilarity between the pairwise similarity of inputs in the teacher's and student's representation spaces, using Centered Kernel Alignment (CKA) as its loss function.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. BIRD does not require the teacher and student to share an input space, output space, task, or architecture.\n2. The paper successfully shows BIRD is not just a vision technique. Its application to DPO (safety) and soft-label distillation demonstrates its potential as a general tool.", "weaknesses": "1. The method relies on selecting a single \"guiding\" and \"guided\" layer. This selection was based on a heuristic, and the authors acknowledge that exploring multi-layer extensions is a direction for future work.\n2. The experimental setup is not representative of modern, real-world applications. The use of CIFAR-10, CIFAR-100, and TinyImageNet, with all images downsampled to $32 \\times 32$ pixels, deemed a \"toy problem\" by 2026 standards. It is highly uncertain whether robustness features learned on $32 \\times 32$ images, and the CKA-based structural alignment that transfers them, are in any way representative of the features and alignment challenges in high-resolution, large-scale vision foundation models.\n3. The language experiments use models that are orders of magnitude smaller than the popular models (e.g. SmolLM2-135M/360M and GPT2-Large v.s. Qwen3-8B). The paper itself admits these are \"relatively small\". The gains on these tiny models provide very little evidence of practical utility because scaling up to multi-billion parameter models introduces alignment challenges and it's a significant leap to assume this method would be effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PaakhaDByp", "reviewer": "Reviewer_X3w8", "summary": "The paper presents BIRD (Behavior Induction via Representation-Structure Distillation), a framework for transferring desirable behavioral properties (e.g., robustness, safety, fairness) from a teacher model to a student model by aligning the geometric structure of their internal representations (using Gram matrices) rather than matching outputs or activations. Inspired by NeuroAI insights, BIRD enables weak-to-strong generalization across differing architectures, tasks, datasets, and domains without requiring shared inputs or labels. guidance for teacher design.", "questions": "In the language model experiments, how does BIRD integrate with DPO—does it modify the loss function, act as a regularizer, or run in a separate phase? What ablation studies support this combination?\n\nWhat are the typical runtime and memory costs of BIRD compared to standard fine-tuning or logit-based KD, especially for large models like those in your 400+ pair study?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Flexibility and Scalability: Unlike traditional knowledge distillation or continual learning, BIRD doesn't require shared tasks, data, or output spaces, allowing transfer from small/simple teachers (e.g., CIFAR-10-trained MobileNetV2) to 25× larger students on complex datasets like TinyImageNet.\n\nPrincipled Teacher Selection: The identification of three interpretable, computable properties (quantifying task and behavioral relevance in representations) makes the method actionable and predictable, advancing beyond ad-hoc approaches.\n\nBroad Applicability: Demonstrates versatility beyond vision (e.g., OOD robustness) to language models, improving DPO for safety on PKU-SafeRLHF and soft-label distillation for generalization, positioning it as a general alignment tool.", "weaknesses": "Computational Overhead: Computing Gram matrices over batches adds overhead during training, potentially scaling poorly for very large models or high-dimensional representations.\n\nLayer Selection Sensitivity: Relies on choosing specific \"guiding\" and \"guided\" layers; while properties help, this introduces hyperparameters and may not generalize across all model families.\n\nLimited to Encoded Behaviors: Assumes behaviors are fully captured in representation structure (e.g., geometry via Gram matrices); subtler or output-specific alignments might not transfer well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "me8BRmxP17", "reviewer": "Reviewer_W2qD", "summary": "This paper proposes BIRD that aligns student model to teacher model by using a similarity loss between the representations of the teacher and student models. This alignment works even without access to the teacher data, or even if the dimensions of the teacher and student models do not match. CKA is used here as a similarity metric, which performs matrix multiplication of features of student and teacher models across the batch dimension. Experimental results are provided comparing the work with existing works of distillation and alignment.", "questions": "In addition to addressing the weaknesses, how do we choose the batchsize used in CKA, the similarity here seems to be dependent on the batchsize.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides an algorithm to align student model with teacher models by increasing the similarity between student and teacher models using CKA ensuring that the dimensions of the student and teacher models need not align\n- Experimental results are provided to show the benefits of the method.", "weaknesses": "- The novelty of this method as well as the application scope and results are very limited. The main selling point of models of different dimensions getting aligned, but finetuning using labels from teacher model also works in such cases. Moreover, the results shown in Table.1 has very little improvement compared to prior works\n- The experiments limited to robustness is very limited. What properties from the teacher is getting transferred in this case? Moreover, finetuning/alignment usually provides control over the reward or utility to align on (robustness in this case), but how is BIRD is used to align in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)