Learning Task-Agnostic Representations through Multi-Teacher Distillation
Authors: Philippe Formont
Maxime DARRIN
Banafsheh Karimian
Eric Granger
Jackie CK Cheung
Ismail Ben Ayed
Mohammadhadi Shateri
Pablo Piantanida
Venue: NeurIPS 2025 poster ()
Keywords: molecular representation
nlp
knowledge distillation
embedding models
representation learning
Primary area: deep_learning
TL;DR: We show that interval estimation based methods produce better distilled embedders in multi-teacher distillation settings compared to MSE or Cosine base methods.

Abstract: Casting complex inputs into tractable representations is a critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. We introduce a task-agnostic framework based on a ``majority vote" objective function. 
We demonstrate that this function is bounded by the mutual information between the student and the teachers' embeddings, leading to a task-agnostic distillation loss that eliminates dependence on task-specific labels or prior knowledge. 
Comprehensive evaluations across text, vision models, and molecular modeling show that our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such as classification, clustering, or regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities.

Learning Task-Agnostic Representations through Multi-Teacherr Distillation
Philippe Formont"
Maxime Darrin"
Universite Paris-Saclay ETS Montreal
McGill University- Universite Paris-Saclay
Mila Quebec AI Institute
Mila Quebee AI Institute
LIVIA- ILLS
ILLS
Banafsheh Karimian
Jackie CK Cheung McGill University
Eric Granger ETS Montreal ILLS LIVIA
Ismail Ben Ayed ETS Montreal
ETS Montreal
ILLS LIVIA
Mila Quebec AI Institute
ILLS LIVIA
Mohammadhadi Shateri ETS Montreal
Pablo Piantanida
CNRS CentraleSupelec Universite Paris-Saclay ILLS Mila Quebec Al Institute
LIVIA
Abstract
Casting complex inputs into tractable representations is critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. In this paper, we introduce a task- agnostic framework based "majority vote" objective function. We demonstrate that this function is bounded by the mutual information between student and teachers' embeddings, leading to a task-agnoste distillation loss that eliminates dependence on task-specific labels or prior knowledge. Our evaluations across text, vision models, and molecular modeling show that method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such classification, clustering, regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities.
Introduction
Transforming complex inputs into tractable representations is crucial for numerous applications
across different domains, from natural language processing (Li & Li 2023 Pimentel al. 2023).
computer vision (Kubota et al. 2024 Bhalla et al., 2024) to bioinformatics (Morgan 1965 Wang
eet al. 2022a). This done using embedders, often large pretrained models (Touvron al. 2023
Jiang et al. 2023), that project objects (image, text, molecules, into numerical representations,
enabling various downstream tasks (Murphy 2013 Vilnis & McCallum 2015).
Variations in model architecture, training paradigms (e.g.. unsupervised vs. supervised), and objective functions (e.g., masked language modeling and contrastive learning) result in embedders that capture different aspects of the same input. To leverage this diversity, a common practice is to combine them into a single model through multi-teacher Knowledge Distillation (KD) (Zhang et al. 2023).
Equal Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Not only are these methods cost-effective at inference time (Hinton et al. 2015 Frosst & Hinton 2017). they are also extremely useful to compress knowledge from larger models into smaller ones for cessource-consrraine mend the weights of models whose architectures have been altered (Muralidharan et al,; 2024). Most existing approaches, however, focus on single-task distillation. In this setting, the student model either learns to mimic teacher representations for a specific task (Dvornik et al. 2019), or the distillation process is explicitly paired with task-specific information. While effective, such methods cannot be used for or generalized to unseen tasks, requiring a new distillation process to be performed for every new task. Our goal is to learn a highly informative representation that retains maximal utility across a wide range of downstream tasks. In other words, we aim to maximize information density within a single representation, enabling
environments (Pan et al. 2022 Wang et al. 2023; Zhang et al. (023). or
purpose adaptability without sacrificing performance.
Task agnostic multi-teacher distillation aims to compress teacher representations into single student embedder, such that the student representation captures as much information as all the teachers combined. To our knowledge, few works address task-agnostic distillation from multiple teachers. Existing approaches often rely on mean squared error (MSE) loss and cross-encoder heads (Navaneet) ct al. 2022). which can be unstable in high-dimensional spaces (Farebrother et al. 2024).
To overcome these limitations, we introduce a novel task-enabling setting to task-agnostic multi- teacher distillation. Our goal is develop representations that capture the maximum amount of information about the data distribution, ensuring their upplicability to a wide range of tasks, even in the absence of prior knowledge about those tasks. We train the student model learn representations that, when applied to downstream tasks, generate predictions consistent with the majority of predictions from the teachers' representations. This approach allows our method to leverage the collective knowledge of the teachers' ensemble. To achieve this, we introduce an ensembling loss that measures the agreement between the Bayesian predictor based on the student's embeddings and the Bayesian predictors based on the teachers' embeddings. We show that this loss can be bounded independently of the task, using the conditional differential entropy of the teachers' embeddings given the student's output, thus providing task-agnostic student-teacher reconstruction loss.
Contributions. In this study, we investigate the following research question: How can the knowledge from multiple large embedding models be effectively distilled and integrated into a smaller one to produce a more general-purpose purpose representation? Our main contributions are threefold:
1. A task-enabling setting. We frame the multi-teacher distillation problem in a task-enabling setting, in which we study the relationship between the Bayes classifiers obtained from the students and the teachers' embeddings. We prove a simple, yet powerful result: the conditional entropy of the teachers given the student's output controls the probability of the student's ' Bayesian predictor disagreeing with the teachers' for any task.
2. A tractable implementation. We leverage recent differentiable high-dimensional Gaussian-Mixen based estimator of the differential conditional entropy to -ooomattion-thorettii loss. This loss maximizes the mutual information between the student and all teachers, resulting in a principled, task-agnostic distillation objective.
formulate an
3. High-quality generalized embedders. Our method enhances distillation capabilities across three application domains: molecular modeling, natural language processing and computer vision. We release trained students achieving competitive performance on a wide range of downstream tasks, e.g., classification, regression, clustering, and sentence similarity.
2 Related Work
Task-oriented distillation. KD is widely used for transferring knowledge from one set of teachers to a student model (Gou et al., (02T) to improve the performance of the student on a given task (Zhang et al. 2019 Yim et al. 2017). This is typically done by transferring logits (Sun et al. 2024): i.e. the models' output, features (Wang et al., 2023) Sarkar & Etemad 2024), relational information (Dong al. 2024 2021),
mixture of them (Liu al. [021aa) Similarly, (Qiu et al.
2024) uses a regularization term to distill the task-relevant information from the large teacher to the small student. We depart from these methods by focusing on distilling task-agnostic representations.
Task-oriented multi-teacher distillation. A common method for multi-teacher KD is averaging the teachers' logits and transferring the result to the student (Dvorik et al. 2019 Hinton et al. 2015).
2

---

Unlubeled Dutoset
S
Inputs
Cr T.....T)
fo
T
=
log Nit (s:), (s))
fox
- -log (x((s)), Ex(s.))
Figure 1: Unsupervised training of our student through task-agnostic distillation. The student embedder S is trained to minimize the negative log-likelihood of multiple teachers' outputs condi- tioned on the student's predictions. During this multi-teacher distillation procedure, both the student's weights and those of the teacher-specific Gaussian kemelss {fe, h&Ek are updated in an end-to-ent to-end fashion. Post-trade tost-training, we discard the Gaussian kemelss and evaluate the student embedders by freezing their weights and training a feed-forward network on their embeddings for an unseen dataset.
However, this approach is not ideal when the performance of the teachers is uncertain. Alternative methods include using gate networks (Zhu et al. 2020). reinforcement learning agents Yuaan et al. 2020), and other methods (Ma al., [2024a] Borza et al. 2022 Zhang al. (023) to perform teacher selection or evaluation. Due to challenges in distilling knowledge among diverse architectures, multi-teacher KD research mainly focuses on logit distillation. Other techniques were also explored, such as multi-teacher feature ensemble (Ye et al. 2024), contrasting feature distillation (Li et al. 2024). and cosine similarity- -based methods for various tasks (Ma al.) 2024b) Aslam et al. 2024
2023). Ensemble-based methods have also been proposed to mitigate over wer-smothhing and leverage
teacher diversity, such as by aggregating structured predictions before distillation (Shayegh et al.) 2024). Although successful, most multi-teacher feature distillation methods remain oriented to only
one or a few tasks.
Task-agnostic and self-supervised features distillation. To the best of our knowledge, few works address task-agnostic representation distillation. Several approaches assume strong limitations, such as requiring the student to have the same architecture as the teachers (Liang et al. 2023] 3, Xu et al. 2022b), or requiring fine-tuning the teachers to then distill their representations (Liu al. (023)
Other methods induce requirements on the students, limiting their extension to a general mu teacher setting. Notably (Gao al. 2022) relies on vision-specific data augmentation, RoB (Duval al.) 2023) focuses on the distillation of joint-embedding approaches, AttnDist (Wang et al.) 2022b) only applicable to single teacher, (Song al. 2023) need the teacher and student to have the same architecture, and SEED (Fang et al. 2021) requires the student and the teacher to have the same embedding dimension. Finally, CompRess (Abbasi Koohpayegani et al. 2020) introduced distillation method ensuring that the embeddings of the student and the teacher encode a similar nearest-neighbor graph, which would be unstable a multi-teacher setting. Other approaches such as
contrastive learning (Feng et al. 2024 Liu et al. 2022 Xu et al. (022a) focus on distilling relational relationships between the samples, such as nearest neighbors preservation (Noroozi al.) (0018) or angle preserving distillation (Park et al., 2019a). SimReg (Navaneet et al.) (202)). however, trains the student jointly with cross-encoding heads to directly reconstruct the teacher's features using an MSE loss.
Interval estimation. While SimReg performs its distillation through pointwise estimation with MSE, is well known in the reinforcement learning literature that these standard regression methods are difficult to train (Farebrother et al.) 2024). On the other hand, replacing traditional regression scheme by maximum-likellhood training of Gaussian kernels appears to more stable (Stewart)
et al. 2023 and effective in Value learning (Bellemare et al. 2017). We extend this idea in the
3

---

context of embedder distillation by using Gaussian kernels to estimate the conditional distribution of the teachers' embeddings given the student embedding and show that is directly connected to maximizing the mutual information between the student and the teacher.
3 Distilling Representation Through Gaussian Kernels
We denote the imput space by and the corresponding imput distribution by Px. We assume we have access to dataset D {x, } f==1 where samples are drawn i.id. according to Px. We consider set of K different teacher embedders, Tı : + Rd for & {I,..., K}. each mapping inputs to potentially different embedding spaces dimension dę-
3.1 From a task-oriented setting to a task-agnostic loss
Our goal is to train representation model capable of effectively handling any downstream task, by leveraging diverse representations from diverse pretrained teachers (Figure 1). To do so, we first measure the agreement between the student's 's Bayes classifier and the teachers' for any given task. First, we demonstrate that it can be bounded by the conditional entropy of the teacher's embedding given the student's, s, which does not depend on the considered task.
Let us consider a task characterized by a target y of discrete concepts and the feature space X' with joint probability measure Pyx E P() x X). For every projection of the features through the different teachers, the Bayes decision rule is given by ct2 - arg max Ex,y (c(T&(X)) Y]] and
c-
for the student: dg - arg max Ex,y [1 [c(S(X)) = YJI.
cR-++
Our goal to minimize the probability that the student's Bayesian classifier deviates from the predictions of the teachers'. This approach has been shown to enhance performance in most cases by reducing both bias and variance, while improving robustness and generalizability (Dietterich 2000 Sccimeca al. [2023] Allen-Zhu & Li 2020) Theisen et al. 2024). In other words, aim to minimize the probability that the student's 's decision differs from that each teacher:
K
1 K
Y,S,, Tk) =
Pr (r(((((X)) + of (Tx())))
Probability that Bayesian classifier's
output is different from the At teacher's
where the loss depends on the joint distribution (X, Y), through the definition of the Bayesian classifiers.
We leverage recent results the performance of the Bayes classifiers to bound the probability of getting two different outcomes using the Bayes classifiers operating on two different projections of the input space.
Proposition 3.1 (Darrin et al. (2024)) )- Lei CT = = (X)) and Cs c5((())) denote the out- come of the Bayes classifee observing the output of the teacher Tg and the student on a given task Y, respectively.
(Cs & CT,) < 1- exp
( (Tx (Tx(X)S((X))
Corollary 3.2 (Training objective). By applying Prop. 3. to Eq. 7 for any given joint distribution Pxy, we have
K K Σ ((T(()))((X)) Am
Y, Tx) < exp (-
(2)
Negative likelihood
This corollary directly follows from the concavity oft + 1 exp(-t) (see Appendix A).
Remark 3.3. This bound over our ideal loss L is independent on the conditional entropy of the teacher embeddings given the student embeddings. Therefore, optimizing the student minimize this loss provides a task-agiostic approach to aligning Bayesian classifier predictions with the ensemble of teachers' predictions, regardless of the downstream task.
specific task and depends solely

---

3.2 Student training
Estimation of the conditional entropy. To evaluate the conditional entropy of the teachers' embeddings given the student's 's embedding, we need a kernel to learn their conditional distribution (X) as presented in Figure I, To this end, we use a parametric Gaussian model whose parameters Ma (S(X)) and Ex (S))) are learned during the student's training (Pichler et al. 2022)
Loss function. Following the above reasoning, we propose to train the student embedder by minimizing the negative log-likelihood of the teachers' embeddings given the student's 's embedding, where the likelihood estimated using Gaussian Kernels as follows:
K h ((Tx(X))((())
L(X, S, T
Tx)
K
1=1
Ex log.N (x0)) (5())) Ex
(3)
where (-|u,, Σ) is the Gaussian distribution with mean μ and covariance Σ. our setting, min- imizing the conditional entropy (XT)x(X)((())) exactly corresponds to maximizing the mutual information (X): constant w.r.t of the student. This also applies to the bound in Eq.2
& (X)) (X)((()))(X))) since for each teacher h(T&(()) is
Training procedure. We train both the student and the different kernels in an end-to-end fashion by minimizing the loss function É. boils down minimizing the negative log- aggllikelhhodd of the teachers' embeddings given the student's 's embedding. We use the Adam optimizer to minimize the loss function. See See/Appendix for the detailed training algorithm. To reduce the computational cost, we first embedded the entirety of the training set using the teachers and store them. We can then build training batches by sampling from the pre-computed embeddings.
Baselines and Evaluation. We consider two widely used multi melti-teachhr feature distillation methods, MSE, used SimReg (Navaneet et al. 2022) and Cosine similarity (see Appendix G for more infor- mation). To evaluate the representations learned by the student, for each modality, benchmarks evaluating its performance on a wide variety of downstream tasks. For classification and regression tasks, train small feedforward network top of the embeddings (the backbones considered frozen) on different tasks and evaluate its performance.
different
4
Text Embedders
4.1
Experimental setting
We focus on distilling high-performing and large models into significantly smaller ones. Indeed, modern models in NLP are extremely large and costly train]] Thus, we aim to produce the best possible models for given weight category, pushing the size/performance of the Pareto frontier (Figure 2a), and not necessarily competing with the largest models. We distill from four teachers ranging from 433M parameters to 7B into students ranging from 20M to 335M parameters based on the nowflakes (Merrick et al. 2024) embedders.
Teachers and student. We select four freely available embedding models from the Huggingface hub (Woll et al. 2020) (See [Sec. C.1.2 for detailed list of the teachers) whose evaluations are available in the MTEB benchmark (Muennighoff et al.) 2023). To ensure having point of comparison, we select teachers of different sizes and performances. Notably, SFR Embeeddiddn
more than ten points stronger than the other three (smaller) teachers. As students we use snowflakes (Merrick) 2024; Merrick et 2024) models xs (22M), s (33M), (109M) and (335M) and we further train them using our distillation method (See Secc. C.1.4).
Embedder evaluation. Evaluating NLP models is notably challenging, and the common practice of evaluating a model using multi-task benchmarks may not be indicative of model capabilities (Liu et al.
5

---

Table Performance of our distilled models compared to the stronguest models of similar sizes from the MTEB Benchmark on classification tasks. Our 109M parameters model outperform significantly models times bigger exhibiting exceptional information density.
Amazon
Coumerfactuo
Amazon
Polarit
Amaazo
Reviews
Kanking??
Emotion
MTTOPIAN
MTOPPment
Mussive
Entent
Missivve
Toxis
Conversation
Tweet
Scatimeat
Extraction
Lmdb
Task
Avg.
Model
GIST
23M
22.9
87.2
42.6
84.2
52.1
78.5
9488
27.2
73.2
76.7
72.9
59.9
72.7
Bas.
Ivysaur
23M
72.1
86.7
42.7
81,9
45.4
80.8
92.1
71.9
70.3
74.9
65.5
58.7
70.2
gte-tiny
23M
71.8
86.6
42.6
81.7
44.7
80.5
91.8
69.9
70.1
74.9
71.0
58.6
70.3
MSB
Student-as
23M
71.6
42.3
83.6
57.5
83.5
94.5
74.3
59.3
72.9
NLJ.
Student-xs
23M
76.5
84.9
42.4
85.8
58.0
81.1
95.2
79.9
75.8
80.4
68.1
60.1
74.0
bge-small-co--1.5
33M
73.8
92.8
47.0
85,7
47.8
90.6
93.4
74,8
74.8
78.7
69.9
60.5
74.1
GIST
75.3
93.2
49.7
56.7
89.5
95.5
79.1
75.5
79.2
72.8
61.0
76.1
Noinstruct
33M
25.8
93.3
50.0
86.4
55.1
90.2
95.3
79.6
76.0
79.3
69.4
61.3
76.0
MSB
Student-s
33M
72.6
90.3
44.3
84.2
56.5
88. 8
94.9
77.2
75.4
81.2
64.9
60.4
74.2
NLL
Student-s
33M
27.3
89.2
43.8
86.7
58.0
88.3
95.5
81.9
76.7
80.7
66.1
60.6
75.4
hge- poe-hhseeee-v1.5
109M
76.2
93.4
48.9
87.0
51.9
90.8
94.2
76.9
76.2
80.2
71.6
59.4
75.5
Bas.
e5- -S-base-4k c5-base-v2
GIST
112M
76.0
77.8
92.8
50.5
46.7
83.5
87.3
47.0
89.7
86.2
93.7
95.3
78.1
75.3
73.0
76.0
79.6
77.7
72.1
72.4
59.3
60.4
76.0
73.8
m
110M
27.8
92.8
46.7
83,5
47.0
86.2
93.7
75.3
73.0
77.7
72.1
60.4
73.8
MSB
Student-m
109M
76.6
89.1
87.2
88.0
95.7
81.66
27.2
82.2
60.5
76.0
NLL
Student-m m
109M
79.6
89.5
45.8
88.0
59.7
88.3
96.2
83.9
78.6
82.7
67.1
61.3
76.7
bgc-large-on--1.5
335M
75.8
92.4
48.2
87.88
92.8
94.6
77.6
80.5
59.9
76.0
GIST
335M
75.6
93.4
49.1
$88.1
54.7
91.2
95.2
78.2
76.2
79.3
71.9
59.2
76.0
Bas.
UAF-I arr--e
335M
75.5
92.88
48.3
87.7
51.8
92.8
94.0
76.9
76.5
79.8
71.1
59.8
75.6
ember-yl
335M
76.1
92.0
47.9
$7.9
92.8
94.6
79.3
77.4
80.5
60.0
mwhii-emhedd-argee.
335M
75.0
93.8
49.2
87.8
50.9
92.8
94.0
76.8
76.2
80.0
71.5
59.7
75.6
MSB
Student-1
335M
27.3
84.5
43.4
86.0
60.0
82.7
95.1
79.8
76.3
81.3
65.8
60.2
74.4
Studens-1
335M
81.5
88.1
45.9
86.9
27.5
67.7
62.2
76.5
2024). For lack of better options and because it is currently the most widely accepted benchmark, we rely on the evaluation provided by the MTEB benchmark (Muennighoff et al., (203) 33 tasks encompassing clustering (11 datasets), sentence similarity (10 datasets) and classification tasks (12 datasets). We compare our models with distilled and non-distilled ones from the MTEB leaderboard.
Training set. We gathered different common datasets used for training embedders and collected 6 million entries from the Huggingface Hub, including Specter (Cohan et al. 20020), T5 (Ni et al. 202T), Amazaon QA (McAuley & Leskovec 2013), IMDB (Maas et al. 2011). SNLI (Bowman et al. 2013), QQP triplets from Quora, AG News (Zhang et al. 2015)), MEDI dataset (Su et 20.33 and the DAIL Emotion dataset (Saravia et al, 2018). We provide the dataset statistics in Sec. C.I.I The datasets are all flattened, such that if the original had two columns (e.g., sentence 1 and 2 in the SNLI dataset), we end up with twice the number of entries, one for each sentence, and we deduplicated the dataset. Models are trained for two epochs with batch size 16 on NVIDIA V100.
4.2 Distillation performance
Task performance. Our method produces models that exhibit strong performance on a large variety of tasks, ranking first amongst all models of similar size in the MTEB benchmark on most of the tasks Figuure 2b). Notably, we observe that our method produces models that are competitive for almost all the tasks, whereas other models appear more specialized. We provide the actual accuracy of our models on classification tasks in Tab. 1
We provide the full results for all model sizes in Sec.. C
Pareto frontier. Our goal with distillation is to increase information density of models to reduce computational costs and memory footprint, we show in Figure 2a that our method can pack more information into fixed-size models. Interestingly, our medium-sized model (109M parameters) outperforms all the models three times its size and even our 335M model under the same training setting. In addition, our small models outperform all previous model of their weight category, notably yielding 2-point gain on average classification accuracy on the MTEB over the previous state-of-the-ar efficient GIST-based embedders (Solatorio) 2024).

---

Classification
Clustering
80
Stadl NL
US1 MMNUV
Sted. X5 MSE
20
CTE
70
STS
Models
MTEB
65
MSE
NLL
METOR
M Masia
Madien
Tweate
Gual
Swebs
Temple
Ang Ches
75
50
100
Traid
Model Size (M)
Model Size (M)
Ang
(a)
(b)
Figure 2: (a) Pareto frontier size/performance in NLP. Our method (in blue) yields Pareto optimal model. (b) Global ranking of embedders on clustering and classification tasks for our xs model (23M). The NLLL-dstiiled model rank in most tasks and in average, outperforming all other baselines of its weight category and closing the gap with models 10 times bigger.
Comparison with standard MSE distillation. Consistent with results from reinforcement learning and interval (ateaationSStwaaaat et al.) 2023), training the student to match the teachers' embeddings using MSE loss results in consistently worse models.
Limitations of the embedding space structure. Our metric, which optimizes mutual information between the student and teachers, does not impose structure on the embedding space. Given that information remains invariant under invertible transformations, let fi and fa be differentiable and invertible mapping functions (diffcomorphissms); thus, IY) [(fi (X): fY)).. Consequently, our objective does not ensure the preservation of structural properties, such as pairwise cosine simi- larity, in the teachers' embedding space. Nonetheless method maintains competitive performance in both clustering and Semantic Textual Similarity (STS) (see Appendix C.2)
Molecular Embedders
We further method in molecular modeling, enabling the distillation of a student with models leveraging different modalities to represent a molccule: text, graph, and 3D point clouds.
InfoGraph
8.2
12
12
11
9.6
8.2
9.8
10
8.8
11
8.4
98
9
8.6
9
98
9.
11
11
9.8
17 11
6.5 11
68
10
ChemBenMMM
8.2
5.2
12
FRAD QMy"
11
6.2
9
11
4.2
5.6
8.2
5.
8
7
7.2
8.6
6
8.6
5
7.2
9.4
ChamGPT
9.2
8.2
7.2
5.2
8.4
9.4
8.N
8
7.b
8.6
7 11
4.6 6.9
8.
10
B.6
10
31
4.5
5.8
10
11
9.6
8B
9.2
9.8
3.4
6.2
8.2
8.5
GROVER
9
6.6
84
10
8.4
4.2
6.4
7.6
7.5
10
10
6.2 5.9
7.2
8.3
8
b.8
6.4
7.4
0.4
6.6 9.2 7.6
11
9.8
7.4
4.9
7:2
8
7.9
GraphCL
82
6.6
5.4
3.4
8.4
6.4
5.8
666
6.4
GrophLog
0.2
6.8
3.6
GraphMypM
6.4
3.8
34
3.2
4.6
6.2
5.2
5.2 4
4.6
6
6.5
8
6.2 8
5.4
4.2
6.2
32
4.2
6
5.6
MclR gat
5
4.4
5.8
8
64
5.6
3.0
5.6
6.2
6
5.3
2.2
3.6
5
4.6
7.6
6.2
5.7
6.2
4.4
5.2
7.4
6
8.6
2.6
4.2
7
7.0
2
5.1
Th
4.2
3.2
36
42
5
7.6
9
5:
3.0
4.4
5.4 8
7
67
1.
5.3
24
1.4
4
5.2
3
3.2
3.7
4.4
5
9.4
5:4
6.8
7.4
4.6
ChemRerMMRR
2.9
26
5.
2.2
4.2
2.2
Student-2M
1.6
1.6
3.4
1.2
LDS0
Canz
Syppplalctty
Sokubility FreaSde
PPER
VDas
Half Life
Chuaraere (H)
Clearance (M)
Average (me))
NERG
NERG/ (K)
AMES
Caronegens
DIU
Too21
Cla
PRMPA
HAA
Bioerallabin liblity
Pgp
BBB
CYP2c10
CYPSM
CYPLA2
CYP2C9
CYP2EO (s)
CYP208 (s)
CYP344 (s)
Average (cls
HiY
Figure 3: Ranking the TDC ADMET tasks. Our student consistently achieves competitive
performances across the evaluated tasks compared to its teachers (denoted by (0) and the other baselines, achieving the best average rank for both regression and classification tasks.
5.1 Experimental setting
Teachers and architecture. We use eight teachers trained on different modalities: SMILES (textual representation of the molecular graph) (Ahmad et al., (022). 2D molecular graphs Youu al. 2020
Xu et al. 2021 Liu et al. 2022 Stärk et al. 2021), and 3D structures (Feng et al. 2023). We identify
7

---

300
Tox
Absorption
Metabolism
0.60% 0.56% 0.75% 0.61%
0.82
0.825
Time/step (ms)
200
AUROC
0.82
0.80
0.800
HTS
Distribution
0.81
0.800
0.89
0.80
0.775
0.88
0
1-3dinfo
1-BertMTR
2-Teachers
student-2M
1
CompRess
MSE
Number of teachers
Cosine
(b) Computational overhead. Evolution of runtime for training step as function of the number of teachers. The computational overhead induced by an additional teacher represents less than 1% of the total runtime on a batch.
(a) Number of teachers and classification performances. Performance of students trained with teacher "1-.", 7, 2 teachers, and 8 teachers (student-2M). We compare our NLL distillation method to MSE, Cosine, and CompRess for cight teachers.
the teachers with (1) such as ChemBERTaMTR and use a 2D-GNN (Graph Isomorphism Network: GIN (Hu et al. (0200) for our student (for more details see Sec. B.1
Evaluation setting. We evaluated all models on the ADMET (Absorption, Distribution, Metabolism, Excretion, Toxicity) tasks of the Therapeutic Data Commons platform (TDC) (Huang et al. (2021) and on a high-throughput screening task (HTS), (HIV (Wu al. 2018)). We record the test performance over five runs (details on the evaluation procedure in Sec. B.3y. We trained our models on MOSES, a processed version of the ZINC Clean Leads dataset (Polykovskiy et al.) 2018), containing million samples, and on ZINC-250k (Irwin & Shoichet (005), consisting of 250,000 samples. The performances of the model trained on 250k samples can be found in InfSec. B. Both are public datasets of commercially available compounds designed to be used in various therapeutic projects.
5.2 Results
Overall performance. compare the performance the student model with the teachers and other baseline embedders on the different tasks. The results (average rank) for each task are presented infFigure Our student model achieves the best performance on both the regression and classification tasks, delivering the most accurate predictions across majority of tasks. This suggests that our method generates informative representations, providing high-quality molecular descriptors.
Single teacher vs. multi teati- To assess the impact of training a student with multiple teachers, we trained students distill the knowledge of a single teacher and two teachers, and compared the results to those of our student trained with eight teachers. We selected two of the best rest-performing baselines as teachers: ChemBERTaMTR-77 2021). We then trained student models on the 2M-molecule dataset. Figure 4a displays the performances of each of these student models on the regression tasks. Training with multiple teachers consistently outperforms training with single teacher, except on the Blood- Brain Barrier (BBB) task (the only Distribution classification task), which is also one of the tasks our model struggles the most with. For the BBB benchmark, we noticed it is one of the datasets where all results are among the most tightly packed (variations within 1.45 times the average standard deviation the results), and whose data distribution differs the most from the training set, which could explain the slightly lower average performance of the 8-teacher student compared to the or 2-teacher students. Overall, using multiple teachers significantly improves performance, with the best performance achieved when training with all eight teachers (additional results are available in See. B.4).
(Ahmad et al., 2022) and 3D-infomax (Stärk al.
Comparison to baselines. Figure 4a also compares the performance of our NLL distillation method to MSE, cosine, and CompRess distillation for eight teachers. Overall, in the evaluation of classifi- cation tasks, our NLL distillation method outperformed the Cosine and MSE distillation methods. This observation goes beyond the results of classification tasks, as we also observed that the NLL distillation method consistently outperforms the other two methods on all evaluated task categories (see Sec. B.1.3 for more details).
8

---

CIFARIO
DTD
STL10
SVHN
FGVCAircraft
CUB
Accuracy
100
100
100
100
100
100
50
50 # Params (M)
50
50
50
50
Params (M)
Params (M)
# Params (M)
# Params (M)
# Params (M)
Baselines
ViT-Student
Figure 5: Pareto frontier of vision models. The figure compares the performance of student model distilled using our method (named ViT-Student shown with color blue) with baselines (shown in yellow) across various datasets. The distilled student consistently lies on the Pareto frontier.
Computational complexity. Training our molecular embedders on the largest dataset (2 M molecules) takes approximately 50 hours A6000 GPUs. We evaluated the computational overhead induced by the multi-teacher setting in Figure 4b) The runtime a training step increases linearly with the number of teachers: +1.57ms per teacher, representing less than 1 %% of the total runtime.
6 Image Embedders
For our final modality, vision, we aim to assess whether our method can deliver competitive per- formance compared to other baseline models (teachers, and MSE, Cosine, and CompRess student), especially on fine-grained vision classification tasks. In the following subsections, we outline the experimental setup used to investigate these questions and present the results. Additional details, including hyperparameter tuning and the augmentations applied, can be found in Sec. D.3]
6.1 Experimental setting
Teachers and evaluations. Given the increasing of Vision Transformers, used large trans- former models (Swin (Liu et al. 20216), DINOv2 (Oquab et al. 2023), ViT (Dosovitskiy et al.) 2021) and BEiT (Bao et al., 2022), with around 87 million parameters) teachers, and selected smaller Vision Transformer, PVTv2 [Wang et al. (022c), with 3.7 million parameters, as the student. We also use some CNN based modes with different sizes as baselines have more comprehensive comparison of our student's representation abilities (refer to Sec. D.I for more details).
Training set. We include fine-grained datasets such DTD (Cimpoi et al. 2014), FGVCAireraft Maji et al. 2013), and CUB (Welinder et al., 2010), alongside CIFARIO (Krizhevsky et al. 2009). SVHN (Netzer et al. 2011). STL10 (Coates al. (2011) for the vision experiment. These allows us to assess the performance our approach on a variety of challenging and detailed classification tasks. Refer to Sec. D.2 details of the datasets meta-data
6.2 Results on Vision Transformer
To further evaluate our method, we conducted experiments using Vision Transformer (ViT) teachers. As shown in Figure 5] the distilled student model trained with our approach consistently lies on the Pareto frontier, for each task, showing a superior trade-off between accuracy and model size. Notably, our distilled student achieves the best performance among other distillation methods and other baseline models within its respective size categories, with results comparable to large VIT teachers (20x more parameters). This demonstrates our method's ability to effectively transfer knowledge from large, complex teacher models to smaller, more efficient student models, while maintaining comparable performance. Additional results Seec. D.4 show that our method generalizes well to unseen vision datasets, improving other distillation baselines, and effectively integrates diverse task-specific teachers without performance conflicts, confirming its robustness across domains.

---

7 Limitations
Our method focuses on training student embedding models for diverse, unknown tasks; for single, pre-defined tasks, task-specific distillation may be more effective. As with any distillation ap- proach-especially especially multi-teacher distillation there is overhead, either computational (if teacher embeddings are generated on-the-fly) or memoor-intensive (if precomputed). We mitigate this by precomputing and storing embeddings, requiring approximately 100GB of disk space for our largest text-based teacher. The quality of our student embeddings depends on the relevance of the teachers to the downstream tasks. While task-specific teachers provide limited benefits outside their domain, they do not degrade performance when combined with task-relevant teachers (Sec. D.4 Our opti- mization metric maximizes mutual information between student and teachers but does not explicitly structure the embedding space, potentially limiting performance in tasks like clustering. For textual embeddings, we observe significant gains in classification (where embeddings train a small classifier) but more modest improvements in clustering and STS tasks, which rely embedding dot products for similarity assessment Seec. C.2.2).
Conclusions and Future Work
We proposed a theoretically grounded task-agnostic distillation mechanism that leverages interval estimation through Gaussian kernels in high dimensions to distill more informative representation from multiple teachers single student. We demonstrated that our objective serves as a proxy for maximizing the mutual information and reconstructive capacity of the student model in relation to the teachers. We experimentally validated that our method more efficient than point sdimatioo--bsed multi-teacher feature distillation methods such as MSE or cosine-based distillation mechanisms. We demonstrated the superior performance of our method compared to others across three different modalities and numerous downstream tasks. In future work, we aim to extend this distillation approach to cross-modal distillation, enhancing the model's capabilities by leveraging task-agnostic cross-modal information.
10

---

References
Abbasi Koohpayegani, S., Tejankar, A., and Pirsiavash, H. Compress: Self-supervised learning by compressing representations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12980-12992. Cur- ran Associates, Inc., 2020. URL U hewpeeeeeddinnneeeersp...... O000FiiieePaaaccceeeeeccceee2622 Pappee pdf
Ahmad, W., Simon, E., Chithrananda, S., Grand, G., and Ramsundar, B. Chemberta-2; Towards chemical foundation models, 2022.
Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint urXiv:2012.09811 2020.
Aslam, M. H., Zeeshan, M. O., Pedersoli, M., Koerich, A. L.,, Bacon, S., and Granger, E. Privileged knowledge distillation for dimensional emotion recognition in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3338-3347, 2023.
Aslam, M. H., Pedersoli, M., Koerich, A. L.., and Granger, E. Multi teacher privieeeedd knowledge distillation for multimodal expression recognition, 2024. URL http.tttps//aaaii.orr////// 2408 09035
Axelrod, S. and
R. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022. doi: 10.1038/ 04155-022--0128--0 URL htRpsstts: wrg. 111111/////1111110 02000022-02222--111111
Bao, H., Dong, L., Piao, S., and Wei, Beit: Bert pre-training of image transformers, 2022. URL
http:///aaxx....oog//b///0000000 08254
Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning,
2017. URL URL https://aaxii...oo orgo aba/aaa//110 11t//17777.0887777
Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F. P., and Lakkaraju, H. Interpreting clip with sparse linear concept embeddings (splice), 2024.
Borza, D.-L., Darabant, A., Ileni, T., and Marinescu, A.-L. Effective online knowledge distillation via attention-based model ensembling. Mathematics, 10:4285, 11 2022. doi: 10.3390/math 10224285.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food- mining discriminative components with random forests. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014, pp. 446 461, Cham, 2014. Springer International Publishing. ISBN 978-3-319. 10599-4.
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Mårquez, L., Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 332-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 00.18653/v1/D15-1000 URL battps://aclantaaae
org org/D15-1775
Brody, S., Alon, U., and Yahav, E. How attentive are graph attention networks?, 2022. URL https://arxiv.org abs//
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object
detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S.. and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Coates, A., Ng. A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Cohan, A., Feldman, S., Beltagy, I., Downey, D., and Weld, D. S. SPECTER: Document-level Representation Learning using Citation-informel Transformers. ACL, 2020.

---

Darrin, M., Formont, P., Cheung, J. C. K., and Piantanida, P. COSMIC: Mutual information for task-agnostic summarization evaluation, 2024. URL httpt://axxiiiie http:///arxii..oraaa//////////// abs// 19457
Dietterich, G. Ensemble methods machine learning. In International workshop on multiple
classifier systems, pp. 1-15. Springer, 2000.
Dong, C.. Li, Y., Shen, Y., and Qiu, M. HRKD: Hierarchical relational knowledge distillation for cross-domain language model compression. In Moens, M.-F., Huang. X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 3126-310 3136, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/11/021..cmm .emnlp- mai..255.. URL https main.
//aclanthology org/ 2021.
Dong, Y., Miller, K., Lei, Q. and Ward, R. Cluster-aware semi-supervised learning: relational knowledge distillation provably learns clustering. Advances in Neural Information Processing Systems, 36, 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021. URL https:///arxs org/ abs/ 2010. 11929
Duval, Q., Misra, I., and Ballas, N. A simple recipe for competitive low-compute self supervised vision models, 2023. htttss:///axiii/
htt//bbss/2/111 094511
Dvomikk N., Schmid, C., and Mairal, J. Diversity with cooperation: Ensemble methods for few-shot classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3723-3731, 2019.
Fang, Z., Wang. J., Wang, L., Zhang, L., Yang, Y., and Liu, Z. Seed: Self distillation for visual representation, 2021. URLhttps arxiv. org/ /tt.a//.../...//....1...1/ 04731
Farebrother, J., Orbay, J., Vuong, Q., Taïga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, S., Faust, A., Kumar, A., and Agarwal, R. Stop regressing: Training value functions via classification for scalable deep rl, 2024. URL https://arxiv..t org. bbrgg//bb///0 2403. 24002039330000
Feng, S., Ni, Y., Lan, Y., Ma, Z.-M., and Ma, W.-Y. Fractional denoising for 3D molecular pre- training. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. https://pprcceeding mlr wwwppseeeeeeeepeepeppppp fetp:///eeng2cc.. html
PMLR, 23 23-29 Jul 2023. URL
Feng, S. Ni, Y., Li, M., Huang, Y., Ma, Z.-M., Ma, W.-.,, and Lan, Y. Unicorn: A unified contrastive learning approach for multi-view molecular representation learning, 2024. URL
https://arxiv. org/arg/ htt//bbs/24000 10343
Frosst, N. and Hinton, G. Distilling a neural network into a soft decision tree, 2017.
Gao, Y., Zhuang, J.-X., Lin, S., Cheng, H., Sun, X., Li, K., and Shen, C. Disco: Remedy self- supervised learning on lightweight models with distilled contrastive learning, 2022. URL https //ar //arxiv. org/ abs /2//21144.00 09124
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 1786))118--18 March 2021. ISSN 1573-1405. doi: 10.1007/ sl 11126-001-01155-
URL http:///:/ doi http:://///aaao.ooooo/////////// 00t.:////000.0011111222--0-----0
Hamilton, W. L., Ying, R., and Leskovec, J. Inductive representation learning on large graphs, 2018
URL https: htt///axxvvi wwwexiiv.orggaaao wtwp://////./..0//22222
He, K., Zhang. X., Ren, S., and Sun, Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Herbold, S. Autorank: A python package for automated ranking of classifiers. Journal of Open
Source Software, 5(48):2173, 2020. doi: 10.21105/joss.0217. URI. hhtps:ttp/.. ://oii// org/ 10 21105/ joss/joss.o.. 02173
12

---

Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint 1503002303.. 2015.
Hu, W., Liu, B., Gomes, Zitnik, M., Liang, P., Pande, V., and Leskovec, J. Strategies for pre-training graph neural networks, 2020. URL
12265
Hu, W., Fey, M., Ren, H., Nakata, M., Dong, Y,, and Leskovec, J. Ogb-Ise: A large-scale challenge for machine learning on graphs, 2021.
Huang, G., Liu, Z., Van Der Maaten, L.,, and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4700-4708, 2017. 2017.
Huang, K., Fu, T., Gao, W. Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks, 2021.
landola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ;0.5mb model size, 2016. URL https //arxiv. org/ abs abs/1602.c.. 11....02.770 07360
Irwin, J. and Shoichet, B. K. ZINC Free Database of Commercially Available Compounds for
Virtual Screening. Journal of chemical information and modeling, 45(():177-182, 2005, ISSN 1549-9596. doi: 10.1021/ci049714, htLhhtt:s///wrt ncbi nlm. nih, www.nniii...io..ee wathhhooo/mmmaaaa/e gov/pmc/articles/ PMC1360656/
Isert, C., Atz, K., Jiménez-Lunne J., and Schneider, G. Qmugs: Quantum mechanical properties of drug-like molecules, 2021.
Jiang. A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., las Casas, D., Bressand, F. Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q., Shoemaker, B. A., Thiessen, A., Yu, B., Zaslavsky, L., Zhang, J., and Bolton, E. E. PubChem 2023 update. Nucleic Acids Research, 51(DI)) https: http://ooo....rr/00000///////aae
10 2022. ISSN 0305-1048. doi: 1..00///ar//kac0556. URL
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine- categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 2013. doi: 10.1 N0.110//CCCW.20113..
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images, 2009.
Kubota, Y., Haraguchi, D., and Uchida, S. Impression-clip: Contrastive shape-impression embedding for fonts, 2024.
Li, S., Yang, X., Cheng, G.,, Liu, W., and Hu, H. Sa-mdrad: sample-adaptive multi-teacher dynamic rectification adversarial distillation. Multimedia Systems, 30(4), July 2024. ISSN 1432-1882. doi: 00.1007/0003300024-1111111
|ttt:tt:://ax http://t.............0000//0/000
X. and Li, J. Angle-optimized text embeddings. 2023.
Liang, C., Jiang, H., Li, Z., Tang, X., Yin, B., and Zhao, T. Homodistil: Homotopic task-agnostic distillation of pre-trained transformers. The Eleventh International Conference on Learning Representations, 2023.
Liu, L., Huang, Q., Lin, S.. Xie, H., Wang, B., Chang, X., and Liang, X. Exploring inter-channel correlation for diversity-preserve knowledge distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8271-8280, 2021a.
Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022. ht http://///pppeeeeeeeeeeee/r//r// wttt://orrmmiiiidxieeeeeelp
13

---

Liu, W., Chen, X., Liu, J., Feng, S., Sun, Y. Tian, H., and Wu, H. Emie 3.0 tiny: Frustratingly simple method to improve task-agnostic distillation generalization. arXiv preprint arXiv:2301.03416. 2023.
Liu, Y. L., Blodgett, L., Cheung, J., Liao, Q. Olteanu, A., and Xiao, Z. ECBD: Evidence- centered benchmark design for NLP. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 16349-16365, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022.aad 2ag.555o112222...looog...6.11 861. URI. https: htttps://acllanthooogg.o. 2rgp:///ooooooooo/222224
acl acl-long.86G 861
Liu, Z., Lin, Y. Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, Swin transformer: Hierar- chical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Ma, D., Zhang, K., Cao, Q., Li, J., and Gao, X. Coordinate attention guided dual-teacher adaptive knowledge distillation for image classification. Expert Systems with Applications, 250:123892. 2024a. ISSN 0957-4174. doi: https://ooi.org// wtwp://///o....a..... 123892. /Rt/ttps::/ httpt:///ww/ sciencedirect.com
Ma, N., Zhang, X., Zheng. H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference computer vision (ECCV), pp. 116-131, 2018.
Ma, Z., Dong, J., Ji. S., Liu, Z., Zhang, X., Wang, Z., He, S., Qian, F., Zhang, X., and Yang. L. Let all be whitened: Multi-teacher distillation for efficient visual retrieval. Proceedings of the AAAI Conference Artificial Intelligence. 38(5):4126 4135, March 2024b. ISSN 2159-5399, doi: 10. 1001660/aaai.v w3w...28..2.220 URL http://dx. doi org/10. 1609 aaai. v3815. 28207
Maas, A. L., Daly, R. E., Pham, P. T., Huang. D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 450-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: www. aclweb org/ anthology/ P11 1015
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 330651551, 2013.
McAuley, J. and Leskovec, J. Hidden factors and hidden topies: understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, pp. 165-172, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781 9781450324090. doi: 10. 10.1145/25071 0570077771115557776 13.. Uttpst/::://aol org/10. 1145/2507157 2507163
Merrick, L. Embedding and clustering your data can improve contrastive pretraining, 2024. URL
httts::/aaxxx org/ abs///b//// 2bbs 18887
Merrick, L., Xu, D., Nuti, G., and Campos, D. Arctic-embed: Scalable, efficient, and accurate text embedding models, 2024. URL httpstttts///aziiii orgee/iiiiorrrrreeeeeeeeeeeeeeee awwpaa/b..../// 05374
Mobley, D. L. and Guthrie, J. P. FreeSolv: a database of experimental and calculated hydration free energies, with input files. Journal of Computer-Aided Molecular Design, 28(7):111-700, July 2014. ISSN 1573-4951, doi: 10.10077/11022--14-97777 s 810822-014-9747-2
URL /ttp://////01..00000000000/
Morgan, H. L. The generation of a unique machine description for chemical structures-a at developed chemical abstracts service. Journal of Chemical Documentation, 5(2):107-113, 1965. doi: 10.1021//16001/all
URL httL:https///ot
org/10. 1021/c1600177011
Morris, C., Ritzert, M., Fey, M., Hamilton, W. L.,, Lenssen, J. F.., Rattan, G., and Grohe, M. Weisfeiler and leman go neural: Higher-order graph neural networks, 2021. URI, httpt::tt//aaaa Abss/810 02244
14

---

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark, 2023.
Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro,
B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407. 14679, 2024.
Murphy, K.
MIT Press, Cam- bridge. Mass. [u.a.]. 2013. ISBN 9780262018023 8029 026201 8020.
Machine learning : a probabillsstic perspective.
URL https
//www.a. amazon www.eaciieeaaeeeaaaaeee.cae
0b/0262018020/reffss:r 2 2tta:///ee2220001111333337777770
Navaneet, K. I.., Koohpayegani, S. A., Tejankar, A., and Pirsiavash, H. Simreg: Regression as a
simple yet effective tool for self-supervised knowledge distillation, 2022. URL https://axxxi org/abs/2201 05131
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng. A. Y., et al. Reading digits in natural images with unsupervised feature learning- In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Ni, J., Ábrego, G. H., Constant, N., Ma, J., Hall, K. B., Cer, D., and Yang, Y. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, 2021.
Noroozi, M., Vinjimoor, A., Favaro, P., and Pirsiavash, H. Boosting self-supervised learning via knowledge transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9359-9367, 2018.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint SEEi:::300.07999 2023.
Pan, Z., Zhou, X., and Tian, H. Extreme generative image compression by learning text embedding from diffusion models. arXiv preprint 1:X17:221..07733, 2022.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976, 2019a.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation, 2019b. URL https //arxiv. org/ abs/1904. 1904. 05068
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In 2012 IEEE
Conference on Computer Vision and Pattern Recognition, pp. 3498-3503 2012. doi: 10. 109/ CVPR.2012.6248092.
Peng, B., Jin, X., Liu, J., Li, D., Wu, Y., Liu, Y., Zhou, S., and Zhang, Z. Correlation congruence for knowledge distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
Pichler, G., Colombo, P., Boudiaf, M., Koliander, G., and Piantanida, P. A differential entropy estimator for training neural networks, 2022.
Pimentel, T., Meister, C., and Cotterell, R. On the usefulness of embeddings, clusters and strings for text generator evaluation, 2023.
Polykovskiy, D., Zhebrak, A., Sänchez-Lengeling, B., Golovanov, S., Tatanov, O., Belyaev, S.,
Kurbanov, R., Artamonov, A., Aladinskiy, V., Veselov, M., Kadurin, A., Nikolenko, S. L., Aspuru- Guzik, A., and Zhavoronkov, A. Molecular sets (MOSES): A benchmarking platform for molecular generation models. CoRR, abs/1811..2822 2018 URL http: www./ogg////aa.e.e.oeo.o
12823
Qiu, S., Han, B., Maddix, D. C., Zhang, S., Wang, Y., and Wilson, A. G. Transferring knowledge from large foundation models to small downstream models. arXiv preprint arXiv:2406.077 073337 2024.
15

---

Sandler. M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., and Chen, Y.-.. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18- 1404. URL https 7/ wralwweb.ogggaane www..r//whhoooggg/i-------
Sarkar, P. and Etemad, A. Xkd: Cross-modal knowledge distillation with domain alignment for video representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14875- 14885, 2024.
Scimeca, L., Rubinstein, A., Teney, D., Oh, S. J., Nicolicioiu, A. M., and Bengio, Y. Short- cut bias mitigation via ensemble diversity using diffusion probabilistic models. arXiv preprint arXiv:2311.16176 2023.
Shayegh, B., Cao, Y., Zhu, X., Cheung, J. C., and Mou, L. Ensemble distillation for unsupervised constituency parsing, International Conference on Learning Representations (ICLR), 2024.
Solatorio, A. V. Gistembed: Guided in-sample selection of training negatives for text embedding fine- tuning, arXiv preprint arXiv:2402. 16829, 2024. httpl//tte///aa...ooor/a/aa/////
16829
Song, K., Xie, J., Zhang, S., and Luo, Z. Multi-mode online knowledge distillation for self-supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11848-1 11557, 2023.
Stewart, L., Bach, F., Berthet, Q., and Vert, J.-P. Regression as classificattin: Influence of task formulation on neural network features, 2023.
05641
Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., and Liò. P. 3d infomax improves gnns for molecular property prediction. arXiv preprint arXiv:2110.04126 2021.
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., tau Yih, W., Smith, N. A., Zettlemoyer, L., and Yu, T. One embedder, any task: fineettion-finetueeed text embeddings, 2023. URL /taps://arxizn www.ii..oggaaaae 22212
2212 09741
Sun, S., Ren, W., Li, J., Wang, R., and Cao, X. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15731-15740, 2024.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 1-9, 2015.
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. Mnasnet: Platform- aware aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2820-2828. 2019.
Theisen, R., Kim, H., Yang. Y., Hodgkinson, L., and Mahoney, M. W. When are ensembles really effective? Advances in Neural Information Processing Systems, 36, 2024.
Touvron, H., Martin, L.. Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov. N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunoy, S., and Scialom, T. Illaa 2: Open foundation and fine-tuned chat models, 2023.
16

---

Vilnis, L. and McCallum, A. Word representations via gaussian embedding. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv org/aba 111//1111..0 6623
Wang. H., Li, W., Jin, X., Cho, K., Ji, H., Han, J., and Burke, M. D. Chemical-reaction-avaaee molecule representation learning. In International Conference on Learning Representations, 2022a. net/ wow.tforrmmmiiiia wett/Eorrmmiid=6hh3p12....
URL htttts:://ppprreeeeeee//
Wang, H., Ma, C., Zhang, J., Zhang. Y., Avery, J., Hull, L., and Carneiro, G. Learnable cross-modal knowledge distillation for multi-modal learning with missing modality. International Conference on Medical Image Computing and Compuler-Assisted Intervention, pp. 216-226. Springer, 2023.
Wang, K., Yang, F., and de Weijer, Attention distillation: self-supervised vision transformer students need more guidance. arXiv preprini arXiv:2210.00000000000 2022b.
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 833)4115-24, 2022c.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., and Perona, P. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Wenlock, M. and Tomkinson, N. Experimental in vitro dmpk and physicochemical data set of publicly disclosed compounds, 2021. URL httph::/ttw.w..... ebi ac. uk/ httk/chhbmm/dcmmett.. ssportt_carr/CCEEMMB3330118111
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jermite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. M. Huggingface's 's transformers: State-of- the-art natural language processing, 2020. URL httph:ttt////iiti
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V. MoleculeNet: A Benchmark for Molecular Machine Learning, October 2018. URL
00564
Xie, E., Wang, W., Yu. Z., Anandkumar, A., Alvarez, J. M., and Luo, Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34: 34: 14411-077--9999 2021.
Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Xu, H., Fang. J., Zhang, X., Xie, L.. Wang, X., Dai. W., Xiong, H.. and Tian, Q. Bag of instances aggregation boosts self-supervised distillation, 2022a. URL http:::////xxxvoo/////bba 01691.
Xu, H., Koehn, P., and Murray, K. The importance of being parameters: An intra-distillation method for serious gains. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 170-183, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10. URL
htt.://...rrr022222..mmmmmaaaaaa wwnlp-mainnnn..... 13
Xu, K., Hu, W., Leskovee, J., and Jegelka, How powerful are graph neural networks? In International Conference on Learning Representations, 2019. /ttp://e///// net/ forum?id Corumiid=ryGs6iA5ki
Xu, M., Wang, H., Ni, B., Guo, H., and Tang. J. Self-supervised graph-eeel representation learning with local and global structure. arXiv preprint arXiv:2106.04111 2021.
Ye, X., Jiang, R., Tian, X., Zhang, R., and Chen, Knowledge distillation via multi- feature ensemble. IEEE Signal Processing Letters, 2024.
17

---

Yim, J., Joo, D., Bae. J., and Kim, J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7130-7138, 2017. doi: 10.1109/CVPR.20177754
You, Y., Chen, T., Sui, Y., Chen, T., Wang. Z., and Shen, Y. Graph contrastive learning with augmentations. In Larochelle, H.. Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812-5823, Cur- ran Associates, Inc., 2020. URL ptooe://rrooo neurips cc/ ww/papprr/002/fii/// 33223378881223117777633334422222 .df
Yuan, F, Shou, L., Pei. Lin, W., Gong, M., Fu, Y., and Jiang, D. Reinforced multi-teacher selection for knowledge distillation, 2020.
Zagoruyko, S. and Komodakis, N. Wide residual networks, 2017. URL httpptt///xx/ii.orrr /bs/ $605.07144
Zhang. H., Chen, D., and Wang. C. Adaptive multi-teacher knowledge distillation with meta-learning. In 2023 IEEE International Conference Multimedia and Expo (ICME), pp. 1943-1948. IEEE, 2023.
Zhang, L., Song. J., Gao, A., Chen, J., Bao, C., and Ma, K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
Zhang. X., Zhao, J. J., and LeCun, Y. Character-level convolutional networks for text classification. In NIPS, 2015.
Zhu, J., Liu, J., Li, W. Lai, J., He, X., Chen, L., and Zheng, Z. Ensembled etr prediction via knowledge distillation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp.
2020.
18

---

Appendix
Table of Contents
Proofs
20
Proof Theorem 3.2
20
B Molecular Modelling
21
B.
Model architecture
21
1.1.1
Chosen Teachers
................................
21
1.2
Architecture influence
21
B. 1.3
Additional results on the TDC datascts
22
B.2
Kernel's predictive power
23
Evaluation details
24
B.3.1
Benchmark Choice
24
B33.2
Evaluation Procedure
24
B.3.3
Evaluation Metries
24
B.4
Single- Teacher setting
24
B.5
Comprehensive results
26
Natural Language Processing
29
Training set and hyperparameters
29
C.I.I
Training set
................................
29
C.1.2
Teachers and based students performance
................................
30
C.1.3
Single teacher distillation
30
C.1.4
Hyperparameters)
30
C.2
Detailed evaluation results
30
C.2.1
Evaluation on classification tasks
30
C.2.2
Evaluation on similarity and clustering tasks
30
C.2.3
Analysis and compare most recent embedders
31
D
Vision
37
D.1 Model architecture
37
D.2 Training Set
37
D.3 Vision Details
37
D.4
Complementary Results
37
F Detailed Method
40
F
Computaional ressources
40
G Baselines
40
Discussion On MSE distillation
41
Funding
47
19

---

A Proofs
We denote as the random variable over that describes the input distribution. We suppose we have access to dataset D = [x, )= C X' of inputs drawn following px and different embedders Tk X' + Rdx, k E (1.....K3 K,, that map the inputs to different embedding spaces. We denote Z = (X) as the random variable over RRdx that describes the embedding of the input distribution in the k-th embedding space and by = S(X) the random variable over Rd that describe the embedding of the input distribution the student embedding space, We denote by z Tx (x;) embedding of x; in the k-th embedding space. We are interested in learning representation that captures the information contained in all the embeddings.
Let us consider a task characterized by a target set Y of discrete concepts and the feature space X with joint probability measure Pyx E P(Y X). For every projection the features through the different teachers, the Bayes decision rule cT _ arg max Exy [c(TT((X) Y|] and similarly
cR +y
for the student: o5 - arg Exy [1[c(S(X)) Y
cR-
We leverage the following recent result from (Darrin et al. 2024):
Proposition A.1. Let CT, = of (X)( and Cs os (x)) denote the outcome of the Bayes classifier observing the output of the teacher Tk and the student S, respectively
Pr(Cs # CT2) exp (()((()))()))))
A.1 Proof of Theorem 3.2
By applying the above proposition to all the terms in Eq. we obtain the following bound on the loss function:
Proposition 1 (Upper bound).
K
L. (XY, S T....., Tx) <
Σ
exp (x)(((()))(())))
(5)
K
k=
K
< - exp
K (T((X))((
(6)
Negative log likelthood
Proof.
K
L S Tx) <
exp (((((())))(()))
K
k=1
1
exp ( (-h((x(X))((X))
K
1 K
<1 +
Σ
exp
(-N(T((X))S(())
K
k=1
K {((x(X)))(X))
< 1 - exp
K
km
We simply rearrange the terms and use the fact that x exp(-2) is concave to interchange the sum and the exponential.
20

---

B Molecular Modelling
B.1 Model architecture
We trained a 10-layer GINE (Hu al. 2020) neural network with a 512 hidden dimension, using 2-layer network for the message passing process. We use the atomic number of each node as input, as well as possible chirality information, and the nature the bond between each pair of nodes. We batch size of 256 and learning rate of 1e 4 to train the model for 400 epochs the 250k dataset and 200 epochs on the 2M dataset. For the teacher-specific kernels, we used a 3-layer MLP with hidden size of 1024.
B.1.1 Chosen Teachers
The teachers used to train our molecular modeling students are summed up in Tab. 2 We gathered various representation models for molecular modeling, with different pre-training objectives, input modalities, architectures, and training datasets.
Table 2: Description of all teachers used our experiments.
Model name
SMILES
2D-GNN
3D-GNN
Architecture
Out size
Dataset (size)
GraphCL
300
GEOM
GraphLog
GIN
300
GEOM
Andod Beabest
50k)
GraphMVPPss
GEOM
Astrod Baskwel
(5k))
3D-infomax
PNA
800
QMugs
(220k
ChemBERT MTR Atrataal
RoBERTs
384
PubChem Kinata
SM. 10M. 77M)
fractional Regurt
TorchMD
PCQMAM
(3.7M)
B.1.2 Architecture influence
Sum
ChemBenMTR-77N
FRAD_QM9
name
EMME-student
GAT-student
1.8
train Val
spiit
Train
name
GCN-student
TAG student
SAGE student
1.8
spitt
GIN studett
train
300
100
300
300
300
400
Epoch
Epoch
Epoch
Figure 6: Training loss of different students using different GNN architectures on the ZINC-250K
dataset.
Figure 6 shows the training loss of the student model with different GNN architectures on the ZINC-250k dataset. In particular, we compared the GINE architecture with Graph Convolutional
Network (GCN) (Morris et al., (021). Graph Attention Network (GAT) (Brody et al., 2022).
GraphSAGE (SAGE) (Hamilton et al. 2018), a Toplogy Adaptative Graph Convolutional Network (TAG) Brody et al. 2022). and GIN Network, that separates from the GINE architecture by the fact that it does not take edge features into account (Xu et al. 2019). We observe that the GINE architectures outperform the other architectures, with lower training loss, faster convergence, and lower validation loss. The Graph attention network (GAT) is the second best performing architecture, but it is still outperformed by the GINE architecture. These two architectures are the only ones to use the edge embeddings in the message passing process, which could explain their better performance.
Models aiming at incorporating 3D information into 2D-GNNs models. We used the three versions of ChemBERT-MITE models trained 5M, 10M, and 77M.
21

---

Indeed, all other architectures perform worse, especially when considering their validation loss computed on 10% of the training set. Specifically, the GIN architecture, not using edge feature, performs significantly worse than the GINE architecture, while having similar architecture.
For experiments, decided to the GINE architecture, as it performs the best during training and converges faster than the other architectures.
B.1.3 Additional results the TDC datasets
Table 3: Average rank of each model on the ADMET and HTS downstream tasks from the TDC (Huang et al. 2021) platform. Our student outperforms all baselines, including teachers, on average.
Absorption
Distribution
Metabolism
Excretion
Tox
HTS
Avg
InfoGraph
13,50
13.27
13.32
11.40
11.98
9.40
12.16
ChemBer1MLM-100
10.65
11.00
10.70
13.80
11.11
14.60
11.98
FRAD (M9
10.57
11.13
10.38
8.33
10.04
7.80
9.71
ChemGPT-1.2B
9.55
11.73
11.75
10.73
10.86
11.20
10.97
GROVER
10.43
8.33
11.25
8.53
10.38
11.00
9.99
GraphCL
10.89
8.53
9.45
10.13
8.70
9.80
9.58
GraphLog( (1)
11.05
7.80
9.07
10.53
8.93
14.00
10.23
GraphMVP(t)
7.20
6.20
7.85
9.80
7.49
8.80
7.89
MolR gat
6,95
7.60
8.30
8.53
ThreeDInfomax
4.17
6,00
7.58
7.13
6.16
10.40
6.91
ChemBertMTTR-7MM
3,50
4.27
5.75
5,00
6.03
4.20
4.79
MSE
8.07
6.40
5.55
6,33
7.55
3.00
6.15
Cosine
5.51
6.13
3.60
4.33
4.97
6.20
5.13
student-250k
3.55
6.20
2.70
2.40
3.80
3.94
student-2M
4.40
5.40
2.75
3.00
2.40
3.72
The average rank of each model in each task category can be found in Tab. 3
Surprisingly, the performances of the "student-250k" and "student-2M" models are similar average. Specifically, the student-250K model outperforms the student-2M model on regression datasets notably, by achieving the best performances Tomkinson, 202I) tasks. This suggests that our method can leverage the diversity the teachers to learn more informative representations, even when trained smaller dataset of 250k datapoints.
the FreeSolv (Mobley & Guthrie 2014) and Lipophilicity (Wenlock &
Sum
GraphMVP
ChemBertMTR-774
1.5
Train Loss
1.0
0.5
0.0
name
FRAD QM9
ThreeDinfomax
GraphCL
5-layers
2-layers-kemel
2.0
3-layers kemell
splt
train
1.6
2
400
Epoch
Epoch
Epoch
Figure Training loss of the student model along the training with different kernel-size on the ZINC-2506 dataset.
22

---

LD50
Half Life
AMES
Solubility
0.55
0.83
0.2
0.89
0.82
0.0
0.88
HIV
hERG (k)
Tox21
Free
0.925
0.80
0.82
0.875
CYP2C9
Carcinogens
Pgp
CYP1A2
0.94
0.85
0.80
0.92
0.925
0.75
CYP2C19
Skin R
BBB
CYP2C9 (s)
0.885
0.80
0.88
0.70
0.880
0.65
AUROC/R2
0.86
0.75
PPBR
Clearance (M)
HIA
Caco2
0.45
0.40
0.3
0.950
0.2
0.925
0.65
Clin Tox
DILI
CYР3А4
hERG
0.7
0.90
0.90
0.85
0.6
0.85
0.80
0.75
CYP2D6
Bioavailability
VD
PAMPA
0.70
0.2
0.775
0.750
0.725
CYР3A4 (s)
CYP2D6 (s)
Clearance (H)
Lipophilicity
0.80
0.15
0.76
0.75
0.10
0.74
0.60
0.05
4
2
2
2
4
n_ pluluster
n_cluster
n_ clusteer
n_ nlcluster
Figure 8: Test AUROC/R2 score of the students on the classificationreegrssion tasks, trained with different kernel-size the ZINC-250K dataset.
B.2 Kernel's predictive power
Our method relies teacher-specifie heads to distill the knowledge of each teacher. In this section, we wish to evaluate the impact of the choice these kernels and their predictive power (in terms of depth) on the performance and training of the student model.
We performed this experiment with kernels of depth 2, 3, and 5, and we trained the student model with these kernels on the ZINC-250K dataset and evaluated the performance of the student model on the ADMET and HTS downstream tasks.
First, during the training, as expected, the more powerful the kernel, the lower the training loss is (see Figure 7), even though difference significant, especially between the students using kernels of depth 3 and 5. Overall, the performances of each student on the downstream tasks are similar, underlining the robustness of our method regarding the choice of the kernel's depth (see Figure 8). For our experiments in the main paper, used a kernel of depth 3, as it enables the best trade-off
23

---

between computational complexity, and training convergence while providing competitive results on the downstream tasks.
B.3 Evaluation details
B.3.1 Benchmark Choice
We selected total of 32 tasks, extracted from the Therapeutic Data Commons (Huang al., 2021) platform, 8 absorption tasks, 3 distribu- tion tasks, metabolism tasks, excretion tasks, 9 toxicity tasks and high-throughpus screening task. summary the tasks considered can be found in Tab. 4] with their corresponding size (total number samples) and type (classifica- tion regression). For all tasks, we computed 5 conformations for each molecule, and used the least energetic as input of our 3D models.
Table 4: Tasks extracted from the Therapeutic Data
Commons platform considered in experiments.
Calegory
Model
Task
reg
P-glycopratein Inhibition
1212
AqSolDB
9982
Lipophilicity
4200
Absorption
Caco-2 Permeability
906
Human Intostinal Absorption
578
FreeSolv
642
PAMPA Permeability
2035
Oral Bioovailability
640
Plasma-Proted BDR
1614
Distribation
Glood-Bratt barrier
VDss
1130
B.3.2 Evaluation Procedure
CYPP450 Inhib.
12328
CYPP450 IA2 Tahih
12579
For every task, we opted for a random split since we obtained similar results to a scaffold split, with a faster computation time, with a ratio of 70/10/20 for the train/validationntess sets. For all tasks, we compute the embeddings generated by each model on the task. We then train 2 layer perceptron with a hidden size of 128 on the task 5000
Inhih.
CYPP450 2CY Inhib.
12092
Metabolism
CYPP450 Inhih.
CYPP450 213 Substrate
664
CYPP450 3A4 Substrate
CYPP450 2C9 Substrate
666
Clearance hepatncyte Half Life
Excrction
667
for min(100, 200 task size epochs (to limit the compute time on large tasks) with a learning rate of 1e We then select the best checkpoint according to the validation performances and report the test metries of this checkpoint.
Tox21
7831
bERG
13445
648
Acule Toxicity LD50
7385
Toxicity
Ames Mutagenicity
7255
Coi
1484
Carcinogens
278
Drug Induced Liver Injury
Skin Reaction
404
B.3.3 Evaluation Metrics
HTS
HIV
40000
We repeat this process five times with different seeds in the train-val-test splits in order to enable
the establishment of robust rankings using autorank (Herbold 2020). We decided to report the ranks of the models enable the comparison of the models on both classification and regression by simply averaging the rank. To compute the rank on all tasks, we rely on the AUROC score for classification tasks and the R² score for regression tasks. For the excretion tasks, since the regression labels have large variance, we decided to apply the regression on the log-values and report the R² score on the log-values.
B.4 Single-Teacher setting
To assess the impact of the multi-teacher setting on the performance of the student model, we trained students to distill the knowledge of a single teacher. We used only the two best performing teachers. 3D-infomax (Stärk et al.) (2021) and ChemBERTaMTR (Ahmad et al.) (2022). to train the student model on the 2M datapoints dataset. We also train student with both teachers, to see if those two teachers are sufficient to achieve the same performance as the models we presented in the core of the paper.
Figure 9 shows how these students underperform compared to a student trained with all teachers, in terms of AUROC for classification tasks and R22 for regression tasks respectively. These tables also show that the student trained with both teachers performs better than each student trained with only one teacher. All results are aggregated in Tab. 6 and Tab.
24

---

Table 5: Performance of the student models trained with only the best teacher ("I-ChemBertMTTRR)
the second-best teacher ("I-3dinfo"), both teachers together ("2-teachers"), and "student-2M" on regression tasks (R2).
Absorption
Lipophilicity
Absorption
Absorption FreeSoly
Absorption
Solubility
Caco2
LD50
Tox
avg
1-3dinfo
0.392 0.317
0.654
0041
0.822
001
0.583
0017
0.798
0.100
0,471 =
0,048
1-BertMTR
0.405 0.309
0.660
0.066
0.829
0.31
0.582
0044
0.803
0.000
0.480
0.025
2- 2-Teachers
0.449 0.312
0.692
0
0.882
0.034
0.688
0,22
0.812 0.012
0.497
0.033
student-2M
0.476 :: 0.301
0.687
0.445
0.878
0.739 +
0.822 + 0.005
0.543
0.041
Clearance (H))
Clearance (M)
Distribution
Distribution
Excretion
Exerction
PPBR
VDss
Excretion
Half Life
1-3dinfo
0.316 0.062
0.130
0.146
0.048
0.095
0.137
-0.037
1-BertMTR
0.347 + 0.070
0.145 +: 1,772
0.051
+ 0.148
0.136
0.110
0.017
0.195
2- 2-Teachers
0.419 +
0.172
0,098
0.066
0.199
0.061 -
student-2M
0.389 0,s0
0.138 0.115
0.069
0.660
0.348
0.022
0.144
0.285
Table Performance of the student models trained with only the best teacher "(1-ChemBBrtMTTR")
the second-bes teacher (" ("1-3dinfo"), both teachers together ("2-teachers") and "student-2M"" on classification tasks (AUROC).
Bioavailability
Absorption
Ahsorption
Absorption
PAMPA
Ahsorption PEP
Distribution BBB
avp
HTS
HIV
1-BertMTR
0.801
H: 0100
0.631 + 0.009
0.910:1:
9,144
0.719.t 0.000
0.93311 6m11
0.886::
6,122
0.756.t
0006
1-3dinfo
0.803
0.000
0.631 + aoen
0.899+: onss
0.737
0.930+
0434
0.890-:
8.114
0.763:t
o010
2- Teacheers
0.808
+ 0u9
0.641 + uses
0.868+ 0:51
0.749- cum
0.927
0.881+
+ ents
0.786
6012
student-2M
0.825
0.66
0.653+ a8e5
0.959+
0.730 0004
0.936:
6.24
0.8824
0.800
Metaholism
CYP1A2
Metabolism
CYP2C19
Metabolism
CYP2C9 (
Metabolism
CYP2C9
Metaholism
CYP2D6 (5)
Metabolism
CYP206
Metabolism
CYP344
Metabolism
CYP3A4
1-BertMTR
0.916:
0.866
0.622-:
0.874
0.729
0.839
0.638
0,848::
1-3dinfo
0.91 6sb 0o0e
0.870:t 0006
0.630 808
0.874
0005
0.721
0.238
0.836
0.99
0.638
0015
0.855
0000
2- 2-Teachers
0.924
0.8711
0.627
0.883
0.725
0.848
0.632
0.881
student-22 2M
1
0
0.697
0.893
0.766
0.868
0.639
0.892
Tox Careinogens
ClinTox
NERG (K)
Tox AMES
DILI
TOx
Skin
Tox
Tox bERG
Tox
I-BertMTR
0.862:
0,831
0.601
0.831
0.815
0.801
0.826:: 0025
0.877 6012
1-3dinto
0.864+ 4er1
0.826
0.603-+ euna
0.841 n39
0.835
eust
0.802+ ooss
0.831 + 6a19
0.875 +
oven
2-Teuchers
0.875
0.827
0.6574
0.865
0.770::
0.814:
0.826
0.898
Student-2M
0.891
0.839
0.662:1
0.856:1
0,801
0.819:1
0.834
0.911
25

---

LD50
Half Life
AMES
Solubility
HIV
hERG (k)
Tox21
FreeSolv
Test port.
0.55
0.925
0.90
0.1
0.925
0.50
0.0
0.850
0.800
0.75
0.875
CYP2C9
Carcinogens
Pgp
CYP1A2
CYP2C19
Skin R
BB8
CYP2C9 (s)
0.90
0.94
0.94
0.88
0,90
0.85
0.70
0.92
0.65
0.86
0.86
0.90
PPBR
Clearance (M)
HIA
Caco2
ClinTox
DIU
CYP3A4
hERG
0.70
0.40
0.3
0.850
0.35
0.90
0.85
0.60
0.825
0.82
0.30
0.85
CYP2D6
Bioavailability
VDss
PAMPA
CYP3A4 (s)
CYP2D6 (s)
Clearance (H)
Lipophilicity
lest pert.
0.875
0.6
0.175
0.750
0.64
0.075
0.7
0.150
0.725
0.050
0.825
0.625
0.125
0.82
0.8
Student-2M
2-7837
1-3dinio
student-2M
T-Tachhrr
1.Bertaii
1-3dinfo
studentzM
2: Soorre
1-3dinfo
student-2M 2-Teachers
1-8ertMTT R 1-3dinfo
student 2-Teachers 1 1-BeitMTH 1-3dinfo
student-2M Sreacher
1-3dinfo
Student-2M
2-Teachers
1.BortMTR
1-3dinfo
studentzah iieeaa Sorttn Sagnnn
Figure 9: Test AUROC/R score of the students on the classfiiationnreerrssioon tasks, trained with all
teachers (student-2M), two teachers (2-Teachers) and teacher (1 (1-ChemBertMITR for the model
trained with ChemBertMTR--77M and 1-eacher-3dinfomax for the model trained with 3D-infomax).
B.5
Comprehensive results
The following tables provide the raw results of the different evaluated models on the ADMET and HTS downstream tasks. TTb.. 7 and Tab. 8) display test performances of the models on the classification and regression tasks respectively. All regression tasks are evaluated using the Rt2 score, while the classification tasks are evaluated using the AUROC score. We report the mean values the metrics over 5 runs for each task, as well as the standard deviation.
We display in Figure 10 the evolution of the average rank the embedders when separating the tasks based on the amount of samples, and the class imbalance (for classification tasks). Our student appears robust both setups, even though as the class imbalance becomes more important, or as the amount of samples in the task decreases, the difference between the top- tep-pefforming embedders becomes less significant.
26

---

Table 7: AUROC of each model on the ADMET and HTS downstream classification tasks. The best emhedder for each task is highlighted in bold and underlined, and the second best is highlighted in bold.
Metaholism CYP3A4
0.817
one
0,555 um
0.846
um 0-4462 un
0,447
0.474 MA
0.733L un
0283: am
0.71 dh
0,89
0 S
0.892 MAM
BERG
0.501 nos
0.567 noo
0,873 iou
0.867 soo
0.550 noe
0.854
0,510
0.8 se
0.574 on
0.997 ane
0,855 sno
0,000 so
0,012 sm
691 kmo
01.977 und
01600 ars
0.6077 use
0.607 and
0.6194 am
0.598 und
0,60 aet
0.645 um
0.637 wes
0.646e aN
0.639L Wn
0.77KL ogr
0.789 aae
0317L am
0.799 ax
0.774 TMu
am
032296 cm
03822 an
03244 axe
0,500 3m
0.928 U
0:834 .R.
CYP344 (s)
Metabolism
0.00 0,606.
0.024
0.64 12
WERG
0,7996
0.7971
0.815 5 oee
0.841 MAE
0.:445 oue
0.868 aneo
0.70 aes
00971 3m0
0779 sns
07002 MO
030E 2014
0,182 aNO
0,807 & M0L
0814# S0A
0.175 a01
03195 MA
Metabolism
CYP206
(1.00
0.806
0.33 0.81 3.1
01,822
0.830
00.0-22
0. 0,00% (6.649 -
Tox
TOx
0762
07777
0,001
0.
(1
0 K0
0.700- om
0 A.e
Metabolism CYP2D6
(.674
0.21 1: m11
0.753
0,75012
0,723 1m
UL760
0:747 20u
0.71
0.734 aec.
0.7422
0,758
0,767 was
0. 6.70612 ore
Tox
0.221
0.133
0341
02
0770
0.75
0.750
0.770
0.000
2
MM
AM
MM
AM
03422 sm
0,58 am
0.79- u
0355% uE
03556 cm
Metabulism CYP2C9
0.8601 sx0
02062:5: so0
0.870 a00
0.889LL 1MM
0N77- ever
0,89 aas
0.893 eaa
DILT
Tox
0-337
0337
0.452
0.79
0.442
03275
0,552
0358-
0.558
0.851
01,6611-
0,631 1. MnS
M02
0657 MO
0 and
0.754 200
0,54 MAN
04 aN0
0453 and
0 and
am
0.600 0.609 s0e
0.6-52 sins
0.622 a00
0.615 M407
0,571 ave
0.600- and
0,009 aae
0.697 2 wns
Tox
Tax
0.11
0,553
0
06392
CYP2C9 (s)
Metebolism
0,0244
0.663
0.616+
0.663
0,673
0.728 one
0,785 OZ
mu
PM
cm
0.001 AN
0.703 AA
0,00 AA
0310 CN
0339 AN
0.845 003
0SS5 eve
0.865 0o0
0 0.859 0099
0.853 ovn
1. is wre
0,882 2: eNr
Carsinggss
Tox
0,772
0,778
0.779
01477
0,793
0.776
Metabolism
CYP2C19
0.852
0..33
0.843
01355
11 16-0-
0.80 0,899
3M2
L1 3M0
0.044 am
0.00 201
03P22 aHE
0.67 Lt. 20
0.88
Tox AMES
0,906 oo-
000
0.899: enL
0,8991 o00
0,909 oo-
0.917 out
09919 exr
0,93 exe
0:933 en0
0-085
0.815
05
858
0.007
0
0.009-
0.
0.80
Metabolism CYP1A2
0.870
0.88611
0.800
0.90 dh
0,928
0,886 we2
0,006 meY
21 MCV
091 1- amS
09202 MCS
are
0008- arE
8376 KIA
0914 4 ViN
0,336 sm
6936 and
Absorption
Per
0.9
09182
0920
0,700 ane
ax.
0.750 am
2014
0.765- axe
0,731 axS
0.797 anD
exe
0.797 wxa
0,796 and
0,000 aod
HTS HIV
0.738
0,722--
0,797
0,7Wie
0.055L o/
0,005 1
1M0
0.745 on
0.735 WAY
0,755 M
0,7421 u
0,73H 0E
Alsorption
0,0992
0.155
02003
0.7091
0.703:
0
om
0 0.959 om
ons
0 8KS om
0.574 ons
0 0676 ozs
on
0.994 ana
WTee orn
0,875 ura
0.882 ona
Distribation BBB
04445
0 868
0,867-
00 til
Absorption
0-822 1er
033 km
kM
an
09974 am
0.98G- Kn
0914 kom
0.008 son
0.90P Mr
019592 sm
HIA
0.5024
0.901
0.3633
0. 78.85 o1
00s
0.7922 on0
0.800 o00
0.808 ooe
ere
0.816 om
M0 one
0,823 une
0,825 do ons
0.688 L PM
4 m
0670 and
0 we
0 1 IN
06:3 mm
04671 axz
0USSE em
avg
707
(,779)-
0. 785
1.79)--
0.8
Bioovailahiliy
Abscpption
06
0
0600
0
0
0.725
InfaChaph
ChenGiP-: 1:33
TRAD Qlayro
ChemBeerMLL Mi 10M
GROVER
Goppling
GrahMVPp
MalR gat
Three Bafomax
ChemBerhiTR- 77M1
MSE
Cosine
stodknt-22 2M
JatoGnght
C 20
FRAD QND
ChemlleMM Ne 10M
CHCOVER ER
Gnghlagg
GraphMyp'
gn
Chem 7
MSE
Coaine
muden 200k
stodkat-20 2M
27

---

Table R²² score of each model on the ADMET downstream regression tasks. The best embedder for each task is highlighted in bold and underlined, and the second best is highlighted in bold.
BA
Absorption
FreeSoly
Absorption
Lipophilicity
Absorption
Solubility
Absorption
Caco2
InfoGraph
0.275 0.254
0,491
0,131
0.639
0,088
0.341
0.055
0.700
ChemBertMLMM- 10M
0.264 0.364
0.543
0.076
0.776
0:088
0.363
0063
0.774
0.07
FRAD QM9())
0.332
0.564 0.051
0.686
0.483
0.29
0.758 =
ChemGPT-1.2
0.340 0.329
0.567 0.079
0,831
0.048
0.487 0.000
0.798 0.009
GROVER
0.350
0.575 0.058
0.708
0.470
0.733
GraphLog(
0.350
0.545 + 0.155
0.811
0.486 0.087
0.765 =
GraphCL,0
0.355+ 0.202
0.559 + 0.051
0.764- 0,018
0,467 0067
0.745 + 0.021
GraphMVp(O)
0.397 0.330
0.592 0,064
0.861 0.036
0.590 0064
0.791 = 0.009
MolR gat
0.394
0.65 + 0.804
0.518 0.077
0.822
ThreeDInfomax
0.425 0.322
0.700
0.138
0.852
0,624 0.011
0.848 = 0.004
ChemBerMMTR-77M'
0.459 0,308
0.725 + 0.027
0.874
0.670 0.025
0.839 0.007
MSE
0.420#: 0.299
0.642
0,060
0,851 0,063
0.605 0.021
0.792
Cosine
0.460::
0.699
0.056
0.893
0.721
0.028
0.815 = 0.009
student-250k
0.482++
0.298
0.712 + 0.040
0.900-
n.35
0.742 + 0.019
0.823 + 0.007
student-2M
0.476
0.301
0.687
0.045
0.878+
0.036
0.739 + 0.021
0.822
0.005
Distribution
Distribution
Cl Clarancee (H)
Cl Clearance (M)
Half Hallliite Life
Excretion
LD500
Tox
PPBR
VDss
Excretion
Excrction
InfoGraph
0.093:1 6073
0.018
0.190
-0.048
0.133
0.070 0.4s
-0.011 1: 0.61
0.458 0,39
ChemBertMLM.
0.112:: 0:35
0.066 0.01
-0.185
0.122
0.040
0.178
-0.240 0.27
0.390 0.44
FRAD (M9
0,180 0031
-0.004 0.000
0.006 0,95
0.124 0x59
0.104
0.415 0/09
ChemGPT-
0,1 i 61066
0.046 0.173
-0.018 o.m
0.117 0199
-0.047 +t:
0.442-t :.442:t0omm 0,03
GROVER
0,185:t :::5:::000 01056
0.186
-0.034 0.095
0.197 om12
0.035 0.167
0.447 11: 0,05s
GraphLog(9)
0.240
0.202:t
-0.094:t
0.068
0.018
0,457:1:
GraphCL (e)
0.237:1 0.488
0.158
-0.022-: 0.127
0.123 0.108
0.007 0.165
0,508 0,026
(r)phMVp(e)
0.327 0.36
0.168 0.881
-0.009 0.135
0.144 0.21
-0.017= 0.226
0.527 0,042
MolR gat
0.28411 0,998
0.155 0.180
-0.0241: 0,093
0.174 0
0.059 0.232
0.496.1 0,940
IhreeDDnfomax
0.314 0053
0.152.t
0.071 +: 0,349
0.195 0.114
-0.004 0.264
0.500 0,940
C
0.393
0.138
0.011 +
0.250
0.196
0.491
MSE
0.362-t: 0/077
0.135 +: 0.87
0.034:t 0.087
0.244 0103
0.060 0.116
0.470:t 0,00
Cosine
0.382:1: 0/032
0.108 0.034
0.079 11: 0.002
0.275 0x054
0.111 0.158
0,515 0.09
student-250k
0.390
0.125
0.113
0.283
0.207
0.529
student-2M
0.389
0.138
0.069
0.348
0.144
0.543 6.41
28

---

12-
24
Averaage Bank
10
12
<1k
1k-10k
>10k
12-
24
Average Rank
10
20
50-40%
40-10%
10.0%
Class imbalance (only for ciassification)
Model
student 2M
MeiR_gat
FRAD_ FRAD_OM9
GROVER
ChemBertMTR- 77M
GraphMVvp3
GraphLag
ChemGGPT-1228
GraphCl'0
ChemBertMMLM-
InfoGraph
Figure 10: Average ranking of our models when grouping tasks based on the number of samples in task and the class imbalance (for classification tasks).
C Natural Language Processing
C.1 Training set and hyperparameters
C.1.1 Training set
Dataset sources. We experiments with two training home-made dataset combining different training sets of different embedders and the GISTEmbed dataset. We provide the statistics of our dataset in Tab. 9 and the GISTEmbed dataset is described in (Solatorio 2024).
Dataset construction. Most embedding datasets consists positive and negative samples, ques- tions and answers, or sentences and their labels. We flattened the datasets to have only column of sentences and deduplicated the dataset. For the MEDI dataset for example, given query, positive and negative samples we build dataset with three times the number of entries, one for each sentence. We then deduplicated the dataset to remove any duplicate entries.
Table 9; Number of samples in each dataset
Number samples
https: http:///iggattcccc///rsssssssssr
190872
https:
3264474
https:/
203755
triplets
328188
356409
223901
120000
67349
416809
https:
100304
45000
25000
6342061
29

---

Table 10: Performance of the teachers we used and of the base students. Experiments with single
teacher distillation were performed with the stronger teacher SFR- PFR-Embedding- R.
Counterfactall
MTOPDOROMn
Amson
Antazon
Polority
Amazon
Reviets
Banking?7
Emnation
Massive
Mussive
Toxts
Tweet
1ndd
SPR-Pabbstding22.R
96.8
98.6
Teacher
stella, en en_400M_y
435.0
92.4
97.2
59.5
89.3
78.8
96.3
98.8
92.3
85.2
89.5
$6.9
73.6
86.7
UAE- VI
335,0
75.5
92.8
48.3
87.7
51.8
92.8
94.0
76.9
76.5
79.8
71.1
59.8
75.6
stmodel_et
335.0
54.9
73.5
embed in
46.5
Student (Base)
snow flae arctic embed
33.0
71.2
78.8
38.3
79.1
45.8
69.5
90.9
58.6
64.8
70.0
62.0
53.9
65.7
show flake- www.rrcccceceed-....
23.0
65.1
70.0
35.3
76.4
41.8
62.8
90.8
58.0
63.5
71.0
64.3
56.2
62.9
C.1.2 Teachers and based students performance
Teachers. We selected 4 teachers from the MTEB benchmark (Muennighoft et.l. 2023) as teachers for our distillation method. We provide the list of the teachers and their performance in Tab. 10 The teachers of widely different sizes (335M, 435M and 7B) have display strong but different performances on the MTEB benchmark.
C.1.3 Single teacher distillation
Single teacher Multi-Teachers. Since some teach- ers yield strong performance on their own, distilling only from the strongest could yield similar results the multi- teacher setting involving weaker teachers. We applied our method in a single-teacher setting using the strongest teacher by far (SF-Embeddings-R 2) as teacher and com- pared the results the multi-teacher setting. Consistently with results in computer vision and molecular representa- tions, we found that adding weaker teachers did improve our results (Figure IT). supporting our hypothesis that en- forcing reconstruction capabilities for a diversity of models indeed leads to more informative representations.
80
Wherels Large-V1
75
@mesgpp model 85
Accuracy
Baselines
65
MSE NLL-Single NLL
Training method
C.1.4 Hyperparameters
Figure 11: Comparison of distilled small model with the performance of the initial backbone, baselines in the MTEB, with our teachers' performance.
Training hyperparameters. We trained our models us- ing the Adam optimizer with constant learning rate of 5.10-5 and an effective batch size of 16 for all our models.
C.2 Detailed evaluation results
We ran different parts of the MTEB benchmarks and report the overall results for all our models in this section.
C.2.1 Evaluation on classification tasks
Small models' performance. Tab. 11 and Tab. 12 we provide the classification accuracy of distilled models on the MTEB classification benchmark for smaller models xs (22M) and s (33M). Our smallest model significantly improves SOTA performance for models of its size by increasing the average score of 2 points compared to the previous best model.
C.2.2 Evaluation similarity and clustering tasks
Limited structure of our embedding spaces. Our method only seeks to pack as much (statistical) information into the embeddings possible without any constraints on the underlying structure of the embedding space. therefore not surprising that methods that relies on metrics on the embedding space such as similarity tasks do not perform well the classification tasks. However,
30

---

Table 11: Performance of our distilled models compared to models of similar sizes 16M to 30M
parameters from the MTEB Benchmark on classification tasks.
Countefacuu
MTOPDonadn
MTOPIntent
Comsrratioon
Amazon
Polurity
Amazon
Roviews
Banking77
Emation
Massive
Mussive
Secmario
Toxic
Twrest
Soatiment
Extraction
Task
Size
Avg.
Model
GIST
23M
72.9
87.2
42.6
84.2
52.1
78.5
94.8
77.7
73.2
76.7
72.9
59.9
72.7
Bullassur
17M
71.9
78.8
39.3
80.6
44.8
71.5
90.8
68.7
68.8
73.8
66.3
59.5
67.9
leysur
86.7
70.2
Squintle
16M
69.6
82.1
41.9
67.1
45.8
75.0
87.3
54.7
61.5
67.0
64.3
61.
64.9
Venusaur
16M
73.2
80.0
39.7
78.0
44,4
73.0
89.9
71.0
67.8
72.4
64.4
39.7
67.8
17M
8200
46.8
54,9
65.7
gte-micro
17M
68.8
77.1
40.9
69
46.2
62.2
86.7
49.7
59.0
66.6
66.1
60.8
62.8
МТЕB
gie tie-micro-v4
mic-micro-v2
17M
19M
71.4
77.7
80.0
39.0
80.4
44,9
44.5
70.6
90.5
67.5
68.5
73.5
74,2
66.7
59.3
67.5
snowllake eretic-embeed-x xs
23M
65.1
70.0
35.3
76.4
41.8
62.8
90.8
58,0
63.5
71.0
64.3
56.2
62.9
nge-miera
17M
75.4
80.6
42.5
90.2
68.0
67.8
73.0
69.2
56.7
66.3
bge-micro-v2
17M
67.8
79,8
81.2
44.5
90.7
68.1
68.6
73,9
70.2
57.6
68.0
gle tiny
23M
71.8
86.6
42.6
81.7
44.7
80.5
91.8
69.9
70.1
74,9
71.0
58.6
70.3
slx-v0.1
64.3
62.6
multi qu Li-MiniLL--L--oss1
23M
61.8
62.4
29.6
78.6
39.6
61.2
90.0
59.6
66.8
73,8
65.1
$1.6
61.7
all Minil M-1.6-v2
23M
63.6
64.3
30.9
80.0
40.8
61.8
91.7
61.5
66.9
73.8
62.1
54.0
62.6
Studenl-xs
75.4
NLE
Stndent-xs
23M
76.5
84.9
42.4
85.8
58.0
81
95.2
79.9
75.8
80.4
68.1
60.1
74.0
Table 12: Performance of distilled models compared models of similar sizes 30M to 50M
parameters from the MTEB Benchmark on classification tasks.
Counteractual
MTOPDonmain
Amaron
Amaron
Polarity
Amazon
Reviews
Banking77
Emocion
MTOPInteaa
Masive
Inint
Masive
Scaariio
Toxic
Conversation
Tieeer
Seniment
Extraction
Task
Size
Avg.
Model
1ge-smalllen-v1.55
GIST
33M
75.3
93.2
49.7
86.7
55.9
89.5
95.5
79.1
75.5
79.2
72.8
61.0
76.1
NoInstruct
33M
75.8
93.3
50.0
86.4
55.1
90.2
95.3
79.6
76.0
79.3
69.4
61.3
76.0
snow fleee---ttti-emmmd--
33M
71.2
78.8
38.3
79.1
45.8
69.5
90.9
58.6
70.0
62.0
65.7
bge-small-4096
35M
68.8
$1.3
38.6
80.0
40.1
S0.1
90.4
66.5
67.6
73.5
69.3
$7.6
67.8
LASER
43M
76.8
61.0
28.7
$7.8
24.8
57.6
75.4
49.5
47.9
55.9
54.0
53.2
c5-small
33M
76.2
87.5
42.6
81.9
46.9
75.5
92.0
73.2
72.2
75,8
72.8
71.7
c5-small-v2
33M
77.6
91.3
45.9
81.6
47.1
86.0
92.7
72.6
71.6
76.4
71.1
61.5
72.9
jist-cmbbedding--sq---
35M
64.3
74.6
jina enweeeiingg----smalllll
33M
71.4
82.9
40.9
78.2
44.0
73.6
94.0
72.5
67.6
69.8
71.5
68.8
Mi-.Minil
33M
65.3
63.0
30.8
80.4
41.2
59.8
91.9
62.8
67.2
74.6
67.5
54.2
63.2
gie-small
33M
70.3
72.3
MSE
Student-s
33M
72.6
90.3
44.3
812
56.5
88.8
949
77.2
754
812
61.9
604
74.2
NLL
Student-s
33M
27.3
89.2
43.8
86.7
580
58.1
95.5
819
76.7
80.
66.1
60.6
75.4
our embedder are still competitive on these tasks achieving average performance for their respective size categories.
we show that our very small model actually outperforms baselines and sits on the pareto frontier for clustering tasks. This is a surprising result as we did not optimize our models for clustering tasks and the embeddings are not designed to have a meaningful structure.
Clustering with very small model. In Tab. 15
C.2.3 Analysis and compare with the most recent embedders
The results at Tab. 22 show that our medium model (STUDENT-M-NLL. 109M) achieves an av- erage of 80.2 the selected MTEB classification tasks, tracking much larger recent embedders within single-digit margins. In particular, QWEN3-EEMEDDDING--. point gain ~ 5.5x the parameters. Substantially larger improvements appear only beyond ~1B parameters (JASPER_ES EN_ VISION OWEN3-EMBEDDING-AA
(595M) reaches 85.8, a +5.6
1.0B: 90.3; B.ELLA.EN.I.5B... 1.5B: 89,4; 4.0B: 89.8). Overall, the 109M model delivers competitive accuracy relative to 4-6x larger embedders, supporting our claim that multi-teacher distillation yields high information density compact scales.
31

---

Table 13: Performance of our distilled models compared to models of similar sizes 100M to 120M
parameters from the MTEB Benchmark on classification tasks.
MTOPDeesinn
MTOPIntent
Coneec colions
Amazonn
Polarnt
Bunkingh?
Enostoon
Mosihve
Inxes
Mossive
Sceiani
Toxic
Tweet
Extraction
Indh
Size
Avg.
Model
hgo--bossoon-11.5
109M
93.4
51.9
76.9
76.2
80.2
71.6
GIST
105M
76.0
93.5
50.5
87.3
54.7
89.1
95.3
78.1
76.0
79.6
72.4
59.3
76.0
bilingual-zmbeen con--dddinnnngenee small
118M
74.3
82.2
40.2
S013
40.8
73.7
39.7
66..5
68.9
14.5
62.5
59.6
67.8
walliiinguul-----mallll
118M
73.8
88.7
44.7
79.4
42.5
80.8
91.1
71.1
70.3
74.5
69.4
62.6
70.7
105M
80.3
68.2
aretic
109M
73.1
mi-nlp-ebeer html
100
74.2
61.9
32.1
82.0
46.6
65.0
93.2
71.1
68.5
75.0
68.2
$3.6
65.9
base- 4k
112M
77.8
92.8
46.7
83.5
47.0
86.2
93.7
75.3
73.0
77.7
72.1
60.4
73.8
instructor base
110M
86.2
$8.4
14.6
77.0
$1.8
81.2
93.7
70.3
67.5
72.6
7L8
63.3
72.4
bert hase-uncosed
110M
14.2
71.3
33.6
63.4
35.5
65.3
82.6
68.1
59.9
64.3
70.0
51.8
61.7
base
109M
$8.0
83.3
49.4
76.0
74.8
72.2
76.8
74.1
c5-base-22
110M
75.0
jina- jina-embeddding-- b-a-g--------
110M
66.7
84.1
71.1
76.2
66.1
contriever- base msmanoa
110M
72.2
68.6
37,4
800
44.8
67.0
93.2
69.3
67.8
76.0
67.8
56.1
66.7
aup-simooo-B sarr---oo---mcsll
110M
75.8
82.5
39.6
75.8
44.8
73.5
84.3
63.1
66.0
70.8
72.0
59.7
67.5
unsup simuse bert base uncased
110M
67.1
74.5
33.9
73.5
42.2
68.6
$1.7
59.2
59.8
66.2
68.8
53.4
62.5
all mlllmmentt-te hose-x2
110M
65.0
67.1
31.4
81.7
42.2
71.2
91.9
68.3
69.8
75.7
61.0
55.0
65.0
allenai- specter
110M
26.3
66.7
24.8
51.7
58.6
57.4
52.4
gtr-t5 buse
110M
42.2
75,4
mamurrc-- bett-co-comdmmoe
110M
66.1
66.9
34.9
82.3
41.9
60.2
91.3
71.1
704
73.7
61.0
55.7
64.7
paraphrase multilingual MiniLM-L.12-2
118M
71.5
09.2
35.1
79.8
42.3
60.5
87.0
85.5
66.9
71.5
60.1
56.1
63.8
sentence 15 base
110M
75.8
$5.1
44.9
76.5
51.4
77.3
90.3
63.3
69.7
72.3
68.2
62.7
69.8
118
71.0
66.1
33.1
78.1
43.4
59.4
31.0
62.8
63.8
67.0
66.0
55.2
62.2
BEKT
109M
75.5
45.2
64.5
20.6
109M
48.6
ALL 8627767
118M
364
22.8
50.8
20.7
54.9
40.8
MSE
Sindel-m
T0RM
76.5
89.]
14.7
57.2
606
58.0
95.7
31.6
77.7
82.2
67.3
60.5
76.0
NLL
Sunknt-m
100
79.5
80.5
458
88.0
39.7
88.3
96.2
839
78.6
12.7
671
61.3
76.7
Table 14: Performance of our distilled models compared to models of similar sizes 200M to 420M
parameters from the MTEB Benchmark on classification tasks.
Amazon
MTOPDumain
MTOPimeent
Copv ensvensiionss
Ausa2o
Polunit
Roviews
BankingT
Emosion
Massive
Intent
Maxive
Scenarto
Tweet
Sentiment
Extrction
Inds
Ang.
Model
gle multilinuual base
305M
76.0
50.7
43.6
85.4
48.0
75,9
92.5
72.6
72.1
76.3
71.0
$7.6
70.9
hge- 1oe-Iargo-en-vl..5
335M
75.8
92.4
48.2
87.8
51.5
92.8
94.6
79.5
77.6
80.5
70.9
59.9
76.0
GIST
335M
75.6
93.4
88.1
54.7
91.2
95.2
78.2
762
79.3
26.0
MUG-B--16
73.9
shinguul-mmbolldingg---
278M
77.4
29.5
16.1
78.5
47.1
87.4
92.9
64.8
75.2
63.1
62.5
71.1
snowdhake arctic eembed-l
354M
74.8
78.4
36.7
80.1
46.5
72.9
92.6
64.5
65.8
71.1
64.7
56.7
67.1
LAB Large V1
335M
75.5
92.8
45.3
87.7
51.8
92.8
94.0
76.9
76.5
79.8
71.1
59.8
75.6
embbedeer 100p
278M
67.1
53.2
62.7
43.5
67.3
9L.
74.7
71.8
77.8
67.5
55.6
67.0
instructor-lagge
335M
e5-ler
82.1
92.9
73.2
MTEB
c5 35-hare--22 multilingual-S bause model 25 jina embedking à eennv cmbeer-v1
335M
79.2
93.8
48.6
84.5
49.5
91.7
91.6
77.1
73.8
78.1
70.9
60.9
75.2
278M
77.4
91.8
47.5
73.5
45.7
84.3
90.9
61.6
65.7
71.6
61.3
62.8
69.8
335M
70.8
91.8
48.9
84.6
54.9
93.1
93.6
66.0
73.5
77,4
71.2
61.5
74.0
335M
85.3
45.8
92.8
76.1
72.7
53.2
67.5
77.4
mn mabai-emmed--- 1
335M
74.8
93.3
46.3
W6.7
49.3
90.4
93.1
73.2
73.9
78.2
71.5
592
74.1
mahii-enhod-brgg--
115M
75.D
93.8
40.3
87.8
50.9
92.8
940
76.8
76-
71.5
59.7
7566
paraphrase multilingual -mpnetbbase-22 base-v2
278M
75.8
76.1
38.5
81.1
45.8
61.5
89.2
68.7
69.
15.3
71.0
59.0
67.9
gle-large
335M
72.6
92.5
4991
86.1
47.9
88.5
93.5
733
718
76.8
7066
5666
73.3
Made--mbed
335M
Snident-
1H5M
0010
82.7
19.
763
81.
NEL
Stndmi-]
335M
31.3
38.1
43.9
86.9
60.4
88.2
95.6
83.2
77.5
81.4
67.7
62.2
76.5
32

---

Table 15: Performance of our distilled models compared of models of similar sizes 16M to 30M
parameters from the MTEB Benchmark on clustering tasks.
Newsgroups
Clustering
Clostering
Clustering
Clustering
Exchange
Clustering
Exchunge
Clustering
Twenty
Clustering
S2S
Reddit
Reddit
Stack
P2P
Stack
Task
Size
Avg.
Model
Bulbasaur
17M
40.3
31.1
51.4
45.9
30.7
52.2
39.4
41.6
Iyysaur
23M
46.4
35.4
56.0
47.5
33.6
53.9
40.8
44,8
Squirtle
16M
33.0
24.7
43.7
31.4
29.2
28.2
Venusaur
16M
31.8
21.1
44.1
26.7
27.5
32.8
26.1
30.0
Wartortle
17M
35.8
27.3
46.1
35.9
29.9
45.3
31.7
36.0
gte-miero
17M
35.2
31.1
47.9
45.6
30.1
52.6
40.8
gte-micro-v4
19M
42.9
32.5
53.6
48.3
31.9
41.4
43.6
MТEB
snow
23M
43.5
32.1
$7.8
48.3
34.6
57.5
36.3
44.3
bge-micut
17M
44.6
34.5
54.5
45.3
34.7
39.4
43.7
bge-micro-v2
17M
44.5
33.2
55.2
34.1
54.5
40.2
gte-tiny
23M
46.6
36,0
56.5
S0.2
35.7
57.5
43.3
46.6
LEST-all-Minl----LE
23M
45.3
35.5
48.7
44.1
33.9
53.1
41.1
slx-v0.1
23M
46.5
37.7
54.8
34.2
46.5
46.2
23M
37.8
27.7
51.0
46.3
33.4
48.1
40.8
40.7
all-Minil M-L.6v2
23M
46.5
37.9
54.8
50.7
34.3
53.1
46.5
46.3
rubert-tiny-uut
29M
24.8
16.7
40.5
26.3
28.0
MSE
Student-Xs
23M
42.4
30,9
55.2
49.2
32.7
53.5
41.9
43.7
NLL
Student-xs
23M
45.2
33.9
SK.1
52.1
33.1
39.9
44.3
46.7
Table 16: Performance of distilled models compared of models of similar sizes 30M to 50M
parameters from the MTEB Benchmark on clustering tasks.
Clustering
P2P
Antiv
Flustering
525
Reddit
Clussering
P2P
Reddit
Clostering
Stack
Exchange
Clustering
P2P
Exchange
Clossering
Twenty
Newsgroups
Clustering
Task Model
Size
Avg.
bge enall-en-v1.5
33M
47.4
40.0
60.6
$2.3
35.3
60.8
48.5
49.3
www.flik--nccii-embbee.d
33M
44.9
35.9
60.5
50.5
34.0
60.7
38.3
46.4
bgo-small-4006
35M
43.9
29.6
54.3
43.7
33.3
51.8
36.6
41.9
MUST-ssall-Enmeeddiin
33M
47.6
39.9
60.6
55.5
36.2
61.9
50.0
S0.2
Doonstruct-small-Emmhddd
47.8
40.1
55.4
36,6
62.0
50.4
MTEB
e5-small
33M
44.1
37.1
57.2
43.3
30.8
59.6
37.6
44.3
e5-smull-v2 jina-
33M
42.1
34.8
59.7
45.7
32.0
58.5
41.1
44.8
34.2
38.0
31.5
36.9
lina-embeeddingg-2--mllll
33M
44,0
35.2
57.1
49.3
34.4
55.4
41.6
45.3
all-MiniLM-Li2--2
46.1
37.5
51.2
33.1
53.0
47.5
gte-small
33M
47.9
40.3
61.4
55.6
36.3
62.6
50.0
50.6
MSE
Student-s
33M
43.1
33.3
37.1
50.8
32.3
55.7
42.8
45.0
NLL
Student-s
45.9
35.2
32.3
47.4
33

---

Table 17: Performance of our distilled models compared of models of similar sizes 100M to 120M
parameters from the MTEB Benchmark clustering tasks.
Azxiv
Clusering
p2p
Auxiv
Cluseriig
Reddit
Cluseering
ReddH
Clostcning
Exchange
Clustering
PIP
Stack
Exchange
Clastering
Tweuty
Newagoups
Clustering
Task
Size
Avg.
Model
hge-base-en-vle
109M
48.8
42.8
62.7
56.6
66.1
50.8
51.8
bilingual-on satall---meeddiigsmaaaa
118M
41.8
31.6
58.4
47.4
33.6
52.5
40.5
43.7
multilingua -o5-small
118M
39.2
30.8
59.0
39.1
32.1
53,5
33.2
41.0
47.5
snowllake arctic eeteee-eeeedddee m-v1.5
109M
45.0
34,1
61.8
51.9
33.8
61.2
38.1
46.6
GIS T- T-ET---ddiing--
109M
48.3
42.7
62.4
59.1
35.6
66.1
52.2
$2.4
ml nip himll
110M
35.3
23.2
51.9
38.7
28.7
42.7
27.8
35.5
25-base-48
112M
46.1
39.7
63.4
56.2
32.5
65.2
48.2
50.2
instructor-aase
110M
39.7
29.2
63.2
59.3
35.3
65.0
51.3
49.0
hert-base-unenne
110M
35.2
27.5
43.3
27.2
26.6
43.6
23.4
32.4
c5-base
45-base-v2
110M
46.1
39.7
63.2
56.5
33.0
64.6
49.9
50.4
MTTBB
jina emberkling- i
110M
39.2
29.1
52.5
42.9
48.1
38.1
40.2
contrever base mimarcu
110M
42.6
32.3
$7.6
54.9
32.2
63.1
46.8
47.1
sup-simese bert-
unsup cut--bae-unncse)
110M
35.2
27.5
47.7
40.2
29.4
47,5
34,9
37.5
24.7
all-m all-mpnet- 1l-mpnet-base-v22
110M
48.4
39.7
56.8
54.8
34.3
53.8
49.7
48.2
allenai-specte
35.3
39,0
33.4
gtr-t5-base
11M
35.5
27.2
58.5
56.1
33.0
64.2
46.7
45.9
hert co-denddensee
110M
36.9
29.0
53.5
48.0
30.5
59.5
38.7
42.3
paraphrase-munn saseeeemmlillinnull www.ttlingul-MMiiiMMi..... MiniLMM-L122222
38.3
31.6
50.1
42.6
40.5
sentence-tF -S-ease
110M
39.3
27.3
59.7
$2.9
35.7
63.1
48.1
46.6
muliilinuual
Angle BERT
109M
35.3
27.7
46.0
40.3
28.9
48.3
33.1
37.1
62.6
59.3
52.3
52.6
ALL 862873
118M
14.8
12.2
27.1
18.4
27.3
23.7
20.2
20.5
MSE
Student-m
10PM
46.3
37.1
60.4
34,3
33.4
62.0
46.1
48.6
NLL
Student-m
38.7
56.3
64.7
Table 18: Performance of distilled models compared of models of similar sizes 16M to 30M parameters from the MTEB Benchmark on STS tasks.
TTBBenchmark
BIOSSES
SICK-R
STS12
STS13
STS14
9755
97500
51577
S952
Task
Size
Avg.
Model
Bulbasaur
17M
85.0
76.0
69.5
81.0
77.1
85.4
82.3
88.0
64.1
83.3
79.2
Ivysaur
23M
87.3
75.6
68.6
80.5
77.6
86.2
82.8
88.6
67.4
84.2
79.9
Squirtle
16M
71.8
77.3
70.2
78.4
74.8
78.3
85.8
61.2
79.2
75.9
Venusaur
16M
77.6
74.7
54.4
74.2
70.0
75.7
73.7
84.8
62.6
76.7
72.4
Wartortle
17M
78.2
75.2
79.3
76.6
84.7
81.4
86.6
63.4
81.8
snowffake-arttic-embell
23M
84.0
69.3
65.9
77.9
72.8
83.5
80.6
84.5
66.3
79.2
76.4
MTEB
bge-micro
17M
83.4
72.4
71.9
80.9
76.6
84.9
80.7
85.6
65.9
81.3
78.4
bge-micro-v2
17M
73.6
71.9
79.8
76.9
81.9
86.8
65.4
82.5
78.7
gte-tiny
23M
86.6
75.8
72.6
82.4
86.5
83.3
88.3
66.7
84.4
80.5
GIST- MElT--ll-Mini..... Minil.M--.6--- M--16-v2
23M
81.3
79.1
75.0
83.3
78.6
87.0
83.0
87.4
68.1
84.4
80.7
23M
79.8
70.0
64.4
76.4
69.3
79.6
81.2
65.5
76.0
74.2
ul-MMiiLLM-L6-12
23M
72.4
80.6
75.6
87.6
67.2
82.0
78.9
MSE
Student-xs
23M
76.8
79.2
72.2
X0.3
75.9
85.0
83.0
87.1
66.4
82.9
78.9
NLL
Student-xs
23M
78.8
77.8
71.6
80.2
77.0
82.8
89.3
65.8
83.5
79.3
34

---

Table 19: Performance of our distilled models compared of models of similar sizes 30M to 50M parameters from the MTEB Benchmark on STS tasks.
STSBenchmark
BIOSSES
SICK-R
STS13
S
STS15
S1S16
51517
S1S21
Task
Size
Avg.
Model
bge-small-en-v1.5
33M
79.4
77.4
84.9
81.6
33M
86.3
68.8
79.6
69.5
78.4
bge-small-4096
35M
81.6
74.2
72.2
80.5
76.2
85.2
81.9
86.6
65.5
81.9
78.6
GIST-small Embedding-vv
75.6
86.3
82.3
83.0
Nolnstruct-small-Ennne -
33M
87.2
80.3
75.8
86.1
82.3
88.9
85.2
88.7
68.5
87.0
83.0
MTEB
e5-small
33M
84.2
78.9
75.2
81.8
78.5
84.6
87.9
86.4
80.9
e5-small--o v2
33M
79.4
78.5
76.2
82.4
79.0
87.8
83.8
87.7
63.1
86.0
80.4
jina- jun--enbedding--enee
35M
83.0
76.3
74.3
78.5
73.8
83.7
80.0
87.5
64.2
79.2
78.1
jina embeddings 22-small-en
all-Minil M-1.12-v2
33M
83.6
79.3
73.1
82.1
76.7
85.6
88.6
65.7
81.1
79.8
gte-small
33M
58.2
77.9
75.
85.1
83.9
87.6
08.0
MSF
Student-s
31
78.9
70.6
75.4
84.1
81.8
66.6
78.6
NEL
Student-s
33M
81.3
79.3
73.0
81.4
78.2
86.3
84.2
90.0
66.0
84.8
80.5
Table 20: Performance of our distilled models compared of models of similar sizes 100M 120M parameters from the MTEB Benchmark on STS tasks.
BIOSSES
SICK-R
STS12
STS13
57514
S1S15
STSI6
S1S17
5753
Task
Size
Avg.
Model
age-basc-on-v1.5
109M
86.9
80.3
78.0
84.2
82.3
88.0
85.5
86.4
66.0
86.4
82.4
dlingual-emhedding
83.9
67.2
86.1
81.9
multilingull gual e5 small
118M
82.3
77.5
76.6
77.0
75.5
87.1
83.6
86.4
60.9
84.0
79.1
lake wwwtir--eeeeeeeee.....e.e
snowflake arctic embed m-vi.5
109M
86.4
69.9
61.8
82.7
69.0
75.5
77.3
75.0
69.1
69.7
73.6
GIST Embedding-vo
109M
S8.0
81.3
76.2
87.8
83.4
89.4
85.3
88.6
67.8
87.3
83.5
mll-nlp-elserhhmml
110M
83.5
68.8
64.8
80.1
75.0
83.7
80.5
85.7
67.5
79.3
76.9
c5-base-4k
112M
81 4
78.3
75.8
83.6
80.0
88.8
84.5
87.6
64.1
86.5
81.0
instructor
bert-base-nsased
110M
54.7
$8.6
30.9
39.9
47.7
60.3
63.7
64.1
56.4
47.3
54.4
ea-ease
109M
79.7
74.2
83.3
78.5
88.5
84.2
87.2
62.9
86.2
81.0
base v2
110M
81.4
78.3
75.8
83.6
80.0
88
84.5
87.6
64.1
86.5
81.0
MTEB
jina mon--mmeedding. b-e---------- contriever hase msmmarco
110M
83.6
79.1
75.
80.9
76.
85.5
81.2
89.0
66.
82.6
79.9
79.7
sup simese bert base uncased
110M
68.4
80.8
75.3
$4.7
80.2
85.4
80.8
89.4
62.0)
84.2
79.1
unsup-simese-eee basc-uncased
79.7
mpmet base-v2
110M
80.4
80.6
72.6
83..5
78.0
85.7
80.0
90.6
68.0
83.4
80.3
allenai- specter
110M
65.0
56.4
62.5
58.7
54.9
62.5
64.3
69.6
55.1
61.3
61.0
gtr-t--bacc
79.0
71.5
68.6
77..1
74.6
84.8
81.6
85.8
66.2
79.6
77.1
msmarco-hert www.-eeer---ooeeeee co-condensor
H10
77.3
72.0
68.2
80.4
74.0
82.6
79.8
85.9
67.5
77.0
76.5
pongbrras-muttilingu--Miii
85.8
sentence-tS-hen -arencce-----beee
110M
75.9
80.2
78.0
85.8
82.2
87.5
84.0
89.6
62.7
85.5
81.1
text2ver: base multilingual
118M
66.2
80.0
80.9
82.9
87.4
88.3
81.6
85.8
63.0
86.5
80.2
gie- buse
A_
109M
118M
21.3
87.6
48.5
789
55.6
75.7
85.7
18.4
81.5
28.8
293
88.8
83.8
3000
67.5
0113
444
17.3
244
L5.7
82.3
39.1
86.6
85.
66.4
NLL
Student-m
109M
83.2
80.
13.2
834
80.4
88.3
86.0
393
66.2
864
82
35

---

Table 21: Performance of our distilled models compared of models of similar sizes 200M 400M
parameters from the MTEB Benchmark on STS tasks.
STSBenchmard
BIOSSES
SICK-R
STS12
STSIS
STS14
STS15
STS16
51517
S1522
Task
Size
Avg.
gle muliiiingulll-aaee buse
305M
81.2
79.3
77.5
85.5
81.7
89.0
84.3
88.9
67.2
86.5
82.1
bge large-en
79.0
MUG-B-1.6
335M
88.4
83.0
79.2
89.4
84.8
89.5
86.7
89.6
70.3
89.0
85.0
bllingullemllodding-bees
278M
87.1
79.6
83.9
89.9
84.9
88.7
64.3
87.4
83.0
snowflake -rcharrrtiic-ce--11111----------
334M
86.3
69.3
67.8
77.5
69.8
80.2
77.9
82.3
68.0
75.7
75.5
UAE Large VI
335M
86.1
82.6
79.1
89.6
85.0
89.5
86.6
89.0
68.8
89.1
84.5
CIST--urge-BBmeeddee
335M
89.2
82.8
77.1
89.3
83.8
89.7
86.4
89.7
69.6
88.3
84.6
embedder-1000
278M
75.3
80.9
77.0
82.6
77.8
85.9
80.7
89.0
68.3
84.2
80.2
instructor-lagee
76.3
89 0
90.3
86.9
MTEB
e5. large
335M
84.7
80.5
75.9
8.5.2
80.5
88.8
85.3
89.4
63.0
87.2
82.1
c5-large-22
77.0
multalingual-cS-usse
278M
85.0
78.5
76.77
78.0
76.6
88.
84.3
87.8
62.3
85.6
80.3
sf
335M
86.8
82.3
77.6
88.0
83.8
88.5
86.5
88.7
88,3
83.8
jina wwweeeeeee----eeeeee --en--l
335M
84.4
79.2
74.5
83.2
78.1
86.9
83.7
90.2
64.9
84.6
81.0
ember-vl
335M
85.8
81.8
78.5
86.6
83.1
88.4
86.8
87.9
66.8
87.8
83.3
mxbai-cmbed woweemmn---dn------------------
335M
88.1
82.0
78.8
90.4
85.5
90.0
87.4
88.8
89.2
84.9
mxhai-embed-llagee
335M
88.4
82.9
78.8
90.3
85.5
89.6
86.6
89.5
69.3
89.1
85.0
paraphrase wwaeeeemmmlliiiiiin linginul wwg.gnalmmmee----eeee mpper--r----ee--eeee
77.9
87.5
87.0
80.8
gle large
335M
88.7
79.8
76.8
88.1
82.7
88.9
84.2
88.5
69.7
86.1
83.3
blade-embed
Student-] MSE
335M
79.1
80.6
73.7
82.1
78.
874
84.2
89.1
67.0
83.3
80.7
NLL
Student-1
335M
83.8
74.4
83.0
79.6
88.0
85.2
86.2
81.5
Table 22: Head-to-head comparison on selected MTEB classification tasks, with large embedders (over x5 times the number of parameters).
AmazonCCi
Banking77
MTOP Dom.
Massive Int.
Massive Soen.
Toxic Conv.
Tweet Sent.
IMDB
Model
Size
Avg.
Qwen3- Beenn--Ebbedding-e
4.0B
93.7
86.3
97.2
97.8
85.0
88.8
91.4
78.4
89.8
stella.en.1.SB.v
1.5B
94.1
89.8
96.7
98.7
84.5
89.7
86.8
74.8
89.4
jasper_en
1.0B
93.8
87.2
97.0
99.2
85.3
91.2
91.3
77.2
90.3
Qwen3-Embedee Eeee--mmedddddn....
595M
91.5
81.0
95.4
96.0
80.4
83.6
82.1
76.0
85.8
jina-embeeddins-v3
572M
90.9
84.1
91.9
75.2
84.1
91.3
71.4
84.
snovilakc-arcti--cmhhdd
568M
81.8
72.8
93.5
71.5
76.2
65.9
59.6
73.4
CaL.M-emhed-minng minnnedd-mmi-instt--22
494M
95.3
89.5
95.2
98.9
77.8
86.0
89.3
78.6
88.8
Kal wwweed-mini-innteeee
494M
81.5
84.9
95.0
92.2
69.8
74.2
89.0
76.5
82.9
494M
76.4
79.2
91.6
92.5
70.9
76.1
70.8
62.7
77.5
stella en 400M_x5
94.3
89.3
96.5
98.3
80.5
89.6
84.0
73.6
88.2
Student-m-nll
109M
79.6
88.0
88.3
96.2
78.6
82,7
67.1
61.3
80.2
NLL.
Student-s-nll
32M
77.3
86.7
88.3
95.5
76.7
80.7
66.1
60.6
79.0
36

---

D Vision
D.1 Model architecture
The models we used for vision as teachers and student are presented in Tab. 23 including the number of parameters of each them.
D.2
Training Set
Tab. 24 presents the statistics, i.e. the number of training and testing samples, of the datasets we used for vision.
D.3 Vision Details
Data processing details: We the official train sets of the datasets for the knowledge distillation part. We split the official training part, if there are no official validation sets, to train and validation with 80 and 20 percents of the data, consequently. For the augmentation we used color jitter with brightness, contrast, saturation and hue equal to 0.2, and random horizontal flip (except for the SVHN dataset).
For training the distillation, extract the embeddings of the train set of each dataset, for each teacher and divide the embeddings 80 train set and percent validation set. For the optimizer we use Adam, with learning rate of 0.001, batch size of 128, trained for 50 epochs.
Distillation details:
Down-stream task fine-tuning: For fine-tuning of down-stream tasks, we add classifier on the frozen embedders. We again Adam optimizer for the fine-tuning of downstream tasks. We perform hyperparameter tuning using grid search to optimize the performance of our models. Our search space includes the learning units with values (0, 1 128), and the type of normalization after the fully connected layer, considering (no optimization, batch normalization, layer normalization). The models are trained for maximum of 1000 epochs with a batch size of 128, but we apply early stopping with patience of 20 to prevent over-fitting and reduce unnecessary computation.
with values (le-2, 1e-3), the number of fully connected layer
D.4 Complementary Results
Tab. 25 shows the detailed results of the Vision Transformer teachers and students. The best among students are shown with an underline, showing that on average and most the cases our method improves the baseline. In addition to the main results, we added additional experiments to answer further informative question:
Table 23: Number of parameters for each model (in million parameters)
Model
# Parameters
Swin (Liu et al. 2021b)
87.77M
DINOv2 (Oquab et al.
2023
86.58M
ViT (Dosovitskıy
2021
86.57M
BEiT (Boo et al. 2022
86.53M
PVTv2 Wang al. 2022c)
3.67M
Wee (Zagoruyko & Komodakis
2017
68.88M
DenseNet (Huang et al. 2017)
28.68M
ResNext (ie al. 2001
25,03M
ResNet18 [He et al. 20166)
11.69M
GoogLeNet SZeggddy al.)
2015
6,62M
MNASNet Tan et al. (2019)
4.38M
MobileNet (Sndder et al.
2018)
3.50M
ShuffleNet Ma et al. 20188
2.28M
SqueezeNet [andola et al.
2016
1.25M
37

---

Table 24: Number of classes, training, validation (if any) and testing samples in each vision dataset
Dataset
classes
training samples
validation samples
test samples
CIFARIO (Krizhevsky
2009)
10
50000
10000
STLIO CCatess et al.
2011)
10
5000
8000
SVHN Netzer al.
2011
10
73257
26032
CUB Welineer et al.
2010
200
5,994
5,794
DTD Cimpo1 al. 2014)
47
1880
1880 3333
1880
FGVCAuraali Maji ct al.
2013)
100
3334
3333
Oxford Pets (Parkhi et al.,
2012
37
3680
8041
Food101 (Bossard et al. 2014)
101
750
250
Stanford Cars (Krause ct al.
2013
196
8144
8041
Table 25: Comparison of Vision Transformer teachers, CNN baselines and the VIT student, with their corresponding parameter size, with the underline showing the best students.
Method
Model
# Parameters
CIFARIO
DTD
STL10
SVHN
FGVCAireratt
CUB
Swin
87.77
97.67
76.33
99.60
52.45
87.11
86.57
96.90
71.65
99.40
54.97
41.71
82.67
DINOv2
86.58
98.57
83,30
99.45
63.01
79.40
89.02
BETT
86.53
97.89
77.34
99.60
66.61
55.45
39.52
NoKD
PVTv2
3.67
89.27
65.05
95.80
62.03
38.58
68.97
wide resnet
68.88
85.65
65.37
95.85
57.77
30.82
60.55
densenet
28.68
87.49
67.93
97.11
66.91
46.84
68.62
resnet18
11.69
83.22
61.54
92.98
51.01
36.09
59.89
googlenet
6.62
82.07
66.38
93.95
55.90
35.85
59.09
CompRess
PVTv2
3.67
94.6
52.7
93.5
61.9
32,7
48.8
MSE
PVTv2
3.67
96.1
65.1
96.4
70.3
34,4
67.7
Cusine
PVTv2
3.67
95.89
65.4
96.7
70.7
35.9
67.1
RKD
PVTv2
3.67
87.64
52.23
89.63
61.66
30.54
47.85
CC grbf
PVTv2
3.67
84.07
61.86
93.03
59.96
33.48
57.55
CC bilinear
PVTv2
3.67
92.95
61.22
95.42
63.71
35.16
64.70
NLL
PVTv2
3.67
94.76
65.85
96.45
76.91
48.13
69.37
How will our method work in vision for unseen datasets? Tab. 26 shows the accuracy of our student compared to various distillation baselines: MSE distillation, Cosine distillation, Correlation Congruence (CC rbf and CC dot) Peng et al. (2019), CompRess Abbasi Koohpayegani et al. (2020) and relational KD Park et al. (019b).
for three unseen datasets. As we can see, our method improved the baselines considerably for unseen datasets.
How our method works for a setting with diverse teachers specialized in different task, and if will be able to avoid confficts? We evaluated the student model's classification performance using three specialized vision teachers: ViT (classification), DETR ( (Carion et al. 2020 object detection),
Table 26: Comparison of ViT student of our method (NLL), and various distillation baselines for the unseen datasets.
Method
Oxford Pets
Food101
Stanford Cars
CompRess
70.23
45.48
19.43
MSE
85,58
58.04
31,96
Cosine
84.38
56.37
30.92
RKD
69.99
43.48
18.24
CC rbf
85.09
58.47
30.08
CC dot
67.42
45.93
20.88
NLL
87.46
62.62
41.29
38

---

and SegFormer ( (Xie et al., 2021). segmentation). We also included DINOv2, embedding model known for strong performance across multiple benchmarks. As shown Tab. 27 adding DETR or SegFormer alongside ViT did not significantly improve or degrade classification performance compared to using ViT alone. This suggests that while task-specific teachers may offer limited benefit outside their domain, they do not negatively impact the student's learning.
To further validate this, we incorporated DINOv2 into the teacher set TTb. 28 This addition improved overall performance, while the inclusion of DETR and SegFormer continued to have minimal effect, confirming that our earlier observations hold even in a more competitive setting with strong general-purpose purpose teacher. These results are consistent with [Sec. 5.2 and FFigure C.I.3) where we observe that adding teachers typically boosts student performance. In molecular and text domains, where all teachers are general-purpost purpose embedders, improvements are more uniform. However, in vision tasks, specialized teachers contribute gains primarily in their area of expertise, yet without harming performance elsewhere. Overall, these findings suggest that our method can effectively integrate knowledge from both specialized and generalist teachers without conflict.
Table 27: Performance of different teacher combinations across datasets (accuracy %).
Teachers
CIFAR-10
DTD
STL-10
SVHN
FGVC
CUB
Average
ViT + Segformer + DETR
94.03
63.62
95.86
65.63
38.79
67.67
70.93
VIT + Segformer
94.23
63.24
95,91
65.79
38.31
67.35
70.81
VIT + DETR
94.71
61.28
95.80
64.14
37.89
65.90
69.95
VIT
94.69
61.70
95.75
64.13
39.42
69.23
70.82
DETR + Segformer
87.87
63.72
94.81
54.71
37.89
62.43
66.91
Table 28: Comparison of ViT-based teacher combinations including DINO on multiple datasets (accuracy %). Bolded values indicate best per column.
Teachers
CIFAR-10
DTD
STL-100
SVHN
FGVC
CUB
Average
VIT + Segformer + DETR DINO
95.39
64.31
96.14
72.88
50.38 50.59
69.69
74.80
VIT DINO
95.83
61.92
96.06
73.60
69.21
74.54
As another additional experiment, CNN based teachers for resnet18. for different relevant datasets. [Tab. shoss the performance improvements, and the effectiveness using our distillation method, compared to other.
Table 29: Comparison of the performance with CNN-based teacher (accuracy %). Bolded values indicate best per column.
Method
Model
CIFARIO
FMNIST
MNIST
STL10
SVHN
QMNIST
KMNIST
CelebA
resnet18
81.89
86.94
96.6
92.98
$1.01
96.89
80.43
90.82
squeezenet
79.23
86.65
97.51
85.82
47.77
97.59
84.05
61.35
densenet
87.49
88.69
96.80
97.11
66.91
97.72
86.33
93.98
googlenet
81,94
86.38
96.71
93.95
55.9
97.2
79.27
92.93
NoKD
shufflenct
81.61
87.57
95.77
71.51
49.08
95.96
76.97
92.42
mobilenet
81.67
88.07
96.U5
92.26
48.57
97.5
85.64
91.02
mmasnet
81.41
88.76
96.09
92.79
57.63
97.00
82.35
89.01
resmext50-322ee
83. 42
87.32
95.37
95.97
$2.87
96.65
83.37
91.74
wide-resnet50-2
84.30
87.40
95.16
95.85
$7.77
96.74
76.23
90.22
Cosine
resnct18
84.57
89.90
98.58
88.34
76.34
98.95
91.97
95.00
T.2
resnet18
82.90
89.75
98.25
88.
74.84
98.61
88.21
94.89
NLL
resnet18
87.51
90.64
99.15
88.45
81.99
99.15
95.21
95.47
39

---

E
Detailed Method
Algorithm Distillation through Gaussian Kernels
Input: Dataset D = } Embedders (Tx) I Student embedder S, Number of iterations T,
Learning rate 1) Initialize the parameters 0, of the student embedder E, and the parameters 0g of the parametric Gaussian kernels for t to T do
Sample a batch of imputs {x; }
Compute the embeddings {t, (x;) Jiggk
Compute the student embeddings {s; = S(xi)}
Compute the loss LNLL- Σ k= Σx= log Mtg, Iu (s.). (s()))
Update the parameters 0g and 0x using the Adam optimizer. end for
F Computaional ressources
Our experiments were conducted in single GPUs settings. We used NVIDIA V100 GPUs for about 3000 GPUs hours to train our different models.
G
Baselines
For the MSE, we will optimize the following loss function following SimReg strategy (Navaneet) ct al. 2022).
LMSE
SS(x;) Tx(xx)]1
where it calculates the summation of MSE between the representation produced by each teacher and the student, for each instance of the batch.
Variant of SimReg can be implemented for Cosine multi-teacher feature distillation Gao al. 2022, Navancet et al. 2022). we optimize the summation cosine of teachers and the students representations of each instance of the batch, i.e.:
5(x;). (
LCosine
(8)
NLL Distil ation training loss
0.16
training
0.14
0.10
Distilatton
0.08
0.04
0.02
epoch
Figure 12: Training curves for the MSE baseline and the NLL student for the molecular experiments.
40

---

H Discussion On MSE distillation
We observed that when training with the MSE loss, the loss reaches minimum in only few epochs ( (0)), but the distilled students achieve lower performances downstream tasks. This could be due to the fact that the NLL loss more expressive, and harder to optimize (see below). As a result the student learns more informative features compared to when trained with the MSE loss Figure 12
We can provide theoretical insight explain this phenomenon. Training using the negative log- likelihood over a Gaussian kernel is simple generalization of the MSE. For a given multivariate Gaussian kernel parameterized by j and E. we have:
-log(pm.xxx)) = log(C)) log det E + 5(2 (60-~))-- =) (x 1))
Minimizing the MSE loss boils down to minimizing this equation over only, with Σ = I. There- fore, minimizing the negative log-likelln log-likelihood of a Gaussian kernel strictly more expressive than minimizing the MSE directly, which could account for the performance gains we observe.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose objective train model through multi teacher distillation, give theoretical grounding to it and validate it experimentally that it works by traning embedders in 3 modalities for wide range of size and domains.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, for example "Embedding space structure' Sec. 4.2, and "Computa- tional complexity." in Sec. 5.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
41

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide proofs for corollary in the supplementary materials. See Sec. 3.I and Appendix E
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All datasets, checkpoints and architectures are open source and under permis- sive licences. The code for our particular experiments is available in different repositories for each modalities with the specific hyperparameters and how to rerun the experiments. Our code is publicly available through anonymous links provided in the core of the paper. Regardless of the code we provided the details of datasets, models, hyperparameters and
tuning setting (see Secc. D.2
Sec. D.3
Sec. C.1.4 Sec. C.1.I Sec. B.3 and Sec. B.I).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
42

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., . a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We leveraged datasets and models available the huggingface hub and torchvision datasets for computer vision and NLP (apache 2.0, CC BY-NC-SA 4.0, CC BY-SA, CC BY-NC 4.0, CC BY 4.0 license), in molecules datasets under the apache 2.0 license and MIT license. All models and datasets are correctly cited in the paper. Code and data are publicly available in different repositories available in the main body of the paper.
gathered publiely available
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
coooooooPccy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
43

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: For every modality we share hyperparameter choices and specifics of the model in the specific sections well as the detailed experiments appendices (see Sec. D.2 Sec. D.3] Sec. C.1.4 and Secc. B.3)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report comprehensive results in appendices for every task with error bars and subset specific evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We both discuss the method complexity in the main paper and used ressources in Appendix F and Sec. 5.2
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
44

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///aaarrg...cpopaaa//ssaaa
Answer: [Yes]
The research conducted in the paper conforms with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
45

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We attribute all artefacts used and provide direct links to them for repro- ducibility purposes. All the ressources we used were publicly available under permissive licences.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Source code is shared through anonymous git repositories and trained models will be released the Huggingface hub.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
46

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standardd components.
Please refer to LLM policy http://uurrppss..ooeeeeeeeeeeeee / for what should or should not be described.
I Funding
This work granted access the HPC resources of IDRIS under the allocation AD01 1013290R,, and enabled by support provided by Calcul Quebec and the Digital Research Alliance Canada. This work was funded through scholarships by "École Technologie Supérieure Montreal", "Université Paris "aris-Saclay" and "MeGill University".
47


Reviews:
[{"review_id": "9C37ZBA13a", "reviewer": "Reviewer_dFG5", "summary": "This paper proposes a task-agnostic framework for multi-teacher distillation that learns general-purpose representations without requiring task-specific labels or supervision.\nThe approach introduces a novel loss function based on a “majority vote” principle, which is shown to be bounded by the mutual information between the student and teacher embeddings.\nThis results in a task-agnostic objective that encourages the student to align with the ensemble of teachers across a wide range of potential downstream tasks.\n\nThe method leverages a differentiable, Gaussian mixture-based estimator of conditional entropy to implement this loss in practice.\nThe training procedure minimizes the negative log-likelihood of the teacher embeddings conditioned on the student’s output, allowing end-to-end learning of a compact, informative student embedder.\n\nThe paper evaluates the approach across three domains - natural language processing, computer vision, and molecular modeling - using a range of classification, regression, and clustering tasks.\nThe results demonstrate that the distilled student models achieve competitive or superior performance relative to both teacher models and size-matched baselines.\nThe distilled model by the proposed method also shows strong size-performance trade-offs, advancing the Pareto frontier for efficient representation learning.", "questions": "1. Despite the strengths of the work, there is no dedicated discussion of its limitations, either theoretical or practical. For example, the method may face challenges in embedding structure preservation (as the authors briefly mentioned), memory costs from storing teacher embeddings, or reliance on high-quality teachers. Could the authors add an explicit limitations section/paragraph discussing practical boundaries of the approach, including cases where it may underperform or become inefficient? Can the authors elaborate on the practical implications of unstructured embeddings? Have the authors observed cases where this limitation actually causes issues?\n2. It may also be related to the above - Line 253, Page 8: Have any reasoning behind why the default 8-teachers model struggles and performs worse than other 1 or 2-teachers models with the BBB (Distribution) task?\n\nMinor issues\n- Line 39, Page 2: “o our knowledge, …”\n- No caption for Figure (probably) 4", "limitations": "Please refer to the weaknesses and the questions above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper introduces a theoretically grounded, task-agnostic distillation objective based on minimizing the conditional entropy of teacher embeddings given the student’s. This formulation allows the student to learn diverse, informative representations without reliance on task-specific labels, offering a general and conceptually sound approach.\n- The method is validated across three domains - language, vision, and molecular modeling - demonstrating its versatility. The breadth of evaluation strengthens the claim that the learned representations are useful across a wide range of downstream tasks.\n- The distilled student models consistently achieve high performance for their parameter count. In many cases, they match or outperform significantly larger models, indicating the method’s effectiveness in capturing information from multiple teachers.\n\n**Weaknesses**\n- While the authors acknowledge that the proposed objective does not preserve structural relationships (e.g., cosine similarity) in the embedding space, which helps clarify the scope and nature of the learned representations, the paper does not include a dedicated discussion of the limitations of the method in general. For instance, the handling of teacher inconsistency, conditions where the method may underperform, etc., are not discussed.", "was_revised": true, "final_justification": "This paper proposed a novel approach for task-agnostic multi-teacher distillation method.\nThe paper was well written, while the authors addressed the concerns raised, mainly about discussion of limitations.\nThe paper should be accepted so that the community benefits from the work and extend it.", "rebuttal": "We thank reviewer dFG5 for their insightful review and are pleased that they found our work interesting. Following the reviewer's advice, we will include a section dedicated to the limitations of our approach, referencing some results of other sections.\n\n## Limitation discussions\n\n**While we discussed some limitations in the appendices and experimental section, we agree our work would benefit from a dedicated limitation section. We will add a specific section covering the following limitations of our method:**\n- **Application Scope:** Our method develops student embedding models primarily for diverse, unknown tasks. For single pre-defined tasks, task-specific distillation approaches might be more suitable.\n- **Overhead in Distillation:** Like any distillation setting, especially multi-teacher distillation, there is an overhead due to the distillation of large teachers into a smaller model. This overhead can be computational (if teacher embeddings are obtained by running inference for each training) or memory-related (if teacher outputs are precomputed and used during training). We opted for the latter as it drastically speeds up student training, initially storing all necessary embeddings on disk. For text (our most computationally demanding application), this amounts to approximately 100GB of embeddings for the largest teacher.\n- **Teacher Quality:** Our approach requires high-quality teachers relevant to the downstream tasks. In Section D.4, Table 26, we explore the impact of training a student embedder with task-specific teachers (classification, object detection and segmentation). We demonstrate that while task-specific teachers may offer limited benefits outside their domain, they do not negatively impact the students’ learning when used alongside task-relevant teachers.\n- **Structural relationship:** As pointed out in Section 4.2, our metric only optimizes the mutual information between the student and the teachers, it does not directly enforce any structure on the embedding space, which could harm the performances of our models for clustering tasks for instance. For textual embeddings benchmark, we observe clear gains in classification tasks (where a small classifier is trained on top of the embeddings), but the gains are less clear for clustering and STS tasks that rely on the dot product between embeddings to assess text similarity (See full results in App. C.2.2).\n- **Representative dataset:** To effectively embed data for future tasks, our method requires a training set that is representative of the data distributions of these tasks. This limitation, however, is common to all embedding models, which all require a diverse and representative dataset for training.\n\n\n## Comparison of 8-teachers to 2-teachers on the BBB benchmark.\n\nThe student trained in molecular modeling with 8 teachers indeed shows slightly lower performance on the BBB (blood-brain barrier) benchmark compared to students trained with 1 or 2 teachers.\nThe BBB benchmark data distribution significantly differs from our training set, representing a domain shift compared to the training set of the students.\nFurthermore, it is one of the benchmarks where teacher performance is most tightly packed, with variations within 1.45 times the average standard deviation of the results. This could explain why training with 8 teachers performs closely to 1 or 2 (the differences in AUROC being half the standard variation), as all teachers demonstrate comparable performance on this specific task. \nWe believe this explains the slightly lower average performance of the 8-teacher student compared to the 1 or 2-teacher students.\n\nThank you again for your valuable feedback. We believe these clarifications and updates will address your concerns and enhance the quality and relevance of our work. We remain available to clarify any additional points if needed."}, {"review_id": "AGdH4jSaME", "reviewer": "Reviewer_Aw1B", "summary": "This paper proposes a multi-teacher distillation technique that is task agnostic. Through a majority vote objective function and ensembling loss, they show that this loss can be bounded independently of the task, making the distillation process task agnostic.", "questions": "please see weakness", "limitations": "no", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper formalizes the problem of task agnostic distillation well.\n- The use of Gaussian mixture based estimator to formulate loss is quite interesting and novel\n- They show a good understanding of the problem in designing embedding models, and show the relevance of this technique. \n- The paper shows application of this method for distilling molecular embedders, which is an interesting and important application. \n\nWeaknesses:\n- The paper is a bit outdated. \n- The paper does not motivate the need for task agnostic distillation well. A good comparison of modern embedding models on the benchmarks would have been useful, with more elaborate benchmarks. \n- The existing evaluation has multi-teacher distillation techniques as the baseline, but overall, it should have newer embedding models in the baseline as well.  \n- the evaluations are done on really small models and small benchmarks.", "was_revised": true, "final_justification": "I would like to keep the initial score. This paper addresses tiny models and from this discussion it's pretty clear that the technique proposed by the authors of this paper wont be beneficial on models that are >1B params. and even >700M params. So it casts doubts on the motivation for the paper and the relevance of the technique in light of the modern embedding models out there.", "rebuttal": "We thank reviewer Aw1B for their thorough review, and we are glad they appreciate the novelty in our proposed method. We politely disagree our experiments focused on 'really small models and small benchmarks', and we aim to provide a clear justification below. \n\n## Small Models\nWe trained models up to 300M parameters for textual embeddings, while this is not particularly large (<1B), **it is a standard size for embedding models in text**.\nContrary to text generation where clear performance gain can be seen when using very large models, performances of embedding models in text can achieve very competitive scores, while having fewer parameters  (e.g., the Stella-500M model outperforming several 8B models on the MTEB). \nThis scale is comparable to other well-established and widely used embedders, such as Stella and GIST, which are recognized for their strong performance in the field.\n\n## Small benchmarks\n\nWe politely disagree with this statement, across the three modalities evaluated in our study **we conducted comprehensive assessments on a total of 71 widely used datasets.** These included 6 datasets for computer vision, 32 for molecular modeling, and 33 for text analysis (ie **the Massive Text Embedding Benchmark, which constitutes the reference for textual embedders evaluation**). \n\n## Motivation of the task-agnostic distillation\nIn our introduction, we aim to motivate task-agnostic distillation in two steps. First, we highlight the development of embedding models, which are inherently task-agnostic. These models compress objects into numerical representations, thereby facilitating a wide array of downstream tasks. Next, we argue that the diversity of embedding models available in each field can be leveraged to build an embedder that benefits from these diverse representations through distillation.\n\nWe will extend the paragraph motivating the use of embedding models by mentioning their current applications in a wide range of scenarios, including classification, clustering, and information retrieval. We will also emphasize their key advantage in terms of computational efficiency, particularly in scenarios with limited labeled data.\n\n## 'The paper is a bit outdated'\n\nWe are not sure what part of the paper you are referring to when you say it is a bit outdated. Could you be more specific so we can update and improve our work accordingly? Nevertheless, we would like to make a few remarks as the field of text embedding models have gained a lot of focus in research lately:\n- We chose our baselines based on their leading performance on the MTEB benchmark at the time of submission. Notably, **Stella 400m v5 was the top-performing model in its weight category, only recently surpassed by the Qwen embedders released on June 15th 2025.** \n- We compared our students against the best models in each category, **including GIST models, which were among the highest-performing embedders across most weight categories until early 2025.**\n\nIn any case, we will update the MTEB benchmark tables in our paper to include the most recent results published.\n\n## Choice of baselines\n**We did not only compare our approaches with multi-teacher distillation approaches.** While our analysis mainly focuses on the comparison of our method with multi-teacher distillation methods, we also compared its performances with several embedders for each modality (of the same weight category).\n**The objective of our experimental section is to validate that our distillation approach is efficient in compressing the information of several teachers into a smaller student,  thus our focus on distillation baselines.** It is hard to provide a fair comparison of methods when comparing with other embedders since they have been trained with widely different settings, datasets, infrastructures and training objectives, whereas we provide distillation comparison in a valid controlled setting for fair comparisons to answer our initial scientific question. \n\nWe hope we have successfully addressed all your concerns, and we would be grateful if you could reconsider your score based on our revisions. Thank you again for your insightful review."}, {"review_id": "hyxV0J4E2L", "reviewer": "Reviewer_zNKm", "summary": "This paper presents a novel approach to Multi-Teacher Knowledge Distillation (KD), addressing the problem of generating general-purpose, task-agnostic embeddings.  Specifically, the core contribution is a task-enabling setting for multi-teacher distillation. Instead of traditional MSE-based losses, which can be unstable in high-dimensional spaces, the proposed method trains a student model to align its downstream task predictions with the collective predictions of an ensemble of teacher models. This is achieved through an ensembling loss that measures agreement between Bayesian predictors derived from student and teacher embeddings. A key theoretical finding is that this loss can be bounded independently of the specific task, utilizing the conditional differential entropy of the teachers' embeddings given the student's output, thereby providing a robust, task-agnostic student-teacher reconstruction loss. To evaluate the conditional entropy of the teachers’ embeddings given the student’s embedding, the authors propose using a parametric Gaussian model whose parameters are learned during the student’s training. Finally, the paper demonstrates high-quality generalized embedders across molecular modeling, natural language processing, and computer vision, with trained student models achieving competitive performance on a range of downstream tasks (e.g., classification, regression, clustering, sentence similarity).", "questions": "I feel the paper should be accepted (I gave it a 5) but I am not sure if there's anything that could be added for me to give a higher score.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: This well-crafted paper proposes a fairly novel and interesting method, complemented by comprehensive and compelling experimental findings.\n\nWeaknesses: No major weaknesses.", "was_revised": true, "final_justification": "Having read the authors' rebuttals and the other comments from the reviewers, I still maintain my score and positive opinion about this paper — which I feel that should be accepted.", "rebuttal": "We thank reviewer zNKm for their review, and we are glad they appreciated our work, and believe the paper should be accepted. We remain available to answer any new question if needed."}, {"review_id": "ViKYOador5", "reviewer": "Reviewer_3Aq4", "summary": "This paper proposes a novel framework for task-agnostic multi-teacher distillation that leverages a majority vote loss formulation, theoretically grounded in mutual information between student and teacher embeddings. The authors develop a principled loss function using conditional entropy and demonstrate the approach across three domains: NLP, CV, and molecular modeling. Extensive experiments show that the distilled student models achieve competitive or superior performance compared to existing baselines of similar sizes, improving the Pareto frontier of model size vs performance.", "questions": "1. Have you considered adapting other feature-based distillation methods as task-agnostic baselines? For example, Correlation Congruence (ICCV 2019) and Relational Knowledge Distillation (CVPR 2019) could be easily applied in your setup by averaging over teacher features. Including them on a single domain like vision is enough to provide a more comprehensive comparison.\n\n2. Could you analyze the stability of training compared to MSE or cosine baselines? Maybe you can provide convergence curves and variance across runs.\n\n3. Could the proposed method be extended to cross-modal distillation, as hinted at in the conclusion? What challenges do you anticipate?", "limitations": "Yes. The authors acknowledge that maximizing mutual information does not explicitly preserve the structure of the embedding space (e.g., cosine similarity), since information-theoretic objectives are invariant to invertible transformations. While this is theoretically valid, it does not appear to pose a practical problem in the reported experiments. The student models still achieve strong performance across tasks, suggesting that the learned representations remain effective despite the lack of explicit structure preservation. As such, I view this limitation as mostly theoretical and not impactful to the empirical validity or applicability of the method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper connects Bayesian classifier disagreement bounds with conditional entropy and mutual information, offering a principled and task-agnostic loss formulation.\n\n2. The method is validated on three diverse domains (text, vision, molecules), strengthening the claim of generality. Across multiple benchmarks (MTEB, TDC, fine-grained vision tasks), the distilled models consistently outperform or match larger models, pushing the size-performance tradeoff frontier. The comparison to standard MSE, cosine, and CompRess baselines is thorough and convincing.\n\nWeaknesses:\n\n1. While the proposed method is compared to MSE, cosine similarity, and CompRess, the set of baselines remains relatively narrow. There exist several established feature-distillation methods from the literature (e.g., Correlation Congruence for Knowledge Distillation [ICCV 2019]) that, while originally developed for task-specific distillation, can be straightforwardly adapted to a task-agnostic multi-teacher setup (e.g., by averaging teacher features). Including one or two such baselines (even only on a single modality like vision) would strengthen the empirical case and more convincingly isolate the advantages of the proposed information-theoretic loss.\n\n2. Although the method is claimed to be more stable than MSE or cosine methods, the paper lacks deeper analysis (e.g., convergence curves, variance across runs) to support this point.\n\n3. Additionally, there are some minor presentation issues. Several figures (e.g., in Section 4 and 5) are rasterized and appear blurry when zoomed in — using vector graphics would improve readability. There are also a few small typos and formatting inconsistencies: 1) Line 39: missing the “T” in “To our knowledge”; 2) Line 97: verb tense (“introduce”) is in past tense, inconsistent with surrounding text; 3) Line 176–177: the formatting of the letter “M” is inconsistent (italicized in one line but not the other). These issues do not affect the technical content but could be improved in the final version.", "was_revised": true, "final_justification": "I appreciate the authors' response. The paper is overall well-executed, and the experimental section is quite thorough. That said, I believe that a score of 4 (borderline accept) remains the most suitable assessment.", "rebuttal": "We thank reviewer 3Aq4 for their thorough and insightful review. We appreciate that they found our method principled and our experimental setup thorough and convincing. We added additional baselines to our vision experiments and we are thankful to the reviewer for pointing out the unclear reference to the MSE training stability, as well as the mistaken reference in the conclusion. We will correct these in the revised version of the paper.\n\n## MSE distillation stability\n\nWe would like to clarify that our reference to the instability of Mean Squared Error (MSE) was drawn from previous works in the field of reinforcement learning \\[1, 2, 3\\], where stability is context-specific. **Our intention was to use this as motivational background for our work, rather than making a direct claim about stability within our own research. We acknowledge that this distinction was not clear in our initial presentation, and we will revise those references. Most importantly, we mistakenly referenced stability in our conclusion, which we will correct in the revised version.**\n\n*Additionally we will add a dedicated section comparing the MSE loss and our NLL loss (with the training curves):*\n\n*We observed that when training with the MSE loss, the loss reaches a minimum in only a few epochs (\\~40), but the distilled students achieve lower performances on downstream tasks. This could be due to the fact that the NLL loss is more expressive, and harder to optimize (see below). As a result the student learns more informative features compared to when trained with the MSE loss. (training curves will be included)*\n\n*We can provide a theoretical insight to explain this phenomenon. Training using the negative log-likelihood over a Gaussian kernel is a simple generalization of the MSE. For a given multivariate Gaussian kernel parameterized by \\\\mu and \\\\Sigma, we have:*\n\n$\\-log (p\\_{\\\\mu, \\\\Sigma}(x)) \\= \\-\\\\log C \\+ \\\\frac12 \\\\det \\\\Sigma \\+ \\\\frac12 (x-\\\\mu)^T \\\\Sigma^{-1} (x-\\\\mu)$\n\n*Minimizing the MSE loss boils down to minimizing this equation over *$\\\\mu$* only, with *$ \\\\Sigma \\= I$*. Therefore, minimizing the negative log-likelihood of a Gaussian kernel is strictly more expressive than minimizing the MSE directly, which could account for the performance gains we observe.*\n\n\\[1\\] Stop Regressing: Training Value Functions via Classification for Scalable Deep RL  \nJesse Farebrother and al. 2024  \n\\[2\\] A Distributional Perspective on Reinforcement Learning, Marc G. Bellemare and al. 2017  \n\\[3\\] Regression as Classification: Influence of Task Formulation on Neural Network Features  \nLawrence Stewart and al. 2022\\.\n\n## Extension to multi-modal setting\n**The method can indeed be extended to a multimodal setting, where modality specific encoders would share the same backbone to embed different types of data.**\nThe challenge we would like to address is training a distillation model when limited cross-modal labels are available, to enable pretraining on larger datasets. So far, we did not observe a significant advantage compared to single modality training, and we are running additional experiments to answer the question: “How much cross-modal information is required to distill multi-modal representation that outperform the single modality ones?”\n\n## Additional baselines\n\nWe have added relational KD (RKD) \\[Park et al., CVPR 2019\\], Correlation Congruence with Gaussian RBF, and Bilinear (normalized features) kernels (CC-grbf, CC-Bilinear) \\[Peng et al., ICCV 2019\\] for vision. **As shown in the following table, our method (NLL) has better performance compared to the additional baselines.** Our intuition for the difference of accuracy is that both RKD and CC are proposed to work alongside the task loss, which could be an important signal for their optimization in practice.\n\n| Method | CIFAR10 | DTD | STL10 | SVHN | FGVCAircraft | CUB |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| RKD | 87.64 | 52.23 | 89.63 | 61.66 | 30.54 | 47.85 |\n| CC-grbf | 84.07 | 61.86 | 93.03 | 59.96 | 33.48 | 57.55 |\n| CC-Bilinear  | 92.95 | 61.22 | 95.42 | 63.71 | 35.16 | 64.7 |\n| NLL | **94.76** | **65.85** | **96.45** | **76.91** | **48.13** | **69.37** |\n\n## Presentation Issues\nThank you for your thoroughness. You are absolutely right, we’ll update the figures with vectorized versions, and we’ll fix the formatting inconsistencies you pointed out.\n\nThank you again for your valuable feedback and suggestions, which have significantly improved our paper. Especially, thank you for pointing out the lack of clarity in our reference to the instability of MSE value estimation in reinforcement learning, that we have corrected.  \nWe hope we addressed all of your concerns so that you might consider raising the score of your review."}]

Author rebuttal:

Meta-review: This paper proposes a task-agnostic method for multi-teacher distillation utilizing a newly proposed loss function which is bounded resulting in a task agnostic objective. All reviewers agree that this is a paper with good methodological contribution and novelty (which is rare for the fairly saturated topic of knowledge distillation), and strong results across multiple datasets and domains. The authors did a good job with rebuttal addressing most of the authors' concerns. There is only some issue raised pertinent to what extent the method can be used for distilling very big models but the authors have already shown results with a 300M model, so this limitation can be considered only minor. Overall, a clear accept.
Decision: Accept (poster)