Title: Learning Task-Agnostic Representations through Multi-Teacher Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: molecular representation

nlp

knowledge distillation

embedding models

representation learning
Abstract: Casting complex inputs into tractable representations is a critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. We introduce a task-agnostic framework based on a ``majority vote" objective function. 
We demonstrate that this function is bounded by the mutual information between the student and the teachers' embeddings, leading to a task-agnostic distillation loss that eliminates dependence on task-specific labels or prior knowledge. 
Comprehensive evaluations across text, vision models, and molecular modeling show that our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such as classification, clustering, or regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for task-agnostic multi-teacher distillation that leverages a majority vote loss formulation, theoretically grounded in mutual information between student and teacher embeddings. The authors develop a principled loss function using conditional entropy and demonstrate the approach across three domains: NLP, CV, and molecular modeling. Extensive experiments show that the distilled student models achieve competitive or superior performance compared to existing baselines of similar sizes, improving the Pareto frontier of model size vs performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper connects Bayesian classifier disagreement bounds with conditional entropy and mutual information, offering a principled and task-agnostic loss formulation.\n\n2. The method is validated on three diverse domains (text, vision, molecules), strengthening the claim of generality. Across multiple benchmarks (MTEB, TDC, fine-grained vision tasks), the distilled models consistently outperform or match larger models, pushing the size-performance tradeoff frontier. The comparison to standard MSE, cosine, and CompRess baselines is thorough and convincing.\n\nWeaknesses:\n\n1. While the proposed method is compared to MSE, cosine similarity, and CompRess, the set of baselines remains relatively narrow. There exist several established feature-distillation methods from the literature (e.g., Correlation Congruence for Knowledge Distillation [ICCV 2019]) that, while originally developed for task-specific distillation, can be straightforwardly adapted to a task-agnostic multi-teacher setup (e.g., by averaging teacher features). Including one or two such baselines (even only on a single modality like vision) would strengthen the empirical case and more convincingly isolate the advantages of the proposed information-theoretic loss.\n\n2. Although the method is claimed to be more stable than MSE or cosine methods, the paper lacks deeper analysis (e.g., convergence curves, variance across runs) to support this point.\n\n3. Additionally, there are some minor presentation issues. Several figures (e.g., in Section 4 and 5) are rasterized and appear blurry when zoomed in — using vector graphics would improve readability. There are also a few small typos and formatting inconsistencies: 1) Line 39: missing the “T” in “To our knowledge”; 2) Line 97: verb tense (“introduce”) is in past tense, inconsistent with surrounding text; 3) Line 176–177: the formatting of the letter “M” is inconsistent (italicized in one line but not the other). These issues do not affect the technical content but could be improved in the final version.\n\nQUESTIONS:\n1. Have you considered adapting other feature-based distillation methods as task-agnostic baselines? For example, Correlation Congruence (ICCV 2019) and Relational Knowledge Distillation (CVPR 2019) could be easily applied in your setup by averaging over teacher features. Including them on a single domain like vision is enough to provide a more comprehensive comparison.\n\n2. Could you analyze the stability of training compared to MSE or cosine baselines? Maybe you can provide convergence curves and variance across runs.\n\n3. Could the proposed method be extended to cross-modal distillation, as hinted at in the conclusion? What challenges do you anticipate?\n\nLIMITATIONS:\nYes. The authors acknowledge that maximizing mutual information does not explicitly preserve the structure of the embedding space (e.g., cosine similarity), since information-theoretic objectives are invariant to invertible transformations. While this is theoretically valid, it does not appear to pose a practical problem in the reported experiments. The student models still achieve strong performance across tasks, suggesting that the learned representations remain effective despite the lack of explicit structure preservation. As such, I view this limitation as mostly theoretical and not impactful to the empirical validity or applicability of the method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to Multi-Teacher Knowledge Distillation (KD), addressing the problem of generating general-purpose, task-agnostic embeddings.  Specifically, the core contribution is a task-enabling setting for multi-teacher distillation. Instead of traditional MSE-based losses, which can be unstable in high-dimensional spaces, the proposed method trains a student model to align its downstream task predictions with the collective predictions of an ensemble of teacher models. This is achieved through an ensembling loss that measures agreement between Bayesian predictors derived from student and teacher embeddings. A key theoretical finding is that this loss can be bounded independently of the specific task, utilizing the conditional differential entropy of the teachers' embeddings given the student's output, thereby providing a robust, task-agnostic student-teacher reconstruction loss. To evaluate the conditional entropy of the teachers’ embeddings given the student’s embedding, the authors propose using a parametric Gaussian model whose parameters are learned during the student’s training. Finally, the paper demonstrates high-quality generalized embedders across molecular modeling, natural language processing, and computer vision, with trained student models achieving competitive performance on a range of downstream tasks (e.g., classification, regression, clustering, sentence similarity).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This well-crafted paper proposes a fairly novel and interesting method, complemented by comprehensive and compelling experimental findings.\n\nWeaknesses: No major weaknesses.\n\nQUESTIONS:\nI feel the paper should be accepted (I gave it a 5) but I am not sure if there's anything that could be added for me to give a higher score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a multi-teacher distillation technique that is task agnostic. Through a majority vote objective function and ensembling loss, they show that this loss can be bounded independently of the task, making the distillation process task agnostic.

STRENGTHS AND WEAKNESSES:
Strengths:
- This paper formalizes the problem of task agnostic distillation well.
- The use of Gaussian mixture based estimator to formulate loss is quite interesting and novel
- They show a good understanding of the problem in designing embedding models, and show the relevance of this technique. 
- The paper shows application of this method for distilling molecular embedders, which is an interesting and important application. 

Weaknesses:
- The paper is a bit outdated. 
- The paper does not motivate the need for task agnostic distillation well. A good comparison of modern embedding models on the benchmarks would have been useful, with more elaborate benchmarks. 
- The existing evaluation has multi-teacher distillation techniques as the baseline, but overall, it should have newer embedding models in the baseline as well.  
- the evaluations are done on really small models and small benchmarks.

QUESTIONS:
please see weakness

LIMITATIONS:
no
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a task-agnostic framework for multi-teacher distillation that learns general-purpose representations without requiring task-specific labels or supervision.\nThe approach introduces a novel loss function based on a “majority vote” principle, which is shown to be bounded by the mutual information between the student and teacher embeddings.\nThis results in a task-agnostic objective that encourages the student to align with the ensemble of teachers across a wide range of potential downstream tasks.\n\nThe method leverages a differentiable, Gaussian mixture-based estimator of conditional entropy to implement this loss in practice.\nThe training procedure minimizes the negative log-likelihood of the teacher embeddings conditioned on the student’s output, allowing end-to-end learning of a compact, informative student embedder.\n\nThe paper evaluates the approach across three domains - natural language processing, computer vision, and molecular modeling - using a range of classification, regression, and clustering tasks.\nThe results demonstrate that the distilled student models achieve competitive or superior performance relative to both teacher models and size-matched baselines.\nThe distilled model by the proposed method also shows strong size-performance trade-offs, advancing the Pareto frontier for efficient representation learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper introduces a theoretically grounded, task-agnostic distillation objective based on minimizing the conditional entropy of teacher embeddings given the student’s. This formulation allows the student to learn diverse, informative representations without reliance on task-specific labels, offering a general and conceptually sound approach.\n- The method is validated across three domains - language, vision, and molecular modeling - demonstrating its versatility. The breadth of evaluation strengthens the claim that the learned representations are useful across a wide range of downstream tasks.\n- The distilled student models consistently achieve high performance for their parameter count. In many cases, they match or outperform significantly larger models, indicating the method’s effectiveness in capturing information from multiple teachers.\n\n**Weaknesses**\n- While the authors acknowledge that the proposed objective does not preserve structural relationships (e.g., cosine similarity) in the embedding space, which helps clarify the scope and nature of the learned representations, the paper does not include a dedicated discussion of the limitations of the method in general. For instance, the handling of teacher inconsistency, conditions where the method may underperform, etc., are not discussed.\n\nQUESTIONS:\n1. Despite the strengths of the work, there is no dedicated discussion of its limitations, either theoretical or practical. For example, the method may face challenges in embedding structure preservation (as the authors briefly mentioned), memory costs from storing teacher embeddings, or reliance on high-quality teachers. Could the authors add an explicit limitations section/paragraph discussing practical boundaries of the approach, including cases where it may underperform or become inefficient? Can the authors elaborate on the practical implications of unstructured embeddings? Have the authors observed cases where this limitation actually causes issues?\n2. It may also be related to the above - Line 253, Page 8: Have any reasoning behind why the default 8-teachers model struggles and performs worse than other 1 or 2-teachers models with the BBB (Distribution) task?\n\nMinor issues\n- Line 39, Page 2: “o our knowledge, …”\n- No caption for Figure (probably) 4\n\nLIMITATIONS:\nPlease refer to the weaknesses and the questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}