When is an Embedding Model  More Promising than Another?

Authors: maxime  darrin
philippe  formont
ismail  ben ayed
jackie  chi
kit  cheung
pablo  piantanida
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:38:31

Abstract: Embedders play a central role in machine learning, projecting any object into numerical representations that can, in turn, be leveraged to perform various downstream tasks. The evaluation of embedding models typically depends on domain-specific empirical approaches utilizing downstream tasks, primarily because of the lack of a standardized framework for comparison. However, acquiring adequately large and representative datasets for conducting these assessments is not always viable and can prove to be prohibitively expensive and time-consuming.  In this paper, we present a unified approach to evaluate embedders. First, we establish theoretical foundations for comparing embedding models, drawing upon the concepts of sufficiency and informativeness. We then leverage these concepts to devise a tractable comparison criterion (information sufficiency), leading to a task-agnostic and self-supervised ranking procedure. We demonstrate experimentally that our approach aligns closely with the capability of embedding models to facilitate various downstream tasks in both natural language processing and molecular biology. This effectively offers practitioners a valuable tool for prioritizing model trials.

Main paper: 
 introduction: 
 Embeddings are a prominent tool in machine learning and are used in multiple fields, such as natural language processing [64, 83] , computer vision [93, 59, 12, 53] or bioinformatics [67, 3, 23, 112] . These models embed objects such as images, texts, or molecules into numerical representations that can be used to perform numerous downstream tasks by preserving key features of the object [76, 111] .Depending on the data modalities, intended purpose, and available resources, embedders showcase a wide variety of architectures, training settings (unsupervised, supervised, self-supervised, etc.), objectives (masked language modeling, contrastive learning, etc.) [20, 100, 78, 112, 39] , and datasets [65, 86, 31, 38, 5, 117] . And more recently, foundation models have become a natural starting point to create embedders [21, 106, 52, 73] . This diversity and variety of options makes selecting the most promising embedders for a data distribution challenging [75] . Most work evaluates embedders focusing on the performance they enable on a finite set of downstream tasks [85, 17, 90, 91, 81, 24] . Nevertheless, this evaluation process encounters two primary limitations. Firstly, it is not scalable concerning the number of embedders and tasks, as it requires fitting a downstream model for each task. Hence, prioritizing the evaluation of the most promising models becomes essential to mitigate computational costs. Secondly, acquiring high-quality labels can be a time-consuming and notably expensive endeavor in various applications. To overcome these limitations, in this paper, we explore task-agnostic evaluation metrics for embedders relying solely on pairwise comparisons between embedders, i.e., without the need for labeled data in downstream tasks.More specifically, our contributions can be summarized as follows:1. An innovative theoretical framework for comparing embedding models: We cast the problem of ranking embedders into the noisy communication channels ordering (Sec. 2.2) and statistical experiments comparison settings (Sec. 2.3). We exploit the notions of sufficiency and informativeness and relax them, leveraging the concept of deficiency introduced by Le Cam [63] (Sec. 2.4), which is reframed to account for concepts and features. These concepts provide us with tools to establish an embedder ranking. 2. A practical relaxation: Estimating deficiency presents significant challenges. We propose the concept of information sufficiency (IS), which quantifies the information required to simulate one embedder from another (Sec. 3). We estimate the information efficiency to get a task-agnostic and label-free comparison tool for embedders evaluation. 3. Extensive experimental validation: The expected IS correlates with the ability of embedders to enable a wide range of downstream tasks. In NLP (Sec. 5) and molecular modeling (Sec. 6), our method respectively achieves Spearman ranking correlations of 0.90 (56 tasks) and 0.94 (31 tasks) ; providing an efficient model trial prioritization tool for practitioners.Embedding evaluation. Embedding evaluation is mainly performed based on a limited set of downstream tasks [22, 91, 81, 24] , for which the embeddings are used as inputs to smaller models. Therefore, embedders evaluation is field-and task-specific. In NLP, [41, 85] they rely on a limited set of tasks; more recently, the Massive Text Embedding Benchmark (MTEB) [75] followed this task-oriented trend and offered standardized test bed for embedders encompassing various downstream tasks in NP. Devising statistical tests to compare models and learning algorithms has a long history [30] . However, most works propose statistical tests relying on the performance of the downstream tasks of interests [60, 11] . Other works study the expressiveness of embedders and connect it to performance on downstream tasks [107, 25] , but mostly focus on geometrical properties of the high dimensional representation in self-supervised learning settings [2, 42, 45] .Probing. While probing methods do not aim at comparing embedders, they evaluate their representations to discover what these models have learned. They train small models on the internal representations of large models to perform specific downstream tasks. These procedures allow researchers to assess what information is present and recoverable from these embeddings [10, 1, 88, 84] .Other work proposed measuring mutual information (MI) between internal representations and labels. It has been used to evaluate the difficulty of a dataset as the predictiveness of the labels using the features [35] . For instance, [97] evaluates the utility of representations in astrophysics to predict physical properties. Following this trend, [54] leverages the point-wise MI between Gaussian distributions to evaluate text-to-images and image-to-text generative models. However, none of these methods have focused on comparing embedders in the general case to the best of our knowledge.2 Theoretical Foundations for Comparing Embedding Models 
background and notation: 
 We assume that all considered spaces are standard Borel [28] Each such space U is equipped with its Borel σ-algebra B(U). The set of all probability measures on U is denoted by P(U) The total variation distance between P and Q is denoted by ∥P -Q∥ TV . Given a joint probability measure P XY induced by two random variables X ∈ X and U ∈ U, the Mutual Information [27] is denoted by I(X; U ). A Markov (or transition probability) kernel between X and U is a mappingP U |X : B(U) × X → [0, 1].The space of all such P U |X is denoted by K(U|X) and (M •P U |X )(V |x) indicates the composition of Markov kernels M ∈ K(V|U) and P U |X ∈ K(U|X). For further details, refer to Appendix A.We aim to compare embedding models without relying on labeled data for downstream tasks. Let us consider two embedding models represented by their Markov kernels (or transition probabilities) P U |X ∈ K(U|X) and P V |X ∈ K(V|X), any target set Y of (discrete or continuous) concepts and feature space X with joint probability measure P Y X ∈ P(Y × X) induced by random variables (Y, X) ∈ Y × X, as illustrated in Figure 1 . First, we study the question:What sufficient conditions must be met by the embedding model U relative to V to guarantee that I(Y ; U ) ⩾ I(Y ; V ) for all distributions P Y X ?< l a t e x i t s h a 1 _ b a s e i g j q Q Y B y 3 g k E t 7 V Q k d 8 k C r O 4 i D i / W D + 2 u T 7 Z 1 x l Q i b v 9 C L l p z G b J m I i Q q a J G l X e 1 F x f z e S o V 3 f 9 m O l Z M F n 2 V g 3 X Z 2 m q 5 I d 1 b u m / F 2 M + Y 5 p S q / p N c F u / a t R G l a r X 9 O x y 7 4 N W A a o o 1 o m s f I G P M S R C 5 I j B k U A T j s C Q 0 T N E C x 5 S 4 k 6 x J E 4 R E j b P s U K Z t D l V c a p g x M 7 p O 6 V o W L A J x c Y z s + q Q d o n o V a R 0 8 Z I 0 k u o U Y b O b a / O 5 d T b s 3 7 y X 1 t O c b U H / o P C K i d W Y E f s v 3 X X l / + p M L x o T v L I⇢ V (V) ⇡ ⇢ b V ( b V)< l a t e x i t s h a 1 _ b a s e 6 4 = " E X U 9 f z 5 From an information-theoretic perspective [27] , the quality of an embedding model can be likened to the capacity of a noisy communication channel with an uncoded input (e.g., a text, a molecule...), where a downstream task of interest is performed at the output (the embedding) of the channel. Let Y ∈ Y represent the message (the source) to be communicated over both channels; X represents the transmitted signal; and P U |X and P V |X the communication channels with outputs U and V , respectively. This process is illustrated in Figure 1 . It naturally satisfies the Markov chain Y ↔ X ↔ (U, V ). A desirable property is that the embedding models U and V retain as much pertinent information as feasible to predict Y .9 v g k 1 o n C 4 s 6 t w E 9 W N R Z 8 = " > A A A C z H i c j V H L T s J A F D 3 U F + I L d e m m E U x c k d Y F 6 o 7 E j S u D i Q U M E t I O A z b 0 l X Z q Q i p b f 8 C t f p f x D / Q v v D O W R C V G p 2 l 7 5 t x 7 z s y 9 1 4 k 8 N x G G 8 V r Q F h a X l l e K q 6 W 1 9 Y 3 N r f L 2 T i s J 0 5 h x i 4 V e G H c c O + G e G 3 B L u M L j n S j m t u 9 4 v O 2 M z 2 S 8 f c f j x A 2 D K z G J e M + 3 R 4 E 7 d J k t i L q u N v u Z d d + Z V v v l i l E z 1 N L n g Z m D C v L V D M s v u M E A I R h S + O A I I A h 7 s J H Q 0 4 U J A x F x P W T E x Y R c F e e Yh i c = " > A A A C y n i c j V H L S s N A F D 2 N r 1 p f V Z d u g q 3 g q i Q u q u 4 K b l y 4 q G A f U E t J 0 m k d m h e T i V B C d / 6 A W / 0 w 8 Q / 0 L 7 w z p q A W 0 Q l J z p xP E z c M r s Q k 4 j 3 f H g X u 0 G W 2 I O q 6 3 O h n r f v O t N w v l q y K p Z Y 5 D 6 o a l K B X I y y + 4 A Y D h G B I 4 Y M j g C D s w U Z C T x d V W I i I 6 y E j L i b k q j j H F A X S p p T F K c M m d k z f E e 2 6 m g 1 o L z 0 T p W Z 0 i k d v T E o T B 6 Q J K S 8 m L E 8 z V T x V z p L 9 z T t T n v J u E / o 7 2 s s n V u C W 2 L 9 0 s 8 z / 6 m Q t A k O c q B p c q i l S j K y O a Z d U d U X e 3 P x S l S C H i D i J B x S P C T O l n P X Z V J p E 1 S 5 7 a 6 v 4 m 8 q U r N w z n Z v i X d 6 S B l z 9 O c 5 5 0 D q q V G u V 2 u V R q X 6 q R 5 3 H H v Z x S P M 8 RK l D H i 2 + s D D d U T N n L v S h v W L 4 8 o = " > A A A C z H i c j V H L T s J A F D 3 U F + I L d e m m E U x c k d Y F 6 o 7 E j S u D i Q U M E t I O A z b 0 l X Z q Q i p b f 8 C t f p f x D / Q v v D O W R C V G p 2 l 7 5 t x 7 z s y 9 1 4 k 8 N x G G 8 V r Q F h a X l l e K q 6 W 1 9 Y 3 N r f L 2 T i s J 0 5 h x i 4 V e G H c c O + G e G 3 B L u M L j n S j m t u 9 4 v O 2 M z 2 S 8 f c f j x A 2 D K z G J e M + 3 R 4 E 7 d J k t i L q u N v t Z 6 9 6 a V v v l i l E z 1 N L n g Z m D C v L V D M s v u M E A I R h S + O A I I A h 7 s J H Q 0 4 U J A x F x P W T E x Y R c F e e Yz I s R M R d Y U p c T M h V c Y 4 Z c q R N K Y t T B i N 2 T N 8 h 7 b q a D W g v P R O l d u g U j 9 6 Y l C a O S B N S X k x Y n m a q e K q c J f u b 9 1 R 5 y r t N 6 G 9 r L 5 9 Y g R G x f + n m m f / V y V o E B j h T N b h U U 6 Q Y W Z 2 j X V L V F X l z 8 0 t V g h w i 4 i T u U z w m 7 C j l v M + m 0 i S q d t l b p u J v K l O y c u / o 3 B T v 8 p Y 0 4 P L P c S 6 C V q V U r p a q j U q h d q J H n c U B D n F M 8 z x F D RWe shall be interested in the underlying information relationships between those embedding models that can be interpreted as channel U being "more informative" for communicating Y than channel V . The first attempt to introduce an ordering between communication channels appears in Shannon [94] . Körner and Marton later introduced [57] the concepts of "less noisy" (or more informative) and "degraded" (or sufficiency) orderings between channels. Definition 1 (Sufficiency and informativeness orderings [57] ). Let P U |X and P V |X be two Markov kernels (embedding models). Motivated by the concepts of sufficiency and informativeness between embedding models, we can inquire about their statistical consequences for a learner conducting an inference task on these embeddings. More precisely, given a finite set of concepts Y, if U ≽ S V , is the Bayes risk expected to be smaller when the inference is based on U than when it is based on V ?The pursuit of comparing statistical experiments originated from the seminal paper by Bohnenblust, Shapley, and Sherman [16] , followed by subsequent contributions by Blackwell [13, 14] . They formally established the relationships between sufficiency (Def. 1) and inference procedures.In our framework, a statistical experiment [13] consists of a mathematical abstraction (see Appendix A for further details) intended to represent a downstream task where a learner aims at inferring a concept y ∈ Y from the embeddings U or V . Deciding what embedder should be used to perform a given task is too general. In this work, we do not take into account the computational cost or the size of an embedder and solely focus on the following question:What are the necessary and sufficient conditions that ensure that employing the embedding U for any task P Y X leads to lower risk compared to using the embedding V ?Drawing parallels with the theoretical framework established for comparing statistical experiments, a relationship can be derived between the concept of sufficiency and the expected risk for a specific task (see Sec. B.5 for further discussion).We concentrate on the scenario where Y consists of a finite number of concepts (e.g., classification tasks), as it is a significant case in its own right [104] Pr ŶU ̸ = Y ⩽ inf ρ V :V→P(Y) Pr ŶV ̸ = Y ,where ŶU and ŶV are distributed according to ρ U (U ) and ρ V (V ), respectively. Remark 2. In other words, if we can fully simulate an embedder V from another embedder U , the expected risk across all potential classification tasks cannot be greater when using U compared to V . The proof of this Proposition is given in Sec. B.3. It is worth mentioning that various versions of this result are available in the literature [104] . However, our extension here, in a simpler setting, incorporates concepts and features into the experiment comparison framework.According to the notion of "sufficiency", we can distinguish the three following possibilities:• Equivalence: U ≽ S V and V ≽ S U denoted U ≈ V ; U and V can simulate each other. • Comparability: U ≽ S V but V ̸ ≽ S U only V can be simulated from U . • Non-comparability: U ̸ ≽ S V and V ̸ ≽ S U , neither U nor V can simulate each other.Our results up to now only account for the two first possibilities. However, two embedders are generally not comparable (Sec. B.4). This issue was addressed by Le Cam [63] , who introduced the notion of "deficiency". Definition 2. The deficiency δ(P U |X → P V |X ) of P V |X relative to P U |X is defined as [63] δ(P U |X → P V |X ) ≜ inf M ∈K(V|U) E∥M •P U |X -P V |X ∥ TV ,where the infimum is taken over all Markov kernels (or transition probabilities) M ∈ K(V|U), mapping stochastically U and V, and δ measures error between the simulated and true embedders.δ indicates how well one model can be reconstructed from the other, it induces a natural relaxation of the sufficiency where the reconstruction does not have to be perfect 2 for us to obtain guarantees on the downstream tasks performance (See Corollary 1). It avoids the non-comparability problem by evaluating "how much information" we lose when passing from one model to the other one.Le Cam [63] showed that, for a given task Y , the deficiency δ(P U |Y → P V |Y ) is directly related to the expected Bayes risks on the task (see Sec. B.6). We extend this result to the comparison of two embedding models P U |X and P V |X in a task-agnostic manner and build the relation to the expected Bayes risks for any classification task Y .Corollary 1. Given two embedding models P U |X and P V |X satisfying:(i) The deficiency δ(P U |X → P V |X ) ⩽ γ. (ii) For any conditional distribution P Y |X on finite alphabets Y, inf ρ U :U→P(Y) Pr ŶU ̸ = Y -ε ⩽ inf ρ V :V→P(Y)Pr ŶV ̸ = Y .Statement (ii) implies (i) provided that γ ⩾ 2|Y|ε and conversely, (i) implies (ii) provided that γ ⩽ ε.The proof of this Corollary is relegated to Sec. B.3. Remark 3. In particular, we can infer that for any classification task Y , the expected Bayes risk of the embedding model U , denoted by R U , is upper bounded by the expected Bayes risk of the embedding model V , denoted by R V :R U -R V ⩽ δ(P U |X → P V |X ), for all conditional distributions P Y |X ,and similarly,|R U -R V | ⩽ max δ(P U |X → P V |X ), δ(P V |X → P U |X ) , for all conditional distributions P Y |X .If both deficiencies are small, the resulting expected Bayes risks of the embedding models U and V will be close to each other for any target task Y . 
quantifying information sufficiency between embedding models: 
 We want to compare embedding models using the concept of deficiency, leveraging Prop. 2 and Corollary 1. These propositions suggest that the performance on any classification task of an embedding model U relative to the model V is bounded by δ(P U |X → P V |X ). However, estimating the deficiency from data samples is notably challenging [95] , and while upper bounds derivation exists, they do not necessarily make it tractable.The deficiency δ(P U |X → P V |X ) between two embedding models P U |X and P V |X , measures how well U can be used to simulate V using a Markov kernel M ∈ K(V|U). This section aims to build a tractable proxy for this reconstruction cost. To this end, we estimate how much we can reduce the uncertainty about Z by observing U by learning an appropriate Markov kernel. This corresponds to the information sufficiency [29, 4] and can be interpreted as the information-theoretic counterpart of the deficiency. The information deficiency between U and V is then defined as:Definition 3 (Information sufficiency). The information sufficiency I S (U → V ), relative to parametric classes of distributions F Θ (V) and K Θ (V|U) (multivariate Gaussian mixtures [82] ) is defined:EQUATION)Remark 4. When the information sufficiency I S (U → V ) is large, it signifies that U offers a substantial amount of information to simulate V , a proxy for a small deficiency. Conversely, when I S (U → V ) is lower, it implies that the channel P V |Y is subject to considerable noise or randomness, leading to a greater loss of statistical information.We hence attempt to simulate V from U by learning a Markov kernel M ∈ K Θ (V|U), via a mixture of multivariate Gaussians, and measure the uncertainty reduction it induces. Pairwise embedder evaluation. For set of embbeders (Z k ) k represented by their Markov kernels {P Z k |X } k , we compute the pairwise information sufficiency I S (Z k → Z l ). The pairwise information sufficiency matrix defines the adjacency matrix of a directed graph of embedders (Figure 2 ). Corollary 1 shows that embedders sharing high information sufficiency are expected to perform similarly on any downstream tasks, motivating the identification of communities in the graph. While the graph construction is in O(N 2 ); where N is the number of embedders, it is in practice tractable for a reasonable number of embedders (refer to Sec. E.6) for more details).Practical embedding evaluation. We construct the set of all information sufficiency using Z k :S I S (k) = {I S (Z k → Z l )} l̸ =k .We build our information sufficiency score (I S score) by taking the median of S I S (k). Details on the I S score's estimation can be found in Sec. E.1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We only propose model selection method for embedders. The main concern, that is addressed in the paper, is that while our method is theoretically supported and empirically validated, it might not be applicable to all downstream tasks and should be used with caution.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setup: 
 We aim to evaluate the practical utility of the I S score to rank and select the best embedders for a given data distribution. We compare this ranking to those obtained on various downstream tasks. Our experimental protocol is divided into three main steps:1. We evaluate the I S score of the models by identifying a large and diverse dataset that is supposed to be representative of the data distribution of interest. 2. We train a small feedforward neural network (ρ Z k ) per embedder P Z k |X to perform each downstream task and record its performances (R 2 score for regression, AUROC/accuracy for binary/multiclass classification). 3. We compare the models' performances on the downstream tasks and the I S score by measuring three types of correlations: the Pearson correlation, the Spearman correlation, and the Kendall-Tau coefficient.  
experimental setting: 
 Embedders & Datasets. We compared 34 models with different training objectives, training datasets, and architectures. We included embedders derived from modern LLM such as LLaMA [106] , Mistral [52] , Gemma [102] , Croissant [37] and T5 encoders [77] ; common embedders derived from BERT architectures [31, 38, 85] or RobERTa [41] and embedders trained on specific embeddings objectives such Angle [64] , Stella4 , E5 models [113] , LaBSE [38] . A comprehensive list of the models can be found in Sec. C.1, Tab. 1 with their main characteristics and links to the Huggingface Hub for reproducibility. We used them to extract embeddings for many different datasets from the MTEB benchmark such as Banking77 [19] , Sickr [122] , Amazon polarity [72] , SNLI [120] and IMDB [70] . We provide the datasets statistics in Sec. C.1, Tab. 2. Downstream tasks evaluation. We rely on the results released on the MTEB leaderboard 5 and compare our rankings to the rankings and scores obtained by the different models on the different tasks. We evaluate additional tasks that are not included in the MTEB benchmark, such as tweet_eval [8, 74, 7, 109, 9] , DAIR Emotion [92] , agnews topic classification [123] , Clinc intent detection [62] PAWS-X [118] and Rotten Tomatoes [79] . Figure 4b displays the performance on additional downstream tasks and models not evaluated in the MTEB leaderboard. Figure 4c shows that instruction finetuning positively impacts the models' performance on the downstream tasks and that this improvement is captured by I S .Correlation with downstream tasks performance. The MTEB Benchmark offers a natural starting point to compare models' ranking according to their performance on downstream tasks and their I S score. In Figure 3c , we show that the I S score of an embedder correlates positively with its performance on a wide range of downstream tasks, from classification and similarity tasks to retrieval and clustering tasks. Overall, our I S score correlates strongly with MTEB's average score (Spearman correlation of 0.90 and a Pearson correlation of 0.94, see Figure 3c ) and with the subtask performance Figure 3a ). We extended our experiments to a more extensive set of models not included in the MTEB benchmark and observed a similar trend (Figure 4b ). Per-datasets results are reported in Sec. C.3.1 and ablations in Sec. C.3.2. All our results show that our estimation of the information sufficiency between models is a good proxy for the performance of the models on a wide range of tasks.Embedder communities. The pair-wise information sufficiency evaluation between the models can be used to cluster them into communities [15] (Figure 4a , Figure 2 ) 6 . We observe that the extracted clusters group together models that are similar in their training objectives and architectures. LLM-based models such as LLaMA, Mistral, Gemma, and Croissant are clustered together, while BERT-based models share another cluster. Similarly, models trained specifically for embedding purposes, such as UAE-Large-V1 and ember-v1, are grouped together. This suggests that the ordering induced by information sufficiency is meaningful and can be used to identify models with similar properties and behaviors. Consistently with Corollary 1, we observe that the performance of the models on the downstream tasks is similar within the same cluster (Figure C.3.5). In addition, we found that it captures improvements by both steps of pretraining and instruction fine-tuning (Figure 4c , Sec. C.3.2)6 Molecular Modeling 
experimental setting: 
 Embedders. To process molecular data, embedders can leverage different representations of the molecules, providing an interesting benchmark to evaluate the I S score. We evaluated models derived from the molecular representation learning literature, summed up in Sec. D.1. We considered various input modalities such as string representations (SMILES [114] , SELFIES [58] ), 2D-graphs by using graph neural networks (GNNs), and 3D-representations (using the TorchMD-net architecture [80] ). We added a randomly initialized baseline GNN model that was not trained on any dataset.Datasets. To evaluate the information sufficiency between embedders, we compared the models on the ZINC 250k dataset [50] , designed to gather compounds that could be relevant to a wide range of therapeutic projects. This dataset contains 250k commercially available compounds meant to be used in diverse therapeutic projects.Downstream tasks. We evaluated the embedders on 31 downstream tasks extracted from the Therapeutic Data Commons [49] platform. This section focuses on ADMET tasks (Absorption, Distribution, Metabolism, Excretion, and Toxicity). Results on Drug-Target interaction tasks can be found in Sec. D.4. Datasets collected are split into a training, validation, and test set, following the scaffold-split strategy, further described in see Sec. D.3.Global results. The I S score ranking is consistent with the results of the embedders on the ADMET downstream tasks, achieving a Spearman correlation of 0.95 and a Kendall-tau coefficient of 0.80, as reported in Figure 3d . Detailed results for each of the 31 tasks are available in Sec. D.3 in Tab. 6. Table 3b shows the correlation between the I S score rankings and the performances obtained on the ADMET tasks within each category. High correlations are achieved within most task categories, especially when large tasks are available (containing an important number of molecules). On excretion tasks, the correlation is lower (below 0.8), which can be explained by the fact that these tasks are the most challenging regression tasks available, where the fine-tuned models reach the lowest R 2 scores between 0 and 0.2 (see Sec. D.3).Most / Least promising models. We observe in Figure 5b that the most promising models are the (Chem)Bert-MTR models[3]7 and MolR [112] , the former trained on SMILES representations to predict a variety of computationally available molecular properties, and the latter trained on 2D graphs to preserve equivalence of molecules w.r.t chemical reactions. Surprisingly, these models share high predictive mutual information (being assigned to the same Louvain community in Figure 5a ), (a) suggesting that they capture similar information despite significant differences in their training methods. These models also appear to be the most competitive on the ADMET tasks. On the other hand, and consistently with Sun et al. [99] 's observation, training methods for 2D-GNNs such as following attribute masking and context prediction objective are deemed as the least informative according to the I S score. This is explained by the simplicity of these pretraining objectives for this data modality. These methods are also among the least competitive methods on the ADMET downstream tasks.10 20 ADMET rank ContextPred GPT-GNN MoleOOD_SAGE BertMLM-5M GPT-4.7M InfoGraph MolBert GPT-19M GraphMVP 3D-denoising 3D-Infomax GPT-1.2B BertMTR-5M BertMTR-77M AttributeMask MoleOOD_GIN BertMLM-77M MoleOOD_GCN Not-trained BertMLM-10M GROVER GraphCL GraphLog FRAD_QM9 MolR_gcn MolR_gat BertMTR-10M MolR_tag (b)NLP-inspired models. (Chem)Bert-MLM [3] , MolBert [36] and (Chem)GPT [40] leverage masked language model objective applied to string representations (SMILES and SELFIES). Unsurprisingly, as seen in Figure 5a , these models are clustered, suggesting they capture similar information. However, they fail to simulate other models in the pool, resulting in low I S scores, a result consistent with the known limitations of these pretraining objectives [23, 105] . A noticeable exception is (Chem)GPT-1.2B (the biggest model of the pool by far), which displays a significantly higher I S score."Not-trained" GNN. Figure 5b helps visualize the performances of the different models relative to our baseline "Not-trained" GNN. Surprisingly, some models are ranked less promising than this baseline by the I S score. However, all of these less promising models obtain poorer performances on the downstream tasks. Similarly, except for InfoGraph [98] , every model ranked more promising than the "Not-trained" GNN baseline and obtained better results on ADMET tasks. This surprising result validates evaluation of the I S score w.r.t this baseline. 
limitations and conclusions: 
 We proposed a principled approach to embedding model evaluation by framing model ranking as a variation of comparing statistical experiments. Utilizing concepts of sufficiency, informativeness, and deficiency, we developed mathematically grounded metrics for pairwise comparisons between embedders without relying on labeled data in downstream tasks. Our tractable relaxation, termed information sufficiency, demonstrated strong correlations with rankings based on downstream task performance in extensive experiments. Although successful, our method still has at least two primary limitations. First, its effectiveness depends on the number and diversity of available embedders (see Sec. E.4). Future work could explore using randomly initialized embedders (random projections) instead of pre-trained ones. Second, we can enhance our proxy for predicting the deficiency between models by exploring better methods (e.g., estimating the f -divergence) to directly learn the Markov kernel that minimizes the total variation distance, which we leave for future research. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide the citations and direct references of all the datasets and models used in our work. To ensure reproducibility, all the datasets and models used are publicly available and under open licenses.Guidelines: 
 
 Appendix: We consider all alphabets to be standard Borel [28] (i.e., isomorphic to a Borel subspace of a Polish space), encompassing virtually all practical scenarios. Each such space Y is equipped with its Borel σ-algebra B(Y). The set of all probability measures on Y is denoted by P(Y). The total variation distance between P and Q in P(Y) is defined asEQUATIONand the Kullback-Leibler divergence is defined byKL P ∥Q =    X log dP dQ dP if P ≪ Q +∞ otherwise.(EQUATIONGiven a joint probability measure P XY in P(X×Y) induced by two random variables X ∈ X and Y ∈ Y with product measures P X P Y , the Mutual Information is defined as I(X; Y ) = KL P XY ∥P X P Y .If P X ∈ P(X) is a probability measure induced by X ∈ X, the Differential Entropy is defined byEQUATIONwhere µ denotes the Lebesgue measure. Similarly, it is possible to define the conditional entropy of Y given X which is denoted by h(Y |X). The mutual information satisfies the identities [27] for further details).I(X; Y ) = h(Y ) -h(Y |X) = h(X) -h(X|Y ) and h(Y |X) ⩽ h(Y ) (seeA Markov (or transition probability) kernel between X and Y is a mappingT : B(Y) × X → [0, 1],satisfying T (•|x) ∈ P(Y) for all x ∈ X and T (B|•) being a measurable function on X for any B ∈ B(Y). The space of all such T is denoted by K(Y|X). In cases where both Y and X are finite, any K(Y|X) is represented as a stochastic matrix with elements T (y|x), (x, y) ∈ X × Y.Every T ∈ K(Y|X) induces a mapping P(X) -→ P(Y), denoted by T , mapping any P ∈ P(X) toEQUATIONEQUATIONWe denote the composition of Markov kernels by juxtaposition: for M ∈ K(Z|Y) and T ∈ K(Y|X), their composition M •T ∈ T (Z|X) is defined byEQUATIONWe define the average of the total variation distance between two Markov kernels T, T ′ ∈ K(Y|X) as follows:EQUATIONA statistical model is a triple EQUATIONProposition (Relationships of sufficiency and information). The following relationships hold:(i) Sufficiency ⇒ informativeness. If the embedding model P U |X is sufficient for the embedding model P V |X , i.e. U ≽ S V , then U ≽ I V . However, Informativeness ⇏ sufficiency.(ii) Informativeness ⇒ higher capacity to distinguish concepts. If the embedding model P U |X is more informative than embedding model P Z|X , i.e. U ≽ I V , thenKL P U |Y (•|y 0 )∥P U |Y (•|y 1 ) ⩾ KL P V |Y (•|y 0 )∥P V |Y (•|y 1 ) ,for any pair of concepts (y 0 , y 1 ) ∈ Y × Y and all probability distributions P Y X .Proof. It is immediate to check that the data-processing inequality and the Markov chain Y ↔ X ↔ U ↔ V implies the relation in claim (i). On the other hand, the non-equivalence is proved by means of an explicit counterexample [57] . Given any 0 < p < 1/2 with p = 1p. For some ϵ, δ > 0, consider two discrete embedding models defined by the following matrices:EQUATIONBy taking ϵ, δ > 0 small enough, both P U |X and P V |X are stochastic matrices. It follows that P V |X is not a degraded version of P U |X but provided that ϵ, δ are sufficient small, the embedding model P U |X is more informative than P V |X , which proves the claim.In order to show (ii), let P U |Y and P V |Y be the corresponding probability measures induced by P X|U (•|u) via the embedding models:EQUATIONandEQUATIONfor any y ∈ Y = {y 0 , y 1 }. For a 0 ⩽ λ ⩽ 1, let P X|Y (•|y 0 ) ∈ P(X) and P X|Y (•|y 1 ) ∈ P(X) be two arbitrary probability measures on X. Let P X|Y (X|y) be defined byP X|Y (X|y) = 1[y = y 0 ]P X|Y (X|y 0 ) + 1[y = y 1 ]P X|Y (X|y 1 ).By replacing it into equations ( 9) and ( 10), we obtainEQUATIONEQUATIONand let P Y (y 0 ) = λ and P Y (y 1 ) = 1λ. The above probability measures correspond to a quadruple of random variables:(Y λ , X λ , U λ , V λ ) ∈ P(Y × X × U × V).Consider the function f (λ) defined byf (λ) = I(Y λ ; U λ ) -I(Y λ ; V λ ).It is not difficult to check that f (λ) ⩾ 0 for all 0 ⩽ λ ⩽ 1, and f (0) = 0 which requires that f ′ (0) ⩾ 0. By taking the differentiation, we obtainf ′ (0) = KL P U |Y (•|y 0 )∥P U |Y (•|y 1 ) -KL P V |Y (•|y 0 )∥P V |Y (•|y 1 ) ⩾ 0,which implies the claim (ii). This concludes the proof.Notice that the KL divergence between the induced distributions of the resulting embedding is not less for the embedding model U than Z. Indeed, consider the case of binary classification Y = {y 0 , y 1 } with uniformly distributed concepts. Pinsker's inequality [108] together with claim (iii) implyKL P U |Y (•|y 0 )∥P U |Y (•|y 1 ) ⩾ 2∥P V |Y (•|y 0 ) -P V |Y (•|y 1 )∥ 2 TV .From which, it is easy to verify that the accuracy of the expected Bayes accuracy of the optimal classifier based on V is upper bounded by [108, Lemma 2.1]:sup ψ Pr(ψ(V ) = Y ) ⩽ 1 - 1 2 exp -KL P U |Y (•|y 0 )∥P U |Y (•|y 1 ) ,where the exponent in the upper bound is subject to the discriminating capacity through the KL divergence of the embedding model U on U.We begin with the proof of Proposition 2.Proof. Clearly, the assumption (i) implies the statement (ii) by Data-Processing. Conversely, let us assume point (ii) holds. This means that, for every probbaility distribution P X and all conditional probability distributions P Y |X , there exists ρ U : U → P(Y) such thatEQUATIONwhere ρ V : V → P(Y) is a (possibly randomized) inference procedure is transition probabilities, which the learner can optimize to maximize the guessing probability.Let the decision rule ρ V (y|v) = 1[v ∈ A y ] for any partition {A y } y∈Y of V with A y ∈ B(V). Then, for any P Y |X , expression (13) implies the existence there exists ρ U (y|u) such thatEQUATIONEQUATIONHowever, we can rewrite the last expression as:EQUATIONBy applying the minimax theorem [87] , it is possible to exchange the order of the inf and the sup, which yields:EQUATIONWe observe that y∈Y Γ (y, x), ρ U = 0,for each x ∈ X and thus, maxEQUATIONwhere the equality holds if and only if Γ (y, x), ρ U = 0 for all y ∈ Y, for each x ∈ X, since by contradiction otherwise y∈Y Γ (y, x), ρ U < 0.Therefore,EQUATIONwhich means the maximum is achieved by degenerate random variables Y = f (X) achieving the maximum for each x ∈ X. Consequently, we have thatEQUATIONhence infρ U E∥P V |X -ρ U •P U |X ∥ TV = 0,and so the existence of the transition probability ρ U such that P U |X is sufficient for P V |X . This concludes the proof of the Proposition 2.We now show the proof of Corollary 1.Proof. Continuing from the proof of Proposition 2, which remains unchanged, until one demonstrates that for ε > 0, infEQUATIONTo proceed from this point, let us now examine the following quantity:y∈Y |Γ (y, x), ρ U |.The above quantity is the induced ℓ 1 -norm distance between Ay P V |X (dv|x) and U ρ U (y|u)P U |X (du|x). Since, for all x ∈ X, Γ (y, x), ρ U , for all x ∈ X which implies that, for the strategy ρ U achieving the left-hand side of Eq. ( 24),sup {Ay} E y∈Y |Γ (y, X), ρ U | ⩽ 2|Y| sup {Ay} E max y∈Y Γ (y, X), ρ U ⩽ 2|Y|ε, for all x ∈ X Hence, sup {Ay} E y∈Y Ay P V |X (dv|X) - U ρ U (y|u)P U |X (du|X) ⩽ 2|Y|ε.This concludes the proof of the Corollary 1.Example 1 (Statistical experiments with Gaussian embedding models). Let U |x and V |x be independently normally distributed as N (x, σ 2 ) and N (x, ϵ 2 σ 2 ) , respectively, with 0 < ϵ < 1.• Case of σ 2 = σ 2 0 known. Here U ≽ S V since V + ν|x has the same distribution as U |x when ν ∼ N (0, (1ϵ 2 )σ 2 0 ). That is V is strictly more informative than U . However, U is strictly more informative than V .• Case of x = 0 known. One can observe that U ≈ V since the variables V /ϵ have the same distribution as the U , and the variables ϵU have the same distribution as the V .• Case of x and σ unknown. Surprisingly, in this case U and V are not comparable.Embedding Models [13, 63] Proposition 3 (Sufficiency and risks of a given task on embedding models [13, 63] ). An embedding model P U |Y ∈ K(U|Y) is deemed to be sufficient for another one P V |Y ∈ K(V|Y) if and only if, for any bounded loss function ℓ where ∥ℓ∥ ∞ ⩽ 1, and for any inference procedure ρ V : V → Y, there exists a inference procedure (possibly randomized) ρ U : U → P(Y) such that the resulting statistical risks satisfy R y (P U |Y , ρ U , ℓ) ⩽ R y (P V |Y , ρ V , ℓ), for all y ∈ Y.(25) Here we denote by R y (P U |Y , ρ Y , ℓ) and R y (P V |Y , ρ V , ℓ) the statistical risks for the corresponding inference frameworks, respectively. Remark 5. The restriction ∥ℓ∥ ∞ ⩽ 1 is irrelevant here. However, we opt for simplicity and limit our focus to situations where one encounters dominated statistical models with Polish sample spaces. In essence, various extensions do not significantly alter the conceptual aspects of the underlying statistical problem (see [103] for further details). Rather, they primarily reflect the complexity of its measure-theoretic formulation.In 1964, Le Cam [63] clarified the relationship between the sufficiency of an embedding model on a given task and its expected risk on this task. The following theorem provides a formal statement of this relationship. Theorem 1 (Le Cam [63] ). Let ε > 0 be fixed. Then, δ(P U |Y → P V |Y ) < ε if and only if, for any bounded loss function ℓ where ∥ℓ∥ ∞ ⩽ 1, and for any inference procedure ρ V using the embedding model P V |Y , there exists a inference procedure (possibly randomized) ρ U based the embedding modelP U |Y such that the risks satisfy R y (P U |Y , ρ U , ℓ) -ε ⩽ R y (P V |Y , ρ V , ℓ), for all y ∈ Y.In this section, we provide all the necessary experimental details to reproduce the experiments in NLP. For the I S score estimation, please see Appendix E. First, we detail the models and datasets used in the experiments. We provide the training details of the downstream tasks, and finally, we present the comprehensive results of the NLP experiments.In Tab. 1, we provide the metadata of the models used in the NLP experiments and their scores on the MTEB benchmark when they exist. We provide in Tab. 2 the statistics of the datasets used to evaluate the I S score.All the downstream tasks are trained in the exact same way. We use a dense classifier with two hidden layers of dimension 256 and train for two epochs using ADAM [56] with a learning rate of 10 -3 , on the official training set and evaluated on either the validation or test set when they are available (with respect to the Huggingface datasets). We do not perform early stopping or selection using the validation set.We provide in this section the unaggregated results for the main NLP experiments presented in the main text, then we provide numerous ablation studies and additional results to address different aspects of the I S score of the embeddings in NLP.The strength of the MTEB benchmark is that it evaluates embedders on a very large and diverse set of downstream tasks. We provide an Tab. 4 and Figure 6 the full results of the MTEB benchmark (English) for the models used in the NLP experiments.Table 1 : Metadata of the evaluated models and their information sufficiency. We obtain significant positive correlations in all categories of downstream tasks. We noticed that we obtained significantly poorer results on STS, Clustering, and Reranking tasks than on classification tasks. We believe this behavior is due to the nature of these tasks. Indeed, they do not rely on training an additional model on top of the embeddings but rather directly use the embeddings as is in dot products or similarity measures. An embedder could produce very informative embeddings, i.e., it is possible to extract the useful information using a small model, and at the same time, these embeddings not be adequate for dot-product-based similarity measures. We believe further investigation is needed to understand the behavior of the models on these tasks. Especially to see if training a small model on top of the embeddings can improve the performance of these tasks.Many factors can impact the estimation of the I S score of the models, such as the dimensions of the different embeddings and the number of available embedders to evaluate. the I S score can capture many different aspects of the embeddings, such as the quality of the embeddings. We provide in this section a comprehensive set of ablation studies to evaluate the impact of these different factors on the I S score of the embeddings. Impact of instruction finetuning. Instruction finetuning is now a common practice to improve the alignment of the base of models and expand the models' reasoning capabilities. In Figure 4c , show that instruction fine-tuning positively impacts the models' performance on the downstream tasks and that the I S score captures this improvement. In addition to studying the impact of instruction finetuning, we evaluated models at different checkpoints during their initial pretraining in Sec. C.3.3 using the CroissantLLM checkpoints [37] .Surprisingly, we found that the number of training steps does not significantly impact the models' performance on the downstream tasks nor on the I S score of the embeddings. The I S score correctly captures this behavior as shown in Figure 7 . We hypothesize that the I S score in terms of embeddings is, in this case, determined by a few numbers of training steps (the first 5000) and the overall architecture of the model. Training the model further even leads to a decrease in performance on the downstream tasks, which is not captured by the I S score of the embeddings; this could be due to the very small variation in the performance.We found that considering the amount of information packed by an embedding per coordinate is crucial to obtain a good ranking of the models. In Figure 16b , we show the correlation between the performance of the models on the MTEB benchmark and their I S score, not normalized by embedding size. While positive significative correlation is still present, the correlation is much weaker than when the dimension of the embeddings normalizes the information sufficiency. We postulate that models clustered together by information sufficiency are likely to behave similarly on the downstream tasks. We evaluate this hypothesis by grouping the models by clusters discovered using the information sufficiency and reporting their performance on the downstream tasks. In Figure 8b and Figure 9 , we observe that models within the same cluster tend to have similar behaviors on the downstream tasks.The I S score is evaluated with a fixed dataset supposed to represent the data distribution of interest (either a very diverse set or a subset following a distribution specific to a subfield like medical or legal texts). We cross-evaluated the I S of the models on different datasets and the performance of the models on the downstream tasks in Figure 10 . We find that closer datasets in terms of the data distribution lead to a higher correlation between the I S score of the models and their performance on the downstream tasks. It is especially highlighted when comparing the correlations we get when evaluating I S on the AG News and Amazon polarity datasets. The first one corresponds to news articles, and the task is to guess the topic, whereas Amazon Polarity corresponds to product reviews, which is a sentiment analysis task. We find that the I S score evaluated on Amazon Polarity tends to yield way better correlation with the performance on the sentiment analysis downstream tasks GEOM [6] (50k) ContextPred [99] ✓ GIN 300 GEOM [6] (50k) GPT-GNN [48] ✓ GIN 300 GEOM [6] (50k) InfoGraph [98] ✓ GIN 300 GEOM [6] (50k) GraphCL [119] ✓ GIN 300 GEOM [6] (50k) GROVER [89] ✓ GIN 300 GEOM [6] (50k) GraphLog [116] ✓ GIN 300 GEOM [6] (50k) GraphMVP [67] 1 ✓ GIN 300 GEOM [6] (50k) 3D-infomax [96] 1 ✓ PNA 800 QMugs [51] (620k) MolR [112] ✓ GCN, GAT, TAG 1024 USPTO [112] ( 1.5M) MoleOOD [117] ✓ GIN , SAGE, GCN 256 BACE [47] (400k)ChemBERT MLM [3] ✓ RoBERTa 600 PubChem [55] (5M, 10M, 77M) ChemBERT MTR [3] ✓ RoBERTa 384 PubChem [55] (5M, 10M, 77M) MolBert [36] ✓ BERT 256 GuacaMol [18] (1.6M) ChemGPT [40] ✓ GPT 128, 256, 2048 PubChem [55] (10M)✓ TorchMD-net 256 PCQM4Mv2 [46] (3.7M) 3D-fractional [39] ✓ TorchMD-net 256 PCQM4Mv2 [46] (3.7M)Table 5 : Models evaluated on the ZINC dataset.We considered 28 models for the molecular experiments, summed up in Tab. 5. Some models were used in different versions (architectures, number of parameters, pretraining dataset's size), such as the ChemBert models, followed by the size of their datasets, or ChemGPT, followed by their number of parameters.Most 2D-GNNs were trained on the GEOM [6] dataset and were gathered from the repository of GraphMVP [67] model. Note that the MoleOOD [117] model was trained on the BACE [47] dataset, with a supervised task specific to the β-secretase enzyme. As a result, this model can be seen as "already specialized", explaining its poor performance in our evaluation.Ability to predict denoising 3D modelsBertMLM-77M ContextPred BertMLM-5M GPT-GNN GPT-4.7M MoleOOD_GIN AttributeMask MoleOOD_SAGE BertMLM-10M MolBert MoleOOD_GCN GPT-19M GROVER InfoGraph Not-trained GraphMVP GraphCL 3D-Infomax GraphLog GPT-1.2B BertMTR-5M BertMTR-10M BertMTR-77M MolR_gcn MolR_gat MolR_tagAbility to be predicted by denoising 3D modelsGPT-1.2B ContextPred GPT-4.7M GPT-19M BertMLM-10M GPT-GNN BertMLM-5M MolBert BertMLM-77M BertMTR-77M BertMTR-10M MoleOOD_GIN BertMTR-5M MoleOOD_SAGE Not-trained InfoGraph MoleOOD_GCN MolR_tag MolR_gat MolR_gcn GROVER 3D-Infomax AttributeMask GraphCL GraphLog GraphMVP (a) 0 1 S (Z U) ContextPred AttributeMask GPT-GNN BertMLM-77M MoleOOD_GIN MoleOOD_SAGE BertMLM-5M MoleOOD_GCN GPT-4.7M Not-trained InfoGraph BertMLM-10M MolBert GROVER GPT-19M GraphCL GraphMVP GraphLog 3D-denoising FRAD_QM9 3D-Infomax GPT-1.2B MolR_gcn MolR_gat BertMTR-5M BertMTR-10M BertMTR-77M MolR_tag Models Z 0 1 S (U Z) (b)Figure 11 : (a) Information sufficiency of embedders over 3D-Denoising models (left) and of 3D-Denoising models over the other embedders (right). (b) Information sufficiency of embedders in both directions. We see the 3D denoising models are among the least predicted models. We used the RD-Kit and Datamol tool-kits [61, 71] to pre-process the molecules and to generate three-dimensional conformers for 3D-models. To run the models using 3D views of the molecules, we generated five conformers (possible 3D configuration of the molecule) for each SMILES and kept the conformer with the lowest energy. Note that this methodology is imperfect, as the 3D coordinates might be noisy; however, we followed the same procedure to pre-process the ZINC dataset, to evaluate the information sufficiency, and on the datasets corresponding to each downstream task.Finally we considered a variety of models architecture for 2D-GNNs notably graph isomorphism network (GIN) [115] , principal neighbor aggregation networks (PNA) [26] , graph convolutional network (GCN) [33] , graph attention network (GAT) [110] , topology adaptive graph convolutional networks (TAG) [32] and GraphSAGE [44] . For SMILES-based models, backbones are inspired by BERT [31] , RoBERTa [69] , and GPT [40] . Finally, both our 3D models use TorchMD-net [80] as a backbone.3D models. The two 3D models considered (FRAD [39] and Denoising [121] ) obtain high I S scores while being among the least predictable (Figure 11b and Figure 12 ). This suggests that these models capture 3D-specific features inaccessible from other modalities while maintaining sufficient overlap to predict them.2D-3D models Some 2D-GNNs we considered (GraphMVP and 3D-infomax) are trained to maximize the mutual information between their embeddings and 3D representations of the molecule. Hence, we expect these models to be related to the 3D-denoising models we considered. However, we observe in Figure 11a that these models do not achieve particularly high information sufficiency scores over 3D-denoising models. On the other hand, the 3D models achieve high information sufficiency scores over them, which might suggest that these 2D models and 3D-denoising models share information that is easier to access from the 3D models. However, we want to point out that GraphMVP and 3D-infomax are both among the most predicted models; that is to say, among the other models in our pool, they achieve the highest information sufficiency scores. The datasets chosen for the molecular experiments are extracted from the Therapeutics Data Commons (TDC) [49] platform. We focused our experiments on ADMET tasks, crucial for drug discovery and development, which results in a total of 31 tasks, described in Tab. 6. A grid search is performed on each dataset individually to maximize the average AUROC or R 2 score across all models for binary classification and regression. We chose a maximum number of epochs depending on the task size to ensure all models have time to converge, limiting this amount to grow to at most 200 epochs.Tab. 6 also displays the variation of the correlation coefficient between the ranking obtained on the I S score and the performances obtained on the downstream tasks regarding Spearman and Kendall correlations. We can see that the I S score correlates well with the performance of downstream tasks when the amount of data available is large.Finally, we can see in Figure 13 that by grouping models based on their performances on these tasks, we obtain a similar clustering to the one obtained on the I S score in Sec. 6. [49] considered in our experiments. We report the correlation between the informativeness score and the performances of the embedders on the downstream tasks in terms of Pearson correlation ϱ p , Spearman correlation ϱ s and Kendall-Tau τ . We also report the average metric of the models on each task across the grid search runs, in terms of R 2 for regression tasks and AUROC for classification tasks. The tasks are ordered within each category by the correlation with the informativness score (in terms of Spearman correlation). We propose further evaluating the embedders on yet another type of downstream task: Drug-Target Interaction. This task aims to predict the binding affinity between a given pair (drug, target). Since none of our models can process protein sequences, we decompose each dataset into multiple regression tasks on a single target by querying all molecules associated with a label for this target. Each task is then formulated as a set of molecules: X = {x i } i∈{0,...,N } , and their labels Y = {y i } i∈{0,...,N } However, such tasks can be small, making it hard to build proper models from the embeddings. In contrast, the number of tasks is very large, making it computationally expensive to proceed to an adequate hyperparameter selection. To bypass these limitations, we propose to estimate the embedded space's clustering quality for each model by measuring how close the labels of a molecule are compared to its nearest neighbors for each task. In other words, we measure:EQUATIONand N i,nneigh (X ) is the set containing the n neigh closest neighbors of x i ∈ X , which would be the performances of a K-nearest neighbors regressor on the task when using one data sample.This quantity can be interpreted as a proxy of the embedding's capability to perform a similarity search, a classic chemo-informatic method using the similarity between different molecular projections to perform predictions. This training-free and computationally inexpensive approach allows us to evaluate the models on many tasks/targets.We focused on 4 DTI datasets: KIBA [101] , BindingDB-Kd, BindingDB-Ki, and BindingDB-IC50 [68] , with a total of 1496 tasks. We removed all tasks containing less than 128 molecules to ensure minimum data for the clustering evaluation. We obtain similar results as in Sec. 6.1, our metric correlating with the performances on the different tasks considered. Figure 14 sums up all results by establishing a ranking across models and the number of neighbors, where we can see that MolR and ChemBerta-MTR appear as both the most promising models according to their I S score, and the best models evaluated. Furthermore, the outliers observed inSec. 6.1 show different behaviors in this setting. For instance, while 3D-Infomax seemed under-estimated and InfoGraph over-estimated by the I S score after seeing the results on the ADMET tasks, Infograph appears under-estimated in this setup, and 3D-infomax over-estimated.As stated in Sec. 2, the deficiency δ(P U |X → P Z|X ) is an intractable object measuring the cost of the reconstruction of Z from U . Due to this intractability, we propose to estimate the information sufficiency I S (U → Z), which is a tractable proxy for the deficiency.We recall the definition of the information sufficiency:I S (U → Z) ≜ inf f ∈FΘ(Z) E [-log f (Z)] Uncertainty of Z -E inf M ∈KΘ(Z|U) E [-log M (Z|U )|U ]Uncertainty when simulating Z from U with M .We denote C the number of modes chosen for the Gaussian mixture distributions and GM µ,Σ,w the Gaussian mixture distribution with C components, parametrized by µ, Σ, w, with 1 T w = 1, such that We evaluate the strength or I S score of an embedder with respect to all the others by relaxing the "for all" conditions with the median I S score. Thus, the number of available embedders might impact the performance of our method. Indeed, if too few embedders are available, it is likely that our evaluation would be biased by favoring models similar to the few available ones. We evaluate the impact of the number of available models by sampling subsets of our global model pool to compute the I S score.In Figure 22 , we found that when fewer models are available, the rankings obtained using the I SEvaluating the I S score of the models is computationally inexpensive. Evaluating the I S score requires only a single (small) GPU. All our experiments were conducted on NVIDIA V100 and NVIDIA A6000 GPUs.Our method's main (computational) shortcoming stems from the need to compute the information sufficiency between all pairs of models. This is a quadratic operation in the number of models. However, in practice, optimizing and estimating the information sufficiency presented in Sec. 3 is cheap. The complete evaluation of the 45 NLP models can be done in less than 6 hours on a single GPU.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.The code used to perform all experiments is available at https://github.com/ills-montreal/emir 38th Conference on Neural Information Processing Systems (NeurIPS2024).If U ≽S, V , then δ(P V |X → P U |X ) = 0, while if U ̸ ≽S V , then δ(P V |X → P U |X ) > 0.For the experiments in molecular modeling, in each subset regression and classification tasks are mixed. Hence, we do not compute the Pearson correlation to avoid mixing scores obtained for different metrics.https://huggingface.co/infgrad/stella-base-en-v2https://huggingface.co/spaces/mteb/leaderboardWe rely on the Louvain community detection implementation from networkx[43]BertMTR-XM stands for a (Chem)Bert-MTR model trained on XM molecules.In practice since this operation does not depend on U , we store the weights of this distribution, and use it for all pairs IS(. → Z). 


Decision: This paper introduces a novel, information-theoretic framework for assessing the relevance of embedding models. The authors introduce the notion of sufficiency of a model A relatively to a model B (used to rank embedders) and tie this to a related notion of informativeness (information efficiency). 

Overall, the proposed approach is quite interesting & novel and a refreshing (theoretical) way to quantify the effectiveness of different embedding models. The authors supplement the theory with good analysis & combine it with experimental results to validate the efficacy of the proposed metric. They also did a very nice job of addressing comments/concerns during the rebuttal phase. All the reviewers agree that it’s a solid paper, no concerns.

Reviews: review: paper_summary: paper_summary: This paper proposes a new metric to compare embedding models without relying on labeled data. The approach involves embedding data using two separate neural networks Z and U, and then trying to use embedding model U to simulate/match the output of embedding Z. They then use this to calculate an information sufficiency criterion. They motivate this with theory, and show that their information sufficiency metric aligns nicely with established embedding benchmarks across two separate domains, NLP and molecular biology.
strength_weakness: strength_weakness: 1. This paper puts a heavy emphasis on the theoretical motivation without giving enough detail about the actual algorithm for calculating the information sufficiency metric I_s. Line 211 states “We hence attempt to simulate Z from U by learning a Markov kernel M via a mixture of multivariate Gaussians, and measure the uncertainty reduction it induces.” It would be helpful to expand on this in detail. The reader is left only guessing how exactly this was done, and in my opinion it does not seem quite replicable. The details in the Estimation method section E.1 and Hyperparameter selection section E.2 are somewhat minimal.

2. This method is touted as a principled way to compare embedding models without using data labels (but still using data in the desired domain). However it is unclear what the other tradeoffs are. Aside from the question of data labels, Is this approach more computationally efficient?
questions: questions: 1. How exactly do you calculate the information sufficiency metric I_s for a given dataset in MTEB and two models? Can you expand on this in more detail (as mentioned above)?

2. In the discussion section, you mention that there are other potentially promising methods for learning the Markov kernel. What do you think some other promising approaches might be?

3. In Section C.3.1, the authors admit that their results are better for MTEB classification tasks when compared to STS, clustering, and reranking tasks. What numbers/tables are they referring to here?
limitations: limitations: It would be helpful for the authors to share code upon publication so that others can use this metric/approach.

### Minor comments:

Typo Figure 3(b) “Molecular Modelling”
Typo line 213 “embbeders”
paper_summary: paper_summary: Evaluating embedding models is challenging because it typically relies on various downstream task data, despite the embedding models being trained for general purposes. This paper introduces an information-theoretic metric for comparing embedding models, eliminating the need for labeled datasets in their evaluation. The core idea is that if one embedding model can simulate another in most cases, it indicates a higher capacity for distinguishing concepts by its embeddings. Empirical results in both text and molecule embedding models show that the proposed metric closely aligns with downstream task rankings. Additionally, the community analysis facilitated by this metric reveals clusters of embedding models, illustrating their relationships.
strength_weakness: strength_weakness: - The connection between the concept of deficiency and information sufficiency is somewhat unclear.
- The computation of information sufficiency in practice is not well-explained. Adding more details would be beneficial.
- (Minor) The arrangement of tables and figures could be improved.
questions: questions: - Does transitivity (or a similar weaker concept) hold for information sufficiency? For example, if $I_S(U → Z) > I_S(W → Z)$ and $I_S(W → Z) > I_S(X → Z),$ does $I_S(U → Z) > I_S(X → Z)$?
- Are there specific reasons for using multivariate Gaussians in Markov kernel learning? Are there any alternatives?
- What are the requirements for data in estimating information sufficiency? How do the quantity, quality, and diversity of datasets affect the effectiveness of the proposed metric?
- Line 114: Should $P_{U|X}$ be $P_{Z|X}$?
- Lines 188-189: Isn't this bound practically vacuous due to the size of X?
limitations: limitations: The authors adequately address the limitations.
paper_summary: paper_summary: This article introduces a novel framework, grounded in information theory, for assessing the relevance of embedding models (*embedders*). The authors begin by introducing the notion of *sufficiency* of a model A relatively to a model B, which can be used to rank embedders. They prove:
1- 	sufficiency implies (i) a higher capacity to distinguish concepts (Prop. 1)
2- 	sufficiency is equivalent to (ii) having a lower expected Bayes risk (Prop. 2)
They reasonably assume that (i) and (ii) is equivalent to “being a good embedder”.
The authors then introduce the *deficiency* between two models, a relaxed version of sufficiency. Theoretical bounds on the Bayes Risk, as a function of the deficiency, are given (Corr. 1). As deficiency is hard to compute, they develop a surrogate estimator termed *information efficiency* (Definition 3), which can be empirically estimated.

The authors perform experiments to validate their theoretical findings using datasets from NLP and molecular modelling. In the NLP domain, they employ datasets from the MTEB benchmark to demonstrate the correlation between their new ranking scores and model performance. For molecular modelling, they use the ADMET dataset. The authors also propose a graph visualization of the models build from the pairwise information efficiency values.
strength_weakness: strength_weakness: - Several key points of the method are not explicit:
    - While the main metric is well described, there is little information on how it is estimated in practice. Besides the fact the KNIFE estimator from [82] is used to quantify mutual information, no procedure is described and the reader is left to try to rebuild it. 
    - How to choose reference data is not elaborated on, besides L229-230.  
    - The choice the median, discussion about number of models, or embedding dimensions are not discussed within the core of the article while it would help a lot to grasp intuitions about how the proposed metric behaves.
- On motivation and applicability: 
     - It would be interesting for the authors to expand on the practical utility of such a method. While it seems to be an interesting alternative to benchmarks which does not require any labelled data, I am not sure of how interesting it could be for applications, especially considering the paper does not expand on how difficult and computationally costly the estimation procedure is.
questions: questions: - You mention (L211-212) that “We hence attempt to simulate $Z$ from $U$ by learning a Markov kernel”: do you need to learn a Markov Kernel for each model ? Is that computationally expensive ?
- While framing the comparison between embedding models this way is new, it would be interesting to have . However, it would be interesting to have a baseline, even the naivest one (e.g. the number of parameters maybe?) to be able to compare your results.
limitations: limitations: Limitations have been adequately addressed.
paper_summary: paper_summary: The authors propose a task-agnostic method for the evaluation of embedding models called "information sufficiency". The general notion is to generate a pairwise matrix that effectively measures how well each embedding model can be used to generate the information content of the other. They compute an overall "information sufficiency score" for each model by computing the median of this pairwise score along one axis. They demonstrate that their metric correlates well with overall downstream task performance.
strength_weakness: strength_weakness: My primary concern centers on the potential pitfalls from using this method, described in more detail in the questions below. I am very open to raising my current score of 6 if those concerns can be addressed sufficiently.
questions: questions: The metric proposed here is a pairwise metric where the embedding model that can most effectively simulate other embedding models is ranked as the most effective. I wonder if it is possible that if a particular embedding model contains unique information that is not represented by any other embedding model in the chosen set, then is it possible that said embedding model would be ranked poorly? The thrust of my concern is that the adoption of this metric would favor embeddings that are more "central" even in edge cases where more niche information might be more useful. 

As a follow up to the last question, I am wondering in what sense the question of whether one embedding model is better than another is generally useful. For most applications and downstream tasks, one does not want an embedding model that is better on average for a host of applications, they want one that is better for their proscribed use case, whatever that may be. Of course, there is the challenge that it is impractical to "try every model", given the breadth of available models. Is there some middle ground here, where a similar method can be used to narrow the number of models that are tested rather than to recommend a single one?  

The authors acknowledge that the usefulness of their methods depends on sampling a large number of diverse embedders with which to compare. To what degree is this circular - does this constraint require that you evaluate the set of embedders using a more traditional downstream benchmarking procedure?

The underlying method has some real similarity to the task affinity matrix from the "Vision Taskonomy" work by Zamir et al. (which is currently not cited), although there the idea is not so much to better compare embedding models as it is to understand the informatic structure of tasks in vision literature. Is there perhaps a way of using this pairwise similarity matrix to embed the differences in informatic content across these embedding models using this related approach?


Minor aesthetic/typographical comments: 

There is the floating phrase "inforatmation sufficiency" on page 26 in the supplement. 

Holy Boldface Batman. That's a lot of boldface. Especially the sentences that start as normal and lead into boldface.
limitations: limitations: Please see my questions. No obvious negative societal impacts of the work.

Comments: title: Raising my score
Further details about the deficiency and the mutual information/information sufficiency
comment: I would like to thank the authors for their rebuttal. I find it (along with rebuttals to other reviewers) has clarified things for me. Given this, and the proposed changes, I have updated my rating and recommend acceptance.
Dear Reviewers,

We would like to sincerely thank all the reviewers for their diligent work and the efforts they put into reviewing this paper. We understand that the review and discussion process is highly time-consuming, and we sincerely appreciate Reviewers o2cb, 4AaU, and UcaH for engaging with us.

We also hope our responses alleviated the concerns of Reviewer id3c, and we remain available for any further clarifications until the end of the discussion period.

We will ensure to include all the valuable feedback we got to improve the final version of our work.
Best regards, The Authors of Paper #12245
Thank you for your detailed response! My questions and concerns are well-addressed and I actually learned a lot. I've raised my rating to 7.
Thank you to the authors for their rebuttal answers. I believe my concerns have mostly been addressed. I am raising my score to a 7 to reflect this. Regarding my mention of the Zamir work, my referral was merely prompted by the fact that the high level idea is simply very similar (essentially, building an embedding for a representation out of profile of transfer properties), and was not meant to request any particular analysis for you to pursue.
**This is a complement to our answer to Reviewer o2cb regarding the connection between our estimator and the deficiency.**

## Review of the Distortion-Rate Function

The rate-distortion (RD) function of a random variable --the source-- $X$ for a given distortion function $\ell(\cdot, \cdot)$ is defined as [Cover 2006]
$$
R_{X,\\ell}(D) \,\triangleq \inf_{\rule{0mm}{4.3mm}\substack{p_{\widehat{X}|X}:\\ \mathbf{E}[\ell(X,\widehat{X})] \leq D}} I(X;\\widehat{X}).
$$

Furthermore, we assume that there exist $D>0$ such that $R_{X,\ell}(D)$ is finite.
We denote the infimum of those $D$ by $D_{\min}$ and $R_{\max}\triangleq R_{X,\ell}(D_{\min})$ (or, more precisely, $R_{\max}\triangleq \lim_{D\rightarrow  D_{\min}+}R(D)$).
The following properties (see [Lem.~1.1, Csiszar 1974]) of the RD function will be used in what follows.

__Theorem 1:__
The RD function $R_{X,\ell}(D)$ is a non-increasing convex function of $D$ on the interval $(D_{\min}, \infty)$. 
It is monotonically decreasing on the interval $(D\_{\\min},D\_{\\max})$ and constant with $R_{X,\ell}(D)=R\_{\min}$ on $[D_{\max},\infty)$ (here $D_{\max}=\infty$ and $D_{\min}=0$ are possible). The inverse function $R_{X,\ell}^{-1}(r)$ is well defined on $(R_{\min}, R_{\max})$ and monotonically decreasing.

The inverse function $R_{X,\ell}^{-1}(r)$ is known as the distortion-rate (DR) 
function of the random variable $X$ for the given distortion function $\ell(\cdot, \cdot)$.

## Deficiency and Information Sufficiency

Assume two Markov (or transition probability) kernel between $\mathsf{U}$ and $\mathsf{Z}$ which is a mapping $M : \mathcal{B}(\mathsf{Z}) \times \mathsf{U} \rightarrow [0,1]$,  and between $\mathsf{Z}$ and $\mathsf{U}$ which is a mapping $M^\prime : \mathcal{B}(\mathsf{U}) \times \mathsf{Z}  \rightarrow  [0,1]$. From the embedder definition, we have the Markov chain  $Z \leftrightarrow X \leftrightarrow U$.
 
We begin by the deficiency  $\delta(P_{U|X} \rightarrow P_{Z|X} ) $. Let us define a suitable distortion function: 
$$
\ell(x, M) \triangleq  \\|  M \circ P_{U|X=x} - P_{Z|X=x} \\|_{\text{TV}}. 
$$
 From which, it is easy to check that 

__Equation 1__
$$
 \delta(P_{U|X} \rightarrow P_{Z|X} ) \triangleq  \inf_M \\| M \circ P_{U|X} - P\_{Z|X} \\|_{\text{TV}} \geq \inf_M\mathbf{E}[\ell(X,M)],
$$
where the last inequality follows by replacing the supremum over $x\in \mathsf{X}$ by the expectation. Using the data processing inequality and the definition of the RD function, we obtain the following bound for any Markov kernel $M$:

__Equation 2__
$$
I(Z,X;U)  \geq 
\inf_{\\rule{0mm}{4.3mm}\substack{p_{\widehat{X}|X}:\\ \mathbf{E}[\ell(X,\widehat{X})] \leq \mathbf{E}[\ell(X,M)]}}  I(X;\widehat{X})
= R(\mathbf{E}[\ell(X,M)]).
$$
For $\mathbf{E}[\ell(X,M)]\in (D_{\min},D_{\max})$, 
we can invert the RD function, and thus we obtain from it the fundamental bound $R_{X,\ell}^{-1}(I(Z,X;U))\leq \mathbf{E}[\ell(X,M)]$  or, equivalently,

__Equation 3__:
$$
 \delta(P_{U|X} \rightarrow P_{Z|X} ) \geq \inf_M \mathbf{E}[\ell(X,M)] \geq R_{X,\ell}^{-1}(I(Z,X;U)) ,
$$
which follows from inequality Eq.1.
For $\mathbf{E}[\ell(X,M)] < D_{\min}$, Equation 2 reduces to $I(Z,X;U) \geq +\infty$ which shows that in order to achieve an expected distortion below $D_{\min}$ the random variables $(Z,X,U)$ must have a joint distribution that is not absolutely continuous with respect to the product of their maginal distributions $(Z,X)$ and $U$. 
For $\mathbf{E}[\ell(X,M)] \geq D_{\max}$ we obtain the trivial bound $I(Z,X;U) \geq 0$.
**The lower bound in Equation 3 is a non-increasing function in the mutual information. This suggests that the mutual information $I(Z;U)$ should be as large as possible as a necessary condition to achieve lower deficiency.** Similarly, we can obtain that the symmetric bound: 

__Equation 4__
$$
\rule{0mm}{5mm}\rule{3mm}{0mm}
 \delta(P_{Z|X} \rightarrow P_{U|X} ) \geq \inf_{M^\prime} \mathbf{E}[\ell(X,M^\prime)] \geq R_{X,\ell}^{-1}(I(U,X;Z)). 
$$

We will incorporate these suggestions into the final version of the paper.

## References
T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New York, NY, 2nd edition, 2006. 

I. Csiszar, “On an extremum problem of information theory,” Studia Scientiarum Mathematicarum Hungarica, vol. 9, no. 1, pp. 57–71, 1974.